[postgis-devel] Union/Collect/Aggregates

[Darafei "Komяpa" Praliaskouski]

Hi,

There are two issues that I believe should be resolved in different ways:

 - Memory free on cancellation: let's check how it really works. Having 2x
memory used on GSERIALIZED (disk) -> LWGEOM (palloc) and LWGEOM (palloc) ->
GEOS (malloc) might have been non-issue when there was a hard limit of 1GB
memory usage, but 6GB vs 12GB means swap versus non-swap (hours vs days
potentially, but proper benchmark is yet to do) on many machines. I believe
there has to be a cancellation callback or a wrap-up flag of some kind.

Can GEOS live in palloc'd memory?

 - Upgrade issues: pushing all the internal functions to keep the same
signature, when their nature is different, just for the sake of backward
compatibility is not a sustainable way forward.
Thanks Sandro for providing a test case for upgrades, I'll try to tackle it
within two weeks when we have holidays in Belarus.

Another reason to not keep signature and instead work on extension install
code is that
https://lists.osgeo.org/pipermail/postgis-devel/2019-April/027887.html suggests
that parallel union may be possible, and that improvement by itself will
also require ST_Union signature alteration. I believe I've seen comments
from Martin on GEOSUnion redesign, so even an old implementation may
suddenly become feasible.

Thoughts?

 On Mon, Apr 29, 2019 at 8:52 PM Paul Ramsey <pramsey at cleverelephant.ca>
wrote:

> So, this issue I feel is still not resolved,
>
> https://github.com/postgis/postgis/pull/394
>
> We've got an open ticket now now on upgrades potentially having issues
> if they use ST_Union() in aggregates. I also have an open question
> with respect to memory leaks in the case of query cancellation.
>
> I feel like a less intrusive implementation would be to not convert
> the individual geometries to GEOS in the transfer function, but to
> collect them in an LWCOLLECTION instead, and do the work of conversion
> and processing in the final function.
>
> This would allow us to keep the shared transfn signature and avoid the
> upgrade issues, and would also keep all the memory in the PgSQL memory
> manager right up until finalfn, so cancellation during collection
> becomes a non-issue.


> I'll do this work if nobody else will, gritting my teeth as I do so,
> since it's churn for a relatively rare use case (collecting more than
> 1GB of inputs for an output that will fit in less than 1GB).
>
> P.
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel



-- 
Darafei Praliaskouski
Support me: http://patreon.com/komzpa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20190430/54775fd8/attachment.html>