[postgis-devel] Status of custom memory allocators based interruptable calls

[Chris Hodgson]

It seems like a pretty big wall to put up: "If you want to be used by 
PostGIS, you must use memory in a supported way and/or provide for 
external memory management" - especially since it is PostGIS that wants 
to use the library, not the other way around.

It seems like the goal of interruptible calls may suffer from the same 
problem, worded slightly differently: "If you want to be used by 
postgis, you must support a way to interrupt long-running processes". 
Besides just the interruptibility, which we could inject using the 
overridden new/delete approach, we also need to allow for memory 
cleanup... so it seems to further compound the problem.

How do other PostgreSQL extensions deal with this? There must be some 
XML database extension that uses an external (possibly C++) library? Do 
they actually allow for interrupting long running XML parsing, etc? 
Either we're in uncharted waters, in which case I suggest you get back 
in the boat and motor on... or there is some example out there of how to 
do this. Sorry but I don't have one off-hand.

I appreciate the effort you're putting into this, Good Luck!

Chris

On 12-05-24 09:35 AM, Sandro Santilli wrote:
> This mail is to share the current status of the battle for interruptable
> GEOS calls through use of custom memory allocators. The battle started
> in Paris last week and it's still on.
>
> The naive approach of redefining operators 'new' and 'delete' in the
> global namespace turned out to affect _any_ C++ code linked to the
> modules. This means also GDAL. So you set allocator/dellocator through
> the GEOS C-API and end up changing allocations everywhere.
>
> This is a bug, but didn't stop me from still trying to see it as a feature
> (we also want to stop CDAL/OGR don't we?) and keep walking in the jungle.
>
> The jungle is due to the fact that pooled memory allocations don't play
> nicely neither with GEOS nor with GDAL. This is because some long-lived
> objects in those libraries have components which are allocated lazily,
> on first use, thus resulting in objects which are partially allocated in
> a pool and partially in another pool. As long as their explicit
> destruction happens before any of these pools is cleared by postgresql
> we are fine, but if any of the pools reaches its end-of-life the next
> destruction would segfault (at best).
>
> I could handle this for GEOS as I can easily see which calls trigger
> the lazy allocations in long-lived objects, but isn't as simple with
> GDAL, not knowing the internals enough. For example now I get this:
>
>   = Invalid read of size 8
>   =    at 0xA6794E1: swq_op_registrar::DeInitialize() (stl_vector.h:533)
>   =    by 0xA5AD8AA: OGRCleanupAll (ogrsfdriverregistrar.cpp:150)
>   =    by 0xA09DF6E: ??? (in /usr/local/lib/libgdal.so.1.9.0)
>   =    by 0xA6872F0: ??? (in /usr/local/lib/libgdal.so.1.9.0)
>   =    by 0x563D261: exit (exit.c:78)
>   =    by 0x6542B5: proc_exit (ipc.c:137)
>   =    by 0x673EA5: PostgresMain (postgres.c:4136)
>   =    by 0x5C640C: main (main.c:197)
>   =  Address 0x69d8380 is 4,880 bytes inside a block of size 8,192 free'd
>   =    at 0x4C270BD: free (vg_replace_malloc.c:366)
>   =    by 0x754B62: AllocSetDelete (aset.c:551)
>   =    by 0x755B9C: MemoryContextDelete (mcxt.c:201)
>   =    by 0x9B5DE92: RASTER_dumpAsPolygons (rt_pg.c:1037)
>
> Which means that GDAL libary cleaner is trying to access, from a global
> destructor called automatically (OGRCleanupAll), memory which was
> allocated during execution of some code triggered by RASTER_dumpAsPolygons.
>
> I did all I could to have OGR drivers fully loaded before installing the
> custom memory management functions (calling OGRRegisterAll), but evidently
> that wasn't enough.
>
> I could go on trying to figure where memory of long-lived objects would
> be augmented and how to make it happen sooner, but I'm not sure this
> approach is viable in the long run as it can break as soon as an external
> library decides to change an allocation from immediate to lazy...
>
> Suggestions or even simple words of comfort are welcome :)
>
> --strk;
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-devel