[postgis-devel] Status of custom memory allocators based interruptable calls

[Sandro Santilli]

This mail is to share the current status of the battle for interruptable
GEOS calls through use of custom memory allocators. The battle started
in Paris last week and it's still on.

The naive approach of redefining operators 'new' and 'delete' in the
global namespace turned out to affect _any_ C++ code linked to the
modules. This means also GDAL. So you set allocator/dellocator through
the GEOS C-API and end up changing allocations everywhere.

This is a bug, but didn't stop me from still trying to see it as a feature
(we also want to stop CDAL/OGR don't we?) and keep walking in the jungle.

The jungle is due to the fact that pooled memory allocations don't play 
nicely neither with GEOS nor with GDAL. This is because some long-lived
objects in those libraries have components which are allocated lazily,
on first use, thus resulting in objects which are partially allocated in
a pool and partially in another pool. As long as their explicit
destruction happens before any of these pools is cleared by postgresql
we are fine, but if any of the pools reaches its end-of-life the next
destruction would segfault (at best).

I could handle this for GEOS as I can easily see which calls trigger
the lazy allocations in long-lived objects, but isn't as simple with 
GDAL, not knowing the internals enough. For example now I get this:

 = Invalid read of size 8
 =    at 0xA6794E1: swq_op_registrar::DeInitialize() (stl_vector.h:533)
 =    by 0xA5AD8AA: OGRCleanupAll (ogrsfdriverregistrar.cpp:150)
 =    by 0xA09DF6E: ??? (in /usr/local/lib/libgdal.so.1.9.0)
 =    by 0xA6872F0: ??? (in /usr/local/lib/libgdal.so.1.9.0)
 =    by 0x563D261: exit (exit.c:78)
 =    by 0x6542B5: proc_exit (ipc.c:137)
 =    by 0x673EA5: PostgresMain (postgres.c:4136)
 =    by 0x5C640C: main (main.c:197)
 =  Address 0x69d8380 is 4,880 bytes inside a block of size 8,192 free'd
 =    at 0x4C270BD: free (vg_replace_malloc.c:366)
 =    by 0x754B62: AllocSetDelete (aset.c:551)
 =    by 0x755B9C: MemoryContextDelete (mcxt.c:201)
 =    by 0x9B5DE92: RASTER_dumpAsPolygons (rt_pg.c:1037)

Which means that GDAL libary cleaner is trying to access, from a global
destructor called automatically (OGRCleanupAll), memory which was
allocated during execution of some code triggered by RASTER_dumpAsPolygons.

I did all I could to have OGR drivers fully loaded before installing the
custom memory management functions (calling OGRRegisterAll), but evidently
that wasn't enough.

I could go on trying to figure where memory of long-lived objects would 
be augmented and how to make it happen sooner, but I'm not sure this
approach is viable in the long run as it can break as soon as an external
library decides to change an allocation from immediate to lazy...

Suggestions or even simple words of comfort are welcome :)

--strk;