[gdal-dev] Experience with slowness of free() on Windows with lots of allocations?

[Even Rouault]

Hi,

while investigating 
https://github.com/OSGeo/gdal/issues/9510#issuecomment-2010950408, I've 
come to the conclusion that the Windows heap allocation mechanism sucks. 
Basically if you allocate a lot of heap regions of modest size with 
malloc()/new[], the time spent when freeing them all with corresponding 
free()/delete[] is excruciatingly slow (like ~ 10 seconds for ~ 80,000 
allocations). The slowness is clearly quadratic with the number of 
allocations. You only start noticing it with ~ 30,000 allocations. And 
interestingly, another condition for that slowness is that each 
individual allocation much be strictly greater than 4096 * 4 bytes. At 
exactly that value, perf is acceptable, but add one extra byte, and it 
suddenly drops. I suspect that there must be a threshold from which 
malloc() starts using VirtualAlloc() instead of the heap, which must 
involve slow system calls, instead of a user-land allocation mechanism.

Anyone has already hit that and found solutions? The only potential idea 
I found until now would be to use a private heap with HeapCreate() with 
a fixed maximum size, which is a bit problematic to adopt by default, 
basically that would mean that the size of GDAL_CACHEMAX would be 
consumed as soon as one use the block cache.

Even

-- 
http://www.spatialys.com
My software is free, but my time generally not.