[gdal-dev] Memory fragmentation and memory consumption when warping with GCPs

[Joaquim Luis]

Frank Warmerdam wrote:
> Joaquim Luis wrote:
>> So one question is, why warping with GCPs takes such large amount of 
>> memory (apparently a chunk
>> of 500 Mb was not enough to process 5000 GCPs)?
> 
> Joaquim,
> 
> Is the memory fragmentation really related to the use of many GCPs?  Does
> the same problem occur with 4 GCPs?

Frank,

Maybe I was not clear enough, but the memory segmentation has nothing to do
with the GCP warping. It results from loading the gdal.dll

For example, on a fresh start if I call (one other MEX for reading) gdalread
without arguments, it prints the usage on screen. After this, which did not
perform any calculus, the memory is already fragmented.

The issue with the GCPs is that with a largest chunk of ~500 Mb I'm not able
to warp with 5000 GCPs.


Joaquim


> 
>> I also tried with the command line GDAL gdaltransform and here I had 
>> no problems. So I'm a bit puzzled.
>> Do all programs depend on the memory fragmentation issue, or is it 
>> just a Matlab limitation?
>> (Ah, Windows XP here)
>>
>> As a side note, I also have MEXs that link with the OpenCV library, 
>> but those do not sensibly
>> fragment the memory.
> 
> It is possible that OpenCV uses a private heap thereby minimising the
> amount of fragmentation in the "master heap".  It might also be that OpenEV
> just doesn't use that much memory.
> 
> I suspect you are seeing some heap fragmentation and consumption due to
> GDAL's block caching mechanism but it is really hard to know.  You can
> influence the amount of memory available for the block cache using
> GDALSetCacheMax(). I'm not in the habit of monitoring heap fragmentation
> on Win32.  I will say that GDAL normally uses malloc() and free() for
> memory allocation so that may have some implications for heap effects.
> 
> Best regards,