[gdal-dev] Memory use in GDALDriver::CreateCopy()

[Even Rouault]

Ozy,

Did you try with gdal_translate -of NITF src.tif output.tif -co 
BLOCKSIZE=128 ? Does it give similar results ?

I'm a bit surprised that you even managed to read a 40Kx100K large NITF 
file organized as scanlines. There was a limit until very recently that 
prevented to read blocks whose one dimension was bigger than 9999. This 
was fixed recently in trunk ( see ticket 
http://trac.osgeo.org/gdal/ticket/3263 ) and branches/1.6, but it has 
not yet been released to an officially released version. So which GDAL 
version are you using ?

Does the output of gdalinfo on your scanline oriented input NITF gives 
something like :
Band 1 Block=40000x1 Type=Byte, ColorInterp=Gray

Is your input NITF compressed or uncompressed ?

Anyway, with latest trunk, I've simulated creating a similarly large 
NITF image with the following python snippet :

import gdal
ds = gdal.GetDriverByName('NITF').Create('scanline.ntf', 40000, 100000)
ds = None

and then creating the tiled NITF :

gdal_translate -of NITF scanline.ntf tiled.ntf -co BLOCKSIZE=128

The memory consumption is very reasonnable (less than 50 MB : the 
default block cache size of 40 MB + temporary buffers ), so I'm not 
clear why you would have a problem of increasing memory use.

ozy sjahputera a écrit :
> I was trying to make a copy of a very large NITF image (about 40Kx100K 
> pixels) using GDALDriver::CreateCopy(). The new file was set to have 
> different block-size (input was a scanline image, output is to have a 
> 128x128 blocksize). The program keeps getting killed by the system 
> (Linux). I monitor the memory use of the program as it was executing 
> CreateCopy and the memory use was steadily increasing as the progress 
> indicator from CreateCopy was moving forward.
>
> Why does CreateCopy() use so much memory? I have not perused the 
> source code of CreateCopy() yet, but I am guessing it employs 
> RasterIO() to perform the read/write?
>
> I was trying different sizes for GDAL  cache from 64MB, 256MB, 512MB, 
> 1GB, and 2GB. The program got killed in all these cache sizes. In 
> fact, my Linux box became unresponsive when I set GDALSetCacheMax() to 
> 64MB.
>
> Thank you.
> Ozy
>
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev