[gdal-dev] Memory use in GDALDriver::CreateCopy()

[ozy sjahputera]

Hi Even,

yes, I tried:
gdal_translate -of "NITF" -co "ICORDS=G" -co "BLOCKXSIZE=128" -co
"BLOCKYSIZE=128"  NITF_IM:0:input.ntf output.ntf

I monitored the memory use using top and it was steadily increasing till it
reached 98.4% (I have 8GB of RAM and 140 GB of local disk for swap etc.)
before the node died (not just the program, but the whole system just
stopped responding).

My GDAL version is 1.6.2.

gdalinfo on this image shows the raster size of (37504, 98772) and
Block=37504x1.
The image is compressed using JPEG2000 option and contains two subdatasets
(data and cloud data ~ I used only the data for gdal_translate test).

Band info from gdalinfo:
Band 1 Block=37504x1 Type=UInt16, ColorInterp=Gray

Ozy

On Tue, Jan 12, 2010 at 5:38 PM, Even Rouault
<even.rouault at mines-paris.org>wrote:

> Ozy,
>
> Did you try with gdal_translate -of NITF src.tif output.tif -co
> BLOCKSIZE=128 ? Does it give similar results ?
>
> I'm a bit surprised that you even managed to read a 40Kx100K large NITF
> file organized as scanlines. There was a limit until very recently that
> prevented to read blocks whose one dimension was bigger than 9999. This
> was fixed recently in trunk ( see ticket
> http://trac.osgeo.org/gdal/ticket/3263 ) and branches/1.6, but it has
> not yet been released to an officially released version. So which GDAL
> version are you using ?
>
> Does the output of gdalinfo on your scanline oriented input NITF gives
> something like :
> Band 1 Block=40000x1 Type=Byte, ColorInterp=Gray
>
> Is your input NITF compressed or uncompressed ?
>
> Anyway, with latest trunk, I've simulated creating a similarly large
> NITF image with the following python snippet :
>
> import gdal
> ds = gdal.GetDriverByName('NITF').Create('scanline.ntf', 40000, 100000)
> ds = None
>
> and then creating the tiled NITF :
>
> gdal_translate -of NITF scanline.ntf tiled.ntf -co BLOCKSIZE=128
>
> The memory consumption is very reasonnable (less than 50 MB : the
> default block cache size of 40 MB + temporary buffers ), so I'm not
> clear why you would have a problem of increasing memory use.
>
> ozy sjahputera a écrit :
> > I was trying to make a copy of a very large NITF image (about 40Kx100K
> > pixels) using GDALDriver::CreateCopy(). The new file was set to have
> > different block-size (input was a scanline image, output is to have a
> > 128x128 blocksize). The program keeps getting killed by the system
> > (Linux). I monitor the memory use of the program as it was executing
> > CreateCopy and the memory use was steadily increasing as the progress
> > indicator from CreateCopy was moving forward.
> >
> > Why does CreateCopy() use so much memory? I have not perused the
> > source code of CreateCopy() yet, but I am guessing it employs
> > RasterIO() to perform the read/write?
> >
> > I was trying different sizes for GDAL  cache from 64MB, 256MB, 512MB,
> > 1GB, and 2GB. The program got killed in all these cache sizes. In
> > fact, my Linux box became unresponsive when I set GDALSetCacheMax() to
> > 64MB.
> >
> > Thank you.
> > Ozy
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20100113/6b468a8f/attachment.html