[Gdal-dev] Increasing Buffer Size in GDAL's Project2

[Frank Warmerdam]

Mubin wrote:
> Hi,
> 
> I am using GDAL's project2.exe file to convert BIP (bytes interleaved by
> pixels) formatted files to BSQ (bytes sequential).  A high resolution image
> file of size 30GB takes about ~12-15hrs to convert.  I could use ENVI to do
> this conversion for me which would take about ~5-6hrs but, as I have many
> files to convert so I want to batch the process.  I remember someone once
> telling me that the buffer size is ~64k in project2.  If that is true, is
> there a way to increase this buffer size in the code for faster processing? 
> If not, is there any other way to increase the speed of project2.  Here is
> my call to project2:
> 
> project2 -of ENVI -s_srs "+proj=utm +zone=16 +datum=wgs84" -t_srs "+proj=utm
> +zone=16 +datum=wgs84" DSS_Image_BIP.bip DSS_Image_BSQ.bsq

Mubin,

I believe project2 is a customized version of gdalwarp prepared for you, so it
isn't likely to be familiar to most GDAL users.

But, assuming it supports all the normal gdalwarp commandline options, you
should be able to add "-wm 400" to assign 400M of warp memory.  This
essentially establishes the size of imagery chunk warped at a time.  Bigger
chunks are somewhat more efficient than small chunks, though the difference
isn't necessarily large.

The other thing that can help is to increase the block cache size.  This
is the IO subsystem cache of blocks on the source and destination file.
This can be set at the commandline using the "--config GDAL_CACHEMAX 400"
switch or by setting the GDAL_CACHEMAX environment variable in advance.

I would suggest setting both values to 400 (400MB) since you have lots of
RAM.  Setting them larger is unlikely to improve performance noticably.

PS. if you aren't really doing any resampling or reprojecting, you might
get much better performance just using gdal_translate directly.  The
GDAL_CACHEMAX item can be helpful for that program as well.  Going through
the warp api adds a great deal of extra processing work.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org