[Gdal-dev] I/O Performance issues with ENVI data format

[Frank Warmerdam]

Ozy Sjahputera wrote:
> I have a C++ program that conducts UTM reprojection and resampling of 
> large images before creating the pansharpened multispectral images.  
> Initially, I was using GTiff format for the pansharpened image.  For 
> 16Kx16K panchromatic image (and its corresponding multispectral), it 
> took me about 50 minutes to finish the process.  Then I tried using ENVI 
> format as a solution to the 4 GB limit of GTiff.  The whole process 
> (using the same input images) now take more than 3 hours to complete.  
> Both programs were built using the release mode with the same 
> optimization options in VC.Net.
> 
> Both trials were run on the same machine with XP-SP2.  Any idea why the 
> disk I/O with the ENVI format is much slower compared to the disk I/O 
> with the GTiff format?

Ozy,

What sort of access pattern are you using for the data?  Perhaps you
are doing a RasterIO() for each pixel you want?

Was your geotiff file tiled or striped?

I see two likely issues:

1) The "rawrasterband" family of formats, including ENVI, has optimized
    logic for reading small bits of data.  But the downside to this
    optimization is that it skips the block cache, and could substantially
    degrade performance if you do many many very small requests.

2) Possibly you are seeing cache thrashing with the ENVI driver which has
    scanline sized blocks where as you *might* have being used tiled geotiffs
    which would be less subject to cache thrashing if you are doing local
    windowed reads.

Of course, as you can see, much depends on subtle issues of data organization
and the pattern of your reads.

One very quick thing to tweak is to increase the cache size.  For instance
setting the GDAL_CACHEMAX environment variable to 200 (for 200MB).

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org