[Gdal-dev] I/O Performance issues with ENVI data format

[Ozy Sjahputera]

Hi Frank,

I have to use RasterIO since we try to capture only certain parts of the 
image.  But the buffer size I use is simply the block size prescribed by 
GDAL (if the block size is too large for the area we try to capture, I 
use the smaller size of the two). 

I think the GeoTiff is striped (each block is simply a row of image).

Forgive my ignorance, but where and how do you set the GDAL_CACHEMAX 
parameter?

Thanks.

Oz



Frank Warmerdam wrote:
> Ozy Sjahputera wrote:
>> I have a C++ program that conducts UTM reprojection and resampling of 
>> large images before creating the pansharpened multispectral images.  
>> Initially, I was using GTiff format for the pansharpened image.  For 
>> 16Kx16K panchromatic image (and its corresponding multispectral), it 
>> took me about 50 minutes to finish the process.  Then I tried using 
>> ENVI format as a solution to the 4 GB limit of GTiff.  The whole 
>> process (using the same input images) now take more than 3 hours to 
>> complete.  Both programs were built using the release mode with the 
>> same optimization options in VC.Net.
>>
>> Both trials were run on the same machine with XP-SP2.  Any idea why 
>> the disk I/O with the ENVI format is much slower compared to the disk 
>> I/O with the GTiff format?
>
> Ozy,
>
> What sort of access pattern are you using for the data?  Perhaps you
> are doing a RasterIO() for each pixel you want?
>
> Was your geotiff file tiled or striped?
>
> I see two likely issues:
>
> 1) The "rawrasterband" family of formats, including ENVI, has optimized
>    logic for reading small bits of data.  But the downside to this
>    optimization is that it skips the block cache, and could substantially
>    degrade performance if you do many many very small requests.
>
> 2) Possibly you are seeing cache thrashing with the ENVI driver which has
>    scanline sized blocks where as you *might* have being used tiled 
> geotiffs
>    which would be less subject to cache thrashing if you are doing local
>    windowed reads.
>
> Of course, as you can see, much depends on subtle issues of data 
> organization
> and the pattern of your reads.
>
> One very quick thing to tweak is to increase the cache size.  For 
> instance
> setting the GDAL_CACHEMAX environment variable to 200 (for 200MB).
>
> Best regards,