[Gdal-dev] I/O Performance issues with ENVI data format
sjahputerao at missouri.edu
Wed Mar 7 12:41:50 EST 2007
I have to use RasterIO since we try to capture only certain parts of the
image. But the buffer size I use is simply the block size prescribed by
GDAL (if the block size is too large for the area we try to capture, I
use the smaller size of the two).
I think the GeoTiff is striped (each block is simply a row of image).
Forgive my ignorance, but where and how do you set the GDAL_CACHEMAX
Frank Warmerdam wrote:
> Ozy Sjahputera wrote:
>> I have a C++ program that conducts UTM reprojection and resampling of
>> large images before creating the pansharpened multispectral images.
>> Initially, I was using GTiff format for the pansharpened image. For
>> 16Kx16K panchromatic image (and its corresponding multispectral), it
>> took me about 50 minutes to finish the process. Then I tried using
>> ENVI format as a solution to the 4 GB limit of GTiff. The whole
>> process (using the same input images) now take more than 3 hours to
>> complete. Both programs were built using the release mode with the
>> same optimization options in VC.Net.
>> Both trials were run on the same machine with XP-SP2. Any idea why
>> the disk I/O with the ENVI format is much slower compared to the disk
>> I/O with the GTiff format?
> What sort of access pattern are you using for the data? Perhaps you
> are doing a RasterIO() for each pixel you want?
> Was your geotiff file tiled or striped?
> I see two likely issues:
> 1) The "rawrasterband" family of formats, including ENVI, has optimized
> logic for reading small bits of data. But the downside to this
> optimization is that it skips the block cache, and could substantially
> degrade performance if you do many many very small requests.
> 2) Possibly you are seeing cache thrashing with the ENVI driver which has
> scanline sized blocks where as you *might* have being used tiled
> which would be less subject to cache thrashing if you are doing local
> windowed reads.
> Of course, as you can see, much depends on subtle issues of data
> and the pattern of your reads.
> One very quick thing to tweak is to increase the cache size. For
> setting the GDAL_CACHEMAX environment variable to 200 (for 200MB).
> Best regards,
More information about the Gdal-dev