[gdal-dev] reading large image files

[Jonathan Greenberg]

I want to back up Even's statement about trying to read a file 
piece-by-piece.  About 99% of raster processes are better handled with 
"tiled" processing -- a nice balance between i/o, memory and sanity 
would be to read the image one line at a time, perform your processing 
on that line, write that line out, then move to the next one:

(generically)
for i = 1, number of lines
    read line i
    analyze line i
    write results of line i to output file
end for

If you want to get more creative (or are doing spatial processing), you 
might want to read a set of lines, restricted by your memory usage.  
There are almost no raster processes that ever need an entire image read 
into memory at one time.

--j

Even Rouault wrote:
> Stephen,
>
> My first answer was "Maybe you have not waited long enough... ;-)"
>
> But I could also reproduce the slowness with GDAL <= 1.5.2 while reading such 
> a huge amount of data into one call.
> There was an inefficient reading strategy in GDALRasterBand::IRasterIO that 
> your dataset precisely hits, that is to say raster with big raster dimensions 
> and tiled data. This is bug #2457 and it has been fixed recently in the SVN 
> branch of 1.5 (in r14863) and trunk. I can confirm that updating to r14863 of 
> branches/1.5 fixes the slowlyness.
>
> So several solutions come to mind :
> * try to read your file piece by piece (ideally by adjusting your requests on 
> block boundaries)
> * update to the SVN version 
> * wait for the next release 
> * try with GDAL_CACHEMAX=54, because 54 MB = 56 623 104 bytes > (1024 * 1024 * 
> 2) * (27552 / 1024). See #2457 for the explanation of that formula ;-) 
>
> (Side note, I see one error in your code. You ask for GDT_Byte data but you 
> allocate a float array. )
>
> Best regards
>
> Le Wednesday 16 July 2008 17:53:21 stephen tan, vous avez écrit :
>   
>> I have a nitf image that I am trying to read into an array using
>> gdal-1.5.1.  The file is about 1.5G .
>>
>> gdalinfo returns the following information:
>>     Size is 27552, 25196
>>     ...
>>     Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Undefined
>>
>> I have tried to read the image into an array using the following code:
>>     GDALDataset  *poDataset;
>>
>>     GDALAllRegister();
>>
>>     poDataset = (GDALDataset *) GDALOpenShared( fname.c_str(),
>> GA_ReadOnly );
>>     if( poDataset == NULL )
>>     {
>>         cerr << "Could not open file: " << fname << endl;
>>         exit(1);
>>     }
>>
>>     int width = poDataset->GetRasterXSize();
>>     int height = poDataset->GetRasterYSize();
>>     int bands = poDataset->GetRasterCount();
>>
>>     GDALRasterBand  *poBand;
>>     int nBlockXSize, nBlockYSize;
>>
>>     poBand = poDataset->GetRasterBand( 1 );
>>
>>     poBand->GetBlockSize( &nBlockXSize, &nBlockYSize );
>>
>>     //float* data;
>>     //data = (float *) CPLMalloc(sizeof(float)*width*height*bands);
>>
>>     float* data;
>>     data = new float[width*height*bands];
>>
>>     //GByte* data;
>>     //data = (GByte *) CPLMalloc(sizeof(float)*width*height*bands);
>>
>>     //GByte data[width*height*bands];
>>
>>     if(poBand->RasterIO( GF_Read, 0, 0, width, height, data, width,
>> height, GDT_Byte, 0, 0 ))
>>     {
>>         cerr << "Error: Could not read image data." << endl;
>>         exit(1);
>>     }
>>     GDALClose(poDataset);
>>
>> No errors are returned.  I let it run for several mins before I hit CTRL+C.
>> Any C++ suggestions on getting the image into a 1D array and/or 2D array
>> would be most helpful.  The above code works for smaller images.  I have
>> also tried different GDALSetCacheMax with no success.
>>
>> Thanks
>>
>>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>     
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>