[gdal-dev] IO Overhead when reading small subsets from Global Files

[Julian Zeidler]

Hi Even,

Thanks for the quick reply.

I should have mentioned, that I also tried converting it to a compressed 
tiled Tiff.
There i can see the same kind of overhead. I extract a 1MB Subset from 
the File and depending on the Tiles 1 read between 9-12 MB via the Network.
The Process only reads one 500x500 block from every single File

Cheers Julian



Am 08.12.2014 11:10, schrieb Even Rouault:
> Le lundi 08 décembre 2014 10:44:41, Julian Zeidler a écrit :
>> Dear Gdal-mailinglist,
>>
>> I am currently trying to optimize a Global Modell.
>> The Modell reads small chunks (500x500) from lots (One for each day) of
>> Global Datasets (40000x20000)
>> These Dataset are compressed NetCDFs with a tilling activated (100x100).
>> (See output oif gdalinfo attached)
>> However when I measure the File-IO via NFS i get a Factor of ~10
>> compared to the uncompressed Output image when testing with gdal. Inside
>> teh Modell using the netCDF library diretyl i measure an even worst
>> Factor of ~60 compared to compressed outputs). This is better than using
>> untiled Inputs where the overhad was ~80x, but still a larger overhead
>> than I expected.
>> I tested it using gdal_translate in.nc out.tif -srcwin 6000 6000 500 500
> Julian,
>
> I'm not sure how chunck indexing works internally in netCDF, but there might
> be an overhead when reading the "index" the first time. So perhaps if you do
> your reads from the same GDAL dataset object, without closing it between
> different requests, the overhead will decrease. If you were already doing that,
> then I'm not sure what you can do, except converting into another format, like
> GTiff.
>
> Even
>