[gdal-dev] IO Overhead when reading small subsets from Global Files

[Julian Zeidler]

Dear Gdal-mailinglist,

I am currently trying to optimize a Global Modell.
The Modell reads small chunks (500x500) from lots (One for each day) of 
Global Datasets (40000x20000)
These Dataset are compressed NetCDFs with a tilling activated (100x100). 
(See output oif gdalinfo attached)
However when I measure the File-IO via NFS i get a Factor of ~10 
compared to the uncompressed Output image when testing with gdal. Inside 
teh Modell using the netCDF library diretyl i measure an even worst 
Factor of ~60 compared to compressed outputs). This is better than using 
untiled Inputs where the overhad was ~80x, but still a larger overhead 
than I expected.
I tested it using gdal_translate in.nc out.tif -srcwin 6000 6000 500 500

Is there anything that can be done to decrease this Overhead?
  For a global Modell Run the Overhead would add up to about 300TB*

*I am thankful for any suggestions.

Cheers Julian




Gdalinfo:

Driver: netCDF/Network Common Data Format

Files: LAI_Y2014_C10.nc

Size is 40320, 20160

Coordinate System is `'

Origin = (-179.999993460545994,90.000001090156800)

Pixel Size = (0.008928571482636,-0.008928571915164)

Metadata:

LAI#Fill_Value=-999

LAI#long_name=Leaf Area Index

LAI#units=m2/m2

LAI#valid_range={1,10}

latitude#long_name=Latitude

latitude#units=degrees_north

longitude#long_name=Longitude

longitude#units=degrees_east

NC_GLOBAL#comment=email:markus.tum at dlr.de

NC_GLOBAL#Conventions=CF-1.4

NC_GLOBAL#institution=DLR-DFD

NC_GLOBAL#title=10 Day LAI Composite based on Geoland2

Corner Coordinates:

Upper Left (-179.9999935,  90.0000011)

Lower Left (-179.9999935, -90.0000087)

Upper Right ( 180.0000087,  90.0000011)

Lower Right ( 180.0000087, -90.0000087)

Center ( 0.0000076,  -0.0000038)

Band 1 Block=100x100 Type=Float32, ColorInterp=Undefined

NoData Value=-999

Metadata:

Fill_Value=-999

long_name=Leaf Area Index

NETCDF_VARNAME=LAI

units=m2/m2

valid_range={1,10}

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20141208/1a7fd269/attachment.html>