[gdal-dev] slow netCDF read times

[Julien Demaria]

Hi,

Maybe this is a problem with your NetCDF internal chunks cache too small and related to this ticket: https://trac.osgeo.org/gdal/ticket/5291
You can change this per-variable cache using this C function: http://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#ga2788cbfc6880ec70c304292af2bc7546
Else a workaround may be to rechunk your data using nccopy to have chunks of the same size than your reading window.
Another solution is to recompile your NetCDF library to set more chunks cache.

Regards,

Julien

De : gdal-dev [mailto:gdal-dev-bounces at lists.osgeo.org] De la part de Pablo Rozas Larraondo
Envoyé : mardi 22 novembre 2016 08:53
À : gdal-dev at lists.osgeo.org
Objet : [gdal-dev] slow netCDF read times

Hello,

I've come across some NetCDF4 files where GDAL is taking a surprisingly long time to read data from them. For example this is an example public file containing precipitation data:

ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0/global_dekad/netcdf/chirps-v2.0.2015.dekads.nc

If I use GDAL to read a small top left block (500x500) from one of its time bands, it takes approximately 1 minute on my computer. Source code is available here:

https://gist.github.com/monkeybutter/769a24bcf87682171eb87ac05c9347c5

The equivalent operation is completed in less than a second using the NetCDF library and even reading the whole file takes around 6 seconds with the same library.

I've tried to profile the GDAL program to get more insight and understand what's causing the overhead with not much success. All I know is that the deflate function is using 96% of the resources. I also guess that the way this file is chunked has something to do with its performance. Can anyone suggest any idea for better understanding what's happening here?

Thank you for your help,
Pablo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161122/5933a843/attachment.html>