[gdal-dev] gdalinfo's and rasterio's reading problem in LUSTRE FS with NetCDF file (ubuntu:bionic)

[Even Rouault]

On mercredi 5 juin 2019 13:44:07 CEST Erick Palacios Moreno wrote:
> Hi everyone,
> 
>  My team and I have been facing some problems when reading netcdf files in
> `LUSTRE` filesystem with `ncdump` tool regarding an HDF5 1.10.1 issue in
> `ubuntu:bionic` (see for example:
> https://github.com/ALPSCore/ALPSCore/issues/410)
> 
> We made a workaround by installing `netcdf-bin` using repositories of
> `ubuntu:xenial` in `bionic`. Although `ncdump` have successfully read the
> netcdf file, we couldn't fix it (sort of...) for `gdalinfo` and  `rasterio`
> for example:
> 
> this works:
> 
> ncdump /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc|head -n
> 20
> 
> gdalinfo /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc |head
> -n 20
> 
> gdalinfo -sd 1 /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc
> |head -n 20
> 
> but this neither of next lines work:
> 
> gdalinfo
> netcdf:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_m
> ean

That one is weird, I mean that 'gdalinfo -sd 1 ...' works, but not 'gdalinfo 
netcdf:....', since the former internally invokes the later, but with the 
difference of an extra GDALOpen()/GDALClose().

I think you've already found the root cause of all this, that its that HDF5 
1.10 doesn't play well with LUSTRE, which causes those kind of weird/random 
failures. Nothing that GDAL can do about. So your best option is to build GDAL 
and libnetcdf against a libhdf5 version that plays well with LUSTRE, or copy 
locally the files.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com