[gdal-dev] gdalinfo's and rasterio's reading problem in LUSTRE FS with NetCDF file (ubuntu:bionic)

Erick Palacios Moreno epalacios at conabio.gob.mx
Wed Jun 5 11:44:07 PDT 2019 

Hi everyone,

 My team and I have been facing some problems when reading netcdf files in `LUSTRE` filesystem with `ncdump` tool regarding an HDF5 1.10.1 issue in `ubuntu:bionic` (see for example: https://github.com/ALPSCore/ALPSCore/issues/410)

We made a workaround by installing `netcdf-bin` using repositories of `ubuntu:xenial` in `bionic`. Although `ncdump` have successfully read the netcdf file, we couldn't fix it (sort of...) for `gdalinfo` and  `rasterio` for example:

this works:

ncdump /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc|head -n 20

gdalinfo /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc |head -n 20

gdalinfo -sd 1 /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc |head -n 20

but this neither of next lines work:

gdalinfo netcdf:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean

gdalinfo NetCDF:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean

```
ERROR 4: NetCDF:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean: No such file or directory
gdalinfo failed - unable to open 'NetCDF:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean'.
```

This ultimately leads to error with `rio insp`  in both next lines :

rio insp netcdf:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean

rio insp NetCDF:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean

```
ERROR:root:Exception caught during processing
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 214, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 64, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 205, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: netcdf:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/rasterio/rio/insp.py", line 77, in insp
    with rasterio.open(input, mode) as src:
  File "/usr/local/lib/python3.6/dist-packages/rasterio/env.py", line 423, in wrapper
    return f(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py", line 216, in open
    s = DatasetReader(path, driver=driver, **kwargs)
  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: netcdf:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc:blue_mean: No such file or directory
Aborted!

```

But this works for both `gdalinfo` and `rio insp` tools:

```
gdalinfo hdf5:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc://blue_mean

Driver: HDF5Image/HDF5 Dataset
Files: /LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc
Size is 1667, 1667
Coordinate System is `'
Metadata:
  Conventions=CF-1.6, ACDD-1.3
  date_created=2019-04-26T21:07:47.595416
  geospatial_bounds=POLYGON ((-98.8618335626467 19.8093029971201,-98.8722811078025 19.3587577175966,-98.3949028300917 19.3481841431337,-98.382861955109 19.7986884105649,-98.8618335626467 19.8093029971201))
  geospatial_bounds_crs=EPSG:4326
...
```

```
rio insp hdf5:/LUSTRE/MADMEX/dir_test_mount/madmex_003_37_-32_1996-01-01.nc://blue_mean

/usr/local/lib/python3.6/dist-packages/rasterio/__init__.py:216: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, **kwargs)
Rasterio 1.0.23 Interactive Inspector (Python 3.6.7)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.meta
{'driver': 'HDF5Image', 'dtype': 'int16', 'nodata': None, 'width': 1667, 'height': 1667, 'count': 1, 'crs': None, 'transform': Affine(1.0, 0.0, 0.0,
       0.0, 1.0, 0.0)}

>>> a=src.read()
>>> a
array([[[632, 603, 586, ..., 505, 483, 471],
        [567, 538, 536, ..., 468, 478, 523],
        [614, 580, 537, ..., 540, 633, 675],
        ...,
        [828, 810, 804, ..., 275, 268, 299],
        [857, 844, 823, ..., 310, 290, 307],
        [840, 854, 836, ..., 320, 288, 294]]], dtype=int16)

```

This is related with LUSTRE FS because if we copy our NetCDF file to /root location then:

```
this works:

gdalinfo netcdf:/root/madmex_003_37_-32_1996-01-01.nc:blue_mean

Driver: netCDF/Network Common Data Format
Files: /root/madmex_003_37_-32_1996-01-01.nc
Size is 1667, 1667
Coordinate System is:
PROJCS["unnamed",
    GEOGCS["WGS 84",
...
```

```
rio insp netcdf:/root/madmex_003_37_-32_1996-01-01.nc:blue_mean
Rasterio 1.0.23 Interactive Inspector (Python 3.6.7)
Type "src.meta", "src.read(1)", or "help(src)" for more information.
>>> src.meta
{'driver': 'netCDF', 'dtype': 'int16', 'nodata': -32767.0, 'width': 1667, 'height': 1667, 'count': 1, 'crs': CRS.from_wkt('PROJCS["unnamed",GEOGCS["WGS 84",DATUM["unknown",SPHEROID["WGS84",6378137,6556752.3141]],PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["standard_parallel_1",17.5],PARAMETER["standard_parallel_2",29.5],PARAMETER["latitude_of_origin",12],PARAMETER["central_meridian",-102],PARAMETER["false_easting",2500000],PARAMETER["false_northing",0]]'), 'transform': Affine(30.0, 0.0, 2827530.0,
       0.0, -30.0, 876410.0)}
>>> a=src.read()
>>> a
array([[[632, 603, 586, ..., 505, 483, 471],
        [567, 538, 536, ..., 468, 478, 523],
        [614, 580, 537, ..., 540, 633, 675],
        ...,
        [828, 810, 804, ..., 275, 268, 299],
        [857, 844, 823, ..., 310, 290, 307],
        [840, 854, 836, ..., 320, 288, 294]]], dtype=int16)
```

We are using `ubuntu:bionic` docker image so you can reproduce this error using next lines:

```
sudo docker run -v /LUSTRE/:/LUSTRE/ \
-v /LUSTRE/MADMEX/dir_test_mount:/temporal \
--name mounting_volume_test_bionic \
--hostname antares3-datacube \
-dit ubuntu:bionic /bin/bash
```

enter to docker container:

```
sudo docker exec -it mounting_volume_test_bionic bash
```

and execute:

```
apt-get update

export DEBIAN_FRONTEND=noninteractive && echo "America/Mexico_City" > /etc/timezone && apt-get install -y tzdata
apt-get update && apt-get install -y \
            wget curl \
            openssh-server \
            openssl \
            sudo \
            nano \
            software-properties-common \
            git \
            vim \
            vim-gtk \
            htop \
            build-essential \
            libssl-dev \
            libffi-dev \
            cmake \
            python3-dev \
            python3-pip \
            python3-setuptools \
            ca-certificates \
            postgresql-client \
            libudunits2-dev \
            nodejs && pip3 install --upgrade pip

#install netcdf:

echo -e 'deb http://security.ubuntu.com/ubuntu/ xenial-security universe\ndeb http://archive.ubuntu.com/ubuntu/ xenial universe' >> /etc/apt/sources.list
apt update
apt install -t xenial-security libcurl3-gnutls
apt install -y -t xenial netcdf-bin


#Install spatial libraries
add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable && apt-get -qq update

apt-get install -y \
            ncview \
            libproj-dev \
            libgeos-dev \
            gdal-bin \
            libgdal-dev
pip install numpy && pip install GDAL==$(gdal-config --version) --global-option=build_ext --global-option='-I/usr/include/gdal' && pip install rasterio
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export GDAL_DATA=/usr/share/gdal/
```

Hope you can help us

Cheers,

Erick

More information about the gdal-dev mailing list