[gdal-dev] NetCDF driver getting wrong values if compared to HDF5

[Ivan Lucena]

Hi Even,

Thank you very much for your attention on this issue. Thank you for your time.

I did what you told and I rebuild netCDF SDK with support for NC-4. In order to do that I needed to build the HDF4 SDK and install a lot of things I didn't know I didn't have on that machine.

Anyway, the problem persist but I suspect that the issue is not type conversion.

Like you said, the netCDF should handle the the offset/scale float32 to uint8 conversion correctly, with or without NC-4.

I think it is safe to say that the important part of the raster, the land, is correctly handled by HDF5 or NetCDF.

It think the problem has to do with empty blocks. I got that impression just by looking at that image:

[cid:9386b87f-0aba-469b-a152-893fc1d19888]

This image for example was produce by converting using the HDF5 driver. I applied a color table where blue represents Ocean with pixel value 254. Grey represents nodata with value 255.

I think that that the HDF5 driver and/or the HDF5 SDK is handling the ocean data correctly but when it finds an empty block it refers to nodata.

In the other hand, the netCDF driver and/or the netCDF SDK is doing something else. Look at that one image produced by netCDF:

[cid:079aa948-1407-416d-8333-c69fca256fd6]

It is giving me 255 in all the Ocean, where it supposed to be 254. That is weird. I could be applying offset/scale to the flag values.

The netCDF ncdump is showing most the data as NaN. The MEP REST API, the same thing:

import requests
url = 'https://proba-v-mep.esa.int/api/timeseries/v1.1/ts/PROBAV_L3_S10_TOC_NDVI_333M/geometry?startDate=2019-07-11&endDate=2019-07-11'
geom = """{
        "type": "Polygon",
        "coordinates": [[
            [44.1830357,  -1.0937500],
            [44.1830357,  -1.1830357],
            [44.2723214,  -1.1830357],
            [44.2723214,  -1.0937500],
            [44.1830357,  -1.0937500]
        ]]
    }"

{'totalCount': 900, 'validCount': 0, 'average': 'NaN'}

The Proba-V data is not for Oceanography anyway. so it should not be a problem whatever driver I use.

But maybe the netCDF and HDF5 driver's authors would like to take a look.

Thank you again,

Ivan


________________________________
From: Even Rouault <even.rouault at spatialys.com>
Sent: Wednesday, July 31, 2019 11:19 AM
To: gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
Cc: Ivan Lucena <ivan.lucena at outlook.com>
Subject: Re: [gdal-dev] NetCDF driver getting wrong values if compared to HDF5

On mardi 30 juillet 2019 21:48:45 CEST Ivan Lucena wrote:
> Unsupported netCDF datatype (7), treat as Float32.​

Given that

#define NC_UBYTE        7       /**< unsigned 1 byte int */

and

https://github.com/OSGeo/gdal/blob/master/gdal/frmts/netcdf/netcdfdataset.cpp#L357

it seems you would have a GDAL build against a netCDF build without NC4 support.

That said, even with that, I would have expected the ubyte to float32 conversion to be lossless.

but according to
https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#ga0ab808702d73b7f52be14b1e54473902
"""
The nc_get_vara() function will read a variable of any type, including user defined type.
For this function, the type of the data in memory must match the type of the variable - no data conversion is done.
"""

So you could get pretty much garbage.

Solution: build GDAL against a netCDF build that has NC4 support

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190801/5e582ddd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 15250 bytes
Desc: image.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190801/5e582ddd/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 18698 bytes
Desc: image.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190801/5e582ddd/attachment-0003.png>