<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Even,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you very much for your attention on this issue. Thank you for your time.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I did what you told and I rebuild netCDF SDK with support for NC-4. In order to do that I needed to build the HDF4 SDK and install a lot of things I didn't know I didn't have on that machine.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Anyway, the problem persist but I suspect that the issue is not type conversion.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif">
Like you said, the netCDF should handle the the offset/scale float32 to uint8 conversion correctly, with or without NC-4.</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif">
<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif">
I think it is safe to say that the important part of the raster, the land, is correctly handled by HDF5 or NetCDF.</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif">
<br>
</div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It think the problem has to do with empty blocks. <span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I got that impression just by looking at that image:</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<img size="15250" contenttype="image/png" unselectable="on" tabindex="-1" style="max-width: 100%; user-select: none;" data-outlook-trace="F:1|T:1" src="cid:9386b87f-0aba-469b-a152-893fc1d19888"><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif">
<span style="margin: 0px">This image for example was produce by converting using the HDF5 driver. </span><span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I applied a color table where blue represents Ocean with
pixel value 254. Grey represents nodata with value 255.</span></div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I think that that the HDF5 driver and/or the HDF5 SDK is handling the ocean data correctly but when it <span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">finds an empty block it refers to nodata.</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
In the other hand, the netCDF driver and/or the netCDF SDK is doing something else. <span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Look at that one image produced by netCDF:</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<img size="18698" contenttype="image/png" unselectable="on" tabindex="-1" style="max-width: 100%; user-select: none;" data-outlook-trace="F:1|T:1" src="cid:079aa948-1407-416d-8333-c69fca256fd6"><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It is giving me 255 in all the Ocean, where it supposed to be 254. That is weird. I could be applying offset/scale to the flag values.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The netCDF ncdump is showing most the data as NaN. The MEP REST API, the same thing:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace; font-size: 10pt;">import requests</span><span><br>
</span>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">url = 'https://proba-v-mep.esa.int/api/timeseries/v1.1/ts/PROBAV_L3_S10_TOC_NDVI_333M/geometry?startDate=2019-07-11&endDate=2019-07-11'</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;">geom = """{</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "type": "Polygon",</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> "coordinates": [[</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> [44.1830357, -1.0937500],</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> [44.1830357, -1.1830357],</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> [44.2723214, -1.1830357],</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> [44.2723214, -1.0937500],</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> [44.1830357, -1.0937500]</span><br>
</div>
<div><span style="font-family: "Courier New", monospace; font-size: 10pt;"> ]]</span><br>
</div>
<span style="font-family: "Courier New", monospace; font-size: 10pt;"> }"</span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>
<pre style="box-sizing: border-box; overflow: auto; font-family: monospace; font-size: 14px; display: block; padding: 1px 0px; margin: 0px; word-break: break-all; overflow-wrap: break-word; background-color: rgb(255, 255, 255); border-radius: 0px; text-align: left"><span style="font-family: "Courier New", monospace; font-size: 10pt;">{'totalCount': 900, 'validCount': 0, 'average': 'NaN'}</span></pre>
<br>
</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">The Proba-V data is not for Oceanography anyway. so it should not be a problem whatever driver I use. </span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
But maybe the netCDF and HDF5 driver's authors would like to take a look.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you again,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ivan</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Even Rouault <even.rouault@spatialys.com><br>
<b>Sent:</b> Wednesday, July 31, 2019 11:19 AM<br>
<b>To:</b> gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br>
<b>Cc:</b> Ivan Lucena <ivan.lucena@outlook.com><br>
<b>Subject:</b> Re: [gdal-dev] NetCDF driver getting wrong values if compared to HDF5</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">On mardi 30 juillet 2019 21:48:45 CEST Ivan Lucena wrote:<br>
> Unsupported netCDF datatype (7), treat as Float32.<br>
<br>
Given that<br>
<br>
#define NC_UBYTE 7 /**< unsigned 1 byte int */<br>
<br>
and <br>
<br>
<a href="https://github.com/OSGeo/gdal/blob/master/gdal/frmts/netcdf/netcdfdataset.cpp#L357">https://github.com/OSGeo/gdal/blob/master/gdal/frmts/netcdf/netcdfdataset.cpp#L357</a><br>
<br>
it seems you would have a GDAL build against a netCDF build without NC4 support.<br>
<br>
That said, even with that, I would have expected the ubyte to float32 conversion to be lossless.<br>
<br>
but according to <br>
<a href="https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#ga0ab808702d73b7f52be14b1e54473902">https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html#ga0ab808702d73b7f52be14b1e54473902</a><br>
"""<br>
The nc_get_vara() function will read a variable of any type, including user defined type.<br>
For this function, the type of the data in memory must match the type of the variable - no data conversion is done.<br>
"""<br>
<br>
So you could get pretty much garbage.<br>
<br>
Solution: build GDAL against a netCDF build that has NC4 support<br>
<br>
Even<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href="http://www.spatialys.com">http://www.spatialys.com</a><br>
</div>
</span></font></div>
</body>
</html>