[gdal-dev] parsing of netcdf4 no-data values in hdf5 reader

[Julien Demaria]

Hi Piyush,

Thanks for your help proposal!
I am working on porting changes to latest GDAL, I think it will be available tomorrow on GitHub.
For this step I prefer finish it myself because I have already worked a lot on the NetCDF driver and it’s a big piece of code (10k lines!) not always easy to navigate...
My modifications are “generic” and CF-oriented so they should handle all sort of NetCDF4 files. Can I have access to some examples of the NC4 files you want to read?

Julien

De : Agram, Piyush S (334D) [mailto:Piyush.Agram at jpl.nasa.gov]
Envoyé : dimanche 14 octobre 2018 03:01
À : Julien Demaria; gdal-dev at lists.osgeo.org
Objet : Re: parsing of netcdf4 no-data values in hdf5 reader

Hi Julien,
     If you want to put up your code on github, I can volunteer to help port it to 2.x. Generating unittests would not be a problem. Will probably take existing datasets in the test suite and add a group around the datasets to begin with.
Not very familiar with MODIS/VIIRS but looks like its nc4 with groups but all the dimensions are datasets under root. Maybe we can generalize this too?

Piyush

From: Julien Demaria <julien.demaria at acri-st.fr>
Date: Saturday, October 13, 2018 at 4:09 PM
To: "Agram, Piyush S (334D)" <Piyush.Agram at jpl.nasa.gov>, "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
Subject: RE: parsing of netcdf4 no-data values in hdf5 reader


Hi,



I have already already implemented NetCDF4 groups support for GDAL in 2015 and use it to read for example NASA OceanColor MODIS/VIIRS L2 products, see the attached email.

Unfortunately I never found enough time to add corresponding GDAL autotests (covering all my modifications) to be able to provide these improvements to the GDAL community.

The other problem is that I implemented this in GDAL 1.11 so porting to latest GDAL need some work, I started to do that in 2016 but never finished...

But I think my code is good and it will be a pity if not reused...

I think I can try to finish the porting to the latest GDAL and provide it to GitHub, but I'm still not sure to have enough time to implement all needed corresponding GDAL autotests. If you want to contribute you are welcome! ;-)



Best Regards,

Julien



-----Message d'origine-----
De : gdal-dev [mailto:gdal-dev-bounces at lists.osgeo.org] De la part de Agram, Piyush S (334D)
Envoyé : samedi 13 octobre 2018 21:00
À : gdal-dev at lists.osgeo.org
Objet : Re: [gdal-dev] parsing of netcdf4 no-data values in hdf5 reader



Hi Even,

     Looking into the issue, I think the NETCDF reader does not read datasets located insides groups for Rasters. There are 2 specific parts of the reader that seems to be the issue:



1. If the drivers does not find 2 arrays that are dimensions under root it bails. This would need to be updated if data is organized within groups and there are no dims/vars at the root level.

2. It only looks into the groups when used for reading vectors and when it does it only parses the string types for geometry definitions. This would need to be updated.



Some related questions to see if anybody is already planning to contribute these capabilities:



1. Update to NETCDF raster driver to look into sub-groups as well? The spec does not seem to talk about this. Assuming they are trying to be backward compatible with formats that did not support groups - is this acceptable?

2. Support for CF conventions in HDF5 reader? NASA/ESRI seem to be happy working with CF-compliant HDF5.



Adding either support will let people use GDAL with xarray etc.



Piyush





_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181015/94dee198/attachment.html>