<div dir="ltr">Hi Julien, thank you for the clarification - I had missed your reply while I was involved in things. <br><div><br></div><div>It turns out that the L2 products are probably of more interest to us generally so eventually my L3 adventures might not matter. </div><div><br></div><div>I'll email you privately about the sources. </div><div><br></div><div>Cheers, Mike. </div></div><br><div class="gmail_quote">On Wed Feb 11 2015 at 22:24:17 Julien Demaria <<a href="mailto:Julien.Demaria@acri-st.fr">Julien.Demaria@acri-st.fr</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="FR" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Mike,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sorry, I’ve written “</span><span lang="EN-US">full support of NetCDF-4”</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
but in fact NetCDF-4 compound types are not supported by my changes at this time, so NASA L3 binned products are not handled by GDAL.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">NASA L3 binned products store pixels in vector of structures (compound types), with my changes GDAL can now list the subdatasets, but the compound
types are still not handled and GDAL raise a warning and uses Float32 as default type, which is wrong, see the example below.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ideally, we could wrap each field of a compound type as a separate GDAL band in the subdataset. But if we want support all cases we have also to
handle compound types using other compound types, which is not easy technically, I think.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Note that NASA L3 mapped products are fully supported by GDAL (my changes only add capability to read metadata stored in specific groups, but main
variables should be readable without my changes).<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Example of NASA L3 binned product:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">NetCDF representation of level-3_binned_data group:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">group: level-3_binned_data {<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> types:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> compound binListType {<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> uint bin_num ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> short nobs ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> short nscenes ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> float weights ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> float time_rec ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> }; // binListType<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> compound binDataType {<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> float sum ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> float sum_squared ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> }; // binDataType<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> compound binIndexType {<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> uint start_num ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> uint begin ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">uint extent ;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> uint max ;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> }; // binIndexType<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> dimensions:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">binListDim = UNLIMITED ; // (1671755 currently)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binDataDim = UNLIMITED ; // (1671755 currently)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binIndexDim = UNLIMITED ; // (4320 currently)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> variables:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binListType BinList(binListDim) ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binDataType chlor_a(binDataDim) ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binDataType chl_ocx(binDataDim) ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> binIndexType BinIndex(binIndexDim) ;<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> } // group level-3_binned_data<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Corresponding GDAL Subdatasets:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_1_NAME=NETCDF:"<a href="http://V2015024.L3b_DAY_NPP_CHL.nc" target="_blank">V2015024.L3b_DAY_NPP_CHL.nc</a>":/level-3_binned_data/BinList<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_1_DESC=[1x1671755] /level-3_binned_data/BinList (x.</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">�</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_2_NAME=NETCDF:"<a href="http://V2015024.L3b_DAY_NPP_CHL.nc" target="_blank">V2015024.L3b_DAY_NPP_CHL.nc</a>":/level-3_binned_data/chlor_a<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_2_DESC=[1x1671755] /level-3_binned_data/chlor_a (x.</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">�</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_3_NAME=NETCDF:"<a href="http://V2015024.L3b_DAY_NPP_CHL.nc" target="_blank">V2015024.L3b_DAY_NPP_CHL.nc</a>":/level-3_binned_data/chl_ocx<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_3_DESC=[1x1671755] /level-3_binned_data/chl_ocx (x.</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">�</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_4_NAME=NETCDF:"<a href="http://V2015024.L3b_DAY_NPP_CHL.nc" target="_blank">V2015024.L3b_DAY_NPP_CHL.nc</a>":/level-3_binned_data/BinIndex<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> SUBDATASET_4_DESC=[1x4320] /level-3_binned_data/BinIndex (x.</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Tahoma","sans-serif";color:#1f497d">�</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Julien<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Michael Sumner [mailto:<a href="mailto:mdsumner@gmail.com" target="_blank">mdsumner@gmail.com</a>]
<br>
<b>Envoyé :</b> samedi 7 février 2015 02:03<br>
<b>À :</b> Julien Demaria; <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a></span></p></div></div><div lang="FR" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Objet :</b> Re: [gdal-dev] NetCDF driver improvements (including groups support)<u></u><u></u></span></p></div></div><div lang="FR" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi, this is really interesting to me. <u></u><u></u></p></div></div></div><div lang="FR" link="blue" vlink="purple"><div><div>
<div>
<p class="MsoNormal">On Fri Feb 06 2015 at 03:11:02 Julien Demaria <<a href="mailto:Julien.Demaria@acri-st.fr" target="_blank">Julien.Demaria@acri-st.fr</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi GDAL team,<br>
<br>
I've implemented several improvements to the NetCDF driver and I would like to provide them to the community.<br>
Main goal of the changes is to add full support of NetCDF-4 including groups.<br>
NetCDF-4 is the future format of ESA Sentinel-3 products (no groups) and NASA Ocean Color team is switching their L2/L3 products to NetCDF-4 with groups (VIIRS has already switched to the new format in December).<br>
With the changes NASA L2 products geolocation is automatically handled as geolocation arrays and can be reprojected using gdalwarp.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This sounds great, have you also worked with the L3 binned products? I have written my own suite of tools in R to read the NASA L3 bin files from the HDF4 files (using Rcpp), and have made some progress into the NetCDF4 versions of these.
Does your work on the GDAL driver include compound types like those in the L3 NetCDF4 bin files. The structure is described here: <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html" target="_blank">http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I prefer to work with the raw bin values usually rather than always apply a regridding - but having that capacity via GDAL would be great. I also wonder if it would be worth providing a vector driver for L3 bins, that would be my preferred
usage ultimately - I need to apply different algorithms for chlorophyll-a to the RRS values in the equal-area L3 bins, my R package is here, it's quite primitive and relies on the Rcpp route for C++, but at least does what I need for now.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://github.com/mdsumner/roc" target="_blank">https://github.com/mdsumner/roc</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'll try to have a closer look at your efforts here soon, but just wanted to reply now with my perspective. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers, Mike. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">I validated with autotests that nothing is broken in tests netcdf.py (excepting test 13 but see my point 5), netcdf_cf.py and hdf5.py, using NetCDF-3 and 4 libraries.<br>
I've also tested the new functionalities on various NetCDF-4 files.<br>
I think the only possible regression could be for marginal cases where a file was seen directly as a dataset and is now seen as multiple subdatasets (for example if a file has only one var in the top group and has nested groups containing variables), but I
think this is not very common.<br>
<br>
For the moment I have all these changes in local GIT separated commits on the latest gdal-1.11 branch, let me know what changes you want and how can I provide them.<br>
<br>
Changes :<br>
<br>
1) Implement full support for NetCDF-4 groups on reading:<br>
- explore recursively all nested groups to create the subdatasets list<br>
- subdatasets in nested groups use the /group1/group2/.../groupn/var standard<br>
NetCDF-4 convention, excepting for variables in the root group which do not<br>
have a leading slash for backward compatibility<br>
- when accessing a subdataset using NETCDF:$file:$path, the leading slash is optional<br>
- global attributes of each nested group are also collected in the GDAL dataset<br>
metadata, using the same convention /group1/group2/.../groupn/NC_GLOBAL#attr_name,<br>
excepting for the root group which do not have a leading slash for backward compatibility<br>
- when searching for a variable containing auxiliary information on the selected subdataset,<br>
like coordinate variables or grid_mapping, we now also search in parent groups (using NCDFResolveVar).<br>
I now this is something not specified at this time in the CF convention because CF does not know groups,<br>
but it seems logical to me to support this: NetCDF-4 specifies that dimensions of a group are<br>
shared to its nested groups, so associated coordinate variables could be defined as the same level of its<br>
corresponding dimension.<br>
- reference to coordinate variables using the "coordinates" attribute support now also absolute paths,<br>
this allow for example to specify coordinate variables located outside the group of the selected variable<br>
or its parents. Relative paths could be implemented if needed.<br>
This feature is used to add support for new NASA Ocean Color L2 products.<br>
<br>
2) Implement full read/write support for new NetCDF4 types NC_UBYTE, NC_USHORT, NC_UINT and NC_STRING, only if NETCDF_HAS_NC4 is defined (and only if format=NC4 for writing).<br>
Support implemented for variables and attributes.<br>
NC_STRING type is supported for reading (scalar and arrays) attributes and is used for writing only for array attributes (scalar are still written as NC_CHAR).<br>
If NETCDF_HAS_NC4 is not defined or format!=NC4, NC_STRING array attributes are written as a single NC_CHAR string using the GDAL {v1,v2,...} convention.<br>
Add missing support for NC_BYTE in CreateBandMetadata() and NC_BYTE/SHORT in NCDFPut1DVar().<br>
<br>
3) Add support for new NASA Ocean Color L2 products and ESA Sentinel-3 L1 or<br>
L2 products which use the NetCDF-4 format (with groups for NASA, see<br>
<a href="http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html" target="_blank">
http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html</a>):<br>
- NASA products: simulate a "coordinates" variable attribute to detect CF<br>
geolocation arrays, and set bBottomUp to FALSE<br>
- ESA products: set bBottomUp to FALSE and disable warning on missing<br>
Conventions attribute<br>
<br>
4) Fix bug #4554 with a more generic solution by disabling the installation of the HDF5 atexit() cleanup routine using H5dont_atexit().<br>
Previous fix was to call GDALExit() (for the moment only defined gdalwarp.cpp) at the end of every program, which is more painful.<br>
<br>
5) Fix implementation of GetScale/Offset to not always return pbSuccess=TRUE.<br>
Fix CopyMetadata to handle bands with only scale or offset.<br>
==> WARNING this commit breaks the autotest netcdf_13 (check for scale/offset = 1.0/0.0 if no scale or offset is available), but for me it is not logical to return always pbSuccess=TRUE<br>
<br>
6) Optimize IReadBlock() and CheckData() handling of partial blocks in the x axis by re-using the GDAL block buffer instead of allocating a new temporary buffer for each block.<br>
<br>
7) Force block size to 1 scanline for bottom-up datasets if nBlockYSize != 1 instead of raising a fatal error<br>
==> Solve a recent problem raised on the mailing list<br>
<br>
8) Implement Get/SetUnitType using the standard "units" NetCDF attribute<br>
<br>
9) Change default block size to 256x256 instead of scanline (only affect file without NetCDF chunking)<br>
==> because I think this is better for a random access to the data, but I'm not sure if the community want this change which could impact performances<br>
<br>
10) I've also implemented for my needs support for 1D variables by simulating 2D datasets with only one row (dimensionless variables are not supported for the moment),<br>
but this breaks backward compatibility because files containing only one variable and associated 1D coordinate variables are now seen as multiple sub datasets...<br>
and maybe this is not the goal of GDAL to give access to not-2D-raster variables (but sometimes it's useful ;-) )<br>
<br>
Thanks for GDAL!<br>
<br>
Julien<br>
<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
</div></div></div></blockquote></div>