[gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

Sun Aug 28 17:56:03 EDT 2011

I have created a new wiki page at http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements , following Even's suggestion (Thanks Even).

The following topics have been included:

Topics:

1 - UNIDATA NetCDF Conventions

2 - Metadata
3 - Datum issues
4 - Dimension issues
5 - Compatibility with other software
6 - Test files
7 - Contribution
Discussion

I encourage interested parties in participating in this discussion in the wiki and in the mailing list. 

In particular, it would be useful to collect information on contributors and obtain test files with issues and the expected outcome. 

regards, Etienne

----- Original Message -----
From: Even Rouault <even.rouault at mines-paris.org>
To: gdal-dev at lists.osgeo.org; Etienne Tourigny <etiennesky at yahoo.com>
Cc: 
Sent: Friday, August 26, 2011 4:20:06 PM
Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a écrit :

My thoughts, just from a process point of view, not being myself deeply 
involved in the NetCDF format :

* A wiki page (I'd suggest either a new page - 
http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements - or at least a clearly 
identified section in the current page, to avoid confusion for outsiders about 
what is currently implemented and what is not) can be a convenient way of 
structuring and editing the content by anyone with a osgeo id. But discussions 
on the mailing list are also OK, at least to notify new content has been added 
(provided that it doesn't cause flooding on the mailing list ;-))

* A http://download.osgeo.org/gdal/data/netcdf directory would be the right 
place to store NetCDF sample files, provided they are free under the terms of 
http://download.osgeo.org/gdal/data/COPYING . As write access to it is 
restricted to people with appropriate rights, having those rights, I can help 
uploading files over there. Having small (typically < or ~ 10 KB ) sample files 
is also good so they can be directly included in SVN under the 
autotest/gdrivers/data directory and thus easily used by regression tests.

* Tickets can be used to track the implementation of specific items and 
bugfixes. But I've found they are not generally suitable for brainstorming when 
the discussion is lasting, as you cannot edit already posted content. Wiki is 
clearly better for this.

* Looking at past history, I see we haven't yet used the RFC formalism for 
work specific to one driver, while it doesn't require changes or extensions to 
the GDAL API. But it wouldn't hurt to create one if the need arises. Anyway, a 
point that might require care and some consensus among the interested parties 
is about how backward compatibility issues, in particular with respect to files 
produced by older GDAL versions, are adressed. 
http://trac.osgeo.org/gdal/wiki/rfc1_pmc details when a vote is required.

Best regards,

Even

> Hi all,
> 
> Since a few people have shown an interest in helping out, I would suggest
> to set up space where we can post ideas, goals and test files.  This could
> take the form of one of the following:
> 
> - a new wiki entry in trac, or the existing one at
> http://trac.osgeo.org/gdal/wiki/NetCDF - a file repository at
> http://download.osgeo.org/gdal/data/netcdf as suggested some time ago by
> warmerdam
> 
> and/or
> 
> - a new ticket in trac (which allows to post files)
> 
> What are you thoughts about this?  Would it make sense to eventually
> prepare an RFC?
> 
> 
> 
> I would suggest that each group identify the problems with the current
> driver in terms of importing and exporting, post the files that they would
> like to be supported, which other software they use with the netcdf files,
> and their proposed contribution (code, test, ideas, etc.).
> 
> Those that cannot participate in code development can help out in testing
> and discussion. Personally, I can write and test code, given example files
> and expected outcome. 
> 
> 
> Now to your specific comments, Knut-Frode:
> 
> 1) Yes CF-1.5 should be the goal, although I'm not sure we need to support
> upgrading/downgrading.
> 
> 5) The WGS84 issue needs further research. Weather and ocean data are in
> lat/lon, but what is the sphere radius assumed by the various data
> providers and  the models which output/import netcdf files? I agree that a
> small error does not make a difference with low-resolution dataset, but
> I'd like to be sure in any case. We also need a means to translate the
> CF-1.x datum information into EPSG/Proj.4 values.
> 
> 
> 7) There is some support of a z-axis right now, in fact they driver assumes
> that a third dimension is vertical if I understand correctly. However, I
> have not tested nor looked into that.  This issue definitely needs come
> planning and an RFC, IMHO.
> 
> 
> Kyle, thanks for you help and interest!
> 
> 
> cheers,
> Etienne
> 
> 
> ________________________________
> From: Knut-Frode Dagestad <knutfrodesoppel at hotmail.com>
> To: gdal-dev at lists.osgeo.org
> Sent: Friday, August 26, 2011 4:41:02 AM
> Subject: [gdal-dev] Re: discussion on improvements to the NetCDF driver and
> CF-1 convention
> 
> Etienne,
> 
> My group is also very much in support of the suggested improvements, as
> NetCDF/CF is becoming the most important standard for data
> storage/exchange in our community (climate, meteorology, oceanography). My
> group is not much into GDAL driver development (at least not yet), but we
> are now building our future very much around GDAL, so we are interested to
> join the discussion around improved support for NetCDF/CF reading and
> writing.
> 
> About some of the mentioned issues:
> 
> 1) The latest version of CF is 1.5, so it would be better to target 1.5 (or
> 1.x) than 1.0. Ideally the driver could be made generic for smooth upgrade
> to newer (or downgrade to older) versions of the CF conventions.
> 
> 5) It sounds meaningful to assume WGS84 datum if not specified. I believe
> at least 99% of data typically stored in NetCDF/CF will have this datum,
> and even when that is not the case, positioning errors less than 100 m
> should normally have little impact for the types of data. Precision on
> datum level is probably more relevant for more typical GIS-data with
> roads, borders etc. A warning could however be issued each time the driver
> has to assume WGS84.
> 
> 7) We agree in particular that proper handling of time axis is of high
> value. This is one of the issues which is highly relevant for climate-data
> (often 4-dimensional), but not well supported by typical GIS-software or
> standards designed with 2-dimensional data in mind. A general
> representation of a time-axis (or even vertical axis) in GDAL would be
> ideal, but that would probably be too much an effort for anyone to
> develop.
> 
> 
> Knut-Frode
> Nansen Environmental and Remote Sensing Center
> 
> On 24/08/2011 22:37, Etienne Tourigny wrote:
> > Hi all,
> > 
> > I would like to start a discussion with those interested about fixing
> > various issues in the NetCDF driver.
> > 
> > As it stands, it is not fully CF-1.0 compliant, and produces
> > geographical grids that are not valid for other software.
> > Also, the infamous "up-side down" problem has been addressed for
> > importing netcdfs, but needs to be addressed for exporting.
> > 
> > Related ticket: http://trac.osgeo.org/gdal/ticket/2129
> > I have been submitting patches for some time to fix the related issues.
> > 
> > I have identified a number of issues, of which the following have been
> > fixed partially in my proposed patch.
> > 
> > 1- conform to the cf-1.0 standard for geographical grids (creating lat
> > and lon dimensions and variables)
> > 2- make netcdf grids "bottom-up" by default, as all software out there
> > uses that convention
> > 3- fix problems related to floating-point / string conversions (also
> > reported in issue #4200 )
> > 4- fix metadata handling and duplication (as reported in issue #4204),
> > and keep band (variable) metadata outside of the global metadata
> > 
> > Pending issues:
> > 
> > 5- how to deal with netcdfs which have no datum? Can we assume WGS84?
> > These files are extremely common, and CF-1.0 has provisions for
> > identifying the key variables, but no Proj/Wkt codes.
> > 6- fix proper export of projected CRS (this is not fully necessary
> > though) with lat/lon values respecting CF-1.0
> > 7- handle time axis properly (at least when creating a netcdf) - that
> > would be great for import/export of netcdf
> > 
> > Regards,
> > Etienne
> > 
> > 
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev