[gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

Tue Aug 30 09:18:38 EDT 2011

Brian,

I am not familiar with madis data.  I assume that it is point-data for meteorological stations?

If you like, please post your ideas and a short description in a new topic in the wiki entry at
http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements .

Etienne

----- Original Message -----
From: Brian Case <rush at winkey.org>
To: Etienne Tourigny <etiennesky at yahoo.com>
Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
Sent: Sunday, August 28, 2011 7:54:13 PM
Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

any thoughts on a ogr netcdf driver? I have a simple one for madis data
that needs work in my sandbox

Brian

On Sun, 2011-08-28 at 14:56 -0700, Etienne Tourigny wrote:
> I have created a new wiki page at http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements , following Even's suggestion (Thanks Even).
> 
> The following topics have been included:
> 
> 
> Topics:
> 
> 1 - UNIDATA NetCDF Conventions
> 
> 2 - Metadata
> 3 - Datum issues
> 4 - Dimension issues
> 5 - Compatibility with other software
> 6 - Test files
> 7 - Contribution
> Discussion
> 
> I encourage interested parties in participating in this discussion in the wiki and in the mailing list. 
> 
> 
> In particular, it would be useful to collect information on contributors and obtain test files with issues and the expected outcome. 
> 
> regards, Etienne
> 
> 
> ----- Original Message -----
> From: Even Rouault <even.rouault at mines-paris.org>
> To: gdal-dev at lists.osgeo.org; Etienne Tourigny <etiennesky at yahoo.com>
> Cc: 
> Sent: Friday, August 26, 2011 4:20:06 PM
> Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention
> 
> Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a écrit :
> 
> My thoughts, just from a process point of view, not being myself deeply 
> involved in the NetCDF format :
> 
> * A wiki page (I'd suggest either a new page - 
> http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements - or at least a clearly 
> identified section in the current page, to avoid confusion for outsiders about 
> what is currently implemented and what is not) can be a convenient way of 
> structuring and editing the content by anyone with a osgeo id. But discussions 
> on the mailing list are also OK, at least to notify new content has been added 
> (provided that it doesn't cause flooding on the mailing list ;-))
> 
> * A http://download.osgeo.org/gdal/data/netcdf directory would be the right 
> place to store NetCDF sample files, provided they are free under the terms of 
> http://download.osgeo.org/gdal/data/COPYING . As write access to it is 
> restricted to people with appropriate rights, having those rights, I can help 
> uploading files over there. Having small (typically < or ~ 10 KB ) sample files 
> is also good so they can be directly included in SVN under the 
> autotest/gdrivers/data directory and thus easily used by regression tests.
> 
> * Tickets can be used to track the implementation of specific items and 
> bugfixes. But I've found they are not generally suitable for brainstorming when 
> the discussion is lasting, as you cannot edit already posted content. Wiki is 
> clearly better for this.
> 
> * Looking at past history, I see we haven't yet used the RFC formalism for 
> work specific to one driver, while it doesn't require changes or extensions to 
> the GDAL API. But it wouldn't hurt to create one if the need arises. Anyway, a 
> point that might require care and some consensus among the interested parties 
> is about how backward compatibility issues, in particular with respect to files 
> produced by older GDAL versions, are adressed. 
> http://trac.osgeo.org/gdal/wiki/rfc1_pmc details when a vote is required.
> 
> Best regards,
> 
> Even
> 
> > Hi all,
> > 
> > Since a few people have shown an interest in helping out, I would suggest
> > to set up space where we can post ideas, goals and test files.  This could
> > take the form of one of the following:
> > 
> > - a new wiki entry in trac, or the existing one at
> > http://trac.osgeo.org/gdal/wiki/NetCDF - a file repository at
> > http://download.osgeo.org/gdal/data/netcdf as suggested some time ago by
> > warmerdam
> > 
> > and/or
> > 
> > - a new ticket in trac (which allows to post files)
> > 
> > What are you thoughts about this?  Would it make sense to eventually
> > prepare an RFC?
> > 
> > 
> > 
> > I would suggest that each group identify the problems with the current
> > driver in terms of importing and exporting, post the files that they would
> > like to be supported, which other software they use with the netcdf files,
> > and their proposed contribution (code, test, ideas, etc.).
> > 
> > Those that cannot participate in code development can help out in testing
> > and discussion. Personally, I can write and test code, given example files
> > and expected outcome. 
> > 
> > 
> > Now to your specific comments, Knut-Frode:
> > 
> > 1) Yes CF-1.5 should be the goal, although I'm not sure we need to support
> > upgrading/downgrading.
> > 
> > 5) The WGS84 issue needs further research. Weather and ocean data are in
> > lat/lon, but what is the sphere radius assumed by the various data
> > providers and  the models which output/import netcdf files? I agree that a
> > small error does not make a difference with low-resolution dataset, but
> > I'd like to be sure in any case. We also need a means to translate the
> > CF-1.x datum information into EPSG/Proj.4 values.
> > 
> > 
> > 7) There is some support of a z-axis right now, in fact they driver assumes
> > that a third dimension is vertical if I understand correctly. However, I
> > have not tested nor looked into that.  This issue definitely needs come
> > planning and an RFC, IMHO.
> > 
> > 
> > Kyle, thanks for you help and interest!
> > 
> > 
> > cheers,
> > Etienne
> > 
> > 
> > ________________________________
> > From: Knut-Frode Dagestad <knutfrodesoppel at hotmail.com>
> > To: gdal-dev at lists.osgeo.org
> > Sent: Friday, August 26, 2011 4:41:02 AM
> > Subject: [gdal-dev] Re: discussion on improvements to the NetCDF driver and
> > CF-1 convention
> > 
> > Etienne,
> > 
> > My group is also very much in support of the suggested improvements, as
> > NetCDF/CF is becoming the most important standard for data
> > storage/exchange in our community (climate, meteorology, oceanography). My
> > group is not much into GDAL driver development (at least not yet), but we
> > are now building our future very much around GDAL, so we are interested to
> > join the discussion around improved support for NetCDF/CF reading and
> > writing.
> > 
> > About some of the mentioned issues:
> > 
> > 1) The latest version of CF is 1.5, so it would be better to target 1.5 (or
> > 1.x) than 1.0. Ideally the driver could be made generic for smooth upgrade
> > to newer (or downgrade to older) versions of the CF conventions.
> > 
> > 5) It sounds meaningful to assume WGS84 datum if not specified. I believe
> > at least 99% of data typically stored in NetCDF/CF will have this datum,
> > and even when that is not the case, positioning errors less than 100 m
> > should normally have little impact for the types of data. Precision on
> > datum level is probably more relevant for more typical GIS-data with
> > roads, borders etc. A warning could however be issued each time the driver
> > has to assume WGS84.
> > 
> > 7) We agree in particular that proper handling of time axis is of high
> > value. This is one of the issues which is highly relevant for climate-data
> > (often 4-dimensional), but not well supported by typical GIS-software or
> > standards designed with 2-dimensional data in mind. A general
> > representation of a time-axis (or even vertical axis) in GDAL would be
> > ideal, but that would probably be too much an effort for anyone to
> > develop.
> > 
> > 
> > Knut-Frode
> > Nansen Environmental and Remote Sensing Center
> > 
> > On 24/08/2011 22:37, Etienne Tourigny wrote:
> > > Hi all,
> > > 
> > > I would like to start a discussion with those interested about fixing
> > > various issues in the NetCDF driver.
> > > 
> > > As it stands, it is not fully CF-1.0 compliant, and produces
> > > geographical grids that are not valid for other software.
> > > Also, the infamous "up-side down" problem has been addressed for
> > > importing netcdfs, but needs to be addressed for exporting.
> > > 
> > > Related ticket: http://trac.osgeo.org/gdal/ticket/2129
> > > I have been submitting patches for some time to fix the related issues.
> > > 
> > > I have identified a number of issues, of which the following have been
> > > fixed partially in my proposed patch.
> > > 
> > > 1- conform to the cf-1.0 standard for geographical grids (creating lat
> > > and lon dimensions and variables)
> > > 2- make netcdf grids "bottom-up" by default, as all software out there
> > > uses that convention
> > > 3- fix problems related to floating-point / string conversions (also
> > > reported in issue #4200 )
> > > 4- fix metadata handling and duplication (as reported in issue #4204),
> > > and keep band (variable) metadata outside of the global metadata
> > > 
> > > Pending issues:
> > > 
> > > 5- how to deal with netcdfs which have no datum? Can we assume WGS84?
> > > These files are extremely common, and CF-1.0 has provisions for
> > > identifying the key variables, but no Proj/Wkt codes.
> > > 6- fix proper export of projected CRS (this is not fully necessary
> > > though) with lat/lon values respecting CF-1.0
> > > 7- handle time axis properly (at least when creating a netcdf) - that
> > > would be great for import/export of netcdf
> > > 
> > > Regards,
> > > Etienne
> > > 
> > > 
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > 
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev