[gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

Fri Sep 2 09:57:32 EDT 2011

There has been recent work on incorporating netcdf support for point and
trajectory data with Talend, maybe that can help?

http://www.neogeo-online.net/blog/archives/1350/
http://talendforge.org/svn/sdi/trunk/org.talend.sdi.designer.components.sandbox/components/sNetCDFInput/

Etienne

On Tue, Aug 30, 2011 at 10:18 AM, Etienne Tourigny <etiennesky at yahoo.com>wrote:

> Brian,
>
> I am not familiar with madis data.  I assume that it is point-data for
> meteorological stations?
>
> If you like, please post your ideas and a short description in a new topic
> in the wiki entry at
> http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements .
>
> Etienne
>
>
>
> ----- Original Message -----
> From: Brian Case <rush at winkey.org>
> To: Etienne Tourigny <etiennesky at yahoo.com>
> Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
> Sent: Sunday, August 28, 2011 7:54:13 PM
> Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver
> and CF-1 convention
>
> any thoughts on a ogr netcdf driver? I have a simple one for madis data
> that needs work in my sandbox
>
> Brian
>
>
>
> On Sun, 2011-08-28 at 14:56 -0700, Etienne Tourigny wrote:
> > I have created a new wiki page at
> http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements , following Even's
> suggestion (Thanks Even).
> >
> > The following topics have been included:
> >
> >
> > Topics:
> >
> > 1 - UNIDATA NetCDF Conventions
> >
> > 2 - Metadata
> > 3 - Datum issues
> > 4 - Dimension issues
> > 5 - Compatibility with other software
> > 6 - Test files
> > 7 - Contribution
> > Discussion
> >
> > I encourage interested parties in participating in this discussion in the
> wiki and in the mailing list.
> >
> >
> > In particular, it would be useful to collect information on contributors
> and obtain test files with issues and the expected outcome.
> >
> > regards, Etienne
> >
> >
> > ----- Original Message -----
> > From: Even Rouault <even.rouault at mines-paris.org>
> > To: gdal-dev at lists.osgeo.org; Etienne Tourigny <etiennesky at yahoo.com>
> > Cc:
> > Sent: Friday, August 26, 2011 4:20:06 PM
> > Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF
> driver and CF-1 convention
> >
> > Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a écrit :
> >
> > My thoughts, just from a process point of view, not being myself deeply
> > involved in the NetCDF format :
> >
> > * A wiki page (I'd suggest either a new page -
> > http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements - or at least a
> clearly
> > identified section in the current page, to avoid confusion for outsiders
> about
> > what is currently implemented and what is not) can be a convenient way of
> > structuring and editing the content by anyone with a osgeo id. But
> discussions
> > on the mailing list are also OK, at least to notify new content has been
> added
> > (provided that it doesn't cause flooding on the mailing list ;-))
> >
> > * A http://download.osgeo.org/gdal/data/netcdf directory would be the
> right
> > place to store NetCDF sample files, provided they are free under the
> terms of
> > http://download.osgeo.org/gdal/data/COPYING . As write access to it is
> > restricted to people with appropriate rights, having those rights, I can
> help
> > uploading files over there. Having small (typically < or ~ 10 KB ) sample
> files
> > is also good so they can be directly included in SVN under the
> > autotest/gdrivers/data directory and thus easily used by regression
> tests.
> >
> > * Tickets can be used to track the implementation of specific items and
> > bugfixes. But I've found they are not generally suitable for
> brainstorming when
> > the discussion is lasting, as you cannot edit already posted content.
> Wiki is
> > clearly better for this.
> >
> > * Looking at past history, I see we haven't yet used the RFC formalism
> for
> > work specific to one driver, while it doesn't require changes or
> extensions to
> > the GDAL API. But it wouldn't hurt to create one if the need arises.
> Anyway, a
> > point that might require care and some consensus among the interested
> parties
> > is about how backward compatibility issues, in particular with respect to
> files
> > produced by older GDAL versions, are adressed.
> > http://trac.osgeo.org/gdal/wiki/rfc1_pmc details when a vote is
> required.
> >
> > Best regards,
> >
> > Even
> >
> > > Hi all,
> > >
> > > Since a few people have shown an interest in helping out, I would
> suggest
> > > to set up space where we can post ideas, goals and test files.  This
> could
> > > take the form of one of the following:
> > >
> > > - a new wiki entry in trac, or the existing one at
> > > http://trac.osgeo.org/gdal/wiki/NetCDF - a file repository at
> > > http://download.osgeo.org/gdal/data/netcdf as suggested some time ago
> by
> > > warmerdam
> > >
> > > and/or
> > >
> > > - a new ticket in trac (which allows to post files)
> > >
> > > What are you thoughts about this?  Would it make sense to eventually
> > > prepare an RFC?
> > >
> > >
> > >
> > > I would suggest that each group identify the problems with the current
> > > driver in terms of importing and exporting, post the files that they
> would
> > > like to be supported, which other software they use with the netcdf
> files,
> > > and their proposed contribution (code, test, ideas, etc.).
> > >
> > > Those that cannot participate in code development can help out in
> testing
> > > and discussion. Personally, I can write and test code, given example
> files
> > > and expected outcome.
> > >
> > >
> > > Now to your specific comments, Knut-Frode:
> > >
> > > 1) Yes CF-1.5 should be the goal, although I'm not sure we need to
> support
> > > upgrading/downgrading.
> > >
> > > 5) The WGS84 issue needs further research. Weather and ocean data are
> in
> > > lat/lon, but what is the sphere radius assumed by the various data
> > > providers and  the models which output/import netcdf files? I agree
> that a
> > > small error does not make a difference with low-resolution dataset, but
> > > I'd like to be sure in any case. We also need a means to translate the
> > > CF-1.x datum information into EPSG/Proj.4 values.
> > >
> > >
> > > 7) There is some support of a z-axis right now, in fact they driver
> assumes
> > > that a third dimension is vertical if I understand correctly. However,
> I
> > > have not tested nor looked into that.  This issue definitely needs come
> > > planning and an RFC, IMHO.
> > >
> > >
> > > Kyle, thanks for you help and interest!
> > >
> > >
> > > cheers,
> > > Etienne
> > >
> > >
> > > ________________________________
> > > From: Knut-Frode Dagestad <knutfrodesoppel at hotmail.com>
> > > To: gdal-dev at lists.osgeo.org
> > > Sent: Friday, August 26, 2011 4:41:02 AM
> > > Subject: [gdal-dev] Re: discussion on improvements to the NetCDF driver
> and
> > > CF-1 convention
> > >
> > > Etienne,
> > >
> > > My group is also very much in support of the suggested improvements, as
> > > NetCDF/CF is becoming the most important standard for data
> > > storage/exchange in our community (climate, meteorology, oceanography).
> My
> > > group is not much into GDAL driver development (at least not yet), but
> we
> > > are now building our future very much around GDAL, so we are interested
> to
> > > join the discussion around improved support for NetCDF/CF reading and
> > > writing.
> > >
> > > About some of the mentioned issues:
> > >
> > > 1) The latest version of CF is 1.5, so it would be better to target 1.5
> (or
> > > 1.x) than 1.0. Ideally the driver could be made generic for smooth
> upgrade
> > > to newer (or downgrade to older) versions of the CF conventions.
> > >
> > > 5) It sounds meaningful to assume WGS84 datum if not specified. I
> believe
> > > at least 99% of data typically stored in NetCDF/CF will have this
> datum,
> > > and even when that is not the case, positioning errors less than 100 m
> > > should normally have little impact for the types of data. Precision on
> > > datum level is probably more relevant for more typical GIS-data with
> > > roads, borders etc. A warning could however be issued each time the
> driver
> > > has to assume WGS84.
> > >
> > > 7) We agree in particular that proper handling of time axis is of high
> > > value. This is one of the issues which is highly relevant for
> climate-data
> > > (often 4-dimensional), but not well supported by typical GIS-software
> or
> > > standards designed with 2-dimensional data in mind. A general
> > > representation of a time-axis (or even vertical axis) in GDAL would be
> > > ideal, but that would probably be too much an effort for anyone to
> > > develop.
> > >
> > >
> > > Knut-Frode
> > > Nansen Environmental and Remote Sensing Center
> > >
> > > On 24/08/2011 22:37, Etienne Tourigny wrote:
> > > > Hi all,
> > > >
> > > > I would like to start a discussion with those interested about fixing
> > > > various issues in the NetCDF driver.
> > > >
> > > > As it stands, it is not fully CF-1.0 compliant, and produces
> > > > geographical grids that are not valid for other software.
> > > > Also, the infamous "up-side down" problem has been addressed for
> > > > importing netcdfs, but needs to be addressed for exporting.
> > > >
> > > > Related ticket: http://trac.osgeo.org/gdal/ticket/2129
> > > > I have been submitting patches for some time to fix the related
> issues.
> > > >
> > > > I have identified a number of issues, of which the following have
> been
> > > > fixed partially in my proposed patch.
> > > >
> > > > 1- conform to the cf-1.0 standard for geographical grids (creating
> lat
> > > > and lon dimensions and variables)
> > > > 2- make netcdf grids "bottom-up" by default, as all software out
> there
> > > > uses that convention
> > > > 3- fix problems related to floating-point / string conversions (also
> > > > reported in issue #4200 )
> > > > 4- fix metadata handling and duplication (as reported in issue
> #4204),
> > > > and keep band (variable) metadata outside of the global metadata
> > > >
> > > > Pending issues:
> > > >
> > > > 5- how to deal with netcdfs which have no datum? Can we assume WGS84?
> > > > These files are extremely common, and CF-1.0 has provisions for
> > > > identifying the key variables, but no Proj/Wkt codes.
> > > > 6- fix proper export of projected CRS (this is not fully necessary
> > > > though) with lat/lon values respecting CF-1.0
> > > > 7- handle time axis properly (at least when creating a netcdf) - that
> > > > would be great for import/export of netcdf
> > > >
> > > > Regards,
> > > > Etienne
> > > >
> > > >
> > > > _______________________________________________
> > > > gdal-dev mailing list
> > > > gdal-dev at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > >
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20110902/a47771de/attachment-0001.html