[gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention

Brian Case rush at winkey.org
Fri Sep 2 11:49:03 EDT 2011


Etienne

I can read it with ogr. I am wondering if others have interest too

http://svn.osgeo.org/gdal/sandbox/winkey/ogr-netcdf/ogrsf_frmts/netcdf/


Brian

On Fri, 2011-09-02 at 10:57 -0300, Etienne Tourigny wrote:
> There has been recent work on incorporating netcdf support for point
> and trajectory data with Talend, maybe that can help?
> 
> 
> http://www.neogeo-online.net/blog/archives/1350/
> http://talendforge.org/svn/sdi/trunk/org.talend.sdi.designer.components.sandbox/components/sNetCDFInput/
> 
> 
> Etienne
> 
> On Tue, Aug 30, 2011 at 10:18 AM, Etienne Tourigny
> <etiennesky at yahoo.com> wrote:
>         Brian,
>         
>         I am not familiar with madis data.  I assume that it is
>         point-data for meteorological stations?
>         
>         If you like, please post your ideas and a short description in
>         a new topic in the wiki entry at
>         http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements .
>         
>         Etienne
>         
>         
>         
>         
>         ----- Original Message -----
>         From: Brian Case <rush at winkey.org>
>         To: Etienne Tourigny <etiennesky at yahoo.com>
>         Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
>         Sent: Sunday, August 28, 2011 7:54:13 PM
>         Subject: Re: [gdal-dev] Re: discussion on improvements to the
>         NetCDF driver and CF-1 convention
>         
>         any thoughts on a ogr netcdf driver? I have a simple one for
>         madis data
>         that needs work in my sandbox
>         
>         Brian
>         
>         
>         
>         On Sun, 2011-08-28 at 14:56 -0700, Etienne Tourigny wrote:
>         > I have created a new wiki page at
>         http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements ,
>         following Even's suggestion (Thanks Even).
>         >
>         > The following topics have been included:
>         >
>         >
>         > Topics:
>         >
>         > 1 - UNIDATA NetCDF Conventions
>         >
>         > 2 - Metadata
>         > 3 - Datum issues
>         > 4 - Dimension issues
>         > 5 - Compatibility with other software
>         > 6 - Test files
>         > 7 - Contribution
>         > Discussion
>         >
>         > I encourage interested parties in participating in this
>         discussion in the wiki and in the mailing list.
>         >
>         >
>         > In particular, it would be useful to collect information on
>         contributors and obtain test files with issues and the
>         expected outcome.
>         >
>         > regards, Etienne
>         >
>         >
>         > ----- Original Message -----
>         > From: Even Rouault <even.rouault at mines-paris.org>
>         > To: gdal-dev at lists.osgeo.org; Etienne Tourigny
>         <etiennesky at yahoo.com>
>         > Cc:
>         > Sent: Friday, August 26, 2011 4:20:06 PM
>         > Subject: Re: [gdal-dev] Re: discussion on improvements to
>         the NetCDF driver and CF-1 convention
>         >
>         > Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a
>         écrit :
>         >
>         > My thoughts, just from a process point of view, not being
>         myself deeply
>         > involved in the NetCDF format :
>         >
>         > * A wiki page (I'd suggest either a new page -
>         > http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements - or at
>         least a clearly
>         > identified section in the current page, to avoid confusion
>         for outsiders about
>         > what is currently implemented and what is not) can be a
>         convenient way of
>         > structuring and editing the content by anyone with a osgeo
>         id. But discussions
>         > on the mailing list are also OK, at least to notify new
>         content has been added
>         > (provided that it doesn't cause flooding on the mailing
>         list ;-))
>         >
>         > * A http://download.osgeo.org/gdal/data/netcdf directory
>         would be the right
>         > place to store NetCDF sample files, provided they are free
>         under the terms of
>         > http://download.osgeo.org/gdal/data/COPYING . As write
>         access to it is
>         > restricted to people with appropriate rights, having those
>         rights, I can help
>         > uploading files over there. Having small (typically < or ~
>         10 KB ) sample files
>         > is also good so they can be directly included in SVN under
>         the
>         > autotest/gdrivers/data directory and thus easily used by
>         regression tests.
>         >
>         > * Tickets can be used to track the implementation of
>         specific items and
>         > bugfixes. But I've found they are not generally suitable for
>         brainstorming when
>         > the discussion is lasting, as you cannot edit already posted
>         content. Wiki is
>         > clearly better for this.
>         >
>         > * Looking at past history, I see we haven't yet used the RFC
>         formalism for
>         > work specific to one driver, while it doesn't require
>         changes or extensions to
>         > the GDAL API. But it wouldn't hurt to create one if the need
>         arises. Anyway, a
>         > point that might require care and some consensus among the
>         interested parties
>         > is about how backward compatibility issues, in particular
>         with respect to files
>         > produced by older GDAL versions, are adressed.
>         > http://trac.osgeo.org/gdal/wiki/rfc1_pmc details when a vote
>         is required.
>         >
>         > Best regards,
>         >
>         > Even
>         >
>         > > Hi all,
>         > >
>         > > Since a few people have shown an interest in helping out,
>         I would suggest
>         > > to set up space where we can post ideas, goals and test
>         files.  This could
>         > > take the form of one of the following:
>         > >
>         > > - a new wiki entry in trac, or the existing one at
>         > > http://trac.osgeo.org/gdal/wiki/NetCDF - a file repository
>         at
>         > > http://download.osgeo.org/gdal/data/netcdf as suggested
>         some time ago by
>         > > warmerdam
>         > >
>         > > and/or
>         > >
>         > > - a new ticket in trac (which allows to post files)
>         > >
>         > > What are you thoughts about this?  Would it make sense to
>         eventually
>         > > prepare an RFC?
>         > >
>         > >
>         > >
>         > > I would suggest that each group identify the problems with
>         the current
>         > > driver in terms of importing and exporting, post the files
>         that they would
>         > > like to be supported, which other software they use with
>         the netcdf files,
>         > > and their proposed contribution (code, test, ideas, etc.).
>         > >
>         > > Those that cannot participate in code development can help
>         out in testing
>         > > and discussion. Personally, I can write and test code,
>         given example files
>         > > and expected outcome.
>         > >
>         > >
>         > > Now to your specific comments, Knut-Frode:
>         > >
>         > > 1) Yes CF-1.5 should be the goal, although I'm not sure we
>         need to support
>         > > upgrading/downgrading.
>         > >
>         > > 5) The WGS84 issue needs further research. Weather and
>         ocean data are in
>         > > lat/lon, but what is the sphere radius assumed by the
>         various data
>         > > providers and  the models which output/import netcdf
>         files? I agree that a
>         > > small error does not make a difference with low-resolution
>         dataset, but
>         > > I'd like to be sure in any case. We also need a means to
>         translate the
>         > > CF-1.x datum information into EPSG/Proj.4 values.
>         > >
>         > >
>         > > 7) There is some support of a z-axis right now, in fact
>         they driver assumes
>         > > that a third dimension is vertical if I understand
>         correctly. However, I
>         > > have not tested nor looked into that.  This issue
>         definitely needs come
>         > > planning and an RFC, IMHO.
>         > >
>         > >
>         > > Kyle, thanks for you help and interest!
>         > >
>         > >
>         > > cheers,
>         > > Etienne
>         > >
>         > >
>         > > ________________________________
>         > > From: Knut-Frode Dagestad <knutfrodesoppel at hotmail.com>
>         > > To: gdal-dev at lists.osgeo.org
>         > > Sent: Friday, August 26, 2011 4:41:02 AM
>         > > Subject: [gdal-dev] Re: discussion on improvements to the
>         NetCDF driver and
>         > > CF-1 convention
>         > >
>         > > Etienne,
>         > >
>         > > My group is also very much in support of the suggested
>         improvements, as
>         > > NetCDF/CF is becoming the most important standard for data
>         > > storage/exchange in our community (climate, meteorology,
>         oceanography). My
>         > > group is not much into GDAL driver development (at least
>         not yet), but we
>         > > are now building our future very much around GDAL, so we
>         are interested to
>         > > join the discussion around improved support for NetCDF/CF
>         reading and
>         > > writing.
>         > >
>         > > About some of the mentioned issues:
>         > >
>         > > 1) The latest version of CF is 1.5, so it would be better
>         to target 1.5 (or
>         > > 1.x) than 1.0. Ideally the driver could be made generic
>         for smooth upgrade
>         > > to newer (or downgrade to older) versions of the CF
>         conventions.
>         > >
>         > > 5) It sounds meaningful to assume WGS84 datum if not
>         specified. I believe
>         > > at least 99% of data typically stored in NetCDF/CF will
>         have this datum,
>         > > and even when that is not the case, positioning errors
>         less than 100 m
>         > > should normally have little impact for the types of data.
>         Precision on
>         > > datum level is probably more relevant for more typical
>         GIS-data with
>         > > roads, borders etc. A warning could however be issued each
>         time the driver
>         > > has to assume WGS84.
>         > >
>         > > 7) We agree in particular that proper handling of time
>         axis is of high
>         > > value. This is one of the issues which is highly relevant
>         for climate-data
>         > > (often 4-dimensional), but not well supported by typical
>         GIS-software or
>         > > standards designed with 2-dimensional data in mind. A
>         general
>         > > representation of a time-axis (or even vertical axis) in
>         GDAL would be
>         > > ideal, but that would probably be too much an effort for
>         anyone to
>         > > develop.
>         > >
>         > >
>         > > Knut-Frode
>         > > Nansen Environmental and Remote Sensing Center
>         > >
>         > > On 24/08/2011 22:37, Etienne Tourigny wrote:
>         > > > Hi all,
>         > > >
>         > > > I would like to start a discussion with those interested
>         about fixing
>         > > > various issues in the NetCDF driver.
>         > > >
>         > > > As it stands, it is not fully CF-1.0 compliant, and
>         produces
>         > > > geographical grids that are not valid for other
>         software.
>         > > > Also, the infamous "up-side down" problem has been
>         addressed for
>         > > > importing netcdfs, but needs to be addressed for
>         exporting.
>         > > >
>         > > > Related ticket: http://trac.osgeo.org/gdal/ticket/2129
>         > > > I have been submitting patches for some time to fix the
>         related issues.
>         > > >
>         > > > I have identified a number of issues, of which the
>         following have been
>         > > > fixed partially in my proposed patch.
>         > > >
>         > > > 1- conform to the cf-1.0 standard for geographical grids
>         (creating lat
>         > > > and lon dimensions and variables)
>         > > > 2- make netcdf grids "bottom-up" by default, as all
>         software out there
>         > > > uses that convention
>         > > > 3- fix problems related to floating-point / string
>         conversions (also
>         > > > reported in issue #4200 )
>         > > > 4- fix metadata handling and duplication (as reported in
>         issue #4204),
>         > > > and keep band (variable) metadata outside of the global
>         metadata
>         > > >
>         > > > Pending issues:
>         > > >
>         > > > 5- how to deal with netcdfs which have no datum? Can we
>         assume WGS84?
>         > > > These files are extremely common, and CF-1.0 has
>         provisions for
>         > > > identifying the key variables, but no Proj/Wkt codes.
>         > > > 6- fix proper export of projected CRS (this is not fully
>         necessary
>         > > > though) with lat/lon values respecting CF-1.0
>         > > > 7- handle time axis properly (at least when creating a
>         netcdf) - that
>         > > > would be great for import/export of netcdf
>         > > >
>         > > > Regards,
>         > > > Etienne
>         > > >
>         > > >
>         > > > _______________________________________________
>         > > > gdal-dev mailing list
>         > > > gdal-dev at lists.osgeo.org
>         > > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>         > >
>         > > _______________________________________________
>         > > gdal-dev mailing list
>         > > gdal-dev at lists.osgeo.org
>         > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>         > > _______________________________________________
>         > > gdal-dev mailing list
>         > > gdal-dev at lists.osgeo.org
>         > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>         > _______________________________________________
>         > gdal-dev mailing list
>         > gdal-dev at lists.osgeo.org
>         > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>         _______________________________________________
>         gdal-dev mailing list
>         gdal-dev at lists.osgeo.org
>         http://lists.osgeo.org/mailman/listinfo/gdal-dev
>         
> 
> 




More information about the gdal-dev mailing list