There has been recent work on incorporating netcdf support for point and trajectory data with Talend, maybe that can help?<br><div><br></div><div><a href="http://www.neogeo-online.net/blog/archives/1350/">http://www.neogeo-online.net/blog/archives/1350/</a></div>
<div><a href="http://talendforge.org/svn/sdi/trunk/org.talend.sdi.designer.components.sandbox/components/sNetCDFInput/">http://talendforge.org/svn/sdi/trunk/org.talend.sdi.designer.components.sandbox/components/sNetCDFInput/</a></div>
<div><br></div><div>Etienne<br><br><div class="gmail_quote">On Tue, Aug 30, 2011 at 10:18 AM, Etienne Tourigny <span dir="ltr"><<a href="mailto:etiennesky@yahoo.com">etiennesky@yahoo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Brian,<br>
<br>
I am not familiar with madis data. I assume that it is point-data for meteorological stations?<br>
<br>
If you like, please post your ideas and a short description in a new topic in the wiki entry at<br>
<a href="http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements" target="_blank">http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements</a> .<br>
<font color="#888888"><br>
Etienne<br>
</font><div><div></div><div class="h5"><br>
<br>
<br>
----- Original Message -----<br>
From: Brian Case <<a href="mailto:rush@winkey.org">rush@winkey.org</a>><br>
To: Etienne Tourigny <<a href="mailto:etiennesky@yahoo.com">etiennesky@yahoo.com</a>><br>
Cc: "<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>" <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>><br>
Sent: Sunday, August 28, 2011 7:54:13 PM<br>
Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention<br>
<br>
any thoughts on a ogr netcdf driver? I have a simple one for madis data<br>
that needs work in my sandbox<br>
<br>
Brian<br>
<br>
<br>
<br>
On Sun, 2011-08-28 at 14:56 -0700, Etienne Tourigny wrote:<br>
> I have created a new wiki page at <a href="http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements" target="_blank">http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements</a> , following Even's suggestion (Thanks Even).<br>
><br>
> The following topics have been included:<br>
><br>
><br>
> Topics:<br>
><br>
> 1 - UNIDATA NetCDF Conventions<br>
><br>
> 2 - Metadata<br>
> 3 - Datum issues<br>
> 4 - Dimension issues<br>
> 5 - Compatibility with other software<br>
> 6 - Test files<br>
> 7 - Contribution<br>
> Discussion<br>
><br>
> I encourage interested parties in participating in this discussion in the wiki and in the mailing list.<br>
><br>
><br>
> In particular, it would be useful to collect information on contributors and obtain test files with issues and the expected outcome.<br>
><br>
> regards, Etienne<br>
><br>
><br>
> ----- Original Message -----<br>
> From: Even Rouault <<a href="mailto:even.rouault@mines-paris.org">even.rouault@mines-paris.org</a>><br>
> To: <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>; Etienne Tourigny <<a href="mailto:etiennesky@yahoo.com">etiennesky@yahoo.com</a>><br>
> Cc:<br>
> Sent: Friday, August 26, 2011 4:20:06 PM<br>
> Subject: Re: [gdal-dev] Re: discussion on improvements to the NetCDF driver and CF-1 convention<br>
><br>
> Le vendredi 26 août 2011 17:15:47, Etienne Tourigny a écrit :<br>
><br>
> My thoughts, just from a process point of view, not being myself deeply<br>
> involved in the NetCDF format :<br>
><br>
> * A wiki page (I'd suggest either a new page -<br>
> <a href="http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements" target="_blank">http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements</a> - or at least a clearly<br>
> identified section in the current page, to avoid confusion for outsiders about<br>
> what is currently implemented and what is not) can be a convenient way of<br>
> structuring and editing the content by anyone with a osgeo id. But discussions<br>
> on the mailing list are also OK, at least to notify new content has been added<br>
> (provided that it doesn't cause flooding on the mailing list ;-))<br>
><br>
> * A <a href="http://download.osgeo.org/gdal/data/netcdf" target="_blank">http://download.osgeo.org/gdal/data/netcdf</a> directory would be the right<br>
> place to store NetCDF sample files, provided they are free under the terms of<br>
> <a href="http://download.osgeo.org/gdal/data/COPYING" target="_blank">http://download.osgeo.org/gdal/data/COPYING</a> . As write access to it is<br>
> restricted to people with appropriate rights, having those rights, I can help<br>
> uploading files over there. Having small (typically < or ~ 10 KB ) sample files<br>
> is also good so they can be directly included in SVN under the<br>
> autotest/gdrivers/data directory and thus easily used by regression tests.<br>
><br>
> * Tickets can be used to track the implementation of specific items and<br>
> bugfixes. But I've found they are not generally suitable for brainstorming when<br>
> the discussion is lasting, as you cannot edit already posted content. Wiki is<br>
> clearly better for this.<br>
><br>
> * Looking at past history, I see we haven't yet used the RFC formalism for<br>
> work specific to one driver, while it doesn't require changes or extensions to<br>
> the GDAL API. But it wouldn't hurt to create one if the need arises. Anyway, a<br>
> point that might require care and some consensus among the interested parties<br>
> is about how backward compatibility issues, in particular with respect to files<br>
> produced by older GDAL versions, are adressed.<br>
> <a href="http://trac.osgeo.org/gdal/wiki/rfc1_pmc" target="_blank">http://trac.osgeo.org/gdal/wiki/rfc1_pmc</a> details when a vote is required.<br>
><br>
> Best regards,<br>
><br>
> Even<br>
><br>
> > Hi all,<br>
> ><br>
> > Since a few people have shown an interest in helping out, I would suggest<br>
> > to set up space where we can post ideas, goals and test files. This could<br>
> > take the form of one of the following:<br>
> ><br>
> > - a new wiki entry in trac, or the existing one at<br>
> > <a href="http://trac.osgeo.org/gdal/wiki/NetCDF" target="_blank">http://trac.osgeo.org/gdal/wiki/NetCDF</a> - a file repository at<br>
> > <a href="http://download.osgeo.org/gdal/data/netcdf" target="_blank">http://download.osgeo.org/gdal/data/netcdf</a> as suggested some time ago by<br>
> > warmerdam<br>
> ><br>
> > and/or<br>
> ><br>
> > - a new ticket in trac (which allows to post files)<br>
> ><br>
> > What are you thoughts about this? Would it make sense to eventually<br>
> > prepare an RFC?<br>
> ><br>
> ><br>
> ><br>
> > I would suggest that each group identify the problems with the current<br>
> > driver in terms of importing and exporting, post the files that they would<br>
> > like to be supported, which other software they use with the netcdf files,<br>
> > and their proposed contribution (code, test, ideas, etc.).<br>
> ><br>
> > Those that cannot participate in code development can help out in testing<br>
> > and discussion. Personally, I can write and test code, given example files<br>
> > and expected outcome.<br>
> ><br>
> ><br>
> > Now to your specific comments, Knut-Frode:<br>
> ><br>
> > 1) Yes CF-1.5 should be the goal, although I'm not sure we need to support<br>
> > upgrading/downgrading.<br>
> ><br>
> > 5) The WGS84 issue needs further research. Weather and ocean data are in<br>
> > lat/lon, but what is the sphere radius assumed by the various data<br>
> > providers and the models which output/import netcdf files? I agree that a<br>
> > small error does not make a difference with low-resolution dataset, but<br>
> > I'd like to be sure in any case. We also need a means to translate the<br>
> > CF-1.x datum information into EPSG/Proj.4 values.<br>
> ><br>
> ><br>
> > 7) There is some support of a z-axis right now, in fact they driver assumes<br>
> > that a third dimension is vertical if I understand correctly. However, I<br>
> > have not tested nor looked into that. This issue definitely needs come<br>
> > planning and an RFC, IMHO.<br>
> ><br>
> ><br>
> > Kyle, thanks for you help and interest!<br>
> ><br>
> ><br>
> > cheers,<br>
> > Etienne<br>
> ><br>
> ><br>
> > ________________________________<br>
> > From: Knut-Frode Dagestad <<a href="mailto:knutfrodesoppel@hotmail.com">knutfrodesoppel@hotmail.com</a>><br>
> > To: <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> > Sent: Friday, August 26, 2011 4:41:02 AM<br>
> > Subject: [gdal-dev] Re: discussion on improvements to the NetCDF driver and<br>
> > CF-1 convention<br>
> ><br>
> > Etienne,<br>
> ><br>
> > My group is also very much in support of the suggested improvements, as<br>
> > NetCDF/CF is becoming the most important standard for data<br>
> > storage/exchange in our community (climate, meteorology, oceanography). My<br>
> > group is not much into GDAL driver development (at least not yet), but we<br>
> > are now building our future very much around GDAL, so we are interested to<br>
> > join the discussion around improved support for NetCDF/CF reading and<br>
> > writing.<br>
> ><br>
> > About some of the mentioned issues:<br>
> ><br>
> > 1) The latest version of CF is 1.5, so it would be better to target 1.5 (or<br>
> > 1.x) than 1.0. Ideally the driver could be made generic for smooth upgrade<br>
> > to newer (or downgrade to older) versions of the CF conventions.<br>
> ><br>
> > 5) It sounds meaningful to assume WGS84 datum if not specified. I believe<br>
> > at least 99% of data typically stored in NetCDF/CF will have this datum,<br>
> > and even when that is not the case, positioning errors less than 100 m<br>
> > should normally have little impact for the types of data. Precision on<br>
> > datum level is probably more relevant for more typical GIS-data with<br>
> > roads, borders etc. A warning could however be issued each time the driver<br>
> > has to assume WGS84.<br>
> ><br>
> > 7) We agree in particular that proper handling of time axis is of high<br>
> > value. This is one of the issues which is highly relevant for climate-data<br>
> > (often 4-dimensional), but not well supported by typical GIS-software or<br>
> > standards designed with 2-dimensional data in mind. A general<br>
> > representation of a time-axis (or even vertical axis) in GDAL would be<br>
> > ideal, but that would probably be too much an effort for anyone to<br>
> > develop.<br>
> ><br>
> ><br>
> > Knut-Frode<br>
> > Nansen Environmental and Remote Sensing Center<br>
> ><br>
> > On 24/08/2011 22:37, Etienne Tourigny wrote:<br>
> > > Hi all,<br>
> > ><br>
> > > I would like to start a discussion with those interested about fixing<br>
> > > various issues in the NetCDF driver.<br>
> > ><br>
> > > As it stands, it is not fully CF-1.0 compliant, and produces<br>
> > > geographical grids that are not valid for other software.<br>
> > > Also, the infamous "up-side down" problem has been addressed for<br>
> > > importing netcdfs, but needs to be addressed for exporting.<br>
> > ><br>
> > > Related ticket: <a href="http://trac.osgeo.org/gdal/ticket/2129" target="_blank">http://trac.osgeo.org/gdal/ticket/2129</a><br>
> > > I have been submitting patches for some time to fix the related issues.<br>
> > ><br>
> > > I have identified a number of issues, of which the following have been<br>
> > > fixed partially in my proposed patch.<br>
> > ><br>
> > > 1- conform to the cf-1.0 standard for geographical grids (creating lat<br>
> > > and lon dimensions and variables)<br>
> > > 2- make netcdf grids "bottom-up" by default, as all software out there<br>
> > > uses that convention<br>
> > > 3- fix problems related to floating-point / string conversions (also<br>
> > > reported in issue #4200 )<br>
> > > 4- fix metadata handling and duplication (as reported in issue #4204),<br>
> > > and keep band (variable) metadata outside of the global metadata<br>
> > ><br>
> > > Pending issues:<br>
> > ><br>
> > > 5- how to deal with netcdfs which have no datum? Can we assume WGS84?<br>
> > > These files are extremely common, and CF-1.0 has provisions for<br>
> > > identifying the key variables, but no Proj/Wkt codes.<br>
> > > 6- fix proper export of projected CRS (this is not fully necessary<br>
> > > though) with lat/lon values respecting CF-1.0<br>
> > > 7- handle time axis properly (at least when creating a netcdf) - that<br>
> > > would be great for import/export of netcdf<br>
> > ><br>
> > > Regards,<br>
> > > Etienne<br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > gdal-dev mailing list<br>
> > > <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> > > <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
> ><br>
> > _______________________________________________<br>
> > gdal-dev mailing list<br>
> > <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> > <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
> > _______________________________________________<br>
> > gdal-dev mailing list<br>
> > <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> > <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
> _______________________________________________<br>
> gdal-dev mailing list<br>
> <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
> <a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</div></div></blockquote></div><br></div>