<div dir="ltr">Thanks Bruce for the inputs, I realize that it is a wide domain there. <div><br></div><div>In the meanwhile, I start to get good result at netcdf generation using python pandas and xarray. I will keep you informed of how this turns in production use with some data, we will tes both netcdf and postgres models. </div><div><br></div><div>BTW, I'm starting to think that pandas could be a great addition to OSGEO4W, I add it every time in my deployement package, together with ipython and jupyter notebooks. OSGEO4W with R, grass and the python glue is now a great framework for reproducible research. </div><div><br></div><div>Cheers</div><div>Régis</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-06-20 2:11 GMT+02:00 Bruce Bannerman <span dir="ltr"><<a href="mailto:B.Bannerman@bom.gov.au" target="_blank">B.Bannerman@bom.gov.au</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Hi Regis,</div>
<div><br>
</div>
<div>This is something that many of us in the MetOceans/Climate world are dealing with.</div>
<div><br>
</div>
<div>I don’t know of an simple answer at this stage.</div>
<div><br>
</div>
<div>For background, within the World Meteorological Organisation we are doing some foundation work to support our future time-series spatial information needs. </div>
<div><br>
</div>
<div>Have a look at WMO #1131, Climate Data Management System Specifications [1]. It provides a high level architectural overview of modern CDMS requirements.</div>
<div><br>
</div>
<div>While this document explicitly targets CDMS, you will see that they are quite broad and cover the needs of most of the MetOceans and other environmental domains as well.</div>
<div><br>
</div>
<div>I’m currently trying to get some funding to support the establishment of a reference open source CDMS called Open-CDMS. But this is a long uphill slog. Contact me off line if you’d like to know more.</div>
<div><br>
</div>
<div>So with that background, the short answer would be to store this data in a CDMS. However, these are typically not spatially or temporally enabled at present.</div>
<div><br>
</div>
<div>My gut feel is that Postgres/PostGIS may provide an excellent platform for observations data, but we have substantial work to do.</div>
<div><br>
</div>
<div>Let’s keep in contact. I’d be interested in your findings.</div>
<div><br>
</div>
<div><br>
</div>
<div>Bruce</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:16pt"><b><span lang="EN-US">Bruce Bannerman </span></b><span lang="EN-US">|
</span><span lang="EN-US" style="font-size:15pt;font-family:Arial"><span style="font-family:Calibri,sans-serif;font-size:14px">Data Director </span><span lang="EN-US" style="font-family:Calibri,sans-serif;font-size:14px">(ac</span><span lang="EN-US" style="font-size:15pt"><span style="font-family:Calibri,sans-serif;font-size:14px">ting) </span></span><br>
</span><span lang="EN-US" style="font-size:16pt;font-family:Calibri"><img width="193" height="62" src="cid:C02A9194-8D9D-43F9-B59E-45F69787F002" type="image/png"></span><span lang="EN-US" style="font-size:16pt;font-family:Calibri"><br>
</span><span lang="EN-US">Bureau of Meteorology<br>
GPO Box 1289, Melbourne, Victoria 3001<br>
700 Collins Street, Docklands, Victoria 3008</span><span lang="EN-US"><br>
</span><span lang="EN-US">Australia</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="http://www.bom.gov.au" target="_blank">www.bom.gov.au</a></span></p>
</div>
</div>
</div>
<div>[1] <a href="http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300" target="_blank">http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300</a> </div>
<div><br>
</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Qgis-user <<a href="mailto:qgis-user-bounces@lists.osgeo.org" target="_blank">qgis-user-bounces@lists.osgeo.org</a>> on behalf of Régis Haubourg <<a href="mailto:regis.haubourg@gmail.com" target="_blank">regis.haubourg@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, 17 June 2016 at 19:29<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:qgis-user@lists.osgeo.org" target="_blank">qgis-user@lists.osgeo.org</a>" <<a href="mailto:qgis-user@lists.osgeo.org" target="_blank">qgis-user@lists.osgeo.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Qgis-user] best data storage fo time series visualisation in QGIS<br>
</div>
<div><br>
</div>
<blockquote style="BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5">
<div>
<div>
<div dir="ltr">Hi,
<div>I need the communities lights!</div>
<div><br>
<div>I'm starting to work with huge meteo datasets composed of a grid of point layers, and hundred of millions of rainfall / temperature data. </div>
<div><br>
</div>
<div>Datasets are delivered in a custom text format, so I'm digging around on what are the best formats for storage, use in postgis and QGIS. </div>
<div>I would like to be able to :</div>
<div> - run timeManager to generate videos</div>
<div> - display data averaged on day / month / year (or any other) timeframe</div>
<div> - feed R analyses. </div>
<div><br>
</div>
<div>Up to now, I tried the following paths: </div>
<div> - netcdf / grib: ideal for data storage: </div>
<div> Pros : GDAL and QGIS can view it. R And python scipy have providers for that</div>
<div> Cons : not easy to generate from exotic datasources, Current QGIS Netcdf explorer or core date visualisation (time frame = raster bands) are not handy for daily data over decades (about 10 000 days available in my dataset). I didn't manage
to build netcdf yet, FME or GDAL are a bit dry.. </div>
<div><br>
</div>
<div> - load all in postgres / postgis relationnal model: </div>
<div> Pros: available for all clients and fast, if data is correctly indexed and designed/ </div>
<div> Cons: performance requires a table (not a view because of lack of estimated metadata for extent computing) with redondancy over point location. I tried a first approach with a small geographical table for my point grid and a value table. With
correct indexing and clustering, I get good performance in psql but very poor in QGIS. First load is slooow because of the st_extent query, but also every fetch afterwards, even if I filter on a date frame (with good index). I didn't expect it to be slow on
fetch.. </div>
<div><br>
</div>
<div>Another point with postgis storage, TimeManager plugin does not like true date datatype, date cast to char truncate date to first character, so I have to expose my datasets with a text format in my view, which is not quite efficient. (I will create a
ticket upstream) </div>
<div><br>
</div>
<div><b>Does anyone has any experience and advices on that field ? </b></div>
<div><b><br>
</b></div>
<div> I saw that postgis has a datacube type, could that be a way to store data more efficiently? Could QGIS read it? Should I stick with netcdf ? </div>
<div><br>
</div>
<div>
<div>Thanks a lot </div><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div data-smartmail="gmail_signature">Régis <br>
</div>
</font></span></div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Régis Haubourg<br><br>Attention, changement d'adresse mail! <br>Mon adresse principale devient désormais regis.haubourg at <a href="http://gmail.com" target="_blank">gmail.com</a><br><br></div>
</div>