<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hi Regis,</div>
<div><br>
</div>
<div>This is something that many of us in the MetOceans/Climate world are dealing with.</div>
<div><br>
</div>
<div>I don’t know of an simple answer at this stage.</div>
<div><br>
</div>
<div>For background, within the World Meteorological Organisation we are doing some foundation work to support our future time-series spatial information needs. </div>
<div><br>
</div>
<div>Have a look at WMO #1131, Climate Data Management System Specifications [1]. It provides a high level architectural overview of modern CDMS requirements.</div>
<div><br>
</div>
<div>While this document explicitly targets CDMS, you will see that they are quite broad and cover the needs of most of the MetOceans and other environmental domains as well.</div>
<div><br>
</div>
<div>I’m currently trying to get some funding to support the establishment of a reference open source CDMS called Open-CDMS. But this is a long uphill slog. Contact me off line if you’d like to know more.</div>
<div><br>
</div>
<div>So with that background, the short answer would be to store this data in a CDMS. However, these are typically not spatially or temporally enabled at present.</div>
<div><br>
</div>
<div>My gut feel is that Postgres/PostGIS may provide an excellent platform for observations data, but we have substantial work to do.</div>
<div><br>
</div>
<div>Let’s keep in contact. I’d be interested in your findings.</div>
<div><br>
</div>
<div><br>
</div>
<div>Bruce</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom: 16pt;"><b><span lang="EN-US">Bruce Bannerman </span></b><span lang="EN-US">|
</span><span lang="EN-US" style="font-size: 15pt; font-family: Arial;"><span style="font-family: Calibri, sans-serif; font-size: 14px;">Data Director </span><span lang="EN-US" style="font-family: Calibri, sans-serif; font-size: 14px;">(ac</span><span lang="EN-US" style="font-size: 15pt;"><span style="font-family: Calibri, sans-serif; font-size: 14px;">ting) </span></span><br>
</span><span lang="EN-US" style="font-size: 16pt; font-family: Calibri;"><img width="193" height="62" src="cid:C02A9194-8D9D-43F9-B59E-45F69787F002" v:shapes="Picture_x0020_1" type="image/png"></span><span lang="EN-US" style="font-size: 16pt; font-family: Calibri;"><br>
</span><span lang="EN-US">Bureau of Meteorology<br>
GPO Box 1289, Melbourne, Victoria 3001<br>
700 Collins Street, Docklands, Victoria 3008</span><span lang="EN-US"><br>
</span><span lang="EN-US">Australia</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">www.bom.gov.au</span></p>
</div>
</div>
</div>
<div>[1] <a href="http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300">http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300</a> </div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Qgis-user <<a href="mailto:qgis-user-bounces@lists.osgeo.org">qgis-user-bounces@lists.osgeo.org</a>> on behalf of Régis Haubourg <<a href="mailto:regis.haubourg@gmail.com">regis.haubourg@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, 17 June 2016 at 19:29<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>" <<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Qgis-user] best data storage fo time series visualisation in QGIS<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div dir="ltr">Hi,
<div>I need the communities lights!</div>
<div><br>
<div>I'm starting to work with huge meteo datasets composed of a grid of point layers, and hundred of millions of rainfall / temperature data. </div>
<div><br>
</div>
<div>Datasets are delivered in a custom text format, so I'm digging around on what are the best formats for storage, use in postgis and QGIS. </div>
<div>I would like to be able to :</div>
<div> - run timeManager to generate videos</div>
<div> - display data averaged on day / month / year (or any other) timeframe</div>
<div> - feed R analyses. </div>
<div><br>
</div>
<div>Up to now, I tried the following paths: </div>
<div> - netcdf / grib: ideal for data storage: </div>
<div> Pros : GDAL and QGIS can view it. R And python scipy have providers for that</div>
<div> Cons : not easy to generate from exotic datasources, Current QGIS Netcdf explorer or core date visualisation (time frame = raster bands) are not handy for daily data over decades (about 10 000 days available in my dataset). I didn't manage
to build netcdf yet, FME or GDAL are a bit dry.. </div>
<div><br>
</div>
<div> - load all in postgres / postgis relationnal model: </div>
<div> Pros: available for all clients and fast, if data is correctly indexed and designed/ </div>
<div> Cons: performance requires a table (not a view because of lack of estimated metadata for extent computing) with redondancy over point location. I tried a first approach with a small geographical table for my point grid and a value table. With
correct indexing and clustering, I get good performance in psql but very poor in QGIS. First load is slooow because of the st_extent query, but also every fetch afterwards, even if I filter on a date frame (with good index). I didn't expect it to be slow on
fetch.. </div>
<div><br>
</div>
<div>Another point with postgis storage, TimeManager plugin does not like true date datatype, date cast to char truncate date to first character, so I have to expose my datasets with a text format in my view, which is not quite efficient. (I will create a
ticket upstream) </div>
<div><br>
</div>
<div><b>Does anyone has any experience and advices on that field ? </b></div>
<div><b><br>
</b></div>
<div> I saw that postgis has a datacube type, could that be a way to store data more efficiently? Could QGIS read it? Should I stick with netcdf ? </div>
<div><br>
</div>
<div>
<div>Thanks a lot </div>
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">Régis <br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
</body>
</html>