[Qgis-user] best data storage fo time series visualisation in QGIS [SEC=UNCLASSIFIED]

[Bruce Bannerman]

Hi Regis,

This is something that many of us in the MetOceans/Climate world are dealing with.

I don't know of an simple answer at this stage.

For background, within the World Meteorological Organisation we are doing some foundation work to support our future time-series spatial information needs.

Have a look at WMO #1131, Climate Data Management System Specifications [1]. It provides a high level architectural overview of modern CDMS requirements.

While this document explicitly targets CDMS, you will see that they are quite broad and cover the needs of most of the MetOceans and other environmental domains as well.

I'm currently trying to get some funding to support the establishment of a reference open source CDMS called Open-CDMS. But this is a long uphill slog. Contact me off line if you'd like to know more.

So with that background, the short answer would be to store this data in a CDMS. However, these are typically not spatially or temporally enabled at present.

My gut feel is that Postgres/PostGIS may provide an excellent platform for observations data, but we have substantial work to do.

Let's keep in contact. I'd be interested in your findings.


Bruce
Bruce Bannerman | Data Director (acting)
 [cid:C02A9194-8D9D-43F9-B59E-45F69787F002]
Bureau of Meteorology
GPO Box 1289, Melbourne, Victoria 3001
700 Collins Street, Docklands, Victoria 3008
Australia
www.bom.gov.au
[1] http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300


From: Qgis-user <qgis-user-bounces at lists.osgeo.org<mailto:qgis-user-bounces at lists.osgeo.org>> on behalf of Régis Haubourg <regis.haubourg at gmail.com<mailto:regis.haubourg at gmail.com>>
Date: Friday, 17 June 2016 at 19:29
To: "qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>" <qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>>
Subject: [Qgis-user] best data storage fo time series visualisation in QGIS

Hi,
I need the communities lights!

I'm starting to work with huge meteo datasets composed of a grid of point layers, and hundred of millions of rainfall / temperature data.

Datasets are delivered in a custom text format, so I'm digging around on what are the best formats for storage, use in postgis and QGIS.
I would like to be able to :
 - run timeManager to generate videos
 - display data averaged on day / month / year (or any other) timeframe
 - feed R analyses.

Up to now, I tried the following paths:
  - netcdf  / grib:  ideal for data storage:
          Pros : GDAL and QGIS can view it. R And python scipy have providers for that
          Cons : not easy to generate from exotic datasources, Current QGIS Netcdf explorer or core date visualisation (time frame = raster bands)  are not handy for daily data over decades (about 10 000 days available in my dataset).  I didn't manage to build netcdf yet, FME or GDAL are a bit dry..

 - load all in postgres / postgis relationnal model:
          Pros: available for all clients and fast, if data is correctly indexed and designed/
          Cons: performance requires a table (not a view because of lack of estimated metadata for extent computing) with redondancy over point location. I tried a first approach with a small geographical table for my point grid and a value table. With correct indexing and clustering, I get good performance in psql but very poor in QGIS. First load is slooow because of the st_extent query, but also every fetch afterwards, even if I filter on a date frame (with good index). I didn't expect it to be slow on fetch..

Another point with postgis storage, TimeManager plugin does not like true date datatype, date cast to char truncate date to first character,  so I have to expose my datasets with a text format in my view, which is not quite efficient.  (I will create a ticket upstream)

Does anyone has any experience and advices on that field ?

 I saw that postgis has a datacube type, could that be a way to store data more efficiently? Could QGIS read it?  Should I stick with netcdf ?

Thanks a lot

--
Régis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160620/e992babf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1BDB09EC-D554-4372-9C80-E25125AC6C3A.png
Type: image/png
Size: 7308 bytes
Desc: 1BDB09EC-D554-4372-9C80-E25125AC6C3A.png
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160620/e992babf/attachment.png>