[Qgis-user] best data storage fo time series visualisation in QGIS [SEC=UNCLASSIFIED]

Mon Jun 20 11:09:29 PDT 2016

On pandas, I totally agree!

https://trac.osgeo.org/osgeo4w/ticket/472

Cheers
Chris

From: Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org] On Behalf Of Régis Haubourg
Sent: Tuesday, 21 June 2016 2:17 a.m.
To: qgis-user at lists.osgeo.org
Subject: Re: [Qgis-user] best data storage fo time series visualisation in QGIS [SEC=UNCLASSIFIED]

Thanks Bruce for the inputs, I realize that it is a wide domain there.

In the meanwhile, I start to get good result at netcdf generation using python pandas and xarray. I will keep you informed of how this turns in production use with some data, we will tes both netcdf and postgres models.

BTW, I'm starting to think that pandas could be a great addition to OSGEO4W, I add it every time in my deployement package, together with ipython and jupyter notebooks. OSGEO4W with R, grass and the python glue is now a great framework for reproducible research.

Cheers
Régis

2016-06-20 2:11 GMT+02:00 Bruce Bannerman <B.Bannerman at bom.gov.au<mailto:B.Bannerman at bom.gov.au>>:
Hi Regis,

This is something that many of us in the MetOceans/Climate world are dealing with.

I don’t know of an simple answer at this stage.

For background, within the World Meteorological Organisation we are doing some foundation work to support our future time-series spatial information needs.

Have a look at WMO #1131, Climate Data Management System Specifications [1]. It provides a high level architectural overview of modern CDMS requirements.

While this document explicitly targets CDMS, you will see that they are quite broad and cover the needs of most of the MetOceans and other environmental domains as well.

I’m currently trying to get some funding to support the establishment of a reference open source CDMS called Open-CDMS. But this is a long uphill slog. Contact me off line if you’d like to know more.

So with that background, the short answer would be to store this data in a CDMS. However, these are typically not spatially or temporally enabled at present.

My gut feel is that Postgres/PostGIS may provide an excellent platform for observations data, but we have substantial work to do.

Let’s keep in contact. I’d be interested in your findings.

Bruce
Bruce Bannerman | Data Director (acting)
 [cid:image001.png at 01D1CB83.5FFD11B0]
Bureau of Meteorology
GPO Box 1289, Melbourne, Victoria 3001
700 Collins Street, Docklands, Victoria 3008
Australia
www.bom.gov.au<http://www.bom.gov.au>
[1] http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300

From: Qgis-user <qgis-user-bounces at lists.osgeo.org<mailto:qgis-user-bounces at lists.osgeo.org>> on behalf of Régis Haubourg <regis.haubourg at gmail.com<mailto:regis.haubourg at gmail.com>>
Date: Friday, 17 June 2016 at 19:29
To: "qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>" <qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>>
Subject: [Qgis-user] best data storage fo time series visualisation in QGIS

Hi,
I need the communities lights!

I'm starting to work with huge meteo datasets composed of a grid of point layers, and hundred of millions of rainfall / temperature data.

Datasets are delivered in a custom text format, so I'm digging around on what are the best formats for storage, use in postgis and QGIS.
I would like to be able to :
 - run timeManager to generate videos
 - display data averaged on day / month / year (or any other) timeframe
 - feed R analyses.

Up to now, I tried the following paths:
  - netcdf  / grib:  ideal for data storage:
          Pros : GDAL and QGIS can view it. R And python scipy have providers for that
          Cons : not easy to generate from exotic datasources, Current QGIS Netcdf explorer or core date visualisation (time frame = raster bands)  are not handy for daily data over decades (about 10 000 days available in my dataset).  I didn't manage to build netcdf yet, FME or GDAL are a bit dry..

 - load all in postgres / postgis relationnal model:
          Pros: available for all clients and fast, if data is correctly indexed and designed/
          Cons: performance requires a table (not a view because of lack of estimated metadata for extent computing) with redondancy over point location. I tried a first approach with a small geographical table for my point grid and a value table. With correct indexing and clustering, I get good performance in psql but very poor in QGIS. First load is slooow because of the st_extent query, but also every fetch afterwards, even if I filter on a date frame (with good index). I didn't expect it to be slow on fetch..

Another point with postgis storage, TimeManager plugin does not like true date datatype, date cast to char truncate date to first character,  so I have to expose my datasets with a text format in my view, which is not quite efficient.  (I will create a ticket upstream)

Does anyone has any experience and advices on that field ?

 I saw that postgis has a datacube type, could that be a way to store data more efficiently? Could QGIS read it?  Should I stick with netcdf ?

Thanks a lot

--
Régis

--
Régis Haubourg

Attention, changement d'adresse mail!
Mon adresse principale devient désormais regis.haubourg at gmail.com<http://gmail.com>

________________________________
This message contains information, which may be in confidence and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or info at linz.govt.nz) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160621/08fdb7d2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7308 bytes
Desc: image001.png
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160621/08fdb7d2/attachment.png>