[Qgis-user] best data storage fo time series visualisation in QGIS

[Blumentrath, Stefan]

Hi,

What about using Grass Data Explorer (https://bitbucket.org/huhabla/grass-data-explorer)?
Please find a demo here:
https://www.youtube.com/watch?v=xxHt3jJbnYw

Lots of visualization, animation and analysis possibilities thanks to the temporal framework in GRASS (TGIS)...
However, for vector data TGIS might need some love...

If you go for PostGIS, use table partitioning for efficiency!

Cheers
Stefan

From: Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org] On Behalf Of Régis Haubourg
Sent: 17. juni 2016 11:29
To: qgis-user at lists.osgeo.org
Subject: [Qgis-user] best data storage fo time series visualisation in QGIS

Hi,
I need the communities lights!

I'm starting to work with huge meteo datasets composed of a grid of point layers, and hundred of millions of rainfall / temperature data.

Datasets are delivered in a custom text format, so I'm digging around on what are the best formats for storage, use in postgis and QGIS.
I would like to be able to :
 - run timeManager to generate videos
 - display data averaged on day / month / year (or any other) timeframe
 - feed R analyses.

Up to now, I tried the following paths:
  - netcdf  / grib:  ideal for data storage:
          Pros : GDAL and QGIS can view it. R And python scipy have providers for that
          Cons : not easy to generate from exotic datasources, Current QGIS Netcdf explorer or core date visualisation (time frame = raster bands)  are not handy for daily data over decades (about 10 000 days available in my dataset).  I didn't manage to build netcdf yet, FME or GDAL are a bit dry..

 - load all in postgres / postgis relationnal model:
          Pros: available for all clients and fast, if data is correctly indexed and designed/
          Cons: performance requires a table (not a view because of lack of estimated metadata for extent computing) with redondancy over point location. I tried a first approach with a small geographical table for my point grid and a value table. With correct indexing and clustering, I get good performance in psql but very poor in QGIS. First load is slooow because of the st_extent query, but also every fetch afterwards, even if I filter on a date frame (with good index). I didn't expect it to be slow on fetch..

Another point with postgis storage, TimeManager plugin does not like true date datatype, date cast to char truncate date to first character,  so I have to expose my datasets with a text format in my view, which is not quite efficient.  (I will create a ticket upstream)

Does anyone has any experience and advices on that field ?

 I saw that postgis has a datacube type, could that be a way to store data more efficiently? Could QGIS read it?  Should I stick with netcdf ?

Thanks a lot

--
Régis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160617/3eabce08/attachment.html>