[Qgis-user] best data storage fo time series visualisation in QGIS

[Régis Haubourg]

Hi,
I need the communities lights!

I'm starting to work with huge meteo datasets composed of a grid of point
layers, and hundred of millions of rainfall / temperature data.

Datasets are delivered in a custom text format, so I'm digging around on
what are the best formats for storage, use in postgis and QGIS.
I would like to be able to :
 - run timeManager to generate videos
 - display data averaged on day / month / year (or any other) timeframe
 - feed R analyses.

Up to now, I tried the following paths:
  - netcdf  / grib:  ideal for data storage:
          Pros : GDAL and QGIS can view it. R And python scipy have
providers for that
          Cons : not easy to generate from exotic datasources, Current QGIS
Netcdf explorer or core date visualisation (time frame = raster bands)  are
not handy for daily data over decades (about 10 000 days available in my
dataset).  I didn't manage to build netcdf yet, FME or GDAL are a bit dry..

 - load all in postgres / postgis relationnal model:
          Pros: available for all clients and fast, if data is correctly
indexed and designed/
          Cons: performance requires a table (not a view because of lack of
estimated metadata for extent computing) with redondancy over point
location. I tried a first approach with a small geographical table for my
point grid and a value table. With correct indexing and clustering, I get
good performance in psql but very poor in QGIS. First load is slooow
because of the st_extent query, but also every fetch afterwards, even if I
filter on a date frame (with good index). I didn't expect it to be slow on
fetch..

Another point with postgis storage, TimeManager plugin does not like true
date datatype, date cast to char truncate date to first character,  so I
have to expose my datasets with a text format in my view, which is not
quite efficient.  (I will create a ticket upstream)

*Does anyone has any experience and advices on that field ? *

 I saw that postgis has a datacube type, could that be a way to store data
more efficiently? Could QGIS read it?  Should I stick with netcdf ?

Thanks a lot

-- 
Régis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160617/f68087c3/attachment.html>