[Qgis-user] best data storage fo time series visualisation in QGIS

[Andrew Harfoot]

I saw a demo of the Crayfish plugin at the FOSS4G-UK conference earlier 
in the week, and it provides 'time slider' functionality for netCDF 
based data, plus additional tools such as video generation. Worth taking 
a look?

Andy

On 17/06/2016 10:29, Régis Haubourg wrote:
> Hi,
> I need the communities lights!
>
> I'm starting to work with huge meteo datasets composed of a grid of 
> point layers, and hundred of millions of rainfall / temperature data.
>
> Datasets are delivered in a custom text format, so I'm digging around 
> on what are the best formats for storage, use in postgis and QGIS.
> I would like to be able to :
>  - run timeManager to generate videos
>  - display data averaged on day / month / year (or any other) timeframe
>  - feed R analyses.
>
> Up to now, I tried the following paths:
>   - netcdf  / grib:  ideal for data storage:
>           Pros : GDAL and QGIS can view it. R And python scipy have 
> providers for that
>           Cons : not easy to generate from exotic datasources, Current 
> QGIS Netcdf explorer or core date visualisation (time frame = raster 
> bands)  are not handy for daily data over decades (about 10 000 days 
> available in my dataset).  I didn't manage to build netcdf yet, FME or 
> GDAL are a bit dry..
>
>  - load all in postgres / postgis relationnal model:
>           Pros: available for all clients and fast, if data is 
> correctly indexed and designed/
>           Cons: performance requires a table (not a view because of 
> lack of estimated metadata for extent computing) with redondancy over 
> point location. I tried a first approach with a small geographical 
> table for my point grid and a value table. With correct indexing and 
> clustering, I get good performance in psql but very poor in QGIS. 
> First load is slooow because of the st_extent query, but also every 
> fetch afterwards, even if I filter on a date frame (with good index). 
> I didn't expect it to be slow on fetch..
>
> Another point with postgis storage, TimeManager plugin does not like 
> true date datatype, date cast to char truncate date to first 
> character,  so I have to expose my datasets with a text format in my 
> view, which is not quite efficient.  (I will create a ticket upstream)
>
> *Does anyone has any experience and advices on that field ? *
> *
> *
>  I saw that postgis has a datacube type, could that be a way to store 
> data more efficiently? Could QGIS read it? Should I stick with netcdf ?
>
> Thanks a lot
>
> -- 
> Régis
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-user


-- 
Andy Harfoot

GeoData Institute
University of Southampton
Southampton
SO17 1BJ

Tel:  +44 (0)23 8059 2719
Fax:  +44 (0)23 8059 2849

www.geodata.soton.ac.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160617/f97e8b38/attachment.html>