[Qgis-user] best data storage fo time series visualisation in QGIS [SEC=UNCLASSIFIED]

Mon Jun 20 07:17:09 PDT 2016

Thanks Bruce for the inputs, I realize that it is a wide domain there.

In the meanwhile, I start to get good result at netcdf generation using
python pandas and xarray. I will keep you informed of how this turns in
production use with some data, we will tes both netcdf and postgres models.

BTW, I'm starting to think that pandas could be a great addition to
OSGEO4W, I add it every time in my deployement package, together with
ipython and jupyter notebooks. OSGEO4W with R, grass and the python glue is
now a great framework for reproducible research.

Cheers
Régis

2016-06-20 2:11 GMT+02:00 Bruce Bannerman <B.Bannerman at bom.gov.au>:

> Hi Regis,
>
> This is something that many of us in the MetOceans/Climate world are
> dealing with.
>
> I don’t know of an simple answer at this stage.
>
> For background, within the World Meteorological Organisation we are doing
> some foundation work to support our future time-series spatial information
> needs.
>
> Have a look at WMO #1131, Climate Data Management System Specifications
> [1]. It provides a high level architectural overview of modern CDMS
> requirements.
>
> While this document explicitly targets CDMS, you will see that they are
> quite broad and cover the needs of most of the MetOceans and other
> environmental domains as well.
>
> I’m currently trying to get some funding to support the establishment of a
> reference open source CDMS called Open-CDMS. But this is a long uphill
> slog. Contact me off line if you’d like to know more.
>
> So with that background, the short answer would be to store this data in a
> CDMS. However, these are typically not spatially or temporally enabled at
> present.
>
> My gut feel is that Postgres/PostGIS may provide an excellent platform for
> observations data, but we have substantial work to do.
>
> Let’s keep in contact. I’d be interested in your findings.
>
>
> Bruce
>
> *Bruce Bannerman *| Data Director (acting)
>
> Bureau of Meteorology
> GPO Box 1289, Melbourne, Victoria 3001
> 700 Collins Street, Docklands, Victoria 3008
> Australia
>
> www.bom.gov.au
> [1] http://library.wmo.int/opac/index.php?lvl=notice_display&id=16300
>
>
> From: Qgis-user <qgis-user-bounces at lists.osgeo.org> on behalf of Régis
> Haubourg <regis.haubourg at gmail.com>
> Date: Friday, 17 June 2016 at 19:29
> To: "qgis-user at lists.osgeo.org" <qgis-user at lists.osgeo.org>
> Subject: [Qgis-user] best data storage fo time series visualisation in
> QGIS
>
> Hi,
> I need the communities lights!
>
> I'm starting to work with huge meteo datasets composed of a grid of point
> layers, and hundred of millions of rainfall / temperature data.
>
> Datasets are delivered in a custom text format, so I'm digging around on
> what are the best formats for storage, use in postgis and QGIS.
> I would like to be able to :
>  - run timeManager to generate videos
>  - display data averaged on day / month / year (or any other) timeframe
>  - feed R analyses.
>
> Up to now, I tried the following paths:
>   - netcdf  / grib:  ideal for data storage:
>           Pros : GDAL and QGIS can view it. R And python scipy have
> providers for that
>           Cons : not easy to generate from exotic datasources, Current
> QGIS Netcdf explorer or core date visualisation (time frame = raster bands)
>  are not handy for daily data over decades (about 10 000 days available in
> my dataset).  I didn't manage to build netcdf yet, FME or GDAL are a bit
> dry..
>
>  - load all in postgres / postgis relationnal model:
>           Pros: available for all clients and fast, if data is correctly
> indexed and designed/
>           Cons: performance requires a table (not a view because of lack
> of estimated metadata for extent computing) with redondancy over point
> location. I tried a first approach with a small geographical table for my
> point grid and a value table. With correct indexing and clustering, I get
> good performance in psql but very poor in QGIS. First load is slooow
> because of the st_extent query, but also every fetch afterwards, even if I
> filter on a date frame (with good index). I didn't expect it to be slow on
> fetch..
>
> Another point with postgis storage, TimeManager plugin does not like true
> date datatype, date cast to char truncate date to first character,  so I
> have to expose my datasets with a text format in my view, which is not
> quite efficient.  (I will create a ticket upstream)
>
> *Does anyone has any experience and advices on that field ? *
>
>  I saw that postgis has a datacube type, could that be a way to store data
> more efficiently? Could QGIS read it?  Should I stick with netcdf ?
>
> Thanks a lot
>
> --
> Régis
>
>

-- 
Régis Haubourg

Attention, changement d'adresse mail!
Mon adresse principale devient désormais regis.haubourg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160620/2a1ca696/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1BDB09EC-D554-4372-9C80-E25125AC6C3A.png
Type: image/png
Size: 7308 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20160620/2a1ca696/attachment.png>