[Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

Sun Sep 4 12:13:48 PDT 2022

Hi Richard,

I think the underlying issue here is that QGIS does not do data management... It just accesses data (whether not managed, managed or mismanaged).

You are correct, the traditional GIS model of a feature with attributes does not map to the real world very well in many use cases, and timeseries data is one. GIS is seldom used to manage data these days, with very powerful spatially enabled databases readily available. To use your IOT approach where location is just an attribute, in a relational db, the location IS just another attribute. Unlike the GIS model, where the spatial feature is somehow different, here it isn't. A record can have multiple geometries, times, dates, strings, numerics, etc... like start time, end time, start location, end location, trackline between them, if you want... impossible without nasty hacks in a GIS centric data store.

So you have a pretty standard situation where you have a set of sites where data is captured in an ongoing basis.

I have designed Postgis databases to do this... most recently, when Postgres/Postgis was staring to slow down a bit with 600,000,000 readings I moved to Postgres/Postgis/TimescaleDB which is returning typical query results in 10's of milliseconds with 4 billion readings.

You don't just have sites with readings, you have sites, sensors, calibrations, instruments, readings, personnel, etc... there is a whole lot of metadata/data pertaining to your setup that should probably be managed in a database. I don't know if you will have enough data to justify using TimescaleDB, but you should be using some sort of db to manage your data effectively & efficiently. And Postgis/Postgres is a hard combination to beat as a tool for doing this well.

Then you point QGIS at your database for the mapping, cartography, visualisation, etc. You can use database views to simplify queries, but there are issues with QGIS and views. I find using the QGIS DB manager with a query instantiated as a QGIS layer works much better than accessing a view in many cases.

And for an alternative mapping of IOT data to a data model, the Object Relational approach provided by Postgres supports non-relational structures like key/value (hstore) or jsonb data storage.

52North have a (somewhat complex) Postgis database design to provide a data store for their SOS (Sensor Observation Service) software. Perhaps overkill for your use case, but a very complete and robust solution. See: https://www.researchgate.net/figure/The-52-North-SOS-schema_fig14_327124727

Hope this helps, there are certainly effective ways to do what you want out there.

Cheers

Brent Wood

Principal Technician, Fisheries
NIWA
DDI:  +64 (4) 3860529

________________________________

On 2 Sep 2022, at 13:07, Richard Duivenvoorde via QGIS-Developer <qgis-developer at lists.osgeo.org> wrote:

Hi All,

Anybody is aware of a model in which data owns a location, but THAT location has a (growing) set of [timestamp, value] observation data to it....

In GIS/QGIS everything is tabular (talking about vector here, I know meshes can have more shapes).

But in the IOT world the 'location' is 'just an attribute', and the 'other' data is actually more of interest (and growing in time).

Currently to view measurements, 'we' often replicate the location for every value/measurement: we create a 'Feature' for every time step (eg for example via a WFS)

The OGC has a SensorThingsApi (STA) standard, in which for given location you can request all values/observations (of filter a certain sensortype/parameter). So one geometry has a full table of data to it.

BUT to 'work' with that data in QGIS, you always have to 'flatten' it, one way or another, (I think), for every time step: create a feature with: location, time, value...

My Question:

- isn't there some model (or can we come up with one) in which a location can actually have a range of time/value data (like some table join like), and when you use the Temporal Controller, you style your layer by RE-using the one location, but getting the value for current Timestamp (in the TimeController). Some sort of indirect filtering....?

- or is this the mesh model (I know netcdf's in which you have often predefined meshes/location and the data/time vector is growing)?
And would a solution be to have some kind of in memory vector -> mesh loader or so???

I hope this makes sense to others. Any input appreciated.

Regards,

Richard Duivenvoorde

PS, if you want to play with SensorThingsApi:
- install: https://github.com/AirBreak-UIA/SensorThingsAPI_QGIS-plugin
- connect it to: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Locations
(around 5000 location and 500-million 'observations')
- load all locations in one layer, and click on a location
My point: the data is then viewable/selectable by table and graph, but NOT loadable in a sense way (yet) in QGIS...
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer at lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[https://www.niwa.co.nz/static/niwa-2018-horizontal-180.png] <https://www.niwa.co.nz>
Brent Wood
Principal Technician - GIS and Spatial Data Management
Programme Leader - Environmental Information Delivery
+64-4-386-0529

National Institute of Water & Atmospheric Research Ltd (NIWA)
301 Evans Bay Parade Hataitai Wellington New Zealand
Connect with NIWA: niwa.co.nz<https://www.niwa.co.nz> Facebook<https://www.facebook.com/nzniwa> LinkedIn<https://www.linkedin.com/company/niwa> Twitter<https://twitter.com/niwa_nz> Instagram<https://www.instagram.com/niwa_science>
To ensure compliance with legal requirements and to maintain cyber security standards, NIWA's IT systems are subject to ongoing monitoring, activity logging and auditing. This monitoring and auditing service may be provided by third parties. Such third parties can access information transmitted to, processed by and stored on NIWA's IT systems
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20220904/7674e2b7/attachment.htm>