[QGIS-Developer] QgsDataProvider and caching: any defined behaviour?

[Régis Haubourg]

Hi Richard,

 one guy said that once:

There are only two hard things in Computer Science: cache invalidation and
naming things.

-- Phil Karlton
Joke apart, current situation is not good for sure, especially for WFS
which has a cache, and that should be handled better in transactional WFS-T
use cases.

For Postgres, we recently hit a problem, not related to big datasets, but
to very laggy database and network, that led the client to switch back to
WFS, with a its cache giving an impression of reactive application.
Cache invalidation seems especially complicated currently when you have
some business logic inside the database (triggers), and even more when
using Transaction groups, where any action is commited on the fly in the
database. It starts to be tricky keeping in sync the snapping index cache
and the attribute table cache only for features concerned by the
transaction.
Matthias also pointed out that we don't have very specific cache
invalidation signals for attribute table, and we should have some to be
able to invalidate only the features concerned by a transaction (for
instance).

So I think we must go that way, at least to consolidate current features,
but I think should be handled very seriously by a group of dev knowing each
parts of the application. That is an excellent candidate topic for Madeyra.

Régis

2017-11-06 11:05 GMT+01:00 Richard Duivenvoorde <rdmailings at duif.net>:

> Working with a rather large dataset (16M timebased points), I have trouble
> working with this in QGIS, using both the Postgis-provider OR the
> WFS-provider.
>
> As data is properly indexed, retrieving a selected area is fine.
>
> BUT as soon as you try to use the attribute table, I-tool or for example
> the TimeManager, QGIS stalls.
>
> I think(!) because apparently both providers are requesting all data again
> from their datasource, sometimes even ignoring the current extent (WFS).
>
> While, it seems more appropriate to 'just use what you already have': the
> dataprovider/QGIS already has all features (both geom + attributes in case
> of WFS for example), so WHY should it request all data again
> (data-freshness?)?
>
> Are others having this problems too?
>
> Could adding a 'Caching-option' for the QgsDataProvider be a
> reasonable/viable option:
> - if 'data caching' is enabled, the data will be retrieved only once (per
> extent probably). So showing the Attribute table, I-tool info, or using
> TimeManager/Query should be relatively easy/fast as the features are
> locally (or in memory) cached.
> - as soon as you want 'fresh' data you disable the caching temporarily,
> and new data will be requested (as it is doing now apparently).
>
> Or am I missing something (I see there is an option for attributetable
> caching...)?
>
> Not sure about other countries, but here in NL governmental institutes
> start serving huge dataset more and more, either as WMS/WCS/WFS services,
> or more and more as REST/GeoJSON services.
>
> Anybody has some insights ideas about this?
>
> Regards,
>
> Richard Duivenvoorde
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20171106/594a8e95/attachment.html>