[Qgis-developer] QgsVectorLayerCache

[Sandro Mani]

(Forwarding message to list...)

On 15.06.2015 16:52, Sandro Mani wrote:
> Hi Matthias
>
> On 15.06.2015 15:34, Matthias Kuhn wrote:
>> Hi Sandro
>>
>> On 06/15/2015 02:42 PM, Sandro Mani wrote:
>>> Hello Matthias and List
>>>
>>> I have two questions about the QgsVectorLayerCache, which you
>>> (@Matthias) have implemented [1].
>>>
>>> 1. First the easy one:
>>> https://github.com/qgis/QGIS/blame/master/src/core/qgsvectorlayercache.cpp#L97 
>>>
>>>
>>> -> I'm not sure what this block of code is supposed to do. As far as I
>>> can see it just performs an empty iteration over all layer features,
>>> but has no effects otherwise. Am I missing something? This block of
>>> code is executed when loading the attribute table. If I comment it, I
>>> can't spot any side effects, except that the loading of the attribute
>>> table is faster.
>> It's a QgsCachedFeatureWriterIterator which fills the cache when
>> iterating over it.
>> It's only being invoked when full caching is requested to avoid
>> incremental population of the cache with a lot of subsequent requests.
>> If you have slow round-trips and disable this code the effect should be
>> noticeable. If it's not, there's something wrong with it.
> Uhm ok, I'll need to investigate this together with my AFS provider 
> implementation, somehow the result of that code block was that all 
> features were fetched from the server twice.
>>
>>> 2. Secondly, to the vector layer cache in general.
>>> Some background: I've done an initial implementation of an ArcGIS
>>> Feature Service ("AFS") data provider, and similarly to the WFS
>>> provider, the question of intelligent caching arises, to reduce
>>> round-trips with the server. The WFS provider just caches all features
>>> in memory (if the corresponding option is checked), which is
>>> suboptimal for large datasets.
>>> I've hence been thinking about implementing a local disc-based cache
>>> (say in the form of an SpatiaLite DB), which acts as a local feature
>>> cache. The usefulness of this could however go beyond just WFS and
>>> AFS, to include all non-local data sources. So my idea is to implement
>>> something like a QgsVectorDataProviderProxy which
>>>
>>> - overrides getFeatures to return a QgsCacheProxyFeatureIterator: this
>>> iterator first checks whether the Feature is cached, and if not, only
>>> then fetches it from the data provider. If the QgsFeatureRequest
>>> includes an extent, entire pages of features could be loaded from the
>>> disk to memory (up to a specified threshold).
>>>
>>> - overrides all add/change/delete methods to ensure that the cache
>>> remains consistent.
>>>
>>> Actually I think the most elegant approach would be to have
>>> QgsVectorLayer::mDataProvider be an instance of this
>>> QgsVectorDataProviderProxy. If the data source is local, the calls are
>>> simply forwarded to the actual data provider, otherwise, the above
>>> outlined behavior applies.
>>>
>>> So (@Matthias): such an implementation would pretty much overlap with
>>> what you have implemented, but does the work directly at provider
>>> level. What are your thoughts on this? From your experience
>>> implementing [1], do any alarm bells start ringing?
>> I have thought about this approach as well as it seems to be very nice
>> to have one shared cache which is able to provide several consumers with
>> cached data (canvas, attribute table...). Do you think you will be
>> introducing a size limit?
> Speaking of the disk-cache: Yes, I suppose that would make sense, 
> perhaps as a configurable option in the user preferences. For memory 
> cache, there clearly would be a size limit.
>>
>> One risk I see is, that if you have different consumers (with a shared
>> cache), they have different requirements.
>> For the canvas the requirement is usually to have some spatial index
>> that keeps track of which regions are cached and if a new request can be
>> satisfied. It would be even easy/nice to do some look-ahead to pre-load
>> features or only load part of the canvas if a big region is already
>> loaded or do tiling.
> Right, I'd like to model the cache around an idea of "pages", i.e. 
> entire spatial regions which can be swapped in and out of memory 
> depending on the current region of interest.
>>
>> If another consumer then does a second request without a spatial filter
>> (none or attribute filter instead) it may fetch a lot of features and
>> pollute your cache with these features. If there's a size limit of the
>> cache it can then be cleaned of previous features which would still be
>> more important for drawing then the ones fetched for a different
>> consumer which may have been requested just once.
> Yes I see the problem. First, one would need to investigate how 
> expensive such cache trashings are compared to the situation with no 
> cache at all. Then I suppose the usual ideas are things like having an 
> access time stamp on the page loaded in memory, and if a page needs to 
> go, the one last accessed furthest back will get thrown out.
>>
>> You will also have to take care of multithreading since multiple
>> iterators can run at the same time.
> Definitely.
>>
>> It's probably also required to spend some thoughts on how to invalidate
>> the cache if the source data changes. (A time-based limit, a button to
>> trigger a reload...).
> Perhaps it would generally be a good idea to have a clear user-facing 
> entry in the layer context menu to re-sync the entire provider data 
> with the data source.
>>
>> If this is implemented, it would surely be nice to have it not only for
>> AFS but also for other providers. Either way I would leave the choice to
>> the user if he wants to use it or not.
> Sure, this would be a user-configurable option, which ideally would 
> just decide whether QgsVectorLayer::mDataProvider receives an actual 
> provider instance of a cache proxy instance.
>
>
>> If there's a request with a subsetOfAttributes set or without geometry,
>> it's important to know if the request is going to be changed before
>> sending it to the provider (so the cache contains all the information
>> but the request may take longer) or if the request is going to be sent,
>> requesting a reduced amount of information but not going to be cached.
>> Or if it's going to be cached with reduced information, but then it has
>> to be ensured later on that a subsequent request does not receive less
>> information than it requires.
>
> I'd go with fetching the reduced feature from the data provider, and 
> not caching it, for a start at least. There are clearly more nifty 
> approaches to be explored later on ;)
>
>>
>> I hope there are some good inputs in here
> Yes definitely, thanks!
>
> Sandro