[Qgis-developer] New Vector API

[Matthias Kuhn]

Hi Martin,

see answers below

On 01/16/2013 10:13 PM, Martin Dobias wrote:
> Hi Matthias
>
> On Tue, Jan 15, 2013 at 11:46 PM, Matthias Kuhn <matthias.kuhn at gmx.ch> wrote:
>> Hi Martin,
>>
>> This mail concerns the vectorlayercache outlined in section 8) of your mail.
>>
>> I'm currently working on the attribute table which has some internal caching
>> already. As I'm currently rewriting the attribute table, I moved this code
>> outside and generalized it as good as possible. This work is currently
>> available in my branch vectorlayercache [0]. Do you think this could be used
>> instead of your geometry cache? It should be able to cache geometries of
>> features (although I only tested attributes so far) and update and
>> invalidate them if required.
>>
>> I have also the idea to create a QgsVectorDataProviderCacheProxy class which
>> inherits QgsVectorDataProvider and which you can use as a proxy between your
>> app and the DataProvider. I've some unfinished code here which provides such
>> functionality. But for now, the cache in front of the VectorLayer should
>> already help a lot.
> Just out of curiosity, what are you rewriting in the attribute table -
> is it just the cache or does it involve more changes?

Have a look at the dual view discussion on this list. It's explained 
more there.
>
> Your idea of having a cache proxy vector data provider is interesting.
> Such transparent approach would enable us to remove caching from
> vector data provider implementations (e.g. wfs) and transparently add
> caching to the ones where caching would be useful (e.g.
> delimitedtext). I need to give it some more thoughts what impact that
> would have.
Let me know when you do. I have now implemented such a thing in my 
plugin and would love to have this functionality in core.
>
> The code for cache in your github repository implements cache with
> feature ids as keys. That is useful for attribute table queries,
> however for rendering it does not help much. In the cache we will need
> also a spatial index for fast spatial queries used for map rendering,
> identification and geometry editing. Without spatial index, the cache
> is quite slow if the whole cache need to be traversed in order to find
> features in a specified rectangle. As of now, the cache you've done is
> bound to a one particular query it gets: I would be most interested in
> a general cache to avoid having situations where cache would be filled
> by results of a first query and then flushed and filled with results
> of a following query.
Yes it is bound to feature ids by now. My idea was, that you will keep 
track of keys of features of interest anyway. Let's say, the keys of 
features of a particular rectangle.

> The caching behavior should be ideally configurable: e.g. which
> layers/providers to cache (probably we don't want to cache huge
> layers), what to cache (everything / only geometries / only attributes
> / geometries and some attributes), cache size limit.
My cache approach does most of this: only geometries, only attributes or 
both (A subset of attributes is still solved unsatisfactory, but that 
wouldn't be too hard to add) and cache limits. The only thing missing is 
the possibility to configure which layers to cache (Needs to be a higher 
level approach maybe? Layer settings?)
>
> The next question is how to invalidate the cache, so that if data have
> been changed in the backend, the user will not end up with a cached
> version forever. In some providers I think it's possible to detect the
> updates automatically, but some providers do not have any such
> notifications. Maybe we should allow users to set a timeout - after
> specified interval the cache would need to check again with the
> backend whether the features have not been altered.
It would be nice to get this information from providers which do have 
the possibility, for others, see below.
>
> Back to your original question whether such improved cache could be
> used instead of the current geometry cache: yes, it could, in case it
> will include a spatial index. We are really in a need of a general
> caching mechanism, so I would welcome any work in this area. The
> speedup of rendering from cache should be significant. The current
> caching of geometries is there just to preserve the functionality of
> moveVertex() and other functions from vector layer that were using
> cached geometries before.

To solve these problems I was thinking about adding an abstract 
interface ( QgsCachItemManager or something) which provides at least the 
method
* bool requiresCaching( QgsFeature& )

It could then also be extended to check if the cache is able to answer a 
certain select operation (in which case it would return a QgsFeatureIds 
list). So it could keep track of a spatial index, cache age and so on. 
Speak custom tailored caching depending on the special use-case. 
Accomplishing that would require some small internal changes, as we need 
to keep track of removed items. It's using QCache now, which as far as I 
know does not communicate, when a certain item is removed (it cleans 
silently). So either we'd have to replace QCache by some other mechanism 
or let the cache-item destructor emit this signal.

I have to say, that I'm a bit limited in time. At the moment, the 
attribute editor has a higher priority for me (I would really like to 
have this ready by 2.0) so the caching was more of a side-effect.

>
> Regards
> Martin
Regards
Matthias