MS RFC 22a: Feature cache for long running processes and query processing (update)

[Howard Butler]

Steve,

The SDE driver in trunk now gets all of the items all of the time.   
The WhichItems approach wasn't really feasible once I started putting  
joins in play (items are appended to the list if they are not found,  
causing a segfault because the items now mismatch the internal  
columns list that is maintained...).  If I recall correctly, you said  
the SDE driver was the reason WhichItems existed in the first place.   
If the surgery to remove it isn't too difficult, I think we should  
take the opportunity to ditch it.

Howard

On Jun 29, 2007, at 2:11 PM, Steve Lime wrote:

> The one gotcha with one-pass queries regardless of approach is that  
> presently the items
> retrieved for the first pass don't match those on the second pass.  
> In the first pass we grab just
> enough to classify (if necessary) features via msLayerWhichItems.  
> While the second pass gets
> all items for presentation. We really should ditch  
> msLayerWhichItems and get all items all the
> time.
>
> The main impact of that would be needing to derive item (e.g.  
> classitem, labelitem, etc...) index
> values at runtime. Basically:
>
>    if (classitem && classitemindex == -1) /* lookup */
>      classitemindex = msLayerGetItemIndexByName(...);
>      if(classitemindex == -1) { /* item not found, throw an error */
>        ...
>      }
>    }
>
> This would affect a few places in the code, principally mapdraw.c  
> and maputil.c.
>
> The initItemInfo functions for each of the vector drivers might  
> need changes, although I hope
> minor ones since we already support getting all items.
>
> In reality this is a simplification of things which is a good  
> thing. I would be willing to take this modification
> on in consultation with driver owners.
>
> Regarding RFC 22 what do folks think about implementing a simple  
> one-pass cache using
> the resultCacheMemberObj for 5.0? Then considering RFC 22  
> independently of queries independently
> since there are plenty of good ideas there.
>
> Steve
>
>
>>>> On 6/28/2007 at 12:55 PM, in message <4683F62D. 
>>>> 7070401 at mapgears.com>, Daniel
> Morissette <dmorissette at MAPGEARS.COM> wrote:
>> Steve Lime wrote:
>>> That could be as simple as the shape just sitting there along  
>>> with the
>> indexes, that is, the
>>> resultCacheMemberObj gets a new member, a shapeObj.
>>>
>>> The result cache is nothing more than an array of those members.  
>>> (e.g.
>> layer->results) That
>>> could be made more high performance and hidden behind accessor  
>>> methods in
>> mapscript. Users
>>> use getResult(int i) already.
>>>
>>
>> I like that. And you enable this query cache by default for all data
>> providers that suffer from the double-pass query problem (postgis,
>> oracle, etc, but *NOT* shapefiles) and provide a PROCESSING directive
>> that can be used to turn off the query cache if someone wants to  
>> disable
>> it to perform very large queries or for whatever reason, e.g.
>>
>>    PROCESSING "QUERY_RESULT_CACHE" "OFF"
>>
>> Daniel