MS RFC 22a: Feature cache for long running processes and query processing (update)

[Stephen Woodbridge]

Daniel Morissette wrote:
> Hi Tamas,
> 
> Tamas Szekeres wrote:
>>
>>> and even if it has a NextItem method
>>> to walk through all objects, the order of objects is not maintained by a
>>> hashtable, so if a user has data sorted (by sortshp) then the sort oder
>>> will be lost and rendering order will become pseudo-random if done via a
>>> cache layer (unless I'm missing something?).
>>>
>>
>> That's true. I'm not aware of the order of the renderings in this case.
>> In my practice I haven't found such a problem it was required.
>> However we could use an additional list to treat this issue if it is
>> significant.
>>
> 
> This ordering of shapes at render time is a feature of MapServer, hence 
> the command-line program sortshp. I don't use it myself but some users 
> must rely on it otherwise it would not exist. I think it's a sad 
> side-effect to not try to maintain the ordering but I'll let those who 
> need this feature fight for it.

Shape ordering is a problem!

This is a good point as I do use ordered data. I primarily use ordered 
shapefiles to control the rendering and labeling of data. It would be a 
little disconcerting to render an image and then when it is rendered out 
of the cache all the labels would change because the object ordering has 
changed. I think this would be very bad. At Where2getit we serve 500-750 
Million map draws a year and I would hate to have to deal with questions 
and issues from those users and clients. I suppose I do not need to use 
the cache in which case I would get the old behavior which might be OK.

A few additional thoughts on this RFC:

1) Would it make sense or be possible to construct a cache provider that 
  also maintained a spatial index for its cache items?

2) On a zoom in, I assume that the cache will be used exclusively since 
the extents will be in side the previous extents, or does the new scale 
invalidate the cache and require it to be rebuilt. Would it make sense 
for the cache to be "aware" of the zoom scales it can support?

3) Performance is a BIGGY for me. It takes a lot of servers to process 
the map draws mentioned above. Mapserver is basically I/O bound in our 
case of using shapefiles 30-50 GB per map dataset. Mapserver does NOT 
use much memory. I would trade memory for performance any day. I have 
found that I can get a 4-5x performance benefit by creating a 12 GB ram 
disk and moving frequently hit data into it. That still leave 4 GB for 
OS and mapserver processes on 16 GB system and I would like to see 
mapserver use more memory if we can make it go faster.

-Steve
-Steve