RFC 33: Removing msLayerWhichItems()...

[Steve Lime]

Dan made a comment somewhere in this thread about just modifying msLayerWhichItems() and I've
been thinking about that and, as usual, I think he may have a good point. If you look at the code
you might notice a little todo comment about requesting a list of all items first and then using them
to build an item list. Functionally you end up with the same result but you know what you're looking
for as opposed to having to look for it.

So, what about adding a 'getall' flag to msLayerWhichItems()? If true you get 'em all but still build
you're index relationships and expression item lists so no other code would have to be touched, except
for the actual calls to the function. With getall=true the shapeObj's read later would be cachable for
presentation. Drawing routines would want just the required attributes, queries would get all of 'em.

msLayerWhichItems() would call msLayerGetItems() first and loop through that list. This should simplify
the function. One other benefit is that regex grouping (e.g. [a|A] or [1..8]) would be allowed in logical
expressions.

Steve

>>> On 7/12/2007 at 3:52 PM, in message <46964E4A.5157.008F.0 at dnr.state.mn.us>,
Steve Lime <Steve.Lime at DNR.STATE.MN.US> wrote:
> I'm starting to think this is a 5.2 change. It's definitely gnarly and the 
> effects are not understood
> well enough...
> 
> Steve
> 
>>>> On 7/12/2007 at 1:08 PM, in message <46966E28.2050306 at mapgears.com>, Daniel
> Morissette <dmorissette at MAPGEARS.COM> wrote:
>> Steve Lime wrote:
>>> The performance concern is real although I don't know how big a deal it is 
>> for all drivers. I
>>> believe with shapefiles you get everything anyway.  Not sure about others. 
>> How would OGR
>>> be impacted?
>>> 
>> 
>> Your comment made me doubt so I ran both shapefiles and OGR data sources 
>> in the debugger and confirmed that when rendering a map, they both load 
>> only the attributes that are required.
>> 
>> In the shapefile case I believe you still need to do the file I/O to 
>> read the full attribute record from disk, but only the values of the 
>> required attributes are strdup'd and stored in the shape->values[] array.
>> 
>> OGR works in a similar way: the OGRFeature object returned by OGR 
>> contains all the attributes but we only copy the required attributes to 
>> the shape->values[] that is seen by MapServer.
>> 
>> I think the performance gain/loss would be more significant for 
>> shapefiles than for OGR data sources for this reason.
>> 
>>> msLayerWhichItems sets up the item indexes and you still need that for 
>> queries since CLASSes
>>> come into play.
>>> 
>> 
>> Plus, OGR uses special item indexes for some special OGR attributes that 
>> are extracted from the style strings (OGR:LabelText, OGR:LabelAngle, and 
>> OGR:LabelSize), so if we get rid of item indexes we'll need to think of 
>> another way to handle that.
>> 
>>> 
>>> One way to mitigate the performance impact might be to allow a user to 
>> explicitly set the ITEMS 
>>> array in the configuration file. I'm hoping that with PostGIS and Oracle 
>> Spatial that's essentially 
>>> what you do with the sub-selects. An added benefit of that would be that 
>> inline features could
>>> now have attributes:
>>> 
>>> LAYER
>>>   NAME 'anInlineLayer'
>>>   TYPE POLYGON
>>>   STATUS OFF
>>>   ITEMS name, type, area END
>>>   LABELITEM 'name'
>>>   FEATURE
>>>      POINTS ... END
>>>      VALUES 'St. Paul', 'CITY', '53045098.12' END
>>>   END
>>>   ...
>>> END
>>> 
>> 
>> I thought about providing the ability to specify the list of items in 
>> the layer as a possible solution as well, but then I realized that this 
>> was a move backwards for users since MapServer used to determine the 
>> list of items automatically, and from now on users would be required to 
>> do that manually for all layers if they care about performance.
>> 
>> 
>> I find it odd for me to be arguing *for* keeping msLayerWhichItems() 
>> since I never really understood nor liked it until today. In the 
>> interest of getting this done for 5.0 and if everybody wants to get rid 
>> of it then I won't object any more. I'll be happy with an optional 
>> ITEMS...END block for those who really care about that little bit of 
>> performance.
>> 
>> Daniel