RFC 33: Removing msLayerWhichItems()...

[Steve Lime]

I have attributed the initial idea to Dan and instead Tamas made that post. Apologies...

Steve

>>> On 7/13/2007 at 10:57 AM, in message <46975A93.5157.008F.0 at dnr.state.mn.us>,
Steve Lime wrote:
> Dan made a comment somewhere in this thread about just modifying 
> msLayerWhichItems() and I've
> been thinking about that and, as usual, I think he may have a good point. If 
> you look at the code
> you might notice a little todo comment about requesting a list of all items 
> first and then using them
> to build an item list. Functionally you end up with the same result but you 
> know what you're looking
> for as opposed to having to look for it.
> 
> So, what about adding a 'getall' flag to msLayerWhichItems()? If true you 
> get 'em all but still build
> you're index relationships and expression item lists so no other code would 
> have to be touched, except
> for the actual calls to the function. With getall=true the shapeObj's read 
> later would be cachable for
> presentation. Drawing routines would want just the required attributes, 
> queries would get all of 'em.
> 
> msLayerWhichItems() would call msLayerGetItems() first and loop through that 
> list. This should simplify
> the function. One other benefit is that regex grouping (e.g. [a|A] or 
> [1..8]) would be allowed in logical
> expressions.
> 
> Steve
> 
> >>> On 7/12/2007 at 3:52 PM, in message <46964E4A.5157.008F.0 at dnr.state.mn.us>,
> Steve Lime <Steve.Lime at DNR.STATE.MN.US> wrote:
> > I'm starting to think this is a 5.2 change. It's definitely gnarly and the 
> > effects are not understood
> > well enough...
> > 
> > Steve
> > 
> >>>> On 7/12/2007 at 1:08 PM, in message <46966E28.2050306 at mapgears.com>, Daniel
> > Morissette <dmorissette at MAPGEARS.COM> wrote:
> >> Steve Lime wrote:
> >>> The performance concern is real although I don't know how big a deal it is 
> >> for all drivers. I
> >>> believe with shapefiles you get everything anyway.  Not sure about others. 
> >> How would OGR
> >>> be impacted?
> >>> 
> >> 
> >> Your comment made me doubt so I ran both shapefiles and OGR data sources 
> >> in the debugger and confirmed that when rendering a map, they both load 
> >> only the attributes that are required.
> >> 
> >> In the shapefile case I believe you still need to do the file I/O to 
> >> read the full attribute record from disk, but only the values of the 
> >> required attributes are strdup'd and stored in the shape->values[] array.
> >> 
> >> OGR works in a similar way: the OGRFeature object returned by OGR 
> >> contains all the attributes but we only copy the required attributes to 
> >> the shape->values[] that is seen by MapServer.
> >> 
> >> I think the performance gain/loss would be more significant for 
> >> shapefiles than for OGR data sources for this reason.
> >> 
> >>> msLayerWhichItems sets up the item indexes and you still need that for 
> >> queries since CLASSes
> >>> come into play.
> >>> 
> >> 
> >> Plus, OGR uses special item indexes for some special OGR attributes that 
> >> are extracted from the style strings (OGR:LabelText, OGR:LabelAngle, and 
> >> OGR:LabelSize), so if we get rid of item indexes we'll need to think of 
> >> another way to handle that.
> >> 
> >>> 
> >>> One way to mitigate the performance impact might be to allow a user to 
> >> explicitly set the ITEMS 
> >>> array in the configuration file. I'm hoping that with PostGIS and Oracle 
> >> Spatial that's essentially 
> >>> what you do with the sub-selects. An added benefit of that would be that 
> >> inline features could
> >>> now have attributes:
> >>> 
> >>> LAYER
> >>>   NAME 'anInlineLayer'
> >>>   TYPE POLYGON
> >>>   STATUS OFF
> >>>   ITEMS name, type, area END
> >>>   LABELITEM 'name'
> >>>   FEATURE
> >>>      POINTS ... END
> >>>      VALUES 'St. Paul', 'CITY', '53045098.12' END
> >>>   END
> >>>   ...
> >>> END
> >>> 
> >> 
> >> I thought about providing the ability to specify the list of items in 
> >> the layer as a possible solution as well, but then I realized that this 
> >> was a move backwards for users since MapServer used to determine the 
> >> list of items automatically, and from now on users would be required to 
> >> do that manually for all layers if they care about performance.
> >> 
> >> 
> >> I find it odd for me to be arguing *for* keeping msLayerWhichItems() 
> >> since I never really understood nor liked it until today. In the 
> >> interest of getting this done for 5.0 and if everybody wants to get rid 
> >> of it then I won't object any more. I'll be happy with an optional 
> >> ITEMS...END block for those who really care about that little bit of 
> >> performance.
> >> 
> >> Daniel
>