[mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles
using Style Order
Zac Spitzer
zac.spitzer at gmail.com
Fri Sep 11 22:15:26 EDT 2009
following up on the PSC discussion about this RFC
http://trac.osgeo.org/mapguide/wiki/PscMeeting09-10-2009
it was noted that this RFC depends on FDO RFC 40
http://n2.nabble.com/RFC-40-Add-standardized-ordering-support-td3223241.html
to which Orest replied that this is already part of the FdoISelect API
given that it seems that maybe RFC 40 is therefore potentially redundant
and it's only a question there whether the given FDO provider in use supports
ordering, i guess we can focus back on the core of the RFC?
It seems that everyone is happy with approach of pre-ordering of data
and adding support to a layer def for defining which column to order by
the implementation would be as follows?
1. add ordering to the layer definition
2. add support to apply the ordering, throwing an exception when
the provider doesn't support it
thoughts?
z
On Thu, Jul 9, 2009 at 2:08 AM, Traian Stanev
<traian.stanev at autodesk.com> wrote:
>
> The draw priority of labels depends on the draw order of your layers. Labels for the topmost layers get priority. So you have to make sure that the layer with the major highways draws on top of the layer containing the rest of the roads -- this assumes that you have one layer for major and one layer for secondary roads. I'm probably repeating what was already said -- this is the approach that will work with the current implementation, but it has the overhead of having to do a data query for each of the layers. The alternative is to pre-order the data so that your major roads come last in your SHP file (or whatever data source you are using).
>
> If all you are worried about is label draw order, it will probably make sense to ask for something like "label priority" setting for each theme rule, which would give you control over which labels draw and which don't regardless of the underlying theme/layer/filter ordering.
>
> Traian
>
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Scott Hameister
> Sent: Wednesday, July 08, 2009 11:34 AM
> To: 'MapGuide Internals Mail List'
> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>
> That’s a little clearer...I thought I had seen a comment (which I cant find) that multiple layers with filters ran more efficiently than a them with the same number of rules...Thats why I was utterly confused...
> I assumed Creating the image from the objects would be much faster when things weren’t stylized one line at a time differing from one line to the next, but instead stylized in groups...especially with end matching, complex line styles and the like...I will admit it sounds like my idea is a bad one, even tho it would take care of objects rendering as intended, it sounds like it would have had no effect on a labeling hierarchy anyways, which is what was my primary concern....I wanted Major HWYS...i.e. Upper Level rules to take labeling precedence over Local roads(lower Level rules).
>
> I'll Leave it to the pros...I wanted to make sure I understood the whys before I gave up the fight...Thanks guys...Youve all enlightened me.
>
> PS has there been any discussions about creating a HWY/Symbol Shield interface in Maestro...or has anyone seen a tool I could give customers for doing this easier than teaching them XML?
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
> Sent: Wednesday, July 08, 2009 10:15 AM
> To: 'MapGuide Internals Mail List'
> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>
>
> Pushing the filtering to the side of the database will not make the filtering any faster.
>
> In the cases of the fastest current data providers (SDF and SHP), the filtering code is the same as the filtering code used by MapGuide to execute the filter, so you gain nothing -- the data provider will have to do the filter linearly across all the features in order to return the 15 features your filter is looking for.
>
> In the case of other (slower) data providers, the overhead of the multiple select commands (as opposed to a single select like the code does now) will kill you, even if the underlying database is indexing the columns that your filter is based on. There is a high overhead for selects, since each one will also do a spatial query, and there are other overheads as well.
>
> Unless you pre-compute the bottleneck away by reordering the original data, it will always be there, just in a different place.
>
>
> Traian
>
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Scott Hameister
> Sent: Wednesday, July 08, 2009 11:08 AM
> To: 'MapGuide Internals Mail List'
> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>
> Lets assume say 15 rule categories on 10,000 objects...
>
> Walt not trying to be a pest...I understand your math...But doesn t the current method by iterating through every record individually "N=10,000" as the outer loop...thus make 10,000 selections Which Should always be more than the number of "M" rules? (M=15) If the overhead is in Selecting wouldn t selecting 15 be a better number than 10,000...
> The stylization is done without a filter because its pre filtered, so shouldn t that also save on time as well?
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Walt Welton-Lair
> Sent: Wednesday, July 08, 2009 12:33 AM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>
> Assume a themed soil layer with M rules being applied against a dataset with N features. You're wondering how that would perform against M layers, each using a filter corresponding to one of the rules, and each against the same dataset.
>
> In the first scenario we make one FDO select operation and it returns all the features. That's O(N) work. We also make O(M*N) filter evaluations in the stylization code.
>
> In the second scenario we make M FDO select operations (each with the appropriate filter) and each returns a subset of the features. Unless you have a simple filter that the provider can efficiently handle it's still going to be O(N) work per select operation. Since FDO applies the filter we don't do any filter evaluations during stylization.
>
> For the two scenarios many things are the same:
> * the overall number of features processed by the server
> * the overall amount of data sent back from the provider
> * the overall filtering work of O(M*N)
>
> The difference in the scenarios is the number of FDO select operations: 1 versus M. Each select operation has overhead (e.g. latency for a dataset remote from the MG server, opening the data connection, etc.) and that IMO is what will cause the second scenario to be slower. Of course it will vary with the provider and whether the dataset is local to the MG server. In general it's important to keep the number of data requests low.
>
> Obviously the first scenario - which is how MG currently works - doesn't give you the desired draw order behavior, while the second scenario does. With the data massaging workaround you have only one FDO select. It's the same as the first scenario, but now the FDO select returns the features in the order you want.
>
> Why don't you do a test for us (and prove me wrong :-]). Try out the two scenarios for one of your themed layers, measure the performance, and share the results.
>
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of scotth at mpowerinnovations.com
> Sent: Wednesday, July 08, 2009 12:24 AM
> To: Zac Spitzer; MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>
> Im missing something big here...why cant filtering like happens on individual layers work? I only use themes when individual layers with filters is overwhelming like soils, or to avoid grouping for legend reasons...id guess 80 pct of users use a wizard to theme, but 20 pct use advanced rules...i like a pre-cooker..but wht cant the default action be to itterate thru the rules as an outside query than to itterate thru geometry individually? Im trying to understand a pre cooker on live data..or often updated load procedures..fills in a column called rule..and puts in a 1 for rule 1?...and we apply my pre mentioned filter..why not making filter default instead of geometry looping?
>
> -----Original Message-----
> From: Zac Spitzer <zac.spitzer at gmail.com>
> Sent: Tuesday, July 07, 2009 10:49 PM
> To: MapGuide Internals Mail List <mapguide-internals at lists.osgeo.org>
> Subject: Re: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles
> using Style Order
>
> I am all in favour of the massaging approach. It puts the user in control.
>
> Whether the user pre-cooks their data in a db via sql or runs a script which adds a column with a chosen layer key, it ends up being quite flexible.
>
> Being able to extract all features matched by a theme in a layer would make such massaging rather straightforward.
>
> z
>
> On Wed, Jul 8, 2009 at 1:38 PM, Jason Birch<Jason.Birch at nanaimo.ca> wrote:
>> Maybe this kind of scenario is a good argument for a configurable data caching layer (using sqlite of course!) in MapGuide.
>>
>> ----- Original Message -----
>> From: mapguide-internals-bounces at lists.osgeo.org
>> <mapguide-internals-bounces at lists.osgeo.org>
>> To: MapGuide Internals Mail List <mapguide-internals at lists.osgeo.org>
>> Sent: Tue Jul 07 20:31:47 2009
>> Subject: RE: [mapguide-internals] MapGuide RFC 69 - Rendering Layer Styles using Style Order
>>
>> Keep in mind that the filters used by a theme can be very broad:
>> * they can be expressions involving multiple attributes
>> * they can use any functions supported by FDO's expression engine
>> * the rules may be against one attribute but with out-of-order ranges:
>> rule 1: all cities with population < 1000
>> rule 2: all cities with population >= 100000
>> rule 3: all cities with population >= 1000 and < 100000
>>
>> In each of these cases the provider ordering support cannot give you the draw order behavior you want. The ordering support only works for a subset of themes; you would have to analyze the theme to see if it's "compatible" with the provider ordering support, and if it isn't fall back to some other algorithm to give you the draw order behavior you want.
>>
>> I'm becoming more and more fond of the data massaging workaround:
>> * it handles all possible filters
>> * it only has to be done once as a preprocessing step (assuming your
>> data is static)
>> * you will get top performance when rendering the data since only one
>> rendering pass is needed
>>
>>
>>
>> -----Original Message-----
>> From: mapguide-internals-bounces at lists.osgeo.org
>> [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Zac
>> Spitzer
>> Sent: Tuesday, July 07, 2009 11:01 PM
>> To: MapGuide Internals Mail List
>> Subject: Re: [mapguide-internals] MapGuide RFC 69 - Rendering Layer
>> Styles using Style Order
>>
>> On Wed, Jul 8, 2009 at 7:56 AM, Tom Fukushima<tom.fukushima at autodesk.com> wrote:
>>> Hi Zac,
>>>
>>> Please also update the following statement in the RFC: "The only work around currently is to break road networks into separate layers." to read as...
>>>
>>> There are three possible known work arounds:
>>> 1) Break the road networks into separate layers.
>>> 2) Render the roads using composite type styles and specify the RenderingPass appropriately (see RFC 29 for details).
>>> 3) Set up the data source so that it returns readers with the features in the correct draw order.
>>
>> ok, I added three workarounds in.
>>
>>> For #3, Walt gave a description of this in one of his emails. I have extracted it for you here for your reference...
>>> Implement a utility which creates a data set optimized for a theme:
>>> * the utility will read the layer definition containing the themes
>>> * it will query the original data set once using each rule's filter,
>>> and write those features to a new data set (it needs to be a format
>>> which returns features in the order they were written)
>>
>> this can be done pretty easily in php or c# via the mg api. going this
>> approach, massaging the data would also most likely end up elimating
>> any complex expression filters,
>>
>>> * update your layer to use the new data set - the features will come
>>> in with the right order
>>>
>>> Also, we're not sure which, if any, data source #3 will work with. But if it does work, it will be the most efficient one. So perhaps, we should leave out #3 above until someone tries it out and verifies that it works.
>>
>> do you the potential datasource problem is that they don't perform ordering?
>>
>> SDF does via
>> http://fdo.osgeo.org/files/fdo/docs/SDF/_i_extended_select_8h-source.h
>> tml
>>
>> http://trac.osgeo.org/fdo/wiki/FDORfc26
>>
>> pretty much all database servers do natively?
>>
>>>
>>> Thanks
>>> Tom
>>>
>>> -----Original Message-----
>>> From: mapguide-internals-bounces at lists.osgeo.org
>>> [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Zac
>>> Spitzer
>>> Sent: Friday, June 26, 2009 7:02 PM
>>> To: MapGuide Internals Mail List
>>> Subject: Re: [mapguide-internals] MapGuide RFC 69 - Rendering Layer
>>> Styles using Style Order
>>>
>>> I have updated the RFC and added a FDO RFC for Ordering support
>>>
>>> http://trac.osgeo.org/mapguide/wiki/MapGuideRfc69
>>> http://trac.osgeo.org/fdo/wiki/FDORfc40
>>>
>>> z
>>>
>>>
>>> On Sat, Jun 27, 2009 at 12:40 AM, Walt
>>> Welton-Lair<walt.welton-lair at autodesk.com> wrote:
>>>> They either split things up into multiple layers, as you say, or they use the rendering pass behavior in the enhanced stylization (currently no UI for the latter, so that complicates that one). Both will have similar performance impact as what's proposed in the RFC - both end up making the same number of FDO queries. And if you read ticket 618 referenced by the RFC, the number of queries is a concern.
>>>>
>>>> I would love to see MapGuide be able to support this behavior. But without a good / novel approach for doing this the performance impact will be atrocious. There are customers who have themed layers with 100's of rules, and changing the default stylization behavior to have draw order match rule order will not work for those maps performance-wise. I think at minimum the RFC will need to be updated to make this behavior optional (default is off). And that means a layer definition schema change.
>>>>
>>>> -----Original Message-----
>>>> From: mapguide-internals-bounces at lists.osgeo.org
>>>> [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of
>>>> Paul Spencer
>>>> Sent: Friday, June 26, 2009 9:18 AM
>>>> To: MapGuide Internals Mail List
>>>> Subject: Re: [mapguide-internals] MapGuide RFC 69 - Rendering Layer
>>>> Styles using Style Order
>>>>
>>>> How would someone achieve the desired effect now? From what I
>>>> understand (and I am not a power user of mapguide by any standard of
>>>> measurement), you would need to create several layer definitions
>>>> each with a filter and theme those layers for each road class?
>>>>
>>>> how would performance for:
>>>>
>>>> - for each rule, construct an FDO query filtered by the rule's
>>>> condition
>>>> - render all features using the rule's style
>>>>
>>>> differ from having separate layers?
>>>>
>>>> Yours confusedly ;)
>>>>
>>>> Paul
>>>>
>>>>
>>>> On 25-Jun-09, at 11:06 PM, Walt Welton-Lair wrote:
>>>>
>>>>> "Can some elaborate on how the server currently handles this?"
>>>>> Handles what?
>>>>>
>>>>> The current code iterates once over the features (ignore the
>>>>> compound line style example). For every feature it evaluates the
>>>>> filters defined by the rules (in the order that the rules are
>>>>> specified). Once it finds a matching rule it applies its style to
>>>>> the feature.
>>>>>
>>>>> With compound line styles we make a pass over all the features for
>>>>> each line style. So for a compound line style containing M styles
>>>>> we make M FDO queries.
>>>>>
>>>>> With rendering passes (RFC 29) we also iterate over all the layer's
>>>>> features for each rendering pass. The more passes you define the
>>>>> more queries you make.
>>>>>
>>>>> How to implement your RFC?
>>>>>
>>>>> option 1
>>>>> - Make a pass over the features for each rule.
>>>>> - During each pass only render the features that satisfy the rule
>>>>> for that pass.
>>>>> - If there are M rules you will make M FDO queries.
>>>>> => performance (speed) will be unacceptable for anything more than
>>>>> a few themes
>>>>>
>>>>> option 2
>>>>> - Make one query against all the features, remembering (in memory?)
>>>>> all the feature information (attributes / geometry) needed by
>>>>> stylization.
>>>>> - During the initial pass render the features for the first rule.
>>>>> - Iterate over the remaining rules in order, rendering the features
>>>>> for each rule.
>>>>> => performance (memory use) will be unacceptable for data sources
>>>>> containing large numbers of features => this approach goes against
>>>>> the MG architecture of not keeping feature data in memory during
>>>>> stylization (we already break that rule with labels which is what
>>>>> leads to high memory usage spikes for some maps)
>>>>>
>>>>> option 3
>>>>> - Allocate one image per rule.
>>>>> - Make one query against all the features, and render each feature
>>>>> into the image corresponding to the rule it satisifes.
>>>>> - Merge all images at the end.
>>>>> => performance (speed + memory) will be unacceptable for anything
>>>>> more than a few themes
>>>>>
>>>>> option 4
>>>>> ???
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: mapguide-internals-bounces at lists.osgeo.org
>>>>> [mapguide-internals-bounces at lists.osgeo.org
>>>>> ] On Behalf Of Zac Spitzer [zac.spitzer at gmail.com]
>>>>> Sent: Thursday, June 25, 2009 10:36 PM
>>>>> To: MapGuide Internals Mail List
>>>>> Subject: Re: [mapguide-internals] MapGuide RFC 69 - Rendering Layer
>>>>> Styles using Style Order
>>>>>
>>>>> At the moment the only way to achieve this is to split the road
>>>>> network out into multiple layers which is rather ugly.
>>>>>
>>>>> Taking a road network as the example here, it's only at the
>>>>> intersections where this problem occurs.
>>>>>
>>>>> Can some elaborate on how the server currently handles this?
>>>>>
>>>>> z
>>>>>
>>>>> On Fri, Jun 26, 2009 at 12:28 PM, Walt
>>>>> Welton-Lair<walt.welton-lair at autodesk.com> wrote:
>>>>>> "Proposed solution: Add support to the rendering engine to render
>>>>>> layer styles in ordered passes."
>>>>>>
>>>>>> I'd like to see some concrete suggestions in the RFC on how to
>>>>>> actually do this efficiently. Just consider a conservative use
>>>>>> case - say around 10 rules.
>>>>>>
>>>>>> Walt
>>>>>> ________________________________________
>>>>>> From: mapguide-internals-bounces at lists.osgeo.org
>>>>>> [mapguide-internals-bounces at lists.osgeo.org
>>>>>> ] On Behalf Of Zac Spitzer [zac.spitzer at gmail.com]
>>>>>> Sent: Thursday, June 25, 2009 9:29 PM
>>>>>> To: MapGuide Internals Mail List
>>>>>> Subject: [mapguide-internals] MapGuide RFC 69 - Rendering Layer
>>>>>> Styles using Style Order
>>>>>>
>>>>>> I have posted a new RFC
>>>>>>
>>>>>> http://trac.osgeo.org/mapguide/wiki/MapGuideRfc69
>>>>>>
>>>>>> --
>>>>>> Zac Spitzer -
>>>>>> http://zacster.blogspot.com
>>>>>> +61 405 847 168
>>>>>> _______________________________________________
>>>>>> mapguide-internals mailing list
>>>>>> mapguide-internals at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>>>> _______________________________________________
>>>>>> mapguide-internals mailing list
>>>>>> mapguide-internals at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zac Spitzer -
>>>>> http://zacster.blogspot.com
>>>>> +61 405 847 168
>>>>> _______________________________________________
>>>>> mapguide-internals mailing list
>>>>> mapguide-internals at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>>> _______________________________________________
>>>>> mapguide-internals mailing list
>>>>> mapguide-internals at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>>
>>>>
>>>> __________________________________________
>>>>
>>>> Paul Spencer
>>>> Chief Technology Officer
>>>> DM Solutions Group Inc
>>>> http://research.dmsolutions.ca/
>>>>
>>>> _______________________________________________
>>>> mapguide-internals mailing list
>>>> mapguide-internals at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>> _______________________________________________
>>>> mapguide-internals mailing list
>>>> mapguide-internals at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>>
>>>
>>>
>>>
>>> --
>>> Zac Spitzer -
>>> http://zacster.blogspot.com
>>> +61 405 847 168
>>> _______________________________________________
>>> mapguide-internals mailing list
>>> mapguide-internals at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>> _______________________________________________
>>> mapguide-internals mailing list
>>> mapguide-internals at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>>
>>
>>
>>
>> --
>> Zac Spitzer -
>> http://zacster.blogspot.com
>> +61 405 847 168
>> _______________________________________________
>> mapguide-internals mailing list
>> mapguide-internals at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>> _______________________________________________
>> mapguide-internals mailing list
>> mapguide-internals at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>
>> _______________________________________________
>> mapguide-internals mailing list
>> mapguide-internals at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>>
>>
>
>
>
> --
> Zac Spitzer -
> http://zacster.blogspot.com
> +61 405 847 168
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
--
Zac Spitzer -
http://zacster.blogspot.com
+61 405 847 168
More information about the mapguide-internals
mailing list