Map file with 35,000+ Lines... management issue

Thu Jan 24 10:38:57 EST 2008

Yeah, you pointed out a very valid point.
As open layers handled my map file in an efficient manner, I didn't
overloaded myself with implementation of Tiling.

But, there is one more serious thought. Lets try to understand this problem:

Example:

Layer 1 : [shapefile] Roads of a country.

The attributes of this road layer changes w.r.t. zoom-levels. For instance,
at higher zoom level roads seems thinner, and at lower it seems broader.
So, for these zoom-levels, I used Maxscale/Minscale values and developed
multiple layers for this Layer.

Layer 2: [shapefile] Roads of a state

The above case is true with this layer too.

I hope this could magnify my problem.

Regards,
Ritz

On Jan 24, 2008 8:55 PM, Fawcett, David <David.Fawcett at state.mn.us> wrote:

> You mention creating individual layers for each shapefile.  So, does
> this mean that if you have a shapefile of road data for each state you
> are creating a MapServer layer for each shapefile?
>
> If this is the case, you can definitely reduce the number of layers (and
> likely increase performance) by using a tileindex.
>
> A tileindex is a polygon dataset, usually a shapefile, with a polygon
> that defines the boundaries of each individual dataset.  In other words,
> you would use a utility to create a new shapefile with polygons that
> define the boundaries of each of your state road shapefiles.  In the
> attribute table of your tileindex, there is a column that tells
> MapServer where to find the actual shapefile represented by a particular
> feature.
>
> You then create just one layer using the tileindex as the data source.
> Take a look at:  http://mapserver.gis.umn.edu/docs/howto/tileindex if
> you haven't already.
>
> David.
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
> Behalf Of riteshambastha
> Sent: Thursday, January 24, 2008 9:16 AM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> management issue
>
>
> Dear Stephen,
>
> The number of layers exceeded much because I am including each
> individual shapefile multiple times for different Maxcale/Minscale
> values.  So, a shapefile is now called by 3-4 layers, each layer having
> different Maxscale/Minscale values.
>
> I hope I made my point clear.
>
> Regards,
> Ritz
>
>
> Stephen Woodbridge wrote:
> >
> > Ok, I need to ask the obvious question, WHY? do you feel you need 658
> > layers. Is this because you have lots of shapefiles? and most of the
> > layer definitions are the same except for the data source?
> >
> > For Tiger data I have 33000 shapefiles, but I only have about 20+-
> > layers. Are you using tileindexes? Do you know what they are? Just
> > trying to diagnose your situation a little better so we can help.
> >
> > -Steve W
> >
> > ritesh ambastha wrote:
> >> Thanks Bob,
> >>
> >> The map file consists of 658 Layers.
> >> It runs with openlayers and postgis.
> >>
> >> Now, am trying to sort out the best way for solving this issue. Your
> >> reply helped me to view at the problems+solutions in broad spectrum..
> >>
> >> Warm Regards,
> >> Ritesh
> >>
> >> On Jan 24, 2008 1:04 AM, Bob Basques <Bob.Basques at ci.stpaul.mn.us>
> >> wrote:
> >>>
> >>> Ritz,
> >>>
> >>> Whew, 35k lines, that big.  How many layers is that anyway?   The
> >>> Googlish
> >>> mapfile I just did only has 1100 lines in it, and that's mostly for
> >>> readability.  I could probably get it down to half that size if I
> >>> tried.
> >>>
> >>> Don't know what I can contribute as a "Best Practice", because I
> >>> feel that in most cases, that form follows function, if you need a
> >>> capability, you build it.  Anyway, here are some of my thoughts.
> >>>
> >>> These same sorts of performance questions crossed my mind too.  The
> >>> Googlish mapfile I've been working on has 72 separate STYLE
> >>> definitions for example.
> >>> Mostly ranged around threshholding of certain styles.  I can see
> that
> >>> adding
> >>> in the Water bodies, Railroad, Parks, and such, is really going to
> make
> >>> this
> >>> thing big.  I may just do those as separate MapFiles though since
> >>> GeoMoose
> >>> handles things like these separations very nicely.
> >>>
> >>> These are some of the primary reasons that contributed to the way
> >>> we've built GeoMoose as a client and why it runs against MapServer
> >>> CGI, so that it can abstract the layer calls in this fashion.  We're
>
> >>> running 135+ layers internally at the moment, and they all have
> >>> their own MAPFILE and are all
> >>> called separately from the client.  It has made life much easier
> with
> >>> regard
> >>> to MapFile creation and maintenance, since each data custodian
> handles
> >>> their
> >>> respective MAPFILE.  The performance issues are minimized well since
> >>> even
> >>> the data intensive layers are not too bad from a performance
> standpoint.
> >>>
> >>> But even my Googlish looking mapfile got prettty big (in my opinion)
> for
> >>> simply displaying centerlines of streets.   I've learned quite a bit
> >>> from
> >>> these exercises about these types of questions.  While I have yet to
>
> >>> attack the performance side of things, I anticpate that I'll need to
>
> >>> segregate the
> >>> data out at differing thresholds in order to gain some performance
> >>> boots.
> >>> We're all about doing dynamic requests here since many of our
> datasets
> >>> change very frequently, in some cases, down to the minute.  I may
> look
> >>> into
> >>> tiling at some point in the future, but it will still be only for
> some
> >>> of
> >>> the layers, there will still be a need to have this dynamic request
> >>> structure in place.
> >>>
> >>> bobb
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Bob Basques
> >>> GIS Systems Developer
> >>> City of Saint Paul, MN
> >>>
> >>>
> >>> GISmo
> >>> Powered by
> >>> GeoMOOSE
> >>>
> >>>
> >>>>>> riteshambastha <ritesh.linux at GMAIL.COM> wrote:
> >>> Dear Readers,
> >>>
> >>> I have developed a map file with more than 35,000 Lines. Its size
> >>> will grow by double/triple in next few months. Now, I am trying to
> >>> tune my map file by
> >>> removing unwanted lines. Still, I am bit confused about its
> maintenance.
> >>>
> >>>
> >>> Please throw some lights over writing map files by following best
> >>> practices.
> >>>
> >>> Thanks in advance.
> >>>
> >>> Regards,
> >>> Ritz
> >>> --
> >>> View this message in context:
> >>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issu
> >>> e-tp15048892p15048892.html
> >>>
> >>> Sent from the Mapserver - User mailing list archive at Nabble.com.
> >>>
> >>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-issu
> e-tp15048892p15065742.html
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>

-- 
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20080124/2b3e1306/attachment.html