Map file with 35,000+ Lines... management issue

[Stephen Woodbridge]

OK, I think there is a simple solution for you, You need to create 
tileindexes. tileindexes are a way to group a collection of shapefiles 
that have the same content like roads into a single layer. Say you have 
a file structure like:

data/city1/roads.shp
data/city2/roads.shp
data/city3/roads.shp
etc

And if you have a layer for each city/roads.shp you can combine all 
these into a single layer like this:

find data -name roads.shp -print > roads-idx.in
tile4ms roads-idx.in roads-idx.shp
shptree roads-idx.shp

And in your mapfile create a layer like:

LAYER
   ...
   TILEINDEX "roads-idx.shp"
   # instead of a DATA statement
   ...
END

Now all these shapefiles in the tileindex will be considered in a very 
efficient manner.

-Steve

riteshambastha wrote:
> Dear Stephen, 
> 
> The number of layers exceeded much because I am including each individual
> shapefile multiple times for different Maxcale/Minscale values.  So, a
> shapefile is now called by 3-4 layers, each layer having different
> Maxscale/Minscale values. 
> 
> I hope I made my point clear.
> 
> Regards,
> Ritz
> 
> 
> Stephen Woodbridge wrote:
>> Ok, I need to ask the obvious question, WHY? do you feel you need 658 
>> layers. Is this because you have lots of shapefiles? and most of the 
>> layer definitions are the same except for the data source?
>>
>> For Tiger data I have 33000 shapefiles, but I only have about 20+- 
>> layers. Are you using tileindexes? Do you know what they are? Just 
>> trying to diagnose your situation a little better so we can help.
>>
>> -Steve W
>>
>> ritesh ambastha wrote:
>>> Thanks Bob,
>>>
>>> The map file consists of 658 Layers.
>>> It runs with openlayers and postgis.
>>>
>>> Now, am trying to sort out the best way for solving this issue.
>>> Your reply helped me to view at the problems+solutions in broad
>>> spectrum..
>>>
>>> Warm Regards,
>>> Ritesh
>>>
>>> On Jan 24, 2008 1:04 AM, Bob Basques <Bob.Basques at ci.stpaul.mn.us> wrote:
>>>> Ritz,
>>>>
>>>> Whew, 35k lines, that big.  How many layers is that anyway?   The
>>>> Googlish
>>>> mapfile I just did only has 1100 lines in it, and that's mostly for
>>>> readability.  I could probably get it down to half that size if I tried.
>>>>
>>>> Don't know what I can contribute as a "Best Practice", because I feel
>>>> that
>>>> in most cases, that form follows function, if you need a capability, you
>>>> build it.  Anyway, here are some of my thoughts.
>>>>
>>>> These same sorts of performance questions crossed my mind too.  The
>>>> Googlish
>>>> mapfile I've been working on has 72 separate STYLE definitions for
>>>> example.
>>>> Mostly ranged around threshholding of certain styles.  I can see that
>>>> adding
>>>> in the Water bodies, Railroad, Parks, and such, is really going to make
>>>> this
>>>> thing big.  I may just do those as separate MapFiles though since
>>>> GeoMoose
>>>> handles things like these separations very nicely.
>>>>
>>>> These are some of the primary reasons that contributed to the way we've
>>>> built GeoMoose as a client and why it runs against MapServer CGI, so
>>>> that it
>>>> can abstract the layer calls in this fashion.  We're running 135+ layers
>>>> internally at the moment, and they all have their own MAPFILE and are
>>>> all
>>>> called separately from the client.  It has made life much easier with
>>>> regard
>>>> to MapFile creation and maintenance, since each data custodian handles
>>>> their
>>>> respective MAPFILE.  The performance issues are minimized well since
>>>> even
>>>> the data intensive layers are not too bad from a performance standpoint.
>>>>
>>>> But even my Googlish looking mapfile got prettty big (in my opinion) for
>>>> simply displaying centerlines of streets.   I've learned quite a bit
>>>> from
>>>> these exercises about these types of questions.  While I have yet to
>>>> attack
>>>> the performance side of things, I anticpate that I'll need to segregate
>>>> the
>>>> data out at differing thresholds in order to gain some performance
>>>> boots.
>>>> We're all about doing dynamic requests here since many of our datasets
>>>> change very frequently, in some cases, down to the minute.  I may look
>>>> into
>>>> tiling at some point in the future, but it will still be only for some
>>>> of
>>>> the layers, there will still be a need to have this dynamic request
>>>> structure in place.
>>>>
>>>> bobb
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Bob Basques
>>>> GIS Systems Developer
>>>> City of Saint Paul, MN
>>>>
>>>>
>>>> GISmo
>>>> Powered by
>>>> GeoMOOSE
>>>>
>>>>
>>>>>>> riteshambastha <ritesh.linux at GMAIL.COM> wrote:
>>>> Dear Readers,
>>>>
>>>> I have developed a map file with more than 35,000 Lines. Its size will
>>>> grow
>>>> by double/triple in next few months. Now, I am trying to tune my map
>>>> file by
>>>> removing unwanted lines. Still, I am bit confused about its maintenance.
>>>>
>>>>
>>>> Please throw some lights over writing map files by following best
>>>> practices.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Regards,
>>>> Ritz
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>>>>
>>>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>