[mapserver-dev] Upper Limit to Number of Layers in Map file

Tue Apr 21 16:41:21 EDT 2009

Hello,

Thank you Stephen for that additional tile index information.
I will return to the docs and see if I can modify things to
utilize the tile indexing features.

The database, and front end, we are using pre-dates the
tile indexing features of mapserver. Thus queries are
being made with respect to the original file metadata
rather than tile index data.

Applying tile indexes and using that data is a future project
that we may take up at a later date. Unfortunately it requires
a bit of a rewrite that I don't currently have a time
allocation for.

I certainly would like to focus more attention on the
content management side of the mapserver/middleware things.
The tile indexing would get done in that context.

All the best,

Peter

Stephen Woodbridge wrote:
> Peter,
> 
> Are you aware that you can use a database as a TILEINDEX. From your 
> description is seems like you could create an appropriate query that 
> selects the needed rasters and then reference this as a TILEINDEX for a 
> raster layer. This would allow for most of the dynamic selection to 
> happen in the database and mapserver could be pretty immune to 
> everything else.
> 
> Look at creating a LAYER that will be your TILEINDEX and using variable 
> substitution to help construct the appropriate SQL. You should then be 
> able to reference that LAYER as your TILEINDEX "my_db_tindex" for a 
> raster layer.
> 
> I believe other people have done things like this to dynamically create 
> a sliding time window over a large set of raster so they could select 
> just the raster within the windowed timeframe. It sounds like you may be 
> doing something similar, but with some additional dimensions to it.
> 
> -Steve W.
> 
> Peter Willis wrote:
>> Frank Warmerdam wrote:
>>> Peter Willis wrote:
>>>> Hello,
>>>>
>>>> I am attempting to use mapserver as a WMS server for
>>>> a large number of rasters.
>>>>
>>>> The system I am using has a collection of over 21000
>>>> geo-referenced  rasters.
>>>>
>>>> These can be sorted into various groups, however
>>>> any specific group of rasters may exceed 400 or more.
>>>>
>>>> I have tried modifying the value MS_MAXLAYERS in
>>>> map.h and recompiling but am unsure if I will exceed
>>>> the binary size of whatever variables are affected by
>>>> that constant.
>>>>
>>>> What is the maximum, safe, binary size of the
>>>> MS_MAXLAYERS constant that will not overflow
>>>> the variables/procedures that use it?
>>>
>>> Peter,
>>>
>>> First, I would note that MS_MAXLAYERS was made obsolete in MapServer 
>>> 5.0.
>>> If you upgrade to at least 5.0 MapServer defaults to dynamic allocation
>>> of the layers array and in theory millions of layers would work. I 
>>> imagine
>>> you could set MS_MAXLAYERS to a value in excess of 21000 in older 
>>> versions
>>> if you wanted; however, I think that the memory overhead, and 
>>> initialization
>>> startup time would likely become painful.
>>>
>>> But I really think you need to approach this as a tileindexing issue.
>>> Raster layers can reference an index file containing many individual
>>> raster files that are related in some fashion (for instance spatial
>>> tiles making up a mosaic).  There is also some limited support for
>>> selecting files out of a tileindex based on time ranges that might be
>>> valuable.
>>>
>>> If that is not practical - for instance if the rasters are essentially
>>> unrelated and need to be addressed individually - then perhaps you
>>> should consider a dynamic map service using MapScript to to treat each
>>> file/granule/etc as a map service.  This approach might be accomplished
>>> using the MapScript WxS services:
>>>
>>>   http://mapserver.org/ogc/mapscript.html
>>>
>>> Best regards,
>>
>>
>> Hello,
>>
>> As previously stated we can/are breaking down the list of rasters into
>> categorized groups based on the origin and content of their data.
>> So, from a logistical standpoint, there will never be more than a few
>> hundred rasters that ever, theoretically, make their way as layers
>> into a single 'map' file.(usually between 1 and 10)
>>
>> --unimportant background information--
>>
>> The system we are using writes a map file, on the fly,
>> based on the users selected categorical input. Map files
>> are never manually written by anyone. There is a metadata
>> database that holds all layer entries and associated
>> meta information for those layers, regardless of type
>> (ie; raster, point, line, polygon, etc. ), origin, category.
>> This database is queried via CGI to generate the proper
>> 'MAP' file for each user session for the 'non-WMS' mapserver
>> side of things.
>> This works fine. Only rasters selected by the user,
>> or associated (in the relational database) with a specific
>> project/context/content make their way into the map
>> file as layers. This is more like a content management
>> system for mapserver(s).
>>
>> --The WMS Problem--
>> In the current context, I am using mapserver primarily for **WMS**.
>> At this point the system is being expanded to auto-generate
>> dynamic WMS oriented map files. The difference, in this case,
>> from the previous setup is that the WMS map file needs to list
>> ALL the available layers for a particular context.
>> Are *drawing* optimizations, such as tile indexing,
>> relevant in the context of WMS...(??) I didn't
>> think that was the case.
>>
>>
>> I will look at the latest version of mapserver since
>> we are only at version 4.10.1. . That should resolve
>> the maximum layers issue.
>>
>>
>> All the best,
>>
>> Peter
>> _______________________________________________
>> mapserver-dev mailing list
>> mapserver-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> 
> 
> 
> 