[mapserver-dev] Upper Limit to Number of Layers in Map file

[Peter Willis]

Frank Warmerdam wrote:
> Peter Willis wrote:
>> Hello,
>>
>> I am attempting to use mapserver as a WMS server for
>> a large number of rasters.
>>
>> The system I am using has a collection of over 21000
>> geo-referenced  rasters.
>>
>> These can be sorted into various groups, however
>> any specific group of rasters may exceed 400 or more.
>>
>> I have tried modifying the value MS_MAXLAYERS in
>> map.h and recompiling but am unsure if I will exceed
>> the binary size of whatever variables are affected by
>> that constant.
>>
>> What is the maximum, safe, binary size of the
>> MS_MAXLAYERS constant that will not overflow
>> the variables/procedures that use it?
> 
> Peter,
> 
> First, I would note that MS_MAXLAYERS was made obsolete in MapServer 5.0.
> If you upgrade to at least 5.0 MapServer defaults to dynamic allocation
> of the layers array and in theory millions of layers would work. I imagine
> you could set MS_MAXLAYERS to a value in excess of 21000 in older versions
> if you wanted; however, I think that the memory overhead, and 
> initialization
> startup time would likely become painful.
> 
> But I really think you need to approach this as a tileindexing issue.
> Raster layers can reference an index file containing many individual
> raster files that are related in some fashion (for instance spatial
> tiles making up a mosaic).  There is also some limited support for
> selecting files out of a tileindex based on time ranges that might be
> valuable.
> 
> If that is not practical - for instance if the rasters are essentially
> unrelated and need to be addressed individually - then perhaps you
> should consider a dynamic map service using MapScript to to treat each
> file/granule/etc as a map service.  This approach might be accomplished
> using the MapScript WxS services:
> 
>   http://mapserver.org/ogc/mapscript.html
> 
> Best regards,


Hello,

As previously stated we can/are breaking down the list of rasters into
categorized groups based on the origin and content of their data.
So, from a logistical standpoint, there will never be more than a few
hundred rasters that ever, theoretically, make their way as layers
into a single 'map' file.(usually between 1 and 10)

--unimportant background information--

The system we are using writes a map file, on the fly,
based on the users selected categorical input. Map files
are never manually written by anyone. There is a metadata
database that holds all layer entries and associated
meta information for those layers, regardless of type
(ie; raster, point, line, polygon, etc. ), origin, category.
This database is queried via CGI to generate the proper
'MAP' file for each user session for the 'non-WMS' mapserver
side of things.
This works fine. Only rasters selected by the user,
or associated (in the relational database) with a specific
project/context/content make their way into the map
file as layers. This is more like a content management
system for mapserver(s).

--The WMS Problem--
In the current context, I am using mapserver primarily for **WMS**.
At this point the system is being expanded to auto-generate
dynamic WMS oriented map files. The difference, in this case,
from the previous setup is that the WMS map file needs to list
ALL the available layers for a particular context.
Are *drawing* optimizations, such as tile indexing,
relevant in the context of WMS...(??) I didn't
think that was the case.


I will look at the latest version of mapserver since
we are only at version 4.10.1. . That should resolve
the maximum layers issue.


All the best,

Peter