[mapserver-dev] Upper Limit to Number of Layers in Map file

[Stephen Woodbridge]

Peter,

Are you aware that you can use a database as a TILEINDEX. From your 
description is seems like you could create an appropriate query that 
selects the needed rasters and then reference this as a TILEINDEX for a 
raster layer. This would allow for most of the dynamic selection to 
happen in the database and mapserver could be pretty immune to 
everything else.

Look at creating a LAYER that will be your TILEINDEX and using variable 
substitution to help construct the appropriate SQL. You should then be 
able to reference that LAYER as your TILEINDEX "my_db_tindex" for a 
raster layer.

I believe other people have done things like this to dynamically create 
a sliding time window over a large set of raster so they could select 
just the raster within the windowed timeframe. It sounds like you may be 
doing something similar, but with some additional dimensions to it.

-Steve W.

Peter Willis wrote:
> Frank Warmerdam wrote:
>> Peter Willis wrote:
>>> Hello,
>>>
>>> I am attempting to use mapserver as a WMS server for
>>> a large number of rasters.
>>>
>>> The system I am using has a collection of over 21000
>>> geo-referenced  rasters.
>>>
>>> These can be sorted into various groups, however
>>> any specific group of rasters may exceed 400 or more.
>>>
>>> I have tried modifying the value MS_MAXLAYERS in
>>> map.h and recompiling but am unsure if I will exceed
>>> the binary size of whatever variables are affected by
>>> that constant.
>>>
>>> What is the maximum, safe, binary size of the
>>> MS_MAXLAYERS constant that will not overflow
>>> the variables/procedures that use it?
>>
>> Peter,
>>
>> First, I would note that MS_MAXLAYERS was made obsolete in MapServer 5.0.
>> If you upgrade to at least 5.0 MapServer defaults to dynamic allocation
>> of the layers array and in theory millions of layers would work. I 
>> imagine
>> you could set MS_MAXLAYERS to a value in excess of 21000 in older 
>> versions
>> if you wanted; however, I think that the memory overhead, and 
>> initialization
>> startup time would likely become painful.
>>
>> But I really think you need to approach this as a tileindexing issue.
>> Raster layers can reference an index file containing many individual
>> raster files that are related in some fashion (for instance spatial
>> tiles making up a mosaic).  There is also some limited support for
>> selecting files out of a tileindex based on time ranges that might be
>> valuable.
>>
>> If that is not practical - for instance if the rasters are essentially
>> unrelated and need to be addressed individually - then perhaps you
>> should consider a dynamic map service using MapScript to to treat each
>> file/granule/etc as a map service.  This approach might be accomplished
>> using the MapScript WxS services:
>>
>>   http://mapserver.org/ogc/mapscript.html
>>
>> Best regards,
> 
> 
> Hello,
> 
> As previously stated we can/are breaking down the list of rasters into
> categorized groups based on the origin and content of their data.
> So, from a logistical standpoint, there will never be more than a few
> hundred rasters that ever, theoretically, make their way as layers
> into a single 'map' file.(usually between 1 and 10)
> 
> --unimportant background information--
> 
> The system we are using writes a map file, on the fly,
> based on the users selected categorical input. Map files
> are never manually written by anyone. There is a metadata
> database that holds all layer entries and associated
> meta information for those layers, regardless of type
> (ie; raster, point, line, polygon, etc. ), origin, category.
> This database is queried via CGI to generate the proper
> 'MAP' file for each user session for the 'non-WMS' mapserver
> side of things.
> This works fine. Only rasters selected by the user,
> or associated (in the relational database) with a specific
> project/context/content make their way into the map
> file as layers. This is more like a content management
> system for mapserver(s).
> 
> --The WMS Problem--
> In the current context, I am using mapserver primarily for **WMS**.
> At this point the system is being expanded to auto-generate
> dynamic WMS oriented map files. The difference, in this case,
> from the previous setup is that the WMS map file needs to list
> ALL the available layers for a particular context.
> Are *drawing* optimizations, such as tile indexing,
> relevant in the context of WMS...(??) I didn't
> think that was the case.
> 
> 
> I will look at the latest version of mapserver since
> we are only at version 4.10.1. . That should resolve
> the maximum layers issue.
> 
> 
> All the best,
> 
> Peter
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev