[mapserver-users] Thousands of layers?

[Ed McNierney]

James -

I've been meaning to send a message out to the list for a while, and you've given me a good excuse <g>!

We're in the business of hosting MapServer applications for customers.  We like to work with MapServer developers to build the applications, and then we provide the hosting/serving infrastructure.  We've got a LOT of data sets in-house, so we can also make that data available to our customers.

In one recent case we also built the application for the customer.  It's a subscription-based site called MapCard (www.mapcard.com).  For $20 a year you get access to the site.  I didn't bother posting info about it to the list because it was subscription-only.

However, we have now set up a test-drive system for them.  You can register and get free access to the site for 24 hours.  So if you'd like to see what a large-scale MapServer application looks like, you can give it a try.  You need to go to http://topozone.mapcard.com to get the test drive signup page. You can also subscribe for $15 instead of $20 if you go this route (commercial disclaimer: obviously we get paid to host the site, and if you sign up from this test drive page we get a referral fee).

Although I may be mistaken, I will make the claim that (in terms of bytes of data available) this is the largest MapServer application to date.  MapCard offers HTML and Java applet viewers; the applet lets you annotate the maps, and you can save annotated maps for retrieval later.  You can also download high-res (250dpi) map images for printing.

In terms of data, MapCard provides:

	250dpi topographic maps of the entire US (of course <g>)
	1-meter DOQQ aerial photography of the entire US
	Public Land Survey System (PLSS) grids of the entire US with search tools
	Bathymetric maps for several hundred lakes in the Midwest and Texas (more coming)
	GDT's complete Dynamap/2000 street mapping data for the entire US

So - to get back to the question...

The GDT Dynamap/2000 data is comparable to the data sets you're using.  You don't need (or want) one MapServer "layer" for every shapefile.  Take a look at the TILEINDEX statement and use the county-level data that way.  You essentially create a spatial index file that tracks the bounding box of every one of your county shapefiles.  That TILEINDEX is used for a single LAYER object; MapServer then consults the index file when serving a map request.  It quickly figures out the very small number of county files (out of the very large set) required to draw the map currently requested.  Then those county files are drawn.  This technique is the best way to deal with large data sets, and it's exactly how we've managed the GDT data in the MapCard application.

After you've looked into TILEINDEX I'm sure you'll have more questions, so let us know what you need!

	- Ed

Ed McNierney
Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
ed at topozone.com
(978) 251-4242


-----Original Message-----
From: James Jefferson [mailto:jjeffers at deskmedia.com]
Sent: Thursday, May 02, 2002 11:48 AM
To: mapserver-users at lists.gis.umn.edu
Subject: [mapserver-users] Thousands of layers?


Hello everyone,

I've been using mapserver for about a year now and one of the problems I've 
struggled with is making a detailed map of the United States - down to the 
street level.

I have Shapefiles generated from the 1997 TIGER, broken down by county. I've 
noticed in the mapserver documentation that there is a limit of 50 layers per 
mapfile (incidentally, the define in map.h is set to 100). There are a few 
thousand counties in the US, and consquently I think I need a approximately 
20,000 shapefiles to represent the different layers for each county.

If I recompile mapserver with MS_MAXLAYERS set higher, how will that affect 
performance? Looking at source it appears that the MS_MAXLAYERS is just used 
to allocate memory for each layer. How does mapserver decide what maps to 
load? I'm assuming it uses the bounding boxes associated with each shapefile 
to determine what file to load? If so, are those read each time mapserver is 
invoxed, or can it have a index file?

Is anybody else using mapserver for this scale? Advice? Am I on the right 
track at all?

Thanks,

-James Jefferson
Winona, Minnesota