Organizing large temporal datasets (WAS: Increasing Maxlayers limit - anyone have spare Windows binaries? :))

[Brent Fraser]

Bonnie,

  You may want to look into the TILEINDEX capability of MapServer.  A
tileindex is a vector datasource (shapefile, PostGIS database, etc) of
outlines of other data, either raster or vector.  Typically it's used to
show and entire county of mosaicked and clipped airphotos or Census
planimetric mapping vectors.  In those cases MapServer does a spatial query
to select the data files to read for rendering into the output graphic file.
A tileindex could be used to represent data divided by time (as well as
geography) such as satellite imagery.  A recent enhancement (v4.?) allows
the TILEINDEX to be a separate layer form the underlying data, and because
of that can handle a user-specified query on attributes stored in the
tileindex (e.g. satellite id, acquisition time, etc).

  So one possibility might be to have two layers (one for the data and one
for the tileindex) per each of your 10-15 products.  Then query the
tileindex based on the product name/id and the date range.  I'm not sure how
to get the query from the WMS client to the server, so you may have to look
into that...

  Since your data is relatively dynamic, you may want to use a PostGIS
database as the format for your tileindex to avoid any file locking problems
with shapefiles.

Brent Fraser


----- Original Message ----- 
From: Bonnie Stewart
To: MAPSERVER-USERS at LISTS.UMN.EDU
Sent: Monday, July 11, 2005 12:05 AM
Subject: [UMN_MAPSERVER-USERS] Increasing Maxlayers limit - anyone have
spare Windows binaries? :)


Hi all,

I'm working on a way to make a large amount of our vector and raster data
available via WMS and WFS, and I've come up against the MAXLAYERS setting
(200 in our install I think).

I'm already trying to get around the need for having so many layers in a
single mapfile by reducing the granularity available as you get to mapfiles
which cover a larger time period - let me see if I can explain what I mean
without rambling on too much.

At the lowest level of granularity, we're providing access to raster and
vector files which are derived from satellite passes - and there may be
10-40 passes per day, depending on the product. I've already set up a
process to automatically generate a mapfile containing the last 5 days worth
of data, using the GROUP parameter to group them according to the day. This
is jolly nifty for those customers who need to look at the recent info on a
pass by pass basis.

However, if I try to give people access to the entire archive (several years
worth, and 10-15 different products, at 10-40 passes per day) on a pass by
pass basis, the number of layers involved will be horrendous, so I'm trying
to reduce the granularity at which archival data is available.

My current plan is to (programmatically) create a series of weekly mapfiles,
structured in the same way as mentioned above, with individual satellite
passes grouped by day. I would have 52 of those for each year, and then to
wrap them up so that people could have an overview of what's available, I'm
planning to create some mapfiles (either yearly or monthly, maybe both),
which connect to these weekly mapfiles via WMS, and pull in each daily group
from the 52 weekly mapfiles, and define them as a single layer.

So I think I would end up with a mapfile for each year, with a group for
each week, and under each group, 7 individual layers - one for each day.
Hopefully then there are two options:
1) People's clients are able to pull apart the vector data, and maybe do a
query (for instance) to only display the lightning data from one particular
satellite pass - although having used ArcMap and ArcExplorer for testing I'm
not too hopeful about this;
2) Specific file naming conventions could allow users to find the week
they're interested in in the yearly mapfile, and then work out what the
mapfile name/URL for that week's lower level mapfile would be.

So, now I'm finally going to get to my point - even using this structure, a
MAXLAYERS setting of 200 is too low for me - I can do about 5 or 6 days of
satellite passes in 200 layers, but to get to a whole week I would need to
increase the MAXLAYERS parameter. To have a layer for each day in my yearly
mapfile, I'd need a MAXLAYERS of around 370.

I gather it's relatively easy to change the parameter and recompile
Mapserver, but as I have very little experience in this area (and I've seen
so many emails on this list seeking help with compiling mapserver!) I was
hoping somebody might possibly have already done this, and might have
Windows binaries compiled with a higher MAXLAYERS that they would be willing
to share.

Alternatively, if the structure I described above sounds silly or you can
think of something more effective, please let me know :) We do have another
approach in mind for the vector data, which is to produce an ongoing
concatenated shapefile which contains all the features for that product (for
instance, as a new pass's worth of lightning data becomes available, it is
automatically added in to one big Lightning shapefile), and then to use
classes to colour them up a bit according to the time that each event
occurred. It's a nice theory for the vector info, but I'd still need to do
something about the raster info.


Any advice (or binaries!) would be very much appreciated :)


Thanks,


Bonnie

__________________________________
Bonnie Stewart
Research Officer
Satellite Remote Sensing Services
Information Access Division
Department of Land Information
65 Brockway Road, FLOREAT WA 6014
Ph. (08) 9387 0342 | Fx. (08) 9383 7142
bonnie.stewart at dli.wa.gov.au
www.dli.wa.gov.au