Organizing large temporal datasets (WAS: Increasing Maxlayers limit - anyone have spare Windows binaries? :))

[Brent Fraser]

Bonnie,

  I wasn't sure if WMS + tileindex + time attribute was possible.  I've
done similar things with the Mapserver protocol (not WMS).  I'm not sure
is possible; ther emay be a limitation with Mapserver or WMS.  And I
don't know if any WMS clients will allow the user to form a standard
attribute query, much less a temporal one.

  But it may be worth looking into...

Brent

> Hi Brent,
>
> Thanks! I had no idea that TILEINDEXing could be used for time-variant
> data as well as spatial-variant data.
>
> I've been doing some googling trying to find out more information on this
> (no experience whatsoever with TILEINDEX thus far), and this is the best
> I've come up with:
> http://ms.gis.umn.edu/docs/howto/wcs_server
> http://mapserver.gis.umn.edu/cgi-bin/wiki.pl?WCSMapServer
>
> The section which covers "Spatio/Temporal Indexes" in the first document
> is I believe the methodology you're referring to, and while it's very
> helpful, I still have a few questions.
>
> 1) So far I have only read about temporal timeindexing with respect to
> serving WCS (Web Coverage Service) rather than WMS or WFS. Is it possible
> to use the same methodology but serve WMS or WFS layers? I don't know much
> about WCS and I'm not sure what clients (off the shelf - we are not going
> to be developing any client interface for our users) can access it.
>
> 2) Does anybody know of any additional documentation/tutorials/examples
> using TILEINDEX for raster and vector data which is spatially nonvariant
> and temporally variant? :)
>
>
> From what I've seen of WCS, it's very nifty, and I hope to be able to
> provide our data via WCS as well as WMS and WFS - but that will come down
> to whether our users' clients (ArcExplorer, ArcMap, and similar) are able
> to make use of it.
>
>
> Thanks again for the info - looks like I've got some thinking to do!
>
>
> Cheers,
>
>
> Bonnie
>
> __________________________________
> Bonnie Stewart
> Research Officer
> Satellite Remote Sensing Services
> Information Access Division
> Department of Land Information
> 65 Brockway Road, FLOREAT WA 6014
> Ph. (08) 9387 0342 | Fx. (08) 9383 7142
> bonnie.stewart at dli.wa.gov.au
> www.dli.wa.gov.au
>
> ________________________________
>
> From: Brent Fraser [mailto:bfraser at geoanalytic.com]
> Sent: Mon 11/07/2005 11:09 PM
> To: Bonnie Stewart; MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: [Maybe SPAM] Re: Organizing large temporal datasets (WAS:
> Increasing Maxlayers limit - anyone have spare Windows binaries? :))
>
>
>
> Bonnie,
>
>   You may want to look into the TILEINDEX capability of MapServer.  A
> tileindex is a vector datasource (shapefile, PostGIS database, etc) of
> outlines of other data, either raster or vector.  Typically it's used to
> show and entire county of mosaicked and clipped airphotos or Census
> planimetric mapping vectors.  In those cases MapServer does a spatial
> query
> to select the data files to read for rendering into the output graphic
> file.
> A tileindex could be used to represent data divided by time (as well as
> geography) such as satellite imagery.  A recent enhancement (v4.?) allows
> the TILEINDEX to be a separate layer form the underlying data, and because
> of that can handle a user-specified query on attributes stored in the
> tileindex (e.g. satellite id, acquisition time, etc).
>
>   So one possibility might be to have two layers (one for the data and one
> for the tileindex) per each of your 10-15 products.  Then query the
> tileindex based on the product name/id and the date range.  I'm not sure
> how
> to get the query from the WMS client to the server, so you may have to
> look
> into that...
>
>   Since your data is relatively dynamic, you may want to use a PostGIS
> database as the format for your tileindex to avoid any file locking
> problems
> with shapefiles.
>
> Brent Fraser
>
>
> ----- Original Message -----
> From: Bonnie Stewart
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Sent: Monday, July 11, 2005 12:05 AM
> Subject: [UMN_MAPSERVER-USERS] Increasing Maxlayers limit - anyone have
> spare Windows binaries? :)
>
>
> Hi all,
>
> I'm working on a way to make a large amount of our vector and raster data
> available via WMS and WFS, and I've come up against the MAXLAYERS setting
> (200 in our install I think).
>
> I'm already trying to get around the need for having so many layers in a
> single mapfile by reducing the granularity available as you get to
> mapfiles
> which cover a larger time period - let me see if I can explain what I mean
> without rambling on too much.
>
> At the lowest level of granularity, we're providing access to raster and
> vector files which are derived from satellite passes - and there may be
> 10-40 passes per day, depending on the product. I've already set up a
> process to automatically generate a mapfile containing the last 5 days
> worth
> of data, using the GROUP parameter to group them according to the day.
> This
> is jolly nifty for those customers who need to look at the recent info on
> a
> pass by pass basis.
>
> However, if I try to give people access to the entire archive (several
> years
> worth, and 10-15 different products, at 10-40 passes per day) on a pass by
> pass basis, the number of layers involved will be horrendous, so I'm
> trying
> to reduce the granularity at which archival data is available.
>
> My current plan is to (programmatically) create a series of weekly
> mapfiles,
> structured in the same way as mentioned above, with individual satellite
> passes grouped by day. I would have 52 of those for each year, and then to
> wrap them up so that people could have an overview of what's available,
> I'm
> planning to create some mapfiles (either yearly or monthly, maybe both),
> which connect to these weekly mapfiles via WMS, and pull in each daily
> group
> from the 52 weekly mapfiles, and define them as a single layer.
>
> So I think I would end up with a mapfile for each year, with a group for
> each week, and under each group, 7 individual layers - one for each day.
> Hopefully then there are two options:
> 1) People's clients are able to pull apart the vector data, and maybe do a
> query (for instance) to only display the lightning data from one
> particular
> satellite pass - although having used ArcMap and ArcExplorer for testing
> I'm
> not too hopeful about this;
> 2) Specific file naming conventions could allow users to find the week
> they're interested in in the yearly mapfile, and then work out what the
> mapfile name/URL for that week's lower level mapfile would be.
>
> So, now I'm finally going to get to my point - even using this structure,
> a
> MAXLAYERS setting of 200 is too low for me - I can do about 5 or 6 days of
> satellite passes in 200 layers, but to get to a whole week I would need to
> increase the MAXLAYERS parameter. To have a layer for each day in my
> yearly
> mapfile, I'd need a MAXLAYERS of around 370.
>
> I gather it's relatively easy to change the parameter and recompile
> Mapserver, but as I have very little experience in this area (and I've
> seen
> so many emails on this list seeking help with compiling mapserver!) I was
> hoping somebody might possibly have already done this, and might have
> Windows binaries compiled with a higher MAXLAYERS that they would be
> willing
> to share.
>
> Alternatively, if the structure I described above sounds silly or you can
> think of something more effective, please let me know :) We do have
> another
> approach in mind for the vector data, which is to produce an ongoing
> concatenated shapefile which contains all the features for that product
> (for
> instance, as a new pass's worth of lightning data becomes available, it is
> automatically added in to one big Lightning shapefile), and then to use
> classes to colour them up a bit according to the time that each event
> occurred. It's a nice theory for the vector info, but I'd still need to do
> something about the raster info.
>
>
> Any advice (or binaries!) would be very much appreciated :)
>
>
> Thanks,
>
>
> Bonnie
>
> __________________________________
> Bonnie Stewart
> Research Officer
> Satellite Remote Sensing Services
> Information Access Division
> Department of Land Information
> 65 Brockway Road, FLOREAT WA 6014
> Ph. (08) 9387 0342 | Fx. (08) 9383 7142
> bonnie.stewart at dli.wa.gov.au
> www.dli.wa.gov.au
>
>
>
>
>
> This e-mail and any files transmitted with it are intended only for the
> use of the addressee(s).  It may contain information that is confidential
> and privileged.  If you are not an intended recipient, any use,
> interference with, disclosure, distribution or copying of this material is
> unauthorised and prohibited. If you receive this in error, please notify
> the author by Return email to the sender. Information in this message not
> relating to the official business of DLI shall be understood as neither
> given nor endorsed by it.  While every care is taken, it is recommended
> that you scan any attachments for viruses. DLI liability is limited to
> re-supplying affected attachments.
>
>
>