using Mapserver for on-the-fly raster data?

[Frank Warmerdam]

On 7/6/05, Jeff de La Beaujardiere <jeff2005 at svs.gsfc.nasa.gov> wrote:
> I have a set of huge raw binary datafiles containing output from
> someone's numerical simulation (6GB/file, 4files/day). Each file
> contains values for a variety of atmospheric quantities, each at 20
> timesteps, and some at multiple elevations. I would like to offer
> access to at least some of the quantities via WMS protocol using
> Mapserver.
> 
> The brute force way works: I can pre-generate PNG or GeoTIFF images of
> selected quantities, and then generate a mapfile that includes all the
> images. The obvious disadvantage is that I will have many files laying
> around whether they ever get used or not.
> 
> I would like a more elegant approach, wherein Mapserver's WMS service
> metadata can advertise what quantities are available, but the data are
> extracted from the 6GB model file only upon request. Is it possible to
> do something like this? I see a few mentions of database connections
> in the Mapserver documentation but must admit that I am somewhat
> mystified.

Jeff, 

Are the 6GB files encoded in some complex way, or would it be
practical to create header files pointing into the big files for each 
raster product you want to advertise?  ie. are the "bands" uncompressed,
and at predictable locations in the big files? 

If so, it might make sense to create a .vrt "raw file header" for each band
you want to advertise, and then build your map around these.  

Another approach would be to write an actual GDAL driver for the
model files, but that would likely be a fair amount of work.

Another approach would be to intercept the WMS requests with a
script that would take care of extracting desired products in advance
before turning control over mapserv to satisfy the request.  You would
presumably need to extract some information from the url to (LAYERS,
timeslices, etc) to determine what to extract.  I would assume you would
want to cache extracted data so that subsequent requests against the 
same dataset would be fast. 

All in all though, I doubt there is a "canned" solution.  It seems you will
be doing some scripting at one level or another. 

Good luck,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent