Serving hillshaded maps

Wed Jul 6 05:29:43 PDT 2005

Hey Mike,
I've included a email that was in the user group about a month ago which was
very helpful, providing info on how to handle raster datasets in mapserver
(overviews, tileindex, etc..).
If you don't have GDAL go to:
http://fwtools.maptools.org/
and the how to for the utilities at:
http://www.remotesensing.org/gdal/gdal_utilities.html
Good luck
Trav

>From: Gambin Dejan <Dejan.Gambin at PULA.HR>
>Reply-To: Gambin Dejan <Dejan.Gambin at PULA.HR>
>To: MAPSERVER-USERS at LISTS.UMN.EDU
>Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
>dataset performance
>Date: Fri, 10 Jun 2005 08:33:20 +0200
>
>Frank,
>
>I have to say it is great to get such a detalied explanation of
>overviews/tiles/performance problems. I would personally very like to
>see some kind of HOWTO that would explain this in more details end
>technical/theoretical background. For example, how to create overviews
>that exactly match desired output resolution so no resampling occurs
>(with fixed set of zoom scales of course) and similar performance
>things.
>
>I would be happy enough if someone can point me to literature that
>explains all this stuff.
>
>regards
>
>dejan
>
>-----Original Message-----
>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
>Behalf Of Frank Warmerdam
>Sent: Thursday, June 09, 2005 9:53 PM
>To: MAPSERVER-USERS at LISTS.UMN.EDU
>Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
>dataset performance
>
>
>On 6/4/05, Dan Greve <grevedan at hotmail.com> wrote:
> > To everyone,
> >
> > Are GeoTiff overviews taken advantage of by Mapserver?  What's the
> > best way to handle large datasets (300 GB, 260,000 files in my case)
> > when you want the user to be able to view the whole dataset.
>
>
>Dan,
>
>I think that Mark had the right idea with creating new overview layers
>to kick in at various scales.   To show an overview of your whole region
>it would be disaster to have touch all 260,000 of your files.
>
>To answer one specific question, MapServer will take advantage of
>overviews
>built into GeoTIFF files (assuming GDAL is in use).
>
> > I have a lot of data, probably about 300GB spread among 260,000 tiles.
>
> > Let's just say the region is... Texas.  I want the user to be able to
> > see the data set at ANY zoom factor.  He'd start out looking at the
> > entire state of texas, and be able to zoom progressively into a city
> > block, and back out again.  If the data format could handle 300 GB in
> > a single file (I'm using GeoTIFF), theoretically the performance would
>
> > be better than if I created a TILEINDEX (shapefile) of the 260,000
> > tiles.  I've seen this in smaller datasets when requesting the entire
> > scene, even just a 13000x13000 dataset with just over 2500 tiles.
>
>If GeoTIFF supported very large files (there are plans for "BigTIFF
>support" one day) then I might encourage you to just create one huge
>internally
>tiled GeoTIFF file with lots and lots of overview levels.  You could use
>a
>format like Erdas Imagine that does support very large files and build
>one huge mosaic image, with lots of overviews.  It *ought* to work quite
>efficiently though there might be some efficiency hits with such a large
>dataset.  For instance, just processing the block pointers array might
>prove quite a bit of work.
>
>What Mark is suggesting is to:
>
>  o Create a tile index for all your files.  You will likely want a
>spatial
>     index built on this tileindex shapefile.
>
>  o Create a layer in your mapfile using this tileindex, perhaps named
>     "mosaic_fullres".
>
>  o I would suggest building internal overviews on all the individual
>     geotiff files as well.
>
>  o View the resulting layer in MapServer, starting near full resolution.
>     Zoom out till performance degrades unacceptably.  This will be a
>     new resolution at which you you need to build a new "overview
>layer".
>     This isn't an overview within the files in question, it is a whole
>new
>     layer in the map.  There are a variety of ways to build it.  I would
>likely
>     prepare a script to generate it with MapServer itself, by issuing a
>series
>     of scripted render requests at your new chosen overview resolution.
>
>   o If you produce this new layer as a set of tile files, you will also
>need
>     a tileindex for it.
>
>   o In the mapfile you will need to add this new set of tiles as a new
>layer.
>      You will want to use the MINSCALE and MAXSCALE options on this
>      layer and the full resolution layer to ensure that renders start to
>operate
>      from this layer instead of the full resolution data at a suitable
>resolution.
>
>      There is some mechanism (GROUP?  Using the same layer name?) to
>      ensure that this layer and the full res layer will be treated as a
>single
>      layer from a user-visible point of view.  I don't know this details
>of this
>     aspect.
>
>   o you can repeat the above overview layer steps to build additional
>     overview layers if needed till your full scene gives acceptable
>performance.
>
>OK, looking over my garbled explanation, I'm not sure I have helped at
>all. This fairly common situations screams out for some sort of utility
>to help build the overview map layers.  Or at least we should have a
>more detailed HOWTO for this process than I am in a position to prepare
>just now.
>
> >
> > When you say
> >
> > "I create a new tile grid shapefile using that map extent as the size
> > of one tile. I tile the entire map area (in my case the world). "
> >
> > Do you mean you just duplicate the entire dataset with larger tiles
> > when a TILEINDEX search would take longer?
>
>He means to duplicate the whole dataset, but at the much reduced
>resolution at which the render performance started to degrade.  If your
>original files were fairly large, and had internal overviews built, I
>believe your first overview map layer would likely be at something like
>1/128'th of the resolution of the original data. So the overview dataset
>would then be 1/16000th the size of the original data or so.
>
> > When you say
> >
> > "I create a new aggregate image layer using calls to the map server to
>
> > generate an image for each new tile."
> >
> > I have no idea what you meant by "aggregate image layer"
>
>He means a whole new map layer which is at a reduced resolution. It is
>an aggregate of a whole bunch of calls to mapserver to render tiles of
>the total region.  (hence the need for a new tile index).
>
> > Are you downsampling the image at all as you increase the tile sizes?
>
> > The raster howto on the UMN site has a snippet about Frank W. wanting
> > to implement using GeoTIFF overviews in the mapserver.  Does mapserver
>
> > currently take advantage of this? Could you elaborate on your pyramid
> > scheme?
>
>Yes, he means that it would be at a much reduced resolution.  The tile
>sizes in meters is much bigger, but the actual tile sizes in terms of
>pixels need not necessarily be much larger.
>
>Note that there are different types of tiling and overviews coming into
>play.  o Macro tiling: Each tile is a separate TIFF file, and a
>tileindex shapefile
>     is used to associate them to treat them as one layer in the .map
>file.
>  o Internal tiling: A given TIFF file can be internally organized into
>tiles
>     as opposed to strips (scanlines).  This gives
>
>  o  "map level overviews": using mutiple layers in a .map file with
>     MINSCALE/MAXSCALE to select which layer to render from.
>  o "internal overviews": individual TIFF files can have overviews built
>     in and GDAL will automatically take advantage of them if present.
>
>Best regards,
>--
>---------------------------------------+--------------------------------
>---------------------------------------+------
>I set the clouds in motion - turn up   | Frank Warmerdam,
>warmerdam at pobox.com
>light and sound - activate the windows | http://pobox.com/~warmerdam
>and watch the world go round - Rush    | Geospatial Programmer for Rent

----------------------------------------------------------------------------
----

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]On
Behalf Of Mike Davis
Sent: Tuesday, July 05, 2005 8:01 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Serving hillshaded maps

Hello List!

I am trying to decide wether to include a hillshade layer in our
Mapserver applications.  I have played around with using an ESRI grid
hillshade in the past, as with any large raster image, it can slow
things down quite a bit.  Additionally, hillshading may introduce some
problems when serving 8bit imagery.

I am wondering if anyone else out in list land has added a hillshade
layer to their maps.  What approach works best for you?

Thanks in advance

-Mike