Problem trying to serve large .tif dataset

[Brent Fraser]

Glen,

  As Frank has pointed out in his reply (and Ed has said for several years),
the answer is to pyramid your data.  This means constructing layers at
different resolutions, then setting MINSCALE MAXSCALE for each of the layers
in the map file.  The goal is to improve the performance of Mapserver by
limiting the number of files it has to open to render a requested image.

  For example, I've set up a site to render the NASA's Geocover global
Landsat mosaic.  To handle the range of requests from full resolution (14
meters per pixel) to full world-wide coverage (20km per pixel), I set up six
levels (and I may add one more) of the pyramid (view in a fixed-pitched font
such as Courier):

        Largest
Number of     Tile
Lvl:    Map Scale:      Resolution:     Tile Size:
Pixels:     Tiles         Size  Total Size:   Format:
----    ----------      -----------     ---------------------------     ----
----    ----------    ----- -----------   ------
6       1:    50k          14 meters    1:250k-ish (1/16 of 5x6 deg)    10k
x 10k   14064         26mb  = 338     gb  ECW
5            250k          50 meters    1:250k  1 deg H x  2 deg W       2k
x  2k   14921          3mb  =  38.4   gb  ECW
4          1m             200 meters    ~1:1m   5 deg H x  6 deg W       3k
x  2k    36x60 = 2160 20mb  =  26.1   gb  TIFF
3          5m            30 sec ( 1km)         15 deg H x 30 deg W       2k
x  3k    12x12 =  144 20mb  =   2.4   gb  TIFF
2         20m             2 min ( 4km)         45 deg H x 90 deg W       1k
x  2k     4x 4 =   16 10mb  =   0.167 gb  TIFF
1        100m            10 min (20km)        180 deg H x360 deg W       1k
x  2k       1  =    1  7mb  =   0.007 gb  TIFF

Disclaimer: The above is just an example of pyramiding a large raster
dataset (tuning by adding levels or adjusting scale will likely be
required).  And I've broken Ed's Rule #3 ("For best performance, don't
compress your data") since levels 5 and 6 are ECW, and Rule #4 ("Don't
re-project your data on-the-fly") since levels 4,5,6 are in UTM.

Brent Fraser
GeoAnalytic Inc.
Calgary, Alberta


> I am having difficulties with mapserver 4.6 generating output with a
> large .tif dataset.  I'm attempting to setup a large dataset to be served
> up via WMS.
>
> I have a .tif dataset of approximately 2000 .tif files totalling around
> 160k pixels by 375k pixels.  Each tif image is 5000 x 5000 pixels.
>
> Has anyone had any success dealing with datasets of this size?  If so, how
> have you implemented them?
>
> Any help/advise would be greatly appreciated.
>
> Thank you,
> Glen Thompson