Problem trying to serve large .tif dataset

[Ed McNierney]

Brent -

It's OK to break the rules - just as long as you know you're doing it
<g>.

Frank made another point which is also important, but less-often
discussed.  At a certain point pyramids/overviews begin to break down
because a large image requested at a small scale ends up reading a large
number of very small files.  If you're opening more than a few files to
serve an image request, things will slow down, and if you start reading
from hundreds of files you're in trouble.  The time to render the image
will be dominated by the raw file-to-file seek time of your disk
subsystem.  Since finding a new file and opening it is one of the
slowest things you can do in an application server, your performance
gets very bad very quickly.

At a certain point you need to take your tiled overview files and merge
them, creating a smaller number of larger files.  Very broadly speaking,
it's probably a good idea to strive for image files whose extents are in
the range of several times the size of the output images generated.  If
you're making 1,000x1,000 output images, and your files are 4,000x4,000
pixels, then (a) most image requests will be served by only opening one
file, (b) a reasonable chunk of any file opened is used, and (c) you'll
never need to open more than 4 files to render an image.  If your files
are shrunken overviews that are 100 pixels each, then your 1,000x1,000
image may require up to 121 files being opened, and will never require
fewer than 100!

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
ed at topozone.com
(978) 251-4242  

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Brent Fraser
Sent: Friday, October 14, 2005 10:50 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Problem trying to serve large .tif
dataset

Glen,

  As Frank has pointed out in his reply (and Ed has said for several
years), the answer is to pyramid your data.  This means constructing
layers at different resolutions, then setting MINSCALE MAXSCALE for each
of the layers in the map file.  The goal is to improve the performance
of Mapserver by limiting the number of files it has to open to render a
requested image.

  For example, I've set up a site to render the NASA's Geocover global
Landsat mosaic.  To handle the range of requests from full resolution
(14 meters per pixel) to full world-wide coverage (20km per pixel), I
set up six levels (and I may add one more) of the pyramid (view in a
fixed-pitched font such as Courier):

        Largest
Number of     Tile
Lvl:    Map Scale:      Resolution:     Tile Size:
Pixels:     Tiles         Size  Total Size:   Format:
----    ----------      -----------     ---------------------------
----
----    ----------    ----- -----------   ------
6       1:    50k          14 meters    1:250k-ish (1/16 of 5x6 deg)
10k
x 10k   14064         26mb  = 338     gb  ECW
5            250k          50 meters    1:250k  1 deg H x  2 deg W
2k
x  2k   14921          3mb  =  38.4   gb  ECW
4          1m             200 meters    ~1:1m   5 deg H x  6 deg W
3k
x  2k    36x60 = 2160 20mb  =  26.1   gb  TIFF
3          5m            30 sec ( 1km)         15 deg H x 30 deg W
2k
x  3k    12x12 =  144 20mb  =   2.4   gb  TIFF
2         20m             2 min ( 4km)         45 deg H x 90 deg W
1k
x  2k     4x 4 =   16 10mb  =   0.167 gb  TIFF
1        100m            10 min (20km)        180 deg H x360 deg W
1k
x  2k       1  =    1  7mb  =   0.007 gb  TIFF

Disclaimer: The above is just an example of pyramiding a large raster
dataset (tuning by adding levels or adjusting scale will likely be
required).  And I've broken Ed's Rule #3 ("For best performance, don't
compress your data") since levels 5 and 6 are ECW, and Rule #4 ("Don't
re-project your data on-the-fly") since levels 4,5,6 are in UTM.

Brent Fraser
GeoAnalytic Inc.
Calgary, Alberta


> I am having difficulties with mapserver 4.6 generating output with a 
> large .tif dataset.  I'm attempting to setup a large dataset to be 
> served up via WMS.
>
> I have a .tif dataset of approximately 2000 .tif files totalling 
> around 160k pixels by 375k pixels.  Each tif image is 5000 x 5000
pixels.
>
> Has anyone had any success dealing with datasets of this size?  If so,

> how have you implemented them?
>
> Any help/advise would be greatly appreciated.
>
> Thank you,
> Glen Thompson