Raster TIF format Help

[Ed McNierney]

Steve -

I'm not sure who that "al" guy is, but I'll put in my two cents <g>.

My primary data organizational philosophy is that the optimal MapServer
image request would result in reading exactly one input TIFF image file,
and using the entire contents of that image without resampling.

So - step 1 would be to ensure you have TIFF images, preferably
uncompressed, and preprocessed with precomputed overviews for all view
scale settings.  Doing that will get you most of the way there.

I do not think there is any significant difference between GDAL
in-the-file overviews created with gdaladdo or separate files created
using gdalwarp, with each site of files in a LAYER definition limited by
a MINSCALE/MAXSCALE setting and included in a GROUP.  Using the gdaladdo
approach produces files that are a little easier to handle (only one
LAYER definition, no MINSCALE/MAXSCALE fiddling), but it is somewhat
limiting if you have to cover a wide range of view scales.

By "somewhat limiting" I'm referring to the fact that if you support
view scales that are much lower resolution than the original input
files, you can end up (in a zoomed-out situation) needing to read MANY
input files in order to produce one MapServer image, thus drifting a
long way from the optimal arrangement I mentioned above.  This can be a
noticeable performance drag.  If you're using the "external overviews"
approach (via gdalwarp, not internal overviews) then you will have the
option of mosaicing the now-tiny files together to produce larger
images.  This limits your ability to do some shortcuts with TILEINDEX
creation, as you will now have a necessarily distinct set of images that
will need their own index.

I have not found there to be much benefit from internal TIFF tiling
(using TILED=YES in GDAL creation options), but it certainly can make
some difference if your file width is much larger than the size of the
output image.  That's because you will need to read less data from the
TIFF file in order to get the contents of the area you need.  However,
sequential data reads from a file are fast and are very likely to be
"free" in that your disk is going to read several sequential sectors all
at once whether you like it or not!  I don't think it hurts, so I'd go
ahead and adopt it as a general approach.

I believe, however, that internal tiling would be more valuable for
compressed TIFFs, especially LZW-compressed images, due to the need to
read from the beginning in order to do the decompression.  LZW
compression is of little benefit for most imagery and I don't recommend
it.  LZW compression can be appropriate for your topo map images if they
are relatively clean scans.  On the other hand, they tend not to be very
large data sets compared to the imagery so you may not be motivated to
compress them.

So - keep it simple.  Use uncompressed TIFFs precomputed at all
available output resolutions.  Use TILED=YES when creating TIFFs.  Use
gdaladdo internal overviews if the convenience is helpful, but don't
create lots of tiny TIFFs at zoomed-out scales; use gdalwarp and then
mosaic the images together.  Try to produce input images that are, in
general, no smaller than the output image sizes you support.

GDAL does not support as many resampling options as other applications
do, and I have preferred roll-my-own bicubic sampling for zoomed-out
overviews, so the "external overviews" approach can be preferable.  But
it does require more data and mapfile maintenance work.

I hope this helps; I'm being a bit telegraphic as I know you've handled
a few rasters before!  Let me know if anything is especially cryptic.

     - Ed

Ed McNierney
TopoZone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Stephen Woodbridge
Sent: Sunday, March 04, 2007 12:07 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Raster TIF format Help

Ed McNierney, et al,

What is the conventional knowledge on how to organize TIF images 
internally for optimal access by mapserver?

I remember that TIF files can have a lot of different internal 
organizations and that some are much more efficient for mapserver to 
access when serving data.

I am looking at putting together some topo map scans and Landsat imagery

of the middle east. The topo maps are roughly 4000x3000 pixels and in 
various scales like 500k, 200k, 100k, 50k, and the landsat imagery is 
looks to be in 1 deg squares.

Can GDAL be used to reorganize the images and if so what options should 
I specify? or are there other tools I need to use?

Thanks,
   -Steve