Raster TIF format Help

[Stephen Woodbridge]

Hi Ed,

Thank you very much. I think I actually remembered most of this from my 
past raster work, I must have had a great teacher ;) It is really good 
to have a refresher to put it back into context again.

I have added your response to the "Raster Data Access" HowTo on the 
website at http://mapserver.gis.umn.edu/docs/howto/raster_data

I hope this will get integrated into the documentation at some point also.

Thanks again,
   -Steve

Ed McNierney wrote:
> Steve -
> 
> I'm not sure who that "al" guy is, but I'll put in my two cents <g>.
> 
> My primary data organizational philosophy is that the optimal MapServer
> image request would result in reading exactly one input TIFF image file,
> and using the entire contents of that image without resampling.
> 
> So - step 1 would be to ensure you have TIFF images, preferably
> uncompressed, and preprocessed with precomputed overviews for all view
> scale settings.  Doing that will get you most of the way there.
> 
> I do not think there is any significant difference between GDAL
> in-the-file overviews created with gdaladdo or separate files created
> using gdalwarp, with each site of files in a LAYER definition limited by
> a MINSCALE/MAXSCALE setting and included in a GROUP.  Using the gdaladdo
> approach produces files that are a little easier to handle (only one
> LAYER definition, no MINSCALE/MAXSCALE fiddling), but it is somewhat
> limiting if you have to cover a wide range of view scales.
> 
> By "somewhat limiting" I'm referring to the fact that if you support
> view scales that are much lower resolution than the original input
> files, you can end up (in a zoomed-out situation) needing to read MANY
> input files in order to produce one MapServer image, thus drifting a
> long way from the optimal arrangement I mentioned above.  This can be a
> noticeable performance drag.  If you're using the "external overviews"
> approach (via gdalwarp, not internal overviews) then you will have the
> option of mosaicing the now-tiny files together to produce larger
> images.  This limits your ability to do some shortcuts with TILEINDEX
> creation, as you will now have a necessarily distinct set of images that
> will need their own index.
> 
> I have not found there to be much benefit from internal TIFF tiling
> (using TILED=YES in GDAL creation options), but it certainly can make
> some difference if your file width is much larger than the size of the
> output image.  That's because you will need to read less data from the
> TIFF file in order to get the contents of the area you need.  However,
> sequential data reads from a file are fast and are very likely to be
> "free" in that your disk is going to read several sequential sectors all
> at once whether you like it or not!  I don't think it hurts, so I'd go
> ahead and adopt it as a general approach.
> 
> I believe, however, that internal tiling would be more valuable for
> compressed TIFFs, especially LZW-compressed images, due to the need to
> read from the beginning in order to do the decompression.  LZW
> compression is of little benefit for most imagery and I don't recommend
> it.  LZW compression can be appropriate for your topo map images if they
> are relatively clean scans.  On the other hand, they tend not to be very
> large data sets compared to the imagery so you may not be motivated to
> compress them.
> 
> So - keep it simple.  Use uncompressed TIFFs precomputed at all
> available output resolutions.  Use TILED=YES when creating TIFFs.  Use
> gdaladdo internal overviews if the convenience is helpful, but don't
> create lots of tiny TIFFs at zoomed-out scales; use gdalwarp and then
> mosaic the images together.  Try to produce input images that are, in
> general, no smaller than the output image sizes you support.
> 
> GDAL does not support as many resampling options as other applications
> do, and I have preferred roll-my-own bicubic sampling for zoomed-out
> overviews, so the "external overviews" approach can be preferable.  But
> it does require more data and mapfile maintenance work.
> 
> I hope this helps; I'm being a bit telegraphic as I know you've handled
> a few rasters before!  Let me know if anything is especially cryptic.
> 
>      - Ed
> 
> Ed McNierney
> TopoZone.com
> 
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
> Behalf Of Stephen Woodbridge
> Sent: Sunday, March 04, 2007 12:07 PM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: [UMN_MAPSERVER-USERS] Raster TIF format Help
> 
> Ed McNierney, et al,
> 
> What is the conventional knowledge on how to organize TIF images 
> internally for optimal access by mapserver?
> 
> I remember that TIF files can have a lot of different internal 
> organizations and that some are much more efficient for mapserver to 
> access when serving data.
> 
> I am looking at putting together some topo map scans and Landsat imagery
> 
> of the middle east. The topo maps are roughly 4000x3000 pixels and in 
> various scales like 500k, 200k, 100k, 50k, and the landsat imagery is 
> looks to be in 1 deg squares.
> 
> Can GDAL be used to reorganize the images and if so what options should 
> I specify? or are there other tools I need to use?
> 
> Thanks,
>    -Steve