mapserver raster documentation

[Frank Warmerdam]

John Mitchell wrote:
> Frank,
> 
> The reason why I had:
> OUTPUTFORMAT
>    NAME png
>    DRIVER "GD/PNG"
>    MIMETYPE "image/png"
>    IMAGEMODE RGB
>    EXTENSION "png"
>   END
> listed was that for ECW files you had DRIVER "GDAL/JP2ECW" and I am 
> wondering if you need a special driver for Imagine format.

John,

The OUTPUTFORMAT declaration is just for enabling output formats
from MapServer.  I had the GDAL/JP2ECW option previously so that
MapServer could return ECW maps to the user.  You don't need anything
like this for input formats.  If your GDAL is built to support a
format then it will "just work".

> In the next few months I may have to ingest 2500 DVD's of GeoTiff 
> imagery data covering portions of the United States with each DVD 
> contain 1 image of about 1.1 GB in size.  In looking over the MapServer 
> Raster documentation I could do the following:
> 
> a.) Keep as GeoTiff format
>      1.) Convert all the images from whatever their current projection 
> is to WGS84 using GDALWARP.
>      2.) Build internal tiling to each GeoTiff image utilizing: 
> gdal_translate -co TILED=YES orig.tif tiled.tif
>      3.) Build internal overviews using: gdaladdo -r average *.tif 2 4 8 
> 16 32 64 128
>      4.) Build external tiling using: gdal_index doq_index.shp 
> doq/*.tif   This will link all the geotiff's in the doq folder to the 
> doq_index.shp, which then is referenced in tileindex within layer 
> instead of DATA.
>      5.) ?When using tile indexes to manage many raster files as a 
> single file, it is especially important to have a overview layer that 
> kicks in at high (should'nt this say low instead?) scales to avoid 
> having to open a large number of raster files.  Building external 
> overviews use either gdal_merge or gdalwarp (eg gdalwarp -rc -tr 250 250 
> doq/*.tif overview.tif)  The 250 within the gdalwarp statement means a 
> output resolution of 250 meters?
> 
> b.) Convert from GeoTiff to Imagine format
>      1.) Convert all the images from whatever their current projection 
> is to WGS84 using GDALWARP.
>      2.) Mosaic all the GeoTiff's and convert to Imagine format using: 
> gdalwarp -of HFA doq/*.tif tiled.img
>      3.) Build internal overviews using: gdaladdo -r average tiled.img 2 
> 4 8 16 32 64 128
> 
> Which method would you recommend a or b as far as performance?

Whew, that's quite a bit of data to process!

The problem with approach (a) is handling the "no data" areas in the
images after reprojecting them to WGS84.  This can be addressed by
marking "OFFSITE 0 0 0" in your mapfile I believe.

Given that caveat, approach (a) is a pretty direct and practical approach.

Approach (b) will also work, though step (b-1) actually duplicates the
reprojection that can be done in (b-2) so you should really just skip
step (b-1).  I'm a bit nervous about trying to produce a 2.5TB Imagine
file in one pass with gdalwarp.  It seems like this could take a long
time to run, and if anything goes wrong you will be left in an indeterminate
state.  Also, if you use this approach make sure you specify the warp option
"-wo SKIP_NOSOURCE=YES".

To make the process a little more managable, I would suggest using
process (b) but instead of trying to produce one 2.5TB Imagine file,
try producing things in big chunks.  Perhaps 4 degree by 4 degree chunks.
Each of these 50 or so chunks might be around 100GB - a much more
managable amount of data to process.  Then you could, if necessary,
split the processing over a few machines, and verify each chunk after
production.

The chunks would then go into a tileindex, but because there aren't too
many you might not even need an overview layer.

I would stress that this is a lot of data to process, and that you should
try out on the process on a limited region first to ensure the whole
process is working as you expect.

Best regards,
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org