[mapserver-users] How best to serve very large (300GB) raster data?

[Frank Warmerdam]

John Callahan wrote:
> I've been using ESRI products for quite a while. To serve a very large 
> raster dataset I have (about 300 GB), or really any raster or vector, I 
> usually just load the data into ArcSDE installed on Oracle and server 
> through ArcIMS. However, I want to move away from ESRI/Oracle and into a 
> more open environment, with all components being open source if possible.
> 
> How best to serve a large raster dataset of about 300 GB? My data is in 
> 2010 tiles, each a 3-band TIF of about 135 MB.
...
> Ideally, a tile server would be nice but I don't want to create the 
> tiles through an ESRI product (like Arc2Earth or HTMLImageMapper.) I 
> really like the possibility of TileCache (http://www.tilecache.org/) 
> working with MapServer. The immediate question is what would be the 
> raster data source to MapServer? Should I create tiles inside each TIF 
> using gdal_translate and then a shapefile tileindex? Maybe go with 
> compressed JPGs instead of TIFs since they usually have much smaller 
> file sizes? I'm not sure where to start.

John,

My suggestion would be to start by building a shapefile tileindex for
your existing set of TIFs and setup a mapserver instance based on that.

Check the performance.

If you feel a need to squeeze more performance out of it consider:

  o building a .qix spatial index for the tileindex shapefile (likely
    not all that important for 2010 polygons)

  o Possibly reformat the TIFF files to internally tiled format
    (gdal_translate -co TILED=YES in.tif out.tif)

  o Add overviews to each of the TIFFs (gdaladdo target.tif 2 4 8)

Needless to say you should backup the original files before transforming
them or adding overviews.

You could also consider just mosaicing the whole dataset into a single
BigTIFF or .img file for simplier management.  This could likely be
accomplished with gdal_merge.py.

I would advise against converting to JPEG unless you are very pressed
for space.  There are some very bad performance scenarios with JPEG
files and JPEG is a lossy format which can non-trivially damage your
image quality.

If you are going to use TileCache, it likely isn't worth spending too
much time fine tuning performance beyond fixing the gross performance
issues.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org