[Mapserver-users] Large Raster files

[Martin Weinelt]

On Monday 17 May 2004 15:46, Jan Hartmann wrote:
> IMD Listuser wrote:
> > I have several large (250MB) images that I would like to serve using
> > mapserver. They are currently Geotiffs, so I would prefer to use them in
> > the same format. Nonetheless I would be happy to hear about the best
> > strategies for serving imagery.
> >
> > In particular, what is the best method for splitting up such an image
> > into the smaller tiles that can be indexed using gdaltindex?
>
> I use the following rule of thumb: I partition large raster files into
> tiles of 2000*2000 and build an index on them with gdaltindex.
> Partitioning can be done with GRASS or gdal_translate -srcwin. I also
> make downsampled copies of the original raster, resampling it by factors
> of two, also with GRASS or with gdalwarp -tr. If he resulting image
> itself is larger than 2000*2000 I tile that too, and so on. A
> 40000*40000 image would result in 20*20=400 tiles; it would be resampled
> to 20000*20000 pixels (partitioned into 10*10=100 tiles in its turn),
> and so on. The 40000, 20000, 10000, 5000 etc tiled images can be shown
> at their appropriate scales by the MAXSCALE-MINSCALE keyword.
>
> I'm not sure how "optimal" this procedure is; it works nicely for the
> Amsterdam maps, but I'm sure it can be optimized. My impression it that
> it generates *much* more efficient web sites than advanced compressing
> techniques like ECW. The factoring by powers of two makes it very easy
> to script the procedure, so large numbers of big rasters can be
> processed without much programming. This way, the original eighteen
> Amsterdam maps (1.5G) resulted into about 10000 tiles, requiring about
> three times the original disk space.
>
> Perhaps people with really big databases could comment on this?


I am not Ed  ;-) but handling DEM-hillshades in a current project which can 
also get quite big. What I am doing reads almost exactly like the procedure 
decribed by Jan. I stay away from any on-the-fly projection and do as much 
'preprocessing' as can be done. I also change RGB to indexed colors, which 
gives almost the same impression IMHO.

Everything is done by GDAL and the utilities. I decided to have tiles about 
the size of the map, without knowing exactly why this would be better then 
2000px (Jan). 

I also think that compressing the rasters is even contraproductive, because 
'in memory' there is no compression and you add an extra job (decoding) to 
the application.

I keeping the shapefiles in different resolutions aswell, making my map file a 
pain (number of MAXSCALE-MINSCALE-layers for the same data). I will look 
deeper into the new DATAPATTERN and variable substitution mechanisms soon.

Cheers, Martin