[mapserver-users] Ed's Rules for the Best Raster Performance

[Paul Spencer]

Jim, you would think that ;)  However, in practice I wouldn't expect  
the disk access time for geotiffs to be significantly different from  
jpeg if you have properly optimized your geotiffs using gdal_translate  
-co "TILED=YES" - the internal structure is efficiently indexed so  
that gdal only has to read the minimum number of 256x256 blocks to  
cover the requested extent.  And using gdaladdo to generate overviews  
just makes it that much more efficient.

Even if you are reading less physical data from the disk to get the  
equivalent coverage from jpeg, the decompression overhead is enough to  
negate the difference in IO time based on Ed's oft quoted advice (and  
other's experience too I think).  The rules that apply in this case  
seem to be 'tile your data', 'do not compress it' and 'buy the fastest  
disk you can afford'.

Compression is useful and probably necessary if you hit disk space  
limits.

Cheers

Paul

On 15-Sep-08, at 5:48 PM, Jim Klassen wrote:

> Just out of curiosity, has anyone tested the performance of Jpegs  
> vs. GeoTiffs?
>
> I would expect at some point the additional disk access time  
> required for GeoTiffs (of the same pixel count) as Jpegs would  
> outweigh the additional processor time required to decompress the  
> Jpegs. (Also the number of Jpegs that can fit in disk cache is  
> greater than for similar GeoTiffs.)
>
> For reference we use 1000px by 1000px Jpeg tiles (with world files).  
> We store multiple resolutions of the dataset, each in its own  
> directory. We start at the native dataset resolution, and half that  
> for each step, stopping when there are less than 10 tiles produced  
> at that particular resolution. (I.e for one of our county wide  
> datasets 6in/px, 1ft/px, 2ft/px, ... 32ft/px). A tileindex is then  
> created for each resolution (using gdaltindex followed by shptree)  
> and a layer is created in the mapfile for each tileindex and  
> appropriate min/maxscales are set. The outputformat in the mapfile  
> is set to jpeg.
>
> Our typical tile size is 200KB. There are about 20k tiles in the 6in/ 
> px dataset, 80k tiles in the 3in/px dataset (actually 4in data, but  
> stored in 3in so it fits with the rest of the datasets well). I have  
> tested and this large number of files in a directory doesn't seem to  
> effect performance on our system.
>
> Average access time for a 500x500px request to mapserver is 300ms  
> measured at the client using perl/LWP and about 220ms with shp2img.
>
> Machine is mapserver 5.2.0/x86-64/2.8GHz Xeon/Linux 2.6.16/ext3  
> filesystem.
>
> Jim Klassen
> City of Saint Paul
>
>>>> "Fawcett, David" <David.Fawcett at state.mn.us> 09/15/08 1:10 PM >>>
> Better yet,
>
> Add your comments to:
>
> http://mapserver.gis.umn.edu/docs/howto/optimizeraster
>
> and
>
> http://mapserver.gis.umn.edu/docs/howto/optimizevector
>
> I had always thought that all we needed to do to make these pages  
> great
> was to grok the list for all of Ed's posts...
>
> David.
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org
> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Brent
> Fraser
> Sent: Monday, September 15, 2008 12:55 PM
> To: mapserver-users at lists.osgeo.org
> Subject: [mapserver-users] Ed's Rules for the Best Raster Performance
>
>
> In honor of Ed's imminent retirement from the Mapserver Support Group,
> I've put together "Ed's List for the Best Raster Performance":
>
>
> #1. Pyramid the data
>        - use MAXSCALE and MINSCALE in the LAYER object.
>
> #2. Tile the data (and merge your upper levels of the pyramid for  
> fewer
> files).
>        - see the TILEINDEX object
>
> #3. Don't compress your data
>        - avoid jpg, ecw, and mrsid formats.
>
> #4. Don't re-project your data on-the-fly.
>
> #5. Get the fastest disks you can afford.
>
>
> (Ed, feel free to edit...)
>
> Brent Fraser
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users


__________________________________________

    Paul Spencer
    Chief Technology Officer
    DM Solutions Group Inc
    http://www.dmsolutions.ca/