Performance hints please

Thu Apr 6 06:17:35 EDT 2006

Hi,

A very short conclusion first: Put logging on and check the rendering times. When it comes to imagery, if speed is the only thing that matters the use tiled tiff files with overviews and possibly limit the scales to the one of the original imagery + multipied/divided by powers of two. 

These conclusions are based on a quick but rather controlled Geotiff - JPEG 2000 speed comparison I made today.  My hypothesis, often mentioned on mailing list, was that tiled Geotiff with overviews gives you the the best you can get with Mapserver.

Short report follows.

Materials:
- single Geotiff image (10000 by 10000 pixels) created with gdal_translate from FWTools 0.9.8 package with TILED=YES option, planned to be widened to tileindex of more than hundred images
- overviews created by using gdaladdo -r average 2 4 8 16
- Pentium Xeon 3 GHz, single processor
- image on external Lacie Biggest disk F800 in RAID 0 mode, unit connected by standard Firewire only (instead of  Firewire 800)
- MS4W version 1.4.1

Method:
I made the simplest possible mapfile and asked Mapserver to send image in "map" mode to browser while logging rendering times. Output was held in 600 by 600 pixels while I changed the extents of the input.

Results:
This is an example about what I got to the logfile:
msDrawRasterLayerLow(Geotiff): entering.\r
msDrawGDAL(): src=99,8701,1200,1200, dst=0,0,600,600\r
msDrawGDAL(): red,green,blue,alpha bands = 1,2,3,0\r
msDrawMap(): Layer 0 (Geotiff), 0.080s\r
msDrawMap(): Drawing Label Cache, 0.000s\r
msDrawMap() total time: 0.080s\r
msDrawRasterLayerLow(Geotiff): entering.\r
msDrawGDAL(): src=99,8701,1200,1200, dst=0,0,600,600\r
msDrawGDAL(): red,green,blue,alpha bands = 1,2,3,0\r
msDrawMap(): Layer 0 (Geotiff), 0.090s\r
msDrawMap(): Drawing Label Cache, 0.000s\r
msDrawMap() total time: 0.090s\r

So I were asking in this case an area of 1200 by 1200 pixels from somewhere in the middle of the original file to be squeezed to 600 by 600 pixels. I got every time two blocks of information into the log and without any better understandment I just counted the two total times together. I hope that is relevant way to get the total total time :-)

Table 1. Total times needed to generate 600 by 600 pixel output from variable sized source areas inside Geotiff image 

input area   "total time 1"   "total time 2"  "total total"
600x600      0.06                 0.06             0.12
1200x1200  0.08                  0.09             0.17
2400x2400  0.08                  0.07             0.15
900x900      0.25                  0.14             0.39
1500x1500  0.24                  0.16             0.40

Conclusion:
Mapserver performed in this test best with Geotiff files if the requested area in source image is equal to the dimensions of the output, or to those dimensions multiplied with 2 or 4.  I believe the case would be the same with factors of 8 and 16 because of those overviews were as well created for the image. If the area is not sized as "2 to power of something", the time needed to render the output is more than two times longer.

P.S.
I am not going to continue my tests with JPEG 2000 or enlarge the test from single image to tile indexes this time. It was immediately obvious that JPEG 2000 through the driver that MS4W is using is not competative. Renderign from my JPEG 2000 image takes about 10 times more time than from optimised Geotiff image. With a better driver I guess that this speed could be increased to about the same level as it is with ECW format. Now MS4W is at least 2-3 times faster with ECW than with JPEG 2000, but still Geotiff is 2-3 times faster that ECW. 

I just have to believe what the ol' men have said many times here: tiled tiff files with overviews give you the best you can get with Mapserver. I believe that this speed will be beaten in case of single image layer used as backdrop imagery once we have a geospatial JPIP server sending JPEG 2000 streams to the client. I have been awaiting for that for years. However, I presume that this in not the most common way to utilise Mapserver but it is usually used to combine several data sources on server side. In this case JPEG 2000, ECW and the like can only offer some 90% saving in disk space but you have to pay for that in efficiency.

Regards,

-Jukka Rahkonen-
JPEG 2000 fan

>Lähettäjä: UMN MapServer Users List puolesta: Eijnden, B. van den (Bart)
>Lähetetty: to 6.4.2006 11:08
>Re: [UMN_MAPSERVER-USERS] Performance hints please

>One option could be generating overviews for your tiff files using gdaladdo.

>See section 10 of the following HOWTO (Raster Display Performance Tips):

>http://mapserver.gis.umn.edu/docs/howto/raster_data

>Best regards,
>Bart

>>-----Oorspronkelijk bericht-----
>>Van: UMN MapServer Users List
>>Verzonden: donderdag 6 april 2006 10:04
>>Onderwerp: [UMN_MAPSERVER-USERS] Performance hints please

>>G'day all.

>>Does anybody have a howto on Mapserver performance enhancement?

>>My particular environment of interest is a relatively small set of
>>geometries in a PostGIS database and a collection of geotiff aerial
>>photography images onto which I overlay the geometries.

>>The geometries are fully indexed, relatively static and vacuum analyzed
>>every night.

>>One of the options that users have is to request a "movie" of up to 20
>>frames; each of which consists of a background raster plus up to about
>>five circular geometries, about five points and maybe ten other
>>lines/polygons.

>>The amount of geometry involved is so small, the bottleneck has to be
>>the raster processing. The images are 2m/pixel and are typically 20 -
>>150 Mb each. The frame extents include most if not all of an image so
>>tiling does not seem relevant.

>>I have specified 80% JPEG as the output to minimise final image size and
>>this seems successful (at least in terms of size and hence download
>>time)

>>I am looking for ways to reduce the time taken in the image generation
>>phase.

>>TIA,
>>Stephen Davies