Help on raster optimization

Mon Oct 15 13:38:07 PDT 2007

Benoit -

Thanks for such a good description of your problem!  I want to point out that the chief slowdown you will see in a situation like this is the increase in the number of GeoTIFF files being opened and read to serve one request.  Your internal overviews obviously get smaller and smaller at each zoom level.  If you have a 1024x1024 pixel image, the 32x overview is really a 32x32 pixel little image.  If your client requests a 1024x1024 output image, you will need to open and read 1024 individual GeoTIFFs!  Many users overlook this aspect of overviews.  At a certain point your response time is dominated by the number of files you need to seek to and open.

While those other files may be annoying, I doubt they contribute much to the problem.  Yes, you can improve things by cleaning them up.  But I think your biggest win would be to combine your small-scale GeoTIFF overviews into larger, mosaiced images, and use a GROUP layer with MINSCALE/MAXSCALE settings to select the right LAYER.

The random seek time to a new file is the slowest thing your server does.  Minimizing the number of files used to serve a request is MUCH more valuable than minimizing the size of those files.  Reading 10 100K files is much faster than reading 100 10K files.

If you are in the habit of serving data in the lat/lon format WorldWind requires, you would do well to create a shadow copy of your images, all reprojected to the desired output projection, and only advertise that one projection in your service.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
ed at topozone.com
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Benoit Andrieu
Sent: Monday, October 15, 2007 4:26 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Help on raster optimization

Hi all !

We do have a problem on using MapServer on raster datas.

Our datasets are made of multiple GeoTiff files. Each Tiff has his own overviews
(2 4 8 16 32). We do not have an overview level.
Our Tiff are projected in UTM xx depending on the location on earth. Our Tiff
have coordinates in them, they are true GeoTiff files ; however they have tfw
files for compatibility with third-party softwares.

We have an high performance degradation when serving WMS clients as WorldWind
and an home-made client ; more precisely, the degradation is happening on zoomed
out requests.
The cause seems to be the amount of data required to be loaded by mapserver for
the rendering of one single request.
Worldwind is requesting datas in latlong projection whereas our homemade client
is requesting in UTM projection.

Our server is an octo-core with 8Gb of RAM, we are using Win2003 Server and
Mapserver comes from FWTools 1.3.2 (Mapserver 4.x then), it is hosted by an IIS
6.0 in CGI mode (not FastCGI).
I set the GDAL_CACHEMAX to 200.
I monitored the mapserv.exe behaviour in the task manager to see what was
happening and I saw that mapserver was loading a huge amount of data (between
50Mb and 300Mb of data depending on the requests/datasets).
I then monitored the mapserv.exe behaviour in the holy Procmon and there I saw
some interesting things.

First, the epsg file loaded by GDAL is loaded multiple times near the start of
the process, I think it is loaded each time an epsg code is mentionned in the
map file.
As we are serving UTM, their epsg code (32631 for instance) are lying at the end
of the file, so Mapserver was reading the file looking for those codes.
I, then, moved those definitions to the beginning of the file, just after the
latlong definition (epsg 4326).
I launched the requests, and saw that the amount of datas being read was much
lower, but the end-to-end was slow again.

Then, I moved forward, in my procmon report and there I saw IT !!
Mapserver is loading for each of our tiffs files, 6 other files !!
Those files are :
pcs.csv = 400 Kb,
projop_wparm.csv = 165 Kb,
gcs.csv = 35 Kb,
unit_of_measure.csv = 20 Kb,
datum.csv = 70 Kb,
ellipsoid.csv = 10 Kb

So, from what I see, for serving my request, mapserver is loading
approximatively 700Kb of "external" datas and around 50/100 Kb of my 1Mb Tiff
files. In order to serve a zoomed out request, I have 26 Tiff files. So,
finally, mapserver is loading 20 Mo.

So, here is my question : how could I make Mapserv to not load those files again
and again... ???

Thanks in advance !!

Benoît Andrieu

PS : if anyone is interested, I can send my log files from Procmon to show which
files are accessed.

================================================================================================================================================================================
My test requests :

http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:32631&width=512&height=512&bbox=715102,4783144.81842247,716996,4783869.89482688&format=image/gif&version=1.1.1&styles=

http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:4326&width=512&height=512&bbox=5.64654588699341,43.1704597473145,5.66959381103516,43.1764526367188&format=image/gif&version=1.1.1&styles=

================================================================================================================================================================================
My map file :

#This Map Was Generated By Isdas do not modifie
#Generaded : 10/15/2007 1:24:22 PM
MAP
 Name"LigneSudMoins600"
 CONFIG "PROJ_LIB" "c:\Inetpub\wwwroot\Cgi\proj_lib\"
 EXTENT 5.64654588701225 43.1704597472872 5.66959381105485 43.17645263669      

 SIZE 512 512
 IMAGETYPE PNG24
 IMAGECOLOR 255 255 255
 IMAGECOLOR 255 255 255
 UNITS METERS
 OUTPUTFORMAT NAME PNG24 DRIVER "GD/PNG" MIMETYPE "image/png" EXTENSION PNG
IMAGEMODE RGBA  TRANSPARENT ON  END
 PROJECTION    "init=epsg:4326"   END
 WEB
  IMAGEPATH  "c:\ms_tmp\"
  IMAGEURL  "http://ACQUISITIONTEST/ms_tmp"
  METADATA
   "wms_title"  "LigneSudMoins600"   "wms_srs" "epsg:4326 epsg:32631"
   "wms_onlineresource" "http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"
  END
 END
 LAYER
  NAME "Sonar"
  STATUS ON
  TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar"  "wms_srs" "epsg:32631 epsg:27563 epsg:32631
epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_PORT"
  STATUS ON
  TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_PORT.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_PORT"  "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_STARBOARD"
  STATUS ON
  TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_STARBOARD.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_STARBOARD"  "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_FRONT"
  STATUS ON
  TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_FRONT.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_FRONT"  "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
#~~~
END 