Help on raster optimization
Benoit Andrieu
bea at IXSEA.COM
Mon Oct 15 16:26:08 EDT 2007
Hi all !
We do have a problem on using MapServer on raster datas.
Our datasets are made of multiple GeoTiff files. Each Tiff has his own overviews
(2 4 8 16 32). We do not have an overview level.
Our Tiff are projected in UTM xx depending on the location on earth. Our Tiff
have coordinates in them, they are true GeoTiff files ; however they have tfw
files for compatibility with third-party softwares.
We have an high performance degradation when serving WMS clients as WorldWind
and an home-made client ; more precisely, the degradation is happening on zoomed
out requests.
The cause seems to be the amount of data required to be loaded by mapserver for
the rendering of one single request.
Worldwind is requesting datas in latlong projection whereas our homemade client
is requesting in UTM projection.
Our server is an octo-core with 8Gb of RAM, we are using Win2003 Server and
Mapserver comes from FWTools 1.3.2 (Mapserver 4.x then), it is hosted by an IIS
6.0 in CGI mode (not FastCGI).
I set the GDAL_CACHEMAX to 200.
I monitored the mapserv.exe behaviour in the task manager to see what was
happening and I saw that mapserver was loading a huge amount of data (between
50Mb and 300Mb of data depending on the requests/datasets).
I then monitored the mapserv.exe behaviour in the holy Procmon and there I saw
some interesting things.
First, the epsg file loaded by GDAL is loaded multiple times near the start of
the process, I think it is loaded each time an epsg code is mentionned in the
map file.
As we are serving UTM, their epsg code (32631 for instance) are lying at the end
of the file, so Mapserver was reading the file looking for those codes.
I, then, moved those definitions to the beginning of the file, just after the
latlong definition (epsg 4326).
I launched the requests, and saw that the amount of datas being read was much
lower, but the end-to-end was slow again.
Then, I moved forward, in my procmon report and there I saw IT !!
Mapserver is loading for each of our tiffs files, 6 other files !!
Those files are :
pcs.csv = 400 Kb,
projop_wparm.csv = 165 Kb,
gcs.csv = 35 Kb,
unit_of_measure.csv = 20 Kb,
datum.csv = 70 Kb,
ellipsoid.csv = 10 Kb
So, from what I see, for serving my request, mapserver is loading
approximatively 700Kb of "external" datas and around 50/100 Kb of my 1Mb Tiff
files. In order to serve a zoomed out request, I have 26 Tiff files. So,
finally, mapserver is loading 20 Mo.
So, here is my question : how could I make Mapserv to not load those files again
and again... ???
Thanks in advance !!
Benoît Andrieu
PS : if anyone is interested, I can send my log files from Procmon to show which
files are accessed.
================================================================================================================================================================================
My test requests :
http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:32631&width=512&height=512&bbox=715102,4783144.81842247,716996,4783869.89482688&format=image/gif&version=1.1.1&styles=
http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:4326&width=512&height=512&bbox=5.64654588699341,43.1704597473145,5.66959381103516,43.1764526367188&format=image/gif&version=1.1.1&styles=
================================================================================================================================================================================
My map file :
#This Map Was Generated By Isdas do not modifie
#Generaded : 10/15/2007 1:24:22 PM
MAP
Name"LigneSudMoins600"
CONFIG "PROJ_LIB" "c:\Inetpub\wwwroot\Cgi\proj_lib\"
EXTENT 5.64654588701225 43.1704597472872 5.66959381105485 43.17645263669
SIZE 512 512
IMAGETYPE PNG24
IMAGECOLOR 255 255 255
IMAGECOLOR 255 255 255
UNITS METERS
OUTPUTFORMAT NAME PNG24 DRIVER "GD/PNG" MIMETYPE "image/png" EXTENSION PNG
IMAGEMODE RGBA TRANSPARENT ON END
PROJECTION "init=epsg:4326" END
WEB
IMAGEPATH "c:\ms_tmp\"
IMAGEURL "http://ACQUISITIONTEST/ms_tmp"
METADATA
"wms_title" "LigneSudMoins600" "wms_srs" "epsg:4326 epsg:32631"
"wms_onlineresource" "http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"
END
END
LAYER
NAME "Sonar"
STATUS ON
TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar" "wms_srs" "epsg:32631 epsg:27563 epsg:32631
epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_PORT"
STATUS ON
TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_PORT.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_PORT" "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_STARBOARD"
STATUS ON
TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_STARBOARD.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_STARBOARD" "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_FRONT"
STATUS ON
TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_FRONT.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_FRONT" "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
#~~~
END
More information about the mapserver-users
mailing list