Help on raster optimization
Benoît Andrieu
bea at IXSEA.COM
Tue Oct 16 10:21:03 PDT 2007
Thanks for anyone's answer,
I totally agree with your answers concerning the fact that the less files
there is, the better the performance is.
So creating a new layer will help in the future.
Why we are not doing this at this time is that we have a product that is
generating tiff files in real time, those tiff files are quite small (300m *
50m) and after generation they are automatically referenced in a shape file.
So the more we are generating little images the more the perception of a
real time system should be present.
In a near future we will generate an overview layer in order to maximize the
performance but actually we only have one layer.
Actually I was trying to look at optimizing the performance but I think that
without this overview I won't be able to make significant improvements
without cutting down a lot of features in GDAL. So I won't look more at my
actual problem.
For information, from what I see in my logs, GDAL is spending quite as much
time at loading the GTiff than opening the csv files I mentioned.
Thanks again for your time.
Benoît
----- Original Message -----
From: "Ed McNierney" <ed at topozone.com>
To: "Benoit Andrieu" <bea at IXSEA.COM>; <MAPSERVER-USERS at LISTS.UMN.EDU>
Sent: Monday, October 15, 2007 10:38 PM
Subject: RE: [UMN_MAPSERVER-USERS] Help on raster optimization
Benoit -
Thanks for such a good description of your problem! I want to point out
that the chief slowdown you will see in a situation like this is the
increase in the number of GeoTIFF files being opened and read to serve one
request. Your internal overviews obviously get smaller and smaller at each
zoom level. If you have a 1024x1024 pixel image, the 32x overview is really
a 32x32 pixel little image. If your client requests a 1024x1024 output
image, you will need to open and read 1024 individual GeoTIFFs! Many users
overlook this aspect of overviews. At a certain point your response time is
dominated by the number of files you need to seek to and open.
While those other files may be annoying, I doubt they contribute much to the
problem. Yes, you can improve things by cleaning them up. But I think your
biggest win would be to combine your small-scale GeoTIFF overviews into
larger, mosaiced images, and use a GROUP layer with MINSCALE/MAXSCALE
settings to select the right LAYER.
The random seek time to a new file is the slowest thing your server does.
Minimizing the number of files used to serve a request is MUCH more valuable
than minimizing the size of those files. Reading 10 100K files is much
faster than reading 100 10K files.
If you are in the habit of serving data in the lat/lon format WorldWind
requires, you would do well to create a shadow copy of your images, all
reprojected to the desired output projection, and only advertise that one
projection in your service.
- Ed
Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA 01863
ed at topozone.com
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Benoit Andrieu
Sent: Monday, October 15, 2007 4:26 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Help on raster optimization
Hi all !
We do have a problem on using MapServer on raster datas.
Our datasets are made of multiple GeoTiff files. Each Tiff has his own
overviews
(2 4 8 16 32). We do not have an overview level.
Our Tiff are projected in UTM xx depending on the location on earth. Our
Tiff
have coordinates in them, they are true GeoTiff files ; however they have
tfw
files for compatibility with third-party softwares.
We have an high performance degradation when serving WMS clients as
WorldWind
and an home-made client ; more precisely, the degradation is happening on
zoomed
out requests.
The cause seems to be the amount of data required to be loaded by mapserver
for
the rendering of one single request.
Worldwind is requesting datas in latlong projection whereas our homemade
client
is requesting in UTM projection.
Our server is an octo-core with 8Gb of RAM, we are using Win2003 Server and
Mapserver comes from FWTools 1.3.2 (Mapserver 4.x then), it is hosted by an
IIS
6.0 in CGI mode (not FastCGI).
I set the GDAL_CACHEMAX to 200.
I monitored the mapserv.exe behaviour in the task manager to see what was
happening and I saw that mapserver was loading a huge amount of data
(between
50Mb and 300Mb of data depending on the requests/datasets).
I then monitored the mapserv.exe behaviour in the holy Procmon and there I
saw
some interesting things.
First, the epsg file loaded by GDAL is loaded multiple times near the start
of
the process, I think it is loaded each time an epsg code is mentionned in
the
map file.
As we are serving UTM, their epsg code (32631 for instance) are lying at the
end
of the file, so Mapserver was reading the file looking for those codes.
I, then, moved those definitions to the beginning of the file, just after
the
latlong definition (epsg 4326).
I launched the requests, and saw that the amount of datas being read was
much
lower, but the end-to-end was slow again.
Then, I moved forward, in my procmon report and there I saw IT !!
Mapserver is loading for each of our tiffs files, 6 other files !!
Those files are :
pcs.csv = 400 Kb,
projop_wparm.csv = 165 Kb,
gcs.csv = 35 Kb,
unit_of_measure.csv = 20 Kb,
datum.csv = 70 Kb,
ellipsoid.csv = 10 Kb
So, from what I see, for serving my request, mapserver is loading
approximatively 700Kb of "external" datas and around 50/100 Kb of my 1Mb
Tiff
files. In order to serve a zoomed out request, I have 26 Tiff files. So,
finally, mapserver is loading 20 Mo.
So, here is my question : how could I make Mapserv to not load those files
again
and again... ???
Thanks in advance !!
Benoît Andrieu
PS : if anyone is interested, I can send my log files from Procmon to show
which
files are accessed.
===============================================================================================================================================================================
My test requests :
http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:32631&width=512&height=512&bbox=715102,4783144.81842247,716996,4783869.89482688&format=image/gif&version=1.1.1&styles=
http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:4326&width=512&height=512&bbox=5.64654588699341,43.1704597473145,5.66959381103516,43.1764526367188&format=image/gif&version=1.1.1&styles=
===============================================================================================================================================================================
My map file :
#This Map Was Generated By Isdas do not modifie
#Generaded : 10/15/2007 1:24:22 PM
MAP
Name"LigneSudMoins600"
CONFIG "PROJ_LIB" "c:\Inetpub\wwwroot\Cgi\proj_lib\"
EXTENT 5.64654588701225 43.1704597472872 5.66959381105485 43.17645263669
SIZE 512 512
IMAGETYPE PNG24
IMAGECOLOR 255 255 255
IMAGECOLOR 255 255 255
UNITS METERS
OUTPUTFORMAT NAME PNG24 DRIVER "GD/PNG" MIMETYPE "image/png" EXTENSION PNG
IMAGEMODE RGBA TRANSPARENT ON END
PROJECTION "init=epsg:4326" END
WEB
IMAGEPATH "c:\ms_tmp\"
IMAGEURL "http://ACQUISITIONTEST/ms_tmp"
METADATA
"wms_title" "LigneSudMoins600" "wms_srs" "epsg:4326 epsg:32631"
"wms_onlineresource" "http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"
END
END
LAYER
NAME "Sonar"
STATUS ON
TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar" "wms_srs" "epsg:32631 epsg:27563
epsg:32631
epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_PORT"
STATUS ON
TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_PORT.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_PORT" "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_STARBOARD"
STATUS ON
TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_STARBOARD.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_STARBOARD" "wms_srs" "epsg:32631
epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
LAYER
NAME "Sonar_FRONT"
STATUS ON
TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_FRONT.shp"
TILEITEM "location"
TYPE RASTER
UNITS METERS
METADATA "wms_title" "Sonar_FRONT" "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123" END
PROJECTION "init=epsg:32631" END
OFFSITE 0 0 0
END
#~~~
END
More information about the MapServer-users
mailing list