Help on raster optimization

Tue Oct 16 13:21:03 EDT 2007

Thanks for anyone's answer,

I totally agree with your answers concerning the fact that the less files 
there is, the better the performance is.
So creating a new layer will help in the future.
Why we are not doing this at this time is that we have a product that is 
generating tiff files in real time, those tiff files are quite small (300m * 
50m) and after generation they are automatically referenced in a shape file.
So the more we are generating little images the more the perception of a 
real time system should be present.
In a near future we will generate an overview layer in order to maximize the 
performance but actually we only have one layer.
Actually I was trying to look at optimizing the performance but I think that 
without this overview I won't be able to make significant improvements 
without cutting down a lot of features in GDAL. So I won't look more at my 
actual problem.

For information, from what I see in my logs, GDAL is spending quite as much 
time at loading the GTiff than opening the csv files I mentioned.

Thanks again for your time.

Benoît

----- Original Message ----- 
From: "Ed McNierney" <ed at topozone.com>
To: "Benoit Andrieu" <bea at IXSEA.COM>; <MAPSERVER-USERS at LISTS.UMN.EDU>
Sent: Monday, October 15, 2007 10:38 PM
Subject: RE: [UMN_MAPSERVER-USERS] Help on raster optimization

Benoit -

Thanks for such a good description of your problem!  I want to point out 
that the chief slowdown you will see in a situation like this is the 
increase in the number of GeoTIFF files being opened and read to serve one 
request.  Your internal overviews obviously get smaller and smaller at each 
zoom level.  If you have a 1024x1024 pixel image, the 32x overview is really 
a 32x32 pixel little image.  If your client requests a 1024x1024 output 
image, you will need to open and read 1024 individual GeoTIFFs!  Many users 
overlook this aspect of overviews.  At a certain point your response time is 
dominated by the number of files you need to seek to and open.

While those other files may be annoying, I doubt they contribute much to the 
problem.  Yes, you can improve things by cleaning them up.  But I think your 
biggest win would be to combine your small-scale GeoTIFF overviews into 
larger, mosaiced images, and use a GROUP layer with MINSCALE/MAXSCALE 
settings to select the right LAYER.

The random seek time to a new file is the slowest thing your server does. 
Minimizing the number of files used to serve a request is MUCH more valuable 
than minimizing the size of those files.  Reading 10 100K files is much 
faster than reading 100 10K files.

If you are in the habit of serving data in the lat/lon format WorldWind 
requires, you would do well to create a shadow copy of your images, all 
reprojected to the desired output projection, and only advertise that one 
projection in your service.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
ed at topozone.com
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On 
Behalf Of Benoit Andrieu
Sent: Monday, October 15, 2007 4:26 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: [UMN_MAPSERVER-USERS] Help on raster optimization

Hi all !

We do have a problem on using MapServer on raster datas.

Our datasets are made of multiple GeoTiff files. Each Tiff has his own 
overviews
(2 4 8 16 32). We do not have an overview level.
Our Tiff are projected in UTM xx depending on the location on earth. Our 
Tiff
have coordinates in them, they are true GeoTiff files ; however they have 
tfw
files for compatibility with third-party softwares.

We have an high performance degradation when serving WMS clients as 
WorldWind
and an home-made client ; more precisely, the degradation is happening on 
zoomed
out requests.
The cause seems to be the amount of data required to be loaded by mapserver 
for
the rendering of one single request.
Worldwind is requesting datas in latlong projection whereas our homemade 
client
is requesting in UTM projection.

Our server is an octo-core with 8Gb of RAM, we are using Win2003 Server and
Mapserver comes from FWTools 1.3.2 (Mapserver 4.x then), it is hosted by an 
IIS
6.0 in CGI mode (not FastCGI).
I set the GDAL_CACHEMAX to 200.
I monitored the mapserv.exe behaviour in the task manager to see what was
happening and I saw that mapserver was loading a huge amount of data 
(between
50Mb and 300Mb of data depending on the requests/datasets).
I then monitored the mapserv.exe behaviour in the holy Procmon and there I 
saw
some interesting things.

First, the epsg file loaded by GDAL is loaded multiple times near the start 
of
the process, I think it is loaded each time an epsg code is mentionned in 
the
map file.
As we are serving UTM, their epsg code (32631 for instance) are lying at the 
end
of the file, so Mapserver was reading the file looking for those codes.
I, then, moved those definitions to the beginning of the file, just after 
the
latlong definition (epsg 4326).
I launched the requests, and saw that the amount of datas being read was 
much
lower, but the end-to-end was slow again.

Then, I moved forward, in my procmon report and there I saw IT !!
Mapserver is loading for each of our tiffs files, 6 other files !!
Those files are :
pcs.csv = 400 Kb,
projop_wparm.csv = 165 Kb,
gcs.csv = 35 Kb,
unit_of_measure.csv = 20 Kb,
datum.csv = 70 Kb,
ellipsoid.csv = 10 Kb

So, from what I see, for serving my request, mapserver is loading
approximatively 700Kb of "external" datas and around 50/100 Kb of my 1Mb 
Tiff
files. In order to serve a zoomed out request, I have 26 Tiff files. So,
finally, mapserver is loading 20 Mo.

So, here is my question : how could I make Mapserv to not load those files 
again
and again... ???

Thanks in advance !!

Benoît Andrieu

PS : if anyone is interested, I can send my log files from Procmon to show 
which
files are accessed.

 ===============================================================================================================================================================================
My test requests :

http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:32631&width=512&height=512&bbox=715102,4783144.81842247,716996,4783869.89482688&format=image/gif&amp;version=1.1.1&styles=

http://acquisitiontest/Cgi/bin/mapserv.exe?&request=GetMap&layers=Sonar_FRONT,Sonar_STARBOARD,Sonar_PORT&TRANSPARENT=TRUE&BGCOLOR=0xFFFFFF&map=C:\content\surveytemplate\LigneSudMoins600\MAP_62.map&srs=EPSG:4326&width=512&height=512&bbox=5.64654588699341,43.1704597473145,5.66959381103516,43.1764526367188&format=image/gif&version=1.1.1&styles=

 ===============================================================================================================================================================================
My map file :

#This Map Was Generated By Isdas do not modifie
#Generaded : 10/15/2007 1:24:22 PM
MAP
 Name"LigneSudMoins600"
 CONFIG "PROJ_LIB" "c:\Inetpub\wwwroot\Cgi\proj_lib\"
 EXTENT 5.64654588701225 43.1704597472872 5.66959381105485 43.17645263669

 SIZE 512 512
 IMAGETYPE PNG24
 IMAGECOLOR 255 255 255
 IMAGECOLOR 255 255 255
 UNITS METERS
 OUTPUTFORMAT NAME PNG24 DRIVER "GD/PNG" MIMETYPE "image/png" EXTENSION PNG
IMAGEMODE RGBA  TRANSPARENT ON  END
 PROJECTION    "init=epsg:4326"   END
 WEB
  IMAGEPATH  "c:\ms_tmp\"
  IMAGEURL  "http://ACQUISITIONTEST/ms_tmp"
  METADATA
   "wms_title"  "LigneSudMoins600"   "wms_srs" "epsg:4326 epsg:32631"
   "wms_onlineresource" "http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"
  END
 END
 LAYER
  NAME "Sonar"
  STATUS ON
  TILEINDEX "C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar"  "wms_srs" "epsg:32631 epsg:27563 
epsg:32631
epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_PORT"
  STATUS ON
  TILEINDEX 
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_PORT.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_PORT"  "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_STARBOARD"
  STATUS ON
  TILEINDEX
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_STARBOARD.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_STARBOARD"  "wms_srs" "epsg:32631 
epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
 LAYER
  NAME "Sonar_FRONT"
  STATUS ON
  TILEINDEX 
"C:\content\surveytemplate\LigneSudMoins600\IDX_PIC_SS_65_FRONT.shp"
  TILEITEM "location"
  TYPE RASTER
  UNITS METERS
  METADATA     "wms_title" "Sonar_FRONT"  "wms_srs" "epsg:32631 epsg:27563
epsg:32631 epsg:32632 epsg:32633 epsg:32635" "wms_onlineresource"
"http://ACQUISITIONTEST/WMS/Default.aspx?MapId=123"  END
  PROJECTION    "init=epsg:32631"    END
  OFFSITE 0 0 0
 END
#~~~
END