mapserver initialization delays

Mon Jun 12 18:40:54 PDT 2006

Scott -

"I have looked at the mapserver demo_init.html file and have added code
to my servlet that periodically performs the following get
http://www.geonomy.com/cgi-bin/mapserv?map=/var/www/geonomy/geonomy.map&
layer=geonomy_features&program=/cgi-bin/mapserv&map_web_imagepath=/tmp&m
ap_web_imageurl=/tmp
request every N minutes (intervals of 1 minute - 10 minutes)... but,
this does not change the behavior. "

I'm not certain you've quite nailed down the problem correctly, at least
if I'm reading your comment above correctly <g>.  You seem to be saying
(a) performance is fine until there's a pause of "20 - 30 minutes"
between requests, and (b) forcing an automated request every N (1 < N <
10) minutes doesn't fix the problem.  Those both can't be true.

I tried your original URL but it doesn't seem to be working right now.
It would be helpful to understand just what conditions cause the
performance to degrade back to the original "slow" state.  Can you
clarify that as carefully as possible?  Thanks.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Scott Melby
Sent: Monday, June 12, 2006 4:27 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] mapserver initialization delays

Stephen -

Thanks for the response.  Although I agree that there is definitely
something that is getting "evicted" from memory, it does not appear to
me that the amount of pertinent data that is cached in memory is the
issue.  The reason I say this, is that I can perform hundreds of
queries, all with differing bounding boxes and differing dataFilter
parameters (no result set overlap), and only the first query after a
long period is slow.  So... unless the initial request is loading ALL
data into memory (regardless of the FILTER and BBOX params), it seems
that it would incur delays on some requests other than the original.  
Note: When I am using FILTER and BBOX params the problem is a bit less
severe (40+ secs for initial draw, 3+ secs for subsequent is pretty
typical).

Regardless of the cause of the delays, I really can't have my users
hitting a 40+ second delay for an interactive query on the site.  Are
there known strategies for dealing with large datasets?  The response
time I get for all requests other than the first are completely
reasonable.  So... my attempts thus far have been aimed at attempting to
keep things from getting into this state via background hits.  This,
however, has been unsuccessful for some unknown reason.  I can see that
the web server is getting my requests (1 per minute) and I can also see
that mapserver is returning a valid .PNG file for each request.

Thanks again
Scott

Stephen Woodbridge wrote:

> Scott,
>
> What you are likely seeing is the fact the the first long draw is that

> postgres is finding all you data and it is getting cached in memory.
> So the subsequent requests are all fetching the data from the cache. 
> If you wait more and more of the data gets bumped from the cache.
>
> There is very little mapserver specific startup costs. You might have 
> read about database connection costs and pooling, but that is not what

> you are seeing.
>
> -Steve W.
>
> Scott Melby wrote:
>
>> Hello -
>>
>> I am pretty new to MapServer, so I apologize if I am re-hashing old 
>> material.  However, I am having a significant problem and am unable 
>> to find a "reasonable" solution documented anywhere.  I am running 
>> mapserver 4.6, and am serving a single layer that contains about 
>> 350,000 points from a PostGIS database.  The layer has a GiST index 
>> on the geometry column.  The following request 
>> http://www.geonomy.com/cgi-bin/mapserv?map=/var/www/geonomy/geonomy.m
>> ap&layer=geonomy_features&mode=map&dataFilter=geometry%20is%20not%20n
>> ull will draw all points in my table into a 256x256 RGB PNG file.  
>> The issue I am having is that MapServer can take up to 3 minutes to 
>> draw the image the first time it is called(after any delay of 30+ 
>> minutes with no calls).  Once that initial draw completes, subsequent

>> hits draw in about 9 seconds.  In my research I have found a page 
>> that indicates that this is due to MapServer initialization.  But the

>> advice given there was to send a request that causes all map features

>> to be drawn after setting up the server.... which has not solved my 
>> problems.  The delay does not appear to be tied to a particular 
>> client session... because hitting that same URL from another browser 
>> draws quickly if done within a few minutes of the initial query.
>> However, if I wait long enough (seems to be about 20-30 minutes) the 
>> next request will once again take 2-3 minutes instead of 9 seconds.
>> Note: I don't believe that the issue is due to efficiencies of 
>> drawing the same image or accessing the same data repeatedly, because

>> I have found that altering the dataFilter parameter (which is passed 
>> to a FILTER on the layer in the map file) causes the image to change,

>> but does not significantly alter the performance (filters with less 
>> points draw more quickly, but still take a relatively long time for 
>> initial query if delay is long enough before it is called).  I have 
>> looked at the mapserver demo_init.html file and have added code to my

>> servlet that periodically performs the following get 
>> http://www.geonomy.com/cgi-bin/mapserv?map=/var/www/geonomy/geonomy.m
>> ap&layer=geonomy_features&program=/cgi-bin/mapserv&map_web_imagepath=
>> /tmp&map_web_imageurl=/tmp request every N minutes (intervals of 1 
>> minute - 10 minutes)... but, this does not change the behavior.
>> I have also tried performing the following get at 1 minute intervals 
>> to try to avoid problems that could be caused by inactivity (such as 
>> the GiST index having to be reloaded from disk, or mapserver dropping

>> its connection to PostGIS)..  In this case the GET request is being 
>> sent from a Java using a HttpURLConnection that is set to not use 
>> caches.  Note: I have also tried adding a BBOX to this request to 
>> make sure that the GiST index is being used.  However, this does not 
>> fix the problem either.
>>
>> http://www.geonomy.com/cgi-bin/mapserv?map=/var/www/geonomy/geonomy.m
>> ap&REQUEST=GetMap&SERVICE=WMS&VERSION=1.1.1&LAYERS=geonomy_features&F
>> ORMAT=PNG&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SRS=EPSG:4326&BBOX=-115,3
>> 0,-108,37&WIDTH=256&HEIGHT=256&reaspect=false&dataFilter=geometry%20i
>> s%20not%20null
>>
>>
>> Any help is greatly appreciated.
>>
>> Thanks in advance
>> Scott
>>