[mapserver-users] mapserver defunct

Wim Vanbelle wimvanbelle at gmail.com
Wed Jan 12 13:45:47 EST 2011


Thanks for your comments. I'll try switching to CGI.

My current config is setup like this:
MapServer version 5.6.5 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG OUTPUT=WBMP
OUTPUT=SWF OUTPUT=SVG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE
SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT
SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER
SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS SUPPORTS=GEOS
SUPPORTS=RGBA_PNG INPUT=EPPL7 INPUT=POSTGIS INPUT=OGR INPUT=GDAL
INPUT=SHAPEFILE

tmpfs is used over ramfs, because the latter cannot be exported using nfs.
Initially we started using postgres, but switched to shapefiles because that
would scale better with higher load. (no need to setup more databases)

We have a lot of ram available, so any tips on using that better are
appreciated. using iostat/vmstat i can see that disk was no issue though.

On 12 January 2011 16:52, Stephen Woodbridge <woodbri at swoodbridge.com>wrote:

> Have you built mapserver with threads? If so disable this and see if the
> problem goes away.
>
> HTTP 500 probably means that your process is getting a SIGV. Also unless
> you are using a database connection, using fastcgi is not going to give you
> a huge benefit because of the way mapserver works. And threads will probably
> not help either. OK, I admit I am somewhat biased against threads :) Tuning
> for performance in my experience is:
> 1. start by keep it simple
> 2. measure where your bottlenecks are
> 3. fix these bottlenecks
> 4. repeat
>
> Don't throw everything at it because you think think it will help. It just
> makes a complicated mess and you have no way to know if any given pieces is
> helping or hurting.
>
> Using a tmpfs is good if you manage it, I prefer a ramfs, the difference
> being the a ramfs is fixed size and you have to explicitly load it, but you
> have more control over size and behavior for a little more planning and it
> does not have down side of being able to eat all your memory or forcing
> linux to do more swapping which can slow down things.
>
> Also if you need to squeeze an additional 1-1.5% performance and can live
> without swap, disable your swap partition. Linux is a swappy OS and swaps to
> disk before it needs to in anticipation, this is costly if you have a lot of
> memory and do not NEED to swap.
>
> -Steve W
>
>
> On 1/12/2011 7:23 AM, Wim Vanbelle wrote:
>
>> The problem definately seems related to concurrency.
>>
>> Load benching using 1 concurrent requests: 0 fails.
>> Load benching using 2 concurrent requests: +- 30% fails
>> more concurrent = more fails.
>>
>> The failed seem to return with HTTP 500.
>>
>> Running a shp2img during this time, still returns a good image.
>>
>> I'll post more info about this topic and a resolution, if i find one :).
>> Do note that we're dealing with shapefiles with 3 000 000 linestrings.
>> While they consist only of 2 coordinates (begin end obviously), it's
>> still a big thing to load.
>>
>>
>> On 12 January 2011 06:18, Stephen Woodbridge <woodbri at swoodbridge.com
>> <mailto:woodbri at swoodbridge.com>> wrote:
>>
>>    Wim,
>>
>>    I'm not sure I have any magic bullet for this problem, because I'm
>>    not sure what is causing it. If this was my project, I would tackle
>>    it like this:
>>
>>    1. alert people we have a problem that might impact the release date
>>    if you have not already done that
>>    2. divide the problem to try and localize what is causing it
>>
>>    a. do the tiles that fail, always fail? this points at a potential
>>    mapserver related issue as opposed to a load issue
>>    a.1 have you turned on mapfile DEBUG output? can you identify the layer
>>    a.2 you can get some maybe useful output from:
>>
>>    strace /path/to/mapserv QUERY_STRING='<url arguments>'
>>
>>    gdb /path/to/mapserv
>>    run QUERY_STRING='<url arguments>'
>>    where
>>    quit
>>
>>    valgrind /path/to/mapserv QUERY_STRING='<url arguments>'
>>
>>    a.3 you can run it with shp2img ...
>>
>>    b. is it fastcgi, do you see the same problem if you build it as cgi
>>    c. is it lighttpd, do you see the same problem if you run it with
>>    apache fastcgi? what about apache cgi?
>>
>>    Intermittent problems are the hardest to diagnose because they often
>>    are not reproducable in a debuging environment. We might be able to
>>    help you with more information.
>>
>>    I have done a lot of load testing on various versions of mapserver
>>    and tiled all of US and Canada about 4-5 different times using
>>    mapserver. I have not used FastCGI/Lighty and lighttpd, so I'm not
>>    sure about what influence that might have on the problem.
>>
>>    Hope this gives you some ideas.
>>
>>    Best regards,
>>      -Steve W
>>    http://imaptools.com/
>>
>>
>>    On 1/11/2011 3:15 PM, Wim Vanbelle wrote:
>>
>>        Well, the problem is that the requests are not always served.
>>
>>        During load testing, say about 10 concurrent requests I also
>>        load the
>>        map. But there are randomly tiles that are never loaded. I
>>        thought this
>>        would be related to the defunct processes, but that is not a
>>        certainty.
>>
>>        My setup is lighttpd + fastcgi.
>>        To be honest I'm in a pretty bad situation now, considering we
>>        go live
>>        with this in 9 days.
>>
>>        Not sure what else I can do honestly.
>>
>>
>>        On 11 January 2011 19:05, Stephen Woodbridge
>>        <woodbri at swoodbridge.com <mailto:woodbri at swoodbridge.com>
>>        <mailto:woodbri at swoodbridge.com
>>
>>        <mailto:woodbri at swoodbridge.com>>> wrote:
>>
>>            Here is an explanation of defunct processes:
>>
>>        http://www.webmasterworld.com/forum40/1032.htm
>>
>>            I'm sure there are others. So since you are running fastcgi,
>>        it is
>>            the responsibility of the fastcgi parent process to clean up
>>        dead
>>            child processes. It is likely that it is busy because you
>>        are doing
>>            load testing or there is some subtle issue that is
>>        occurring. If you
>>            are getting good responses from mapserver, you probably do
>>        not need
>>            to worry about it unless you are getting tons of them.
>>
>>            If you are using apache, I believe there is a parameter that
>>        you can
>>            set for fastcgi processes which will let them die after N
>>        requests.
>>            This should clean those up and apache will spawn a new
>>        process if
>>            needed to replace the that died.
>>
>>            -Steve W
>>
>>
>>            On 1/11/2011 10:04 AM, Wim Vanbelle wrote:
>>
>>                Hi all,
>>
>>                I'm getting a lot of
>>
>>                27098 x  20   0     0    0    0 Z    1  0.0   0:00.03
>>        mapserv
>>        <defunct>
>>                27104 x  20   0     0    0    0 Z    1  0.0   0:00.03
>>        mapserv
>>        <defunct>
>>
>>                while load testing mapserver. Is there any way I can go
>>        about
>>                checking
>>                out why this is happening? Especially when doing
>> concurrency
>>                tests it
>>                seems to go sideways.
>>
>>                Currently i'm using :
>>
>>                MapServer version 5.6.5 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG
>>                OUTPUT=WBMP
>>                OUTPUT=SWF OUTPUT=SVG SUPPORTS=PROJ SUPPORTS=AGG
>>        SUPPORTS=FREETYPE
>>                SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER
>>                SUPPORTS=WMS_CLIENT
>>                SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER
>>                SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS
>>        SUPPORTS=GEOS
>>                SUPPORTS=RGBA_PNG INPUT=EPPL7 INPUT=POSTGIS INPUT=OGR
>>        INPUT=GDAL
>>                INPUT=SHAPEFILE
>>
>>                using FastCGI/Lighty.
>>
>>                Do note that the mapfile itself seems to be fine, since
>>        it does
>>                render
>>                content when i'm the single user using it.
>>
>>                Thanks for your expert insight :D.
>>
>>
>>
>>                _______________________________________________
>>                mapserver-users mailing list
>>        mapserver-users at lists.osgeo.org
>>        <mailto:mapserver-users at lists.osgeo.org>
>>        <mailto:mapserver-users at lists.osgeo.org
>>        <mailto:mapserver-users at lists.osgeo.org>>
>>
>>        http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>>            _______________________________________________
>>            mapserver-users mailing list
>>        mapserver-users at lists.osgeo.org
>>        <mailto:mapserver-users at lists.osgeo.org>
>>        <mailto:mapserver-users at lists.osgeo.org
>>        <mailto:mapserver-users at lists.osgeo.org>>
>>
>>        http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20110112/2718dc39/attachment-0001.html


More information about the mapserver-users mailing list