[mapserver-users] Slow and degrading performance with radar (but not satellite) on mapserver-6.0.1-3_0.el6.x86_64
Daniel Morissette
dmorissette at mapgears.com
Thu Jul 23 14:24:16 PDT 2020
Actually, I think this specific error means that the map parameter
"MAP=wmst" is missing a ".map" extension in the mapfile name.
I'd suggest renaming the mapfile to wmst.map, and also passing the full
path in the MAP= param.
Daniel
On 2020-07-23 16:54, Stephen Woodbridge wrote:
> Paul,
>
> This sounds like you have error in your mapfile. you might try posting
> it to the list if you can't figure it out yourself.
>
> There also might be some mapfile tools or sites that can validate the
> mapfile, but I've never needed them so I don't have them bookmarked
> anywhere. Can anyone on the list make a suggestion to Paul?
>
> -Steve W
>
>
> On 7/23/2020 12:48 PM, English Paul wrote:
>>
>> Thanks again Steve – progress:
>>
>> $ /usr/libexec/mapserv -nh
>> QUERY_STRING="MAP=wmst?FORMAT=image%2Fpng&LAYERS=winter1km_5min&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&TIME=2020-07-23T16%3A40%3A00Z&SRS=EPSG%3A900913&BBOX=-9392582.0328932,4383204.9527744,-9236038.9989652,4539747.9867024&WIDTH=256&HEIGHT=256"
>> > junk.png
>>
>> Gives the following in junk.png:
>>
>> Content-type: text/html
>>
>> <HTML>
>>
>> <HEAD><TITLE>MapServer Message</TITLE></HEAD>
>>
>> <!-- MapServer version 6.0.1 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG
>> OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=CAIRO SUPPORTS=FREETYPE S
>>
>> UPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT
>> SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPP
>>
>> ORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS SUPPORTS=GEOS
>> INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE -->
>>
>> <BODY BGCOLOR="#FFFFFF">
>>
>> msLoadMap(): Regular expression error. MS_DEFAULT_MAPFILE_PATTERN
>> validation failed.
>>
>> msEvalRegex(): R
>>
>> *From: *mapserver-users <mapserver-users-bounces at lists.osgeo.org> on
>> behalf of Stephen Woodbridge <stephenwoodbridge37 at gmail.com>
>> *Date: *Wednesday, July 22, 2020 at 19:53
>> *To: *"mapserver-users at lists.osgeo.org" <mapserver-users at lists.osgeo.org>
>> *Subject: *Re: [mapserver-users] Slow and degrading performance with
>> radar (but not satellite) on mapserver-6.0.1-3_0.el6.x86_64
>>
>> you need to include in your QUERY_STRING=MAP=....&
>>
>> it is probably hard coded in the fcgi config.
>>
>> -Steve W
>>
>> On 7/22/2020 7:15 PM, English Paul wrote:
>>
>> So you have a complex historical "mess" and its not clear where
>> the performance issue is. So you need to divide the problem into
>> small problems that you can verify are or are not contributing. I
>> would start with something like this:
>>
>> Thank you – I would not have thought to try it outside of CGI
>> entirely!
>>
>> Take one slow image request and try that as cgi or cli and not
>> fcgi and turn on debugging.
>>
>> - copy and rename you mapfile so it doesn't mess with the
>> production requests
>>
>> Not a major worry in this case – I’m working in an entirely dev
>> environment. I did make backup copies though!
>>
>> - turn on the debugging and send it to stderr in the debug mapfile
>>
>> - you can manually run that image request from the commandline like:
>>
>> if mapserv is not in your path you might need to find it and your
>> the path to it below
>>
>> mapserv -nh QUERY_STRING="everything after the ? in the original
>> query" >junk.png 2>error.txt
>>
>> -nh suppresses headers from being output before the image data
>>
>> error.txt will be stderr output and should contain the debug messages
>>
>> I couldn’t quite get there – which this is what I ended up running:
>>
>> # /usr/libexec/mapserv -nh
>> QUERY_STRING="FORMAT=image%2Fpng&LAYERS=winter1km_5min&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&TIME=2020-07-22T22%3A30%3A00Z&SRS=EPSG%3A900913&BBOX=-11584184.507886,4070118.8849183,-11271098.44003,4383204.9527744&WIDTH=256&HEIGHT=256"
>> > junk.png 2>error.txt
>>
>> I first tried it with our current map file(s), and then with
>> MS_ERRORFILE set to stderr and DEBUG set to 5, in both cases, I
>> got an empty error.txt, and the following in junk.png:
>>
>> Content-type: text/html
>>
>> <HTML>
>>
>> <HEAD><TITLE>MapServer Message</TITLE></HEAD>
>>
>> <!-- MapServer version 6.0.1 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG
>> OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=CAIRO SUPPORTS=FREETYPE
>>
>> SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER
>> SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT
>> SUPPORTS=WCS_SERVER SU
>>
>> PPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS SUPPORTS=GEOS
>> INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE -->
>>
>> <BODY BGCOLOR="#FFFFFF">
>>
>> loadMap(): Web application error. CGI variable "map" is
>> not set.
>>
>> </BODY></HTML>
>>
>> I noticed that if I ran it with just -nh, I get the following:
>>
>> #/usr/libexec/mapserv -nh
>>
>> This script can only be used to decode form results and
>>
>> should be initiated as a CGI process via a httpd server.
>>
>> So maybe this older version just really doesn’t want to be run on
>> the CLI? Or maybe I’m messing up the QUERY_STRING, or need to set
>> the CGI variable to something as the junk.png output suggests?
>>
>> Thanks again and thanks SO MUCH for the super quick reply!!
>>
>> Paul
>>
>> On 7/22/2020 1:47 PM, English Paul wrote:
>>
>> Hi,
>>
>> Newbie here, just got a work assignment to look into some
>> performance issues with mapserver-6.0.1-3_0.el6.x86_64 –
>> specifically, at one time, it rendered radar images very fast,
>> then it degraded and seemed to consume a lot more CPU. So –
>> the EC2 instance side was upgraded significantly – and it
>> still performs badly and uses a lot of CPU. It also seems to
>> be getting slowly worse over time (days/months, not
>> seconds/minutes). I asked this on IRC, but it looks like email
>> might be a better route.
>>
>> During all of this, the same mapserver instance renders
>> satellite images quickly. These seem to be a similar, or in
>> some cases larger size png to start with, rendered onto the
>> same final map/size.
>>
>> So – the obvious answer is that it isn’t using the CPU to
>> render/re-render, but rather spending it on something else –
>> I/O most likely – eg: a network request, disk I/O, SQL query?
>>
>> The previous person working on it tried turning on debug at
>> various levels – but unfortunately that made it even slower,
>> making it tricky to answer “what is making it slow when debug
>> is turned off?”
>>
>> My first instinct was to try an strace and nothing was
>> obvious. Next up – a flame graph from strace, and/or trying
>> dtrace – but my understanding is that dtrace is a little weak
>> on RHEL/CentOS/Amazon Linux 6.0 **and** I’m not particularly
>> good at that. Also, we’re using fcgi, so attaching to the
>> correct process is a bit tricky.
>>
>> My next instinct was to look at release notes and see if the
>> current stable has anything fixes/improvements that directly
>> address this – there aren’t any that are obvious to my eyes,
>> but you developers have been busy! So many things! Including
>> some performance fixes and one “significant” performance fix.
>>
>> Current config file:
>>
>> AddHandler fcgid-script fcgi
>>
>> FcgidIPCDir /var/run/mod_fcgid
>>
>> FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm
>>
>> FcgidMaxProcesses 10
>>
>> FcgidMaxProcessesPerClass 10
>>
>> FcgidMaxRequestInMem 196608
>>
>> FcgidInitialEnv PROJ_LIB /usr/share/proj
>>
>> FcgidInitialEnv LD_LIBRARY_PATH
>> "/usr/local/lib:/usr/pgsql-9.1/lib"
>>
>> So – suggestions for my next move? I currently plan to take a
>> quick swing at building 7.6 for RHEL 6.0, knowing there might
>> be old libraries and whatnot that make that a non-starter. Of
>> course – we’ve got other infra running on this same instance,
>> so upgrading everything is a much bigger task.
>>
>> Strace flame graph?
>>
>> Stretch and try dtrace?
>>
>> A better way to use debug?
>>
>> Something else I’m missing – eg: differences between the
>> image types that make them perform so differently?
>>
>> Thanks,
>>
>> Paul
>>
>> _______________________________________________
>>
>> mapserver-users mailing list
>>
>> mapserver-users at lists.osgeo.org <mailto:mapserver-users at lists.osgeo.org>
>>
>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>>
>> _______________________________________________
>>
>> mapserver-users mailing list
>>
>> mapserver-users at lists.osgeo.org <mailto:mapserver-users at lists.osgeo.org>
>>
>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
--
Daniel Morissette
Mapgears Inc
T: +1 418-696-5056 #201
More information about the mapserver-users
mailing list