<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1023017832;
mso-list-type:hybrid;
mso-list-template-ids:1885142660 2066523732 135462915 135462917 135462913 135462915 135462917 135462913 135462915 135462917;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="NL-BE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We are trying to add mapcache to our stack, because in the last years the number of requests on our mapservers kept increasing, and we are looking for smarter ways to scale.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We have a test setup, and on low load everything works perfectly. An ab benchmark on a single requests has also no problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">However, when we try to replay some production data on the mapcache server, with 500 concurrent users, we see the following errors in apache error log:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:50:33.789479 2017] [core:notice] [pid 13:tid 140568766519168] AH00051: child pid 793 exit signal Segmentation fault (11), possible coredump in /etc/apache2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:50:34.791605 2017] [core:notice] [pid 13:tid 140568766519168] AH00051: child pid 1384 exit signal Segmentation fault (11), possible coredump in /etc/apache2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:50:35.792510 2017] [core:notice] [pid 13:tid 140568766519168] AH00051: child pid 1346 exit signal Segmentation fault (11), possible coredump in /etc/apache2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:50:38.795504 2017] [core:notice] [pid 13:tid 140568766519168] AH00051: child pid 1338 exit signal Segmentation fault (11), possible coredump in /etc/apache2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">And also:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:49.804880 2017] [:error] [pid 8256:tid 140568096990976] [client xxx.xxx.xxx.xxx:59612] deleting a possibly stale lock after waiting on it for 30.034 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:49.815800 2017] [:error] [pid 796:tid 140568273237760] [client xxx.xxx.xxx.xxx:59631] deleting a possibly stale lock after waiting on it for 30.037 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:50.052952 2017] [:error] [pid 9502:tid 140568449484544] [client xxx.xxx.xxx.xxx:59900] deleting a possibly stale lock after waiting on it for 30.064 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:50.062164 2017] [:error] [pid 1364:tid 140568264845056] [client xxx.xxx.xxx.xxx:59951] deleting a possibly stale lock after waiting on it for 30.04 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:54.226499 2017] [:error] [pid 796:tid 140568239666944] [client xxx.xxx.xxx.xxx:34636] deleting a possibly stale lock after waiting on it for 30.046 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:55.358419 2017] [:error] [pid 797:tid 140568399128320] [client xxx.xxx.xxx.xxx:35329] deleting a possibly stale lock after waiting on it for 30.047 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:54:55.949560 2017] [:error] [pid 794:tid 140568256452352] [client xxx.xxx.xxx.xxx:35639] deleting a possibly stale lock after waiting on it for 30.037 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Tue Feb 28 12:55:00.062597 2017] [:error] [pid 1408:tid 140568122169088] [client xxx.xxx.xxx.xxx:38816] deleting a possibly stale lock after waiting on it for 30.036 seconds<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In the syslog we see a lot of this:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314200.930086] traps: apache2[7324] general protection ip:7f308fc53fe9 sp:7f3089657b10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314204.940646] traps: apache2[7469] general protection ip:7f308fc53fe9 sp:7f3082ffcb10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314207.948949] traps: apache2[7626] general protection ip:7f308fc53fe9 sp:7f307c7efb10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314209.954537] traps: apache2[7748] general protection ip:7f308fc53fe9 sp:7f30837fdb10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314214.966671] traps: apache2[7954] general protection ip:7f308fc53fe9 sp:7f3088e56b10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314229.004451] traps: apache2[8562] general protection ip:7f308fc53fe9 sp:7f307c7efb10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314235.020377] traps: apache2[8798] general protection ip:7f308fc53fe9 sp:7f307f7f5b10 error:0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314235.020395] traps: apache2[8799] general protection ip:7f308fc53fe9 sp:7f307eff4b10 error:0 in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[51314235.020535] in libmapcache.so.1.4.1[7f308fc10000+56000]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We compile/run apache/mapcache in docker container (base image 14:04) (with –net host), compiling with this parameters:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># Install Mapcache itself<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ADD <a href="https://github.com/mapserver/mapcache/archive/rel-1-4-1.tar.gz">
https://github.com/mapserver/mapcache/archive/rel-1-4-1.tar.gz</a> /<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># Compile Mapcache for Apache<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">RUN mkdir -p /usr/local/src/mapcache && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tar xf rel-1-4-1.tar.gz -C /usr/local/src/mapcache --strip-components=1 && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> mkdir /usr/local/src/mapcache/build && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> cd /usr/local/src/mapcache/build && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> cmake ../ \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> -DWITH_FCGI=0 -DWITH_APACHE=1 -DWITH_PCRE=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> -DWITH_TIFF=0 -DWITH_BERKELEY_DB=0 -DWITH_MEMCACHE=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> -DWITH_SQLITE=0 -DCMAKE_PREFIX_PATH="/etc/apache2" && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> make && \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> make install<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">What we’ve tried:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Change mpm from event to worker, and play with the tuning options<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Change cache and/or locking directory from tmpfs to normal disks.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">Disable all unneeded options in compile<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Mapcache.xml:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><?xml version="1.0" encoding="UTF-8"?><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><!-- see the accompanying mapcache.xml.sample for a fully commented configuration file --><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><mapcache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <cache name="disk" type="disk"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <base>/tmp/mapcache/</base><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <symlink_blank/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </cache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <source name="LOS-all" type="wms"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <getmap><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <params><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <FORMAT>image/png</FORMAT><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <LAYERS>layer1,layer2,layer3</LAYERS><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <MAP>/maps/LOS.map</MAP><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </params><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </getmap><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <http><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <url><a href="http://localhost:81/ms%3c/url">http://localhost:81/ms</url</a>><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </http><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </source><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <tileset name="LOS-all"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <source>LOS-all</source><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <cache>disk</cache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <grid>WGS84</grid><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <grid>GoogleMapsCompatible</grid><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <format>PNG</format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <metatile>5 5</metatile><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <metabuffer>10</metabuffer><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <expires>60</expires><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <auto_expire>60</auto_expire><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </tileset><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <default_format>PNG</default_format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="wms" enabled="false"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <full_wms>assemble</full_wms><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <resample_mode>bilinear</resample_mode><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <format>JPEG</format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <maxsize>4096</maxsize><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </service><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="wmts" enabled="true"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="tms" enabled="true"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="kml" enabled="true"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="gmaps" enabled="true"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="ve" enabled="false"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="mapguide" enabled="false"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <service type="demo" enabled="false"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <errors>report</errors><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <locker type="disk"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <directory>/tmp</directory><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <timeout>30</timeout><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </locker><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"></mapcache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Andy idea what I can do to find exactly what is going wrong?
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks in advance!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Pieter Callewaert<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>