<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.Shkpostityyli19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">You should test the same shapefiles locally for understanding where the time gets spent. Normal time with simple styles is perhaps up to 300 ms. The optimization hints were written long
before there were S3 or Amazon (the company).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Unfortunately I don’t know how to optimize access to S3.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> MapServer-users <mapserver-users-bounces@lists.osgeo.org>
<b>Puolesta </b>Matthew Graber via MapServer-users<br>
<b>Lähetetty:</b> perjantai 8. joulukuuta 2023 21.47<br>
<b>Vastaanottaja:</b> mapserver-users@lists.osgeo.org<br>
<b>Aihe:</b> [MapServer-users] Speeding up msDrawMap() for shapefiles<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm trying to speed up the time required for msDrawMap() to execute for rendering shapefiles. Currently, according to the mapserver logs, it's taking 2.5-3.5 seconds, sometimes up to 5 seconds for it to execute:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal">[Wed Dec 6 18:43:47 2023].699791 CGI Request 53 on process 2641<br>
[Wed Dec 6 18:43:47 2023].717216 msDrawMap(): rendering using outputformat named png (AGG/PNG).<br>
[Wed Dec 6 18:43:52 2023].299580 msDrawMap(): Layer 1010 (Shapefile_Layer), 4.582s<br>
[Wed Dec 6 18:43:52 2023].299792 msDrawMap(): Drawing Label Cache, 0.000s<br>
[Wed Dec 6 18:43:52 2023].299803 msDrawMap() total time: 4.584s<br>
[Wed Dec 6 18:43:52 2023].310853 msSaveImage(stdout) total time: 0.011s<o:p></o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal">[Wed Dec 6 18:43:52 2023].310883 mapserv request processing time (msLoadMap not incl.): 4.611s<o:p></o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The mapfile is currently set up using "CONNECTIONTYPE OGR" and "CONNECTION" to access the shapefiles from an S3 bucket as follows:<o:p></o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal">CONNECTIONTYPE OGR<br>
CONNECTION '/vsis3/s3_bucket_path/my_shapefile%.shp'<o:p></o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I noticed at <a href="https://mapserver.org/input/vector/shapefiles.html">
https://mapserver.org/input/vector/shapefiles.html</a> that it says "Shapefile access is built directly into MapServer. It is also available through OGR, but direct access without OGR is recommended and discussed here."<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">So I tried removing "CONNECTIONTYPE OGR" and changing "CONNECTION" to "DATA" as follows:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal"> DATA '/vsis3/s3_bucket_path/my_shapefile%.shp'<o:p></o:p></p>
</div>
<div style="margin-left:30.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I haven't been able to observe any definitive speed improvements by doing this yet, although I can't say for sure that there aren't any.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Based on <a href="https://mapserver.org/optimization/vector.html">
https://mapserver.org/optimization/vector.html</a>, I also tried using the shptree command to create a spatial index and the coshp command to sort the shapefile and index. Even after doing that, I still see load msDrawMap() times of up to 5 seconds.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Does anyone know of any other optimizations that could be made here? Or if not, would using shptree and coshp on the shapefile and then accessing it in the mapfile using "DATA" to take it from S3 be the optimal way to set things up?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Matt<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>