<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Have you tried to find an explanation for the difference between Spatialite and geopackage? They are both SQLite databases and there is not much difference in how they store the geometries
into BLOBs. Does direct SQL “select * from table” give similar results?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I have not experienced such difference with Mapserver. In some not so scientific tests with 1.2 million polygons I found that shapefile was the fastest and Spatialite/Geopackage were
slightly slower, but still faster than PostGIS with plain small bbox queries. The advantage of shapefile was lost when attribute filters were needed for more than one attribute. Comparison was made between a sorted shapefile and database tables with appropriate
indexes. The BBOX queries returned typically about 10-50 polygons with simple geometries – agricultural parcels.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">Lähettäjä:</span></b><span lang="EN-US"> mapserver-users <mapserver-users-bounces@lists.osgeo.org>
<b>Puolesta </b>Andreas Neumann<br>
<b>Lähetetty:</b> tiistai 27. huhtikuuta 2021 18.40<br>
<b>Vastaanottaja:</b> Wouter Visscher <wouter.visscher@gmail.com><br>
<b>Kopio:</b> mapserver-users <mapserver-users@lists.osgeo.org><br>
<b>Aihe:</b> Re: [mapserver-users] Mapserver installation in cloud environments (kubernetes)<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi all,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks for your reactions and replies to my request if there is experience in running UMN MapServer in the cloud. Sorry about the delay in responding.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I had a look at the MapServerless AWS Lambda Layer project and corresponding video. Thanks for sharing this work!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Pre-rendered tiles are of course good for background maps, but for other layers, esp. frequently updated layers and layers where we want to allow custom styling, feature info, etc. map tiles are not good alternatives.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Interesting to me was the feedback from Wouter about the Dutch NGDI (PDOK), which seems to be more similar to our GDIs (province level and national level OGC services in Switzerland). Interesting to hear that you are
copying data as close as possible to the pods (Geopackage or by bringing Postgis really close to the Pods).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">In the case of QGIS Server it is also interesting to see that the different vector data seem to matter substantially. We have a QGIS Server performance suite published at
</span><a href="http://test.qgis.org/perf_test/graffiti/"><span lang="EN-US">http://test.qgis.org/perf_test/graffiti/</span></a><span lang="EN-US"> --> scroll to the latest date. In the performance test suite there is a section comparing different vector data
sources. See f.e. </span><a href="http://test.qgis.org/perf_test/graffiti/2021_04_27_01_00/report.html#c395b4ae16b447f1b3e6fc127a4531da"><span lang="EN-US">http://test.qgis.org/perf_test/graffiti/2021_04_27_01_00/report.html#c395b4ae16b447f1b3e6fc127a4531da</span></a><span lang="EN-US">
where Postgis and SpatiaLite data sources are substantially faster than Shapefile and Geopackage is clearly the looser. Not sure if the same applies for UMN MapServer? Maybe that is a pattern that applies to QGIS Server only ...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Wouter: I would be interested in learning more about the "ogc-webservice-proxy" you mention for determining the size/duration/location. Do you have more to share regarding this proxy service? What is the purpose of it
and how does it work? Do you also separate short-running requests (e.g. GetFeatureInfo or GetLegendGraphics) vs long-running requests (GetMap, printing services, report generation)?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Is your "traefik" load balancer using "round robin" load balancing or something more advanced based on system load or even by using an estimate/prediction how long a request might take based on the parameters submitted?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks all for the interesting discussion!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, 23 Apr 2021 at 22:07, Wouter Visscher <</span><a href="mailto:wouter.visscher@gmail.com"><span lang="EN-US">wouter.visscher@gmail.com</span></a><span lang="EN-US">> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Andreas,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">To answer your question: "Do you know of any work ...."<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">For the Dutch National Geodata Infrastructure (PDOK) we are running now (over a year) hundreds of OGC WMS/WMTS/WFS servers (primarily Mapserver and Mapproxy) on aks (Azure kubernetes).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">The challenges you are describing sound very similar to ours :)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">To give some small inside on some of our solutions regarding:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">deployment: kustomize and operators<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">cloud optimization: (regarding data) geohashes over GPKG and PostGIS tables and getting the data as close to the pods as possible.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">load balancing: (infrastructure wise) 'standaard' loadbalancing with traefik, (OGC wise) we are working on a 'ogc-webservice-proxy' for determining the size/duration/location, basically what is going to be the impact
of a request.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">resource sharing: (infrastructure) ScaleSets for nodes, and ReplicaSets based on load, (data) because we have most of the data in GPKG we 'copy' the data around. So for a service that scales up to 4 pods we have 4 of
the same GPKG copied to that node.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I will share your knowledge with my colleagues, but it seems you reached a lot of the same conclusions (so far I have read) we have. <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, Apr 23, 2021 at 5:20 PM Andreas Neumann <</span><a href="mailto:andreas@qgis.org" target="_blank"><span lang="EN-US">andreas@qgis.org</span></a><span lang="EN-US">> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">For a small project as part of the Swiss National Geodata Infrastructure (grant project) several people worked on a study document called "Cloud-optimized OGC WMS Server" where we analyzed problems that can arise when
you install an OGC web server in the cloud (e.g. docker image deployed via Kubernetes, OpenShift or the likes). This work had a focus on QGIS Server with it's own set of problems - but some of the issues studied in this document also matter for other OGC WMS
servers, such as UMN Mapserver or Geoserver, such as the load balancing problem, how to share resources, etc.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Here is the link to the document (not in final form yet, but close to being final):
</span><a href="https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2" target="_blank"><span lang="EN-US">https://docs.google.com/document/d/1cOUWgzalRx7CHWTFgHz6-uyScsCcoaEmYC0VBHdZShQ/edit#heading=h.c7gq4lie7ys2</span></a><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I wonder if any similar work has been done specifically around problems, challenges and solutions when you deploy UMN Mapserver in cloud environments? Do you know of any work?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">One major problem that probably all installations of an OGC WMS server have is how to deploy a more intelligent load balancing system? Often, the default load balancer is some kind of round robin load balancer system,
but often this leads to inferior results where "cheap and short" requests (such as a simple GetFeatureInfo or GetLegendGraphics request) can be queued behind a long-running GetMap request (potentially with many layers, many features and a high-dpi, such as
600dpi, where the request can take several seconds to process.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">In our production system we are currently separating the requests to dedicated instances for short requests and potentially long requests, to avoid the above mentioned scenario, but we are not so satisfied with the solution,
as it is a bit inflexible and also a bit harder to maintain. Ideally, we would like to have a more intelligent load balancer with incoming queue that holds back requests as long as all WMS server instances are busy. This would avoid the situation where a
"less intelligent" load balancer would simply forward the requests to instances based on Round-Robin principle.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Do you know of any work in the UMN Mapserver community regarding cloud deployment, cloud optimization, load balancing and resource sharing?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">In our study document I'd like to also include the perspective of other WMS servers besides QGIS server, so any input would be welcome.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Andreas<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">--<o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Andreas Neumann<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><a href="http://QGIS.ORG" target="_blank"><span lang="EN-US">QGIS.ORG</span></a><span lang="EN-US"> board member (treasurer)<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
mapserver-users mailing list<br>
</span><a href="mailto:mapserver-users@lists.osgeo.org" target="_blank"><span lang="EN-US">mapserver-users@lists.osgeo.org</span></a><span lang="EN-US"><br>
</span><a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" target="_blank"><span lang="EN-US">https://lists.osgeo.org/mailman/listinfo/mapserver-users</span></a><span lang="EN-US"><o:p></o:p></span></p>
</blockquote>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
mapserver-users mailing list<br>
</span><a href="mailto:mapserver-users@lists.osgeo.org" target="_blank"><span lang="EN-US">mapserver-users@lists.osgeo.org</span></a><span lang="EN-US"><br>
</span><a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" target="_blank"><span lang="EN-US">https://lists.osgeo.org/mailman/listinfo/mapserver-users</span></a><span lang="EN-US"><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
--<br>
Andreas Neumann<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><a href="http://QGIS.ORG" target="_blank"><span lang="EN-US">QGIS.ORG</span></a><span lang="EN-US"> board member (treasurer)<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>