<div dir="ltr">To follow up on this and provide a possible solution I answer my own question.<div><br></div><div>Some problems were related to the geographically location I was back then when I tried all this. The internet connection was quite good but for some reasons DNS was slow and probably I got some speed penalty by swisstopo after many try and error on their WMTS.</div><div><br></div><div>However I can now say I have a setup of mapproxy which gives me a acceptable seeding speed (around 70Mb/s) now when I'am back in Switzerland.</div><div><br></div><div>Please find attached the config files which are working well for me now. I also played around with several cache types to test i/o speed so the setup is also reflected in the configs. A possible speed up is achieved by the bulk_meta_tiles option for the caches.</div><div><br></div><div>I seed with the following command:<br>mapproxy-seed -f /io/config/mapproxy.yaml -s /io/config/seed.yaml -c 32 --seed swisstopo.mbtiles<br></div><div><br></div><div>As you can see I restrict the BBOX to a specific one around switzerland because the original one (by WMTS capabilities <a href="https://wmts.geo.admin.ch/EPSG/2056/1.0.0/WMTSCapabilities.xml">https://wmts.geo.admin.ch/EPSG/2056/1.0.0/WMTSCapabilities.xml</a>) is huge and contains mostly white tiles. Don't know why this might be necessary. </div><div><br></div><div>Cheers</div><div>Clemens</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 5:33 PM Clemens Rudert <<a href="mailto:clemens@opengis.ch">clemens@opengis.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you for the response. I poked around a bit on that but didn't came to a valid conclusion. What is even more strange-if I use a WMS instead of WMTS as source this runs much faster. Preseeding is using fair ammount of CPU and a reasonable ammount of net traffic (1.2Mb/s). It is more in the region where I would expect such a process. Of course now the tiles have to be "produced".<div><br></div><div>However this does not explain why simply downloading the tiles from WMTS and storing them locally is that much slower.</div><div><br></div><div>Regarding the slower python in docker I found some confirming articles and tried several quick fixes (seccomp off, etc) but with no measurable effect.</div><div><br></div><div>Unfortunately I am not able to use a local plain mapproxy instance for testing because of version conflicts. My system proj is of version 8.8 which seems to conflict with mapproxy even on the master branch.</div><div><br></div><div>Cheers</div><div>Clemens</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 13, 2022 at 1:47 PM GMUER Daniel <<a href="mailto:daniel.gmuer@hexagon.com" target="_blank">daniel.gmuer@hexagon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE-CH">
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Clemens<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Very interesting post of your work as I’m doing exactly the same thing right now
</span><span lang="EN-US" style="font-family:"Segoe UI Emoji",sans-serif">😅</span><span lang="EN-US">. I ported Mapproxy to a Dockercontainer and seed a WMS. In my example I use a WMS from Stadt
Zürich but this should not make any difference to the swisstopo-services. I did not notice any performance issues on my dockercontainer until I read your post. But now I figured out I’m having the same Problem.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Just a small comprehension of what I observed while seeding with a concurrency of 16:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CH">Pure Python: 130’000 Tiles/10min<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CH">Docker : 36’064 Tiles/10min<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CH"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I think this might be a question on the Docker side.. I’ll try to find out more about this and let you know if I find out something.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">If you want to reply, you can do this in german. I’m writing in English as I’ve seen your Opengis’ler are working from all over the place and I’m not sure if you are familiar with german..
Wie immer du magst </span><span lang="EN-US" style="font-family:"Segoe UI Emoji",sans-serif">😉</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Regards<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Daniel<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="color:black">Daniel Gmür</span></b><b><span lang="EN-US" style="color:black"><br>
</span></b><span lang="EN-US" style="color:black">Project Engineer<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:black">Safety, Infrastructure & Geospatial division<u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="FR-CH" style="color:black">Hexagon<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span lang="FR-CH" style="color:black"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span lang="FR-CH" style="color:black">T: </span></b><span lang="FR-CH" style="color:black">+41 43 322 46 46
<br>
<b>E: </b></span><u><span lang="FR-CH" style="color:rgb(68,114,196)"><a href="mailto:daniel.gmuer@hexagon.com" target="_blank">daniel.gmuer@hexagon.com</a></span></u><b><span lang="FR-CH" style="color:rgb(68,114,196)">
</span></b><span lang="FR-CH" style="color:white">*</span><span lang="FR-CH" style="color:gray"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="FR-CH" style="color:gray"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span lang="FR-CH" style="color:black">HxGN Schweiz AG<u></u><u></u></span></b></p>
<p class="MsoNormal"><span lang="FR-CH" style="color:black">Flurstrasse 55<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="FR-CH" style="color:black">8048 Zürich, Switzerland</span><span lang="FR-CH" style="color:black"><br>
</span><u><span style="color:rgb(68,114,196)"><a href="http://hexagongeospatial.com" target="_blank"><span lang="FR-CH" style="color:rgb(5,99,193)">hexagongeospatial</span><span lang="FR-CH" style="color:rgb(5,99,193)">.com</span></a></span></u><span style="color:rgb(68,114,196)">
</span><span lang="FR-CH" style="color:black">| </span><span style="color:gray"><a href="https://blog.hexagongeospatial.com/" target="_blank"><span lang="FR-CH" style="color:rgb(5,99,193)">Blog</span></a></span><span lang="FR-CH" style="color:black">|</span><span lang="FR-CH" style="color:rgb(68,114,196)">
</span><u><span style="color:rgb(68,114,196)"><a href="https://www.linkedin.com/company/hexagon-geospatial" target="_blank"><span lang="FR-CH" style="color:rgb(5,99,193)">LinkedIn</span></a></span></u><span lang="FR-CH" style="color:rgb(68,114,196)"> </span><span lang="FR-CH" style="color:black">|</span><span lang="FR-CH" style="color:gray"> </span><u><span style="color:rgb(68,114,196)"><a href="https://twitter.com/hexgeospatial" target="_blank"><span lang="FR-CH" style="color:rgb(5,99,193)">Twitter</span></a></span></u><span style="color:rgb(68,114,196)">
<u><span lang="FR-CH"><u></u><u></u></span></u></span></p>
<p class="MsoNormal"><span lang="FR-CH"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="DE">Von:</span></b><span lang="DE"> MapProxy <<a href="mailto:mapproxy-bounces@lists.osgeo.org" target="_blank">mapproxy-bounces@lists.osgeo.org</a>>
<b>Im Auftrag von </b>Clemens Rudert<br>
<b>Gesendet:</b> Mittwoch, 13. April 2022 12:14<br>
<b>An:</b> <a href="mailto:mapproxy@lists.osgeo.org" target="_blank">mapproxy@lists.osgeo.org</a><br>
<b>Betreff:</b> [MapProxy] speed of preseeding existing wmts<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<table border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100%">
<tbody>
<tr>
<td style="background:rgb(166,166,166);padding:5.25pt 1.5pt"></td>
<td width="100%" style="width:100%;background:rgb(234,234,234);padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal">
<span style="font-size:9pt;font-family:"Segoe UI",sans-serif;color:rgb(33,33,33)">Sie erhalten nicht oft E-Mail von "<a href="mailto:clemens@opengis.ch" target="_blank">clemens@opengis.ch</a>".
<a href="http://aka.ms/LearnAboutSenderIdentification" target="_blank">Weitere Informationen, warum dies wichtig ist</a><u></u><u></u></span></p>
</div>
</td>
<td width="75" style="width:56.25pt;background:rgb(234,234,234);padding:5.25pt 3.75pt">
</td>
</tr>
</tbody>
</table>
<div>
<div style="border:1pt solid red;padding:2pt">
<p class="MsoNormal" style="line-height:12pt"><span style="font-size:10pt;color:red">This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hello all <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have the situation that I want to cache an existing WMTS to my projects infrastructure to be more secure with 24/7 availability.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I try to cache 4 layers provided by swisstopo, the federal swiss geo data agency. This WMTS's are really fast and usually having no issues with catching the tiles.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I do setup my mapproxy to have this 4 layers definded in epsg:2056 and I also defined the correct grid sucessfully. Maps are cached on request and it works as expected.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">All is running in a Docker-Container<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Internet-connection supports up to 1gbit/s and shouldn't be a bottleneck.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">But now If I try to preseed this 4 layers I can't get real performance out. I defined BBOXES in epsg:2056 to limit the area. I do not reproject anything. I simply store loaded tiles. In my understanding there shouldn't be a bottleneck somewhere.
But there is. I tried the mapproxy-seed command with many different settings. Also with -c from 1-25. But there is no difference. The different spawned processes are using around 1.5% of CPU and net throughput is somewhere around 300kb/s if I use 25 concurrent
tasks. It stays at around 50kb/s when I use one concurrent task.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Machine is a 8 core with 32gb of ram (not running into limits) and a usually fast SSD.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I write here because I can't come to a conclusion what might be wrong. Maybe it's something simple I miss here?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I Attached the 2 config files I use. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Clemens<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqfield.org%2Fget%2F&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=waLyoY%2F6mCz9LuukYZyj%2Fr1teKmPnc2JTApEfLYJMts%3D&reserved=0" target="_blank"><span style="border:1pt solid windowtext;padding:0cm;text-decoration:none"><img border="0" width="100" height="100" style="width: 1.0416in; height: 1.0416in;" id="gmail-m_4725731691573725939gmail-m_4021812579251751183Bild_x0020_1" src="cid:1802388e08bbeff1c801"></span></a><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:Arial,sans-serif">QFIELD 2.0 IS HERE!</span><span style="font-size:12pt;font-family:Roboto">
</span><span style="font-family:Arial,sans-serif">- Hold the power of QGIS in your hand -
<a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.opengis.ch%2F2022%2F04%2F05%2Fqfield-2-0-is-here%2F&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=n5PxZAhK%2FOHwkWI6UBKK6aARuLIKGYrwlNuW5M53Z9I%3D&reserved=0" target="_blank">
learn more</a> - <a href="https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqfield.org%2Fget&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=SCl3%2B%2FYLtV8a0H28ME18lvwpDy5uX7dAkPA2xLirxhc%3D&reserved=0" target="_blank">
get it now</a></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>
<br>
<div style="text-align:left"><a href="https://qfield.org/get/" target="_blank"><img src="https://www.opengis.ch/wp-content/uploads/2022/04/Qfield_Banner_email.png" alt="https://qfield.org/get/"></a><span style="font-family:Arial"><a><br></a></span></div><div style="text-align:left"><span style="font-family:Arial">QFIELD 2.0 IS HERE!<span style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><font size="3"> </font></span>- Hold the power of QGIS in your hand</span><span style="font-family:Arial"> - <a href="https://www.opengis.ch/2022/04/05/qfield-2-0-is-here/" target="_blank">learn more</a> - <a href="https://qfield.org/get" target="_blank">get it now</a><br></span></div>