[MapProxy] speed of preseeding existing wmts

Thu Apr 21 09:29:50 PDT 2022

To follow up on this and provide a possible solution I answer my own
question.

Some problems were related to the geographically location I was back then
when I tried all this. The internet connection was quite good but for some
reasons DNS was slow and probably I got some speed penalty by swisstopo
after many try and error on their WMTS.

However I can now say I have a setup of mapproxy which gives me a
acceptable seeding speed (around 70Mb/s) now when I'am back in Switzerland.

Please find attached the config files which are working well for me now. I
also played around with several cache types to test i/o speed so the setup
is also reflected in the configs. A possible speed up is achieved by
the bulk_meta_tiles option for the caches.

I seed with the following command:
mapproxy-seed -f /io/config/mapproxy.yaml -s /io/config/seed.yaml -c 32
--seed swisstopo.mbtiles

As you can see I restrict the BBOX to a specific one around switzerland
because the original one (by WMTS capabilities
https://wmts.geo.admin.ch/EPSG/2056/1.0.0/WMTSCapabilities.xml) is huge and
contains mostly white tiles. Don't know why this might be necessary.

Cheers
Clemens

On Wed, Apr 13, 2022 at 5:33 PM Clemens Rudert <clemens at opengis.ch> wrote:

> Thank you for the response. I poked around a bit on that but didn't came
> to a valid conclusion. What is even more strange-if I use a WMS instead of
> WMTS as source this runs much faster. Preseeding is using fair ammount of
> CPU and a reasonable ammount of net traffic (1.2Mb/s). It is more in the
> region where I would expect such a process. Of course now the tiles have to
> be "produced".
>
> However this does not explain why simply downloading the tiles from WMTS
> and storing them locally is that much slower.
>
> Regarding the slower python in docker I found some confirming articles and
> tried several quick fixes (seccomp off, etc) but with no measurable effect.
>
> Unfortunately I am not able to use a local plain mapproxy instance for
> testing because of version conflicts. My system proj is of version 8.8
> which seems to conflict with mapproxy even on the master branch.
>
> Cheers
> Clemens
>
> On Wed, Apr 13, 2022 at 1:47 PM GMUER Daniel <daniel.gmuer at hexagon.com>
> wrote:
>
>> Hi Clemens
>>
>>
>>
>> Very interesting post of your work as I’m doing exactly the same thing
>> right now 😅. I ported Mapproxy to a Dockercontainer and seed a WMS. In
>> my example I use a WMS from Stadt Zürich but this should not make any
>> difference to the swisstopo-services. I did not notice any performance
>> issues on my dockercontainer until I read your post. But now I figured out
>> I’m having the same Problem.
>>
>> Just a small comprehension of what I observed while seeding with a
>> concurrency of 16:
>>
>> Pure Python: 130’000 Tiles/10min
>>
>> Docker : 36’064 Tiles/10min
>>
>>
>>
>> I think this might be a question on the Docker side.. I’ll try to find
>> out more about this and let you know if I find out something.
>>
>>
>>
>> If you want to reply, you can do this in german. I’m writing in English
>> as I’ve seen your Opengis’ler are working from all over the place and I’m
>> not sure if you are familiar with german.. Wie immer du magst 😉
>>
>>
>>
>> Regards
>>
>> Daniel
>>
>>
>>
>> *Daniel Gmür*
>> Project Engineer
>>
>> Safety, Infrastructure & Geospatial division
>>
>> *Hexagon*
>>
>>
>>
>> *T: *+41 43 322 46 46
>> *E: **daniel.gmuer at hexagon.com <daniel.gmuer at hexagon.com>* *
>>
>>
>>
>> *HxGN Schweiz AG*
>>
>> Flurstrasse 55
>>
>> 8048 Zürich, Switzerland
>> *hexagongeospatial.com <http://hexagongeospatial.com>* | Blog
>> <https://blog.hexagongeospatial.com/>| *LinkedIn
>> <https://www.linkedin.com/company/hexagon-geospatial>* | *Twitter
>> <https://twitter.com/hexgeospatial>*
>>
>>
>>
>> *Von:* MapProxy <mapproxy-bounces at lists.osgeo.org> *Im Auftrag von *Clemens
>> Rudert
>> *Gesendet:* Mittwoch, 13. April 2022 12:14
>> *An:* mapproxy at lists.osgeo.org
>> *Betreff:* [MapProxy] speed of preseeding existing wmts
>>
>>
>>
>> Sie erhalten nicht oft E-Mail von "clemens at opengis.ch". Weitere
>> Informationen, warum dies wichtig ist
>> <http://aka.ms/LearnAboutSenderIdentification>
>>
>> This email is not from Hexagon’s Office 365 instance. Please be careful
>> while clicking links, opening attachments, or replying to this email.
>>
>>
>>
>> Hello all
>>
>>
>>
>> I have the situation that I want to cache an existing WMTS to my projects
>> infrastructure to be more secure with 24/7 availability.
>>
>>
>>
>> I try to cache 4 layers provided by swisstopo, the federal swiss geo data
>> agency. This WMTS's are really fast and usually having no issues with
>> catching the tiles.
>>
>>
>>
>> I do setup my mapproxy to have this 4 layers definded in epsg:2056 and I
>> also defined the correct grid sucessfully. Maps are cached on request and
>> it works as expected.
>>
>>
>>
>> All is running in a Docker-Container
>>
>>
>>
>> Internet-connection supports up to 1gbit/s and shouldn't be a bottleneck.
>>
>>
>>
>> But now If I try to preseed this 4 layers I can't get real performance
>> out. I defined BBOXES in epsg:2056 to limit the area. I do not reproject
>> anything. I simply store loaded tiles. In my understanding there shouldn't
>> be a bottleneck somewhere. But there is. I tried the mapproxy-seed command
>> with many different settings. Also with -c from 1-25. But there is no
>> difference. The different spawned processes are using around 1.5% of CPU
>> and net throughput is somewhere around 300kb/s if I use 25 concurrent
>> tasks. It stays at around 50kb/s when I use one concurrent task.
>>
>>
>>
>> Machine is a 8 core with 32gb of ram (not running into limits) and a
>> usually fast SSD.
>>
>>
>>
>> I write here because I can't come to a conclusion what might be wrong.
>> Maybe it's something simple I miss here?
>>
>>
>>
>> I Attached the 2 config files I use.
>>
>>
>>
>> Cheers
>>
>> Clemens
>>
>>
>>
>>
>>
>>
>>
>>
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqfield.org%2Fget%2F&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=waLyoY%2F6mCz9LuukYZyj%2Fr1teKmPnc2JTApEfLYJMts%3D&reserved=0>
>>
>> QFIELD 2.0 IS HERE! - Hold the power of QGIS in your hand - learn more
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.opengis.ch%2F2022%2F04%2F05%2Fqfield-2-0-is-here%2F&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=n5PxZAhK%2FOHwkWI6UBKK6aARuLIKGYrwlNuW5M53Z9I%3D&reserved=0>
>> - get it now
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fqfield.org%2Fget&data=04%7C01%7C%7C16e6416b191b42f8a24d08da1d365622%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637854417215960490%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=SCl3%2B%2FYLtV8a0H28ME18lvwpDy5uX7dAkPA2xLirxhc%3D&reserved=0>
>>
>

-- 
 <https://qfield.org/get/>

QFIELD 2.0 IS HERE! - Hold the power of QGIS in 
your hand - learn more 
<https://www.opengis.ch/2022/04/05/qfield-2-0-is-here/> - get it now 
<https://qfield.org/get>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20220421/6620075c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD0002.jpg
Type: image/jpeg
Size: 823 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20220421/6620075c/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mapproxy.yaml
Type: application/x-yaml
Size: 3265 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20220421/6620075c/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: seed.yaml
Type: application/x-yaml
Size: 586 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20220421/6620075c/attachment-0003.bin>