<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Dear PSC,
</p>
<p dir="auto"><br>
</p>
<p dir="auto">As explained previously by Régis, we spent some time
working on a dedicated platform to benchmark performance between
several QGIS Server versions (currently 2.14, 2.18, 3.0 and
master) and it's time to release it now.</p>
<p dir="auto">Many users are asking for information about
performance differences between QGIS Server 2.X and 3.X. However,
they often want to test performance with their own dataset, and
not only with some generic stuff.
And, in this particular case, we realized that there weren't any
simple and convenient way to concretely measure and generate a
performance report. For the purpose of meeting the specific needs
of these requests, we worked on two dedicated projects: Graffiti
[1] and QGIS-Server-Perfsuite [2].</p>
<p dir="auto"><br>
</p>
<h3>
</h3>
<h3 dir="auto">Graffiti</h3>
<p dir="auto">Graffiti is a simple Python tool allowing to generate
a HTML report from a tests scenario described in a YAML file. This
way, anyone with available QGIS Server instances may use this tool
to generate a performance report with custom data, custom .qgs
project and custom WMS parameters. This tool meets the initial
demands. <br>
</p>
<p dir="auto"><br>
</p>
<h3 dir="auto">QGIS Server Perfsuite</h3>
<p dir="auto">We need such reports to be generated on a daily basis
so that we can track regressions or improvements. QGIS Server
Perfsuite allows to easily deploy a whole platform with Dockerfile
to build/execute QGIS Server, some Ansible scripts for a remote
deployment and default configuration files for Graffiti. The
result is the report previously mentioned [0].</p>
<p dir="auto"><br>
</p>
<h3 dir="auto">First results</h3>
<h3>
</h3>
<p dir="auto">Regarding these preliminary results, we don't have
very good results for lines and polygons in 3.X, but all this
requires more investigations because internal test for one
customer sometimes shows different tendencies. We already tackled
one performance issue with line parallel labelling. QGIS 2.18 and
PAL candidates were not taken into account anymore. Once located
and the bugfix merged in master, performances are good again, and
this scenario is clearly visible in the report [3].</p>
<p dir="auto">All this to say that performance regressions can
happen suddenly, and developing without keeping an eye on
performances may be dangerous, especially from a server point of
view. The PerfSuite contains only a few scenarios and datasets,
and we aim at adding new ones progressively.
Of course, we have to be realistic and keep in mind that,
regarding the number of options in QGIS, server tuning and data
providers, we cannot test everything...</p>
<p dir="auto"><br>
</p>
<h3 dir="auto">How does this platform plays with MSPerf from C2C ?</h3>
<p dir="auto">It should be made clear that the C2C performance
platform does not have the same purpose that what we're are
working on. It's not better, it's not worse, it's just a different
tool. Firstly, in our case, we don't want to compare QGIS Server
with other map servers. The underlying aim is clearly to provide a
convenient tool for developers and users to measure the response
time of the server on specific data with a particular
configuration (by the way, it seems that Docker images of QGIS are
private in C2C, which make their facility a tiny difficult to
customize without asking them [5]). Secondly, graffiti measures
the unitary response time per request, not according to a number
of users. Once again, it's a very good thing to have to
contemplate the big picture! However, at this stage, it seems more
suitable to check the unit rendering time, without messing with
web server configurations. Moreover, it allows us to observe the
caching time (like with the trust option [6]). <br>
</p>
<p dir="auto"><br>
</p>
<p dir="auto">The last step from a system point of view, is to run
these tests in a continuous integration environment. Is the PSC
interested in helping us in that direction?</p>
<p dir="auto">Of course, we look forward to hearing the comments and
criticisms :).</p>
<p dir="auto"><br>
</p>
<p dir="auto">Sorry for being so long, and have a good day!</p>
<p dir="auto"><br>
</p>
<p dir="auto">All the best.</p>
<p dir="auto">Paul / the Oslandia Team</p>
<p dir="auto"><br>
</p>
<p dir="auto">[0] <a
href="http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html"
rel="nofollow noreferrer noopener" target="_blank">http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html</a>
<br>
</p>
<p dir="auto">[1] <a href="https://github.com/pblottiere/graffiti"
rel="nofollow noreferrer noopener" target="_blank">https://github.com/pblottiere/graffiti</a>
<br>
</p>
<p dir="auto">[2] <a
href="https://github.com/Oslandia/QGIS-Server-PerfSuite"
rel="nofollow noreferrer noopener" target="_blank">https://github.com/Oslandia/QGIS-Server-PerfSuite</a>
<br>
</p>
<p dir="auto">[3] <a
href="http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html#a98e80fea3074fe19f037adb8e86d35c"
rel="nofollow noreferrer noopener" target="_blank">http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html#a98e80fea3074fe19f037adb8e86d35c</a>
<br>
</p>
<p dir="auto">[4] <a href="https://github.com/KDAB/hotspot"
rel="nofollow noreferrer noopener" target="_blank">https://github.com/KDAB/hotspot</a>
<br>
</p>
<p dir="auto">[5] <a
href="https://github.com/camptocamp/ms_perfs/pull/27"
rel="nofollow noreferrer noopener" target="_blank">https://github.com/camptocamp/ms_perfs/pull/27</a>
<br>
</p>
<p dir="auto">[6] <a
href="http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html#0e8047f450854e73a08851336b21a1d7"
rel="nofollow noreferrer noopener" target="_blank">http://37.187.164.233/qgis-server-perfsuite-report/graffiti/report.html#0e8047f450854e73a08851336b21a1d7</a></p>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 09/06/18 16:07, Régis Haubourg
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CABgOYCe-pMijUW8tDxHwa=KUmLAvOuO3B-bC7Lr-=cN7L6pdyg@mail.gmail.com">
<div dir="ltr">
<div>Hi PSC , <br>
</div>
<div><br>
</div>
<div>Yes we are aware of the MS perf platform since the 2016
QGIS server code sprint.<br>
</div>
<div>
<div><br>
</div>
<div>We have been discussing quite a few times about coding
with performance and waited for some investment from C2C
during 2017. We are very happy they now publish a daily
report, however publishing static tests on static datasets
is not what we seek. Yves showed some results in the QGIS Fr
user days two years back, and I saw the expectations,
frustrations and debates it rose that time. <br>
</div>
<div> Our goal is NOT to compare QGIS server with Mapserver or
GeoServer. We don't want to fall into these of debates that
in my opinion are dangerous for everyone because it's almost
impossible to reach absolute measuring for tools that runs
in different environnement and render data differenly.</div>
<div><br>
</div>
<div> We are talking of a semi scientific approach here, and
being able to reproduce and confirm issues with different
tools is probably a very good thing, if we just can take
time to analyse things.</div>
<div><br>
</div>
<div>Our task was to build a very light platform dedicated to
be integrated in continuous integration, and that is really
easy to enrich with new tests. So we build a comparison
between framework between ltr, release and dev version that
can be run in many contexts (web server type, mutlithread,
multiprocess options, rendering options, data providers,
etc..).</div>
<div><br>
</div>
<div> We also want to keep an history of the developpement
version performance for each test. <br>
</div>
<div><br>
</div>
<div>As you see, this can lead to massive amount of computing
and logs, so we really need something easy to set up and as
light as possible. Moreover, measuring performance needs
light tools to avoid influencing the measure themselves. <br>
</div>
<div><br>
</div>
We believe that QGIS renders data extremely fast, but it has
some glitches due to the desktop design oringin that can be
tackled if we have permanent feedback. <br>
</div>
<div> <br>
</div>
<div>Paul just finished the platform this week, and we now
dedicating efforts on running it on a dedicated server to be
certain of not being influenced by any external load. <br>
</div>
<div><br>
</div>
<div>I hope next week we'll publish it with the first reports. <br>
</div>
<br>
<div>In the end, the question of hosting the platform on
dedicated server still remains, as much as administrating it
correctly so that no tool run in parallel at the same time,
with the risk of altering the measures. I hope you are still
ok with the fact that we need this kind of tool. <br>
</div>
<div><br>
</div>
<div>Best regards, <br>
</div>
<div>Régis<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-06-09 15:10 GMT+02:00 Richard 🌍
Duivenvoorde <span dir="ltr"><<a
href="mailto:richard@duif.net" target="_blank"
moz-do-not-send="true">richard@duif.net</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi PSC,<br>
<br>
On my TODO list there was to ask Yves/Camptocamp about their
QGISserver<br>
tests and make sure Oslandia was eventually aware of that.<br>
<br>
See below:<br>
<br>
- results are available<br>
<a
href="https://gmf-test.sig.cloud.camptocamp.net/ms_perfs/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://gmf-test.sig.cloud.<wbr>camptocamp.net/ms_perfs/</a><br>
and apparenlty updated daily?<br>
Yves promises to do some cleanup and upload to <a
href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">test.qgis.org</a><br>
<br>
- he thinks Oslandia is aware of this work (to be sure, I
bcc Regis :-)<br>
<br>
Regards,<br>
<br>
Richard Duivenvoorde<br>
<br>
<br>
-------- Forwarded Message --------<br>
Subject: Re: Performance tests QGIS Server<br>
Date: Wed, 6 Jun 2018 10:30:39 +0200<br>
<br>
Le 04/06/2018 à 20:44, Richard 🌍 Duivenvoorde a écrit :<br>
> Hi Yves,<br>
><br>
> During PSC meeting we were talking about some
QGIS-Server OWS<br>
> performance-tests/service that Oslandia is doing
currently.<br>
I heard something about this indeed.<br>
> Andreas mentioned that CampToCamp also did something
(for Andreas) last<br>
> year. And that you asked me to put it on the website.<br>
> So Question: did you ask me? And did I answer? :-)<br>
Camptocamp did it something and we discussed about how to
improve it and<br>
share to the community. The topic was "QGIS benchmark"
(07/11/2017). See<br>
<a
href="https://gmf-test.sig.cloud.camptocamp.net/ms_perfs/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://gmf-test.sig.cloud.<wbr>camptocamp.net/ms_perfs/</a>.
Here a quote of<br>
your answer:<br>
<br>
"""<br>
What is the idea? That we (as <a href="http://qgis.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">qgis.org</a>
<<a href="http://qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://qgis.org</a>>)
ourselves run<br>
the benchmark<br>
(docker?) every now and then?<br>
<br>
If NOT, then it is easiest when I give you credentials on
the<br>
<a href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">test.qgis.org</a>
<<a href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://test.qgis.org</a>>
(virtual)Webserver, so you can<br>
rsync/scp the result to a<br>
directory 'benchmarks' at:<br>
<br>
<a href="http://test.qgis.org/" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://test.qgis.org/</a><br>
<br>
As you can see Paul (of Oslandia) also pushes the QGISServer
CITE test<br>
results there into a directory 'ogc_cite':<br>
<br>
<a class="moz-txt-link-freetext" href="http://">http://</a><a href="http://test.qgis.org/ogc_cite/"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://test.qgis.org/<wbr>ogc_cite/</a>
<<a href="http://test.qgis.org/ogc_cite/"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://test.qgis.org/ogc_<wbr>cite/</a>><br>
<br>
The idea is to either add an index.html to <a
href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">test.qgis.org</a><br>
<<a href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://test.qgis.org</a>>
which then<br>
sents you to individual test directories (or the latest in
that<br>
directory), OR we create a (translatable) page in the <a
href="http://qgis.org" rel="noreferrer" target="_blank"
moz-do-not-send="true">qgis.org</a><br>
<<a href="http://qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://qgis.org</a>>
website<br>
which does some description of the different tests, and then
links to<br>
the html-output pages on <a href="http://test.qgis.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">test.qgis.org</a>
<<a href="http://test.qgis.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://test.qgis.org</a>><br>
I think there should really be some explanation at the
performance tests...<br>
<br>
Both is possible, just need some body/time to do it :-)<br>
"""<br>
I am ok with a rsync on the QGIS server. Result need some
love to<br>
improve some graphics (null value gives no graph at all).<br>
<br>
And I should add picture of layer to illustrate the layer
complexity.<br>
<br>
> Andreas was also wondering in how much of the Oslandia
work is actually<br>
> already done by C2C and if it is still handy to
communicatie about this.<br>
I don't know, I have no idea what Oslandia is working on,
well not more<br>
than what they shared on QGIS mailing list. They are aware
of our<br>
project for sure, as we discussed about this at the QGIS
server hackfest<br>
and 3Liz pushed a pull request.<br>
<br>
Y.<br>
<br>
</blockquote>
</div>
<br>
</div>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Qgis-psc mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Qgis-psc@lists.osgeo.org">Qgis-psc@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/qgis-psc">https://lists.osgeo.org/mailman/listinfo/qgis-psc</a></pre>
</blockquote>
<br>
</body>
</html>