[Qgis-user] Crowdfunding Project: Automated tests for QGIS

[Nyall Dawson]

On 10 October 2014 22:58, Matthias Kuhn <matthias.kuhn at gmx.ch> wrote:
> Dear QGIS users,
>
> I started a crowd-funding campaign for automated unit tests in QGIS. Please
> consider to help funding this project in order to improve the software and
> code quality in the long run. More details can be found on the project page
> and in the text below.
> I plan to improve the quality of QGIS sustainably by introducing the
> automatic execution of test procedures which will inform responsible
> developers automatically in case of failure. Your support for this project
> is highly welcome and you can contribute to make QGIS more stable in the
> long run. To implement the necessary changes as well as repairing currently
> broken tests a minimal amount of CHF 5000.- will be required. The following
> paragraphs will describe the project in more detail.

Nice initiative - and I'm strongly in favour of anything that improves
our test suite.

Here's the situation with composer tests:
They all run and pass on my workstation (ubuntu 14.04). I use them
obsessively, before every commit, to test for regressions. But they
are extremely susceptible to platform specific text rendering issues,
and consequently fail for almost all other platforms. I think this
could be fixed by making the pixel tolerance in
QgsRenderChecker/QgsCompositionChecker apply to anomaly images also.
This way it would be relatively simple to add known anomaly images for
each platform, and tweak the tolerance until we hit a sweet spot. We
also need to make sure every test uses the standard test font (I've
got a commit nearly ready to go which does this for
TestQgsComposerHtml).

I'm also curious to see how crowdfunding works for QGIS. I know other
open source projects (eg, Krita [1] ) have had great success with
crowdfunding feature development. I suspect that crowdfunding may be
more successful in consumer, rather than enterprise, focused projects,
but I'd love to be proved wrong. Imagine if we could crowdfund some
full time developer positions for QGIS!

Nyall

[1] https://www.kickstarter.com/projects/krita/krita-open-source-digital-painting-accelerate-deve

>
> ----------
>
> In the last years, QGIS matured from a hobby project to a serious
> alternative with a broad scope of possibilities and a great potential.
>
> With the constant addition of new features new possibilities have been
> created and still are created. Every new feature means changes to the source
> code of the software and is a potential source of errors.
>
> There are possibilities to automatically detect errors. This is done by
> executing a series of operations with known result. If a operation does not
> return the desired (known) result something went wrong and it needs to be
> evaulated why.
>
> QGIS has currently over 100 different tests. However, these are not run
> regularly and nobody realizes when tests fail. This led to the situation,
> that at the moment approximately 2/3 of the tests do not report success and
> nobody knows why and since when.
>
> I intend to take the following measures to ensure that in the future tests
> no longer can silently fail:
>
> * tests will be run on all source code changes in master (main development
> branch)
>
> * tests will be run on all pull requests (changes being evaluated for
> integration in master)
>
> * if tests fail the responsible developers will be notified by mail
>
> * with symbols, the current state of the tests will be indicated
>
> With this system it is possible to react instantaneously when tests fail and
> to revert the responsible changes or to not even integrate these.
>
> To make sure that this system works and is accepted, it is important to
> start with a working suite of tests. How much time it takes to fix all tests
> is difficult to say. The plan is therefore to fix as many tests with the
> available funds and to disable the rest of the tests in the first time.
> Therefore additional funds that are provided on top of the required minimum
> CHF 5000.- will be very welcome for additional working tests.
>
> It would be great if this crowd-funding effort could help to get this
> project to fly.
>
> Kind regards,
>
> Matthias Kuhn
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-user