[Qgis-user] Crowdfunding Project: Automated tests for QGIS

Fri Oct 10 05:25:20 PDT 2014

Hi Joanthan,

Valid points you are raising.
What's currently broken with tests is not, that developers don' care but 
rather, that developers are not aware of failing tests due to the lack 
of visibility. Also it's not possible currently to tell which commit is 
responsible for a failing test.
Another thing is, that there is no reference platform yet. A test may 
work on windows but fail on linux. Then somebody tries to "fix" it and 
breaks it on the other platform. This method will introduce a 
"reference" platform that will have to work. If further reference 
platforms will be added later is open, but something to keep in mind.
These things will be fixed. And I don't assume that somebody will 
disagree if a commit that breaks a test will be reverted as well as it 
will make it easier to review pull requests because a "red" pull request 
will not even have to be considered. But certainly, an official 
statement of the PSC concerning this will be highly appreciated. There 
is also a recent RFC/QEP from Nyall that deals with requirements for 
unit tests. [1]

I hope I could address your concerns and if not, just ask again :-)

Kind regards,
Matthias

[1] 
https://github.com/nyalldawson/QGIS-Enhancement-Proposals/blob/728f5ef8cfc6dd4c7ee830213339842db6891ee1/QEP-5-Unit-tests.rst

On 10/10/2014 02:10 PM, Jonathan Moules wrote:
>
> Hi Matthias,
>
> It’s good to see that some initiative is being taken on this, but if 
> this goes ahead, is there a requirement from the PSC that devs 
> actually do fix things that cause the tests to fail or not have their 
> commit accepted? Because otherwise I can see this devolving into the 
> situation that currently exists (devs don’t bother with unit tests).
>
> Further thought – if this goes ahead it may be worth building on it 
> and requiring that unit tests be created for all new functionality as 
> part of acceptance of that functionality. Are there such plans? 
> (that’s more something for the dev list anyway).
>
> Certainly a good start though!
>
> Cheers,
>
> Jonathan
>
> *From:*qgis-user-bounces at lists.osgeo.org 
> [mailto:qgis-user-bounces at lists.osgeo.org] *On Behalf Of *Matthias Kuhn
> *Sent:* Friday, October 10, 2014 12:58 PM
> *To:* qgis-user at lists.osgeo.org
> *Subject:* [Qgis-user] Crowdfunding Project: Automated tests for QGIS
>
> Dear QGIS users,
>
> I started a crowd-funding campaign for automated unit tests in QGIS. 
> Please consider to help funding this project in order to improve the 
> software and code quality in the long run. More details can be found 
> on the project page and in the text below.
>
> http://blog.vitu.ch/10102014-1046/crowdfunding-initiative-automated-testing
>
> I plan to improve the quality of QGIS sustainably by introducing the 
> automatic execution of test procedures which will inform responsible 
> developers automatically in case of failure. Your support for this 
> project is highly welcome and you can contribute to make QGIS more 
> stable in the long run. To implement the necessary changes as well as 
> repairing currently broken tests a minimal amount of CHF 5000.- will 
> be required. The following paragraphs will describe the project in 
> more detail.
>
> ----------
>
> In the last years, QGIS matured from a hobby project to a serious 
> alternative with a broad scope of possibilities and a great potential.
>
> With the constant addition of new features new possibilities have been 
> created and still are created. Every new feature means changes to the 
> source code of the software and is a potential source of errors.
>
> There are possibilities to automatically detect errors. This is done 
> by executing a series of operations with known result. If a operation 
> does not return the desired (known) result something went wrong and it 
> needs to be evaulated why.
>
> QGIS has currently over 100 different tests. However, these are not 
> run regularly and nobody realizes when tests fail. This led to the 
> situation, that at the moment approximately 2/3 of the tests do not 
> report success and nobody knows why and since when.
>
> I intend to take the following measures to ensure that in the future 
> tests no longer can silently fail:
>
> * tests will be run on all source code changes in master (main 
> development branch)
>
> * tests will be run on all pull requests (changes being evaluated for 
> integration in master)
>
> * if tests fail the responsible developers will be notified by mail
>
> * with symbols, the current state of the tests will be indicated
>
> With this system it is possible to react instantaneously when tests 
> fail and to revert the responsible changes or to not even integrate these.
>
> To make sure that this system works and is accepted, it is important 
> to start with a working suite of tests. How much time it takes to fix 
> all tests is difficult to say. The plan is therefore to fix as many 
> tests with the available funds and to disable the rest of the tests in 
> the first time. Therefore additional funds that are provided on top of 
> the required minimum CHF 5000.- will be very welcome for additional 
> working tests.
>
> It would be great if this crowd-funding effort could help to get this 
> project to fly.
>
> Kind regards,
>
> Matthias Kuhn
>
> This message has been scanned for viruses by MailControl 
> <http://www.mailcontrol.com/>, a service from BlackSpider Technology
>
> Click here 
> <https://www.mailcontrol.com/sr/h0opETQ15z7GX2PQPOmvUgny0%216zNmYYcr+CLPtUcwBrcg72XoIMJrYkKSbFUrl7COZlUrWwm5I94h7CChK4GQ==> 
> to report this email as spam.
>
>
> ------------------------------------------------------------------------
>
> *HR Wallingford and its subsidiaries* uses faxes and emails for 
> confidential and legally privileged business communications. They do 
> not of themselves create legal commitments. Disclosure to parties 
> other than addressees requires our specific consent. We are not liable 
> for unauthorised disclosures nor reliance upon them.
> If you have received this message in error please advise us 
> immediately and destroy all copies of it.
>
> HR Wallingford Limited
> Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom
> Registered in England No. 02562099
>
> ------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20141010/1d56b571/attachment.html>