[Qgis-psc] Handling the Travis CI situation

[Matthias Kuhn]

This would be desirable.

It's unfortunately a bit of complexity included. As Alessandro said, the
rendering tests are a good example of tests which are hard to maintain and
- unfortunately - not well defined functions like we happen to have on e.g.
database engine unit tests.

To recall, when we started with CI some 5 years ago we were in the
situation that we had a set of tests, which randomly passed on some dev
machines and some on others. So effectively you had no chance to know if a
test doesn't pass because of your machine or your patch.

At least now we have a reference platform and know if a patch actually does
cause changes. Which is the most important information.

Currently many of the tests are already cross platform. And nobody is
actively writing *CI centric tests*. It's just a matter of fact that the
tests have a requirement to *at least* pass on the CI env because it's the
only thing that can be enforced.

When tests are updated to a new platform (ci or not), many of them will
pass on a wider variety of platforms. Because bugs are fixed or they are
made more generic. And some will just have reference images for one more
platform added.

So whatever we do we will do a step into the right direction. But there
will be no guarantee that every test passes on your machine except if you
make them all pass on your machine - which would be very welcome!!

What you could also do is listing tests that are generic / platform
independent (e.g. the expression tests would be a very good example) and
then always just run these instead of the whole set of tests.

Matthias


On Fri, Nov 6, 2020 at 11:29 AM Sandro Santilli <strk at kbt.io> wrote:

> On Fri, Nov 06, 2020 at 05:07:40PM +1000, Nyall Dawson wrote:
>
> > - different Qt versions on Github vs Travis
> > - different underlying library versions (e.g. updated proj, gdal, and
> > others vs the older Travis build environment)
> > -  other differences in the build setup, e.g. different limits on the
> > environment under which the tests run
> >
> > Before we can make the switch and move away from Travis, we'll need to
> > update these tests and get as many of them passing as possible on
> > Github, and then handle the tricky job of backporting the test fixes
> > and Github action setup to the 3.16/3.10 branches.
>
> I'd recommend changing the tests to return success results with BOTH
> versions of Qt, proj, gdal etc. The goal would be making the testsuite
> pass on _developers_ machines, whatever libraries they are using.
>
> The CI-centric nature of QGIS testsuite has always been a problem for
> me as it makes it impossible to check the quality of changes locally.
>
> There are likely existing tickets addressing these issues.
> A 'testsuite' label exists. I'm not sure all the relevant tickets
> have the label applied, but it could be a good starting point:
>
>
> https://github.com/qgis/QGIS/issues?q=is%3Aopen+is%3Aissue+label%3Atestsuite
>
> --strk;
> _______________________________________________
> Qgis-psc mailing list
> Qgis-psc at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/qgis-psc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20201106/a35bf4fd/attachment.html>