[Qgis-psc] Handling the Travis CI situation

[Denis Rouzaud]

Le lun. 9 nov. 2020 à 09:06, Alessandro Pasotti <apasotti at gmail.com> a
écrit :

> On Mon, Nov 9, 2020 at 12:55 AM Nyall Dawson <nyall.dawson at gmail.com>
> wrote:
> >
> > On Fri, 6 Nov 2020 at 20:57, Matthias Kuhn <matthias at opengis.ch> wrote:
> > >
> > > This would be desirable.
> > >
> > > It's unfortunately a bit of complexity included. As Alessandro said,
> the rendering tests are a good example of tests which are hard to maintain
> and - unfortunately - not well defined functions like we happen to have on
> e.g. database engine unit tests.
> > >
> > > To recall, when we started with CI some 5 years ago we were in the
> situation that we had a set of tests, which randomly passed on some dev
> machines and some on others. So effectively you had no chance to know if a
> test doesn't pass because of your machine or your patch.
> > >
> > > At least now we have a reference platform and know if a patch actually
> does cause changes. Which is the most important information.
> > >
> > > Currently many of the tests are already cross platform. And nobody is
> actively writing CI centric tests. It's just a matter of fact that the
> tests have a requirement to at least pass on the CI env because it's the
> only thing that can be enforced.
> > >
> > > When tests are updated to a new platform (ci or not), many of them
> will pass on a wider variety of platforms. Because bugs are fixed or they
> are made more generic. And some will just have reference images for one
> more platform added.
> > >
> > > So whatever we do we will do a step into the right direction. But
> there will be no guarantee that every test passes on your machine except if
> you make them all pass on your machine - which would be very welcome!!
> > >
> > > What you could also do is listing tests that are generic / platform
> independent (e.g. the expression tests would be a very good example) and
> then always just run these instead of the whole set of tests.
> >
> >
> > Just to add a few extra thoughts to the already comprehensive replies
> > given by Matthias and Denis:
> >
> > - we had to do a very similar effort with updating existing tests when
> > we moved from Qt 4 -> Qt 5. We'll probably have to do another similar
> > effort in another 18-24 months or so. It's going to be a regular,
> > recurring task to go through and update all the tests which have been
> > introduced since the previous effort and ensure that they are
> > sufficiently tolerant to pass under different environments/software
> > versions. (Maybe we should consider adding "test maintenance" as a
> > regular yearly expense of the nature of 1.5 weeks?)
> > - we really should do a similar effort to get the existing tests
> > passing under mac os and windows too. I suspect there's some valid
> > bugs that the test suites would reveal if we could reliably run them
> > under windows/mac, but the real issues are drowned in the noise of
> > tests which haven't been designed to be cross platform compatible.
> > (This could be a good grant proposal idea for future funding rounds!)
> >
> > And then my personal 2c:
> >
> > We have a great test suite, and fantastic tools for making and
> > managing tests. Sure, there's a learning curve involved with them.
> > Sure, they ARE different to the test suites used by gdal, or geos, or
> > <insert other project here>. But that's just business as usual in
> > software development, and not at all reflective of inferiority in the
> > test suites. Can we please move on from this recurring point once and
> > for all and focus on the current relevant parts of this discussion
> > instead?
> >
> > Nyall
> >
> >
>
>
> Hi,
>
> While I agree on all points, there is one important thing that we
> should make it better while we work on it:
> make sure the test suite can **easily** run locally using the same
> docker images we are using in the CI process, I can do it but I cannot
> say it was easy to set up and it's probably overly complicated for
> most people.
>

The new github workflow is far more simple and easier to read than the
travis config.
Everything is in one file and should be easily replicable.
https://github.com/qgis/QGIS/blob/test-focal/.github/workflows/run-tests.yml#L101-L124


> Also, making the test suite independent from the particular CI we will
> use (GH workflows, Travis & C.) will make it easier to move it to
> another CI if needed.
>

It should!
Let's when we move the CI for 3.16 to Github: the base image will remain
(bionic) and we shouldn't have anything to touch in the tests.


>
> This is of course also a matter of producing a good documentation for
> the process and the tools.
>
> Kind regards.
>
> --
> Alessandro Pasotti
> QCooperative:  www.qcooperative.net
> ItOpen:   www.itopen.it
> _______________________________________________
> Qgis-psc mailing list
> Qgis-psc at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/qgis-psc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20201109/35f6c4e4/attachment.html>