[Qgis-psc] Handling the Travis CI situation

Thu Jan 28 14:28:36 PST 2021

Hi all, no need to vote again, the budget was approved and it included an
item for working on CI.

So the funds are available. I spoke about this yesterday with Andreas
following up a request by Denis.

Nyall do I see it correctly that you and Denis (and Alessandro?) are
probably the ones going to be doing most of the work?

Do you have a concrete plan and how can we help getting things moving?

cheers Marco

On Thu, 28 Jan 2021, 23:11 Tim Sutton, <tim at kartoza.com> wrote:

> Any reason we can't just vote on it here on teh ML Marco?
>
> I am +1 to invest the money to pay the crack team to fix the tests. If
> Microsoft ever shafts us we can consider moving to a self hosted Jenkins
> (my preference) or that thing that Sandro mentioned, but at least we will
> have a working suite.
>
> Regards
>
> Tim
>
> On Thu, Jan 28, 2021 at 8:53 PM Nyall Dawson <nyall.dawson at gmail.com>
> wrote:
>
>> On Thu, 12 Nov 2020 at 23:59, Marco Bernasocchi <marco at qgis.org> wrote:
>> >
>> > thanks everyone for the all the information and feedback, it was really
>> helpful in preparing the budget proposal.
>>
>> Can I ask what the status on this request is? Travis has become
>> completely unworkable over the last week, with new jobs taking 10hrs+
>> to start, and then failing instantly because of the docker limits.
>> It's really slowing down the bug fixing efforts, and I fear that if it
>> degrades any further we'll be forced to just turn off Travis and as a
>> result have NO regression testing protecting the QGIS codebase! argh
>> :O
>>
>> Nyall
>>
>>
>>
>>
>> >
>> > cheers Marco
>> >
>> > On Mon, 9 Nov 2020, 10:40 Alessandro Pasotti, <apasotti at gmail.com>
>> wrote:
>> >>
>> >> Great, thanks!
>> >>
>> >> You've done a really good job there.
>> >>
>> >> Cheers
>> >>
>> >> On Mon, Nov 9, 2020 at 10:20 AM Denis Rouzaud <denis.rouzaud at gmail.com>
>> wrote:
>> >> >
>> >> >
>> >> >
>> >> > Le lun. 9 nov. 2020 à 09:06, Alessandro Pasotti <apasotti at gmail.com>
>> a écrit :
>> >> >>
>> >> >> On Mon, Nov 9, 2020 at 12:55 AM Nyall Dawson <
>> nyall.dawson at gmail.com> wrote:
>> >> >> >
>> >> >> > On Fri, 6 Nov 2020 at 20:57, Matthias Kuhn <matthias at opengis.ch>
>> wrote:
>> >> >> > >
>> >> >> > > This would be desirable.
>> >> >> > >
>> >> >> > > It's unfortunately a bit of complexity included. As Alessandro
>> said, the rendering tests are a good example of tests which are hard to
>> maintain and - unfortunately - not well defined functions like we happen to
>> have on e.g. database engine unit tests.
>> >> >> > >
>> >> >> > > To recall, when we started with CI some 5 years ago we were in
>> the situation that we had a set of tests, which randomly passed on some dev
>> machines and some on others. So effectively you had no chance to know if a
>> test doesn't pass because of your machine or your patch.
>> >> >> > >
>> >> >> > > At least now we have a reference platform and know if a patch
>> actually does cause changes. Which is the most important information.
>> >> >> > >
>> >> >> > > Currently many of the tests are already cross platform. And
>> nobody is actively writing CI centric tests. It's just a matter of fact
>> that the tests have a requirement to at least pass on the CI env because
>> it's the only thing that can be enforced.
>> >> >> > >
>> >> >> > > When tests are updated to a new platform (ci or not), many of
>> them will pass on a wider variety of platforms. Because bugs are fixed or
>> they are made more generic. And some will just have reference images for
>> one more platform added.
>> >> >> > >
>> >> >> > > So whatever we do we will do a step into the right direction.
>> But there will be no guarantee that every test passes on your machine
>> except if you make them all pass on your machine - which would be very
>> welcome!!
>> >> >> > >
>> >> >> > > What you could also do is listing tests that are generic /
>> platform independent (e.g. the expression tests would be a very good
>> example) and then always just run these instead of the whole set of tests.
>> >> >> >
>> >> >> >
>> >> >> > Just to add a few extra thoughts to the already comprehensive
>> replies
>> >> >> > given by Matthias and Denis:
>> >> >> >
>> >> >> > - we had to do a very similar effort with updating existing tests
>> when
>> >> >> > we moved from Qt 4 -> Qt 5. We'll probably have to do another
>> similar
>> >> >> > effort in another 18-24 months or so. It's going to be a regular,
>> >> >> > recurring task to go through and update all the tests which have
>> been
>> >> >> > introduced since the previous effort and ensure that they are
>> >> >> > sufficiently tolerant to pass under different
>> environments/software
>> >> >> > versions. (Maybe we should consider adding "test maintenance" as a
>> >> >> > regular yearly expense of the nature of 1.5 weeks?)
>> >> >> > - we really should do a similar effort to get the existing tests
>> >> >> > passing under mac os and windows too. I suspect there's some valid
>> >> >> > bugs that the test suites would reveal if we could reliably run
>> them
>> >> >> > under windows/mac, but the real issues are drowned in the noise of
>> >> >> > tests which haven't been designed to be cross platform compatible.
>> >> >> > (This could be a good grant proposal idea for future funding
>> rounds!)
>> >> >> >
>> >> >> > And then my personal 2c:
>> >> >> >
>> >> >> > We have a great test suite, and fantastic tools for making and
>> >> >> > managing tests. Sure, there's a learning curve involved with them.
>> >> >> > Sure, they ARE different to the test suites used by gdal, or
>> geos, or
>> >> >> > <insert other project here>. But that's just business as usual in
>> >> >> > software development, and not at all reflective of inferiority in
>> the
>> >> >> > test suites. Can we please move on from this recurring point once
>> and
>> >> >> > for all and focus on the current relevant parts of this discussion
>> >> >> > instead?
>> >> >> >
>> >> >> > Nyall
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> While I agree on all points, there is one important thing that we
>> >> >> should make it better while we work on it:
>> >> >> make sure the test suite can **easily** run locally using the same
>> >> >> docker images we are using in the CI process, I can do it but I
>> cannot
>> >> >> say it was easy to set up and it's probably overly complicated for
>> >> >> most people.
>> >> >
>> >> >
>> >> > The new github workflow is far more simple and easier to read than
>> the travis config.
>> >> > Everything is in one file and should be easily replicable.
>> >> >
>> https://github.com/qgis/QGIS/blob/test-focal/.github/workflows/run-tests.yml#L101-L124
>> >> >
>> >> >>
>> >> >> Also, making the test suite independent from the particular CI we
>> will
>> >> >> use (GH workflows, Travis & C.) will make it easier to move it to
>> >> >> another CI if needed.
>> >> >
>> >> >
>> >> > It should!
>> >> > Let's when we move the CI for 3.16 to Github: the base image will
>> remain (bionic) and we shouldn't have anything to touch in the tests.
>> >> >
>> >> >>
>> >> >>
>> >> >> This is of course also a matter of producing a good documentation
>> for
>> >> >> the process and the tools.
>> >> >>
>> >> >> Kind regards.
>> >> >>
>> >> >> --
>> >> >> Alessandro Pasotti
>> >> >> QCooperative:  www.qcooperative.net
>> >> >> ItOpen:   www.itopen.it
>> >> >> _______________________________________________
>> >> >> Qgis-psc mailing list
>> >> >> Qgis-psc at lists.osgeo.org
>> >> >> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>> >>
>> >>
>> >>
>> >> --
>> >> Alessandro Pasotti
>> >> QCooperative:  www.qcooperative.net
>> >> ItOpen:   www.itopen.it
>> >> _______________________________________________
>> >> Qgis-psc mailing list
>> >> Qgis-psc at lists.osgeo.org
>> >> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>> >
>> > _______________________________________________
>> > Qgis-psc mailing list
>> > Qgis-psc at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/qgis-psc
>> _______________________________________________
>> Qgis-psc mailing list
>> Qgis-psc at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/qgis-psc
>>
>
>
> --
>
> ------------------------------------------------------------------------------------------
> 
>
> Tim Sutton
> Visit http://kartoza.com to find out about open source:
>  * Desktop GIS programming services
>  * Geospatial web development
> * GIS Training
> * Consulting Services
> Tim is a member of the QGIS Project Steering Committee
>
> -------------------------------------------------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20210128/6e22daa5/attachment.html>