[QGIS-Developer] QGIS grant report: Improve test result handling on QGIS CI

[Nyall Dawson]

Hi PSC,

I'm happy to announce that this grant is now complete!

While the original proposal was explicitly stated to be a research project
with no guarantees of success, the end result is predominantly a success
(with some limitations!)

You can see the new failure handling in action in this PR:
https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755

What we have now is that any tests which fail a rendering comparison will
write a descriptive comment to the PR, as shown in the above link. The
comment details which render tests failed, where they are in the code, and
includes some helpful pointers to downloading the full test report and the
QGIS developer documentation.

Originally, I hoped to link directly to the full test report or include it
as an attachment to the comment. Unfortunately this is NOT possible given
the current Github API. There's a bunch of notes I've added to the initial
comment in https://github.com/qgis/QGIS/pull/54906 which link to the
limitations / feature requests on Github's side, so we can monitor the
situation and further improve the reports if/when Github add this
functionality.

As well as the above described improvements on the CI side, I've also
implemented lots of improvements in running the tests locally and how the
render test reports are generated and presented to developers!

Thanks for making this possible!

Nyall
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20231127/f90aeaef/attachment.htm>