[Qgis-psc] QGIS grant report: Improve test result handling on QGIS CI

[Nyall Dawson]

On Mon, 27 Nov 2023, 4:56 pm Alessandro Pasotti, <apasotti at gmail.com> wrote:

> Hi Nyall,
>
> good news, thank you for this improvement!
>
> Just a quick question, in the linked PR:
> https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755
> comment the instructions say:
>
> "The full test report (included comparison of rendered vs expected
> images) can be found under the 'Checks' tab - 'QGIS tests',
> 'Artifacts' section as test-results-5."
>
> But I couldn't find any test-results-5 in the artifacts section, there are
> only:
>
> Artifacts
> build-22.04-qt5.tgz
> build-38-qt6.tgz
>

That's caused by a limitation in GitHub API. We can't retrieve the workflow
run id in an action, so that link will always just point to the most recent
workflow run for the PR.

And in this case I reverted the change causing a test failure, so the most
recent workflow doesn't have the failure report.

Hope that makes sense!

Nyall

>
> Cheers.
>
>
> On Mon, Nov 27, 2023 at 5:29 AM Nyall Dawson via QGIS-PSC
> <qgis-psc at lists.osgeo.org> wrote:
> >
> > Hi PSC,
> >
> > I'm happy to announce that this grant is now complete!
> >
> > While the original proposal was explicitly stated to be a research
> project with no guarantees of success, the end result is predominantly a
> success (with some limitations!)
> >
> > You can see the new failure handling in action in this PR:
> https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755
> >
> > What we have now is that any tests which fail a rendering comparison
> will write a descriptive comment to the PR, as shown in the above link. The
> comment details which render tests failed, where they are in the code, and
> includes some helpful pointers to downloading the full test report and the
> QGIS developer documentation.
> >
> > Originally, I hoped to link directly to the full test report or include
> it as an attachment to the comment. Unfortunately this is NOT possible
> given the current Github API. There's a bunch of notes I've added to the
> initial comment in https://github.com/qgis/QGIS/pull/54906 which link to
> the limitations / feature requests on Github's side, so we can monitor the
> situation and further improve the reports if/when Github add this
> functionality.
> >
> > As well as the above described improvements on the CI side, I've also
> implemented lots of improvements in running the tests locally and how the
> render test reports are generated and presented to developers!
> >
> > Thanks for making this possible!
> >
> > Nyall
> >
> >
> >
> >
> >
> > _______________________________________________
> > QGIS-PSC mailing list
> > QGIS-PSC at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/qgis-psc
>
>
>
> --
> Alessandro Pasotti
> QCooperative:  www.qcooperative.net
> ItOpen:   www.itopen.it
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20231127/7b8d6305/attachment.htm>