[Qgis-psc] QGIS grant report: Improve test result handling on QGIS CI

[Alessandro Pasotti]

Hi Nyall,

good news, thank you for this improvement!

Just a quick question, in the linked PR:
https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755
comment the instructions say:

"The full test report (included comparison of rendered vs expected
images) can be found under the 'Checks' tab - 'QGIS tests',
'Artifacts' section as test-results-5."

But I couldn't find any test-results-5 in the artifacts section, there are only:

Artifacts
build-22.04-qt5.tgz
build-38-qt6.tgz

Cheers.


On Mon, Nov 27, 2023 at 5:29 AM Nyall Dawson via QGIS-PSC
<qgis-psc at lists.osgeo.org> wrote:
>
> Hi PSC,
>
> I'm happy to announce that this grant is now complete!
>
> While the original proposal was explicitly stated to be a research project with no guarantees of success, the end result is predominantly a success (with some limitations!)
>
> You can see the new failure handling in action in this PR: https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755
>
> What we have now is that any tests which fail a rendering comparison will write a descriptive comment to the PR, as shown in the above link. The comment details which render tests failed, where they are in the code, and includes some helpful pointers to downloading the full test report and the QGIS developer documentation.
>
> Originally, I hoped to link directly to the full test report or include it as an attachment to the comment. Unfortunately this is NOT possible given the current Github API. There's a bunch of notes I've added to the initial comment in https://github.com/qgis/QGIS/pull/54906 which link to the limitations / feature requests on Github's side, so we can monitor the situation and further improve the reports if/when Github add this functionality.
>
> As well as the above described improvements on the CI side, I've also implemented lots of improvements in running the tests locally and how the render test reports are generated and presented to developers!
>
> Thanks for making this possible!
>
> Nyall
>
>
>
>
>
> _______________________________________________
> QGIS-PSC mailing list
> QGIS-PSC at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/qgis-psc



-- 
Alessandro Pasotti
QCooperative:  www.qcooperative.net
ItOpen:   www.itopen.it