[Qgis-psc] QGIS grant report: Improve test result handling on QGIS CI

[Nyall Dawson]

On Thu, 4 Jan 2024, 5:23 am Sandro Santilli, <strk at kbt.io> wrote:

> On Mon, Nov 27, 2023 at 05:09:07PM +1000, Nyall Dawson via QGIS-PSC wrote:
> > On Mon, 27 Nov 2023, 4:56 pm Alessandro Pasotti, <apasotti at gmail.com>
> wrote:
> >
> > > https://github.com/qgis/QGIS/pull/55417#issuecomment-1826995755
> > > comment the instructions say:
> > >
> > > "The full test report (included comparison of rendered vs expected
> > > images) can be found under the 'Checks' tab - 'QGIS tests',
> > > 'Artifacts' section as test-results-5."
> > >
> > > But I couldn't find any test-results-5 in the artifacts section
> >
> > That's caused by a limitation in GitHub API.
>
> Would it be possible for that comment to contain instruction about how
> to run that test locally ? Helping developers running failing tests
> locally would reduce load on shared infrastructure.
>

Good idea, but unfortunately non trivial to execute ☹️

Obstacles are:

- We need the ctest name for the test so that associated ctest -R
"^testname\$" command can be determined to run just one test file.

- We don't have access to the ctest name for the current test being
executed during the test.  So we'd have to **hardcode** these into each
test individually. (There's unfortunately no fixed pattern used to specify
the ctest test name from the actual test source file name, so we can't use
qtest methods or macros which give us the source file name here)

- If we want to generate a command which will run just ONE function from
the test, then we hit against
https://gitlab.kitware.com/cmake/cmake/issues/20470 . (This could possibly
be worked around using a variation on https://stackoverflow.com/a/77084304)

So not impossible to do, but also a considerable amount of work. If someone
else takes this on I'm happy to review. 👍

Nyall
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20240104/b61bff99/attachment.htm>