[Qgis-developer] Is the new release schedule a success?

[Nyall Dawson]

On 1 July 2015 at 08:03, Jürgen E. <jef at norbit.de> wrote:
> Hi Sandro,
>
> On Tue, 30. Jun 2015 at 23:05:47 +0200, Sandro Mani wrote:
>> Just one curiosity: are unit tests run on all main platforms before
>> release?
>
> They are run every night - currently for different version of debian/ubuntu,
> windows and osx.
>
> But there virtually always have been a lot of tests (see dash[0]; starting at
> the HF in Lyon), that fail at least on some of the platforms (including only
> some of the debian/ubuntu version), so that's not a clear indicator for real
> bugs.  Hence the outcome has never been reliable enough to mean anything
> without taking a closer look at the results.
>
> Matthias fixed a bunch of the tests, when Travis was introduced.  But still
> disabled a couple of the flaky tests just in the travis configuraton and also
> added some new control images to make some others pass.

I've been meaning to post an update on this, so may as well do it now...

Following up Matthias' work on Travis, he and I (+some friendly
goblins) have been doing a lot of work to get the test suite behaving
well on other platforms. This has mostly consisted of making the test
suites more robust so they behave nicely cross platform, plus a small
amount of identifying and addressing real issues. We're getting
getting much closer, and comparing dash from the start of 2015 [1] to
yesterday [2] shows just how much progress has been made. (Today's
builds should look even better).

Here's where we currently stand:
- On linux platforms almost all of the tests should pass. The
exception is qgis_legendrenderertest, which requires a bit of tweaking
to handle different sized renderered/control images. There's some
(potentially real issues) failures on 2.11.0-wheezy-amd64, which I
still need to investigate and are probably related to dependency
packaging issues on that distro.

- Again, the tests should mostly pass on OSX, with the exception again
of the legend renderer and possibly the server tests (I haven't been
able to get a server build running yet on OSX to verify).
Unfortunately there seems to be major issues with dakcarto's nightly
builds which are causing additional failures - I'm going to chase this
up with Larry and see if he can fix the build environment to clear
these up. The unfortunate thing is that the OSX tests DID correctly
flag the regression addressed in cc13c57, but the failure was buried
in the noise of other failing tests.

Lastly.... Windows. We're getting stacks of failures on the nightly
Windows builds. I think there's two reasons for this:
1. A valid issue regarding font style names on Windows. Basically, the
test suite relies on a reference font "QGIS Vera Sans Roman". It looks
like there's an issue with font style names in Windows, where "Roman"
is incorrectly shown as "Normal", "Oblique" as "Italic", etc. You can
see this even with standard Windows fonts like Calibri - under Windows
all the style names are wrong, but they're fine under other platforms.
This potentially breaks compatibility of project files between Windows
and other platforms, and also causes all tests which rely on a font to
fail. I *think* this is an upstream issue - from my testing it looks
like we aren't the only Qt app affected by it.

2. Locale issues with the test suite. The Windows builds are done
using a locale which uses , instead of . for decimal places, and many
tests aren't expecting this. We could blindly force the locale for the
tests, but I'd rather address this so that the test suite can actually
test to see if there's underlying issues with the methods on different
locales.

So there we go:
Linux = all good
OSX = mostly good
Windows = a bunch of real issues to address

All told, I'm happy with the direction the test suite is going and the
shape it's in. When we finally get through this backlog of failing
tests it should be easy to keep an eye on dash and address regressions
as they occur. We'll even be ready when Travis gives us the option of
OSX/Windows automated builds.

Nyall


[1] http://dash.orfeo-toolbox.org/index.php?project=QGIS&date=2015-01-02
[2] http://dash.orfeo-toolbox.org/index.php?project=QGIS&date=2015-06-30