[OpenLayers-Dev] Automated Testing

[Eric Lemoine]

On Dec 19, 2007 3:34 PM, Christopher Schmidt <crschmidt at metacarta.com> wrote:
> On Wed, Dec 19, 2007 at 06:45:41AM +0100, Eric Lemoine wrote:
> > On Dec 18, 2007 8:44 PM, Paul Spencer <pspencer at dmsolutions.ca> wrote:
> > > Tim,
> > >
> > > I think that the non-functional modifications should exclude any
> > > significant change to the test framework (run-tests, auto-tests) -
> > > i.e. those would require a ticket and review, since we are relying on
> > > the test framework to tell when stuff is broken.  This would include
> > > the changes I made yesterday, which in hindsight I should have
> > > proposed and got some sort of review on.  It would not include the
> > > changes I made to the actual tests to accommodate Safari's CSS quirks
> > > though.
> > >
> > > I also think that the trivial modifications category could be a little
> > > more lax unless the 'obvious typos' includes minor changes that are
> > > not typos?
> > >
> > > That being said, I don't feel strongly about changing it ... but I
> > > don't think it captures the spirit of the change in process that Chris
> > > was aiming for.
> > >
> > > Paul
> >
> > Hello
> >
> > I agree with Paul that changes to the test framework should require
> > tickets. Likewise, I think changes to the build framework (python
> > files) should require tickets.
>
> Really? Why?
>
> I agree that all these things should be tested before commit, but I'm
> not sure I understand why they need tickets. Especially with the build
> system, which I've never seen anyone other than Erik or I modify (since
> Phil/Schuyler wrote it originally Oh-So-Many-Years-Ago).
>
> I can understand reluctance to have the core library modified without
> tickets, review, adequate testing, etc. -- lots of people use it, and if
> I screw something up, other people suffer. But if I screw up the tests,
> users don't suffer, only developers -- and developers have the ability
> to smack me back into line when I break things.

My thinking was the following: the test framework is a core thing, if
someone breaks it then we can end up with the situation where tests
can no longer be run. The build system is something that users rely
on.

> > Now I'm a bit concerned with "Run tests in at least on browser" before
> > commit. I would rather go with "Run tests in at least IE and FF".
>
> This means that I can't commit about 90% of the time, since I do almost
> all of my OpenLayers development at home, where I have no access to an
> IE machine. (The remaining 10% is when I come into work early or stay
> late to run things in IE.)
>
> Again, things which are likely to affect IE should clearly be tested in
> IE -- some of them even require manual testing in IE first, especially
> things like VML renderer changes. However, I think expecting developers
> to test in IE is expecting too much to actually see it happen on every
> commit.

Most of the time I don't write a patch and commit it the same day. I
write and test it on FF one day (at home), and test it on IE and mark
it as "Review" the day after.

>
> > Question: Must commiters make sure the patch is functional on every
> > browser supported by OpenLayers before commit? This seems too strict
> > to me. Maybe we should require that the commiter make sure the patch
> > is functional on at least FF and IE before commit. What do you think?
>
> Clearly testing on all browsers is not something that can seriously be
> expected. (This is the reason for starting work on the automated testing
> to begin with, after all.) My personal position is that requiring
> something be tested on IE and FF before commit -- unless there is
> something in the code that makes it more likely than usual to fail in IE
> -- is ardurous enough to slow down development.

I understand your concern with slowing down development. But I agree
with Tim that it's great to have a reliable trunk. I'd just like to
see it as reliable as it is today.

--
Eric