[OpenLayers-Dev] Automated Testing

[Christopher Schmidt]

On Wed, Dec 19, 2007 at 06:45:41AM +0100, Eric Lemoine wrote:
> On Dec 18, 2007 8:44 PM, Paul Spencer <pspencer at dmsolutions.ca> wrote:
> > Tim,
> >
> > I think that the non-functional modifications should exclude any
> > significant change to the test framework (run-tests, auto-tests) -
> > i.e. those would require a ticket and review, since we are relying on
> > the test framework to tell when stuff is broken.  This would include
> > the changes I made yesterday, which in hindsight I should have
> > proposed and got some sort of review on.  It would not include the
> > changes I made to the actual tests to accommodate Safari's CSS quirks
> > though.
> >
> > I also think that the trivial modifications category could be a little
> > more lax unless the 'obvious typos' includes minor changes that are
> > not typos?
> >
> > That being said, I don't feel strongly about changing it ... but I
> > don't think it captures the spirit of the change in process that Chris
> > was aiming for.
> >
> > Paul
> 
> Hello
> 
> I agree with Paul that changes to the test framework should require
> tickets. Likewise, I think changes to the build framework (python
> files) should require tickets.

Really? Why?

I agree that all these things should be tested before commit, but I'm
not sure I understand why they need tickets. Especially with the build
system, which I've never seen anyone other than Erik or I modify (since
Phil/Schuyler wrote it originally Oh-So-Many-Years-Ago). 

I can understand reluctance to have the core library modified without 
tickets, review, adequate testing, etc. -- lots of people use it, and if
I screw something up, other people suffer. But if I screw up the tests,
users don't suffer, only developers -- and developers have the ability
to smack me back into line when I break things.  

> Now I'm a bit concerned with "Run tests in at least on browser" before
> commit. I would rather go with "Run tests in at least IE and FF".

This means that I can't commit about 90% of the time, since I do almost
all of my OpenLayers development at home, where I have no access to an
IE machine. (The remaining 10% is when I come into work early or stay
late to run things in IE.)

Again, things which are likely to affect IE should clearly be tested in
IE -- some of them even require manual testing in IE first, especially
things like VML renderer changes. However, I think expecting developers
to test in IE is expecting too much to actually see it happen on every
commit.

> Question: Must commiters make sure the patch is functional on every
> browser supported by OpenLayers before commit? This seems too strict
> to me. Maybe we should require that the commiter make sure the patch
> is functional on at least FF and IE before commit. What do you think?

Clearly testing on all browsers is not something that can seriously be
expected. (This is the reason for starting work on the automated testing 
to begin with, after all.) My personal position is that requiring
something be tested on IE and FF before commit -- unless there is
something in the code that makes it more likely than usual to fail in IE
-- is ardurous enough to slow down development. 

Regards,
-- 
Christopher Schmidt
MetaCarta