[OpenLayers-Dev] Automated Testing

[Christopher Schmidt]

On Mon, Dec 17, 2007 at 01:29:50PM -0700, Tim Schaub wrote:
> Hey-
> 
> Christopher Schmidt wrote:
> > Semi-Automated testing is now here.
> 
> This is great Chris.  I do have some thoughts that I'm hoping I can get 
> your feedback on (with regard to rapid development).
> 
> I think this process is pretty rapid:
> 1) I make a change in a working copy.
> 2) I run tests locally.

On what browser?

Everyone can run them on Firefox pretty easily. Some people can run them
on Safari easily, some people can run them on IE easily, some people can
run them on Opera easily.

Most people can't run them on all of: 
 * FF3 (Win/Mac/Linux)
 * FF2 (Win/Mac/Linux)
 * Opera (Win/Mac/Linux)
 * IE6 (Win)
 * IE7 (Win)
 * Safari (Win, Mac)

Additionally, as you pointed out, we don't yet have a mechanism for
testing singlefile builds. I think that this should change, and part of
the testing framework is going to need to be to set up the single file
builds to run automatically as well.

I think that anything that takes over 5 minutes for reviewing whether 
a patch changes existing behavior is unlikely to be routinely performed
by the author of a patch. Even if it is routinely performed, it's still
possible to screw up -- more than once, I've thought I was testing
something in IE, onl yto find out I as only getting cached test results
that weren't affected at all by the patch.

Fixing this situation is likely to help lazy developers find the errors
of their ways more quickly. If 10 minutes after committing, I get an
email saying "IE is throwing errors everywhere", I'm likely to go back
and either roll my change out, or (if it's obvious) file a new ticket
with a patch, and wait until someone can review it and commit it. 

Currently, for me, it takes 20 minutes to run tests. They can't be run
simultaneously in IE and FF, because they fail due to timeouts, and they
each take about 10 minutes to run, depending on how long the browser has
been open. For a small change -- 5-10 lines -- I'm unlikely to let all
the tests run before deciding it's okay. And in that case, the onus
hsould still be on me to fix it if I break something. Automated tests
let a computer tell me I broke something before a human finds out
(hopefully). 

> I like the idea of automated testing.  But I don't know what I'd do if I 
> got an email that a bunch of tests were failing.  Probably just get 
> frustrated that someone committed a change without running tests.

And then, hopefully, go yell at them to fix them. And if that continues,
then they get commit/review rights limited/revoked.  

> Basically, I think the most rapid development course is one where the 
> person making a change confirms that tests pass - and if they don't, 
> that person fixes the issue.

And I think it's too ardurous to expect that this will actually happen
in all cases. It *should*, but asking either a patcher or a reviewer to
build multiple single file builds to make sure something works correctly, 
or to run tests for 20 minutes just to determine that three lines of
code work correctly, is likely as not to be ignored. 

> Again, I appreciate the effort you are putting in to this.  I just want 
> to make sure that this doesn't take the place of committers testing 
> their own changes.

To me, it only affects the *extent* to which committers are expected to
test their own changes: specifically, the variety of platforms against
which committers should feel required to test before committing. I
always run tests in FF, and I typically try to do manual testing in
Safari, but 90% of the time, unless I see something either 1. big or 2.
seemingly likely to break IE or only affect IE, I don't typically let it
sit and wait for an IE-holding developer to shepard it through. The
feeling I have at the moment is that this is a bad thing to do as a
committer, and I'm working to make up for my deficiencies by ensuring
that when I screw up, I can find out about it.

Regards,
-- 
Christopher Schmidt
MetaCarta