[OpenLayers-Dev] Automated Testing

[Tim Schaub]

Hey-

Christopher Schmidt wrote:
> Semi-Automated testing is now here.

This is great Chris.  I do have some thoughts that I'm hoping I can get 
your feedback on (with regard to rapid development).

I think this process is pretty rapid:
1) I make a change in a working copy.
2) I run tests locally.
3) Since I know what code I changed, if a test fails, it generally 
doesn't take that long for me to find the problem.

This is the slowest and most tedious process I can imagine:
1) Someone makes a change on their working copy.
2) They commit.
3) I update, make a change, run tests and get confused by failing tests.
4) I spend time trying to figure out who did what.
5) bla bla bla

I like the idea of automated testing.  But I don't know what I'd do if I 
got an email that a bunch of tests were failing.  Probably just get 
frustrated that someone committed a change without running tests.

Basically, I think the most rapid development course is one where the 
person making a change confirms that tests pass - and if they don't, 
that person fixes the issue.

Again, I appreciate the effort you are putting in to this.  I just want 
to make sure that this doesn't take the place of committers testing 
their own changes.

Tim

> 
> To throw yourself into the automated testing pool, simply go to:
> 
> http://openlayers.org/dev/tests/auto-tests.html?run=all
> 
> in IE or FF. (Safari doesn't work at all, and Opera has a number of
> failures which I haven't yet debugged.) 
> 
> This page will automatically reload every 20 minutes and run again.
> (Okay, this is perhaps a bit much, given our current commit rate, but I
> haven't yet got it to the point where it can check if the code has
> changed and run it again... hopefully I'll get there at some point.)
> You can leave your browser open doing this, and it should act as an
> 'automatic' testing instance. 
> 
> 
> The results are posted to http://openlayers.org/test/results.cgi , with
> failures getting a seperate HTML page with the total test text, and an
> email being sent to the new autotest list: 
> 
> http://openlayers.org/mailman/listinfo/autotest
> 
> I'm going to work on setting up an IE machine at work to run the tests
> -- I haven't actually tested with IE yet -- which would let us get
> reports if we commit anything which causes the tests to fail in IE, and
> hoepfully get one set up the same in FF.
> 
> Next on my list is getting a couple of minor tests running against
> single file builds -- the primary thing with those being just to test
> that they compiled correctly, rather than that they're providing the
> full support of the library. That way, when I commit something that
> breaks the build, I can take responsibility for it and fix it.  
> 
> And then, of course, there's the need to fix the tests in Opera and
> Safari, so we can get testing going on those as well.
> 
> I'm working hard to figure out how we can make OpenLayers more resilient
> against a slightly less weighty commit process, so that we can open up
> development more. For the past several months, the code has not changed
> much, and I think a big reason is just the fact that we don't have the
> infrastructure in place to make rapid development tenable. I'm hopeful
> we can fix this through the use of technology, and accelerate the pace
> of OpenLayers development in the process.
> 
> If any of this seems like a wrong path to anyone, please feel free to
> let me know, and I can back all this out -- there's really not much to
> it, but having 'broken the build' several times this weekend, I felt
> like I needed to make up for it, and this is my first attempt.
> 
> Regards,