[OpenLayers-Dev] Automated Testing

[Tim Schaub]

I like the idea (and if we were making a policy I would suggest) that a 
patch creator does three things:

1) make a ticket (more documentation than commit messages)
2) run at least the tests you know are relevant in at least one browser
3) report what you did on the ticket

Are you suggesting less?

Tim


Christopher Schmidt wrote:
> On Mon, Dec 17, 2007 at 01:29:50PM -0700, Tim Schaub wrote:
>> Hey-
>>
>> Christopher Schmidt wrote:
>>> Semi-Automated testing is now here.
>> This is great Chris.  I do have some thoughts that I'm hoping I can get 
>> your feedback on (with regard to rapid development).
>>
>> I think this process is pretty rapid:
>> 1) I make a change in a working copy.
>> 2) I run tests locally.
> 
> On what browser?
> 
> Everyone can run them on Firefox pretty easily. Some people can run them
> on Safari easily, some people can run them on IE easily, some people can
> run them on Opera easily.
> 
> Most people can't run them on all of: 
>  * FF3 (Win/Mac/Linux)
>  * FF2 (Win/Mac/Linux)
>  * Opera (Win/Mac/Linux)
>  * IE6 (Win)
>  * IE7 (Win)
>  * Safari (Win, Mac)
> 
> Additionally, as you pointed out, we don't yet have a mechanism for
> testing singlefile builds. I think that this should change, and part of
> the testing framework is going to need to be to set up the single file
> builds to run automatically as well.
> 
> I think that anything that takes over 5 minutes for reviewing whether 
> a patch changes existing behavior is unlikely to be routinely performed
> by the author of a patch. Even if it is routinely performed, it's still
> possible to screw up -- more than once, I've thought I was testing
> something in IE, onl yto find out I as only getting cached test results
> that weren't affected at all by the patch.
> 
> Fixing this situation is likely to help lazy developers find the errors
> of their ways more quickly. If 10 minutes after committing, I get an
> email saying "IE is throwing errors everywhere", I'm likely to go back
> and either roll my change out, or (if it's obvious) file a new ticket
> with a patch, and wait until someone can review it and commit it. 
> 
> Currently, for me, it takes 20 minutes to run tests. They can't be run
> simultaneously in IE and FF, because they fail due to timeouts, and they
> each take about 10 minutes to run, depending on how long the browser has
> been open. For a small change -- 5-10 lines -- I'm unlikely to let all
> the tests run before deciding it's okay. And in that case, the onus
> hsould still be on me to fix it if I break something. Automated tests
> let a computer tell me I broke something before a human finds out
> (hopefully). 
> 
>> I like the idea of automated testing.  But I don't know what I'd do if I 
>> got an email that a bunch of tests were failing.  Probably just get 
>> frustrated that someone committed a change without running tests.
> 
> And then, hopefully, go yell at them to fix them. And if that continues,
> then they get commit/review rights limited/revoked.  
> 
>> Basically, I think the most rapid development course is one where the 
>> person making a change confirms that tests pass - and if they don't, 
>> that person fixes the issue.
> 
> And I think it's too ardurous to expect that this will actually happen
> in all cases. It *should*, but asking either a patcher or a reviewer to
> build multiple single file builds to make sure something works correctly, 
> or to run tests for 20 minutes just to determine that three lines of
> code work correctly, is likely as not to be ignored. 
> 
>> Again, I appreciate the effort you are putting in to this.  I just want 
>> to make sure that this doesn't take the place of committers testing 
>> their own changes.
> 
> To me, it only affects the *extent* to which committers are expected to
> test their own changes: specifically, the variety of platforms against
> which committers should feel required to test before committing. I
> always run tests in FF, and I typically try to do manual testing in
> Safari, but 90% of the time, unless I see something either 1. big or 2.
> seemingly likely to break IE or only affect IE, I don't typically let it
> sit and wait for an IE-holding developer to shepard it through. The
> feeling I have at the moment is that this is a bad thing to do as a
> committer, and I'm working to make up for my deficiencies by ensuring
> that when I screw up, I can find out about it.
> 
> Regards,