[OpenLayers-Dev] manual acceptance tests

[Erik Uzureau]

On 11/9/07, Tim Schaub <tschaub at openplans.org> wrote:
> Hey-
>
> Erik Uzureau wrote:
> > This sounds like a brilliant idea to me and I am heavily in favour.
>
> Cool.
>
> >
> > It seems like just adding one directory might be a little short though
> > maybe we could do something like:
> >
> > openlayers/trunk/tests/acceptance
> > and
> > openlayers/trunk/tests/unit (for the current file-by-file tests?)
>
> I agree more structure would make things a bit nicer.  However, being
> the semantic cop, our Test.AnotherWay stuff is not really all unit
> tests.  There are plenty of acceptance tests in there.
>
> Maybe we could divide them up like this
>
> trunk/tests/automated
> trunk/tests/automated/index.html [the current run-tests.html]
> ...
>
> trunk/tests/manual
> trunk/tests/manual/index.html [list with links to test pages]
> ...
>
> Anybody opposed?  I'm happy to set this up.

Yes, it's true that our current tests are a right mish-mash of unit
and "acceptance"
 tests. I think it would be a worthwhile excercise to go through and
sort them out.

Perhaps it would be worthwhile to flush out the terminology. Here is
what I currently
have in my head:

manual -> html files like the ones in examples/ directory that user
has to open and possibly click around. to verify the test passes, user
has to actually *look* at the output.

unit -> test file (right now test.anotherway) that goes through and
tests each function of a file independently. Ideally, there should be
a one-to-one matchup between functions and tests.

automatic -> any test that requires no user observation to determine
if it passes or not. Unit tests qualify under "automatic"

acceptance -> ?? What do we consider acceptance tests? at this point,
it seems to me like maybe these would be all tests which are automatic
but not unit? Maybe acceptance tests can be further divided into
"manual" and "automatic" ?


Maybe y'all can clear some of that up or voice your own definitions.

What I would really like is three categories:

1) automatic unit tests

2) automatic non-unit tests: These tests would be
testing the interactivity of the openlayers library, doing more complex
things like creating map objects and adding layers and perhaps even
panning around, doing transoformations on data sets, etc etc.

3) manual "acceptance" tests -- these are the ones that I believe
you're proposing, tim.


So 1&2 could be run automatically in a post-commit hook, and 3 could
be reserved for (as you say) release time and when you make a patch
that you think might change something but you're not sure.

Please, fire away with feedback
e




It would be a chore, to be sure, but I think we could do it. I might
have some time
not next week but the following.

The trick is we'd have to do it without making things more complicated for
people.





> >
> > Also, I think its probably a good time to start looking into a
> > post-commit hook (only on trunk, of course) that runs the unit tests.
> > I don't personally have the first clue as to how to do that but
> > someone out there must? I think it would be a helpful addition to our
> > process and (realistically) wouldn't hinder the dev process hardly at
> > all.
>
> Unfortunately, since we rely on testing in multiple browsers, this isn't
> all that simple.  I know there are libraries to control IE with Python.
>   Don't know about Firefox.  Since our test output is huge and ugly
> HTML, I wouldn't want to be the one that had to scrape that and make
> sense of it.  I think to make this reasonable, we'd have to change test
> frameworks (to something that had an embedded browser) - but who wants
> to pay for that?
>
> Tim
>
> >
> > e
> >
> >
> >
> > On 11/9/07, Tim Schaub <tschaub at openplans.org> wrote:
> >> Hey-
> >>
> >> Our automated tests are a mix of acceptance and unit tests.  These are
> >> nice (because they are automated), but we need more (manual) acceptance
> >> tests to fully test things like the renderers.
> >>
> >> Instead of scattering manual acceptance tests in the examples directory,
> >> I'd like to have an "acceptance" directory in our "tests" directory (or
> >> "manual" if that makes more sense to folks).  This would make it clear
> >> for developers what acceptance tests needed to be run manually, and
> >> would make it less confusing to users who poke around the examples
> >> directory and stumble into the weird examples that are actually tests
> >> (speaking for myself here).
> >>
> >> Anyone opposed?  Or got a better idea?
> >>
> >> Barring feedback, I think I'll make such a change on Monday.  (This will
> >> only involve adding a directory to the tests directory and starting to
> >> compile regular manual acceptance tests there.)
> >>
> >> Tim
> >> _______________________________________________
> >> Dev mailing list
> >> Dev at openlayers.org
> >> http://openlayers.org/mailman/listinfo/dev
> >>
> >
> > !DSPAM:4033,4734c31d13012090977483!
> >
>
> _______________________________________________
> Dev mailing list
> Dev at openlayers.org
> http://openlayers.org/mailman/listinfo/dev
>