Late last night, a commit to the Mozilla trunk caused the tree to “go orange” (signaling that, while the build was successful, a test had failed). Upon inspection it was determined that one of the recently-integrated Prototype tests was failing.

This is particularly interesting because this is the first time that one of the foreign JavaScript library test suites has caught a possible regression. While the true nature of last night’s error is still being determined, I’m just pleased that it was able to do something.

However, during the course of that event, some outstanding issues, with how the test suites work, were uncovered. For instance, we determined that we need to be able to answer the following questions easily, and quickly, when an error comes up: Which test failed? and What does the test actually test?. Looking at these issues in depth:

Which test failed? In the case of the Prototype suite, this was actually quite challenging, if not impossible, without changing how the testrunner worked. Presently, the suite will print out any tests that fail, with their error message – but not print out any of the passed tests (giving you no context to know which particular test failed).

To alleviate this problem, I wrote up a quick patch for our copy of the test runner to add line numbers to the end of all the failure messages (giving the developers a rough way to figure out which test was causing problems). And then, today, I upstreamed the patch to Prototype, and after a quick chat with the team, it was committed right away. (This is my first patch to Prototype – it makes me quite happy to be able to contribute a little bit back to the great project.)

What does the test actually test? This is a much more challenging question. Frequently the developers who are reading the test case aren’t familiar with the code base (the exception probably being Rob Sayre and the MochiKit tests, since he’s a committer). So while there’s not much that can be done in this area, at the very least library authors should aspire to write simple, granular, test cases that can be easily digest and analyzed. Although, that’s typically a concern for most test suites as it is.

The next step is to patch up the Scriptaculous test suite (since it’s using the same testrunner as Prototype) and then do a survey of the other suites to see if their reporting is up-to-par.