jQuery’s Test Suites
Take the case of the jQuery core testing environment. Our default test suite is an XHTML page (served with the HTML mimetype) with the correct doctype. In includes a number of tests that cover all aspects of the library (core functionality, DOM traversal, selctors, Ajax, etc.). We have a separate suite that tests offset positioning (integrating this into the main suite would be difficult, at best, since positioning is highly dependent upon the surrounding content). This means that we have, at minimum, two test suites straight out of the gate.
Next, we have a test suite that serves the regular XHTML test suite with the correct mimetype (application/xhtml+xml). We aren’t 100% passing this one yet, but we’d like to be able to sometime before jQuery 1.4 is ready. Additionally, we have another version that we’re working on that serves the regular test suite but with its doctype stripped (throwing it into quirks mode). This is another one that we would like to make sure we’re passing completely in time for 1.4.
Both of those tweaks (one with correct mimetype and one with no doctype) would also need to be done for the offset test suite. We’re now up to 6 test suites.
We have another version of the default jQuery test suite that runs with a copy of Prototype and Scriptaculous injected (to make sure that the external library doesn’t affect internal jQuery code). And another that does the same with Mootools. And another that does the same for old versions of jQuery. That’s three more test suites (up to 9).
Finally, we’re working on another version of the suite that manipulates the Object.prototype before running the test suite. This will help us to, eventually, be able to work in that hostile environment. This is another one that we’d like to have done in time for jQuery 1.4 – and brings our test suite total up to 10.
We’re in the initial planning stages of developing a pure-XUL test environment (to make sure jQuery works well in Firefox extensions). Eventually we’d like to look at other environments as well (such as in Rhino + Env.js, Rhino + HTMLUnit, and Adobe AIR). I won’t count these non-browser/HTML environments, for now.
At minimum that’s 10 separate test suites that we need to run for jQuery. Ideally, we should be running every one of them just prior to committing a change, just after committing a change, for every patch that’s waiting to be committed, and before a release goes out…
in every browser that we support.
The Browser Problem
At the time of this post that includes 12 browsers.
- Internet Explorer 6, 7, 8. (Not including 8 in 7 mode.)
- Firefox 2, 3, Nightly.
- Safari 3.2, 4.
- Opera 9.6, 10.
- Chrome 1, 2.
Of course, that’s just on Windows and doesn’t include OS X or Linux. For the sake of sanity in the jQuery project we generally only test on one platform – but ideally we should be testing Firefox, Safari, and Opera (the only multi-platform browsers) on all platforms.
Some test suites (such as Yahoo UI, jQuery UI, and Selenium) have ways of automating pieces of user interaction (you can write tests like ‘Click this button the click this other thing’). For most cases this works pretty well. However all of this is just an approximation of the actual interaction that a user may employ. Nothing beats having real people manually run through some easily-reproducible (and verifiable) tests by hand.
What currently exists?
The only way to tackle the above problem of scale is to have a massive number of machines dedicated to testing and to somehow automate the process of sending those machines test suites and retrieving their results.
There currently exists an Open Source tool related to this problem space: Selenium Grid. It’s able to send out tests to a number of machines and automatically retrieve the results – but there are a couple problems:
- It isn’t able to test against non-desktop machines. Each server must be running a daemon to handle the batches of jobs – this leaves mobile devices out of the picture.
- It can’t test against unknown browsers. Each browser needs special code to hook in to triggering the loading of the browser by Selenium, thus an unknown browser (such IE 8, Opera 10, Firefox Nightly, or Chrome) may not be able to run.
Naturally, this solution doesn’t tackle the problem of manual testing, either.
A solution: TestSwarm
All of this leads up to a new project that I’m working on: TestSwarm. It’s still a work in progress but I hope to open up an alpha test by the end of this month – feel free to sign up on the site if you’re interesting in participating.
All the test suites are collected. For example, 1 “commit” can have 10 test suites associated with it (and be distributed to a selection of browsers).
The nice thing about this construction is that it’s able to work in a fault-tolerant manner. Clients can come-and-go. At any given time there might be no Firefox 2s connected, at another time there could be thirty. The jobs are queued and divvied out as the load requires it. Additionally, the client is simple enough to be able to run on mobile devices (while being completely test framework agnostic).
The libraries can also push manual tests out to the users. A user will be notified when new manual tests arrive (maybe via an audible cue?) which they can then quickly run through.
All of this help from the users wouldn’t be for nothing, though: There’d be high score boards keeping track of the users who participate the most and libraries could award the top participants with prizes (t-shirts, mugs, books, etc.).
The framework developers get the benefit of near-instantaneous test feedback from a seemingly-unlimited number of machines and the users get prizes, recognition, and a sense of accomplishment.
If this interests you then please sign up for the alpha.