Fuzz testing is incredibly cool. Essentially, you throw random data at a program, seeing how it responds. This has a broad range of applicability, especially so in the world of security (finding fringe cases that the program authors didn’t cover, leading to exploits).

For example, if you were testing a web application, you could generate random query strings of input to send to it, to see how the site might respond (possibly detecting an exploit or XSS attack of some sort).

Jesse Ruderman just introduced a fantastic new utility for probing JavaScript implementations called jsfunfuzz. Specifically, this utility generates various JavaScript statements then executes them – seeing what the result will be.

I’ve actually been following the progress of this utility for a while now – seeing it’s outstanding results pop up on various tickets that I’ve watched. At first, I didn’t know about the jsfunfuzz utility and just assumed that Jesse was a wizard of developing obscure ways to break JS engines. It wasn’t until that I found and looked into the tool that his various bug comments made much more sense.

The usefulness of jsfunfuzz falls into two distinct areas: Testing the security of a JavaScript implementation looking for errors capable of crashing a browser and testing the validity of a JavaScript implementation looking for syntax errors.

Security

jsfunfuzz has explicit knowledge of how JS is parsed and interpreted, so it’s able to make educated guesses at how to write code that will break a browser. Because of this, over 280 bugs have been found in Mozilla thus far, with about 2 dozen of them being exploitable.

This aspect of the utility has been making some significant news lately, when it was announced at the Blackhat conference. Tons of examples of the result of this work can be found in the jsfunfuzz ticket.

Here’s one statement, for example, that jsfunfuzz generated that was able to produce a security exploit in Mozilla:

js> this.x setter= new Function; this.watch('x', 
  function(){}); gc(); x = {};
before 9232, after 9232, break 01205000
Bus error

Language Implementation

The second positive side-effect of this utility, and the one that I’m particularly interested in, is that of testing a JavaScript language implementation for correctness.

Update: It should be noted that ‘correctness’ is only hap-hazardly verified through fuzz testing (in that, no specific output is tested or verified – it simply watches for crashes or decompilation errors). It’s this second group of tests (uneval-ing a code block then verifying its contents to be equivalent to the input) that are particularly handy for testing the basic syntactical validity of a JavaScript implementation – but it doesn’t go much beyond that.

For example, look at this recent (large) ticket where Brendan implemented a number of speed improvements to Spidermonkey. He asked Jesse to come in and run the fuzzer to see if it could find any loose-ends, and sure enough, plenty were to be found:

js> [].map(1 for (x in []))
Assertion failure: *pc == JSOP_NULL, at jsopcode.c:3862

—

js> (4).__lookupGetter__('w')  
Crash [@ obj_lookupGetter]

—

js> f = function() { new (delete y) }
function () {
    new delete y;
}
js> eval(uneval(f))
typein:2: SyntaxError: syntax error:
typein:2: (function () {new delete y;})
typein:2: ..................^

—

(The following script caused a crash [@ js_ValueToString].)

function c(gen){
  Iterator;    
  '' + gen;
  for (var i in gen());
}
function gen(){
  ({}).hasOwnProperty();
  yield;
}
c(gen);

Now, all of this is not for waste. What happens now is that for each of these issues that jsfunfuzz discovered (albeit in a patched version of the JS engine) can be converted into test cases, which will be re-run every time a new build of the browser is generated. This is absolutely fantastic.

It’s pretty incredible how a common security testing tool can become good at testing, and generating test cases for, a JavaScript language implementation. This is especially important as development of ECMAScript 4-compliant language implementations start to appear – as new ways to test and break them will need to be discovered; and fuzz testing looks to be a fantastic way to achieve that.