Update: The Acid3 test is now final, I’ve updated the blog post to reflect this.
The new news on the block is that the upcoming Acid 3 test is in the oven, starting to get baked. Traditionally, the Acid test has served as a way to get browser vendors in line by testing them on really-annoying edge cases. This can, sometimes, get people tied up in knots but it actually serves as a devious way of getting people to meet a large part of a spec.
For example, in order for a browser to have some weird padding/margin test case solved – in CSS – they must also have a working box model. So while an Acid test may not, explicitly, test for a working box model, it will be done implicitly (by testing edge cases that result from it).
With that in mind, it’s time to take a look at Acid 3 which primarily focuses on technology that I find to be interesting: ECMAScript and the DOM. Let’s dig in and see what exactly is being tested – specifically, relating to ECMAScript.
- Array Elisions – Making sure that stuff like
[,,]has a length of 2 and[0,,1]has a length of 3. - Array Methods – Doing an unshift with multiple arguments
.unshift(0, 1, 2), joining with an undefined argument.join(undefined). - Number Conversion – Banging against
.toFixed(),.toExponential(), and.toPrecision()– especially with decimals and negative numbers. - String Operations – Negative indicies in substr
.substr(-7, 3), character access by index"foo"[1](part of the ECMAScript 4 spec). - Date – Making sure that certain method calls result in NaN results (like
d.setMilliseconds(), with no arguments) and also enforcing +1900 year offsets. - Unicode in Identifiers – You can’t use escaped Unicode in identifiers, for example:
eval("test.i\\u002b= 1;");(that should throw an exception). - Regular Expressions –
/[]/matches an empty set,/[])]/should throw an exception, backreferences to non-existent captures, and negative lookaheads/(?!test)(test).exec("test test"). - Enumeration – Make sure that object properties are enumerated in the correct order, make sure that you’re able to enumerate properties of certain names (toString, hasOwnProperty, etc.).
- Function Constructors – The user should be able to set custom constructors on the
.constructorproperty,.constructorshould not be enumerable, and.prototype.constructorshould be deletable. - Function Expressions –
(function test(){ ... })();You should be able to call the function by name, within the function itself, you can’t directly overwrite the function name (only with a function-scoped variable), and ‘test’ isn’t leaked into the parent scope. - Exception Scope – Variables within the
catch(){}should interact with the catch arguments primarily, followed by variables in an outer scope. - Assignment Expressions –
s = a.length = "123";– a.length has a return value of 123 (the number) which is assigned to ‘s’, rather than the correct result of the string “123”. - Encoding –
encodeURI()andencodeURIComponent()must gracefully handle null bytes.
All-in-all it’s a comprehensive smattering of weird ECMAScript edge cases – you’re bound to find at least one that fails in your favorite browser-of-choice. I’m sure we’ll see many more test cases coming in, in the upcoming days in weeks.
I’m looking forward to seeing the final results – and the competitive heat that’s been applied to CSS-spec implementors being applied to ECMAScript implementors.
For kicks, here’s the current results in a bunch of major browsers (including the correct reference rendering).
These are the preliminary results of the UNCOMPLETED Acid 3 test in UNCOMPLETED versions of major browsers – take with a grain of salt. Go here to view the final version of this test.
Reference Rendering:
Firefox 2:
Firefox 3b2:
Safari 3:
WebKit Nightly:
Opera 9.5b1:
Internet Explorer 7: (thanks chunghe!)
Julian Calaby (January 11, 2008 at 12:30 am)
It turns out to be rather broken in IE7.
John Resig (January 11, 2008 at 12:39 am)
@Julian: Somehow not surprised – totally forgot to run it in IE7 – what results did you get?
chunghe (January 11, 2008 at 12:55 am)
http://www.flickr.com/photos/birdegg/2184960582/
IE7, by far the worst.
Gaurav (January 11, 2008 at 1:22 am)
The test has already changed. I now get 62/100 in Fx 3.0b2, and the last (largest) box is grey rather than silver. By the way, you can see a report of which tests failed by clicking on the “A” in “ACID3”.
John Resig (January 11, 2008 at 1:24 am)
@Gaurav: Yeah, the numbers are always moving for me, as well. So yeah, take all of this will a huge grain of salt – just look at the physical tests, for now.
Gaurav (January 11, 2008 at 1:51 am)
I had to save the page and play around with the CSS before I could see it, but IE7 is at 25/100. Since 16 tests are unimplemented, that really means it is passing 9 out of 84 tests.
matp75 (January 11, 2008 at 2:31 am)
looks like the test should be run from network
ie if you copy it to test, put it on a web server, not on local disk.
Chris Hubick (January 11, 2008 at 2:49 am)
I have no idea how, but on my x86_64 Fedora 8 machine using provided Firefox 2 with No-Script active, viewing this blog entry on Planet Mozilla causes my whole machine to spontaneously reboot! It did it three times in a row.
lrbabe (January 11, 2008 at 2:55 am)
And what is the result with ie6 ?
Mark G (January 11, 2008 at 3:33 am)
Submitted Acid 3 test url to browsershots.org and ran it through a few different browsers/operating systems, you can check the results here:
http://browsershots.org/http://hixie.ch/tests/evil/acid/003/NOT_READY_PLEASE_DO_NOT_USE.html
jgraham (January 11, 2008 at 5:09 am)
It’s worth noting that Hixie is looking for input on what should go into the last 16 tests. See Anne’s post for more details, but the basic criteria is that tests must only use features described by a spec that went CR (or non W3C-equivalent) in 2004 or earlier and must fail in one or more of Mozilla, Opera or Safari (latest available trunk).
Gaurav (January 11, 2008 at 6:34 am)
@matp75: Even if I did that, I’d not be sure I configured it correctly, I can see that content-types are very important. I only did it because I couldn’t see the results in a particular browser, which I consider a bug in the test.
Stéphane Loeuillet (January 11, 2008 at 7:20 am)
Got 42/100 with Access NetFront 3.4 on an embedded box.
I’ll try to figure if I can make a screenshot.
Milo (January 11, 2008 at 9:59 am)
“object properties are enumerated in the correct order”
Can you, or someone else here, explain where the “correct order” is documented in a specification? ECMA-262 section 12.6.4 states “The mechanics of enumerating the properties (…) is implementation dependent. The order of enumeration is defined by the object.” And I can’t find any JavaScript specification that defines an order on Object properties…
I’d ask Ian, but I’m sure he’s busy :)
Gaurav (January 11, 2008 at 10:47 am)
@Milo: I think those are in the DOM specs, not ECMAScript.
Milo (January 11, 2008 at 10:57 am)
@Gaurav: The test (test 91) doesn’t appear to be using any DOM-related functionality. It just creates a plain JavaScript Object and then enumerates over the properties, expecting to find them in a certain order.
John Resig (January 11, 2008 at 11:17 am)
@Milo: I’ve pinged Ian about that – considering that it isn’t in the ES3 spec, I’m not sure if this is the best place for it. Note that he also tests “kitten”[4] – which also isn’t in the ES3 spec. Granted, both of those things are nearly ubiquitously implemented, but I’m not sure if that’s a good precedent to follow.
Gaurav (January 11, 2008 at 11:34 am)
Silly me. I should have just downloaded and installed an add-in to view the “generated source” in IE7.
Gaurav (January 11, 2008 at 11:40 am)
@Milo, John: Test 91 may be different than before (because the test has been modified again), but right now it sorts the properties before checking them, so the properties did not need to be in any order. It is checking that the sort puts them in alphabetical order correctly. The error messages are misleading.
Gaurav (January 11, 2008 at 11:58 am)
Oops, sorry. Not alphabetical order. Order of value returned as defined in some crazy way I’ve never tried.
Milo (January 11, 2008 at 11:59 am)
@Gaurav: Thanks for pointing that out. (Now?) the sort appears to be ordering them in numeric order (sorting by the functions’ return values).
I’m just going to stop looking at it, and trust that in the end it won’t test something that isn’t actually specified anywhere, no matter how badly we’d like it to be.
John Resig (January 11, 2008 at 1:35 pm)
@Irbabe: You can find IE6 results on Browsershots.org, courtesy of Mark G.
Garrett (January 11, 2008 at 3:19 pm)
@Gaurav
ES4 is attempting to standardize that properties are returned in the
order in which they’re defined, based on the premise that that is what
all impl’s do.
My general comments in response to Acid 3 draft follow.
Unit would offer some advantages:
1. easier to know specifically where a test fails (units, not %Acid)
2. Can track regressions of specific bugs (for impls)
3. more modular — tests and suites can be added, modified, et c.
The tests should serve as a live documentation. An Acid test doesn’t do that.
I’d like to see ECMAScript separated from ECMAScript DOM Bindings.
For example, instead of:
// DOM Range
function () {
instead,
function testDomRange() {
Then organize into suites, for example: http://www.w3.org/DOM/Test/
JSUnit could be better (not stop/pause, frames messes w/history, can’t see test list in suite loaded in UI).
Gaurav (January 11, 2008 at 4:08 pm)
@Garrett
Hey, I’m just an amateur web author who got interested. :) Any feedback really needs to be sent to Ian Hickson.
Besides, I think the ACID tests were never really about making any sort of useful or complete test suite for standards, but to educate developers, users, and particularly the media regarding how broken the web browsers are toward their implementation of the standards. IE7 showing that mess makes the point very spectacularly, don’t you think? I hope you can make a much better, more complete test suite, for actual implementers and testers.
Ian Hickson (January 11, 2008 at 5:39 pm)
I’ve removed the string indexing test. I thought that was in ES3 for some reason.
Test 91 with the properties has always had the .sort(), by the way, I don’t know why you guys thought it was checking object enumeration. :-)
Gaurav (January 11, 2008 at 7:32 pm)
@Ian
I guess we can give you feedback without emailing you. :) I think we thought that because the title comment is “check that properties are enumerable by default” and the error strings are like “failed to find […] in expected position”.
Gaurav (January 11, 2008 at 7:55 pm)
Hehe… Firefox won’t pass if AdBlock Plus is installed.
Shmuma (January 13, 2008 at 3:34 am)
Konqueror (KDE 3.5.8) crashes at test start.
jmdesp (January 13, 2008 at 9:40 am)
@Garrett : The trouble is that all you describe is very nicely done by the tests in http://www.w3.org/DOM/Test/ , but nobody is really interested in the results they give. I opened bmo bug 296391 about getting Gecko based browser to pass them, but nobody volunteered to do any work on that (there’s also bug 125665 about the DOM1 part of those tests). The current status is even that the latest Fx nightlies won’t run the test anymore, they stop after a few test and you get no result (I guess it broke after some security change).
Acid test are all about public relation, they’re made to shine and attract everyone attention, but really that’s the only way to get developers to fix their bugs, even the mozilla.org one.
So long live Acid if it can make things move on that front, after the CSS one.
Garrett (January 13, 2008 at 1:52 pm)
jmdesp –
JSUnit kind of well, it could be better. I think if:
it had lists of individual tests in the suite (sort of like how selenium does), that would be better.
it loaded the tests that were listed in the main document into an iframe, rather than using a frameset, then the tests would show up in search results when developers go searching, just like PPK’s site does.
Search results and linked tests will generate buzz, and also specific argument and examples (though probably not as much hype).
Ian – I didn’t assume that test 91 was not testing Sort Compare, I didn’t read enough of the test (including test 91). I was just responding to Guarav’s and others’ replies.
Guillaume L. (January 14, 2008 at 12:42 pm)
any browser .. ever render the page correctly?
Boris (January 14, 2008 at 12:53 pm)
jmdesp, it doesn’t help that the W3C DOM tests are sometimes buggy… The DOM Working Group got shut down with errata still pending. So there are things that all UAs implement in the same way, and that way differs from the spec, so they would fail the test.
Amir (January 15, 2008 at 2:57 am)
Firefox 2 – why so bad?
Robin (January 15, 2008 at 9:54 am)
Bugs? :) Firefox while very good is by no means perfect. Besides, each of these tests has to currently fails in either Safari or Firefox before it’ll make it in to the test. The point is to force all UA manufacturers to pull their socks up, even if in IE’s case it’s more like a pair of tights.
Dean (January 18, 2008 at 3:25 pm)
Interesting nobody tested it in the only browser that actually passed the Acid 2 test – Safari 2.
I’d like to see how it performs in that browser.
Joe Hamilton (January 18, 2008 at 4:48 pm)
I tried the test in IE6 and it looks exactly the same as IE7. I thought IE7 was supposed to be an improvement!
If none of the browsers work, how do you know what the image is supposed to look like? And if you do have a browser that renders it correctly, can I have it?
Daniel (January 19, 2008 at 1:24 pm)
@Amir: Firefox 3 includes many improvments like reflow refactoring (which lead it to passing Acid2), Firefox 2 on the other hand is using an aging engine version ;)
I wonder why Safari handles the test so badly? Are the rumors true and they’re just passing Acid 2 because they made it pass?
Anyway: IE7 got 24/100 and my standalone version of IE6 got 23/100 and also looks more weird.
Sadly we’re not going to see how good IE8 is in the near future.
Gaurav (January 22, 2008 at 9:37 pm)
@Joe: IE7’s improvements concentrated on CSS and basically ignored ECMAScript/DOM. I think I remember Ian writing a Slashdot comment about how he created the reference “image” (it is actually a CSS rendering). Basically, he worked it out without the benefit of an ideal standards compliant browser. However, at this point the test isn’t finished and might have bugs, so we really don’t know at this point if he got it right. But I think his resume speaks for itself: he probably understands the standards better than any of us, given how many of them he was involved in writing.
Daniel (January 31, 2008 at 2:38 pm)
It seems the test is “released” now. The NOT_READY_XYZ.html site is gone.
John Resig (January 31, 2008 at 4:51 pm)
@Daniel: Thanks, I’ve updated the page to point to the final version of the test.
MySchizoBuddy (February 11, 2008 at 10:37 pm)
Webkit nightly as of r30109 has made some improvements
http://farm3.static.flickr.com/2061/2259746316_e90dfbb2f5_o.png
Solipsism (February 12, 2008 at 1:33 am)
WebKit e30123 is now up to 81/100.
Ed (February 13, 2008 at 4:50 pm)
WebKit r30191 is at 82/100
Stebs (February 19, 2008 at 11:25 am)
Firefox 3.0 (Minefield/3.0b4pre: 2008021904) is now at 64/100
Ed (February 27, 2008 at 8:13 am)
WebKit r30595 is now at 86/100
Nathan (March 6, 2008 at 12:50 pm)
“Interesting nobody tested it in the only browser that actually passed the Acid 2 test – Safari 2.”
Actually… the *first* browser, but not the only: Acid 2 Timeline
I got pretty different results for IE6: IE 6 Acid3
Monica (March 7, 2008 at 3:05 pm)
I get 12/100 for IE6 too and 50/100 for Firefox 2.0.
noraa (March 23, 2008 at 7:24 pm)
latest webkit (as of March 22) is getting a 95/100. The blue and red boxes are showing up grey, and the animation is not entirely smooth.