JavaScript Engine Speeds

Posted: September 17th, 2007

25 Comments (Show Comments)

1. Roberto Saccon (September 17, 2007 at 11:35 pm)

   that’s a great ! I have started to implement a JS compiler for the Erlang virtual machine (right now it just compiles/runs a very small subset of Ecma 262, and haven’t released the code yet). So if things go as planned (but they usually don’t) you can add another engine to your charts in a few weeks …

   regards
   Roberto

2. Peter Kasting (September 18, 2007 at 12:37 am)

   If I read your graphs right, it looks like on most tests you run, Spidermonkey and Tamarin are roughly on par (with a few notable exceptions), while Rhino and JSCore tend to lag well behind, with Rhino worse than JSCore. Is that a fair summary?

3. Mark Rowe (September 18, 2007 at 1:05 am)

   What versions of the various JavaScript engines are being tested here? Looking at the testkjs executable in the SVN repository you reference, it is linked against /Users/jresig/Code/WebKit/WebKitBuild/Release/JavaScriptCore.framework/Versions/A/JavaScriptCore, which clearly will not exist for anyone interested in reproducing your results. The dynamic linker will fall back to whatever version of JavaScriptCore.framework people happen to have on their system which would lead to wildly different results.

4. glandium (September 18, 2007 at 1:13 am)

   > There’s always a significant amount of overhead required to run JavaScript code seurely within a browser, thus the efficiency of that security layer will frequently become a deciding factor in the results.

   … and Firefox is pretty bad at it. And the more extensions you install, the slower it gets…

5. Maciej Stachowiak (September 18, 2007 at 1:25 am)

   Which is fastest seems to depend on which benchmark you look at. JSC seems to do pretty well on array and string operations (with a few exceptions), excellent on regexp operations, and not so good on things that are math-heavy (which most of the “real world” tests are.

6. Justin Meyer (September 18, 2007 at 1:31 am)

   Very interesting! I think that most ‘real world’ tests in javascript are string manipulations.

7. Boris (September 18, 2007 at 1:44 am)

   John, I assume you’ll be filing bugs on cases when Tamarin is slower than SpiderMonkey, right? Array.unshift comes to mind, as do most of the regexp tests, some of the String tests, the partial-sums test, the Fib test.

8. Paul (September 18, 2007 at 3:23 am)

   What -O level are you using with Rhino? With -O 9 (compiled) it’s usually pretty fast (once the jvm gets a chance to JIT).

9. Jonathan Watt (September 18, 2007 at 4:04 am)

   It’s probably not really the case, but the time taken by Spidermonkey seems to increase linearly, while the time taken by Tamarin seems to increase exponentially. Do you fancy running a larger number of iterations? :-)

10. John Resig (September 18, 2007 at 7:22 am)

    @Peter: It all depends on the test. I’m trying to build some better aggregate views of the data, but I’m wary of presenting false assumptions. I’ll figure something out and get it up online.

    @Mark: Ah, damn! Yeah, I built a copy of JavaScriptCore from the WebKit SVN just before I ran these tests, yesterday. Hmm… I’ll have to try and think of another strategy for distributing that binary. Any thoughts are appreciated.

    @Justin: Well, I’d argue that most real-world JavaScript is DOM traversing and manipulation, but removing that, I’d probably say object property access, looping, then string and array manipulation.

    @Boris: I could do that – just straight into bugzilla?

    @Paul: Here’s the options that I’m using for Rhino:
    java -server -jar dep/js.jar -opt 9

    You can see this in the test suite’s Makefile:
    http://ejohn.org/code/js-speed/Makefile

    @Jonathan: Possibly – some of these tests already take a significant amount of time to run – especially the array-related ones. I’ll have to see what’s best.

11. Frankie Robertson (September 18, 2007 at 9:54 am)

    >> and explodes in IE (it requires canvas support)
    http://sourceforge.net/projects/iecanvas

12. John Resig (September 18, 2007 at 10:25 am)

    @Frankie – Yep! I’m lazy and haven’t gotten around to it yet, but thanks!

13. steve (September 18, 2007 at 10:26 am)

    Is there a CLI version/access to the JScript engine (or whatever IE/Trident uses?)

    I would be interested to know, again, at a core level, how their JS engine compares.

14. Asd (September 18, 2007 at 11:20 am)

    Wow. I had thought Rhino was fairly fast.

15. Maciej Stachowiak (September 18, 2007 at 11:26 am)

    John, would you be interested in contribution of more tests for this benchmark? I have a huge pile of JavaScript benchmarks from the web and I think some of them cover interesting things that aren’t covered by these tests (which nontheless are pretty awesome).

    Also, would you be interested in contribution of a harness for browser-based testing? I don’t think there will be a big difference in relative performance from the raw engine numbers, but it would allow fair comparisons to the JS engines in IE, Opera and other browsers.

16. John Resig (September 18, 2007 at 11:49 am)

    @steve: I’m not sure offhand, that’s my area of expertise, but I suspect that there must be something (or, at least, a way to build a simple CLI to JSript). Anyone with the knowhow is welcome to pick this up!

    @Asd: Don’t get it wrong – Rhino is quite speedy, it’s just that these other engines are speedier. :-)

    @Maciej: That’d be fantastic! I’ll drop you an email off-site.

17. Mark Rowe (September 18, 2007 at 12:00 pm)

    John, you should be able to “fix” testkjs by including the built JavaScriptCore.framework along side it in the dep directory and then running the following command to update the linker search path for the framework:

    install_name_tool -change /Users/jresig/Code/WebKit/WebKitBuild/Release/JavaScriptCore.framework/Versions/A/JavaScriptCore @executable_path/JavaScriptCore.framework/Versions/A/JavaScriptCore ./testkjs

    An easy way to verify that this has worked would be to attempt to run testkjs on a Mac OS X machine without the Safari 3 beta installed. Prior to this command, or if the command does not do what is intended, testkjs will refuse to run with an error from dyld.

18. John Resig (September 18, 2007 at 12:16 pm)

    @Mark: Excellent – thanks Mark! I’ll hop on that and get that committed right away.

19. David Golightly (September 18, 2007 at 12:18 pm)

    John – those tests are interesting and quite useful, though there were some surprises: Tamarin’s regex engine appears to be unusually slow! However, I’d have two requests for the tool itself (when you get the chance):

    1. Allow bookmarks to particular views of the tool, so I can link to eg. the graph for Array constructors.

    2. Have some sort of way to get at the specific value of each bar in the graph, whether it’s by mouseover or what have you. It’s somewhat frustrating to look at a graph like that for “Array Construction, new Array()” and not be able to have any idea about the values for the rest of the graph because one engine (in this case, Rhino) is such an outlier.

    Anyway, I know these features are both somewhat complicated to build, but I thought I’d get that in there for the record.

20. Boris (September 18, 2007 at 12:45 pm)

    Yeah, just filing these issues directly in Bugzilla on the Tamarin component is the way to go. Please cc me and Brendan when you do.

21. B C (September 18, 2007 at 4:12 pm)

    John,

    Sun’s JVM’s use a period of profiling code-use before they perform optimizations. The -server mode you used is intended for long-running server applications and sacrifices initial speeds for greater eventual improvements after a long time. (I read once that the server VM needs 10000 executions before it will optimize a given section of code).

    Did you take these issues into account when timing Rhino?

22. John Resig (September 18, 2007 at 11:54 pm)

    @BC: I haven’t seen the speed costs that you mention – the server flag seemed to help, when I tried it. If you’re interested, you could run the tests to see if that flag really is a significant factor. Let me know if it is!

23. db48x (September 19, 2007 at 4:45 am)

    This is really cool John. I especially like the way you show the js source of the tests. However, it seems the js shown is not always complete or correct. For example, the code for the n-body test ends with “for (i = 0; i”.

24. drbobb (September 20, 2007 at 3:58 pm)

    In Rhino, i just noticed that String.replace becomes slow (and I mean _pathetically_ slow) when the string you’re performing the replacements on is large(ish). I tried s = s.replace(sFrom, sTo, 'g'); where s.length was 964949, sFrom.length == 3, sTo.length == 4 — and it turned out that var a = s.split(sFrom); s = a.join(sTo); was faster by a factor of over 1600 (!!). There is clearly some sort of anomaly here.
    Btw. s = ( new java.lang.String(s) ).replace( sFrom, sTo ); did reasonably well, but still took about 70 percent more time than using Array.split. (There were about 9000 instances of s.From in the original string s).

    The reason that made me test this was that in one script, I had tried to perform a replace on a string of length about 3MB. Several _hours_ later it still hadn’t completed (and was hogging the cpu 100%), so I killed the jvm and was left wondering whether there might be some infinite loop bug there. But no, it is just that ridiculously slow.

25. Jason Orendorff (November 7, 2007 at 8:35 am)

    John,

    cscript //E:jscript filename.js

    (Actually, it ought to work even without the //E flag; cscript.exe should recognize the .js extension. But google tells me there are applications that clobber the relevant registry entries, so explicitly saying //E:jscript is safer.)

Comments are closed.
Comments are automatically turned off two weeks after the original post. If you have a question concerning the content of this post, please feel free to contact me.