Performance Comparison on Javascript Frameworks - DBMonster App

Tl;dr: Blog Post comparing performance of the dbmonster web app written in React, Ember, Underscore, Ractive and Paperclip.


Runtime performance is one of the key focus areas for most modern javascript frameworks. During a talk at the recent ReactConf 2015, a simple app simulating DB queries was used as a way to compare angular, ember and react. Many independent implementations of the test application have emerged since, each trying to showcase how fast the corresponding framework is.
Instead of comparing performance visually, I wanted to see if I could quantify the results and have a way to reproduce them. Note that there is a metric measuring memory, but rendering smoothness may not correlate directly to this metric.

Test Environment

The test suite compares the smoothness of dbmonster implementations in each of the framework. I ran the scroll/smoothness tests from telemetry using browser-perf to collect metrics like frame rates, layout times, nodes impacted per layout, etc.
Since some of the implementations in the original repo seemed slow, I picked up the fastest implementations for each of the frameworks that I could find. I was able to compare the apps written for react, ember, underscore/backbone, ractive and paperclip.
To run the tests,
  1. Download Chromedriver and start it.
  2. Clone the gist - git clone
  3. Install all dependencies with npm - $ npm install browser-perf
  4. Run $ node index.js to start the test for all urls. 
  5. Results of all runs will be available in data.json. 
Metrics for each framework were averaged over 10 runs. 4 such averages were collected for each framework to remove anomalies and to ensure that the metrics were similar in relation to each other. Note that I have not looked under the hood for each of the implementations.

Test Results

Here are some of the interesting facts from the tests.
  • Frames Per Second (calculated using RequestAnimationFrame) - Paperclip was the smoothest, followed by ractive, underscore/backbone, ember and finally react.
  • Frames Per Second (using about:tracing benchmarking) - This metric is a little different and is much closer to actual screen frames. I was not able to collect the data for ractive or underscore. This metric indicated that though ember did not do well on animation frames, it definitely had better times when drawing on the screen.
  • Average time spent painting on the page was highest for ember. React was much lower and the other three were even lower, with similar numbers.
  • Layouts - React's virtual DOM shines here, making it spend the minimum time in layout operations. Underscore seems to be the max, with ember and others in the middle.
  • Recalculating Styles - Underscore/Backbone spent the maximum time recalculating styles, while React's Virtual DOM clearly showed the reason for the lesser time. 
  • GC Events - Ember spent the maximum time collecting garbage, while paperclip was much better again. 
  • Nodes changed per layout cycle - Ember seems to change the maximum nodes, while React's Virtual DOM seems to show its work again here.
  • React was the only framework that seemed to emit events and parse HTML, the latter possibly due to JSX. The events may be the cause for lower frame rates in react despite it performing better at layouts and paints.
Here is the entire spreadsheet, showing averages from each test set, with interesting rows marked in green.

Next Steps

If you have trouble running these tests and see different results, please do ping me. If you are also aware of faster implementations or would like to try this on your framework, I would be glad to help.
Clearly, developers are not just optimizing to deliver content to the user fastest, but also working to ensure that the content enables a smooth experience. I also read about the glimmer implementation for ember and was hoping to work on a test suite that would measure the improvements in event incremental commit. I was also hoping to work a little more with React and Radium to profile for performance. 

OpenTweet - GopherGala Hackathon project

I participated in the online Gopher Gala hackathon with Jeff last week, and we came up with an interesting project. The idea was to create an open tweeting platform/protocol. As Jeff explains it, this is like email, but it pulls tweets instead of pushing mails. The tweets can reside on any server and a user, identified by [email protected] can be followed by any client, using the standard, simple protocol.
For the fun of it, we made the protocol on top of TCP sockets, instead of JSON over websocket pull, or other alternative mechanisms. Here is a demo that shows a ChromeApp and an iOS Cordova client interacting with a couple of servers that host the tweets.

Any user can host and control their own servers, and can use any mechanism to authenticate to the server and post tweets to it. For the demo, we used a simple RESTful API, but the real power of such a system would be the ability to do things like scheduling tweets, summarizing or aggregating tweets from other followers, etc. You don't even need an account to follow others!
The client is a ChromeApp with the UI using Ionic and Chrome Sockets. Being a Cordova commiter, I also converted it to a Cordova app using mobile-chrome-app that comes packaged with appropriate plugins to fill in for the Chrome API.
The source is on github, and if you like the idea, please vote for us :)

Scrolljank testing with WebPageTest

A node script to test scroll jank on WebPageTest - (link)

WebPageTest is a great way for testing the loading performance of web page. It provides information like the resource loading waterfall, and actionable metrics like speed index.

Recently, WebPageTest also added the ability to get diagnostic information like the Chrome developer timeline, tracing events and allowed the ability to run custom scripts. With these in place, I wanted to see if I could do scroll jank testing similar to what browser-perf does.

A scroll jank test loads a page on a browser, scrolls it vertically (or horizontally) and records the average frame rate (or the number of frames per second). The number roughly corresponds to how smooth a web page feels, and helps identify any event handlers that may be causing the jank. Chromium's Telemetry performance suite has a good scroll jank test implementation on which browser-perf is roughly based. For this exercise, I looked to reuse a lot of browser-perf code to get the measurement.

For a scroll jank test, we would typically need -

1. A way to scroll a web page

WebPageTest has a custom script parser that allows execution of custom javascript. I could simply inject the same javascript that I use in browser-perf  to make the page scroll. To make it simpler, we could place some restrictions on the scroll action.
  • It would be enough to run the scroll only on Chrome
  • The page would only scroll a fixed number of pixels.
  • The scroll needs to be as close to the real scroll - should generate the same events and follow similar page.
This basically narrows down the script to just use the function, and pass a couple of flags (--enable-gpu-benchmarking) to Chrome to make the function available. 

2. A way to record the frame rates

This is available from the about:tracing information. If enabled before the test, the ensure tracing information will be available as a downloadable file with a URL. To calculate frame rates from this, I simply re-used the parsing logic from browser-perf, which in turn is based on Chromium Telementry's logic.

Trying it on your pages

I ran the tests on a private instance. This was a Windows 7 VM running on a HyperV on my dev box. It had WebPageTest 2.15 and was running Apache 2.2 and wptdriver.exe. I had also set the timeout in the wptdriver.ini to 10 seconds, to try out many sites quickly.

The tests can be started from the Web UI., Fill up these fields in addition to the URL
  • Chrome Tab > In command line args, enter  --enable-gpu-benchmarking --enable-thread-composting
  • Chrome Tab > Check the box that says "Capture Chrome Trace (about://tracing) "
  • In the Script tab, Enter Script text box, use
You can also look at this gist, that uses the REST interface to interact with your instance of WebPageTest. Ensure that you have the required node_modules installed and the URL variables changed appropriately.

Caveats and Future ideas

This script is by no means perfect. There are multiple factors that may add inaccuracies to the results.
  1. The tracing information is capture not just while scrolling, but also when loading a page. If the page has components like auto-scroll, this would impact the frame rates. I am working on seeing if I could manually start and stop collecting traces (code).
  2. On, the script takes a really long time to run. I am trying to figure out how to stop the script slowly, using a permutation of execAndWait, waitForJS, etc. No luck yet :(
  3. Need to use the full scrolling script - it does things like check if the scroll is completed, allow horizontal scrolling, control the scroll speed, etc.
  4. The test script does not scroll certain pages at certain resolutions - no idea why - this needs investigation
  5. To run this across multiple browsers, the frame rates are calculated using requestAnimationFrame trick. I was not able to make "Custom Metrics" access a variable that I was setting in a Script tag. 

I would love to improve this scripts and find answers to the questions above. You can also try out browser-perf to run scroll tests on a selenium server - the mechanism is very similar and the wiki pages have more information.