Continuous Performance testing of Web Frameworks

Performance is a feature; a fast website has direct impact on revenue and user experience. Many of today's fast websites are built on top of front end frameworks (like React or Ember) and assume that they are also fundamentally fast. While many of these frameworks do place a lot of emphasis on performance, the per-commit regression tests that are run in continuous integration environments (like travis) only check for functional correctness.This blog post is about my experiments to set up a complementary continuous integration system for monitoring rendering performance regressions in some popular front end frameworks.

React, Ember, Ionic and Bootstrap are currently tested.

Key metrics like frame rates, paint times, etc are recorded and plotted in a graph. I run the perf-tests for every released version of the framework. Running it for each version can call out abrupt variations or gradual trends in the metrics and can help identify performance regressions. For example, here are some significant variations in the numbers and the potential reason for each case.

• Ember's frame rate graph shows improved frame rates for version 1.13.0. Ember's new rendering engine is fast as shown in this Ember with Glimmer video. The tests validate that with concrete numbers and indicate a 25% performance gain.
• When react moved from 0.5 to 0.8, there was a marked increase in performance as shown in the graph. I suspect there were due to the size reduction and upgrades in browserify and jstransform. The performance metrics have been more or less stable since then. Some good optimizations planned for 0.14, and I am hoping to see the graph show the differences too !!
• Testing all the Ionic components clearly shows that the web apps are much faster on Chrome on Android than a Cordova app, possibly due to the tax of using WebView.
• The tests for bootstrap shows how the project has evolved over time. It shows the cleanup that happened towards the 3.0 release and how the components matured towards the end of the 2.0 release.

I plan to continue running the tests and updating the graphs as more versions are released. Currently, I manually trigger the tests for each version and am exploring ways to automatically kick off the test, either by polling bower, or any other alternate mechanism. 

The test infrastructure is simple to set up. Each framework has canonical web app or a set of components rendered on a web page as the target for testing. These target apps are available on github for react, ionic, ember and bootstrap with instructions on how to set up and run the tests. The metrics were collected using browser-perf and stored in a couchDB database and drawn as a graph using perfjankie.
Though js-perf like test cases that test that test the speed of an event loop exist, there may be significant differences between the event loops and when the browser actually draws a frame.

Help needed - I hope to cover more frameworks and versions, something like todoMVC for performance. If you would like to add something similar for your web framework, ping me and I would love to help. If you have any changes to the canonical app to make it more like a real world app, or additional test cases, please do send in a pull request to the github repositories, or leave a comment.