Karma is a great test runner tool and has the capability to launch local browsers without installing multiple servers or web drivers like selenium does. Karma is a part of the AngularJS tool chain and a lot of websites use it to write unit test cases.
However, most development workflows stop at functional testing. Performance testing usually happens when a site or a component slows down. I thought it would be interesting to include a way to check for performance regressions as a part of this testing workflow. Hence Karma-telemetry.
Karma-telemetry is a karma plugin (like karma-qunit or karma-jasmine) that runs the Chromium Telemetry's smoothness and loading benchmarks for components, widgets and parts of a web page. With web components seeming to play a big part in Angular's future, this framework would also be a way to test how well each component performs and track if any perf regressions are introduced during development.
The idea was inspired by topcoat's benchmarking website. I had done a similar analysis for bootstrap and ReactJS components a few posts ago. With this now being a Karma plugin, it can be run in a NodeJS environment without having to download the entire Chromium Source code.
To add this to your website, follow the instructions in the Readme file.
Technical Details
Chromium telemetry is a python based tool that connects to a running Chrome instance using the remote debugging protocol to run various tests. The loading benchmark calculates various metrics like the time taken to render a page, DOM load time, etc. It looks at the window.performance.timing API and the Chrome timeline. The smoothness benchmark tries to calculate how janky the page scroll is, and the possible frames per second. Interestingly, both these benchmarks also give useful information in case of other browsers too.Though telemetry claims that it can run of other browsers, some of the tests use webkit specific APIs. As a part of karma-telemetry, I was able
- Extract telemetry into a separate node module
- Remove webkit specific code so that it runs across all browsers
- Report test results in a way karma can consume and possibly save in a graph for regression checks.
Inside Karma-telemetry
Karma telemetry is very similar to other karma plugins like karma-qunit and karma-jasmine. The only difference is that it does not have a Grunt build file to construct the final qunit-adapter.js from a qunit-wrapper. Instead, I baked the wrapper into the adapter which was much easier for testing. When loading, the index file loads karma in addition to a few more javascript file. These javascript files are responsible to performing the scroll and recording the metrics. The test metrics themselves are available as individual test results and reporters like karma-junit-reporter can be used to save the XML file. The reporter can also be integrated into continuous integration systems like Jenkins to and alters can be set if changes between commits are drastic.Additions to Karma
Karma-telemetry needed some additional features in Karma. I also sent in several pull requests to add these features to Karma and Karma launchers- Ability to run tests in a new window instead of the usual iFrame. This can be leverage not just by karma-telemetry but also by other frameworks that do not run inside an iFrame - pull
- Firefox Launcher - ability to add custom preferences when launching firefox- pull
- Sauce Launcher - pass custom options to the sauce launcher and disable popups - pull and pull
- Chrome - disable popup blocker by default - pull
Next Steps
I am working on building a node command line utility based on this that is based on selenium to get similar results. The idea is to use services like saucelabs to be able to run the performance tests on commits and generate graphs that would alert us on regressions.Checking for performance on every commit is hard and I think such tooling to integrate performance testing into continuous integration would help a lot of developers keep their sites fast.
Follow my blog's feed for my experiments on performance.
