Scrolljank testing with WebPageTest

A node script to test scroll jank on WebPageTest - (link)

WebPageTest is a great way for testing the loading performance of web page. It provides information like the resource loading waterfall, and actionable metrics like speed index.

Recently, WebPageTest also added the ability to get diagnostic information like the Chrome developer timeline, tracing events and allowed the ability to run custom scripts. With these in place, I wanted to see if I could do scroll jank testing similar to what browser-perf does.

A scroll jank test loads a page on a browser, scrolls it vertically (or horizontally) and records the average frame rate (or the number of frames per second). The number roughly corresponds to how smooth a web page feels, and helps identify any event handlers that may be causing the jank. Chromium's Telemetry performance suite has a good scroll jank test implementation on which browser-perf is roughly based. For this exercise, I looked to reuse a lot of browser-perf code to get the measurement.

For a scroll jank test, we would typically need -

1. A way to scroll a web page

WebPageTest has a custom script parser that allows execution of custom javascript. I could simply inject the same javascript that I use in browser-perf  to make the page scroll. To make it simpler, we could place some restrictions on the scroll action.
  • It would be enough to run the scroll only on Chrome
  • The page would only scroll a fixed number of pixels.
  • The scroll needs to be as close to the real scroll - should generate the same events and follow similar page.
This basically narrows down the script to just use the window.chrome.gpuBenchmarking.smoothScrollBy function, and pass a couple of flags (--enable-gpu-benchmarking) to Chrome to make the function available. 

2. A way to record the frame rates

This is available from the about:tracing information. If enabled before the test, the ensure tracing information will be available as a downloadable file with a URL. To calculate frame rates from this, I simply re-used the parsing logic from browser-perf, which in turn is based on Chromium Telementry's logic.

Trying it on your pages

I ran the tests on a private instance. This was a Windows 7 VM running on a HyperV on my dev box. It had WebPageTest 2.15 and was running Apache 2.2 and wptdriver.exe. I had also set the timeout in the wptdriver.ini to 10 seconds, to try out many sites quickly.

The tests can be started from the Web UI., Fill up these fields in addition to the URL
  • Chrome Tab > In command line args, enter  --enable-gpu-benchmarking --enable-thread-composting
  • Chrome Tab > Check the box that says "Capture Chrome Trace (about://tracing) "
  • In the Script tab, Enter Script text box, use
You can also look at this gist, that uses the REST interface to interact with your instance of WebPageTest. Ensure that you have the required node_modules installed and the URL variables changed appropriately.

Caveats and Future ideas

This script is by no means perfect. There are multiple factors that may add inaccuracies to the results.
  1. The tracing information is capture not just while scrolling, but also when loading a page. If the page has components like auto-scroll, this would impact the frame rates. I am working on seeing if I could manually start and stop collecting traces (code).
  2. On Webpagetest.org, the script takes a really long time to run. I am trying to figure out how to stop the script slowly, using a permutation of execAndWait, waitForJS, etc. No luck yet :(
  3. Need to use the full scrolling script - it does things like check if the scroll is completed, allow horizontal scrolling, control the scroll speed, etc.
  4. The test script does not scroll certain pages at certain resolutions - no idea why - this needs investigation
  5. To run this across multiple browsers, the frame rates are calculated using requestAnimationFrame trick. I was not able to make "Custom Metrics" access a variable that I was setting in a Script tag. 

I would love to improve this scripts and find answers to the questions above. You can also try out browser-perf to run scroll tests on a selenium server - the mechanism is very similar and the wiki pages have more information. 

Protractor-perf: Performance regression testing

Reuse Protractor E2E test cases to check for performance regressions using protractor-perf


Functional testing guards the app from code changes in the future that may break the app. Performance is also a feature and hence it is equally important to ensure that code changes do not slow down a web page.
Angular lays a lot of emphasis on testing. Protractor runs end-to-end test scenarios and verifies the correctness of apps. These end-to-end scenarios can also be used to track performance of Angular app. Protractor-perf is a tool that reuses protractor's test cases to record performance metric and ensure that all key performance indicators stay within the performance budget.

Protractor-perf is based on browser-perf and records website runtime rendering metrics like frame rates, layout times, expensive event handlers, etc.

Using protractor-perf is pretty simple. Ensure that protractor is setup and can run the end to end test cases. Then, add a little code in the test cases to indicate when to start and end performance measurements, and run $ protractor-perf conf.js instead of $ protractor conf.js. Assertions can then to be used to ensure that the performance metrics that are recorded stay within the expected range. Here is a example protractor test case, with the instrumentation statements added.


var ProtractorPerf = require('protractor-perf');
describe('angularjs homepage todo list', function() {
    var perf = new ProtractorPerf(protractor); // Initialize the perf runner
    it('should add a todo', function() {
        browser.get('http://www.angularjs.org');

        perf.start(); // Start measuring the metrics
        element(by.model('todoText')).sendKeys('write a protractor test');
        element(by.css('[value="add"]')).click();
        perf.stop(); // Stop measuring the metrics 

        if (perf.isEnabled) { // Is perf measuring enabled ?
            // Check for perf regressions, just like you check for functional regressions
            expect(perf.getStats('meanFrameTime')).toBeLessThan(60); 
        };

        var todoList = element.all(by.repeater('todo in todos'));
        expect(todoList.count()).toEqual(3);
    });
});

First, protractor-perf is initialized using new ProtractorPerf(protractor). The global protractor reference is passed in. The statements perf.start() and perf.stop() indicate when to start and stop recording the performance metrics. Finally, perf.isEnabled is used to check if the test case was run using protractor-perf. If this test case is executed using regular protractor, the perf statements become no-ops. This way, you can still use the instrumented code for regular functional testing also. The perf.getStats is then used to ensure that the requirement performance metrics stay within the acceptable ranges.
A quick way to include performance tests for all scenarios would be to start the tests in a beforeEach function, and adding the asserts in a helper function that is called towards the end of each individual spec block.
If you have an angular app that is tested using protractor, give protractor-perf a spin. I would love to hear your experience and if the tool was a useful addition to your workflow. Ping me if you have any questions and I would be glad to help you out.

Cordova Apps - Rendering Performance

Try it out for your Cordova apps - setup instructions

When developing a mobile app, one of the concerns of using the Hybrid approach is performance. Achieving smooth experience like a native app is important for Hybrid apps and developer tools for Android and iOS have been helping to a great deal.

To try it out on your Cordova apps, look at these setup instructions.

Background: browser-perf

Browser-perf is a tool to measure performance for websites on browsers like Chrome and Safari. Apart from other things, it leverages the remote debugging protocol of the browsers to look at the various timeline events. It also leverages Selenium to replay typical user interactions. Combining these two, browser-perf gets performance metrics for websites during typical usage scenarios. The metrics include frames per second, time for layouts, style, paint, etc.

Demo

The demo below shows how the cordova apps are created, and how simple it is to hook them up to browser-perf to record the performance metrics. 

 

Running on your Cordova apps

You can test your existing Cordova apps on Android and iOS using the following steps. By default, "scroll-tests" are run, but if app's webview does not have a scroll, or you would like to test other interactions like clicks, swipes, etc, you can follow this tutorial.

Android (4.4)

  1. Download and run Chromedriver
  2. Ensure that the emulator is running, or the device is connected
  3. Ensure that the Cordova app is already installed on the device/emulator
  4. Copy this config file, and change values for androidActivity and androidPackage.
  5. Run browser-perf --config-file=configfile.json

iOS

  1. Install Appium using npm install -g appium and follow its setup instructions.
  2. Note that the bundleId and app must be change. app must point to an ipa or a zip file's absolute location.
  3. Run appium and start browser-perf with a config file like this.

How does it work - the details

The mobile environments are not vastly different, and just like the desktop browsers, we would need the two things
  • A way to automate the cordova apps or mobile browsers
  • A mechanism to record timeline and profiling data when the automation occurs.
Android and iOS are slightly different, but the fact that they have similar remote debugging protocols makes it a little easy.

Android

Chromedriver provides full support for the selenium JSON wire protocol for Chrome and Webviews based on Chrome for Android. The "performance" log type is also fully supported to get timeline events. The tricky part is to pick up the events that are similar to about:tracing on desktop chrome. This can be done using adb, and adb_trace shows how that can be done. In case of browser-perf, the relevant parts of the code were ported to work with Node. Thus, all required data for calculating metrics are now available.

iOS

There are many drivers to automate Safari and Hybrid apps on iOS. I found that appium works great, and is pretty simple to install. Appium also relies on the remote debug protocol to automate Safari or the webviews. The performance log was however not initially implemented.
I was able to hack around a little and my pull request to add this feature finally landed in [email protected] Timeline metrics can now be collected just as easily on Safari and the webviews on iOS emulators and devices.

Next Steps

Now that the metrics are recorded, it would be great to record them every time a change is made to the cordova app. Using a tool like perfjankie, a graph (like this) can be plotted over time to see how the various metrics change as the source code is changed. Watch out this space for more of my experiments with performance.