Scrolljank testing with WebPageTest

A node script to test scroll jank on WebPageTest - (link)

WebPageTest is a great way for testing the loading performance of web page. It provides information like the resource loading waterfall, and actionable metrics like speed index.

Recently, WebPageTest also added the ability to get diagnostic information like the Chrome developer timeline, tracing events and allowed the ability to run custom scripts. With these in place, I wanted to see if I could do scroll jank testing similar to what browser-perf does.

A scroll jank test loads a page on a browser, scrolls it vertically (or horizontally) and records the average frame rate (or the number of frames per second). The number roughly corresponds to how smooth a web page feels, and helps identify any event handlers that may be causing the jank. Chromium's Telemetry performance suite has a good scroll jank test implementation on which browser-perf is roughly based. For this exercise, I looked to reuse a lot of browser-perf code to get the measurement.

For a scroll jank test, we would typically need -

1. A way to scroll a web page

WebPageTest has a custom script parser that allows execution of custom javascript. I could simply inject the same javascript that I use in browser-perf  to make the page scroll. To make it simpler, we could place some restrictions on the scroll action.

• It would be enough to run the scroll only on Chrome
• The page would only scroll a fixed number of pixels.
• The scroll needs to be as close to the real scroll - should generate the same events and follow similar page.

This basically narrows down the script to just use the window.chrome.gpuBenchmarking.smoothScrollBy function, and pass a couple of flags (--enable-gpu-benchmarking) to Chrome to make the function available. 

2. A way to record the frame rates

This is available from the about:tracing information. If enabled before the test, the ensure tracing information will be available as a downloadable file with a URL. To calculate frame rates from this, I simply re-used the parsing logic from browser-perf, which in turn is based on Chromium Telementry's logic.

Trying it on your pages

I ran the tests on a private instance. This was a Windows 7 VM running on a HyperV on my dev box. It had WebPageTest 2.15 and was running Apache 2.2 and wptdriver.exe. I had also set the timeout in the wptdriver.ini to 10 seconds, to try out many sites quickly.

The tests can be started from the Web UI., Fill up these fields in addition to the URL

• Chrome Tab > In command line args, enter  --enable-gpu-benchmarking --enable-thread-composting
• Chrome Tab > Check the box that says "Capture Chrome Trace (about://tracing) "
• In the Script tab, Enter Script text box, use
  navigate http://nparashuram.com/perfslides exec window.chrome.gpuBenchmarking.smoothScrollBy(5000,function(){},0.5,0.5,chrome.gpuBenchmarking.DEFAULT_INPUT,"down",500);

You can also look at this gist, that uses the REST interface to interact with your instance of WebPageTest. Ensure that you have the required node_modules installed and the URL variables changed appropriately.

Caveats and Future ideas

This script is by no means perfect. There are multiple factors that may add inaccuracies to the results.

1. The tracing information is capture not just while scrolling, but also when loading a page. If the page has components like auto-scroll, this would impact the frame rates. I am working on seeing if I could manually start and stop collecting traces (code).
2. On Webpagetest.org, the script takes a really long time to run. I am trying to figure out how to stop the script slowly, using a permutation of execAndWait, waitForJS, etc. No luck yet :(
3. Need to use the full scrolling script - it does things like check if the scroll is completed, allow horizontal scrolling, control the scroll speed, etc.
4. The test script does not scroll certain pages at certain resolutions - no idea why - this needs investigation
5. To run this across multiple browsers, the frame rates are calculated using requestAnimationFrame trick. I was not able to make "Custom Metrics" access a variable that I was setting in a Script tag. 

I would love to improve this scripts and find answers to the questions above. You can also try out browser-perf to run scroll tests on a selenium server - the mechanism is very similar and the wiki pages have more information. 

Protractor-perf: Performance regression testing

Reuse Protractor E2E test cases to check for performance regressions using protractor-perf

Functional testing guards the app from code changes in the future that may break the app. Performance is also a feature and hence it is equally important to ensure that code changes do not slow down a web page.
Angular lays a lot of emphasis on testing. Protractor runs end-to-end test scenarios and verifies the correctness of apps. These end-to-end scenarios can also be used to track performance of Angular app. Protractor-perf is a tool that reuses protractor's test cases to record performance metric and ensure that all key performance indicators stay within the performance budget.

Protractor-perf is based on browser-perf and records website runtime rendering metrics like frame rates, layout times, expensive event handlers, etc.

Using protractor-perf is pretty simple. Ensure that protractor is setup and can run the end to end test cases. Then, add a little code in the test cases to indicate when to start and end performance measurements, and run $ protractor-perf conf.js instead of $ protractor conf.js. Assertions can then to be used to ensure that the performance metrics that are recorded stay within the expected range. Here is a example protractor test case, with the instrumentation statements added.

var ProtractorPerf = require('protractor-perf');
describe('angularjs homepage todo list', function() {
    var perf = new ProtractorPerf(protractor); // Initialize the perf runner
    it('should add a todo', function() {
        browser.get('http://www.angularjs.org');

        perf.start(); // Start measuring the metrics
        element(by.model('todoText')).sendKeys('write a protractor test');
        element(by.css('[value="add"]')).click();
        perf.stop(); // Stop measuring the metrics 

        if (perf.isEnabled) { // Is perf measuring enabled ?
            // Check for perf regressions, just like you check for functional regressions
            expect(perf.getStats('meanFrameTime')).toBeLessThan(60); 
        };

        var todoList = element.all(by.repeater('todo in todos'));
        expect(todoList.count()).toEqual(3);
    });
});

First, protractor-perf is initialized using new ProtractorPerf(protractor). The global protractor reference is passed in. The statements perf.start() and perf.stop() indicate when to start and stop recording the performance metrics. Finally, perf.isEnabled is used to check if the test case was run using protractor-perf. If this test case is executed using regular protractor, the perf statements become no-ops. This way, you can still use the instrumented code for regular functional testing also. The perf.getStats is then used to ensure that the requirement performance metrics stay within the acceptable ranges.
A quick way to include performance tests for all scenarios would be to start the tests in a beforeEach function, and adding the asserts in a helper function that is called towards the end of each individual spec block.
If you have an angular app that is tested using protractor, give protractor-perf a spin. I would love to hear your experience and if the tool was a useful addition to your workflow. Ping me if you have any questions and I would be glad to help you out.

Cordova Apps - Rendering Performance

Try it out for your Cordova apps - setup instructions

When developing a mobile app, one of the concerns of using the Hybrid approach is performance. Achieving smooth experience like a native app is important for Hybrid apps and developer tools for Android and iOS have been helping to a great deal.

To try it out on your Cordova apps, look at these setup instructions.

Background: browser-perf

Browser-perf is a tool to measure performance for websites on browsers like Chrome and Safari. Apart from other things, it leverages the remote debugging protocol of the browsers to look at the various timeline events. It also leverages Selenium to replay typical user interactions. Combining these two, browser-perf gets performance metrics for websites during typical usage scenarios. The metrics include frames per second, time for layouts, style, paint, etc.

Demo

The demo below shows how the cordova apps are created, and how simple it is to hook them up to browser-perf to record the performance metrics. 

 

Running on your Cordova apps

You can test your existing Cordova apps on Android and iOS using the following steps. By default, "scroll-tests" are run, but if app's webview does not have a scroll, or you would like to test other interactions like clicks, swipes, etc, you can follow this tutorial.

Android (4.4)

1. Download and run Chromedriver
2. Ensure that the emulator is running, or the device is connected
3. Ensure that the Cordova app is already installed on the device/emulator
4. Copy this config file, and change values for androidActivity and androidPackage.
5. Run browser-perf --config-file=configfile.json

iOS

1. Install Appium using npm install -g appium and follow its setup instructions.
2. Note that the bundleId and app must be change. app must point to an ipa or a zip file's absolute location.
3. Run appium and start browser-perf with a config file like this.

How does it work - the details

The mobile environments are not vastly different, and just like the desktop browsers, we would need the two things

• A way to automate the cordova apps or mobile browsers
• A mechanism to record timeline and profiling data when the automation occurs.

Android and iOS are slightly different, but the fact that they have similar remote debugging protocols makes it a little easy.

Android

Chromedriver provides full support for the selenium JSON wire protocol for Chrome and Webviews based on Chrome for Android. The "performance" log type is also fully supported to get timeline events. The tricky part is to pick up the events that are similar to about:tracing on desktop chrome. This can be done using adb, and adb_trace shows how that can be done. In case of browser-perf, the relevant parts of the code were ported to work with Node. Thus, all required data for calculating metrics are now available.

iOS

There are many drivers to automate Safari and Hybrid apps on iOS. I found that appium works great, and is pretty simple to install. Appium also relies on the remote debug protocol to automate Safari or the webviews. The performance log was however not initially implemented.
I was able to hack around a little and my pull request to add this feature finally landed in [email protected]. Timeline metrics can now be collected just as easily on Safari and the webviews on iOS emulators and devices.

Next Steps

Now that the metrics are recorded, it would be great to record them every time a change is made to the cordova app. Using a tool like perfjankie, a graph (like this) can be plotted over time to see how the various metrics change as the source code is changed. Watch out this space for more of my experiments with performance.

Introducing Perfmonkey

Check how smooth your website scrolls - perfmonkey.com 

Perfmonkey.com is a service to monitor the rendering performance of web pages. Send a HTTP request every time the site is deployed, and Perfmonkey will run a bunch of rendering tests to ensure that your web pages never have performance regressions.
Perfmonkey.com is currently in private beta and we are currently adding folks to the service. Till we have a fully scalable infrastructure, I wanted to create a way for developers to run performance test for single web pages using browser-perf and get an idea of the kind of metrics that can be measured.
Hence, I built the page that can take any website, run the performance tests on it and show the performance report. The challenge was to build something quickly that can scale well. Since most projects use Github for source code and Travis-ci for continuous integration, I based this service on top of github and travis.

A runner for browser-perf

The first step was to create the code that can run browser-perf. This repository simply invokes browser-perf and runs the tests specified in a config.json file. Since this runs on the cloud, the selenium server was configured to point to Sauce Labs.
This public repository was also configured to run Travis builds for every commit or a pull request. This way, if someone wanted to run the tests on a site, they would just need to change the config file, and send a pull request.

Automating the trigger

A pull request triggers the test run, I wanted a simpler user interface. I created a simple web page that uses github's API to edit a file, and then send a pull request. The Github API for changing files and creating a pull request is also pretty simple, everything was done in the web page.
The only server that I needed to host was an OAuth proxy to authenticate the users. The oAuth tokens are not even saved on the server, they are just cached in a user's browser
Given that Github APIs support cross-origin requests, I could simply use AJAX for all the requests.

Publishing the results

Github also provides an API to check the build status. This can be combined with the Travis API to fetch the actual build job, and the build logs. Like Github, Travis also supports cross origin request. All I had to do was pick up the Travis logs and parse the output.

Sample test runs

Here are the results for the test run on a slow page with parallax effects, and the test on the same page made faster. 

Instead of returning to the Developer tools profiler every time, this tool can be added to your continuous integration system to monitor for performance regressions. You can also get pretty graphs that can indicate the commits of deploys that make the site slower. Sign up for the beta of perfmonkey.com and we could help you get started on monitoring rendering performance.

GPU Composited CSS and browser-perf

I was at the Velocity Conference 2014, Santa Clara, speaking about "Automating Website Performance Measuring and adding it to continuous integration". The slides are available here.

Just before my presentation, I had the chance to catch up with Ariya Hidayat and we spoke about his work on GPU Composited CSS.His article talks about how CSS computations can be offloaded to the GPU and how over-doing it would simply exhaust the GPU.

He suggested that it would be interesting to see how browser-perf worked on his Codepen examples. It is clear from the developer timline tools (as described in Ariya's article) that work is transferred to the GPU. The cooler part would be trying to see if picking up this information can be automated.

I wrote a quick script before the presentation to demonstrate this, but the projectors did not agree with my laptop, and I just had to show static slides. Here are the details of my experiments.

Experiment 1: Impact of number of color changing rectangles

This codepen page shows different number of rectangles on the screen, each changing color using keyframes. browser-perf recorded the metrics for 1, 10 and 100 boxes. Here is the comparison for each case

Metric	One	Ten	Hundred	Units
CompositeLayers	20.00	58.99	171.00	ms
CompositeLayers_count	90	113	136	count
Layers	7	17	106	count
Paint	0.00	2.72	277.99	ms
Paint_count	1	20	7332	count

As seen from the table, the number of layers and composite layers increases with the number of rectangles on the screen. The number of times paint is called and the time per paint also increases.

Experiment 2: Changing Color vs Changing Opacity

As seen from the experiments above, changing the color causes the GPU to redraw the texture. A simpler way to simulate the same effect would be to use two  rectangles and slowly show one while hiding the other. With their opacity changing over time, they approximately show the same effect.

Metric	Color Change	Opacity Change	Unit
CompositeLayers	33.99	10.00	ms
Paint_count	51	5	count
Layers	6	2	count

The same page was used for changing colors, while this codepen was used to try changing the opacity.
The numbers also confirm that the number of times paint happened is much lower when the opacity changes. Similarly, the number of layers and composite layers in case of changing the color is much higher.

To summarize, certain properties like background color, borders, shape, etc make the GPU redraw the rectangle and should be avoided when trying to achieve a smooth web page.

Here is the full gist with all the data and the code to re-run the experiments. Have interesting rendering performance examples and want to measure the metrics for them? Ping me and I would love to help you run browser-perf on your examples.

Perfslides - demo app to show that browser-perf works

Performance Graphs for a website generated using browserperf

I have been working on browser-perf for quite some time now. With browser-perf and perfjankie, front end performance monitoring can be easily added to any continuous integration system. These tools make a lot of interesting metrics available. I always wanted to a project that demonstrates how this information can be actionable and associated to code that can be fixed to improve performance.

Perfslides is a project to show how changes in code across individual commits can impact smoothness and jank of a web page. It also doubles up as a slide deck that was used at conferences where I spoke about these tools.
The project has a simple, long and scrollable webpage on which the performance tests are run. The perf branch has five commits which are compared. Each commit is isolated in functionality to demonstrate a problem with jank or how it was fixed. These code snippets are from real world projects.
A graph over these five commits clearly shows the relation between the changed code and their impact of performance.

The Commits
The five commits are as follows

1. The first commit is a very basic version of the site with minimal bootstrap styling. The site has unscaled images from the slides and the site. 
2. The second commit is a styled version of the page, with parts related to the presentation hidden. This translates to hiding many large pictures that were used for the slide deck. Other styles included aligning the text, resizing images, etc. 
3. The third commit mocks a feature that introduces performance regressions. This code has a scroll handler that tries to position a bookmark indicating the amount scrolled. Amongst other things, it also tries to save this information in a cookie to reload the page at the same position it was scrolled to. 
4. This commit fixes the performance regression by delegating all the expensive work to a requestAnimationFrame call, using CSS transforms and caching all jQuery elements.
5. The final commit shows the impact of third party Javascript to a file. In this case, social sharing buttons and a comments form was added to the site. 

Integrating performance tests was as simple as adding a grunt task from perfjankie. Running grunt perf starts up the web server, connects to local selenium, runs the tests and saves all the data to a local database. This data is also replicated to a cloudant server and shows the impact of each commit on the metric.

The Metrics
Some noticeable and interesting changes to the metrics are

• When adding a scroll handler, the mean frame time improves when work is delegated to requestAnimationFrame.
• The addition of CSS3 transforms in commit 4 is shown by the increase in Layers metric
• The average painted area is also different
• Javascript execution time and navigation metrics increase when third party code was added to the site.
• The main difference between commit 1 and 2 is resized images. Looking at the DecodeImage and ResizeImage clearly indicates this. 

Testing Cordova
The built folder was also copied to the www folder in a cordova project and similar performance tests were run. The results were very similar.

Using it for your sites
Checking the developer tools for performance regressions after every deploy is very hard. Using tools like browserperf and perfjankie make tracking regressions simple. You could simple add a grunt task as shown in this project, connect to hosted browsers on saucelabs and save the data on cloudant. Alternatively, you could also try out this entire setup by signing up for perfmonkey.com, and sending a CURL request every time a new version of the site is deployed. Perfmonkey.com is a hosted service that does all this for you.

Follow the projects on github or watch out this space for more updates.

Front End Ops Conf 2014

I was at the Front End Ops conference on April 24, 2014 at San Francisco and spoke about "Adding Rendering Metrics to Browser Performance".

Here is a video of the session

The slides for the session are available at http://nparashuram.com/perfslides

Browser-perf 0.1.0

Measure the rendering performance of your site with browser-perf

Browser-perf is

• a node based tool
• that measures various performance metrics 
• of a web page or a hybrid (cordova based) application 
• on browsers like IE, Chrome and Firefox
• when running real scenarios (like shopping cart checkout, or scrolling a web page)

Measuring the smoothness or responsiveness of a web page is hard without data to support it. Tools in modern browsers go a long way in helping web developers determine and fix runtime rendering and performance issues. Most of the analysis depends on rules and "wisdom" generated by the web development community over time.Browser-perf is a way to convert those rules into tools to constantly monitor such issues.
As an example, this article is a great checklist of things to watch out for, when trying to develop a smooth web page. It lists some of the most common issues that make a web page janky. However, referring back to these rules every time during a new deployment of the web site is hard. Browser-perf converts the data from a browser into numbers based on the checklist. Some of the metrics derived from the checklist include number of event handlers that take more than 1/60 seconds, expensive GC events during animations, average area of paints, number of nodes calculated during layouts, etc.

Inside Browser-perf

Browser-perf runs tests scenarios based on selenium and collects the data from sources like the chrome developer tools timeline, or about:tracing. It then maps this raw data into actionable information based on ideas derived from various performance checklists.
The test scenario can range from simple page scrolls, to a complex checkout workflow that the user defines. During this scenario, various data points like duration of frame painting, layout cycles, memory growth, etc are measured and reported.
Browser-perf is extensible and developers can add ways to collect data from more sources (like xperf in case of IE), and also generate various other types of metrics.

Using Browser-perf in your environment

Browser-perf is node based and can either be used from the command line, or included as a node module to be a part of the build or continuous integration process. It can be run against a local setup of selenium or simply against "cloudified" browsers. It can also be used for testing Cordova applications on Android 4.4. You can find a lot more information in the wiki pages of the project.
If you would like to try browser-perf out, please do let me know and I would love to see if I can help you out. 

Type of Metrics

Some of the metrics collected when this is run include

• mean time to render a frame
• Paint Times, and average nodes per layout cycle
• Memory growth rate
• Number of layers
• Aggregation of time for all events on Chrome time lines
• Event handlers that take more than 16ms (1/60 of a second)

An exhaustive and growing list of metrics is available here.

Next Steps

On my todo list, are the following items

1. Enable AngularJS end to end tests also measure performance. I am working on writing an adapter for Protractor. Please vote on this issue if you think this could help you.
2. Better documentation :) 
3. Integrate information that Windows Performance Analyzer provides for IE, to get richer data for IE
4. Get more information from about:tracing
5. Help a couple of sites integrate this into their build process to study how they could benefit from such information. Please contact me if you would like to try it out.My previous blog post for more details.

Watch this space for more updates, and my work on browser performance. 

Making Frontend Performance testing a part of Continuous Integration - PerfJankie

Node module for Browser Rendering performance testing- PerfJankie

Real developers ship, the rest of us just introduce bugs into production systems. That is why continuous integration exists; for catching that ever elusive semicolon or that missing comma. When shipping, developers talk about database performance, application scalability and even network optimization after functionality and stability. Front end performance is usually the last in the picture, but thanks to tools the initiative of YSlow (and then pagespeed), people do think about it. Though the end user notices browser rendering performance issues first on any deployment, it is usually handled, in a very ad hoc manner.
A quote on the internet says

During the old days of front end engineering, people may have attributed jank to a faulty, dirt ridden mouse scroll wheel, but with touch devices and polished screens, users have started noticing that the finger slides across the screen while the page is trying to catchup.

It is painfully easy to slow down a smooth feeling webpage in the battle to make it pretty. Would it not be great if browser performance regression testing can also become a part of continuous integration.

PerfJankie is an attempt towards that idea, a way to control how fast a developer slows down their page. Some also call it an attempt to get back at the awesome 'off-by-one-pixel' designers by 'guilting' about that extra color in the gradient, but this is not true :)

Technical Details

Perfjankie does two main things

Run Performance test cases

Chromium Telemetry offers a wealth of metrics for testing web pages. Converting it to node and running them on 'cloudified' Saucelabs or BrowserStack browsers just makes the deal sweeter. Perfjankie delegates this to another node module called browser-perf. The type of metrics calculated today include first paint time, mean time of a frame when scrolling the page, dropped frames, load time, etc.

Save and display those results

Storing and displaying the data is the boring but important part of the module. It uses CouchDB database (or hosted couchDB like cloudant (on Azure) or iriscouch) to store the results as browser-perf hands over. This raw data in itself is not interesting. The way to convert this data into actionable information is in form of a CouchApp that resides in same database. The app displays the metrics in a timeline of commits, singling out the commits that slowed down the web page. It also show how adding features may have gradually made the page heavy. Here is how the graph looks like.

Perfjankie deploys the couch app the first time the tests are run and can also check and update the app or the couchdb views if the node module is updated.
Note: Of Course the UI and usability of this dashboard is bad as there were no designers - it would be awesome if you could send in a pull request making the dashboard better. 

How do I use it?

Integrating this into your projects should be easy, given that this is a node module If you already use grunt, you can take a look at this pull request with a grunt task. If you do not like couchdb, you could directly use the browser-perf command line interface and store the data into your own database.
If you are still stuck, please contact me and I would be more than happy to help you.
For more information about my previous or future work with rendering performance, watch out for this space, or subscribe to my blog :)