A node script to test scroll jank on WebPageTest - (link)
WebPageTest is a great way for testing the loading performance of web page. It provides information like the resource loading waterfall, and actionable metrics like speed index.
Recently, WebPageTest also added the ability to get diagnostic information like the Chrome developer timeline, tracing events and allowed the ability to run custom scripts. With these in place, I wanted to see if I could do scroll jank testing similar to what browser-perf does.
A scroll jank test loads a page on a browser, scrolls it vertically (or horizontally) and records the average frame rate (or the number of frames per second). The number roughly corresponds to how smooth a web page feels, and helps identify any event handlers that may be causing the jank. Chromium's Telemetry performance suite has a good scroll jank test implementation on which browser-perf is roughly based. For this exercise, I looked to reuse a lot of browser-perf code to get the measurement.
For a scroll jank test, we would typically need -
1. A way to scroll a web page
WebPageTest has a custom script parser that allows execution of custom javascript. I could simply inject the same javascript that I use in browser-perf to make the page scroll. To make it simpler, we could place some restrictions on the scroll action.- It would be enough to run the scroll only on Chrome
- The page would only scroll a fixed number of pixels.
- The scroll needs to be as close to the real scroll - should generate the same events and follow similar page.
2. A way to record the frame rates
This is available from the about:tracing information. If enabled before the test, the ensure tracing information will be available as a downloadable file with a URL. To calculate frame rates from this, I simply re-used the parsing logic from browser-perf, which in turn is based on Chromium Telementry's logic.Trying it on your pages
I ran the tests on a private instance. This was a Windows 7 VM running on a HyperV on my dev box. It had WebPageTest 2.15 and was running Apache 2.2 and wptdriver.exe. I had also set the timeout in the wptdriver.ini to 10 seconds, to try out many sites quickly.The tests can be started from the Web UI., Fill up these fields in addition to the URL
- Chrome Tab > In command line args, enter --enable-gpu-benchmarking --enable-thread-composting
- Chrome Tab > Check the box that says "Capture Chrome Trace (about://tracing) "
- In the Script tab, Enter Script text box, use
Caveats and Future ideas
This script is by no means perfect. There are multiple factors that may add inaccuracies to the results.- The tracing information is capture not just while scrolling, but also when loading a page. If the page has components like auto-scroll, this would impact the frame rates. I am working on seeing if I could manually start and stop collecting traces (code).
- On Webpagetest.org, the script takes a really long time to run. I am trying to figure out how to stop the script slowly, using a permutation of execAndWait, waitForJS, etc. No luck yet :(
- Need to use the full scrolling script - it does things like check if the scroll is completed, allow horizontal scrolling, control the scroll speed, etc.
- The test script does not scroll certain pages at certain resolutions - no idea why - this needs investigation
- To run this across multiple browsers, the frame rates are calculated using requestAnimationFrame trick. I was not able to make "Custom Metrics" access a variable that I was setting in a Script tag.
I would love to improve this scripts and find answers to the questions above. You can also try out browser-perf to run scroll tests on a selenium server - the mechanism is very similar and the wiki pages have more information.
