React Native Packager vs Haul

What packages faster ? Haul or React Native ? Jump to results

A few days ago, the good folks at Callstack.io released an new, drop-in replacement for the React Native packager called Haul. Over the weekend, I ran experiments to compare the performance of the two systems, to understand these two packagers better. I also wanted to see if I can add support for Haul into Mobile Center (an end to end devops service for React Native apps) ! 

Goal

As a developer, I would like to see code changes show up on the React Native app as soon as possible. The goal of this experiment was to study the performance characteristics of the two packagers and how they improve the edit-reload development cycle.
Though Haul and React Native packagers perform the same function, the technology stack differs quiet a bit; from HTTP server (connect vs express) to the way bundling is triggered (file watching vs external request). For a fair comparison, the impact of these differences must be eliminated so that we compare the core "packager" of each stack.
Like any scientific experiment, this process needs to eliminate impact of external factors, be statistically significant and available for anyone to replicate to view the results.

Methodology

To run these tests, clone the repository, install npm dependencies and run npm test. The three steps used to design the experiment are as below.

Step1: Instrument

The code in node_modules/haul or node_modules/react-native is patched to include Node's high resolution timer that indicates the start and end of the packaging process. For React Native, the probe is placed around call to the transformer while we rely on progress plugin in case of Haul. I would invite experts on the packagers to re-validate if this is the right place to instrument the code.
I also tried using the v8-profiler to measure memory usage and method calls, but it kept crashing on Haul, so no additional data could be collected there.

Step 2: Observe

In a typical real world scenario, a developer starts up the packager, heads over to their editor, makes changes to files, includes components, and then expects to see the bundle loaded on the phone.
This workflow was automated with the test suite that starts the packager using child_process.fork, and then starts adding components or trees of components to the main file. The results seem to indicate that the occurrence of a change matters more than the contents of the change.


Step 3: Measure

Since our packagers are instrumented, a message with the time to package is printed out on the console. I had initially tried using IPC to send messages, but reading the information from the stdout is much more reliable and accurate. A series of timestamps for every change is collected and tabulated. The entire scenario was repeated 100 times and outliers eliminated to obtain a statically significant result.
For each test run, the packager cache was completely cleared at the beginning and subsequent runs used the transformer cache (in case of React Native), or happypack (in Haul).

Results  #

Here are the graphs from the two test cases. React Native Packager claims that it was designed for fast compilation and targets of sub-second edit-reload cycle - the numbers prove it.


There was not much difference in the initial load with empty cache, indicating that React Native packager was clearly benefiting from storing the modules in memory. However, ReactNative packager is slower when launching the packager a second time since all its in-memory cache is blown away. Haul on the other hand is blazing fast since it picks up the on-disk cache from the previous run on the packager.
The other interesting observation is the rate at which time increases in the second chart when the number of components increase. Also note that the v8-profiler showed that the rate of memory increase in React Native packager was much higher.

Update 1 : @grabbau pointed out that there were some improvements in the master branch of Haul. Repeating the test from Haul master does show promises, but it is still a little slower than the stock React Native packager. 

Update 2: Spencer also said that internally at facebook, they have a global shared cache to make the first load times faster. 

Conclusion

Performance is just one aspect to consider when choosing a packager. Haul addresses other important issues like symlinks, haste duplicates or custom packagers like typescript. While each of these can technically be solved in the React Native packager, I like Haul for bringing it all together and making it simple. Given that Haul is new, it does lack other developer features like systraces, opening debugging in custom editors, etc - parts that don't necessarily belong to the packager component.
Could the React Native transformer be made pluggable to allow a web pack based packager, bringing us best of both worlds ?

Debugging create-react-native-app with VSCode

Expo and Facebook recently released a command line tool called create-react-native-app that makes getting started with React Native easy. With this tool, you can start developing mobile applications for iOS and Android without having to install the SDKs for each of the platforms. This is very similar to the Phonegap Developer App or Ionic View for Cordova apps and works by uploading only the JavaScript part of the application to the player app. Consequently, you can develop and debug iOS apps from a Windows machine without even needing to connect the phone via a USB.
VSCode already supports debugging Expo apps and has support to attach to a packager that is already running.




To get started download VSCode and install the React Native tools extension for VSCode. The extension gives you the ability to debug source code right from inside the editor, supports syntax highlighting and completion and has code snippets for popular React constructs.

Time travel debugging

The extension uses Node to debug the Expo app. If we replace Node with node-chakracore, we also get the ability to use time travel debugging. To try out time travel debugging with create-react-native-app and VSCode

  1. Download node-chakra nightly builds - it supports Mac, Windows and Linux :)
  2. Grab the debugger code and save it as debugger.js - this is the same code that runs when a React Native app is debugged on Chrome
  3. To start recording a trace
    1. Create a folder called logs, adjacent to debugger.js
    2. Run <path-to-node-chakra>/bin/node --record debugger.js
  4. To replay a debugging session 
    1. Start up VSCode, head to the debug pane and create a new configuration to debug a node project
    2. Add the additional key value pairs to the configuration (as shown in the video). This basically sets the node executable to be ChakraCore, points it to the location of the logs that are used for time travel, and sets up source maps
    3. Hit the debug button - you will now see "Reverse Continue" and "Step back" in addition to the usual debug workflow controls
The time travel debugging part is still experimental, but if it sounds fun, let me know, and I would love to make it a part of the extension.Thanks to the amazing folks on the Chakra team for making this possible !

ReactConf - Web like Release and Development Agility

ReactConf 2017, Santa Clara

Links from the slides

Mobile Center and CodePush

VSCode for React Native

User Gesture Mirroring


Slides available on Docs.com - https://doc.co/19Qwun

Exponent Apps - testing on Multiple Form factors

Want to test your exponent app on multiple screen sizes ? Check out maya-kai
Exponent is a great way to build mobile apps - you get all the benefits of React Native, with none of the pain of installing the Android SDK or XCode. Since Exponent is now integrated into VSCode, my developer workflow pretty much involves firing up the VSCode editor and debugging the app running on exponent in my devices.
Despite an efficient authoring workflow, testing my app across devices is still a little cumbersome. With the need to support different devices and screen sizes, I still have to perform the typical test scenarios manually over all the devices.

User gesture mirroring for React Native

Last year, I had blogged about a project that lets you mirror user gestures across devices for React Native apps. This has the same functionality that browsersync has for websites. This library lets you interact with just one device and as the test scenario progresses, you can view the user controls across other devices with different screen sizes. Though this library was for originally meant for React Native, the code is all JavaScript and uses React's Event Model. This blog post explores the idea of trying out the library with apps on Exponent.

Exponent + Maya-kai

Surprisingly, no changes were required to make it work with Exponent! I simply npm install maya-kai --save and import it in main.js using var mk=require('maya-kai'); mk.start(); The devices (2 phones, 1 iPad) were on the LAN, so I just needed to ensure that the maya-kai server was accessible to them.

To automate this even more, I added a task that also launches my app on all the connected Android devices when I start the tests.

adb devices | egrep '\t(device|emulator)' | cut -f 1 | xargs -t -J% -n1 -P5 adb -s % shell am start -a android.intent.action.VIEW -d exp://<exponent URL>

I was unable to find an equivalent for iOS though.

Other developer workflow actions like live reload and hot module replacement also worked on all devices. General developer is also faster since there is no native app "install" process here.

Wishlist for Exponent 

  • If user gesture mirroring is be something built into Exponent, additional setup steps would not be needed anymore.
  • It would also be good to automate the process of launching the app for iOS devices when starting tests.  
  • There is also no dev vs production mode, so I am currently using 2 different apps, one for testing, and the other one being the real, published app, without Maya-kai embedded.

Conclusion

I think that this workflow is great for quickly testing multiple form factors. I could simply publish my exponent app, launch it on multiple devices and interact with them. This is also a neat way to share screen with remote clients - simply enable gesture mirroring and you can walk someone through you app's workflow.
I am also looking at integrating this with Appetize, enabling a virtual device wall, something that I did for Cordova applications.