A/B Testing for React Native apps with CodePush

A/B Testing (or Split Testing) is a way to optimize user behavior by serving different web pages to a random set of visitors and measuring metrics that determine the relative efficacy of each user interface design.
A/B testing on Websites is simple since the changes can be deployed and reverted quickly. On the contrary, doing this on mobile applications may require explicitly defining the scope of the A/B Experiment before submitting the apps to App Stores. Iterating on the experiments based on incoming metrics may also be delayed by the App Store processes. Scoping down the experiments to audience segments can also get tricky and may also need to be premeditated.
Though React Native generates native iOS and Android apps, the underlying JavaScript Engine that drives the app can utilize technologies like CodePush to enable web-like A/B tests. This post describes a way to use CodePush and other services in Mobile Center to set up A/B tests just as easily as one would do for websites.


The three main requirements and the corresponding solutions for running A/B tests for React Native are. 
  1. Setup A/B experiments that can be improved as we start collecting user behavior data. For this, we need the ability to update the app instantly. 
    • Solution: CodePush.
  2. "Push" the experiments to the user rather than wait for the user to requests. We would also want to control the audience to ensure that confounding factors are eliminated
    • Solution: Push Notifications, with the ability to define Audience segments
  3. Collect data about user activity on the app to declare a winner in the A/B tests. For a mobile app, the analytics solution should not simply stream data to the server, but collect it (even when the app is offline) and send them to the server in batches. 
    • Solution: Mobile App Analytics. 
While I am using Mobile Center as it is convenient to pick up all the services from one place, any other solution for Notifications or analytics would also work.

App Setup

To get started, the SDKs need to be installed on the React Native app using npm install react-native-code-push mobile-center-analytics mobile-center-push and added to the project using the react-native link command. The link command also asks for mobile center keys and code-push deployment keys. You may need additional setup like configuring Google Cloud Messaging or enabling APNs for integrating Push notifications with the React Native app.
We would also have to add a snippet to listen to push notifications that are sent from the server.

This would be the basic app that needs to be distributed to our users.

Starting the experiments

  1. Define an A/B experiment case. This can range from simple things like the color of a checkout button to complicated workflow differences.
  2. Make changes to the code to incorporate the experiment. It would also help to save this experiment in the corresponding branch in source control. 
  3. Track user behavior by adding the track method at the appropriate points in code. We may also want to add additional information with the tracking information that identifies the specific case for which data is collected.
  4. Next, we would also create one deployment per case and release a bundle update for each of the deployment. Even if the app uses CodePush, these updates will not yet be available on the apps since they apps use a Staging or a Production Deployment by default.
  5. We would use Push Notifications to let the apps know about the new deployment key and the snippet above that is set up to run on receiving a push notification would do the job. Create a new push notification to be sent to the user as per the tutorial.
  6. Specify the codepush deployment key corresponding to the experiment case as the custom data in the push notification with the key called "codePushDeploymentKey".
  7. Select the right audiences that would receive the push notification and have their app updated with the specific experiment case.  
As users start recieving the notification, codepush's sync method will start picking up the new bundle with the experiment. Once the user performs actions on the app, analytics would be able to send data back to the server for that use case.


While this may not be available as an end-to-end scenario in Mobile Center yet, we can already use the existing infrastructure to A/B test our React Native apps and deliver better apps. Some points to note.
  • As with CodePush, note that you will not be able to add experiments that require changing native code. 
  • App Stores explicitly forbid changing the intent and behavior of apps. Ensure that the A/B tests still stay within what is allowed by the App Stores.
  • Many complex A/B testing scenarios are usually a matrix of features and experiments simultaneously. While this post does not cover those cases, the methodology described above can be used to build such a complex system. 

ChainReact 2017 - Zero to DevOps

Chain React 2017, Portland

Title: Zero to DevOps

React Native brought the web’s enviable development patterns to mobile, without needing to sacrifice native UI. Code Push for React Native brings web like release agility to mobile apps by enabling developers to update apps instantly. This talk will cover the next iteration of Code Push and how it fits into a complete suite of DevOps services built for called Mobile Center, built for React Native. We will look at the end to end workflow - from a single commit on github triggering continuous builds and tests, to the final, signed app distributed to testers, clients or end users. From monitoring services like JavaScript enabled crashes to understanding user behavior with analytics, we will look at ways to get better mobile apps to your users. We will look at the internals of Mobile Center to use with your existing toolchain, other services that are planned as a part of the suite, and integration with popular community tools and services.

React Native Packager vs Haul

What packages faster ? Haul or React Native ? Jump to results

A few days ago, the good folks at Callstack.io released an new, drop-in replacement for the React Native packager called Haul. Over the weekend, I ran experiments to compare the performance of the two systems, to understand these two packagers better. I also wanted to see if I can add support for Haul into Mobile Center (an end to end devops service for React Native apps) ! 


As a developer, I would like to see code changes show up on the React Native app as soon as possible. The goal of this experiment was to study the performance characteristics of the two packagers and how they improve the edit-reload development cycle.
Though Haul and React Native packagers perform the same function, the technology stack differs quiet a bit; from HTTP server (connect vs express) to the way bundling is triggered (file watching vs external request). For a fair comparison, the impact of these differences must be eliminated so that we compare the core "packager" of each stack.
Like any scientific experiment, this process needs to eliminate impact of external factors, be statistically significant and available for anyone to replicate to view the results.


To run these tests, clone the repository, install npm dependencies and run npm test. The three steps used to design the experiment are as below.

Step1: Instrument

The code in node_modules/haul or node_modules/react-native is patched to include Node's high resolution timer that indicates the start and end of the packaging process. For React Native, the probe is placed around call to the transformer while we rely on progress plugin in case of Haul. I would invite experts on the packagers to re-validate if this is the right place to instrument the code.
I also tried using the v8-profiler to measure memory usage and method calls, but it kept crashing on Haul, so no additional data could be collected there.

Step 2: Observe

In a typical real world scenario, a developer starts up the packager, heads over to their editor, makes changes to files, includes components, and then expects to see the bundle loaded on the phone.
This workflow was automated with the test suite that starts the packager using child_process.fork, and then starts adding components or trees of components to the main file. The results seem to indicate that the occurrence of a change matters more than the contents of the change.

Step 3: Measure

Since our packagers are instrumented, a message with the time to package is printed out on the console. I had initially tried using IPC to send messages, but reading the information from the stdout is much more reliable and accurate. A series of timestamps for every change is collected and tabulated. The entire scenario was repeated 100 times and outliers eliminated to obtain a statically significant result.
For each test run, the packager cache was completely cleared at the beginning and subsequent runs used the transformer cache (in case of React Native), or happypack (in Haul).

Results  #

Here are the graphs from the two test cases. React Native Packager claims that it was designed for fast compilation and targets of sub-second edit-reload cycle - the numbers prove it.

There was not much difference in the initial load with empty cache, indicating that React Native packager was clearly benefiting from storing the modules in memory. However, ReactNative packager is slower when launching the packager a second time since all its in-memory cache is blown away. Haul on the other hand is blazing fast since it picks up the on-disk cache from the previous run on the packager.
The other interesting observation is the rate at which time increases in the second chart when the number of components increase. Also note that the v8-profiler showed that the rate of memory increase in React Native packager was much higher.

Update 1 : @grabbau pointed out that there were some improvements in the master branch of Haul. Repeating the test from Haul master does show promises, but it is still a little slower than the stock React Native packager. 

Update 2: Spencer also said that internally at facebook, they have a global shared cache to make the first load times faster. 


Performance is just one aspect to consider when choosing a packager. Haul addresses other important issues like symlinks, haste duplicates or custom packagers like typescript. While each of these can technically be solved in the React Native packager, I like Haul for bringing it all together and making it simple. Given that Haul is new, it does lack other developer features like systraces, opening debugging in custom editors, etc - parts that don't necessarily belong to the packager component.
Could the React Native transformer be made pluggable to allow a web pack based packager, bringing us best of both worlds ?

Debugging create-react-native-app with VSCode

Expo and Facebook recently released a command line tool called create-react-native-app that makes getting started with React Native easy. With this tool, you can start developing mobile applications for iOS and Android without having to install the SDKs for each of the platforms. This is very similar to the Phonegap Developer App or Ionic View for Cordova apps and works by uploading only the JavaScript part of the application to the player app. Consequently, you can develop and debug iOS apps from a Windows machine without even needing to connect the phone via a USB.
VSCode already supports debugging Expo apps and has support to attach to a packager that is already running.

To get started download VSCode and install the React Native tools extension for VSCode. The extension gives you the ability to debug source code right from inside the editor, supports syntax highlighting and completion and has code snippets for popular React constructs.

Time travel debugging

The extension uses Node to debug the Expo app. If we replace Node with node-chakracore, we also get the ability to use time travel debugging. To try out time travel debugging with create-react-native-app and VSCode

  1. Download node-chakra nightly builds - it supports Mac, Windows and Linux :)
  2. Grab the debugger code and save it as debugger.js - this is the same code that runs when a React Native app is debugged on Chrome
  3. To start recording a trace
    1. Create a folder called logs, adjacent to debugger.js
    2. Run <path-to-node-chakra>/bin/node --record debugger.js
  4. To replay a debugging session 
    1. Start up VSCode, head to the debug pane and create a new configuration to debug a node project
    2. Add the additional key value pairs to the configuration (as shown in the video). This basically sets the node executable to be ChakraCore, points it to the location of the logs that are used for time travel, and sets up source maps
    3. Hit the debug button - you will now see "Reverse Continue" and "Step back" in addition to the usual debug workflow controls
The time travel debugging part is still experimental, but if it sounds fun, let me know, and I would love to make it a part of the extension.Thanks to the amazing folks on the Chakra team for making this possible !

ReactConf - Web like Release and Development Agility

ReactConf 2017, Santa Clara

Links from the slides

Mobile Center and CodePush

VSCode for React Native

User Gesture Mirroring

Slides available on Docs.com - https://doc.co/19Qwun

Exponent Apps - testing on Multiple Form factors

Want to test your exponent app on multiple screen sizes ? Check out maya-kai
Exponent is a great way to build mobile apps - you get all the benefits of React Native, with none of the pain of installing the Android SDK or XCode. Since Exponent is now integrated into VSCode, my developer workflow pretty much involves firing up the VSCode editor and debugging the app running on exponent in my devices.
Despite an efficient authoring workflow, testing my app across devices is still a little cumbersome. With the need to support different devices and screen sizes, I still have to perform the typical test scenarios manually over all the devices.

User gesture mirroring for React Native

Last year, I had blogged about a project that lets you mirror user gestures across devices for React Native apps. This has the same functionality that browsersync has for websites. This library lets you interact with just one device and as the test scenario progresses, you can view the user controls across other devices with different screen sizes. Though this library was for originally meant for React Native, the code is all JavaScript and uses React's Event Model. This blog post explores the idea of trying out the library with apps on Exponent.

Exponent + Maya-kai

Surprisingly, no changes were required to make it work with Exponent! I simply npm install maya-kai --save and import it in main.js using var mk=require('maya-kai'); mk.start(); The devices (2 phones, 1 iPad) were on the LAN, so I just needed to ensure that the maya-kai server was accessible to them.

To automate this even more, I added a task that also launches my app on all the connected Android devices when I start the tests.

adb devices | egrep '\t(device|emulator)' | cut -f 1 | xargs -t -J% -n1 -P5 adb -s % shell am start -a android.intent.action.VIEW -d exp://<exponent URL>

I was unable to find an equivalent for iOS though.

Other developer workflow actions like live reload and hot module replacement also worked on all devices. General developer is also faster since there is no native app "install" process here.

Wishlist for Exponent 

  • If user gesture mirroring is be something built into Exponent, additional setup steps would not be needed anymore.
  • It would also be good to automate the process of launching the app for iOS devices when starting tests.  
  • There is also no dev vs production mode, so I am currently using 2 different apps, one for testing, and the other one being the real, published app, without Maya-kai embedded.


I think that this workflow is great for quickly testing multiple form factors. I could simply publish my exponent app, launch it on multiple devices and interact with them. This is also a neat way to share screen with remote clients - simply enable gesture mirroring and you can walk someone through you app's workflow.
I am also looking at integrating this with Appetize, enabling a virtual device wall, something that I did for Cordova applications.

React Native - Zero to DevOps

Using http://mobile.azure.com to set up the DevOps pipeline for React Native applications