A/B Testing for React Native apps with CodePush

A/B Testing (or Split Testing) is a way to optimize user behavior by serving different web pages to a random set of visitors and measuring metrics that determine the relative efficacy of each user interface design.
A/B testing on Websites is simple since the changes can be deployed and reverted quickly. On the contrary, doing this on mobile applications may require explicitly defining the scope of the A/B Experiment before submitting the apps to App Stores. Iterating on the experiments based on incoming metrics may also be delayed by the App Store processes. Scoping down the experiments to audience segments can also get tricky and may also need to be premeditated.
Though React Native generates native iOS and Android apps, the underlying JavaScript Engine that drives the app can utilize technologies like CodePush to enable web-like A/B tests. This post describes a way to use CodePush and other services in Mobile Center to set up A/B tests just as easily as one would do for websites.

Requirements

The three main requirements and the corresponding solutions for running A/B tests for React Native are. 

1. Setup A/B experiments that can be improved as we start collecting user behavior data. For this, we need the ability to update the app instantly. 
• Solution: CodePush.
2. "Push" the experiments to the user rather than wait for the user to requests. We would also want to control the audience to ensure that confounding factors are eliminated
• Solution: Push Notifications, with the ability to define Audience segments
3. Collect data about user activity on the app to declare a winner in the A/B tests. For a mobile app, the analytics solution should not simply stream data to the server, but collect it (even when the app is offline) and send them to the server in batches. 
• Solution: Mobile App Analytics. 

While I am using Mobile Center as it is convenient to pick up all the services from one place, any other solution for Notifications or analytics would also work.

App Setup

To get started, the SDKs need to be installed on the React Native app using npm install react-native-code-push mobile-center-analytics mobile-center-push and added to the project using the react-native link command. The link command also asks for mobile center keys and code-push deployment keys. You may need additional setup like configuring Google Cloud Messaging or enabling APNs for integrating Push notifications with the React Native app.
We would also have to add a snippet to listen to push notifications that are sent from the server.

This would be the basic app that needs to be distributed to our users.

Starting the experiments

1. Define an A/B experiment case. This can range from simple things like the color of a checkout button to complicated workflow differences.
2. Make changes to the code to incorporate the experiment. It would also help to save this experiment in the corresponding branch in source control. 
3. Track user behavior by adding the track method at the appropriate points in code. We may also want to add additional information with the tracking information that identifies the specific case for which data is collected.
4. Next, we would also create one deployment per case and release a bundle update for each of the deployment. Even if the app uses CodePush, these updates will not yet be available on the apps since they apps use a Staging or a Production Deployment by default.
5. We would use Push Notifications to let the apps know about the new deployment key and the snippet above that is set up to run on receiving a push notification would do the job. Create a new push notification to be sent to the user as per the tutorial.
6. Specify the codepush deployment key corresponding to the experiment case as the custom data in the push notification with the key called "codePushDeploymentKey".
7. Select the right audiences that would receive the push notification and have their app updated with the specific experiment case.  

As users start recieving the notification, codepush's sync method will start picking up the new bundle with the experiment. Once the user performs actions on the app, analytics would be able to send data back to the server for that use case.

Conclusion

While this may not be available as an end-to-end scenario in Mobile Center yet, we can already use the existing infrastructure to A/B test our React Native apps and deliver better apps. Some points to note.

• As with CodePush, note that you will not be able to add experiments that require changing native code. 
• App Stores explicitly forbid changing the intent and behavior of apps. Ensure that the A/B tests still stay within what is allowed by the App Stores.
• Many complex A/B testing scenarios are usually a matrix of features and experiments simultaneously. While this post does not cover those cases, the methodology described above can be used to build such a complex system.