What packages faster ? Haul or React Native ? Jump to results
A few days ago, the good folks at Callstack.io released an new, drop-in replacement for the React Native packager called Haul. Over the weekend, I ran experiments to compare the performance of the two systems, to understand these two packagers better. I also wanted to see if I can add support for Haul into Mobile Center (an end to end devops service for React Native apps) !
Goal
As a developer, I would like to see code changes show up on the React Native app as soon as possible. The goal of this experiment was to study the performance characteristics of the two packagers and how they improve the edit-reload development cycle.Though Haul and React Native packagers perform the same function, the technology stack differs quiet a bit; from HTTP server (connect vs express) to the way bundling is triggered (file watching vs external request). For a fair comparison, the impact of these differences must be eliminated so that we compare the core "packager" of each stack.
Like any scientific experiment, this process needs to eliminate impact of external factors, be statistically significant and available for anyone to replicate to view the results.
Methodology
To run these tests, clone the repository, install npm dependencies and run npm test. The three steps used to design the experiment are as below.Step1: Instrument
The code in node_modules/haul or node_modules/react-native is patched to include Node's high resolution timer that indicates the start and end of the packaging process. For React Native, the probe is placed around call to the transformer while we rely on progress plugin in case of Haul. I would invite experts on the packagers to re-validate if this is the right place to instrument the code.I also tried using the v8-profiler to measure memory usage and method calls, but it kept crashing on Haul, so no additional data could be collected there.
Step 2: Observe
In a typical real world scenario, a developer starts up the packager, heads over to their editor, makes changes to files, includes components, and then expects to see the bundle loaded on the phone.This workflow was automated with the test suite that starts the packager using child_process.fork, and then starts adding components or trees of components to the main file. The results seem to indicate that the occurrence of a change matters more than the contents of the change.
Step 3: Measure
Since our packagers are instrumented, a message with the time to package is printed out on the console. I had initially tried using IPC to send messages, but reading the information from the stdout is much more reliable and accurate. A series of timestamps for every change is collected and tabulated. The entire scenario was repeated 100 times and outliers eliminated to obtain a statically significant result.For each test run, the packager cache was completely cleared at the beginning and subsequent runs used the transformer cache (in case of React Native), or happypack (in Haul).
Results #
Here are the graphs from the two test cases. React Native Packager claims that it was designed for fast compilation and targets of sub-second edit-reload cycle - the numbers prove it.There was not much difference in the initial load with empty cache, indicating that React Native packager was clearly benefiting from storing the modules in memory. However, ReactNative packager is slower when launching the packager a second time since all its in-memory cache is blown away. Haul on the other hand is blazing fast since it picks up the on-disk cache from the previous run on the packager.
The other interesting observation is the rate at which time increases in the second chart when the number of components increase. Also note that the v8-profiler showed that the rate of memory increase in React Native packager was much higher.
Update 1 : @grabbau pointed out that there were some improvements in the master branch of Haul. Repeating the test from Haul master does show promises, but it is still a little slower than the stock React Native packager.
Update 2: Spencer also said that internally at facebook, they have a global shared cache to make the first load times faster.
Conclusion
Performance is just one aspect to consider when choosing a packager. Haul addresses other important issues like symlinks, haste duplicates or custom packagers like typescript. While each of these can technically be solved in the React Native packager, I like Haul for bringing it all together and making it simple. Given that Haul is new, it does lack other developer features like systraces, opening debugging in custom editors, etc - parts that don't necessarily belong to the packager component.Could the React Native transformer be made pluggable to allow a web pack based packager, bringing us best of both worlds ?
