How Does Facebook Test its Apps?

 
Using REAL mobile devices.

You must be like “Noooo…, really?” with a sarcastic look on your face.

Well, Facebook using real devices for testing is actually not surprising at all. But, what is really surprising is that there are several mobile app development companies who still do NOT use real devices for testing. Isn’t it enough to test mobile applications on simulators or emulators? Definitely No! If it were, then Facebook would not have invested on its expansive state-of-the-art Mobile Device Lab in Prineville, OR.

Facebook’s journey to the data centre started off with their first setup – The Sled model. The Sled model had drawbacks like inconsistent Wi-Fi connectivity which led them to build the second version named – The Gondola. The Gondola had a lot of issues with tangled USB Cables so they built – The Slatwall. The Slatwall took up an entire room, and we were able to deploy 240 devices, but to accommodate 2,000 phones they need to scale to nine of these rooms, this wouldn’t work. So they decided to move the mobile device lab into a data centre in Oregon.

Currently, the lab in Oregon hosts 60 custom-built racks, each of which holds 32 devices. Four Linux-based OCP Leopard servers are used to test Android apps, while eight Mac Minis are used for iOS apps. Each OCP Leopard server is connected to eight Android devices, and each Mac Mini is connected to four iPhones. Almost 2,000 devices can be run simultaneously for testing new iOS and Android versions of Facebook, Instagram, Messenger and Whatsapp. Each rack also has its own Wi-Fi signal and an EMI enclosure to keep the Wi-Fi local to its rack, according to a post by Facebook developer Antoine Reversat. This not just ensures that Facebook apps functions properly on new and old devices, but also in various network conditions.
 

Why does Facebook require a Mobile Device Lab?

Facebook has a growing 1.6 billion users worldwide. There are tens of thousands of different mobile devices in the market, and Facebook wants its apps to run seamlessly on all of them. Taking into account the variation in hardware, mobile operating system, aspect ratios, and network condition of these mobile devices is an increasingly complex operation.

Additionally, the developers at Facebook make thousands of code changes every week on each mobile platform. This means they also need to be aware of the performance implications of a code change on both high-end and typical devices, as well as on a variety of operating systems. “Given the code intricacies of the Facebook app, we could inadvertently introduce regressions that take up more data, memory or battery usage”, writes Reversat in a blog post. It is crucial to ensure new versions of Facebook’s apps don’t crash or run into performance problems when engineers make updates.

The Rack based Mobile Device Lab is a system built to maintain and improve the development speed while minimising the number of regressions in dimensions of performance, such as speed, data usage, battery consumption, and memory footprint. “In practice, performance issues can be detected in either a lab environment or the real world. Both environments have pros and cons. In the lab, it is difficult to build a controlled environment to reasonably mimic the real world, so catching most perf issues and predicting perf wins before they make it out into the wild is harder. Data that’s gathered in the real world is truer than data gathered in the lab, but the signals can be very noisy. This can make it more difficult to fix a problem or more difficult to figure out the root cause of an issue. It is also less desirable because it requires data collection from user devices. We have worked hard on building tools for lab environments with a belief that such a system can catch most issues, if not all of them, and provide a significant benefit.” writes Zheng Mi, an Engineering Manager at Facebook

The lab is also integrated with configuration management tool named “Chef”, developed by Facebook to deal with the management of hundreds of thousands of servers is used to monitor and maintain a consistent device state during the tests.

The lab also houses the social network’s one of the most ambitious artificial intelligence projects: the Big Sur. It is Facebook’s most advanced Open Rack-compatible hardware specifically designed for AI computing at large scale. Leveraging NVIDIA’s Tesla Accelerated Computing Platform, Big Sur is twice as fast as its previous generation, which means you can train twice as fast and explore networks twice as large.

When Facebook can assemble its own fleet of devices for testing purposes, why can’t you? Want to own a mobile device lab like Facebook’s? pCloudy can help you.