Testing on Real Devices vs. Emulators

Testing a mobile application thoroughly is a crucial step before it hits the market. Testing diminishes the chances that your product will have to face otherwise. By testing your app before launching, you communicate indirectly to the target market that your product is trustworthy. According to the International Data Corporation, 182.7 billion mobile apps will be downloaded by 2015. That’s a substantial 1600% increase from the 10.7 billion apps that were downloaded in 2010.

While there are a number of OS’s and ever aggrandizing variety of mobile devices out there, there are two major ways to test a mobile application i.e. on a real device or use an emulator for that purpose. At first glance, you might think to choose just one or the other, but the fact is that using a combination of both is the route to success.

Need of Emulators & Real Devices

Emulators are now available for each of the common operating systems and are commonly used both during manual and automation testing. Emulators are powerful tools that are mostly available for free from the OS vendors and provide a rich set of development tools. However, emulators can never be identical to the real target environment and as mobile applications are targeted on real devices, so testing on real devices is considered to be important to ensure the maximum application quality. This is similar to the pilots trained on emulators to test such situations as they cannot simulate using real aircraft due to cost or risk coefficients. But would you really trust a pilot who never flew a real airplane before? I guess not.

Pros and Cons:

In order to achieve the acme of quality testing for mobile application, pros and cons of both these approaches should be carefully considered while devising the mobile testing strategy. Following are some of the main positives and negatives of these approaches:

Mobile Emulators: Pros

Simple – Just download the software, install on your system and you’re ready to go.
Price – Mobile emulators are free and provided as a part of the SDK with new OS.
Fast – As emulators are normal client software which runs locally on your system, therefore they are much faster than the real devices connected to the network or in the cloud.

Mobile Emulators: Cons

Increased Risk – While using emulators you are not actually testing on the same devices with hardware resources that will be used by the end users. This implies that even if the application passes tests on emulators, you still can’t fully guarantee that it will actually work on the real devices as well.
Hardware and Software Differences – Emulators are actually a “plain” version of the OS and do not reflect the specific hardware & software features of real devices. Moreover, emulators are not updated accordingly to the new devices being released in the market.
Different Network Environment – In terms of network, mobile emulators run on system (through some firewall) and access the Internet via company firewall. This is not how real devices work and this difference could also affect application behavior.
No way to Test Network Interoperability – In mobile testing it is also critical to test the effect of network-related events on application like call, text message, etc. We cannot test this on the emulators.

Real Devices: Pros

Reliability – Real handsets, when used for testing ensure accurate results without any false positives or false negatives.
Interoperability Testing – Testing on real devices is mostly conducted in a live mobile network.
True User Experience – Testing on real devices is the best way to get a fair idea about the user experience, also considering other factors like the memory usage, CPU, screen size, etc. for a specific device.
Performance Testing – Easier to expose performance related issues with real handsets, as well as defect which are mostly due to the device itself or its environment.

Real Devices: Cons

Cost and Logistics – According to the testing best practices, testing should be conducted on up to 30-40 devices in the market and replacing 30% of each quarter to stay up to date. Cost of procuring and managing these devices is quite high.
Security Issues – If you are using devices connected locally to your workstation, you will have to make sure that USB ports are open. Moreover, mobile devices can be easily stolen, and also provide a way of unauthorized access to your internal network.

Which Approach is the Best?

We have seen both the pros and cons of using emulators for testing in detail. We have learned about the importance of testing on real devices in order to ensure the highest quality. Thus, when it comes to deciding between the emulators or real devices, it appears that the optimal testing solution for mobile testing is not “either or” but rather “a combination of both.”