Types of Testing

Since there are many players in the mobile app market, the competition is endless. To beat the heat and to keep providing an enhanced mobile app experience to the users, the app needs to be constantly updated with the changes happening and for doing this, there are different types of testing techniques. Let us have an insight into the types of testing methods.

1. Compatibility Testing

Compatibility testing is a critical QA task. It assures that a given application works as intended with selected operating systems, selected devices with different screen sizes resolutions, and internal hardware (memory size, processor speed, and button/input differences). It defines the feasible compatibility combinations of devices and interfaces for a specific testing assignment, in concurrence with the customer’s requirement.

Compatibility Factors:

There are different factors that play a very significant role in mobile compatibility testing strategy. Mobile operating systems and their supported versions, mobile devices developed by different manufacturers, different device screen sizes with different resolutions and internal hardware including different input types, processor speeds and memory size of devices.

Let’s develop a test strategy for compatibility testing. The first step is to identify the Operating systems and devices based on key factors described in the previous slide. Once you identified the devices based on different compatibility factors you need to set up your environment for testing. You can either set up your internal test lab or you can go for external cloud providers like pCloudy. Now, the next step is to execute your test cases on a defined set of environment. Report the defects to appear while testing and report these defects to your development team.

The main purpose of compatibility testing is to ensure whether different software is compatible with different configurations and whether the application is compatible with the client’s environment.

2. Installation Testing

Mobile phones come with different types of mobile apps like Native and Web and Hybrid. Installation testing is a type of testing that is done at the initial stage of mobile app life span or maybe it is the first impression on the users, in other words. Installation testing checks whether the mobile app installs, uninstalls and updates properly without any interruption.

Installation Testing Factors:

There are some of the factors that play a significant role in installation testing of an application. Factors like which platforms and operating systems supported by the application and how they will be distributed whether it is directly available in the app store or are going to get the installable files on mail itself or is it available on an app distribution platform and can be accessed through public URL. The user behavior regarding installation changes from person to person. For eg: the developer may install the app (By using ADB install command) and a different user might directly install it from the app store or public URL.

Important points to be noted in Installation Testing:

Some of the common checkpoints which can be converted to standard test cases are:

• Installing the app in phone memory.
• Installing the app in phone memory is full.
• Installing the app on an external SD card.
• Installing the app on an external SD card is full.
• Check the memory space after installing. It should not occupy more than promised.
• Check if it is releasing the space after removing

Another part of the installation and uninstallation testing is the testing of patches/updates
These are the common situation when we need to test app after updates.

Check for updates:

• Can I download an update?
• Can I update the app when there are multiple updates available?
• What happens if I don’t update?

 

When App is upgraded:

• Check that the user is still logged in
• Check that data is still the same
• Backward compatibility:-Check that database changes will not affect the current data.
• Test the update from an older version to the current version

 

When OS is upgraded:

• Check the app compatibility.
• App updated gracefully without a crash.

 

3. Interruption Testing

Interrupt testing is a process to replicate abrupt (Unexpected) interrupt to the application. It is done to understand how the app behaves under certain interruptions before resuming to its original state. This can be achieved in various ways and techniques depending on the application under test.

Following are some Common Interrupts:

1. A phone call when the application is running or is in background
2. Battery removal when the application is running or is in the background, battery high, battery low, incoming call or SMS
3. Plugged in or out of charging
4. Device shut down
5. OS upgrade
6. Network loss and restoration, etc.

Some specific scenarios for an application using a network:

1. Connect to the network but remove LAN connection from the router so the device can sense wifi state on the device but cannot connect to the internet
2. Connection via VPN and VPN disconnected

Scenario for Application using services:

1. Kill service by clicking on the recent button and swiping the application right to kill app and services
2. Kill app using third-party App killer
3. Kill specific services from Settings->Manage Applications
4. Scenarios for Application Linked to account Manager
Remove account from Settings->Account Manager

Incoming and Outgoing SMS and MMS:

An application is running in your mobile and you send an SMS/MMS from another mobile to your mobile. In this situation, SMS/MMS notification should display for a while and then close and the application should continue to run.

Incoming and Outgoing calls:

You are in a login page trying to log in and you have just pressed the submit button that will take you to the application home page. This is a transition period because loading is happening.If we receive a call during this time then the application should not end abruptly or crash.

Battery Removal:

If the application is running, remove the battery of your mobile and it will shut down. After rebooting the mobile, run the application again, it should run smoothly and without a crash.

Data Cable Insertion and Removal:

When the application is running, connect the USB Cable, the system should display “USB Connection Screen” and if the user returns to the application then the application should not crash or end abruptly

Interruption Testing Strategy

Let’s prepare a test strategy to test interruption testing:
The first step in this process is to decide the critical interrupts based on your application components and architecture. For example, if your application is using GPS include common interrupts related to GPS. Let’s Include two type of interrupts:

Application specific interrupts:

Prepare your test cases and try to fit theses interruption test cases with state of the application. For example, interrupt your Network while performing payment.

Interrupt testing process:

List of common Interrupts:

Phone calls, Text messages, App notifications, Battery Warning, Forced updates, Voicemail, Shaking the Phone, Different Gestures, Switching between apps, Locking and unlocking the screen, Changing the orientation, Music playing while using the app, Out of memory (general performance interruptions), Data app interruptions (What Sapp, Viber, Tango), Audio interrupts from multiple sources (iPod, Media player, Other audio apps), phone in standby mode, switch the network to plane mode.

Tools to test interruption Testing:

Use tools like Joule Unit (Android), On iOS use the Energy Usage Function of Instruments, Monkey for Android to stress test your app and see the handling of the interrupts, UI Auto Monkey for iOS

Scenario

A) Once the user connects the mobile earphone jack with Nokia Lumia and starts music Player.
B) Now when the user resumes the application the cart gets empty.

4. Localization Testing

This type of testing is a technique in which we check whether the mobile app adheres to the local cultural settings, customizing the apps according to target country and language and also according to the linguistic aspects. Localization is also known as “L10N” as there are 10 characters between L & N. For eg: Chinese language-search engine offered by Baidu beats Google in China. Baidu does better than Google because it looks and feels fully native to the Chinese speaking audience.

Also, regional language speakers can’t get many games in their own language, so they turn to race games, which are easy to understand. We must have noticed in these examples, language is the common issue. Apart from Translation there are different other elements in the localization, like Usage of proper Time zone, local formats of dates, addresses and phone numbers, Design and layout to fit translated text, Converting to local requirements (such as currencies and units of measure), Adapting graphics to target markets, Modifying content to suit the tastes and consumption habits of other markets, Addressing local regulations and legal requirements.

Why should we localize?

Localization is important in order to understand the cultural and linguistic aspects of a particular region. While testing, testers keep repeating the tests to check typographical, linguistic errors, cultural appropriateness of the UI, etc. Research says:
1.  56.2 percent of consumers say that the ability to obtain information in their own language is more important than price. (Common Sense Advisory, Can’t Read, Won’t Buy: Why Language Matters on Global Websites, 2006)
2.  74 percent of multinational enterprises believe it is either important or most important to achieve increased revenues from global operations. (California State University at Chico, 2007.
3.  95 percent of Chinese online consumers indicate a greater comfort level with websites in their language. (Forrester Research, Translation and Localization of Retail Web Sites, 2009)
4.   A critical success factor for cross-border merger and acquisition deals is the ability to communicate information clearly and accurately in multiple languages. (Merrill Corporation, How to do Better Multinational M&A Deals, 2008)

Types of Localization Testing:

In the localization testing, validating whether a mobile app is capable of performing given a geographical location, in particular, is found out. It includes four major types of testing like linguistic, culture, cosmetic, and functional tests.
1. Linguistic testing– ensures that the UI text appears in the congregation with the language and is not mistranslated or misapplied. This requires language skill and knowledge of the product. Mistranslation, irrelevant usage of the language are few of the errors in this.

2. Cultural Testing– Few words may imply different meanings in different cultures. So, this has to be translated properly in relation to the target culture.

3. Cosmetic Testing– For easy understanding, let us use this scenario. For eg. In the apps used in Afghanistan, the text will be in Arabic and will start from right to left unlike other languages like English, French that are read from Left to right.

4. Functional Testing– In this testing, testers follow prescribed test scripts to run through all aspects of the product to make sure it functions as designed in a localized environment. Often functional testing includes verification that the localized product is compatible with various operating systems and third-party products. Functional testing generally requires a good knowledge of the target language and total familiarity with the product. Some of the major Functional testing problems may include– Incorrect or missing UI elements, graphics, and windows, Incorrect error messages, Localization bugs that may cause software crashes, Errors generated by installing the localized software on a localized OS.

5. Performance Testing

Performance testing is to test the performance of the mobile application in expected workload scenarios and to eliminate the performance hurdles. It checks whether the response of the app is quick, the ability of the app to tale load, and app stability in those load situations. Performance is very critical because if the app is malfunctioning, it is more possible that the user might uninstall the app and might shift to a competitor’s app which is better.

The three main pillars of Mobile app performance testing are:

Device Performance

App start-up time is the most critical performance parameter from the user’s point of view. It shouldn’t take more than 2 seconds for the app’s first screen to pop up after the user taps on the app’s icon. Battery time is the second most important issue. Some apps consume a high amount of battery life. Excessive resource usage creates a burden on the processor and the phone heats up. In some cases, it is observed that the new app installed uses the same amount of battery as the OS. If more features are added in the app its memory consumption also increases. This is why memory consumption should be checked while testing the app.

Network Performance

The app needs to be tested on different mobile networks and network properties. If there is packet loss then the app should generate alerts or resend the request for the information. Jitters or delay in receiving information

Server/API Performance

Performance is directly proportional to the response time from the Server. In this, tests are run to check how efficient the app is to handle data transfers from the server and how quickly and in what format the data is transported by the app. The number of API calls generated by apps should be lesser and how and where the data is saved in the ‘server down’ case is all that is tested here.

Mobile app testing tools make it easier for the testers to identify and rectify any error that comes up. For different OS there are a different set of Mobile app tools. For Android, we can use Robotium and Monkey Runner and for IOS we can use Automator.

6. Usability Testing

Usability testing is a type of testing which is performed to check how user-friendly the mobile application is in terms of the navigation, ease to use the app, flexible app controls, etc. It is also known as user experience testing.
What matters in Usability Testing?

Workflow: To understand, below image represents the workflow of how the user reaches its goal while navigating through the app controls.

We must understand what the user needs to do in order to use the app’s functionality. If the workflow was overlong and included repetitive actions It may annoy the user while performing the task.
Design and Layout: Good layout and design allow a user to easily complete tasks. An app need not offer so many features or content that the app is difficult to navigate. A friendly app has a design that allows less user text entry, helps differentiation between the selected items, finger-sized targets, should provide intuitive hints to texts, etc.

The response time of the application: It is one of the key factors. This means how long the user takes to complete a task. Often long delays while the app interacts with the server hinders the user experience standards.

The emotional state of users: Emotional state of user measures the user’s motivation to use the app. The apps should be smart enough to understand what the users want and how to engage them, this can be achieved by usability testing.

Usability Testing Strategy:

The first step is to identify users who match your expected audience. Watch and listen to them as they use your products. You can use video and audio recording equipment for archiving test sessions, in case later review is needed. However, our primary tool is taking notes with paper and pencil! This is still the fastest way we have found to get information quickly to developers and designers. You have your data. You now need to organize and analyze it. How the results are absorbed as a report in order to be useful for product changes.

Tools and techniques for usability testing

1. Mr. Tappy: It is a tool set up to capture how users interact with the apps naturally. The user data so generated could be used for better user experience.

2. Paper Prototyping: It is a technique that consists of creating hand drawings of user interfaces in order to enable them to be rapidly designed, simulated and tested. As simple as this technique may look like, it has been effectively used since the 1980s and is likely to continue to be used with a guaranteed degree of success for many more years to come.

3. Magitest: It allows you to perform mobile user testing of native apps and websites. Captures the participant’s voice and facial expressions during the session. Here the users complete the tasks you specify and allows to listen to them speak their thoughts aloud as they interact with your mobile app.

4. Reflector: This app allows us to see the iPad, iPod, iPhone screens on MAC or PC. The devices connect wirelessly to proceed with further functions.

7. Conformance Testing

Conformance/compliance testing is a type of testing which is performed to validate if the application is meeting the Market place and Enterprise policy guidelines. It is like an audit done to check whether certain standards are being met or not. Conformance Testing is focused on two main areas- App store guidelines and Enterprise policy compliance. Each Market place has their own guidelines. We need to validate if the application is following the guidelines of each of these app stores.

1. App store Guidelines pertaining to User Interface, Media Content, Privacy, Spyware and Phishing, nudity, religion, culture, violence, etc.

2. Enterprise Policy Compliance pertaining to a different set of industrial standards. For eg: For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, In healthcare, the Health Insurance Portability and Accountability Act (HIPAA) includes an administrative simplification section, which mandates protecting the information assets of patients. So, it is a dire need for the testing organizations to follow the necessary guidelines and compliances to limit the risk that non-compliance might follow.

Reap the benefits of different types of testing

In this day and age, mobile app quality matters the most as the market is very competitive. We need to focus on user experience as well as the performance of the app. If the app contains bugs the app ratings will go down and so will the number of app downloads. If we can make our app bug free than the chances of app gaining popularity will increase. This can be achieved by performing different types of testing on the mobile application. In the next blog in this series, we will talk about mobile app testing strategies.

Watch this video for more information:

Related Articles:

• Mobile App Testing Strategies
• 6 Most Common Problems in Mobile App Testing
• Accelerated Mobile App Testing powered by pCloudy
• Traditional Mobile App Testing Vs New Way of Mobile App Testing with pCloudy
• Mobile Application Testing Trends in 2019
• Types of Mobile Apps: Native, Hybrid, Web and Progressive Web apps