In the last blog in this series we learned how to start the Appium server and launch app from the code. In this blog we will understand how to setup Appium on a Mac machine.
Pre-requisite
• Mac Book
• Xcode
• Simulator or real device
• Appium
• Eclipse
• Java
The first step is to install Xcode in the Mac Book. To do that, open app store and search for Xcode and click on the Get button to download Xcode.
Then click on the Install App option. This will install Xcode on your Mac machine. Now click on the Agree button if you agree with the terms and conditions and it will start installing components.
Once the Xcode is installed, you need to select “Create a new Xcode project” and you will be redirected to the template selection option.
You may select any of the templates or leave default settings as they are. Click on the next button to navigate to the next screen.
Now you will be asked to provide a product name and other product-related settings.
Enter the required details and click on the next button. In the last seen of product settings, you will be asked to give the location in which to save your project.
Select the location anywhere as per your requirements and click on the create button.
Once the project is loaded go to the Product menu option and click on the Run option.
Once you click on the run option, it will launch iOS simulator.
Now right click on the simulator icon on the doc panel and select the keep in doc panel. In this way, you don’t need to start Xcode to launch the simulator. Thus you don’t need to open the Xcode project each time to launch iPhone simulator. If you want to see the list of simulators, got to the Window menu and select the Devices option. You will get a list of all the simulators available with this version of Xcode.
Installing JDK and setting the path
Download JDK for Mac OS and once the file is downloaded, double click on the file and install Java.
Now you need to set the Java installation path in your environment variable. Open a terminal and write “open.bash_profile”.
If “.bash_profile” does not exist then execute the command “touch~/.bash_profile”. This will create the file. So when you execute the command, it opens the “.bash_profile” file.
Copy the path to JDK home, write export JAVA_Home= and then save this file. Open command prompt and execute the java – version command. Open Appium.io and click on Download Appium button.
Now download the latest .dmg file and once downloaded, double click on the .dmg file.
It will open a new window where you need to drag and drop Appium into the application folder.
Search for Appium and double click on the Appium icon. This will launch the Appium server on your Mac machine.
To Launch Appium on a simulator click on the Start Server button.
Now click on the Start Inspector Session button in the Appium server.
Once you select pCloudy as the cloud provider from the list you can click on Done.
Then you need to enter the Hostname, User name, API Key, Desired Capabilities and then click on Start Session.
Your simulator is now connected.
Now you know how to setup Appium on Mac and in the previous blogs in this series we explained how to write the test script and start Appium server. In the next blog we will learn how to use Appium inspector to identify elements in an app.
We are committed to keeping you ahead of others. pCloudy is one of the fastest to release the support of Android Q beta devices on the cloud.
Google has launched the final Android Q beta and Android Q official release is just a few weeks away. Testers and developers can test their app against this final beta version. Android Q beta 6 includes the final API 29 SDK and updated build tools for Android Studio. It also supports all the features, developer APIs and system behaviors that will be in the final release.
Now you can test your mobile apps on devices loaded with Android Q beta version in pCloudy.
In the previous blog in this series, we learned how to write the first Appium script for Android devices. Now in this blog, we will understand how to start the Appium server.
The code structure is segregated into three parts, the first part of the code starts the Appium server, the second part stops the Appium server.
The third part is the main method which is necessary to execute the class and from this main method, all the above method are called.
Starting Appium from code requires a path of two files which are kept inside the Appium folder.
node.exe
js
Starting Appium Server
In the code highlighted in red, we have called process class which is a Java class and declared it static.
We then created a start server variable and pass the path to node.exe and main.js, highlighted in yellow. In the code highlighted in green, we added both paths into the same variable with spaces and created a method called startAppiumServer() which takes care of the Appium server startup process. Next step is to create an object of Runtime class which is again a java class and call the method getRuntime(), highlighted in blue. We will pass the variable “STARTSERVER” into runtime.exec() method. It will start the Appium Server.
Once the process is started, we have to store the current state of the process into this variable. We will give a sleep time of 5 seconds as it takes time to start the process. what we are trying to validate here is, if the process is not null; it means that the process is started as highlighted in the gray box.
Stop Appium Server
Here we need to create another method named stopAppiumServer(). We again verify the state of the process and if it is not null, call a method of process class destroy().
So now we know how to start and stop the Appium server from code. Let’s try to install and launch an app on a mobile device. Let’s see how we can install a .apk file on an Android device and launch it automatically without providing the path in Appium Server.
Installing and launching the application
To begin with, we call the stopAppiumServer() method to verify if any instance of Appium Server is already running and if so, then it closes that instance, as highlighted in red. The method highlighted in yellow starts the Appium Server.
Then store the apk path file into a variable of File class, which is a Java class as highlighted in blue. Create another variable of File class app. It takes two parameters:
The absolute path of the apk file.
Name of the apk file.
Now the absolute path is stored into appDir variable and the name of apk file in the second part as highlighted in the green box. The rest of the things are the same as discussed in the earlier sessions.
The complete code is shown here.
Now that you have learned how to launch app from code, let’s move on to the next blog where we will learn how to set up Appium on a Mac machine.
In the previous blog, we informed about the basics of Appium and its architecture. Now let’s understand how to setup Appium. There are four steps required to setup Appium, they are enabling the Android device for test, test environment setup, Appium Configurations, Launch the app on the device.
Pre-requisites for Appium Mobile Testing
1. An Android device with OS 4.2+
2. AUT(Application Under Test) file (.apk)
3. Phone USB Drivers
4. Java (JDK)
5. Android Studio (SDK)
6. Eclipse
7. Selenium Standalone JAR
8. Appium Java Client
9. Appium for Windows
10. .NET Framework 4.5
Every Android smartphone contains a secret set of Android developer options which are used by app developers who need additional functions to test their apps they are making for Android devices. It allows you to enable debugging over USB, capture bug reports on your Android device and show CPU usage on the screen to measure the impact of your software.
To enable developer option in the phones to go to the settings, click on the about phone options and click on the build number 7 times to enable the developer options.
Doing so will display a toast message for enabling the developer option. the current message appears if the developer message is already enabled on the phone.
Phone Drivers
The phone can communicate with the PC only if there are proper drivers installed for the USB cable. Each phone manufacturers provides its own drivers for the phone. PDA net is a driver which works with all the Android devices. It makes sure that your phone is detected in DDMS.
Open pdanet.co and go to download screen and download the latest version to install it on your PC.
Let’s verify if the phone is prepared.
The ddms.bat file is present in the Android SDK tools folder.
Double click on the file and you will see the Dalvic Debug Monitor window.
If your DDMS option is enabled i.e. if you have installed the PDA net software and connected the device to the machine, that device should be detected in this monitor. Now let’s move on to the second step.
Test Environment Setup
First, you need to download JDK from the Oracle website.
Once downloaded, install it to your machine. Now you need to set the Java installation path in your Environment variable. Right-click on Computer option in the Start menu and select the Properties option. Select Advanced system settings and then select the Environment variables option in the Advanced tab.
Then select the new option and enter the new variable name as JAVA_HOME.
Set the path of JDK to variable value and then click OK.
To get an Android emulator you need to go to www.developer.android.com\studio#downloads and Scroll down to Command-line tools only section to download the zip file of SDK tools package for Windows.
Don’t download the EXE file. Once the SDK file is downloaded, copy it to the C drive, make a new folder and extract the zip file in the new folder.
Now you need to search for SDK manager in the bins folder and open SDK manager. Once the SDK manager window is open, go to tools and then get into the bin folder and in the address bar type cmd and hit enter. Now in the command prompt type SDK manager “platform-tools” “platforms; Android-28 and then hit enter. So now you will be able to see Platform tools and Platform folders in now folder you created for the extracted files.
Click on platform tools and copy the address bar text and then again go to an environment variable. Then go to the path, a new window will pop up, make a new path and enter the copied text there and then click on OK.
To download Appium go to www.appium.io and click on the Download button.
Under the latest version, click on the Appium-windows-1.12.1.exe file. Once the file is downloaded, open the appium.exe file. Click on the Start Server v1.12.1 button.
Then in the server window click on the Start inspection session icon at the top right corner.
Now you can get started with Appium automation testing. Do read our next blog to get more information about Appium mobile automation and how to inspect elements using Appium Desktop.
Appium is an open source mobile app UI testing framework. You can test all types of mobile apps and perform automation test on physical devices as well as on emulators and simulators. Appium mobile testing does not have a dependency on mobile device OS and it supports cross-platform app testing as a single API works for both Android and iOS. Appium supports many popular languages like C, PHP, Python, C#, Java, Ruby, JavaScript, etc.
What is Appium Mobile Testing?
When Appium is installed then a server is set up on your machine that exposes the REST API. It receives a command request from the client and executes that command on Android or iOS mobile devices. Then it responds back with an HTTP response. It uses mobile test automation frameworks like Apple instruments or UIAutomator2 to drive the UI of apps.
Approaches to Test Automation
There is two approaches for mobile test automation, Image-based and Object-based approach. Let’s understand both in detail.
Image-Based Approach for Test Automation
This technique of object identification is based on the image processing attributes of the objects in the Application Under Test (AUT). Example: Automate user options like “click, type, drag-drop, mouse actions, etc.”
Visual verification of the expected output
• Not dependent on the platform underneath
• Can be used to automate emulators as well as a real device.
The object-based approach of test automation
This technique of test automation is based on recognizing the nativity of the objects in AUT. This nativity reorganization process for each individual object in the application is carried out using different attributes that are assigned to the object.
It is used to extract the application object identifier with its properties from the actual native operating system source code, just like the developer used. This is an accurate and fast method to recognize the buttons, lists and other objects used by the application.
One drawback of the object-based approach is that the recognition of the individual attributes of the object involved restricts these techniques ability to function in test scenarios that require third-party application access. This reduces the automation coverage of utilizing this technique.
Which approach should we choose?
As we have seen, both approaches have their pros and cons. To get better results, you can merge both approaches and think about devising a hybrid test automation solution.
The combination of OCR (image based) and native (object-based) approach allows users to build a single script that will be portable across different devices. It will make your automation robust and efficient and allows the users to confidently detect the relevant native and GUI defect within the mobile application.
Appium Overview and Architecture
Appium is an HTTP server that manages WebDriver sessions. On iOS devices, Appium proxies command to a UI automation script running on Mac Instruments environment. Apple provides an application called instruments which are used to do a lot of activities like profiling, controlling and building iOS apps. It also provides an automation component where you can write some commands in JavaScript which uses UIAutomation APIs which interact with the app user interface. Appium mobile automation uses these same libraries to automate iOS apps.
The Webdriver Script sends a command in the form of HTTP (JSWP) to the Appium Server. Then the Appium Server will then decide as per the request which driver should be initiated. So, in this case, the Appium Server will initiate the XCUITest driver and it will pass the request to the WebdriverAgent which is an IPA (WebdriverAgent.xcproj) developed by Facebook. WebdriverAgent is responsible to send the command to the Application Under Test (AUT) to carry out the actions in the app. Then the response will be sent to the Webdriver Script through the Appium server.
Only iOS 9.3 and above version are supported by the XCUITest Driver. You can find all the capabilities for XCUITest Driver in the link mentioned below.
The situation is very similar in the case of Android where Appium proxies command to a UIAutomator2 test case running on the device. UIAutomator2 is Android’s UI automation framework which supports running JUnit test cases directly into the device into the command line. It uses Java as the programing language but Appium will make it run from any of the web drivers supported languages.
To answer the question “what is Appium automation?”, let’s have a look at the Appium philosophy.
Appium mobile testing is required for proper execution of continuous integration as it speeds up the overall testing process. Appium mobile testing helps to run the repetitive test which would take more time to perform manually. This is why appium mobile testing is the preferred choice for the developers when it comes to test automation.
In the next blog, we will learn more about Appium mobile testing and how to install/configure the Appium setup on a windows machine.
There are more than 5.5 Million apps available on Google Play and Apple App Store combined and by 2022, the number of app downloads will be around 250 Billion. The number of smartphone users in the world is more than 3 Billion and the users prefer using smartphones instead of desktop. This data shows that the mobile app market is very competitive and growing. In this age of digital transformation, the focus of enterprise mobility is to provide better user experience at great speed. The launch of 5G will further complicate the app development and testing process initially. But apart from this, there are three main challenges to overcome and pCloudy is leading the path to bring positive change. Let’s look at these challenges and their solutions.
Challenges Faced by Enterprise Mobility
There is a growing demand for quality at speed in the mobile app industry. The current market dynamics are such that it is necessary to provide the best quality product in less time to remain in the competition. When we talk about mobile app testing, quality can be maintained only when the enterprise can achieve scale. There is a myriad of devices in the market with different OS versions, browsers, Custom ROM, etc. and the app should be compatible with all the devices. How can ensure this and most importantly, how can we speed up the testing process on multiple devices? pCloudy can help you overcome this challenge with its best in the class features. Let’s have a look at them one by one.
Parallel Testing to Achieve Scalability
If we are talking about scalability then testing your app on 5,10 or even 20 devices are not enough for the enterprises to ensure a bug-free app. pCloudy enables to testers to run your test scripts concurrently on 50, 100 or maybe more devices. Enterprises have the option to perform parallel testing in our public cloud or in our plug & play on-premise infrastructure solution. With more than five thousand device browser combinations, enterprises can ensure that the application is working properly in all the devices.
Parallel testing on 50 devices
You can see in the picture above, we have set up a device infrastructure for 50 devices for demonstration. Similar device setup can be provided for our on-premise solution and public cloud users can just log into the pCloudy platform and book the devices to perform testing. These real devices can be accessed remotely on the cloud which makes it more convenient for the users as there are no location constraints.
Progressive Reports to Improve Quality
It won’t be efficient to go through each and every report to analyze the data for every device. Instead of that, you get intelligent results where you can do a quick analysis of the errors and then debug quickly. In the pic below you can see the list of results for different builds with their start time, progress, status, and Action.
In the status, you will be able to see how many tests pass, fail, and errors. This way you can concentrate on the tests which failed and you will save time as you don’t have to manually go through all the cases.
You can integrate any reporting framework with your existing Appium setup. The same way you have to configure with pCloudy by adding a one-line code while creating an asset.
In the pic above you can see the test case result in detail. You can see the tags, test case name, build, Automation, and the live status of errors.
Once you expand the test case you can see the logs, Jason format, screenshots, video for that particular session. This information can be used to expedite the test execution.
Integration with AI to Speed up Mobile App Testing
The next challenge is to speed up and simplify the test script creation process. Usually, the procedure is to connect the device to the system and use an automation framework like Appium to use object repositories. If you are using images then the XPath may change depending on the devices.
This AI based utility will help you to get the object repository for the entire page. So if there is an icon for contacts then this utility will automatically recognize the element.
In the code, you just need to declare the driver for Appium and for pCloudy as well. Then you need to initialize the drivers and start using the code generated using the AI utility. This is how the script creation is expedited using AI.
Conclusion
To sum up, pCloudy has come forward with solutions for the enterprise mobility to deliver quality at speed with scale. Whether it’s parallel testing, progressive reports or creating a script using AI-based utility, all these features fix the problems that were mentioned earlier. The proverb “change is the only constant” fits well as pCloudy comes up with updates on a regular basis to resolve the issues and make mobile app testing better.
These days software firms are putting in a lot of effort to attain quality at speed. Whether it’s continuous testing, agile or bringing AI into automation, it is required to refine the software development process to keep up with the rapid technological changes. The best way to do that is to adopt practices like parallel testing which will save time and effort. So let’s get into the details of parallel testing and understand the benefits and challenges faced to implement it into the software development life cycle.
What is Parallel testing?
Parallel testing is defined as a type of software testing where multiple apps or features are tested simultaneously which reduces time consumption.
Mostly it is done when a new version is released for an app. There is a need to check the compatibility of the latest version of different device, browsers and OS combination.
For example, if there is a new version of the OS being launched, then there is a need for testing the app to check if all the features are working fine with the new OS version.
So parallel testing can be used when the app is to be tested on the different version of the operating system or when there is a new version of the app to be launched.
To perform parallel testing you can simply create slave projects for the different functionalities of the app and a master project that will run these projects.
Why and When to Perform Parallel Testing
Parallel testing is done to make sure if the new version of the app works fine or to check the consistency between the old and the new version. It can also be done to check the integrity of the new app and the data format between the versions. When a company is moving from the old system to new system and synchronization is performed, it becomes necessary to check the legacy data is imported from one system to another and all the outcomes are defined more accurately especially in the financial domain.
Benefits of Parallel testing
Parallel testing has become a need for the QA teams as the benefits cannot be neglected. Let’s have a look at some of the benefits of performing parallel testing.
Better Development Flow: Better reporting and quick feedback with parallel testing enable testers to share the results with the development team. This practice also boosts the Continuous Integration efforts which lead to the timely delivery of the product. Parallel testing also helps build better communication between the QA team and the developers as the testing and reporting are done in a timely manner.
Less Effort: Needless to say manual testing is very important but doing regression testing again and again on different devices combinations will be time-consuming. Performing parallel testing will reduce the amount of work that you have to put in as you will have to work on one script only. That script will be replicated on multiple devices.
Time-Saving: Parallel testing saves your time by a lot and that time can be invested in exploratory testing. Even if you are testing your app on two devices, your testing speed increases by two times. Human error can give you incorrect data so it’s better to use parallel testing for such repetitive testing which can also be boring.
Better Test Coverage: The test coverage increases when you test your app on various device browser-OS combinations. The more devices you test your app on the better. This will ensure that the end user gets the best experience irrespective of the device used. In this way, parallel testing ensures better user experience which means better ROI.
Less script: You can just write the script once and rerun the test on multiple devices. The older and newer version of your app can be installed on different devices to check compatibility and consistency. The goal is to make sure that the newer version of the app works fine on the latest and the legacy devices similar to the older version with some updates.
Challenges of Parallel testing
Parallel testing has its own sets of challenges and It is always better to get familiar with the issues before making a decision. Although there are solutions to these challenges which are also mentioned below.
Infrastructure Limitations: The cost to set up the test environment can take a toll on the company’s finance as the infrastructure and maintenance will cost a lot. Mobile and networking devices will add to the cost. Also, the company will have to hire highly skilled professionals just to maintain the setup. Instead of this companies can opt for cloud-based services like pCloudy where they will get all the desired devices to test on which can be accessed from anywhere anytime.
Data Dependencies: It might be difficult to make a strategy for parallel testing if the test cases are dependent on some specific data. So if we take a case of sign up page. There can be test scripts running for a user login but if not done in a sequential manner then any other script might run first which would delete the user credentials form the database. Another script might also run which will require a user to enter the data. In this case, it will be difficult to integrate the scripts into parallel testing due to data dependencies. This is why it is required to write the test scripts which are truly data independent. Therefore the test scripts should have the configured data which is required for the test run and then the scripts can be modified to run in parallel.
Conclusion
To sum up, parallel testing reduces the effort and time spent on testing the app by multiples. It also increases your test coverage and helps provide a better user experience. The best way to do parallel testing is by using the cloud platform as you will not have to invest in the infrastructure and you will be able to access the devices from anywhere anytime. So if your testing team is distributed across the globe, this will be your best option and all that extra money saved in the maintenance can be put to good use for some other feature development process.
Mobile Responsive Test across range of Mobile Devices and platforms is extremely important for superior User Experience. However, Mobile Responsive Test and debugging can get knotty, especially when there are issues like obfuscated HTML/CSS/javascript code, improperly implemented HTTPS, browser inconsistencies, page load performance and so on. So how do you overcome these issues? Using a Web Developer Tools or debugging extensions make debugging your web application easier. Here we’ll talk about the most popular one – Google Chrome Developer Tools.
The Google Chrome Developer Tools is a consolidated environment built into Google Chrome Browser. It is used for web authoring, debugging, monitoring, optimizing, and understanding web applications or websites. (Sourced from Google Chrome DevTools)
A responsive testing simulator provides a virtual environment where you can test the functionalities just like in a real device. You can also test websites on mobile devices online and test websites on different screen sizes. It is easy to perform responsive design testing across devices on a simulator.
Inspect and edit the DOM and CSS of your website in real-time
Run code snippets from any page
Debug Obfuscated code
Set advanced breakpoints and debug JavaScript using graphical debugger
Troubleshoot Security issues
Analyze run-time and page performance to improve the speed of your app
Trace reflows and repaints as your app loads
The Chrome DevTools provides almost all the gears required to inspect, debug and monitor the performance of a Web App/mobile web. It also provides additional capabilities to simulate Mobile Devices to test and build mobile-first, responsive web sites. You can even debug web content on a real Android Device from a development machine. But, is it enough to perform Mobile responsive test on one or two devices?
Remote debugging using Chrome DevTools in integration with pCloudy’s Mobile Device Lab
Device fragmentation is a very real problem when testing Web apps on real devices, particularly for UI and functional tests. The number of devices with unique specifications and proprietary build variations has increased. At an advanced level, to perform Mobile Responsive Testing and to check performance of the Web App on real devices, you will need to experiment your site on various platforms, across a number of device form factors. pCloudy’s Mobile Device Lab offers complete integration with Chrome DevTools to test your apps on real devices. This integration allows you to test Web Apps on a myriad of android devices with different sizes and configurations.
Here’s how most App developers test their Web Apps
Inspect and live edit your site’s design and content
Test and edit your site in live mode by inspecting all the HTML and CSS in your page. In the Elements Panel, you can live-edit a DOM node by simply double-clicking a selected element and making changes.
You can even live-edit style property names and values in the Styles pane. Examine and edit box model parameters.
It is also possible to edit the current element’s box model parameters by making changes to top, bottom, left, right values for the current element’s padding, border, and margin properties using the Computed pane.
Remote Debugging on real devices to fix errors in the code
Debugging Obfuscated Code: Make your code readable and easy to debug even after you’ve combined, minified or compiled it. You can easily change the formatting of your minimized code by clicking on { }.
Test and optimize the performance of the Web App
The Network Panel allows you to see how your page renders, and time taken to render from start to end. To do this click into the “Network” panel, click on the camera icon and refresh the page on the device.
The Timeline panel helps you to record and analyze all the activities in your application as it runs. It’s the best place to start investigating perceived performance issues in your application.
To make a recording of a page load, open the Timeline panel, open the page that you want to record, and then reload the page. The Timeline panel automatically records the page reload. You can also record page interactions on the page and view the performance details through the recorded timeline.
Here are some more details you can view using the Record Timeline
Record the Timeline to analyze every event that occurred after a page load or a user interaction
View FPS, CPU, and network requests in the Overview pane
Click on an event within the Flame Chart to view details about it
Zoom in on a section of a recording to make analysis easier
There are some things to be kept in mind while performing responsive testing:
Device selection
It is better to ask your business analyst or your manager which devices we need to perform the testing. Creating a device matrix is a tedious job due to device fragmentation with frequent browser and OS updates. Even if you spend a lot of time on making a strategy for the combination of devices, that may not get approved by the product manager or scrum master.
Automation testing and Manual testing
It’s better to use both automated and manual approach to perform testing. Based on the type of testing you can share the effort like 70% automated and 30% manual. It is obvious that sometimes automated testing will miss something that a human eye can see. A tool clicking on a link or zooming in and out is different from doing it manually.
Emulators, simulators and real devices
If you wish to test your app on many verities of devices then you can also opt for emulators and simulators. It won’t be possible for you to set up a device lab as it will cost a lot. So you can use a real device that is popular in the market and emulators for the devices that are not that popular.
Prioritize
Mobile app testing including mobile responsive testing needs to be perfectly prioritized as testing is a time-consuming process. Responsive testing priority should be discussed with the BA and the product manager well in advance and take some suggestions from them. The issue is that there are a plethora of devices and while prioritizing, the most popular devices will be used first for testing.
Conclusion
“Using the right tools for the right job” is an old aphorism that applies to all domains including Software Development. As we’ve seen, Chrome DevTools has several features that help you develop better, debug faster, and measure efficiently the performance of your website or application. An ideal choice for you for Mobile Responsive Testing.
The features discussed in this blog are some of the most common ones used by developers, designers, and testers to write, debug, monitor and optimize Websites and Web Apps. Using these features in integration with Mobile Device Lab like pCloudy can transform the entire process of creating, testing and launching Websites and Web Apps.
Since there are many players in the mobile app market, the competition is endless. To beat the heat and to keep providing an enhanced mobile app experience to the users, the app needs to be constantly updated with the changes happening and for doing this, there are different types of testing techniques. Let us have an insight into the types of testing methods.
1. Compatibility Testing
Compatibility testing is a critical QA task. It assures that a given application works as intended with selected operating systems, selected devices with different screen sizes resolutions, and internal hardware (memory size, processor speed, and button/input differences). It defines the feasible compatibility combinations of devices and interfaces for a specific testing assignment, in concurrence with the customer’s requirement.
Want to test your Mobile App?
Join pCloudy Platform!
Signup for Free
Compatibility Factors:
There are different factors that play a very significant role in mobile compatibility testing strategy. Mobile operating systems and their supported versions, mobile devices developed by different manufacturers, different device screen sizes with different resolutions and internal hardware including different input types, processor speeds and memory size of devices.
Let’s develop a test strategy for compatibility testing. The first step is to identify the Operating systems and devices based on key factors described in the previous slide. Once you identified the devices based on different compatibility factors you need to set up your environment for testing. You can either set up your internal test lab or you can go for external cloud providers like pCloudy. Now, the next step is to execute your test cases on a defined set of environment. Report the defects to appear while testing and report these defects to your development team.
The main purpose of compatibility testing is to ensure whether different software is compatible with different configurations and whether the application is compatible with the client’s environment.
2. Installation Testing
Mobile phones come with different types of mobile apps like Native and Web and Hybrid. Installation testing is a type of testing that is done at the initial stage of mobile app life span or maybe it is the first impression on the users, in other words. Installation testing checks whether the mobile app installs, uninstalls and updates properly without any interruption.
Want to test your Mobile App?
Join pCloudy Platform!
Signup for Free
Installation Testing Factors:
There are some of the factors that play a significant role in installation testing of an application. Factors like which platforms and operating systems supported by the application and how they will be distributed whether it is directly available in the app store or are going to get the installable files on mail itself or is it available on an app distribution platform and can be accessed through public URL. The user behavior regarding installation changes from person to person. For eg: the developer may install the app (By using ADB install command) and a different user might directly install it from the app store or public URL.
Important points to be noted in Installation Testing:
Some of the common checkpoints which can be converted to standard test cases are:
Installing the app in phone memory.
Installing the app in phone memory is full.
Installing the app on an external SD card.
Installing the app on an external SD card is full.
Check the memory space after installing. It should not occupy more than promised.
Check if it is releasing the space after removing
Another part of the installation and uninstallation testing is the testing of patches/updates
These are the common situation when we need to test app after updates.
Check for updates:
Can I download an update?
Can I update the app when there are multiple updates available?
What happens if I don’t update?
When App is upgraded:
Check that the user is still logged in
Check that data is still the same
Backward compatibility:-Check that database changes will not affect the current data.
Test the update from an older version to the current version
When OS is upgraded:
Check the app compatibility.
App updated gracefully without a crash.
3. Interruption Testing
Interrupt testing is a process to replicate abrupt (Unexpected) interrupt to the application. It is done to understand how the app behaves under certain interruptions before resuming to its original state. This can be achieved in various ways and techniques depending on the application under test.
Following are some Common Interrupts:
1. A phone call when the application is running or is in background
2. Battery removal when the application is running or is in the background, battery high, battery low, incoming call or SMS
3. Plugged in or out of charging
4. Device shut down
5. OS upgrade
6. Network loss and restoration, etc.
Some specific scenarios for an application using a network:
1. Connect to the network but remove LAN connection from the router so the device can sense wifi state on the device but cannot connect to the internet
2. Connection via VPN and VPN disconnected
Scenario for Application using services:
1. Kill service by clicking on the recent button and swiping the application right to kill app and services
2. Kill app using third-party App killer
3. Kill specific services from Settings->Manage Applications
4. Scenarios for Application Linked to account Manager
Remove account from Settings->Account Manager
Incoming and Outgoing SMS and MMS:
An application is running in your mobile and you send an SMS/MMS from another mobile to your mobile. In this situation, SMS/MMS notification should display for a while and then close and the application should continue to run.
Incoming and Outgoing calls:
You are in a login page trying to log in and you have just pressed the submit button that will take you to the application home page. This is a transition period because loading is happening.If we receive a call during this time then the application should not end abruptly or crash.
Battery Removal:
If the application is running, remove the battery of your mobile and it will shut down. After rebooting the mobile, run the application again, it should run smoothly and without a crash.
Data Cable Insertion and Removal:
When the application is running, connect the USB Cable, the system should display “USB Connection Screen” and if the user returns to the application then the application should not crash or end abruptly
Interruption Testing Strategy
Let’s prepare a test strategy to test interruption testing:
The first step in this process is to decide the critical interrupts based on your application components and architecture. For example, if your application is using GPS include common interrupts related to GPS. Let’s Include two type of interrupts:
Application specific interrupts:
Prepare your test cases and try to fit theses interruption test cases with state of the application. For example, interrupt your Network while performing payment.
Interrupt testing process:
List of common Interrupts:
Phone calls, Text messages, App notifications, Battery Warning, Forced updates, Voicemail, Shaking the Phone, Different Gestures, Switching between apps, Locking and unlocking the screen, Changing the orientation, Music playing while using the app, Out of memory (general performance interruptions), Data app interruptions (What Sapp, Viber, Tango), Audio interrupts from multiple sources (iPod, Media player, Other audio apps), phone in standby mode, switch the network to plane mode.
Tools to test interruption Testing:
Use tools like Joule Unit (Android), On iOS use the Energy Usage Function of Instruments, Monkey for Android to stress test your app and see the handling of the interrupts, UI Auto Monkey for iOS
Scenario
A) Once the user connects the mobile earphone jack with Nokia Lumia and starts music Player.
B) Now when the user resumes the application the cart gets empty.
4. Localization Testing
This type of testing is a technique in which we check whether the mobile app adheres to the local cultural settings, customizing the apps according to target country and language and also according to the linguistic aspects. Localization is also known as “L10N” as there are 10 characters between L & N. For eg: Chinese language-search engine offered by Baidu beats Google in China. Baidu does better than Google because it looks and feels fully native to the Chinese speaking audience.
Also, regional language speakers can’t get many games in their own language, so they turn to race games, which are easy to understand. We must have noticed in these examples, language is the common issue. Apart from Translation there are different other elements in the localization, like Usage of proper Time zone, local formats of dates, addresses and phone numbers, Design and layout to fit translated text, Converting to local requirements (such as currencies and units of measure), Adapting graphics to target markets, Modifying content to suit the tastes and consumption habits of other markets, Addressing local regulations and legal requirements.
Why should we localize?
Localization is important in order to understand the cultural and linguistic aspects of a particular region. While testing, testers keep repeating the tests to check typographical, linguistic errors, cultural appropriateness of the UI, etc. Research says:
1. 56.2 percent of consumers say that the ability to obtain information in their own language is more important than price. (Common Sense Advisory, Can’t Read, Won’t Buy: Why Language Matters on Global Websites, 2006)
2. 74 percent of multinational enterprises believe it is either important or most important to achieve increased revenues from global operations. (California State University at Chico, 2007.
3. 95 percent of Chinese online consumers indicate a greater comfort level with websites in their language. (Forrester Research, Translation and Localization of Retail Web Sites, 2009)
4. A critical success factor for cross-border merger and acquisition deals is the ability to communicate information clearly and accurately in multiple languages. (Merrill Corporation, How to do Better Multinational M&A Deals, 2008)
Types of Localization Testing:
In the localization testing, validating whether a mobile app is capable of performing given a geographical location, in particular, is found out. It includes four major types of testing like linguistic, culture, cosmetic, and functional tests. 1. Linguistic testing– ensures that the UI text appears in the congregation with the language and is not mistranslated or misapplied. This requires language skill and knowledge of the product. Mistranslation, irrelevant usage of the language are few of the errors in this.
2. Cultural Testing– Few words may imply different meanings in different cultures. So, this has to be translated properly in relation to the target culture.
3. Cosmetic Testing– For easy understanding, let us use this scenario. For eg. In the apps used in Afghanistan, the text will be in Arabic and will start from right to left unlike other languages like English, French that are read from Left to right.
4. Functional Testing– In this testing, testers follow prescribed test scripts to run through all aspects of the product to make sure it functions as designed in a localized environment. Often functional testing includes verification that the localized product is compatible with various operating systems and third-party products. Functional testing generally requires a good knowledge of the target language and total familiarity with the product. Some of the major Functional testing problems may include– Incorrect or missing UI elements, graphics, and windows, Incorrect error messages, Localization bugs that may cause software crashes, Errors generated by installing the localized software on a localized OS.
5. Performance Testing
Performance testing is to test the performance of the mobile application in expected workload scenarios and to eliminate the performance hurdles. It checks whether the response of the app is quick, the ability of the app to tale load, and app stability in those load situations. Performance is very critical because if the app is malfunctioning, it is more possible that the user might uninstall the app and might shift to a competitor’s app which is better.
App start-up time is the most critical performance parameter from the user’s point of view. It shouldn’t take more than 2 seconds for the app’s first screen to pop up after the user taps on the app’s icon. Battery time is the second most important issue. Some apps consume a high amount of battery life. Excessive resource usage creates a burden on the processor and the phone heats up. In some cases, it is observed that the new app installed uses the same amount of battery as the OS. If more features are added in the app its memory consumption also increases. This is why memory consumption should be checked while testing the app.
Network Performance
The app needs to be tested on different mobile networks and network properties. If there is packet loss then the app should generate alerts or resend the request for the information. Jitters or delay in receiving information
Server/API Performance
Performance is directly proportional to the response time from the Server. In this, tests are run to check how efficient the app is to handle data transfers from the server and how quickly and in what format the data is transported by the app. The number of API calls generated by apps should be lesser and how and where the data is saved in the ‘server down’ case is all that is tested here.
Mobile app testing tools make it easier for the testers to identify and rectify any error that comes up. For different OS there are a different set of Mobile app tools. For Android, we can use Robotium and Monkey Runner and for IOS we can use Automator.
6. Usability Testing
Usability testing is a type of testing which is performed to check how user-friendly the mobile application is in terms of the navigation, ease to use the app, flexible app controls, etc. It is also known as user experience testing.
What matters in Usability Testing?
Workflow: To understand, below image represents the workflow of how the user reaches its goal while navigating through the app controls.
We must understand what the user needs to do in order to use the app’s functionality. If the workflow was overlong and included repetitive actions It may annoy the user while performing the task. Design and Layout: Good layout and design allow a user to easily complete tasks. An app need not offer so many features or content that the app is difficult to navigate. A friendly app has a design that allows less user text entry, helps differentiation between the selected items, finger-sized targets, should provide intuitive hints to texts, etc.
The response time of the application: It is one of the key factors. This means how long the user takes to complete a task. Often long delays while the app interacts with the server hinders the user experience standards.
The emotional state of users: Emotional state of user measures the user’s motivation to use the app. The apps should be smart enough to understand what the users want and how to engage them, this can be achieved by usability testing.
Usability Testing Strategy:
The first step is to identify users who match your expected audience. Watch and listen to them as they use your products. You can use video and audio recording equipment for archiving test sessions, in case later review is needed. However, our primary tool is taking notes with paper and pencil! This is still the fastest way we have found to get information quickly to developers and designers. You have your data. You now need to organize and analyze it. How the results are absorbed as a report in order to be useful for product changes.
Tools and techniques for usability testing
1. Mr. Tappy: It is a tool set up to capture how users interact with the apps naturally. The user data so generated could be used for better user experience.
2. Paper Prototyping: It is a technique that consists of creating hand drawings of user interfaces in order to enable them to be rapidly designed, simulated and tested. As simple as this technique may look like, it has been effectively used since the 1980s and is likely to continue to be used with a guaranteed degree of success for many more years to come.
3. Magitest: It allows you to perform mobile user testing of native apps and websites. Captures the participant’s voice and facial expressions during the session. Here the users complete the tasks you specify and allows to listen to them speak their thoughts aloud as they interact with your mobile app.
4. Reflector: This app allows us to see the iPad, iPod, iPhone screens on MAC or PC. The devices connect wirelessly to proceed with further functions.
7. Conformance Testing
Conformance/compliance testing is a type of testing which is performed to validate if the application is meeting the Market place and Enterprise policy guidelines. It is like an audit done to check whether certain standards are being met or not. Conformance Testing is focused on two main areas- App store guidelines and Enterprise policy compliance. Each Market place has their own guidelines. We need to validate if the application is following the guidelines of each of these app stores.
1. App store Guidelines pertaining to User Interface, Media Content, Privacy, Spyware and Phishing, nudity, religion, culture, violence, etc.
2. Enterprise Policy Compliance pertaining to a different set of industrial standards. For eg: For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, In healthcare, the Health Insurance Portability and Accountability Act (HIPAA) includes an administrative simplification section, which mandates protecting the information assets of patients. So, it is a dire need for the testing organizations to follow the necessary guidelines and compliances to limit the risk that non-compliance might follow.
Reap the benefits of different types of testing
In this day and age, mobile app quality matters the most as the market is very competitive. We need to focus on user experience as well as the performance of the app. If the app contains bugs the app ratings will go down and so will the number of app downloads. If we can make our app bug free than the chances of app gaining popularity will increase. This can be achieved by performing different types of testing on the mobile application. In the next blog in this series, we will talk about mobile app testing strategies.
‘Metrics’. This term has become one of the widely used terms today in enterprise mobility especially for measuring mobile app performance.
Let us first understand why are the metrics are so important to analyze the performance of an app.
Generally, app developers consider mobile app performance testing as an afterthought. Mobility teams have to meet stringent deadlines and for them, testing the performance of an app has the least priority. To them, passing the functional tests meet all critical criteria for launching an app. Well, let us find out why is performance such an important criterion for mobile app testing.
Reasons why measuring Mobile App performance is such an important criterion:
1. Speed wins: As per the Compuware report, users expect their apps to load in about two seconds. This can even be proven right by the SEO reports from Google, which considers the site speed as an important parameter for Page ranking.
It is important for mobility teams to understand that your users want their apps to respond in about two seconds or less. Anything beyond two seconds, and you might lose your business.
Once the app starts taking anywhere close to five or 10 seconds, users start shifting to your competitors.
2. Mobile App Error Rates are probably much higher than it seems: You must have struggled with issues like images not loading, network errors, content feeds leaving blank holes in the app, timeouts, crashes, freezing, upload failure etc. Most of these issues are not errors of apps but caused by unreliable networks.
The error rates are highly variable and depend on network conditions. In the United States, we’ll typically see 3% error rates, whereas in India, China, and on challenging networks in Europe and Asia, we see error rates up to a frustrating 12%. Not measuring and then getting a handle on mobile app errors will quickly result in frustrated customers, uninstalls and 1-star reviews in the app stores. And no one wants that.
3. 5 star app rating: Reviews and ratings have become important aspects of judgment for any user before downloading your app. And the app should be perfect in all the ways to ensure a 5 star rating.
If you focus on functional tests and miss out on the mobile app performance not knowing, if too much battery is drained on a particular device, or is consuming too much data, then your Playstore might be full of negative reviews even before you market it well.
In this post, you’ll know how app monitoring performance works so that you can get the best out of your Smartphone software.
What is application performance management? 10 APM features that we can tick right off our fingers are:
Troubleshooting of disruption of frequent speed and uptime
Monitoring of the speed at which transactions are performed by end-users and systems
Use of a single integrated suite of software tools
Load-testing
Synthetic monitoring
Real-use monitoring
Root-cause analysis
Web-performance monitoring, to gauge app speed and uptime
An end-to-end overview of bottlenecks
Review of service interruptions
APM is integrated through the lifecycle of an application including the pre-deployment and post-deployment stages. It is therefore important for developers, testers, and business teams. A slow load can be prevalent due to a host of issues related to APIs, servers, or browsers. APM techniques are what help pinpoint the actual problem.
Basic availability monitoring involves testing IP protocols and network services, but a user needs to establish some optimal performance thresholds and real-time alerts to monitor and manage both speed and reliability.
Tips for monitoring app performance
Performance monitoring must take into account certain things in order to optimize user experience. These include measuring from the user’s point of view, i.e. performance, stability, and resource utilization in real time the way the user perceives them.
The context is crucial too. It’s important to determine how many users were affected by a slowdown, and what action on a specific operating device prompted the disruption. To truly experience a ‘mobile moment,’ you should follow these tips for monitoring your app performance:
a) Measure Frame rendering time
Only measuring network time doesn’t cut it out for users, who are interested in when the network resumes as well as when it responds.
Measuring response time by the device or operating system allows the product manager to focus on one platform at a time, respond to performance issues, and deploy resources accordingly.
b) Measure battery usage
It’s crucial to measure the battery consumption of your mobile app, along with data plan impact and memory usage. If the app is responsible for draining your battery, you should bring it in line with best practices.
Frequent updates can also be minimized so that user satisfaction can be increased.
c) Measure cell data consumption
Recent studies by Dimensional Research have shown that 20% of respondents stopped using apps that were heavy on data usage. This indicates that you should measure the consumption by app version, the total number of users, KBs of data consumed per minute, the amount of data being transferred per hit, and the percentage of data being consumed per hit.
d) Measure errors
HTTP errors from certain URLs need to be captured. This will help track the number of users who received an error, the actions that resulted in it, number of users with errors on app launch, the percentage of actions that led to at least one error, and the app version.
pCloudy helps you with performance profiling of your mobile apps. You can view the CPU usage, Memory usage, Data usage and Battery consumption while performing a series of actions on the app. The Real time mobile app performance monitoring feature helps you to keep track of the app’s performance while conducting manual tests. You can even check the performance consistency of the app by running automation test suites in parallel on multiple devices and by also simulating different network environments.
App Performance while conducting Manual Tests
The performance of an app can be viewed real-time while conducting functional tests. You can simply select the application installed on the device from the Tools section to view a graph of Memory and CPU usage, with the packets/bytes consumed in the Data Usage.
All information captured during a testing session is gathered by the platform and stored in a secure location to be used later. When a tester completes his testing session using a device, the performance related information can be found in the Cloud Drive.
Battery Consumption
The Battery Consumption graph indicates Battery consumption by Device, Battery consumption by Application and Battery consumption in %.
CPU Consumption
The CPU consumption of an App is plotted as CPU consumption in percentage against Time Duration in seconds.
Memory Consumption
The CPU consumption of an App is plotted as Memory consumption in ‘Mb’ against Time Duration in seconds.
You can listen to this webinar on Performance testing to find out how a cloud based solution can help your app perform well in all situations.
