Suyash is a content strategist at pCloudy. He is a frequent contributor to the world's leading mobile technology blogs and tech forums. In his spare time, you will find him reading detective novels, watching a documentary or exploring a new destination.
End to end testing is done to make sure that the application behaves as expected and the flow of actions from starting to end is complete without any issue. To explain with an example, suppose you are the end user and you are using the app to book a flight. So first you will launch the app and click on the flight’s icon on the home page. Then you will enter the destination, date and other required details. Now you will choose your preferred flight, book the seat, enter the passenger details and make the payment. Once this is done, you will get the ticket and a follow-up email on your registered email ID.
Now in this scenario, if the screen takes a lot of time to load the next page or if the book button does not work then this will be an unpleasant experience for the user. So we can figure out if there are any issues from starting to the end of the process. It’s not just about the UI functioning but also the data encryption and security, information flow between other internally used platforms integrated to the app, proper functioning of the firewall and more. This is where end to end testing comes into the picture to make sure that everything works well under any circumstances.
How to Perform End to End Testing
While performing end to end testing the first thing to do is to analyze the requirements and set up the test environment. Then you need to evaluate the main system with the connected subsystems and define the responsibilities of all systems involved. Make a list of the testing methods and the description of the standards to be followed. Once that is done you can create the test cases and track the requirement matrix. Finally, you need to save the output and input data before testing each system.
Measuring the success of the end to end testing
To perform end to end testing efficiently and save time and effort, a matrix should be created. Some parameters should be defined to gauge the success of the testing performed. These measurements will help you define the testing matrix:
Test case status: A proper visualization of the test cases prepared can be observed through graphs and compared with the number of tests that were initially planned. Tracking Test progress: This measurement will help you to get the details of the tests completed on a weekly basis. It will be easy to keep a track of the test that failed and those that passed. Defect details: Tracking of the bugs occurred and the issues surfaced should be done one a weekly basis and keeping a track of the defects that were opened and closed on the basis of priority. Environment availability: Here we need to take a note of the amount of time allotted for running a test in an environment and the time actually spent on performing end to end testing.
Best Practices to Perform End to End Testing
End to end testing can be very time consuming and we can save some time and effort by following some steps. It is always better to plan in advance before initiating the testing process.
Prioritize: Chances of defect increases as the test cases become complex with more components and it becomes difficult to debug them. Therefore it is better to perform unit testing and integration testing to resolve issues that are easier to detect. After initiating the End to end testing you can also start doing smoke and sanity testing to keep an eye on the high-risk scenarios. Manage the Environment: The requirements need to be documented and the system administrator needs to be informed about it to meet the requirements before starting the testing. The system setup should be proficient enough to run the test cases and keep your staging environment close to the production. Optimize: After test completion, you need to clear the test data so that the environment is restored and can be used for testing again. This will make the environment ready to start at any moment. Risk Analysis: Focus on the features that are crucial and on failure can cause a high risk. Prepare a risk assessment matrix which will come in handy in these situations. Limit Exception testing: Use low-level unit testing and integration testing for exception scenarios and use end to end testing to deal with typical user scenarios.
Conclusion
With the rapid enhancements in the app technology and the increasing use of technologies like IOT, the development teams have to make sure that the app has a smooth functionality. The end users can be very choosy in this competitive market and so it’s necessary to perform end to end testing of applications to provide a good user experience.
We are glad to announce the release of pCloudy 5.4 with exciting new features to make app testing simpler than ever. There is a possibility that you might be familiar with pCloudy, but for those who are new, pCloudy is a cloud-based mobile app testing platform. We have achieved many milestones in a short period of time because we resolve issues immediately and bring in new features on a regular basis. Case in point: This new version of pCloudy, where we have addressed the concerns of our users and added new features to optimize testing. Let’s have a look at all the features added in pCloudy 5.4.
CTRL + V
To test your app on a pCloudy device you need to login to device.pcloudy.com and then book a device by clicking on the devices tab. Just click on the available button next to the chosen device which will take you to the device interaction page. Earlier in the device window, you would have to use the virtual extended keyboard to enter text in the device. Although now you can just copy any text from your system and paste it in the device directly. So now instead of using an extended keyboard just use two buttons to enter the string in the device. This will save time and effort when working with multiple devices where you need to enter long text many times.
FollowMe new UI
To use the FollowMe feature you will have to instrument the app you want to perform the test on. Under devices section, click on the FollowMe tab then in the next screen choose the app, select the duration and devices on which you will perform the test. In the next screen, you will see the devices which you selected and two options (Softkeys and Screenshots) at the top right corner of the window. Now when you perform any action in the master device, that action gets replicated on the secondary device too. If you enter text, tap on a button, swipe, etc. in the master device then that will happen simultaneously in the secondary device.
In pCloudy, you can run Appium tests on multiple devices in parallel. To perform a test using Appium, you need to feed in the capabilities. Some users want to test their apps more than once to perform different types of tests on the same app. To do that earlier the users had to install the app again and again. But now we have released an update through which you can perform different types of testing using Appium on the preinstalled apps. This will save you time and make the process easier.
So if you have already installed the app in the device and you want to perform a test on the same app then you don’t need to pass the Application Name capability.
pCloudy is integrated with the most popular tools and frameworks to make testing convenient. A lot of users also requested to integrate Github repository and so we did it. Now you can directly upload your tasks and raise defects in the GitHub repository from pCloudy. Again this feature is added to make it convenient for our users to test and manage the tasks through pCloudy.
To make use of GitHub just login to device.pcloudy.com and click on the user name at the top right corner of the window. In the dropdown list click on settings and in the settings window click on GitHub tab and enter the URL and access key to register. Now in the device session screen, you can click on Collaborate and then select Log a bug in GitHub. A new window will open where you will be able to see the list of repositories, enter the issue type, enter the title, add the description and then click on the click on the Log A Bug In GitHub button. Then you can go to the GitHub to check all the issues raised whenever you want.
Summary
We are working on some more new features and very soon we will come up with the new version of pCloudy. This is a constant process and we always try to address the users issues and provide them updates on a regular basis. This is why we are achieving new heights and we will keep on working hard to provide the best service.
DevOps helps enterprises to build software at a fast pace and with minimal issues. The time to market is accelerated and the bugs are fixed faster in continuous deployment with the help of automated tools. AI is much in line with DevOps as the main focus is on automating the process and with AI the system can identify patterns, anticipate issues and provide solutions. The proactive approach improves the overall efficiency of the software development life cycle. So let’s have a look at how AI is transforming DevOps.
Feedback Loop and Correlate Data
The main role of DevOps is to take continuous feedback at every stage of the process. often people use performance monitoring tools to get feedback on running applications. These tools gather much information in the form of log files, data sheets, performance matrix, and other types. The monitoring tools use machine learning to identify the issues early and make suggestions. The DevOps teams use these suggestions to make the necessary improvements to the application. Many times teams use two or more tools to monitor the health of the app and the data from all the platforms can be correlated by the help of machine learning to get a more deep understanding of the app functioning.
Software Testing
AI is changing DevOps for good by enhancing the software development process and making testing more efficient. Whether it is regression testing, user acceptance testing or functional testing, these all produce a large amount of data. AI can figure out patterns in the data collected in the form of results and identify poor coding practices which produce a lot of errors. This information can be used by the DevOps teams to increase their efficiency.
Anomaly Detection
DevSecOps is one of the essential aspects of software development as security is the key to any successful software implementation. Distribution denial of service attacks are increasing and the business needs to prepare themselves to protect their security systems from hackers. DevSecOps can be augmented using artificial intelligence to enhance security by central logging architecture to record threats and running machine learning based anomaly detection. This will help businesses proactively attenuate the attack from hackers and DDOS.
Alerts
DevOps approach might create scenarios where the team receive an overwhelming amount of alerts without any priority tag. This will create ruckus in the teams as it will be very difficult to handle all the alerts in the continuous development environment. AI can help in this scenario by tagging the alerts and prioritizing them so that the urgent ones can be worked upon immediately.
Root Cause Analysis
To fix an issue permanently, a root cause analysis is necessary. Although it might take time to do it compared to fixing the issue with a patch which will provide the instant solution. In order to find the root cause of an issue, the developers will have to spend time which will delay the release of the product. AI can speed up the process by finding patterns in the data collected and implement to fix the root cause.
The collected data can be used by implementing AI to find a pattern and speeding up the development process. The organized data is more useful and makes prediction possible. The best practice is to use machine learning to automate the tasks which are time-consuming which will ensure the smooth and effective functioning of the DevOps teams.
Software testing has evolved a lot since the time when the waterfall model was used. All the work was done in a sequential manner and only after the development phase was complete the testers used to test the product. Testers used to find bugs but a lot of time and energy was wasted in the process to rebuild and code again.
Now companies are using an Agile model where the main goal is to find the bugs in continuous development, fix them quickly and release the app faster. There is a need to improve the automated testing process to complement the manual testing. More emphasis has been given to CI, CD, and DevOps to make the software development effective.
There has been a considerable change in the functioning of testing tools and test automation frameworks. The most important change is the introduction of AI in a test automation strategy.
According to G2Crowd, AI-powered bots are expected to cut business cost by $8 billion by 2022. Testing bots are already empowering automation testing and will play a major role in reducing the time and effort spent in mobile app testing.
Let’s have a look at how AI is breaking new ground for test automation.
1. Running automated tests that matter
It’s not a good strategy to run your entire test suite due to a very small change in your app that you couldn’t trace. You are probably already generating a lot of data from your test runs if you are doing continuous integration. But it will take a lot of time to go through the data and search for common patterns. So you need to know if you make a small change in code then what is the minimum number of test you need to run to figure out if the change is needed or not.
2. Reducing maintenance and eliminating flaky test
We can run several automated tests on a daily basis to ensure the functionalities of the app are still stable. Although, if we find out that half of this test failed. In that case, we would need to spend a lot of time to troubleshoot the failures and investigate the cause. Then there is a need to find ways to fix the failures and then work on the changes.
Using AI we can avoid issues and start detecting issues in the test before they even occur. So instead of reacting to it, we can proactively fix tests. AI can figure out which tests are stable or flaky based on the number of test runs and it can tell us what test needs to be modified to ensure test runs are stable. AI can also handle test running on different resolutions and can optimize the wait time used in the test to wait for the page to load.
3. Dependencies on other modules
Writing a test for systems having dependencies on other modules is also a challenge. AI can help us to mock responses from a database or server. The AI can start recording server responses once we have written the test and have run them for a period of time. So the next time we run the test it will access the stored responses and will continue to run without any obstacles. This will speed up the process as the delay in response is eliminated and the server or physical database is no more needed.
4. Learning from production data
Real user data can be used to create an automated test and with the help of AI, we can observe and learn how the customer is using our product. We can identify common actions such as search option, using filters, login/logout, etc and compile them into reusable components. These components can be used for our test as well. Therefore, we have an actual test written by AI based on the real data along with the reusable components.
5. Easy execution of tests and speeding up the release
In automation testing, the time and effort it takes to write and execute a test is a major challenge due to the complexity of the test automation tools, app, and programing language used. To mitigate these problems AI-based tools are being used. The use of dynamic locators and reusable components has made it possible to write and execute a test in hours which earlier used to take a week.
Conclusion
The DevOps theory says test early, test often, but this puts a lot of responsibility on the testing team. Also, it’s not feasible for testing teams to spend time to do exploratory testing manually for each new release. AI-based tools can perform codeless automation testing which will save us time and resources and give the testers some space to breathe.
Since there are many players in the mobile app market, the competition is endless. To beat the heat and to keep providing an enhanced mobile app experience to the users, the app needs to be constantly updated with the changes happening and for doing this, there are different types of testing techniques. Let us have an insight into the types of testing methods.
1. Compatibility Testing
Compatibility testing is a critical QA task. It assures that a given application works as intended with selected operating systems, selected devices with different screen sizes resolutions, and internal hardware (memory size, processor speed, and button/input differences). It defines the feasible compatibility combinations of devices and interfaces for a specific testing assignment, in concurrence with the customer’s requirement.
Want to test your Mobile App?
Join pCloudy Platform!
Signup for Free
Compatibility Factors:
There are different factors that play a very significant role in mobile compatibility testing strategy. Mobile operating systems and their supported versions, mobile devices developed by different manufacturers, different device screen sizes with different resolutions and internal hardware including different input types, processor speeds and memory size of devices.
Let’s develop a test strategy for compatibility testing. The first step is to identify the Operating systems and devices based on key factors described in the previous slide. Once you identified the devices based on different compatibility factors you need to set up your environment for testing. You can either set up your internal test lab or you can go for external cloud providers like pCloudy. Now, the next step is to execute your test cases on a defined set of environment. Report the defects to appear while testing and report these defects to your development team.
The main purpose of compatibility testing is to ensure whether different software is compatible with different configurations and whether the application is compatible with the client’s environment.
2. Installation Testing
Mobile phones come with different types of mobile apps like Native and Web and Hybrid. Installation testing is a type of testing that is done at the initial stage of mobile app life span or maybe it is the first impression on the users, in other words. Installation testing checks whether the mobile app installs, uninstalls and updates properly without any interruption.
Want to test your Mobile App?
Join pCloudy Platform!
Signup for Free
Installation Testing Factors:
There are some of the factors that play a significant role in installation testing of an application. Factors like which platforms and operating systems supported by the application and how they will be distributed whether it is directly available in the app store or are going to get the installable files on mail itself or is it available on an app distribution platform and can be accessed through public URL. The user behavior regarding installation changes from person to person. For eg: the developer may install the app (By using ADB install command) and a different user might directly install it from the app store or public URL.
Important points to be noted in Installation Testing:
Some of the common checkpoints which can be converted to standard test cases are:
Installing the app in phone memory.
Installing the app in phone memory is full.
Installing the app on an external SD card.
Installing the app on an external SD card is full.
Check the memory space after installing. It should not occupy more than promised.
Check if it is releasing the space after removing
Another part of the installation and uninstallation testing is the testing of patches/updates
These are the common situation when we need to test app after updates.
Check for updates:
Can I download an update?
Can I update the app when there are multiple updates available?
What happens if I don’t update?
When App is upgraded:
Check that the user is still logged in
Check that data is still the same
Backward compatibility:-Check that database changes will not affect the current data.
Test the update from an older version to the current version
When OS is upgraded:
Check the app compatibility.
App updated gracefully without a crash.
3. Interruption Testing
Interrupt testing is a process to replicate abrupt (Unexpected) interrupt to the application. It is done to understand how the app behaves under certain interruptions before resuming to its original state. This can be achieved in various ways and techniques depending on the application under test.
Following are some Common Interrupts:
1. A phone call when the application is running or is in background
2. Battery removal when the application is running or is in the background, battery high, battery low, incoming call or SMS
3. Plugged in or out of charging
4. Device shut down
5. OS upgrade
6. Network loss and restoration, etc.
Some specific scenarios for an application using a network:
1. Connect to the network but remove LAN connection from the router so the device can sense wifi state on the device but cannot connect to the internet
2. Connection via VPN and VPN disconnected
Scenario for Application using services:
1. Kill service by clicking on the recent button and swiping the application right to kill app and services
2. Kill app using third-party App killer
3. Kill specific services from Settings->Manage Applications
4. Scenarios for Application Linked to account Manager
Remove account from Settings->Account Manager
Incoming and Outgoing SMS and MMS:
An application is running in your mobile and you send an SMS/MMS from another mobile to your mobile. In this situation, SMS/MMS notification should display for a while and then close and the application should continue to run.
Incoming and Outgoing calls:
You are in a login page trying to log in and you have just pressed the submit button that will take you to the application home page. This is a transition period because loading is happening.If we receive a call during this time then the application should not end abruptly or crash.
Battery Removal:
If the application is running, remove the battery of your mobile and it will shut down. After rebooting the mobile, run the application again, it should run smoothly and without a crash.
Data Cable Insertion and Removal:
When the application is running, connect the USB Cable, the system should display “USB Connection Screen” and if the user returns to the application then the application should not crash or end abruptly
Interruption Testing Strategy
Let’s prepare a test strategy to test interruption testing:
The first step in this process is to decide the critical interrupts based on your application components and architecture. For example, if your application is using GPS include common interrupts related to GPS. Let’s Include two type of interrupts:
Application specific interrupts:
Prepare your test cases and try to fit theses interruption test cases with state of the application. For example, interrupt your Network while performing payment.
Interrupt testing process:
List of common Interrupts:
Phone calls, Text messages, App notifications, Battery Warning, Forced updates, Voicemail, Shaking the Phone, Different Gestures, Switching between apps, Locking and unlocking the screen, Changing the orientation, Music playing while using the app, Out of memory (general performance interruptions), Data app interruptions (What Sapp, Viber, Tango), Audio interrupts from multiple sources (iPod, Media player, Other audio apps), phone in standby mode, switch the network to plane mode.
Tools to test interruption Testing:
Use tools like Joule Unit (Android), On iOS use the Energy Usage Function of Instruments, Monkey for Android to stress test your app and see the handling of the interrupts, UI Auto Monkey for iOS
Scenario
A) Once the user connects the mobile earphone jack with Nokia Lumia and starts music Player.
B) Now when the user resumes the application the cart gets empty.
4. Localization Testing
This type of testing is a technique in which we check whether the mobile app adheres to the local cultural settings, customizing the apps according to target country and language and also according to the linguistic aspects. Localization is also known as “L10N” as there are 10 characters between L & N. For eg: Chinese language-search engine offered by Baidu beats Google in China. Baidu does better than Google because it looks and feels fully native to the Chinese speaking audience.
Also, regional language speakers can’t get many games in their own language, so they turn to race games, which are easy to understand. We must have noticed in these examples, language is the common issue. Apart from Translation there are different other elements in the localization, like Usage of proper Time zone, local formats of dates, addresses and phone numbers, Design and layout to fit translated text, Converting to local requirements (such as currencies and units of measure), Adapting graphics to target markets, Modifying content to suit the tastes and consumption habits of other markets, Addressing local regulations and legal requirements.
Why should we localize?
Localization is important in order to understand the cultural and linguistic aspects of a particular region. While testing, testers keep repeating the tests to check typographical, linguistic errors, cultural appropriateness of the UI, etc. Research says:
1. 56.2 percent of consumers say that the ability to obtain information in their own language is more important than price. (Common Sense Advisory, Can’t Read, Won’t Buy: Why Language Matters on Global Websites, 2006)
2. 74 percent of multinational enterprises believe it is either important or most important to achieve increased revenues from global operations. (California State University at Chico, 2007.
3. 95 percent of Chinese online consumers indicate a greater comfort level with websites in their language. (Forrester Research, Translation and Localization of Retail Web Sites, 2009)
4. A critical success factor for cross-border merger and acquisition deals is the ability to communicate information clearly and accurately in multiple languages. (Merrill Corporation, How to do Better Multinational M&A Deals, 2008)
Types of Localization Testing:
In the localization testing, validating whether a mobile app is capable of performing given a geographical location, in particular, is found out. It includes four major types of testing like linguistic, culture, cosmetic, and functional tests. 1. Linguistic testing– ensures that the UI text appears in the congregation with the language and is not mistranslated or misapplied. This requires language skill and knowledge of the product. Mistranslation, irrelevant usage of the language are few of the errors in this.
2. Cultural Testing– Few words may imply different meanings in different cultures. So, this has to be translated properly in relation to the target culture.
3. Cosmetic Testing– For easy understanding, let us use this scenario. For eg. In the apps used in Afghanistan, the text will be in Arabic and will start from right to left unlike other languages like English, French that are read from Left to right.
4. Functional Testing– In this testing, testers follow prescribed test scripts to run through all aspects of the product to make sure it functions as designed in a localized environment. Often functional testing includes verification that the localized product is compatible with various operating systems and third-party products. Functional testing generally requires a good knowledge of the target language and total familiarity with the product. Some of the major Functional testing problems may include– Incorrect or missing UI elements, graphics, and windows, Incorrect error messages, Localization bugs that may cause software crashes, Errors generated by installing the localized software on a localized OS.
5. Performance Testing
Performance testing is to test the performance of the mobile application in expected workload scenarios and to eliminate the performance hurdles. It checks whether the response of the app is quick, the ability of the app to tale load, and app stability in those load situations. Performance is very critical because if the app is malfunctioning, it is more possible that the user might uninstall the app and might shift to a competitor’s app which is better.
App start-up time is the most critical performance parameter from the user’s point of view. It shouldn’t take more than 2 seconds for the app’s first screen to pop up after the user taps on the app’s icon. Battery time is the second most important issue. Some apps consume a high amount of battery life. Excessive resource usage creates a burden on the processor and the phone heats up. In some cases, it is observed that the new app installed uses the same amount of battery as the OS. If more features are added in the app its memory consumption also increases. This is why memory consumption should be checked while testing the app.
Network Performance
The app needs to be tested on different mobile networks and network properties. If there is packet loss then the app should generate alerts or resend the request for the information. Jitters or delay in receiving information
Server/API Performance
Performance is directly proportional to the response time from the Server. In this, tests are run to check how efficient the app is to handle data transfers from the server and how quickly and in what format the data is transported by the app. The number of API calls generated by apps should be lesser and how and where the data is saved in the ‘server down’ case is all that is tested here.
Mobile app testing tools make it easier for the testers to identify and rectify any error that comes up. For different OS there are a different set of Mobile app tools. For Android, we can use Robotium and Monkey Runner and for IOS we can use Automator.
6. Usability Testing
Usability testing is a type of testing which is performed to check how user-friendly the mobile application is in terms of the navigation, ease to use the app, flexible app controls, etc. It is also known as user experience testing.
What matters in Usability Testing?
Workflow: To understand, below image represents the workflow of how the user reaches its goal while navigating through the app controls.
We must understand what the user needs to do in order to use the app’s functionality. If the workflow was overlong and included repetitive actions It may annoy the user while performing the task. Design and Layout: Good layout and design allow a user to easily complete tasks. An app need not offer so many features or content that the app is difficult to navigate. A friendly app has a design that allows less user text entry, helps differentiation between the selected items, finger-sized targets, should provide intuitive hints to texts, etc.
The response time of the application: It is one of the key factors. This means how long the user takes to complete a task. Often long delays while the app interacts with the server hinders the user experience standards.
The emotional state of users: Emotional state of user measures the user’s motivation to use the app. The apps should be smart enough to understand what the users want and how to engage them, this can be achieved by usability testing.
Usability Testing Strategy:
The first step is to identify users who match your expected audience. Watch and listen to them as they use your products. You can use video and audio recording equipment for archiving test sessions, in case later review is needed. However, our primary tool is taking notes with paper and pencil! This is still the fastest way we have found to get information quickly to developers and designers. You have your data. You now need to organize and analyze it. How the results are absorbed as a report in order to be useful for product changes.
Tools and techniques for usability testing
1. Mr. Tappy: It is a tool set up to capture how users interact with the apps naturally. The user data so generated could be used for better user experience.
2. Paper Prototyping: It is a technique that consists of creating hand drawings of user interfaces in order to enable them to be rapidly designed, simulated and tested. As simple as this technique may look like, it has been effectively used since the 1980s and is likely to continue to be used with a guaranteed degree of success for many more years to come.
3. Magitest: It allows you to perform mobile user testing of native apps and websites. Captures the participant’s voice and facial expressions during the session. Here the users complete the tasks you specify and allows to listen to them speak their thoughts aloud as they interact with your mobile app.
4. Reflector: This app allows us to see the iPad, iPod, iPhone screens on MAC or PC. The devices connect wirelessly to proceed with further functions.
7. Conformance Testing
Conformance/compliance testing is a type of testing which is performed to validate if the application is meeting the Market place and Enterprise policy guidelines. It is like an audit done to check whether certain standards are being met or not. Conformance Testing is focused on two main areas- App store guidelines and Enterprise policy compliance. Each Market place has their own guidelines. We need to validate if the application is following the guidelines of each of these app stores.
1. App store Guidelines pertaining to User Interface, Media Content, Privacy, Spyware and Phishing, nudity, religion, culture, violence, etc.
2. Enterprise Policy Compliance pertaining to a different set of industrial standards. For eg: For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, For the pharmaceutical industry, the Food and Drug Administration (FDA) passed rule comes into the picture, In healthcare, the Health Insurance Portability and Accountability Act (HIPAA) includes an administrative simplification section, which mandates protecting the information assets of patients. So, it is a dire need for the testing organizations to follow the necessary guidelines and compliances to limit the risk that non-compliance might follow.
Reap the benefits of different types of testing
In this day and age, mobile app quality matters the most as the market is very competitive. We need to focus on user experience as well as the performance of the app. If the app contains bugs the app ratings will go down and so will the number of app downloads. If we can make our app bug free than the chances of app gaining popularity will increase. This can be achieved by performing different types of testing on the mobile application. In the next blog in this series, we will talk about mobile app testing strategies.
Do you know why Google has selected Gradle as the build system for Android Studio? Many Android developers work in a heterogeneous environment with different technology stacks. Gradle solves some of the hardest problems faced by the developers like how to automate the testing of apps and how to manage dependencies and variations that allow professional developers to develop variations of their app with one click. This is why pCloudy came up with a new update where you can now run Espresso with Gradle on pCloudy devices. Let’s get a brief introduction about Espresso and Gradle before we learn how to run Espresso with Gradle on pCloudy devices.
Espresso is a testing framework for Android which automatically synchronizes your test actions with the UI of your app. It also let the test wait until all the background activities have finished.
Gradle is an open source advanced build tool that allows seamless execution of tasks. It uses domain specific language and it is based on Groovy and Kotlin. It is a plugin based system so if you want to automate the task of building some package from sources then you can write the complete plugging in Java and share it with the world.
Gradle allows efficient and repeatable use of espresso and Test Orchestrator which allows automated yet fine-tuned control of the way you run your test. You can decide which specific test suites to be run to distribute the test cases across different devices. It is preferred by developers as it allows deep unit and functional testing rigs.
Power up your DevOps with Espresso and Gradle on pCloudy
You would be running Espresso on your machine using Android Studio and Gradle. However, the test would be running on pCloudy device. There is a pCloudy Espresso script that is provided by us and you need to put that espresso script in the workspace of the project. Along with the Gradle script, you will also get a config file. The gradle script will read input parameters from this config file.
Then you run Gradle to invoke the script which will upload your Application APK, test APK and other APK files to pCloudy. It would acquire a device to run your test on, it would execute those tests it will report back the status of what is happening into the Android Studio. After the test cases are run you can see the detailed reports and after that gradle script will release the device for other users to use.
There is a one-time setup that you need to do to place the gradle script in the workspace and you need to fill in the configuration file. So when you run the gradle script it will complete all the task for you and generate the report.
Steps to run Espresso with Gradle on pcloudy
1. Download the espresso starter pack from here and Unzip it
2. You will find three files,
a. pCloudy_Espresso.jar
b. Config.properties
c. build.gradle.SAMPLE
(This is a sample build.gradle that shows how to change your build.gradle to add the pCloudy Espresso jar.)
3. Copy the contents of the file build.gradle.Sample to the build.gradle file of your Android Application to register the new Gradle task and update as appropriate (see the image below)
4. Place the config.properties file in your android workspace in the same directory as your build.gradle file.
5. Update the config.properties file as guided in the comments in the file.
6. In your Android workspace run the command by typing the name of the gradle task as below.
7. Once the test execution is complete you will get a URL of the test execution report. Some fields in the report are empty right now. We will fix them in subsequent phases.
The use of gradle has made it easier to run Espresso test on pCloudy devices. We can just use the configuration file and easily run the test scripts on pCloudy devices. This will save time and effort as it further simplifies the app testing process in pCloudy.
Mobile app automation testing has evolved as a crucial aspect of the mobile app development process to help deliver better quality solutions, under controlled time cycles and cost schedules. But for delivering bug-free app, choosing the best suitable automation testing framework for your app is very important. There are many automation testing frameworks available in the market with exceptional capacities. This blog is all about Appium vs Espresso and we will analyze which of these two most widely used Automation testing frameworks is preferable for your app testing.
Espresso was not preferred because of its flakiness and instability issues. But, from the time Google has brought Android Test Orchestrator, a Gradle test option, instability and unreliability of Android Espresso tests have vanished. This, in turn, is creating a serious problem for the most popular automation framework Appium.
Let’s find out in this blog if Espresso now comes with a power to kill Appium or Appium can hold its stand in this fiercely competitive market.
Let’s get into the details.
What is Appium?
It is an open source, cross-platform mobile app automation testing framework. Appium allows native, hybrid and web app testing and supports automation test on physical devices as well as emulators or simulators. The Appium server uses selenium web driver which permits platform independence and allows the user to use the same code for Android or iOS.
Advantages of using Appium
Facilitates test execution without server machines
Appium is developed using cross-platform runtime environment like NodeJs which enables programmers to write server-side code in javascript. It is designed as an HTTP server and you can run the test without requiring a server machine.
Does not require app code recompilation
Most of the automation testing tools require testers to alter app code. Some of the test automation frameworks require testing professionals to recompile the code according to the targeted mobile platforms. Appium enables testers to evaluate both cross-platform and native apps without recompiling and altering the code that often.
Automates various types of mobile apps
Testers can avoid using different automation tools for different types of apps as Appium can be used for web apps, hybrid, and native apps too. It facilitates the testing of hybrid and mobile web apps as a cross-platform test automation framework. At the same time, it enables testers to test native apps through web driver protocol.
Testers can use real devices, emulators, and simulators
Testers use real devices to evaluate mobile app’s usability and user experience more precisely. Although, to speed up the mobile app testing one needs to use emulators or simulators too. Appium helps testers to produce reliable test results and reduce testing time by supporting real devices, emulators and simulators.
Provides a record and playback tool
In Appium, testers can use the inspector to accelerate testing through record and playback functionality. Appium inspector can record the behavior of native apps by inspecting their document object model (DOM). Record and playback tool can produce test scripts in a number of programming languages.
Testers can automate apps without adding extra components
Testers can execute the same test across multiple mobile platforms without putting extra time and efforts or adding extra component. Appium simplifies automation by keeping complexities in Appium server.
Supports several web driver compatible languages
You can integrate Appium with many testing frameworks and WebDriver – compatible languages including PHP, Java, Ruby, Javascript, C# and Objective C. Hence, a tester has the option to write test scripts in his preferred programming language.
Disadvantages of using Appium
Common gestures
Appium lacks commonly used gestures like double-clicking in java-client libraries. It also does not support Android alert handling directly and the users cannot evaluate alert handling through native API. Testers have to put extra time and effort to test these gestures.
No script execution on multiple iOS simulators
Simulators make it easier for testers to mimic internal behavior of the underlying iOS devices. Although Appium does not allow users to run multiple test scripts on multiple simulators simultaneously.
Lacks the capability to recognize images
Appium cannot locate and recognize images automatically to evaluate games and apps precisely. The testers have to take help of screen coordinates to make Appium locate and recognize images.
Does not support older versions of android
Appium supports only Android 4.2 and later and does not supports older APIs for Android. There are still many people using devices which run on older versions of Android and developers find it difficult to test mobile apps developed targeting older Android API level.
What is Espresso?
Espresso is a tool developed by Google which is used for testing the UI of Android apps. It automatically synchronizes your test actions with the user interface of the mobile app and ensures that the activity is started before the tests run.
Although when you execute an Espresso test you will have shared state in separate tests and some flakiness. For this Google came up with a solution. Android Test Orchestrator is a Gradle test option that helps in testing and increases the reliability of our automated test suites.
If you use Gradle build tools in any version of Android Studio below 3.0 then you also have to update the dependency setup. Let’s take a look at the advantages of using Android Espresso.
Advantages of using Espresso
Integration with Gradle
The new Android Espresso now has the power of the Android Studio and Gradle that comes along with it. So now invoking your tests, running it or modifying it is just a matter of calling a Gradle command. This gives the full power of command line to the developer and makes testability much easier.
Test Orchestrator
The new Android Espresso comes with the power of Android Test Orchestrator that allows you to run each of your app’s tests within its own invocation of Instrumentor. It ensures that there is minimum shared state and crashes being isolated. It allows you to filter the tests that you want to run and also distribute tests across devices. This implies that you have finer control over how your tests run.
Less flakiness
The scalability of the test cycle in Android Espresso is high due to the synchronized method of execution. A built-in mechanism in Espresso that validates that the object is actually displayed on the screen. This saves test execution from breaking when confronted with “Objects not detected” and other errors.
It’s easy to develop Espresso test automation
Test automation is based on Java and JUnit which Android developers are familiar with. There is no setup or ramping up to implement quality in the in-cycle stage of the app SDLC.
Reliable and fast feedback
Android Espresso does not need any server to communicate with, instead, it runs side by side with the app and delivers fast results. It gives fast feedback to the code changes so that developers can move to the next bug fix.
Simple workflow
Espresso allows developers to build a test suite as a stand-alone APK that can be installed on the target mobile alongside the app under test and be executed quickly.
Disadvantages of using Espresso
It requires access to the application source code
Without the source code, you won’t be able to do anything. Also, There is a risk to get used to the in-built test synchronization and UI – then it might be hard to work with WebDriver.
Narrow focus
If UI tests are required for both Android and iOS, it will be necessary to write twice, for two different systems. If tests require to work with Android outside the application (for example, open a received notification with a text message), you’ll have to use additional tools, such as UIAutomator.
Knowledge of launching Android app on emulators required
It is desirable to have at least minimal experience of building and launching Android applications on emulators.
Conclusion
Appium and Espresso both can be used to perform UI testing on Android app but if you have to choose one of them then you need to decide on the bases of your requirements. What kind of app is it and what kind of testing you want to perform. Developers who want to perform UI testing for their native Android app should go for Android Espresso. Although, if the test needs to support iOS and Android both and you want to test at a functional level then you can use Appium.
Espresso is a testing framework used to write UI test cases. It automatically tests your actions with the UI of your application. The framework also ensures that your activity is started before the test run. It can be used to test across multiple applications. If used for testing outside application, you can perform only black box testing. as you cannot the classes inside the application.
Espresso has three components:
ViewMatchers: Allows to find the view in the current view library. ViewActions: Allows to perform actions on the view ViewAssertions: Allows to assert state of the view
Why Espresso?
It's fast: It is really important to run test cases at a fast pace as there can be many tests. UI test takes time and is costly and it is possible that you might not be able to run all the tests.
It's easy to setup: Setup process on local machines is easy and Android developers feel comfortable in the mobile SDK language which they use every day.
It's less flaky: Automated DevOps process required fast and reliable feedback. Test written in native tools is much less flaky. It gives more control over the application: It is white box testing, Espresso is inside the application, it has access to and knows how to use the code that actually runs the application for more thorough testing of each element.
You can use Espresso cheat sheet for quick reference as it contains most available instances of Matcher, ViewAction, and ViewAssertion.
Steps for running your Test scripts in Espresso using pCloudy:
Login over https://device.pcloudy.com with your registered Email ID & Password.
To schedule "Espresso" over pCloudy, follow the following steps-
Go to the "Automation" page.
Select the Automation tool as "Espresso".
Select "Instrumentation Type" based on the Test Scripts you've written.
Note: pCloudy provides support for Instrumentation Type(InstrumentationTestRunner, AndroidJUnitRunner, and AndroidXJUnitRunner) for Android.
Select the Application APK and Test APK that you must have uploaded in the MY APP/DATA section.
Select the single device execution time and assign a name to your test cycle.
In the next step, Click on "ADD" to add the device for testing and click on " Next".
Now you can select the devices by clicking on the "ADD" button. You can also use the device filter to make it easier to search for a device. Click on next once the devices are selected.
Click on "Schedule" to start the test.
Go to your email inbox and open pCloudy Automation Alert mail. Click on the given link “Click to view Report”.
Now you have the result of your scheduled test automation.
Take a look at this GIF to understand the flow of actions taken to run the test cycle.
Software developers tend to make mistakes as app development is a very complex process. This means that the app will contain bugs. Espresso helps us detect errors that may have been made in the development, ensuring the reliability of the client and his satisfaction with the application.
Since pCloudy is committed to provide to you with the newest the fastest, here is another one. The newly released device Samsung s10 is now available on pCloudy for you to test your app.
This blog and a few upcoming ones in this series will talk about specific aspects of mobile app testing.
[xyz-ihs snippet=”quickLinks-mobile-app-testing”]
Mobile app testing involves many intricacies and in this series of blogs, we will explore this area starting from the basics. The topics that are covered in this blog will be elaborated in the forthcoming blogs. I will be updating this page with links for those topics as we publish them in the near future. Keep watching this space for more information related to mobile app testing. Now let’s get started with the basics.
What is mobile app testing and why is it needed?
These days mobile phones are not used just for making or receiving calls, but they are essential multipurpose gadgets. The mobile technology is evolving rapidly and the mobile app developers need to deliver the best performing app in quick time. To get the best app delivered in the given time frame, the automation team needs to be on their toes. The testers play a critical role in the quality and delivery of apps. Moreover, the market is very fragmented and this makes testing more complicated. The mobile app must be tested on different combinations of platforms, networks, operating systems.
Types of mobile applications:
There are three types of mobile applications: Native apps: These apps are developed for a single mobile operating system so it is native for a particular platform or device. For example, Android apps will not work on iOS apps.
Web apps: These are mobile web applications to render/deliver pages on mobile web browsers. These apps work on different operating systems as they launch on web browsers.
Hybrid apps:These apps have cross-platform compatibility but can still access phone hardware.
Challenges in mobile app testing
All the operating systems like Android, iOS, Windows have different UI and functioning. The usage and layout of elements are verified when publishing the app in the market. Noncompliance with the guidelines can delay the publishing process which in turn increases the cost of development. Lack of access to devices is also a big issue. It is possible that two or more testers need the same device at the same time. Wherein one tester is left out waiting for the other to the tester to complete their work. Procuring multiple devices adds up to the cost and takes time. It is a major issue if the testers are placed across different countries.
Types of mobile app testing
To develop a highly efficient mobile app, you need to ensure that the quality, usability, and security of the app is up to the mark. You can improve your app ratings and customer satisfaction with rigorous testing, which will lead to more downloads.
The mobile app testing methodologies are mentioned below:
1. Compatibility testing
2. Installation testing
3. Interruption testing
mobile app testing strategy that will ensure that your quality assurance activities align with customer expectations, business goals, and industries best practices. Testing teams must consider the following procedures while designing a comprehensive mobile application testing strategy:
UI Testing: A UI friendly application sells more when compared to an app which is best developed but with a nasty UI. If an application has a perfect and a splendid UI on one device but on the other device it is completely twisted just because it has a different size or a different OS, then it will leave a very bad impression. The commercial success of the application will be badly affected.
Security Testing: There’s always a big concern about data safety and security. There is a possibility of the personal data being available to hackers, be it your social sites credentials or your bank account credentials. Hence, the security of these apps becomes very vital for the business of any company. This, in turn, generates the need for security testing of all mobile applications and hence is considered as important testing that is carried out by testers foran app.
Manual vs Automated testing: The speed and reliability of automated testing are helpful for regression testing and executing time-consuming test cases. Most automated tests are reusable, but in an agile environment, automated test scripts have to be reworked. As a mobile app evolves, the product flow changes, as well as the user interface requirements and specific features. As a result, every change calls for an update in the automated test script. Maintaining automated test scripts often causes projects to fall behind in sprint cycles unless you keep a tight hold on the reigns. It is scalable and more cost efficient for larger mobile apps. We can run multiple tests simultaneously. But it is not suitable to test user experience factors. Manual testing Replicates real user experiences and it is faster and more cost- effective for small projects. It is easier to identify and deal with app crashes with manual testing. Manual testing Handles more complicated use-case scenarios.
Device Selection: New versions of mobile devices emerge every day as mobile technology is evolving rapidly. Each model comes with its distinct features in terms of OS, screen size and other aspects. You have to factor in not only the OS of the device but also the OS version. It is critical that you choose the right number and type of device to test your mobile app. It is important to identify the devices that are popular with your target demographics. You need to identify the combination of devices that can get you the broadest coverage in terms of OS, model, and brand. While selecting a device for testing you need to take into account, its screen size, resolution, PPI & other hardware features that can impact your app’s performance. You also need to consider the effect of network and varying conditions posed by location.
Performance Testing:
Test your app under load conditions to ensure high performance while being used by thousands of users. Build real-user flows and test them on real mobile devices. Correlate between different application performance vitals. Simulate the on-the-move scenarios and rapid environmental changes that are unique to mobile users.
Cloud vs In-house:
Cloud solutions provide an instant way to build mobile development and testing labs. The greatest benefit of these solutions is that you don’t have to set up any server hardware or connect any physical mobile device in order to use them seamlessly as a part of your development process. The majority of the costs related to cloud-service are included in the license fee (price) of the service. You very rarely even need any other software to be purchased to use these services.
An in-house solution is something more traditional compared to today’s cloud- based software solutions. It is typical that at the beginning when an in-house solution is acquired in use it involves more expenses as you need to buy a license and get the hardware infrastructure (e.g. servers, devices) in place. The operative expense will be there as well and you need to take care of maintenance of the system, software (and sometimes even hardware) updates, fixing and resolving problems with the system and probably many other smaller tasks that eventually makes financial burden to accumulate.
Network Connectivity:
The unpredictability of wireless networks has a huge impact on the functionality, performance, and user experience of an app. This is why we often see certain functional defects, performance defects and sometimes crashes only when the app is running on a certain network or in some specific locations. Besides, mobile apps are expected to function on everything from an excellent Wi-Fi connection to a 3G/4G connection.
The apps can be installed on real devices and can be tested on various networks like VoLTE, GPRS, UMTS, LTE, CDMA, etc. But the procedure of using real networks for app testing is slow, cumbersome and mostly provides inaccurate results. Using a simulated lab network, testers can easily test their mobile apps by simply selecting and applying the type of network profile to be used. They can also create and customize different network profiles by simply changing parameters like Uplink Bandwidth, Downlink Bandwidth, Latency, Loss Percentage, Packets Corruption, etc.
Conclusion
In the world of mobile apps, testing challenges emerge with evolving mobile technology. The upsurge of new technologies like 5G and foldable screens will push the developers and testers to find new ways to test their apps. In continuous testing and development, the focus should be on optimizing the app performance
In the next blog, we will talk in detail about Android and iOS.
