No comments yet

Introduction

In today’s highly competitive mobile app market, delivering a flawless user experience is essential. Mobile apps are constantly updated with new features, bug fixes, and optimizations to meet user expectations. To ensure quality across diverse devices, operating systems, and networks, different types of testing methods are required. These testing techniques help ensure that apps not only function well but also provide a seamless, reliable, and enjoyable user experience. In this blog, we’ll explore seven essential types of mobile app testing, along with the challenges that arise and the solutions to overcome them.

Compatibility Testing

Key Compatibility Factors

 

Compatibility testing ensures that a mobile app works across a variety of operating systems, device models, screen sizes, and hardware configurations. This type of testing is critical because mobile users access apps on a wide range of devices with varying capabilities, and failure to support even a subset of these can lead to user frustration and lost customers.

Key factors that impact compatibility testing include:

 

• Operating System Versions: iOS, Android, and their various versions.
• Device Models: Different devices (phones, tablets) from manufacturers like Samsung, Apple, Huawei, etc.
• Screen Sizes & Resolutions: Apps must adapt to a variety of screen sizes and pixel densities.
• Internal Hardware: Testing on devices with varying memory, processor speeds, and storage capacity.

Challenges and Solutions

Challenge:

One of the biggest challenges in compatibility testing is the sheer number of device combinations that need to be tested. Managing physical devices in-house is expensive and resource-intensive.

Solution:

Cloud-based testing platforms like Pcloudy provide an efficient solution by giving access to thousands of real devices with different OS versions and hardware configurations. This helps teams to automate compatibility tests and scale their testing efforts without maintaining physical labs. Pcloudy also enables parallel testing across multiple devices, speeding up the overall process.

Installation Testing

Key Focus Areas

Installation testing is one of the first interactions a user has with a mobile app. It ensures that an app installs, uninstalls, and updates without issues. This testing is critical to verify the app’s ability to install smoothly across various devices and handle future updates seamlessly.

Key areas to focus on include:

• App Installation: Testing how the app installs under different conditions, such as with limited storage or in different installation locations (e.g., internal memory, SD card).
• App Updates: Ensuring that the app updates smoothly without causing data loss or crashes.
• Uninstallation: Verifying that uninstallation removes all app data and does not leave residual files.
• Post-Installation: Ensuring the app launches properly after installation and functions as intended.

Challenges and Solutions

Challenge:

The main challenge in installation testing is handling various installation environments, especially on devices with low memory or unstable network connections. Additionally, testing installation scenarios across different OS versions and devices can be complex.

Solution:

Using a cloud-based testing product like Pcloudy, QA teams can test on real devices under real-world conditions. Pcloudy provides access to thousands of actual mobile devices with varying configurations, enabling teams to test scenarios like low-memory conditions, update handling, and different installation environments. Automation tools help execute various user actions during the installation process, ensuring robust testing across multiple environments without manual intervention. This ensures that your app installs, updates, and uninstalls smoothly across different devices and conditions, providing users with a seamless experience.

Interruption Testing

Common Interruptions to Test

 

Interruption testing evaluates how well a mobile app handles unexpected events, such as incoming calls, network disruptions, or battery drains, while the app is running. The goal is to ensure that the app resumes normal functionality after an interruption.

Common interruptions to test include:

• Incoming calls and SMS notifications while the app is in use.
• Battery low, battery removal, or plugging the device into charging.
• OS updates that occur while the app is running in the background.
• Network disconnection and reconnection
• Device shutdown or reboot while using the app.

Interrupt Testing Process

Challenges and Solutions

Challenge:

Replicating real-world interruptions, especially across different devices, OS versions, and network conditions, can be difficult to reproduce consistently.

Solution:

Pcloudy provides a reliable environment to automate and simulate interruptions such as network loss, incoming calls, or device shutdowns. Tools like Monkey (for Android) or UI Auto Monkey (for iOS) help simulate interruption scenarios, allowing testers to monitor how well the app recovers from these events. Automating these tests across multiple devices ensures thorough coverage.

Localization Testing

Types of Localization Testing

Localization testing ensures that a mobile app is tailored to a specific geographic region, considering cultural, linguistic, and regional differences. This testing verifies that the app works seamlessly when localized for various languages, currencies, time zones, and formatting conventions.

Four key types of localization testing include:

• Linguistic Testing: Ensures that all text in the app is properly translated and adapted to the target language. This includes avoiding mistranslations or phrases that don’t make sense in the local context.
• Cultural Testing: Ensures that content is culturally appropriate. Some symbols, colors, or phrases may have different meanings in various cultures, and testing ensures nothing offensive or inappropriate is presented to users.
• Cosmetic Testing: Verifies that the layout and design elements fit well with the localized content. For example, languages like Arabic and Hebrew, which read right-to-left, require changes to app design.
• Functional Testing: Ensures that the app functions correctly in the localized environment, including handling local date formats, currency, and special characters.

Challenges and Solutions

 

Challenge:

Managing translations and ensuring cultural accuracy across multiple regions can be challenging, especially with languages that have different text directions, such as Arabic or Hebrew. Additionally, it’s important to ensure that all text is properly displayed without breaking the app layout.

Solution:

Tools like Pcloudy allow testers to run localization tests across real devices in different regions, ensuring linguistic and functional accuracy. Automated scripts can be used to check for proper translation, layout adaptation, and functionality. Pcloudy provides access to a wide range of devices from different locales, helping to ensure comprehensive localization testing across multiple regions.

Performance Testing

Key Areas in Performance Testing

Performance testing is essential to ensure that the mobile app performs optimally under various conditions, such as high load, different network speeds, and limited device resources. It identifies performance bottlenecks, stability issues, and overall app responsiveness.

The three primary areas of focus in mobile performance testing are:

• Device Performance: Testing how the app behaves on different devices, with a focus on start-up time, memory consumption, and battery usage. High memory or battery consumption can lead to users uninstalling the app.
• Network Performance: Testing how the app handles different network conditions, such as slow or unstable connections. This includes testing the app’s ability to manage packet loss, network delays, and connectivity interruptions.
• Server/API Performance: Testing how efficiently the app communicates with the server and processes API requests. Slow or inefficient API calls can degrade the user experience, especially in data-heavy apps.

Challenges and Solutions

Challenge:

Replicating real-world conditions like varying network speeds or high traffic loads is a significant challenge. Ensuring that the app works well under different device configurations while maintaining performance consistency is also complex.

Solution:

Pcloudy’s network simulation feature allows QA teams to replicate different network conditions, such as low bandwidth or high latency, to test how well the app performs under challenging conditions. Additionally, using tools like Pcloudy to run performance tests across multiple devices ensures that device-specific issues, such as excessive battery drain or memory usage, are identified and addressed early in the development cycle.

Usability Testing

Important Usability Factors

 

Usability testing ensures that the app is user-friendly and provides an intuitive, seamless experience. This type of testing focuses on how easy it is for users to navigate through the app, complete tasks, and interact with the app’s features.

Key factors in usability testing include:

 

• Navigation Ease: Testing the workflow to ensure users can easily navigate through the app with minimal effort. Complex workflows or unintuitive navigation paths can frustrate users.

                                Example of Navigation Path

• Design & Layout: Verifying that the app’s design is user-friendly, with clear, well-organized content. Elements like finger-friendly buttons, minimal text entry, and intuitive visual cues are essential for a positive user experience.
• Response Time: Ensuring that the app responds quickly to user inputs without lag or unnecessary delays. A slow response time can lead to a poor user experience and high uninstall rates.
• User Engagement: Testing how well the app engages users emotionally. A successful app should be smart enough to predict user actions, offer personalized experiences, and keep users motivated to continue using it.

Challenges and Solutions

Challenge:

Usability testing can be subjective, as user preferences and behaviors vary. It’s difficult to ensure that the app will be intuitive for all user types and across different demographics. Additionally, collecting meaningful feedback from users to guide improvements can be challenging.

Solution:

Tools like Mr. Tappy or Reflector can capture real user interactions during usability testing, allowing testers to observe how users navigate and respond to the app. Recording user sessions helps teams identify pain points and optimize the user experience. Pcloudy’s cloud-based platform allows for testing on a wide range of devices, ensuring that the app remains user-friendly across different screen sizes, input types, and configurations.

Conformance Testing

Key Conformance Testing Areas

Conformance testing, also known as compliance testing, ensures that your mobile app adheres to industry standards, regulatory requirements, and marketplace guidelines. This type of testing is critical, especially when submitting apps to app stores or meeting enterprise policy guidelines. Ensuring conformance can prevent rejections from app marketplaces and avoid penalties related to non-compliance with industry regulations.

The two key areas of conformance testing include:

• App Store Guidelines: Every app marketplace, like Google Play or Apple’s App Store, has specific guidelines covering areas such as user interface (UI), privacy policies, content restrictions (e.g., nudity, violence, cultural sensitivity), and data protection. Failure to comply can result in app rejection or removal from the store.
  
  

Enterprise Policy Compliance: In some industries, apps must comply with industry-specific regulations. For instance, healthcare apps may need to comply with HIPAA (Health Insurance Portability and Accountability Act), while pharmaceutical apps may fall under FDA (Food and Drug Administration) guidelines. Meeting these standards is essential to maintaining credibility and avoiding legal issues.

Challenges and Solutions

Challenge:

Staying up-to-date with app store guidelines and ensuring that the app meets the ever-changing standards of different marketplaces can be difficult. Moreover, managing compliance with strict industry regulations can be overwhelming, particularly when apps are released across multiple regions with differing legal frameworks.

Solution:

Pcloudy offers a comprehensive conformance testing solution that helps validate whether your app meets both app store guidelines and industry regulations. Automated checks ensure your app complies with the latest app store rules before submission, while the platform’s flexibility allows for testing specific compliance criteria related to industries like healthcare or finance. Pcloudy’s regular updates keep testers informed of any changes to app store guidelines, reducing the risk of non-compliance.

Conclusion

In an increasingly competitive mobile app market, delivering a high-quality user experience is crucial to success. Testing your app across various dimensions—compatibility, installation, interruptions, localization, performance, usability, and conformance—ensures that it functions seamlessly and meets user expectations across different devices, regions, and conditions. Each type of testing addresses specific challenges that can impact an app’s performance, usability, or compliance with industry standards.

The challenges associated with these testing types can be daunting, but with cloud-based testing platforms like Pcloudy, teams can automate, scale, and simplify the testing process. From testing real-world interruptions to ensuring app store compliance, Pcloudy offers the tools and resources to ensure comprehensive mobile app testing without the hassle of managing physical devices or manual testing efforts.

By incorporating these testing strategies, mobile app developers and QA teams can confidently release bug-free apps that provide a flawless user experience, leading to higher user satisfaction, increased app downloads, and long-term customer retention.

No comments yet

No comments yet

No comments yet

Code Review in a Startup

Here I bring to you the 5th blog in the DevOps series showcasing our learning while #scalingup. Read our previous blog to know about the bunch of tactics that we used at different times during our evolution to achieve a successful clockwork during our DevOps journey.

Proper optimized code review is something that many startups miss. Some take the easy way out and ignore it and others spend years discussing the best practices, conventions and styles without ever committing code. Both these are slightly over exaggerated examples of paths to failure, but I am sure you can relate to these if you have ever tried to answer the all important question about code review “Exactly how much code review should your team do?”

For perfectionists, the answer is that code should always be reviewed and you should always be refactoring code and improving it wherever you see a scope. I have worked in companies, where people used to spend almost as many hours reviewing code as much as they would writing them. There are some practices like pair programming, which has maximizing code review as one of the results. So this is not exactly a wrong direction of thinking. The problem however is that when you are working with time constrained environments like that of a startup, you will not have a luxury where you can keep revisiting and improving code beyond a reasonable amount of time. And I have seen that surprisingly many people altogether avoid code review because timelines are constrained. I don’t even need to mention how risky this is and how dangerous this course of action is. However it seems many people do exactly that.

In this blog post, I will attempt to throw light on our experiences in deciding how much is enough. We too, like other startups have a time constrained environment, and a few (thankfully only a few) customers who want every feature done yesterday. So as a culture, we too try to make sure that we finish everything faster. Development faster, Testing Faster, Deployment to various environments faster. But we do not ignore code review. We ensure that we do code review. We have a few principles that we followed to make sure that code review happens all the time, and is neither too less nor too much. Here are they. What we have seen is that if you follow all these steps, then you have a sane code review process and you can guarantee a stable flow of god code into your repository. These points are in no particular order of importance.

You should do code review: The first principle is not a lame attempt at a joke. The idea is that code review should happen come what may. Without this principle being followed, every other principle in this list breaks down. We use git-flow as one of our developer methodologies with git. One of the advantages of this system is that code review is built in. Unless code is approved by a designated reviewer, the code does not go to the next level be it Testing or Production. The Approver is as responsible for a piece of code as the original developer. This way the reviewer spends more time reviewing and the original coder also reviews and corrects his code in advance to preempt the reviewer. This adds more points in the system where code review can happen and makes sure that code always gets reviewed and the review is not forgotten.

Requirement Matching: Does your code do what the requirements ask it to do? Are we doing less? Are we doing more? A quick inspection reveals mismatches if any. This goes a long way in finding out if there are any problems in the code.

Readability: There is an apocryphal statement that says that code is written once and read hundreds of times. While the statement may not be accurate, you may write more than once, and may not read hundreds of times, you still do get the picture. The basic premise is that code does get read many far more times than it gets written. Also once written, your code will also be read for review, bug fixes, enhancements, etc and not always by you. Also in the IT Sector, jobs switches happen a lot, so a new person should be able to understand and work with the code as soon as possible.. So it makes sense that whoever looks at your code after you have left is able to understand your code well and can alter if needed and maintain the code well till the product lifetime completes.

Reviewability: A further subset of this is that the code should also be reviewable, you and your code should be able to convey to the reviewer what the code is supposed to do and what the reviewer is supposed to review.

Scalability: Will your code be able to stand frequent and/or continuous requirement changes in the future? Will it be able to handle a reasonable amount of requirement change without having to have to rewrite the entire thing? Overall applications are live, especially in the Agile era, your requirements are never frozen, and hence the code also should never be frozen. It should be able to handle changes in requirements. A word of caution here, do not overdo this. While your code should be able to handle requirement changes, you cannot (and should not) make your code so generic that it can handle the proverbial ‘everything under the sun’. Your code should be reasonably scalable. Too much scalability also is as bad as too little. How much to go down this path will depend on your specific business needs. However it is not a bad idea to talk about specifics to your business stakeholders. They can tell you how and more importantly how much a feature will be used. You can then decide how scalable you want the code for that feature to be. The process of arriving at how much is just right, takes time to set, but once done, you will thank yourselves for the foreseeable future.

Improvements: This is one of the basic purposes of code review. This answers the question, “Can we do the same thing in a better way?”. Better way could mean one or more among, faster performance, better readability, more modularity, and others. You need to keep asking this question in a code review. If you can, then your code review is not complete, if you cannot improve any longer, then probably the code reached here after many rounds of reviews. Or was copied from well reviewed code. Again this is one of those things that has the potential to be overdone, so think carefully how far do you want to go down this rabbit hole without losing your wits.

BNBR: This is lifted from one of the policies of Quora, It means Be Nice, Be Right. The point is that while reviewing , you need to be nice and be right. Being Nice First. The point of a code review is to see if things can be done better by putting multiple heads instead of one. It is not to hurt or massage egos. What can be done by just pointing out issues with data should not descend into a shouting match (verbally or through the keyboard). Make sure that your comments are politely worded and are correct.

Code Scanners: Before you give your code for review, your code should be scanned by tools to make sure that basic checks are done for issues like parenthesis matching, formatting, typos, indentation, naming convention etc. Your reviewers will have a tougher time navigating your code if you do not fix these. If your reviewer finds these issues and not a code scanner, then you have not prepared for your review well.

The Unhappy Path: The code works fine, but some scenarios were not tested. How do you know if your code is able to handle most of the basic errors or exceptions. Your review should make sure that this is in place well before time. Again you need to use this judiciously. You should not overdo it.

Timeliness: Your code review should have a deadline. You cannot indefinitely keep reviewing the code, your review should finish within a deadline. If you ship years late, how will better code help you ?

Dark Spots: Every reviewer may not be able to review all aspects of the code. So it is a good practice to tell in the review comments what aspects were reviewed and what were not, so that everyone knows the extent of the code review. If everyone says it looks good to me, but everyone only happened to review the naming conventions, then it probably was better if only one person reviewed. If each reviewer mentioned this small info, then everyone knows in advance if there were any dark spots in that particular review and they will be able to redress it.

Fatigue: Do not review too many pieces of code at a time. If you happen to be spending a long time reviewing, then probably the code under review is too large or you are reviewing too many PRs at the same time. Reviewing is a thought intensive process and you need to make sure that it is done properly. So please take breaks, if you are tired of reviews and are still somehow powering through that will reflect in the quality of the reviews. A rule of thumb is to not review more than 60 minutes at a time or around 400 lines of code at a time.

Checklists: One good shortcut is to use checklist to review a PR or a piece of code against. These checklists ensure that your mind does not wander, wondering what you have missed and you will also be reviewing against pre decided metrics.

Defects: What do you do with the issues you found ? Not every issue needs to be or can be fixed immediately. You have to decide what to do with each review comment. Whether you will be fixing them, ignoring them or putting back into your backlog. Make sure this is a separate backlog for technical debt.

Overall these are the things that we follow while doing code review. Many of these helped us a lot to make sure that we are reviewing just enough to keep our process wrooming along while at the same time, not ignoring major issues.

No comments yet

Testing framework plays a crucial role in the success of any automated testing process. Choosing the right test automation framework is important as it will maximize the test coverage and improve test efficiency which means a better return on investment.

There are some key points you need to keep in mind while choosing a suitable python testing framework. The framework should justify your testing needs and it should be easy to use. Check if the framework has integrations with other tools and frameworks that you might use. The features, support, stability, and extensibility are also important. So let’s compare the most popular python testing frameworks to make it easier for you to choose the right one.

Robot framework

It is still the most popular python testing framework that uses a keyword-driven testing approach and is used for acceptance testing. To run Robot you will have to install python 2.7.14 or any later version, python package manager, and a development framework like Pycharm.
Advantages

• Opensource
• Platform independent
• No need to learn a programming language to write Robot Framework test cases
• Automatic report generation after executing each built
• Supports behavior-driven, data-driven and keyword-driven approaches
• Easy installation

Disadvantages

• • Not enough support for parallel testing
  • It’s difficult to create customized HTML reports

Gauge

It is an opensource tool developed by the team that made Selenium. Gauge is immensely useful while integrating continuous testing in the CI/CD pipeline. It is gaining popularity as it supports many plugins like python runner, IDE plugins, build management, java runner, etc.
Advantages

• • • Quick defect detection
    • Easy to write test cases
    • Supports multiple programming languages
    • Command-line support
    • Supports all major plugins
    • Cross-browser tests can be automated

Disadvantages

• • • It is relatively new so it will evolve in the coming years

Pytest

Although Pytest is used for different types of testing, it is more preferred for functional and API testing. There are no prerequisites needed for Pytest, just knowledge of python will be enough to get started. It has a simple syntax which makes test execution easier.
Advantages

• • • Supports Fixtures and Classes that help in creating common test objects available throughout a module
    • It allows the use of multiple fixtures
    • It does not require a debugger
    • Xdist and other plugins support makes parallel execution easier
    • It supports parameterization, which is essential while executing the same test with different
    • configurations using a simple marker
    • Large community support

Disadvantages

• • • Test written in Pytest cannot be shared with other platforms

Pyunit

It is a unit testing framework much like Junit but for python language. Also referred to as unittest, it has five core modules. The test loader class is used to load all the test cases and test suites. The test runner shows the result of the test executed using an interface. The test suite is a collection of test cases that are clubbed logically based on the functionalities. A test case contains the actual implementation of the code and the test report contains the organized data of the test results.

Advantages

• • • No need for high-level python knowledge for test execution
    • Extensive report generation
    • Pyunit comes with Python package, no need to install any additional module
    • Simple and flexible test case execution

Disadvantages

• • • Requires boilerplate code
    • Pyunit is derived from Junit and so it still uses camelCase naming instead of snake_case naming method
    • It supports abstraction so the code intent sometimes becomes unclear

Behave

In Behave, test cases can be written in simple language and lets teams execute behavior-driven development (BDD) testing with ease. Behavior-driven development encourages quality analysts, developers, and business managers to work in collaboration to achieve higher efficiency.
Advantages

• • • Easy execution of all kind of test cases and easy coordination
    • Better clarity on the developers and testers output as the format of the spec is similar
    • Domain vocabulary that keeps the behavior consistent in the organization and the system behavior is expressed in a semi-formal language
    • Detailed reasoning and thinking promotes better product specs

Disadvantages

• • • Only for black-box testing

To sum it up

All the above-mentioned frameworks have their specialties like Pyunit is used for unit testing and Behave is good for behavior-driven testing. Although Robot framework is the best tool for a beginner to learn the nuances of automation framework. It’s always better to jot down your requirements based on their priority and then choose the right python testing framework.

No comments yet

Test cases are the first step in any testing cycle and are very important for any project. If anything goes wrong at this step, the impact gets proliferated in the entire software testing process. This can be avoided if the testers use proper procedure and guidelines while creating the test case template.

In this blog, I am going to share some simple yet effective tips which you could use for writing effective test cases. These tips will save you time and effort while optimizing the use of resources.

How to write test cases in a better way

Let’s have a look a the tips to write better test case template.

1. Detailed Domain Knowledge

Domain knowledge in information technology means deep knowledge of business and operational dynamics, the risks involved and the opportunities in that particular project. It is required to follow the best practices in the domain.

2. Break long test cases into many smaller ones

It is better to break the test case into a group of smaller ones if it has too many steps. It would be easier for the developer to backtrack and repeat the test steps if an error occurs somewhere in the test script. If not done than there are high chances that the developer will miss the bug.

3. Preconditions

Before starting on the test case it is suggested confirm all the assumptions that apply to the test and the preconditions that must be met before execution. There can be data dependency or the dependencies on the test environment or any other test cases.

4. Attach Artifacts

Relevant artifacts should be attached to the test cases. This can be done using a test management tool. At the time of product delivery, It will help to track the changes in the application. I will be easy to understand the flow of the function when there is a change at any step which will not be easy to relate otherwise.

5. Test data input

While writing a new test case a tester can share test data wherever applicable to be used for the Test Case within the test case description or add with the specific test case step. This will save time as there is no need to look for the test data anywhere else.

If the values are to be verified then testers can specify the value range or describe what values are to be tested for a particular field. Choose a few values from each class which will give good coverage for your test.
It’s better not to mention the real test data value but the type of data which is required to run the test. In projects where multiple teams use the test data and it keeps changing, mentioning only the type of data will be a wise choice.

6. Organize your work

Use a test management tool to manages your test cases instead of using a spreadsheet. There are many test management tools that can be used to organize the test cases in one place which will increase the productivity of the team.

7. Stop Assuming

It is better to refer to the specification document. Assumptions about the features or functionalities can lead to disagreements between the client and the developers. This gap between the client’s requirement and the application under development will impact the business.

8. Test Case Naming Conventions

To write tests which are easy to understand, we have to stop coding on autopilot and pay attention to the naming conventions. It is required to name our test classes, fields of our test classes, test methods and the local variables while writing automated tests for our application.

It does not matter which team member wrote the test, others will know what feature is tested under what scenario without even looking at the test code.

9. Meet Customer’s Requirements

If the testers miss a bug or write test cases that do not relate to the real world scenarios then it’s just a waste of resources and time. The goal is to meet the customer’s expectations and that can be attained only if the testers think from the users perspective.

10. Cover All Verification Points

It is important to write well-defined test case verification steps covering all the verification points for the feature under test. To make sure that the test Case covers all the verification points match your test case steps with the artifacts given for your project.

11. Avoid Repetitions

Do test automation when needed as it will reduce the manual effort and save a lot of time. The test scripts should be written in such a way that they can be used afterward for some other project.

12. Make it Reusable

Create test case template which could be re-used in the future by other teams. Also, before writing a new test case for your module, find out if there are similar test cases written already for some other project. Doing this you will avoid any redundancies in your test management tools. Call the existing test case in pre-conditions or at a specific design step if there is a need for a particular test case to execute some other test case.

13. All-Inclusive Test Coverage

Test cases should include all the features and functionalities mentioned in the software requirement. Requirement traceability matrix will help in finding the untested functions of the application.

14. Group Similar Test Cases

A test run is a collection of test cases that testers should execute in a particular order. Test cases are often grouped in test runs. It’s preferred to put preconditions at the beginning of a test run rather than inserting them into each test case.

Actually, only a few of the test cases need preconditions, so the field is often left empty. A test management tool will help to customize your forms and create a test case template which will save your time and effort when writing test cases. Another thing to keep in mind is to avoid writing the same instructions several times by moving repeated preconditions to a test run.

15. Easy to Understand

The test cases should be well defined with comments where ever needed so that any other software tester can work on it in the future. Whatever project you work on, when designing test cases, you should always consider that the test cases will not always be executed by the one who designs them. Therefore, the tests should be easily understandable and to-the-point.

In a scenario where the person who wrote all those test cases leaves for some reason and you have a completely new testing team to work with, the entire effort spent during the design phase could go down the drain.

16. Test Case Description

In the description, the testers need to mention all the details about what is going to be tested, what needs to be verified, the test environment and test data.

The information mentioned below should be there in a well-written test case description:

• Test to be carried out
• Testing tools
• Test Environment Details
• Behavior being verified
• Any dependencies like preconditions and assumptions
• Test Data to be used

17. Maintenance and Update

All the test cases should be updated with the new requirements so it’s easier to execute them in the future if there is a need. Even if some other tester wants to use the test case he/she wouldn’t have to go through the details of the script.

Conclusion

The tester needs to have good domain knowledge and should write presentable test cases from the users perspective. A good test case template will make it easier for testers to write good test cases. If there are only a few test steps, consider making a checklist instead and have a look at some relevant test case examples before working on your test case. A test case example will be helpful in creating test case templates too. A test management tool will definitely help in improving the way test cases are created and managed.

Related Articles:

• Audio Interaction Testing On Remote Devices
• Selenium Testing For Effective Test Automation
• 5 Benefits of Parallel Testing
• How to Measure the Success of End to End Testing
• Mobile App Testing Strategies

No comments yet

No comments yet

In the year 2028, there will be around 7.8 Billion mobile users which accounts for 70% of the world population. More mobile users mean more apps and more competition and to lead the competition we need to make sure that our app is flawless. If nearly half of the bugs in your mobile app are discovered by the users, your app’s ratings are going to decline and so are the downloads. This is why the right choice of mobile app testing techniques must be followed in the decision-making process.

Mobile App Testing Strategies

Today, the mobile app market is highly competitive. To be better every day and survive for long, the QA team has to follow a mix of plans that would be responsible for taking the right testing decisions. The testers have to formulate testing strategies to face every situation fearlessly and immaculately. Mobile apps have to be perfect before reaching to the end users so there have to be certain decisions to be taken regarding the testing plan. The following model of mobile app testing plans can be considered for better execution.

In the planning Stage, decisions like Selection of Device matrix, Test Infrastructure (In-house vs. Cloud, Simulator vs. Real device), Testing scope, Testing Tools, Automation (Framework/Tool) are taken. Since it is the first stage, it is the most important one as all the further stages would depend on these decisions. In the next stage which is execution and review, decisions regarding Test Case Design, Testing of user stories, testing types as per Sprint Objective, Progressive Automation, Regression Testing, Review and course correction are taken.

We are going to discuss the planning stage aspects more elaborately

Device Matrix:

It is an important factor, choosing the device as per your target audience’s behavior matters in decisions regarding resting. There are different approaches to the selection of the device matrix.

Approach 1- Selection of Devices based on market research.

Determine the set of devices with your target operating System that will have the highest occurrence of accessing your application by using app purchase data and analytics. For Example- if you support both Android and iOS, and your application will be used across millions of Samsung, Google Nexus and Moto G devices but only thousands of iPhones, you prioritize testing on the Google Nexus and Moto G above the iPhone device. So, this test plan will consist of testing on devices which are prioritized by your market analysis.

Approach 2: Categorize the devices based on Key mobile aspects

This approach highlights the categorization of the devices based on certain mobile aspects which can be considered in formulating the testing strategy. The categorization goes as:

Test infrastructure

This is another element of the planning stage. This focuses on Strategizing on the Infrastructure components like hardware, software, and network which are an integral part of test infrastructure. It ensures that the applications are managed in a controlled way.

Real device, Emulators or Mobile cloud-Where to test?

Choosing the right platform to test as per the testing needs is very important i.e whether to test on the Real device or an emulator or on the cloud

Real Devices

Testing on a real device is anytime more reliable than testing on a simulator. The results are accurate as real-time testing takes place on the device in a live environment. It carries its own disadvantages as it is a costly affair and not all the organizations are able to afford a complete real device laboratory of their own.

Pros:

Reliable- Testing on Real devices always gives you an accurate result

Live Environment- Testing on real devices enables you to test your application on the actual environment on which your target audience working on. You can test your application with different network technologies like HSPDA, UMTS, LTE, Wi-Fi, etc.

User experience- Testing on Real devices is the only way to test your Real-time User experience. It cannot be tested through Emulators or devices Available on Cloud.

Cons:
Maintaining the matrix- You cannot maintain such a huge matrix of mobile devices in your own test lab.
Maintenance- Maintaining these physical devices is a big challenge for organizations.
Network providers- There are more than 400 network providers all over the world. Covering all these network providers in their own test lab is impossible.
Locations- You cannot test how your application behaves when it is used in different locations.

Emulators

The emulator is another option to test mobile apps. These are free, open source and can be easily connected with the IDE for testing. The emulator simulates the real device environment and certain types of testing can be run on it easily. However, we cannot say that the results of emulators are as good as those of real devices. It is slower and cannot test issues like network connection, overheating, battery behavior, etc.

Pros:

Price- Mobile emulators are completely free and are provided as part of the SDK on every new OS release.

Fast- As Emulators are available on the local machine so they run faster and with less latency than Real devices connected to a local network or devices available on the cloud.

Cons:

The wrong impression- Even if you have executed all test cases on emulators, you cannot be 100 % sure it will actually work in the real environment.

Testing Gestures- Gestures like Pinching, Swipe or drag, long press using the mouse on simulators are different in using these gestures on real devices. We cannot test these functionalities on emulators.
Can’t test Network Interoperability- With the help of Simulators you cannot test your application with different network technologies. Like HSPDA, UMTS, LTE, Wi-Fi, etc.

Testing on Mobile Cloud

Mobile cloud testing can overcome the cost challenges like purchasing and maintaining mobile devices. It has all different sets of device types are available in the cloud to test, deploy and manage mobile applications. The tests run virtually with the benefit of choosing the right type device-OS combinations. Privacy, security, and dependency on the internet can be a challenge in this case but it has many benefits that can cater to different testing scenarios.

The organization can choose the right mix of above-mentioned platforms as every platform carries its own advantages and disadvantages. Sometimes a combination of real and emulators is preferred and sometimes all three can be considered as per the testing strategy.

Pros:

Devices Availability- Availability of Devices and network providers is a big gain for cloud users.
Maintenance- When you are using cloud services. Forget about maintenance. These providers take responsibility for maintaining these devices.
Pay per use- You don’t need to buy a device. You only have to pay for the duration you use that device.

Parallel Execution- You can test your complete test suite on multiple devices.

Cons:
Cost- Some providers are a bit costly

Automation Tools for Mobile App Testing on Android and iOS

Nowadays, there are so many automation tools available in the market. Some are expensive and some are freely available in the market. Every tool has its own pros and cons. Choosing the right tool for testing would reduce the QA team effort providing seamless performance at the same time. We will discuss the best mobile app testing automation tools for iOS and Android platforms in 2018.

1. Appium: It is one of the preferred MAT tools by testers. It is open source and free tool available for Android and iOS. It automates any mobile app across many languages and testing frameworks like TestNG. It supports programming languages like Java, C# and other Webdriver languages. It provides access to complete back end APIs and database of the test codes.
Top Features:
-Appium supports Safari on Ios and Other browsers on Android
-Many Webdriver compatible languages can be used such as Java, Objective-C, JavaScript to write test cases
-Support languages like Ruby, Java, PHP, Node, Python.

2. Robotium: It is a free Android UI testing tool. It supports in writing powerful black box test cases for Android Applications. It supports Android version 1.6 and above. The tests are written in Java language and basically, Robotium contains a library of unit tests. Apart from this, Robotium takes a little more effort in preparing tests, one must work with program source code to automate tests. Robotium does not have play record and screenshot function.

Top Features:
-The tests can be created with minimum knowledge of the project
-Numerous android exercises can be executed simultaneously.
-Syncronises easily with Ant or Maven to run tests.

3. Calabash: It is an open source MAT tool allowing testers to write and execute tests for Android and iOS. Its libraries enable the test codes to interact with native and hybrid apps. It supports cucumber framework which makes it understandable to non-tech staff. It can be configured for Android and Ios devices. It works well with languages like Ruby, Java, .NET, Flex and many others. It runs automated functional testing for Android and ios. It is a framework that is maintained by Xamarin and Calabash.

4. Espresso: It is a mobile app testing automation tool for Android. It allows writing precise and reliable Android UI tests. It is a tool targeted for developers who believer automated testing is an important part of CI CD process. Espresso framework is provided by the Android X Test and it provides APIs for writing UI tests to simulate user interactions on the target app. Espresso tests can run on Android 2.33 and above. Provides automatic sync of test actions with the app UI.

5. Selendroid: An open source automation framework which drives off the UI of Android native, hybrid and mobile web application. A powerful testing tool that can be used on emulators and real devices. And because it still reuses the existing infrastructure for web, you can write tests using the Selenium 2 client APIs.

6. Frank: Is an open source automation testing tool for the only iOS with combined features of cucumber and JSON. The app code needs not to be modified in this tool. It includes Symboite live app inspector tool and allows to write structured acceptance tests. It is tough to use directly on the device but is flexible for web and native apps. It can run test both on simulator and device. It shows the app in action by showing its recorded video of test runs.

Above are a few promising, popular and most commonly used and mobile app testing automation tools. Choice of tools certainly resolves many testing-related problems faster and efficiently. Implementing these tools requires skill and experience and so an organization needs to have a proper testing team in place to make all of this possible.

No comments yet

We are moving toward a future where everything is going to be autonomous, fast and highly efficient. To match the pace of this fast-moving ecosystem, application delivery times will have to be accelerated, but not at the cost of quality. Achieving quality at speed is imperative and therefore quality assurance gets a lot of attention. To fulfill the demands for exceptional quality and faster time to market, automation testing will assume priority. It is becoming necessary for micro, small, and medium-sized enterprises (SMEs) to automate their testing processes. But the most crucial aspect is to choose the right test automation framework. So let’s understand what a test automation framework is.

What is a Test Automation Framework?

A Mobile Testing automation framework is the scaffolding that is laid to provide an execution environment for the automation test scripts. The framework provides the user with various benefits that help them to develop, execute and report the automation test scripts efficiently. It is more like a system that was created specifically to automate our tests. In a very simple language, we can say that a framework is a constructive blend of various guidelines, coding standards, concepts, processes, practices, project hierarchies, modularity, reporting mechanism, test data injections etc. to pillar automation testing. Thus, the user can follow these guidelines while automating applications to take advantage of various productive results.

The advantages can be in different forms like the ease of scripting, scalability, modularity, understandability, process definition, re-usability, cost, maintenance etc. Thus, to be able to grab these benefits, developers are advised to use one or more of the Test Automation Framework. Moreover, the need of a single and standard Test Automation Framework arises when you have a bunch of developers working on the different modules of the same application and when we want to avoid situations where each of the developers implements his/her approach towards automation. So let’s have a look at different types of test automation frameworks.

Types of Mobile Automated Testing Frameworks

Now that we have a basic idea about Automation Frameworks, let’s check out the various types of Test Automation Frameworks available in the marketplace. There is a divergent range of Automation Frameworks available nowadays. These frameworks may differ from each other based on their support to different key factors to do automation like reusability, ease of maintenance etc.

Types of Mobile testing automation frameworks:

Module Based Testing Framework

Module-Based Testing Framework, as the name implies, depends on a number of modules to function. In order to produce the greatest results from the automation test, you would need to develop unique scripts for each module and ensure that they work together. Changes to the application’s functionality won’t have an impact on the modules. The scripts are safe unless they are manually changed.
Given that a high level of modularization is produced by merging multiple modules, this provides a cost-effective management approach. Productivity is still at its highest level. But, if necessary, it can take a lot of time and effort to make modifications to the test data individually.

Library Architecture Testing Framework

Based on the modular foundation, the library architecture framework for automated testing offers several extra advantages. Instead of separating the programme under test into the many scripts that must be executed, related jobs inside the scripts are found and afterwards grouped by function, allowing the application to be eventually divided up into common goals. The test scripts can access this library of functions anytime they are required.

Data Driven Testing Framework

A number of tests must be run while testing an automation framework before a successful result can be determined. In these situations, you might need to alter the test results to try and draw a different conclusion. You can keep the test data on an external drive and access it later for adding a new script to the test case thanks to the Data-Driven Testing Framework.

Keyword Driven Testing Framework

The keyword-driven testing framework, which is frequently regarded as an extension of the data-driven testing framework, collects your test data from an external source and securely preserves the set of codes. These codes, which are also known as “keywords,” can be used to change the test script and draw additional conclusions from the test framework. Also, these keywords effectively determine what tasks each programme performs.

Hybrid Testing Framework

To maximize the effectiveness of the aforementioned frameworks, the hybrid testing framework combines the data-driven and keyword-driven frameworks. It provides more room for more efficiency and success, making it the ideal automation foundation.

Behavior Driven Development Framework

The goal of the Behavior Driven Development framework is to build a platform that encourages active participation from all users, including developers, testers, business analysts, etc. Also, it improves cooperation on your project between the developers and testers. For this behavior-driven testing, test specifications can be written in plain, non-technical language.

Benefits of a Mobile Testing Automation Framework

Apart from the minimal manual intervention required in automation testing, there are many advantages of using a test automation framework. Some of them are listed below:

1. Faster time-to-market: Using a good test automation framework helps reduce the time-to-market of an application by allowing constant execution of test cases. Once automated, the test library execution is faster and runs longer than manual testing.
2. Earlier detection of defects: The documentation of software defects becomes considerably easier for the testing teams. It increases the overall development speed while ensuring correct functionality across areas. The earlier a defect is identified, the more cost-effective it is to resolve the issue.
3. Improved Testing efficiency: Testing takes up a significant portion of the overall development lifecycle. Even the slightest improvement of overall efficiency can make an enormous difference to the entire timeframe of the project. Although the setup time takes longer initially, automated tests eventually take up a significantly lesser amount of time. They can be run virtually unattended, leaving the results to be monitored toward the end of the process.
4. Better ROI: while the initial investment may be on the higher side, automated testing saves organizations many a lot of money. This is due to the drop in the amount of time required to run tests, which leads to a higher quality of work. This in turn decreases the necessity for fixing glitches after release, thereby reducing project costs.
5. Higher test coverage: In test automation, higher number of tests can be executed pertaining to an application. This leads to a higher test coverage, which in a manual testing approach would imply a massive team, limited heavily with their amount of time. An increased test coverage leads to testing more features and a better quality of application.
6. Reusability of automated tests: The repetitive nature of test cases in test automation helps software developers to assess program reaction, in addition to the relatively easy configuration of their setup. Automated test cases can be utilized through different approaches as they are reusable.

Top ten test automation frameworks

1. Robot Framework
Robot Framework is the best choice if you want to use a python test automation framework for your test automation efforts. The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET). The Robot Framework uses a keyword-driven approach to make tests easy to create. Robot Framework can also test MongoDB, FTP, Android, Appium and more. It has many test libraries including Selenium WebDriver library and other useful tools. It has a lot of API’s to help make it as extensible as possible. The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.

2. WebdriverIO
WebdriverIO is an automation test framework based in Node.js. It has an integrated test runner and you can run automation tests for web applications as well as native mobile apps. Also, it can run both on the WebDriver protocol and Chrome Devtools protocol, making it efficient for both Selenium Webdriver based cross-browser testing or Chromium based automation. As WebDriverIO is open source, you get a bunch of plugins for your automation needs. ‘Wdio setup wizard’ makes the setup simple and easy.

3. Citrus
Citrus is an open-source framework with which you can automate integration tests for any messaging protocol or data format. For any kind of messaging transport such as REST, HTTP, SOAP, or JMS, Citrus framework will be suited for test messaging integration. If you need to interact with a user interface and then verify a back-end process, you can integrate Citrus with Selenium. For instance, if you have to click on a “send email” button and verify on the back end that the email was received, Citrus can receive this email or the JMS communication triggered by the UI, and verify the back-end results, all in one test.

4. Cypress
Cypress is a developer-centric test automation framework that makes test-driven development (TDD) a reality for developers. Its design principle was to be able to package and bundle everything together to make the entire end-to-end testing experience pleasant and simple. Cypress has a different architecture than Selenium; while Selenium WebDriver runs remotely outside the browser, Cypress runs inside of it. This approach helps in understanding everything that happens inside and outside the browser to deliver more consistent results. It does not require you to deal with object serialization or over-the-wire protocols while giving you native access to every object. Cypress can synchronously notify you of every single thing that happens inside the browser as you’re pulling your app into it, so that you have native access to every DOM element. It also makes it easy to simply drop a debugger into your application, which in turn makes it easier to use the developer tools.

5. Selenium
One of the most popular open source test automation frameworks for web apps. Selenium also serves as a base for a lot of other testing tools as it has cross-platform and cross-browser functionality. Selenium supports a wide range of programming languages such as Java, C#, PHP, Python, Ruby, etc. It is easy to maintain as it has one of the largest online support networks. Selenium is highly extendable through a wide range of libraries and APIs to meet everyone’s needs and requirements. Selenium is preferred by testers as it is possible to write more advanced test scripts to meet various levels of complexity. It provides a playback tool for test authoring without the need to learn a specific scripting language.

6. Cucumber
It is a cross platform behavior driven development (BDD) tool which is used to write acceptance tests for web applications. Cucumber is quick and easy to set up an execution and allows reusing code in the tests. It supports languages like Python, PHP, Perl, .NET, Scala, Groovy, etc. Automation of functional validation in easily readable and understandable format. One good feature is that both specification and test documentation are uploaded in a single up-to-date document. Cucumber makes it easy for the business stakeholders, who are not familiar with testing, as they can easily read the code as test reports are written in business readable English. The code can be used together with other frameworks like Selenium, Watir, Capybara, etc.

7. Gauge
It is an open source tool agnostic test automation framework for Mac, Linux and Windows. People who work on TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation. Specs – the Gauge automation tests are written using a markdown language with C#, Java and Ruby within your existing IDEs like Visual Studio and Eclipse. Gauge’s functionality can also be extended with its support of plugins. It was developed as a BYOT (Bring Your Own Tool) framework. So you can use Selenium or you can use anything else for driving your tests UI or API tests. If you want a readable non-BDD approach to automation, you should try Gauge.

8. Serenity
If you are looking for a Java-based framework that integrates with behavior-driven development (BDD) tools such as Cucumber and JBehave, Serenity might be the tool for you. It’s designed to make writing automated acceptance and regression tests easier. It also lets you keep your test scenarios at a high level while accommodating lower-level implementation details in your reports.

Serenity acts as a wrapper on top of Selenium WebDriver and BDD tools. It abstracts away much of the boilerplate code you sometimes need to write to get started which makes writing BDD and Selenium tests easier. Serenity also offers plenty of built-in functionality, such as handling running tests in parallel, WebDriver management, taking screenshots, managing state between steps, facilitating Jira integration, all without having to write a single line of code.

9. Carina
Carina is built using popular open-source solutions like Appium, TestNG and Selenium, which reduces dependence on a specific technology stack. You can test mobile applications (native, web, hybrid), WEB applications, REST services, and databases. Carina framework supports different types of databases like MySQL, SQL Server, Oracle, PostgreSQL, providing amazing experience of DAO layer implementation using MyBatis ORM framework. It supports all popular browsers and mobile devices and it reuses test automation code between IOS/Android up to 80%. API testing is based on the Freemarker template engine and it provides great flexibility in generating REST requests. Carina is cross-platform and tests may be easily executed both on Unix or Windows OS.

10. EarlGray
Developers often face difficulty with some of the existing test automation framework in synchronization of the app and the instrumentation. Also, executing tests on apps as synchronized and advanced only when UI elements are visible on the screen has caused issues for many developers. Google EarlGrey has built-in synchronization that makes test scripts wait for UI events to occur before the script tries to interact with the UI of the app. This type of implementation makes the test script concise as all steps of the test script shows how the test will proceed and UI gets synchronized with it. One more key aspect of EarlGrey is that all actions on UI elements happen only on visible elements. This provides a fast and robust approach to ensure UI testing goes through as clicks, gestures and other user interactions do not get done if the UI element is not fully shown.

In a nutshell

This list of top tools here represents the best tools that are mature, popular, and provide test automation capabilities using AI/ML to address the challenges that organizations are now facing to deliver Quality at Speed. This list also includes the tools that provide API and services testing which is essential for successful DevOps transformation. The emerging technologies like AI, codeless, big data and IoT testing, are making test automation more efficient while creating opportunities for the existing tools and new players to assert value to the testing communities.

The choice of automation tools should not only meet your current needs but should also focus on potential trends and improvements. An efficient test automation tool should support basic optimization, data generation, smarter solutions, and analytics. As of now, the level of test automation in organizations is low at between 14% and 18%. But organizations are working towards increasing the automation coverage upto 80%. API and services testing is also a trend that should see further development in the future.

No comments yet