Six Steps to AI-Powered Testing: A Practical Roadmap to Transform Your QA Practice

Dinakar

Welcome back to the Testing Reality podcast episode 20. Joining me is our dynamic senior enterprise sales manager So Anuja, thank you for joining us. Thank you for being here.

Anuja

Thank you for inviting again, Dinakar.

Dinakar

Uh great. So, let's uh So, let's dive in, right? Let's talk about uh what we've gathered here today for, right? A lot of teams are excited about AI and testing. Uh but excitement and adoption are two very different things, right? And when So, when teams come to you and they approach and ask you, "Where do we exactly start?" What is your answer?

Anuja

So, Dinakar, very good question. Uh we tell them not to think about AI as a single tool or a feature.

Dinakar

It's basically a journey.

Anuja

Uh over time, we have seen uh that the teams getting the most value from AI tend to follow a similar path.

Dinakar

They first build a strong testing

Anuja

Then layer AI capabilities on on top of

Dinakar

So, uh I would summarize that uh into some six steps uh which all this is all

Anuja

So, first three steps focus on getting the fundamentals right.

Dinakar

Next three steps focus on using AI to make testing faster, uh smarter, and more efficient.

Anuja

So, in some sense, this is not like a overnight transformation, but it's a journey that you mentioned.

Dinakar

Exactly. It's about taking a practical uh step-by-step approach and uh building confidence along the way.

Anuja

Perfect. So, let's get started. Let's go straight into step number one, then.

Dinakar

Okay. So, talking about step one, uh that would be connecting uh testing to your delivery pipeline.

Anuja

So, think about this. Whenever a new build is created, testing should start

Dinakar

Uh, no one should have to remember to trigger it manually.

Anuja

It It sounds very basic, but it makes a huge difference.

Dinakar

Because, uh, feedback reaches developers much faster.

Anuja

Mhm. So, no manual intervention, but making that automatic. Okay. So, testing becomes a part of the development journey itself instead of a separate activity in such a way.

Dinakar

Absolutely. Absolutely. So, faster teams get feedback, uh, they can start fixing the issues. That is again going to be much more faster.

Anuja

Okay. So, let's move to step number two. What is step number two?

Dinakar

Okay. So, I would say step two is going to be testing on real devices. So, a lot of teams begin with emulators and

Anuja

Reason being, uh, procuring all these real devices could be a challenge. Right? I mean, like it is going to be much more, uh, expensive procurement overhead.

Dinakar

Which is still fine. I mean, like if you are working, I mean, like testing on emulators and simulators. But at some point, you need to validate how the app behaves in the real world. Because, uh, then again, the problem with emulators and simulators is you won't get a precise, uh, test results. What you can expect is on the real devices. It could be real device on cloud or physical So, what we do is we usually, uh,

Anuja

starting with one critical user journey.

Dinakar

And running it across the devices your customers actually use.

Anuja

So, you have accurate results, and then there's no surprise and that's where most of the surprises come up in some

Dinakar

Almost always. I would say like 97 to 98% uh, you can have the accurate, uh,

Anuja

So, you discover, uh, performance differences. Uh, it could be visual inconsistencies or, you know, like some pixel change in on any of the OEMs. It could be device specific issues because we have Android, we have iOS, it could be low-end devices, Chinese models, etc. So, things that look perfectly fine in a simulated environment suddenly becomes very much visible and evident. So, think about this. Those insights are valuable for the team. I'm not saying like you should not be using emulators or simulators. But again, it would be equally valuable for AI systems learning from your testing data if you test it on real

Dinakar

I mean, I absolutely agree with you because the more accuracy that you provide to AI, the more it makes sense and the foundation is built in some sense. So, let's move to step number three, right? You mentioned step three. So, what is step three?

Anuja

Okay. This is again a very important steps because step three is going to be more of creating baselines. Before AI can identify something unusual, it needs to understand what's How long does a screen usually take to load or it could be what's considered acceptable performance or what should the application look like on different So, once the team has identified, those baselines are established, every future run has something meaningful to compare

Dinakar

Okay. So, instead of saying something changed, we know exactly uh you know, what has changed or where exactly we have to bring that change or

Anuja

Correct. That's where testing starts becoming much more proactive.

Dinakar

Okay, great. I think we have a fair understanding of the foundational steps. Uh so, let's move to the AI driven journey, right? So, let's start with the step number four.

Anuja

Right. So, we have discussed majorly onto the foundation. Now comes the AI layer to it. So, that this is where the foundation it paying off in a in a simple words. I would say step four is basically AI-powered test selections.

Dinakar

So, most teams have larger test suites, but not every test needs to run for every change. So, you have to had that mix and match of, you know, which test suites has to be run on which changes.

Anuja

Right. Right.

Dinakar

AI can analyze what's been modified and intelligently select the test that are most relevant.

Anuja

Which means that teams get test results more faster.

Dinakar

Correct. Instead of waiting hours for a full suite to complete, teams can focus on the tests that matter most for that particular build. Because release cycle and to fix issues is going to be very very prominent. So, you have to be very much cautious, and that's where AI comes into picture.

Anuja

I mean, makes sense. Uh but, how do teams know that it's actually working?

Dinakar

Uh I would say a simple test is whether the test chosen by AI makes sense to experienced testers.

Anuja

Because if your senior QA looks at the selected test and says, "Hey, I would have picked those two." That's usually a good sign.

Dinakar

So, let's So, I think step four is clear about, you know, AI test selection. Let's go to step five, right? What is step five?

Anuja

Okay. Now, we have talked about the positives. Now comes the failure classification, which is the step five.

Dinakar

Uh anyone in testing knows that finding a failure is only the beginning, right? I mean, it's a fun part, though. Just we have like I mean, like found out what the failure is. But, bigger challenge is figuring out why it happened.

Anuja

The root cause analysis, the RCA.

Dinakar

Uh AI can help categorize failures It could be a product issue, a flaky test, environment problem, OEM device, or it could be anything else apart from

Anuja

And I'm sure this kind of removes a lot of the manual investigation and saves a lot of time for our engineers for sure.

Anuja

A humongous amount Dinakar I would say. Instead of spending hours sorting through reports and logs, teams can focus on resolving actual issues.

Dinakar

Perfect. Well, what is step six?

Anuja

Okay, so now we have discussed in detail about all five steps. The last step is going to be the most important thing that is release readiness. This is where all the signals come If I talk about the signals that is test execution results, could be performance trends, visual checks. Failure analysis. So AI looks across all of that information and helps answer a very important question. that is are we ready to release?

Dinakar

Not replacing the entire human judgment in some sense but giving the decision to them, making it smarter.

Anuja

Absolutely. So this is very important for everyone to understand. AI is basically artificial intelligence, right? Human intervention is always needed. So the goal isn't to remove people from the process. It's to give them better insights and more confidence before they ship because that's a myth. People think like once AI has come into picture, their roles will be at stake but that is not the case.

Dinakar

Well, this makes sense. I mean you've given a rough break through of all the steps but you know, what should teams measure as progress in each of these

Anuja

Dinakar, keep it very very simple. Just just check with your teams are test triggering automatically every time. Are you finding issues on real devices that weren't visible before? When when I say visible before meaning on emulators or simulators. Have you established performance and visual baselines? Is AI helping reduce the number of tests you need to run or is it again like going to be adding up your task?

Dinakar

So, that is also one of the very important things you need to check and analyze. Are teams spending less time investigating failures because in one of the steps we have discussed uh, why and what. If they are just spending a lot of time investigating failures, then something is wrong with the AI.

Anuja

And are AI recommendations increasing aligned with the judgment of experienced testers? These are basically I would say a strong indicators that you are moving in the right directions. Once you have all these answers, you are at the right

Dinakar

So, we looked at these six steps as a sprint, right? And so, what is important to remember about this if you want to like sum that up into

Anuja

So, Dinakar, this is basically just a starting point. A 2-week sprint isn't going to transform an entire testing practice overnight.

Dinakar

I agree. Yeah.

Anuja

What it does is uh, I mean like it just provides you proof. Proof that the approach works.

Dinakar

Proof that the data is useful. Proof that AI can create value in your environment, right? I mean, if you go for uh, step four to six, so yeah, that's what it says all about AI.

Anuja

From there, teams can expand gradually and build on that momentum.

Dinakar

Okay. And I think that's something that we're going to be covering in the next

Anuja

Yeah. So, next uh, this week was about getting it started. And if you're inviting me again the next week, we will probably talk about what happens uh, after the sprint.

Dinakar

What we can do is we can discuss more about a 90-day roadmap, like a 3-month

Anuja

Uh, what it looks like for scaling AI-powered testing across teams.

Dinakar

Perfect. I think that wraps up our episode to this week. So, it [snorts] was a pleasure having you on uh uh

Anuja

Always a pleasure, Dinakar. Thank you so much for inviting.

Dinakar

Thank you for being here.