Why Emulators Fail and Real Devices Win: Breaking the Mobile Testing Infrastructure Barrier

Dinakar

Welcome to the testing podcast.

Dinakar

Over the last few weeks, we've been talking about why bugs still reach production uh even though the testing teams are constantly testing them.

Dinakar

Right?

Dinakar

So, and we spoke about the emulator gap.

Dinakar

We discussed about the disease coverage gap.

Dinakar

We covered the visibility gap as well.

Dinakar

And we spoke about this over the weeks.

Dinakar

And today we're going to dive deeper into the actual root cause of it all.

Dinakar

And because a lot of teams don't actually have a testing problem, they have an infrastructure problem.

Dinakar

And joining me today is Jessica who's worked closely with a lot of QA and engineering teams across industries be fintech, e-commerce or uh you know the healthcare or the bank etc.

Dinakar

And she has seen exactly where testing breaks at scale.

Dinakar

So, welcome to the podcast, Jessica.

Jessica

you.

Dinakar

to have you here.

Jessica

you.

Dinakar

all right.

Dinakar

So, let's start with the number that keeps haunting us all the time, right?

Dinakar

4 hours.

Dinakar

And you've said that many teams uh take around 4 hours for a typical device testing cycle.

Dinakar

Uh, but the tests themselves might take only a few minutes.

Dinakar

So, where does the actual I mean, where does the four hours go if it's just like a few minutes that it

Jessica

So this is the part that most people miss.

Jessica

When teams are saying testing takes too long, they assume the scripts are slow.

Jessica

But in reality, the scripts might run in 10 minutes.

Jessica

The four hours comes from everything around testing, right?

Jessica

Finding the right device, the access, uh waiting for someone else to finish the OS versions, device resets and app data issues.

Jessica

Uh right, and coordinating across teams.

Jessica

So you'd be surprised how much time is wasted on that one question.

Dinakar

let me guess.

Dinakar

I know this question.

Dinakar

So, where is the iPhone?

Jessica

Right.

Jessica

Device availability.

Jessica

That question alone is responsible for more delayed releases than some production bugs.

Dinakar

it's funny and scary at the same time, right?

Dinakar

Uh but anyway, over the last few episodes, you know, we've built on this idea that, you know, the teams don't fail because they don't care.

Dinakar

They fail because testing doesn't match reality.

Dinakar

And one thing that we have observed is something that we called the device testing wall.

Dinakar

Uh so could you please elaborate and talk about that for a bit?

Jessica

sure.

Jessica

So um the device testing wall happens when a team scales right.

Jessica

So early stage testing is manageable.

Jessica

You have five or 10 devices a small QA team and things feel fine.

Jessica

But the app grows, your users grow, your markets expand.

Jessica

uh the number of testing uh or the devices that you need to test in the wild explodes, right?

Jessica

And then leadership says we need broader coverage.

Jessica

Now QA has to test on 30 plus devices or maybe 50 or 100 then right and suddenly the entire QA function becomes uh more of a uh device operations where they need to have broader coverage and scale quickly.

Jessica

So basically what you're telling me is that testing becomes less about quality and more about device management and

Dinakar

Yeah.

Dinakar

It ends up and a lot of time gets consumed with that.

Dinakar

Right.

Dinakar

And that's when uh you start hearing the same lines everywhere.

Dinakar

The device lab is a mess.

Dinakar

The phone is missing.

Dinakar

Uh the device is locked.

Dinakar

It's stuck in an old build or the battery is dead again.

Dinakar

Right?

Dinakar

And because of which we can't run the suite today.

Dinakar

And then the biggest one, right?

Dinakar

Testing is slowing down the releases.

Dinakar

So that's the wall.

Dinakar

More devices means more management.

Dinakar

And more management invariably ends up meaning more friction.

Dinakar

And friction kills speed.

Jessica

I'm sure that you know some of the teams listening to this might say, you know, okay, fine.

Jessica

So what's the big deal?

Jessica

Uh why don't we just run everything on emulators?

Jessica

It's cheaper.

Jessica

It's faster.

Jessica

And I mean there's no hassle of device management involved, right?

Jessica

So why doesn't that solve the

Dinakar

So that's a great question, right?

Dinakar

Well, uh, emulators are useful, but they're not reality, right?

Dinakar

They're great for development, iteration or quick checks and fast feedback loops, but production bugs don't happen in emulator environments, right?

Dinakar

They happen in the messy real world conditions.

Dinakar

And emulators can't capture things like memory pressure on real hardware or UI differences between OEMs.

Dinakar

Uh right, a network switching or uh battery constraints.

Dinakar

All right, and there are several other background restrictions and 40 other apps running on a typical you know a real device situation.

Dinakar

So uh and that's what we had clarified in the earlier episodes as well.

Dinakar

You can pass every test and still fail in production.

Dinakar

Well, that's I mean that's that's great.

Dinakar

So that brings us to the focus for today's uh podcast, right?

Dinakar

So if if teams need to uh need real devices but device labs don't scale, right?

Dinakar

Then what's the answer?

Dinakar

Could could you take a dig at and explain what a device cloud actually

Jessica

sure.

Jessica

The breakthrough is simple, right?

Jessica

Teams don't need more devices.

Jessica

uh they need more coverage without the management burden, right?

Jessica

Most device labs fail because the moment you scale devices, you scale the chaos as well.

Dinakar

instead of asking how do we help teams manage a 100 devices, the best better question uh right would be how do we give teams access to a 100 devices without them managing anything right and that's what a device cloud does.

Dinakar

You log into a digital testing platform like Pcloudy and devices are there available instantly for you right be it Samsung or Pixel or OnePlus they're all ready right and old OS versions as well as the new right budget phones as well as uh flagship phones are ready and the best part is parallel execution so instead of testing one by one teams can run tests across dozens of devices at the same

Jessica

well that makes sense.

Jessica

Uh because you know and and that is how I think the wall breaks.

Jessica

Is is that what you're hinting at?

Dinakar

Because uh parallel execution really collapses time.

Jessica

Wow.

Jessica

That's it's it's that simple is it?

Dinakar

In fact, in a way, yes.

Dinakar

Right.

Dinakar

We had uh recently helped a fintech uh firm solve a very similar issue.

Dinakar

This team had the same story we've heard everywhere, right?

Dinakar

Their full device testing cycle took around 4 hours and because the tests were slow uh and the execution was serial the access was painful.

Dinakar

uh right and when they moved to parallel execution on a device cloud uh we could really observe the cycle dropped to 23 minutes same suite more devices so more coverage as well and less time and they ran it again because u they didn't usually trust the first result and they got the same

Jessica

what changed wasn't just speed the entire testing culture had changed

Dinakar

because suddenly testing isn't the thing that that's blocking their releases anymore, I'm sure.

Jessica

So, when you see those kind of results, uh the trust comes back again and you're able to see that testing can be predictable.

Jessica

It can be repeatable.

Jessica

And once testing becomes predictable, teams stop cutting corners.

Jessica

Uh they stop skipping coverage.

Jessica

Uh they stop rushing the last minute sanity test before release and they actually start testing like professionals again.

Dinakar

Now that's interesting because in most organizations right you've seen the QA being the gatekeeper in some sense and uh the thing that slows down their own product right so what happens to the team when you remove this friction

Jessica

uh so first things that you would observe when something like this happens is right you would observe yeah first of all everything improves in a sense right and uh they test more devices uh so you naturally have more coverage without those management overhead and secondly they catch issues earlier because they have real world coverage.

Jessica

So the biggest shift is that testing stops becoming the bottleneck.

Jessica

It becomes an enabler for you and in this fintech team that I was referring to right their deployment frequency tripled

Dinakar

because leadership demanded it but because the constraints were removed.

Dinakar

Uh speed creates confidence and confidence creates momentum.

Dinakar

Well, that I mean speed creates confidence.

Dinakar

I love that line and that line alone is worth another episode.

Dinakar

Uh because you know without confidence teams delay shipping or uh they ship with a lot of fear.

Jessica

Exactly.

Jessica

And when fear is removed, engineering teams actually become faster.

Dinakar

teams start shipping more often uh and they ship faster.

Dinakar

And that's what modern enterprises and product teams

Jessica

So basically uh you're saying that many QA teams aren't really testing.

Jessica

Uh I mean we've already covered this in the previous episode as well.

Jessica

Uh they're running more like a phone museum at the moment.

Dinakar

Yeah.

Dinakar

Uh and a museum where everything is outdated and half the exhibits are missing and the security guard is your QA lead,

Jessica

And somehow uh the iPhone is always the one that is the most revered artifact in that whole museum.

Dinakar

Jessica a final question like so if we were to take this talk and summarize in in this talk uh so what is the one key to key takeaway from today?

Dinakar

Yeah, it's this right.

Dinakar

The biggest bottleneck in mobile testing isn't test writing.

Dinakar

It's the device infrastructure.

Dinakar

And when you remove that device friction, teams don't just test faster, they ship

Jessica

and on next week on the testing reality, we're going to tackle something we've hinted at multiple times, right?

Jessica

uh because some teams want device cloud speed but they can't really use public clouds like banking or healthcare or you know government regulated or other regulated industries where compliance is a big issue and uh and they need the same parallel execution uh but under stricter compliance rules.

Jessica

So we'll unpack what teams do when public cloud is not an option.

Jessica

Uh so thanks for joining and and we'll see you next week.