Shivani
Hello everyone.
Shivani
Welcome to our most awaited uh expert panel discussion on AIdriven testing trends 2023 organized by PC Cloudy.
Shivani
I'm Shivani and I'm I'm excited to be your host for today's event.
Shivani
Uh before we get started, I would like to go over a few housekeeping rules for today.
Shivani
Your line is currently muted, but don't worry.
Shivani
You can submit your questions during the panel discussion through the chat options at the bottom of the screen.
Shivani
We'll answer all your questions as part of the Q&A section at the end of the panel discussion.
Shivani
Please note that this panel discussion is being recorded and we will send out the link of the recording so that you can share it with your friends and colleagues later.
Shivani
For those of you who may not be familiar with Pcloudy, we are a cloud-based testing platform uh that offers a diverse range of more than 5,000 device browser combinations for testing mobile and web applications with advanced automation techniques, integration with popular testing frameworks and tools, and roundthe-clock premium support from product specialists.
Shivani
Uh we are trusted by over 300,000 registered users including small, medium and large enterprises.
Shivani
The uh so uh I mean when we talk about this uh topic for the discussion right today so we understand the world of testing is evolving rapidly now and uh AI has disrupted this field and uh with the artificial intelligence coming in and getting advanced day by day we are witnessing an unprecedented change in the way we test our applications so today uh we have brought together a panel of experts who have contributed Ed to PC Cloudy's annual testing trends report which we come up with every year but this year it has been really a different experience since we had done this study together with with these uh panelists that you see uh on your screen today.
Shivani
This is a comprehensive study on the latest trends and advancements in AIdriven testing and their impact on the future of app development and de delivery.
Shivani
So in this panel discussion we will uh delve deeper into unique insights into the future of AI in testing.
Shivani
Our panel of experts will share their opinions and insights on the state of AIdriven testing.
Shivani
Uh so the impact uh uh that it will bring in the testing industry and how organizations can be prepared for this shift.
Shivani
So uh getting to our panelists over here.
Shivani
So our esteemed panelists over here are so we have Aishwarya Gupta who has 20 more than 20 years of experience in data and AI domain driving business growth and enterprise transformation.
Shivani
Currently she is with Vipro as AI practice head.
Shivani
Welcome EA.
Aishwarya
Hey everybody.
Shivani
Uh we have Kevin Pies who is an experienced professional in QA and artificial intelligence.
Shivani
He's currently serving as director of test engineering with OC Tanner.
Shivani
He's also an international speaker and a promoter of AI as the future of testing.
Shivani
Welcome Kevin.
Kevin
Thanks for having me.
Shivani
Uh we have George Ukuru who chief solutions officer at Test House Limited and is an expert in quality assurance, test automation, RPA, DevOps, functional testing and many more.
Shivani
He's also a thought leader, author of three books and moderates the automation hangout podcast.
Shivani
So George, welcome to the panel today.
George
Thank you.
Shivani
Uh we uh have Jonas Minesclu who's the co-founder and manage managing director of Ask UI a KLA based startup that specializes in selector independent UI automation.
Shivani
He studied industrial engineering with the focus on user centric IT security and QA.
Shivani
He's also the organizer of the meetup group software testing KL Raer.
Shivani
Welcome Jonas.
Jonas
Thanks for having me.
Shivani
Hi D.
Shivani
Uh we had another panelist supposed to join Daniel not but unfortunately he is not uh he is not able to join us today because of an unexpected emergency at this place.
Shivani
Uh so now moving on to our moderator for today's discussion Avinash Dari.
Shivani
He's the co-founder of Pcloudy and a renowned keynote speaker and a thought leader with more than 20 years of experience in product development and testing.
Shivani
He's known for his passion for emerging technologies and brings brings a wealth of experience and knowledge to to this discussion.
Shivani
Uh welcome Avanach.
Shivani
So uh without further delay, let me hand it over to Avinash to take the discussion forward.
Avinash
Thank you so much Shivani.
Avinash
Uh great introduction and welcome everyone.
Avinash
Uh there is so much of interest around this topic and I can see so many of you there as part of the session today.
Avinash
Extremely happy to see that and extremely happy to have this panel here.
Avinash
Uh experts in their own areas.
Avinash
Um they have transformed testing testing over a period of time within their organizations.
Avinash
They are speakers in various conferences.
Avinash
So there is so much to learn from from these experts for me as well.
Avinash
So I'm really excited to be uh moderating this session and getting this opportunity to learn from these experts.
Avinash
Um so with that uh what we will do is over next 25 to 30 minutes we will hear the views and opinions from the experts and I'll be moderating the session uh through some of my questions that I have but you can uh you can send your questions in the chat whenever you feel like you can also direct some some of your questions to specific experts uh depending on what what's your interest area is and I will take up those questions maybe towards the end of the panel discussion and we'll try to keep a good amount of time for your question and answers.
Avinash
Um so that's the format we'll keep.
Avinash
So without kind of delaying further uh let's kind of get into the uh the main agenda of this panel discussion today.
Avinash
Um and uh to begin I would ask a general questions to gender question to all the panelists uh and u I will probably point out the name uh in no particular order and you can share your views and we can go in go in a round robin f um I mean for me uh I think it was 2018 where um uh as cloudy we started investing in AI and one of the first uh use case that we picked up was object identification and we were quite impressed with what what Google vision API was offering and that was the first instance when we started using AI and from there on we built a lot of our own uh proprietary technologies we have also used a lot of open-source but that was I think 2018 first instance where I uh started uh exploring AI and was quite amazed with what it can offer uh within the testing space itself.
Avinash
So my question to the panelists here is uh to begin with if you can share with us when and under what circumstances you first got introduced to AI what really triggered uh this passion towards AI for you and how you started thinking that AI can be part of your uh organization within the testing uh area itself.
Avinash
So probably Kevin you can start share your views for two minutes and then we can go to the next panelist if is that okay?
Kevin
Yeah.
Kevin
Yeah.
Kevin
Yeah.
Kevin
So for me uh I love technology.
Kevin
I just I've been excited about technology for so long and I went to a conference uh Star West in 2016.
Kevin
So I beat you by just a couple years there on on getting started.
Kevin
Um, but there was a presentation by Jason Arban on AI and testing and at the time it was like ah whatever, we'll go check it out.
Kevin
Uh, see what he has to say.
Kevin
And it it was one of those uh visionary presentations.
Kevin
It wasn't here's what we can do today.
Kevin
It's here's where it's going.
Kevin
And it got me so excited.
Kevin
I just walked away just pumped like yeah, this is exactly what we need to do.
Kevin
And it's taken a little while uh for things to to play out u the way he he was hoping they would, but oh, it just gets me excited um what AI can do for us and the excitement now, especially around testing.
Kevin
It seems like it's been so long since we've had real innovation in the testing space.
Kevin
And with AI, I think we're finally starting to see those changes that that we've needed um honestly in the testing space.
Kevin
So that that's where it got me excited.
Kevin
Um where that AI uh he called it AI curious and I was definitely AI curious at the time.
Avinash
Absolutely.
Avinash
I have been following uh him for a long time.
Avinash
He's a great personality and as you rightly mentioned a visionary in testing for sure.
Kevin
Yeah.
Avinash
Uh Jonas maybe you if you can share your views on where did you start or where did you get this bug?
Jonas
Yeah.
Jonas
Uh thanks.
Jonas
So for me it wasn't the talk by Jason Arban but maybe another talk from a professor at university.
Jonas
So uh back then in 2018 2019 I was still studying um and in previous jobs my and my myself had this problem with test automation with selectors was painful and we were visiting like a um some courses about computer vision and we were like man why don't we apply this to uh testing software testing for locating objects and that's actually where aski was born uh and then we dig deeper it turned out to be harder than we thought it but um that's where it all started.
Jonas
I guess that was the kicking off project or moment that clicked in my head.
Avinash
Great to know.
Avinash
Uh Jonas Aishwarya you I mean I I know you from long back when we were just doing manual testing in in Oracle and from that point of time testing has evolved so much.
Avinash
So what was your uh uh what was that point of time when you started exploring AI and how did that happen?
Aishwarya
Yeah, sure.
Aishwarya
So for B there were two Euro moments when when and it was similar time at around 2017 around when we were looking at uh you know um automation was at the peak and uh waterfall was going down whereas more and more people were wanted the agile methodology projects to be delivered with that there is there was a huge impact of speed to market for the testing right at that time uh we were struggling for the the frequent object failures course because of the requirement changes and whatn not.
Aishwarya
I think that was one area where we started looking out that what can help us in making our objects smarter enough to identify the changes itself and autoheal.
Aishwarya
So that's where I think we started to look more and more about how we can do the auto healing of the objects.
Aishwarya
That was the first moment where it was the introduction for me uh AIdriven uh testing can be brought into this area and and then that's what where I started looking out more and more uh conferences where people were talking about how AI can be brought into testing right the second portion was um we were working for few of the heavy insurance projects where uh the the testing data has to be available and because of the compliance and and various issues.
Aishwarya
I think the the real getting the real testing data was a big deal.
Aishwarya
So we started to think over that how we can generate the synthetic data which is close to the real production environments but but you know would also compare and uh include the outliers and that's where I think one of the conferences Jason Jason was talking about that how AI can help in generating the synthetic data that's where the second part of the the you know uh test generation data uh entire journey toward towards looking at uh making that as a offering into the solutioning um was one of my objectives and and and really helped us in um you know looking at the outliers of the testings which we were normally not able to test it in during the integrations of of the software.
Avinash
Great, great to know that uh Asha George over to you.
George
I mean you have been a test automation expert and we have discussed so many problems which yes test automation so when did the AI bug I think I think it was maybe it was during one of the conversations that I was having with the customer back in 2017 uh this topic came up and this came as a problem statement from the customer they had around 25,000 tests that they were executing on a website which was actually used by global users and he was primarily looking out for a solution that can actually tell his development team or his testing team what their developers have changed in the code and then prescribe the test that they should be running.
George
So the customer did not want to run a lot of tests.
George
Uh they already 5,000 tests and they were looking at running a subset of those tests based on the impact analysis and they were asking me whether there's a solution that I could actually recommend maybe an AI based solution either custom or maybe a commercial solution but at that point of time I did not have a solution in place but I went back to the same customer in 2019 and we could actually implement this particular solution for them.
George
They were able to cut down their test significantly using predictive analytics and a bit of AI.
George
So that showed that that the the the field of AI actually got matured I would say in the last couple of years where way back 2016 17 it was mostly I would say a lab topic or research topic now I think I'm seeing organization doing it at scale that is that has been my experience ain great to know journey from each of you and I think that will inspire our audience as well because anytime any point of time they can pick up and start if they have not done already uh like we have seen the journey of most of the panelists here.
Avinash
So moving on to the next uh kind of question that I have and again a kind of 10,000 view of where do we stand in terms of uh AI testing trends.
Avinash
Um so again each of you if you can just summarize what are the top testing trends from the AI perspective that you see in today's time.
Avinash
Uh we are sitting in 2023.
Avinash
Uh so what are the trends that we see right now and maybe over a period of next two years uh that will give us a sense of uh uh direction to the audience that we have and they can also go and look at the report that we have uh generated because that gives all the summary uh that we'll be talking about here but let's hear it from uh experts.
Avinash
So again maybe we can start with Kevin and Kevin if you can share your view of where do we stand in terms of the trends uh that would be a good starting point.
Kevin
So I think uh you already mentioned that they should go check out the report.
Kevin
So uh go check out the report to to get a good summary and and with that in mind I'm just going to to dive into the idea that we're we're still at the infancy of of where we're going.
Kevin
So these are these are trends and and George, Ash, um, and Jonas have already mentioned that there we saw like object recognition as as something or or, uh, data generation and yet these are just little tiny pieces of the puzzle that are being experimented with or or maybe even in production in in various places.
Kevin
But we're still we we haven't even begun to scratch the surface on AI actually doing the testing for us and that's where it gets exciting.
Kevin
So yeah, the AI can help us um right now.
Kevin
It's really exciting.
Kevin
Um I know that this is going to come up.
Kevin
So you we'll just talk about it now.
Kevin
ChatGpt can already do some really fun stuff for us.
Kevin
And when you look at the fact that it can help you generate test cases, that's a wonderful start to a trend where we're going to partner with AI to help us do our testing.
Kevin
And I think that to me is the trend is not AI taking over, but AI partnered software testing, AI partnered software development.
Kevin
And so that trend to me is really exciting where we start to work with the machines to get our jobs done better, more efficiently and and we'll see where that takes us.
Avinash
Absolutely.
Avinash
I think it will be reassuring for the audience that it's AI partnered and AI is not taking over the overall testing and we also feel the same.
Avinash
It's AI augmented rather than uh AI taking over.
Avinash
So it's a good reassurance for everyone.
Avinash
Um again coming back to Jonas um what do you see Jonas you you are part of a product uh uh which is based out of AI itself.
Avinash
So what are the trends that you see?
Jonas
Uh yeah there are actually multiple I think so I think object detection is not a solved problem right now for uh testing.
Jonas
So I think there's still a lot of doing there and I think this will still continue because it allows like platform independence and like all a lot of different kind of benefits but uh I agree with um Kevin here that JBT and assisted like generation of test cases a is a move forward.
Jonas
I would like to think it one step further.
Jonas
I think if you combine both approaches uh you can actually fine-tune models on text to automation.
Jonas
And what I mean by that is that you actually just input natural language automation like for example login with those credentials and they will be executed on the screen by that and we're actually looking into something like that right now.
Jonas
I think we are not there yet.
Jonas
There needs to be like some way of like editing what is the outcome of this.
Jonas
But I think that's where we're moving that you basically describe in natural language what you want to do and then actually get your actionable result on that.
Jonas
And um yeah I think large language models are the first step.
Jonas
Another step would be understanding of user interfaces from a visual perspective and then a clever way to combining it.
Avinash
Absolutely.
Avinash
And I think this is the right time because with open API exposing some of their natural models in a such a simple way I think the AI is available for all of us to use uh in a very democratic way which was not possible few years back when we started.
Avinash
So good time to start looking at those u uh models and start implementing for for our audience.
Avinash
Um Ashara back to you again.
Avinash
Um what are the trends that you see?
Avinash
I mean you work with large enterprises solving some big digital transformation challenges.
Avinash
So what are the trends that you see?
Aishwarya
So uh I think in addition to what what Kevin and and just added I I see more and more companies looking forward for increasing the speed and the accuracy of the of the enterprise transformations as well as in this digital journey when they are moving towards cloud migration or modernization it becomes very important where you have a heavy business impact.
Aishwarya
Uh AI models will definitely help in looking at the similar kind of issues happening in that entire industry trends and helping us generating those kind of test cases which ideally we should be testing in that business flow.
Aishwarya
So I so definitely one step above the componenturics test design techniques and uh if we are able to train our models enough on the various issues which have already been occurred in similar kind of modernization and softwares and learn from it.
Aishwarya
I think the the uh optimized amount of test generation will help in increasing the speed and accuracy of the testing.
Aishwarya
The second portion is that the I'm seeing a trend of new roles which are converging and transforming the tester's job of blending in the AI uh models with the testing knowledge and and I think in the morning itself I was talk you know talking to somebody who who said that chat GPD and AI is very similar to your close best friends when you are in in desperate need of the advices.
Aishwarya
You get a multiple advices but the end of it you have to apply your mind and decide that what works for you.
Aishwarya
So Chad Gibb and other natural language models will definitely come to your rescue and help you in defining the guidelines of the testing and better insights of of the the trends and the the applications.
Aishwarya
But at the end of it, it's the testers uh you know experience and instincts which will help us in getting to the right solutions.
Aishwarya
And the and the third uh trend which which I would see it as the simulation of the complex testing scenarios during the testing integrations which today I think requires a heavy cost in simulation.
Aishwarya
I'm not saying that they're not testable, but it is a big question of how much effort you want to put in into testing, doing those simulations and testing it.
Aishwarya
That's where the AI trends are going to be and helping you to do those simulations beforehand.
Avinash
Great.
Avinash
We we delve deeper into some of the use cases a but great insights there.
George
Uh George uh for you what do you see from your customer track and so I think uh test automation is an area where I've seen the most number of use cases getting implemented uh it includes all the everything around designing test it can be maybe generating intelligent unit tests from code or it can be even generating tests from API definitions or maybe even generating test from maybe a user story or anything of that sort the test design is definitely I would say an area where a lot of were interested.
George
I could actually see the maturity growing in this space.
George
As I mentioned earlier, definitely using predictive analytics and other techniques to select the right amount of test or the right test to test your application based on changes.
George
The test execution is also I would say the second area within test automation.
George
And I would say I think maybe the most implemented use case would be around the maintenance part where you actually where you basically see a lot of tools which actually has the capabilities to do self-healing.
George
When it comes to script maintenance.
George
So these are the three major use cases that I've seen around test automation.
George
I would say healing is the most common commonly used one out of these three areas.
Avinash
Yeah.
Avinash
Yeah.
Avinash
Healing is one of the area I am very interested in and we have been working and we have just launched a solution around that.
Avinash
So very excited about that area at a personal level.
Avinash
So uh I think we have uh covered a 10,000 view of uh 10,000 ft overview of where the the trends are.
Avinash
Uh so we'll go slightly deeper into specific use cases that uh you have used personally and seen because one uh question that I always get get is can you give me some example uh where you have used and got benefits from the audience.
Avinash
So I think this is the right time where we can uh talk about some of the specific use cases that we have used uh across customers or within the products.
Avinash
Uh so maybe again starting with Kevin.
Avinash
Um so Kevin in the report you mentioned that uh AI will impact every aspect of uh SDLC and including testing.
Avinash
So if you can just uh elaborate on some of the specific use cases that you have uh seen getting implemented uh getting some real benefit uh within the customer situations u that would really give some good insights to the audience here.
Kevin
I I expect the rest of the panel to cover some really cool stuff and so I'm gonna go super basic.
Kevin
All right.
Kevin
Um we're gonna we're gonna go back to just super simple stuff.
Kevin
Um, one of the things that, uh, there's lots of code out there.
Kevin
It's super easy to do is sentiment analysis.
Kevin
Um, so here's an area where it can get really complex or you can just keep it really simple.
Kevin
And I I joined a project team, uh, a quick story.
Kevin
I joined a project team where the product had a lot of negativity around it.
Kevin
And my only goal on that team was to to turn the the tide and make make it a positive experience.
Kevin
How would I know if it was a good experience or a bad experience?
Kevin
And what I did is I just grabbed all the feedback and I just ran quick sentiment analysis.
Kevin
You don't even have to know how AI works, by the way.
Kevin
There's uh there's easy code out there that you can just throw your data at it and that model will return if it's positive or negative.
Kevin
And I did that and it was all negative.
Kevin
It the product was not good.
Kevin
Um but over time I was able to watch that trend change.
Kevin
And that's an example where everybody on this um this webinar today, this uh that's listening today, you can go out and do that.
Kevin
So there's an easy way for you to go immediately apply artificial intelligence to your we'll call it testing.
Kevin
I know that technically that wasn't the testing part, it was the results.
Kevin
Um, but you can apply that today.
Kevin
And as soon as you do that, you're going to say, "Wait, I could use this on my test cases because then I can determine, do I have negative test cases, right?
Kevin
Am I testing the the the bad results or the negative scenarios or am I not?" And immediately you've applied artificial intelligence to your test cases as well.
Kevin
So those are very simple ways to get introduced into artificial intelligence today.
Avinash
Oh, amazing example, Kevin.
Avinash
Uh, I think something which is really relevant and and we see a lot of our customers coming back and asking for sentiment analysis.
Avinash
Uh, can you give us a solution?
Avinash
So, I think it's a very very important and uh effective use cases that anyone can uh pick it up.
Avinash
So, thanks for sharing.
Avinash
Um, uh, the next uh, Jonas to you again.
Avinash
You're building a product which is which is using core AI as as one of the component.
Avinash
Um so can you just give us a little more detail about the use case that you're trying to solve and how it benefits a customer.
Jonas
Um um so for us business value was always like the main focus and we just used or had AI as one way in solving it or leveraging like AI to to drive business value.
Jonas
Um and the main area is basically um we've seen a lot of customers trying to automate across different kind of systems.
Jonas
Windows automation it was super painful for a lot of people to get selectors then it's super painful for uh people to somehow get automation ids with the uh development team and it's all around this like hard to access elements especially in a in a native environment so very regulated native environments desktop automation Mac OS automation Android automation and even orchestrating all these together so we had people end to end automation what do we build for example a two-factor au authentication real end to end.
Jonas
So I log in in a web application, then I have my mobile phone with authentication app, scrape the data there and paste it into like the web application again on a different device.
Jonas
Um and basically we solved this problem by being by building an automation endshine uh which can run operating systems instead of applications and then you just run into the problem that you don't know how to access the elements and that's where the AI part come into place.
Jonas
So um and like having selectors wasn't really an option because we're driving operating systems.
Jonas
So we said okay one way to do it is use object detection just take screenshots of the application have a model trained on UI elements and use that for selecting the elements and then we could build a system which can end to end automate and orchestrate through different kind of devices native applications and we wouldn't bother about what text stack the actual application is built in and that's what we are building.
Avinash
Oh awesome.
Avinash
So is it is it uh only applicable for the operating system level transformations or can it be applied to any other applications as well like any any mobile?
Jonas
Yeah, it can be applied basically to any application running on top of that operating system.
Jonas
So we are always like driving the operating system and if you have a mobile application running there you can access that mobile application or that desktop application.
Jonas
So that's the thing.
Avinash
Yeah.
Avinash
Yeah.
Avinash
A big area I mean the we all have seen in test automation how the object identification fails and that's one of the major area why the automation projects are not that successful so if if that's uh that's an area where we are using AI to solve it's a great use case and I think a lot of customers will benefit from it uh coming back to you Aishwarya I think you have mentioned some very interesting use cases in the report especially around large complex insurance uh claim applications uh using test uh combin combinatorial uh design uh to optimize the testing right and optimizing testing I think again has been a problem from quite some time back and a lot of methods have been used I think with AI we will we are seeing a stage where I think we'll find some good solutions so can you just explain some the use cases that you tried for optimizing the the test cases and also um solving some of the test design issues that you mentioned the report.
Aishwarya
Sure.
Aishwarya
So um this all started with you know where we were working for one of the largest US firms for claims processing and the data was huge and we had to build in a uh application where we were seeing frequent failures in production.
Aishwarya
Now because uh of the fact that we started with the objective of uh you know what are the areas in the claims processing which we would like to test.
Aishwarya
We looked at the the normal industry flow of the claims processes and we we gathered some of the the data which were generally available not just only for that particular firm but across the industry so that we are able to benchmark the data first.
Aishwarya
We cleansed it m you know did all of that formatting and other things and after that we put into the machine learning machine learning algorithms to train themselves on the uh claims processing industry uh workflows so that it is able to identify the patterns and the trends in the claims process.
Aishwarya
Now what it helped it actually helped that helped us in identify and pinpointing the major issue areas where most of the insurance softwares fail and that output was taken and feeded into the combinatorial test design technique which actually is a stat statistical organized method with which runs in the algorithms and identify the best optimize uh key features and test cases to be tested so that it is covering across your functionality and if there are any point of variations in your software that it will ensure that it it is tested with that minimal number of test cases.
Aishwarya
Now this not only helped us in arriving at the optimized number of test cases.
Aishwarya
When I say optimized, it's actually kept in mind that not only the positive test cases but the negative test scenarios as well as the outliers are being being combined into the test generation um cases and ultimately the machine algorithms which we trained were also used in generating the synthetic data because it was already trained on the amount of claims processes which were available.
Aishwarya
That helped us in building test generation as well as the test data with almost 98% accuracy and believe me um we we were able to release that uh you know uh software within time with zero defects.
Aishwarya
So that was a quite an achievement but I would say that there was an ample amount of uh testing knowledge blended with the data science uh learnings went in to do that.
Aishwarya
So I just want to give one caution there that when you're planning for such kind of large scale testings uh it is very important to blend in both the expertise and not just co only data scientists or only testers.
Avinash
Oh, I think great example of vertical domain specific AI implementations and we all know AI has been used from quite some time around classifications, right?
Avinash
Classification is one uh use case where AI works really well and I think be bug classification or test classification uh I think most of the audience can start thinking about where they can use within their uh organizations.
Avinash
Uh George coming back to you uh you you have been uh talking about using AI in automation.
George
So uh can you give us some specific use case that you have implemented and kind of benefits with with an example maybe maybe I think we'll talk about again the selfhealing uh which is a common problem or the common problem that you see with organizations who have achieved high amount of automation coverage is the amount of effort that you actually put in in order to maintain your scripts.
George
So you may be running maybe thousands of automation scripts during a regression cycle and you may actually have maybe 10 or 15% of them failing not because of a product defect because of the issue with the script.
George
Then typically what happens is you basically try to actually look into the issue why it has failed and if you're working with maybe an open source kind of a solution you may not get a lot of time to actually fix those 15% of scripts that have failed and you end up doing manual testing for those scripts which you are already automated.
George
So end of the day you're actually you have invested in automation but you're spending a lot of time in actually doing manual testing now because your scripts were not able to do their job because of the issues and scripts fail fails due to multiple reasons.
George
So it can be basically because your object properties have been modified by your developer or the developer decided to move a control maybe from the left side of the screen to the right right hand side or sometimes it can be a synchronization issue where your applications too fast or too slow or you may actually get some kind of what do you say popups or exceptions.
George
So it can be due to multiple reasons.
George
So if you if you're able to implement a self-healing capability in into your application or to your what do you call to your automation tool or framework you can actually get rid of a lot of these maintenance effort that may be required to be done by an automation engineer in a manual mode.
George
You can basically automate the script maintenance by actually developing an algorithm or or a piece of code that can actually first look at why a script has failed then look at the the ways of solving it depending on the type of issue that you have.
George
For example, if you're having a object property change, the algorithm should be able to go ahead and fix the issue or use a different technique to recognize the object or if it's basically an issue with uh maybe what do you call synchronization, you should be able to execute the scripts with maybe a different what do you call time settings or configuration setting.
George
So these are types of issues that I've seen and I mean I I would I wouldn't say it's like 100% accurate but I've seen accuracy levels up to like 90%.
George
That is considerable in the sense like if you're able to solve 90% of your script errors without having a human getting involved I think it it makes a lot of what do you call it gives us a lot of benefit so I would say right uh it it may not be easy to implement it I've I've done this once when we when we try to implement this as a part of product almost took I think a five member team to at least four to five months to get to a decent stage it's not easy but it requires a lot of improvisation but it's definitely doable and it actually saves a lot of your effort and I think that's one of the important use case or the what do you call the best use cases that I've seen around test automation aash great um so I can already see lot of questions coming in um and I think I would like to open that session to uh the audience but before that I think there are a lot of uh people who are uh kind of thinking to start uh implementing AI within their organizations questions.
Avinash
Um so a last concluding remark from the panel here before we open up for the question is if anyone wants to start implementing AI u from the very beginning what advice would you give how uh how should they start thinking about it?
Avinash
Um and there are a lot of questions already around it.
Avinash
How do we start?
Avinash
So probably this question will will answer uh a lot of their doubts.
Avinash
So maybe we can go with Kevin first and then we can go to the other panelists.
Avinash
So Kevin, what advice would you give anyone who is looking to start from the beginning?
Kevin
I would suggest that there are two different ways to look at AI.
Kevin
There are those who are going to use it and there are those who are going to build it.
Kevin
So do you want to see what's happening and how you can use an AI tool?
Kevin
Then uh go try out chatgpt or go do a quick um use something like Apple tools and just try it out to see what it does.
Kevin
But those who are actually curious on how to build AI um the the things that have been suggested today in the panel are things that you're going to have to write some code.
Kevin
You're going to have to understand how it works.
Kevin
And if you want to do that, then understand that it's some work.
Kevin
You're gonna have to learn how it works.
Kevin
And that means taking a few courses, reading a few books, um experimenting, try it out.
Kevin
Um don't be afraid.
Kevin
Uh play around with AI.
Kevin
It's not going to take over your machine.
Kevin
It's not going to ruin the world.
Kevin
Um it's it's just some simple models.
Kevin
You're going to train it with some data.
Kevin
You're going to see what it does.
Kevin
And and solve really really small problems.
Kevin
Um don't start with um how does AI take over testing?
Kevin
That's not a good first problem to solve.
Kevin
Uh start with something simple like hey um you know uh Jonas has mentioned object recognition.
Kevin
Can you get a classifier to work?
Kevin
Just just try to get one classifier to work and see how that helps maybe solve one button in you know um object recognition in your automation and implement that and just just be happy with that.
Kevin
Um, so one option, go use some AI like chatgpt.
Kevin
Option two, know that it's going to take some work and uh start to dig in and learn how the code works.
Avinash
Absolutely.
Avinash
A very nice way to put it, Kevin.
Avinash
U understanding versus building.
Avinash
I think that differentiation is very important for anyone to understand.
Avinash
Um Jonas from your side.
Avinash
What what suggestion do you have for anyone who is looking to start the AI journey there?
Jonas
So um I would like to propose uh don't think about the AI part always think in business value.
Jonas
Uh I actually think the best AI tool out there is a tool where the user doesn't know that it uses AI um but just like generates value for the user.
Jonas
So um if you are think always like okay so I have this problem in my organization don't just implement an AI tool because you think okay AI is like a hype I need to implement it just think okay I have this problem classical approaches cannot solve this problem and then there might be an NI approach which can solve this and if that's the case don't try to reinvent the wheel uh just use look what's out there other existing frameworks because reinventing the wheel in AI is very hard I can promise you But there will be a lot of developer resourcing going into that until you figure out how you actually develop models and stuff like that.
Jonas
Um and then see if that solves your problem.
Jonas
So I would like to propose just think like a user and then go from there.
Jonas
I always remember the quote from Steve Jobs, think of the customer first and thinks of think of the tech later.
Jonas
So I think that's the message which is very clear that don't just use AI for the sake of it.
Jonas
Identify the problem and then see what what are the ways to solve it and AI definitely is is one way which we can uh kind of solve the problem much faster.
Avinash
Great.
Avinash
Uh Jonas Aishwarya to you again what advice do you have for anyone who is looking to start their journey in AI?
Aishwarya
So I think I I already covered a little bit of it in terms of the the skills and other things but um I I'll because the simplest part of it Kevin has already covered.
Aishwarya
I'll go looking at the enterprises who are trying to adopt the AIdriven testing first and foremost.
Aishwarya
I think it's very important to align to the AI strategy of the organization.
Aishwarya
Look at how your organization ha has got a vision to move towards the AI driven.
Aishwarya
Align to that part.
Aishwarya
Second is invest in um skills understanding part of it because your domain knowledge your testing knowledge and understanding of the AI um and specifically building in the responsible AI is very important while you're going to generate and drive the testing scenarios uh utilizing AI it's very important to understand that you are not getting into any kind of a compliance issues or or other things so take care of that look at what are the different use cases which which you would like to uh pick and do the pilots around those uh testing scenarios because see it it's it's not a bulk implementation right it you have to understand that not everything will apply to AIdriven testing so it's important to look at carefully choose the use cases which you would like to go for driving the the testing via AI like the object identification because it's almost the effort versus the the kind of uh return you're going to get out of this uh implementation, right?
Aishwarya
And and last but not the least, continuously uh monitor and validate your implementations for a scalable aspect of it.
Aishwarya
And uh once you have seen the pilots getting successful there are ample amount of innovative areas where you can go and scale it up.
Aishwarya
So I I will leave to that point.
Avinash
Yeah good good suggestions Asha for enterprises.
Avinash
Uh George from your side what suggestions do you have?
George
Yeah.
George
Okay.
George
So I think the suggestions are more applicable around the fact that you're trying to implement a solution on your own.
George
For example, instead of buying a instead of implementing a commercial solution with AI capabilities, if you decide to build it inhouse, there are a couple of things that you need to keep in mind.
George
One is setting the right expectations with stakeholders.
George
Things are not going to happen uh when you actually switch on the application.
George
It will not start testing or not solve your problem immediately.
George
There is a time period that is required to implement and as well as to get to an accuracy level.
George
But very often what I've seen is like the stakeholders lose their patience maybe after two or three weeks.
George
They would definitely expect the the results to come in.
George
That may not be the case.
George
If you're trying to solve a problem, it can be as simple as maybe a problem that you're trying to solve maybe for defect classification for example.
George
It may take maybe like weeks together to get to a good state and you may you may need to educate your stakeholders around the challenges and the timeline that it might it might take to get a decent accuracy level.
George
The second one is around data.
George
For example, the same example around defect what do you call classification.
George
If you don't have enough defect data, your solutions team or your team that is developing the solution will not be able to design or train a model or even test it if there's no data given to them.
George
So you should you need to make sure that if you are actually able to give large volume of data as well as the good quality data your solutions will have better accuracy level.
George
So dealing with the right amount of data with the right quality and quantity is very important if you want to get to your accuracy level.
George
And as I was telling earlier you need to definitely improvise over a period of time.
George
So you should be open for continuously improving your solution.
George
You should actually look at taking feedback from your actual users of these applications or tools that you've developed using AI and they should be continuously improving.
George
So you can actually get to maybe like 90% of accuracy for self feeling but 90% may not come during the first build or the first version of product that you're releasing.
George
It may come maybe after the fifth or sixth but you should be actually receptive to feedback.
George
At the same time you should actually be ready to implement those changes inside your code so that you can actually improve and actually a journey that may take a couple of weeks or months.
George
So these are the three things that I would actually mention around the things that you should be actually taking care when you're actually implementing anything around AI.
Avinash
Absolutely.
Avinash
Absolutely.
Avinash
Um I think great insights from all of you so much that I uh picked up from today's panel and I'm sure audience u has got those great insights as well and I can see lot of questions pouring in.
Avinash
Uh I will try to cover as much as I can because we have a time limit as well.
Avinash
Uh but if you're not able to answer, we will try to send you the answers offline as well.
Avinash
Um so I will probably start with the first question from uh Ab and I think that's directed to Ashwaryia.
Avinash
What industry standard AI based tools do you have now for synthetic test data generation?
Avinash
So do you use any industry standard tool or uh you have in-house ashwara what suggestion do you have?
Aishwarya
Yeah.
Aishwarya
So uh we had a uh proprietary test data generators which we have built in backed by the IBM research labs earlier because I was I was giving the examples and I was closely working with the IBM research labs.
Aishwarya
Um but at the same time I think we were looking at some of the uh the test data generators automatically which are available with their clients right because every client has their preferences.
Aishwarya
So um we we were looking at some of the uh the APIs which can be plugged in directly to the the synthetic data generators and can be utilized but but for our own testing purpose it was the the homegrown uh generator tool.
Avinash
Okay, great.
Avinash
Um, Kevin, maybe you can take this from Somir Rajan.
Avinash
It's uh it's about sentiment analysis.
Avinash
It says that I believe predictive analysis modeling with AI and ML can be applied on customer sentiment analysis and then the data can be utilized to generate test data and test cases for various testing of uh multiple CRM applications for better success and marketing automation product building.
Avinash
Do you have any thoughts on this in real world?
Kevin
Uh George, did do you want to take that or do you want me to take that?
George
Maybe I I can take it up.
George
Maybe that's one of the areas where I was doing some research.
George
So customer sentiment analysis is actually a very good area where you should be able to use generate insights on many things.
George
One is how is your customer using your product and then you can actually decide upon what you should be testing in your subsequent builds or in your subsequent releases and also understand the the feel of the customer in terms of what kind how they are feeling in terms of positive or negative sentiment and also look at what type of issues are actually surfacing in the sentiments or in the the the feedback that your customer has actually given.
George
For example, the customer might have actually provided feedback in a social media forum or inside an app store which can be actually further what you call analyzed and then you can actually get to a state and understand like whether the customers are actually facing functional issues, performance related issues or maybe user experience related issues when he or she is actually using uh your product.
George
So there are a lot of sentiment analysis techniques as well as classification techniques that you can actually apply in order to get what you call insights on how your customers are using their product and how we can further improve the product based on the feedback that the customer is actually giving.
George
But the the sad part is rarely forget about AI rarely the testers spend time in looking at what the customers are talking about your product.
George
Once the build is pushed into production, your job stops there.
George
But that's not the fact.
George
You should be constantly looking at these feedbacks and then looking at ways to improve your product.
George
And customer sentiment is a very good example and I think it can be one of the easiest use cases to solve because there are a lot of algorithms out there that actually has this capability.
Avinash
Awesome.
Avinash
Kevin, you want to add something uh on this question?
Kevin
Yeah, I mean we can we can just take what George just described and then you can bring that back into um what was suggested was um you know now you can create test cases off of that, right?
Kevin
And and again that's just the natural progression as you start down this path.
Kevin
Start simple.
Kevin
Uh that's what I keep mentioning is please start really really simple but then you go if I did that I could do this other thing.
Kevin
If I did that I could do this other thing.
Aishwarya
So if you can the hardest part that we've all described is getting the data um if if you can go out and get the data from your customers meaning you know the feedback if you can get the the data on what they're how they're using your product then the models are there that's actually ironically the easier part is running a model on the data if you can get the data and then creating test cases from that data is actually not that hard but you have to have the data to then generate the test cases and one very interesting point um Kevin what you talked about is we actually try to implement the the sentiment analytics on our defects data because you know what the kind of frustration the the people put in while raising the defects I think you get a lot of insights from that data as well.
Kevin
Yeah.
Kevin
Yeah.
Kevin
Yeah.
Kevin
Yeah.
Avinash
Uh two two similar questions from Nares and Kunal around API uh test automation.
Avinash
So can AI be used uh within API test automation?
Avinash
Uh any anyone has any suggestion?
Avinash
Anyone of you can take it up.
Kevin
I'll leave that to the forum.
Aishwarya
Sure.
Aishwarya
Uh so I think uh there were bit of chunks where we were looking into the API uh APIs to be tested and um there are techniques based on which you can actually generate the test cases from the YAML files directly believe me and we can we can actually read through the YAML files uh the models or or the the crawlers which can go through the YAML files pick up the keywords uh key in the the data sheets to the YAML files and generate the test cases automatically, the test models automatically from the YL files.
Aishwarya
That kind of API automation can be a perfect use case for the AI testing.
Avinash
Great.
Avinash
Great.
Avinash
Uh, another question from Prabhakar.
Avinash
Uh what AI tools or AI based testing tools uh are improving the improving the testing uh better in the area of test test data test execution and test maintenance.
Avinash
Uh are there current any tools currently available in the market or is there any guidance or the checklist which audience can follow?
Avinash
Um George you want to take it up?
George
Yeah, there there are multiple tools and the tools will actually vary depending on what do you say uh the use case.
George
For example, if you're actually looking at maybe something like providing faster feedback to developers, something like a certifier bot like the autonomous bot to provide feedback on mobile apps would actually work.
George
Or if you're actually dealing with something around predictive analytics where you want to decide upon what kind of tests you need to execute based on the changes that your development team has done or based on a previous run you can go with launchuple or a product like seize or if you want to look at c what do you call selfhealing there are multiple tools you can use opkey or functionize now toka selfhealing so there are multiple tools but there's no universal tool or one tool that lackly solve all your problems these are tools that are specifically designed to address a specific problem so you may not be able to solve more than one problem using each of these tools.
Kevin
I I'd say that goes back to what Jonas said earlier as well, and that is what is the problem that you're trying to solve.
Kevin
Um, if you're just asking any of us on the panel right now, what tool should I use?
Kevin
We don't know.
Kevin
Um, we'd love to promote all of our own tools, but the reality is you need to tell us what the problem is, and then we can help you solve the problem.
Kevin
And and and Jonas pointed out, it may not be AI.
Kevin
As much as we all love AI and we want AI to solve all these problems, it may not be the right solution.
Kevin
You know, use the simplest solution.
Kevin
That's the that's the best option.
Avinash
So, I think yeah, lot of questions uh gets answered from what you said Kevin.
Avinash
Uh don't look at the tool in the beginning.
Avinash
Think of the problem statement and then figure out how AI can solve tools will follow.
Avinash
Um so, Kiran had similar kind of question.
Avinash
I hope that gets answered.
Avinash
Um uh then there is a questions around business goals from IAS.
Avinash
So what you as a team decide to take help of AI to achieve a business goal.
Avinash
Um so what steps did you take if you have an existing framework or uh what were some of the pitfalls where AI did not help and you have to come back to the traditional approach.
Avinash
Very interesting question.
Avinash
I think lot of uh times we have seen we start implementing AI and it fails and we come back to the traditional approach.
Avinash
So any examples or any interesting anecdotes there?
Jonas
I mean I might have one.
Jonas
So um I was talking to one customer and he was trying to use like object detection for uh his like test automation and finding objects in OCR for like locating text and stuff like that.
Jonas
It turned out it was completely the wrong approach because he was testing calculators.
Jonas
So um you have to think AI is still a statistical method and what he was trying to do is like actually validate that 2 plus two two is four like in a calculator.
Jonas
Uh but OCR is not 100% correct.
Jonas
So if you need like 100% accuracy on like specific stuffs AI might not be the tool to go with because he got like maybe sometimes it wasn't 2567 sometimes it should detect it as 2568 or something.
Jonas
Um and that's you have to be aware of that it will not be 100% accurate and if you have test cases which need to be 100% accurate um AI might fail.
Avinash
Yeah.
Avinash
Yeah.
Avinash
Okay.
Avinash
Yes.
Aishwarya
So yeah, just to add to to what said uh see there are scenarios and and I was talking about when you are planning to implement AI look at what are the best use cases repetitively man high manual effort which requires AI help should be automated not everything is suitable for uh the AI applications right uh again it's effort versus you know what what you're trying to do classic example what what just talked about and I think the the contrary part of it if you look at the visual testing nowadays I think visual testing has been one of the areas where you have to do the pixel by pixel comparisons changes and other things I think manually doing it is a today's task and that's where AI comes to the rescue where you can easily apply the computer vision technology to do that visual testing comparisons that's a classic example of where you should use AI and where where you should not be using it.
Avinash
Yeah.
Avinash
Yeah.
Avinash
Absolutely.
Avinash
Um so I think in the interest of time we'll take one last question.
Avinash
Uh there are so many uh uh as I promised we'll try to send you the answers uh individually or we'll publish all the answers.
Avinash
Uh so one last questions did you see any use cases for improving mobile application performance or end user experience using AI?
Avinash
If so can you please explain on how do we design and implement the same?
Avinash
Uh interesting one.
Avinash
We deal with mobile application all the time and this is a very relevant question.
Avinash
So anyone wants to add uh or answer this?
George
I think you'll be the right person.
George
Abin can talk to answer that.
Avinash
Yeah.
Avinash
I mean uh so uh as well as just mentioned visual AI and I think visual AI works perfectly not perfectly in the sense visual AI is one of the uh areas that can be used to improve the customer uh experience uh in mobile application cross device compatibility testing is one big area which kind of destroys the user experience in so many applications that we have seen.
Avinash
So visual AI is one way to do that.
Avinash
Self-healing because in mobile application uh you rely a lot on automation uh and again during cross compatibility testing your object locators change your uh scripts fail.
Avinash
So self-healing can be uh a good area to implement in the mobile application uh testing space.
Avinash
So these two we have seen effectively working.
Avinash
Uh we have launched our solutions uh within these two areas.
Avinash
Um so I would suggest go ahead and check it out.
Avinash
Uh so I think with that u I would really thank all of you for this engaging discussion.
Avinash
Um I really enjoyed it and I'm sure looking at the response and the kind of questions uh audience uh have already uh kind of given the verdict.
Avinash
U thank you so much.
Avinash
Thanks for your time.
Avinash
Um lovely having you here.
Avinash
Uh we hope to connect back again and discuss more on this.
Avinash
There is so much to discuss and and take it from here.
Avinash
Um have a uh have a good day, good night depending on where you are.
Panelists
Thank you so much.
Panelists
Thank you all.
Panelists
Thank you.
Panelists
Bye-bye.
Panelists
Good night.
Panelists
Bye.
Panelists
Bye.
