Dinakar
Alrighty. Hello everyone and welcome to Pcloudy's webinar on a QE leaders playbook to building an agentic quality platform. My name is Dinakar and I'll be the host for today. And before we get started, here are a few housekeeping rules. You may have noticed that your line is currently muted. And if you have any questions or clarifications, you can use the Q&A option available on your screen. And we will answer your questions as part of the Q&A section at the end of the webinar. And also please do note that this webinar is being recorded and we'll send the link of the recording to watch again or share with your colleagues later. With that said, let me introduce the speakers for today. We have Avin Tari and Jessica Selvin. Avin is a renowned thought leader, a recognized keynote speaker and the co-founder of Pcloudy. He has 20 plus years of experience in product development and testing. He brings with him a passion for emerging technology and quick adoption, both of which have solidified his reputation as a leader in the app testing space and quite recently in the agentic testing space as well. We also have Jessica who is a solution consultant here at Pcloudy. She will be walking us through the agentic workflow demonstration towards the end of the presentation. So without further ado, let me hand it to our first speaker Avin Dwari. Over to you.
Avinash Tiwari
thanks DK. thanks for your introduction and good afternoon everyone or good evening good morning depending on where are you joining from. I'm really glad to have so many of you interested in this topic and u to be able to find time from your busy workplaces. if you don't know what Pcloudy is, we are one of the leading device cloud provider with inbuilt intelligence. And when I say inbuilt in intelligence, it's powered by the agent quality platform that we have built. And over last couple of years we have we have done extensive work in the agent space which sits on top of our very strong test infrastructure powered by our device cloud and browser cloud. And from that experience we have built this playbook which we are going to kind of talk about and also demonstrate in some way to give you a glimpse of what we are talking about. so I'm very excited about this topic and we would love to get your feedbacks your thought process. I'm sure many of you would have tried something maybe you would have built something better than what we have done. So we I would love to get your thought and experiences and learn from it.
Avinash Tiwari
So with that said I mean this is one of the most exciting times I feel. so I want to ask this question to all of you. do you think we are at one of the most transformative time in the history of technology and I mean to answer that maybe we can do a quick thought exercise. so maybe we can spend 10 seconds or 20 seconds thinking about all the AI announcements, breakthroughs or tool launches that you would have encountered in last maybe 90 days or 30 days whatever you can think of whatever comes to your mind very quickly. so just think about it and maybe note down in your mind keep it ready u and see what comes to your mind. So let's count down. Great. Hope you got some time to think about it. you can you can share in your share in the chat window what all you got. but when I was kind of thinking I mean I got almost so many notifications in my mind in terms of all the announcements and when you think about it sometimes you feel busy right sometimes you feel the speed of change is so difficult to handle can the can the world slow down a bit being a technology leader I'm saying that sometimes we all feel that emotion that things are moving too fast but that's the world of AI for you right things are moving really fast. In last month itself, we have had so many announcements from for example Opus 4.7 launch to so much talk about claude. My thoughts which no one knows what's in store with that model project glossing deepseek recently I mean I think couple of days back released the V4 opensource model with trillion dollar parameters. So things are changing and on top of it there are so many agentic frameworks new methodologies coming up almost every week or so. So, so what I wanted to highlight was and I think we all can agree without any contention that things are moving really fast but one of the announcement that caught everyone's attention and created such a wave was announcement of clawot and why this is particularly important is it kind of demonstrates the power of AI that we all are kind of experiencing as we progress in this technology shift. U if you have followed Y combinator last year they had released a thesis paper and then that's the thesis paper based on which they do any investment. So all the 10 thesis was around AI no guesses there but one of the thesis was one person 1 billion company right and it happened that point of time it was it we all thought like it's kind of pipe dream but it happened Clawot was one person company got acquired by open AI with 1 billion valuation and it's one of the most popular open source project in today's time with more than I mean million downloads and people folking it and using it in different ways and that brings me to the next point that why AI as a technology is so different right I mean we all have in our last I mean I have let's say experienced 20 years in the industry and we have seen so many technology shift from web to cloud to digital transformation and now to AI But why AI is different from some of the other technology shift that we have seen in past?
Avinash Tiwari
I mean and I would probably term it as this technology shift especially with AI is it's not a tool. It cannot be just termed as a technology. It's like a leverage equation. You can think it in terms of leverage equation. So in past we have seen a lot of traditional tools that gave us productivity boost mostly linear improvements. to work hard, get a new tool or add a new tool and get proportionally more output, right? But AI is different, right? The expon the leverage here is exponential like we saw in case of clawbot, right? One person with AI can achieve what probably could be achieved by entire teams, right? And what we are seeing is that it's impacting every industry, right? So we have seen such transformation in marketing content customer support the coding and in that context can the quality engineering remain untouch untouched. So the question is question isn't if quality engineering will be transformed by AI but what's the right way? I think like I said in the beginning itself, I'm sure you all have started implementing AI in some way or other. So we all have seen that there is lot of scope of implementing AI within the testing space itself or within the queueing space itself. But the question is what's the right way? I think that's the answer everyone is searching for and in today's talk that's the focus for me to give you some thought process based on what we have done building agenti systems but before that I mean from the QE perspective some data points lot of surveys shows that especially within the application development space testing is one of the key use cases where AI can be a big advantage right and can organizations can get lot of efficiency by bringing AI within the testing and QE space. So you can see in the left side chart application test automation generating test cases from business requirements are some of the top cases right apart from some of the other cases mentioned here on the right side also if you see generating automated functional tests and other kind of testing like accessibility API test coverage are some of the top use cases within AI which we need to which we can think about and think about and start using it within our ecosystem. So with that data point in mind, I think what I want to move to is understanding a little bit of what we are talking about in terms of agenti systems and some terminologies which are used very commonly within this ecosystem is something I wanted to quickly talk about. So I think we all have been familiar with these terms but these are used interchangeably and it's important to understand and understand these concepts in right context. So traditional AI I mean especially in the context of geni right I mean traditional AI can go back to this the era of ML and traditional AI space but in the concept of geni traditional AI has been the chat bots that we all have been familiar with. So you give a task or it asks for a question and it responds where for example in the case of testing it could be like generate test cases for my payment application for this scenario right so it can generate the test cases for you and it's very command oriented for isolated tasks and u with the knowledge that it has it can respond back and then you can respond back and forth right that's that's typically how we have been using most of the chat bots like chat GBT or claude etc.
Avinash Tiwari
However, most of these chatbots also have started kind of using agenting systems at the back. But when it started it was mostly the isolated command-based execution for the traditional chatbots and that's that we can call it traditional AI in the context of genai. But then we move to the era of agentic AI right where these are like autonomous systems which can I mean perceive decide and act based on certain goals and it can learn and evolve over a period of time. So for example I can say that I have made changes to my let's say a particular feature can you suggest the impacted areas and then suggest test cases for me. So, so now we are talking about a goal for the system and in that context first it can decide that okay I need to understand your feature details and then find out the impacted areas and then suggest test cases. So it can do multiple tasks together to achieve that goal. so that's agentic system but now we are also moving into the era of workflow. You can call it agentic workflow or AI workflows where one agent can give us let's say certain efficiency but if you combine that across a workflow create multiple agents which can work together the efficiency can be exponential right of course it has risks as well but more and more we are seeing the real benefit coming up when these workflows are executed and we are able to successully fully integrated within our enterprise systems. Right? Simple case, I mean extending the agenti description that I talked about. I can say that okay I have made the changes to this feature. Can you run the right test cases for me? Right? So it's not only going to now I mean find the impacted test cases and maybe generative test cases are not there. It can also decide which test cases to run. So then it has to invoke multiple agents. One for maybe impact analysis, one for generating the test case and then finally one for executing the test case. So that's the workflow we are talking about, right? It can be called orchestration, it can be called workflow. That's what I'm saying. A lot of people use these terms differently, but it's important to differentiate between these three terms. So world is moving more and more towards AI workflows which are definitely powered by the agentic AI systems. but that's the goal we are looking at in terms of building more and more AI workflows. Getting a little bit deeper into aenti architecture, we double click on what agent AI system is. so there are primarily four layers within the agent AI system. The top is the orchestration layer, the decision center. The center is the kind of heart of it where u any command that you send to the agent AI system is divided into three parts.
Avinash Tiwari
First it senses what is happening what's the perception what's the context then it thinks what I need to do based on the reasoning and planning and then finally it acts and it's not sequential I mean it can get into a loop for example from sense to think to act and then going back to sense or sense to think and going to sense again it fig until it figures out what's the right planning and then acts so it can decide the sequence between sense, think and act and finally when it executes you get the final goal executed. Now all this three stages of sense, think and act is powered by the LLM. Of course it helps the a term in sensing to thinking to act all the actions are powered by the LM interaction layer. And then finally the last layer is learning and memory because now more and more agentic systems require very large memories so that you can maintain the context and also that helps the agentic system learn over a period of time so that you can make better decisions. So that's in a nutshell what agentic architecture could be. of course it's defined you can go more glamor and more deeper into it but at a very high level that's how we can think about in terms of what agent architecture could be now that with that as a base u I want to move to the agentai system that I was talking about right and this is the way we developed our agentic system within pclouding and I mean just for my personal interest I've named it like a testing galaxy because I have my personal interest into space and u universe related u terms. So I've used those terms to describe this galaxy. So let's start with the bottom right. The bottom layer is where we exist right now. Right? So that's the foundation layer for us. the last what I call it your current planet where your current tools it could be your let's say your automation tools it could be device cloud it could be browser cloud it could be your execution grids it could be your test management Jira all the tools that we are using right now in enterprise exist in the current foundation layer right now the second part of this foundation is the data and the third part is your processes etc. So these three things are our existing foundation layer. Now over a period of time one of the thing that we have come across is the data part has to be a separate layer in itself. What we have started calling it data fabric. Right? So the foundation layer can be broken into two part. One is existing tool layer and the data fabric because more and more complex aenting systems require knowledge of your existing data. So it could be your domain knowledge. It could be your let's say test strategy documents. It could be your test plan documents, existing test scripts, your past logs, your past test results. All that has to be part of your data fabric. Could be a rag system, whatever. But there has to be a data fabric powering your existing system. So that the knowledge that you have built over a period of time is utilized. So though here it's mentioned in the foundation layer itself, but you can think of data fabric as a separate layer. all together.
Avinash Tiwari
Now once this foundation I mean I'm sure we all operate here but once this foundation layer is clear to you can think of the agents that you need to build right and agents are like constellations where you can you have different kind of agents for different tasks there are functional quality agents for example for your UI automation to your API testing to accessibility testing there could be different agent for different purposes Right? there could be agents for your performance. there could be agents for your monitoring non-functional requirements, security, etc. Right? So you can think of your I mean digital testing space and break it down into different agents and these agents could be either your inbuilt custom agents or you can think of some third party agents which you can integrate within your system. So for example, we have so many agents which can be integrated within any enterprise on top of your existing tools, right? And you'll see a glimpse of it when Jessica shows the demonstration. That's the second layer which we need to think about. And then these two layers a foundation layer and the agent constellation has an integration using something we call it integration mess. Right? So how do we integrate the agent with for example your existing tool called Jira right? So most of these tools have started exposing their MCPs. So that has made the integration between these two layers very simple. So you have to of either you build your own integration layer or use the existing third party MCP gateways or A2A gateways to create this integration mess because if you don't do that then you're not kind of getting the right outcomeing these two layers. So you have to think of what are the integration meth that I need to think about and how do we achieve that. Then comes the third layer what we call it the gravity field which kind of brings every everything together right we I talked about AI workflow in my previous slide. So think of this third layer as the AI workflow where agents are discovering each other. So this layer is basically your orchestration layer. So if you are creating a workflow where based on your PR commit the test has to be executed. It requires multiple agents to be to be combined together and that can only happen if you have a good orchestration layer built where the agents are able to discover each other and pass information and then call each other right and complete the workflow. So that's the third layer which is called here the gravity field. And then the next layer is the kind of layer which powers everything your star engine in my cont in my terminology where all your LLMs your frameworks or models reside right and here it one of the important thing that has started happening especially in the case of enterprise is differentiation between where can you use a public frontier model and where can use maybe open-source fine-tuned model for your own need and that's extremely important when you start thinking about agenti system because many enterprises have a lot of security concerns some of the data you're not allowed to send outside so how do you still achieve the AI agentic AI workflows considering that you're not allowed to use public models so there you can think of open source models so you can always build a multimodel approach where part of it can be handled by public model, part of it can be handled by custom models and in some cases custom models are more than sufficient. Right? we have been using custom model for our automation agent even for very complex applications. We've done the finetuning of that model and it works extremely well saves a lot of cost for us for our customers. So you can always think of maybe going only for the custom models but there are a lot of situations where I mean public models are much more efficient. So depending on the use case you can decide a multimodel approach frameworks for example agent there are many agentic frameworks like lang chain and there are third party ones like ai so you think you can also write your own custom. So again do when we think about the star engine which the fabric which powers everything you can think whether you want to build something in house or you want to build completely on your own sorry inhouse versus third part third party and the last one is what we call it orbiting interface. So all this agentic system how are your users going to access it right are these going to be through CLI are these going to be through some plugins are these going to be through some chatbots natural language commands or APIs so what is the access interface that you need to think about is the last or the first layer in this stack the top layer in this stack what we call it orbiting interface and also on top of it the monitoring of all the agents right So it's also important that when you have built these systems for example you have deployed your models what if the models are not working right something goes down how do you monitor these systems so that itself brings lot of challenges maybe your agent is running but your model is not responding there are there are more load I mean models are very u hardware intensive so what if there are multiple concurrent connections happening how much load can it can it handle so there are a lot of monitoring that needs to be done. So there has to be some real-time dashboards which can help you understand the status of these agentic systems and so that you what is going on and if there is something wrong you can quickly debug it. So that's a new dimension to this whole agentic AI system coming into play. So that's that's the approach that we have taken where we have divided our agenti systems into these five layers or in fact now six layers with data fabric as a separate layer your foundation to your agent constellation your orchestration your LLM fabric or all the tool fabric and then finally the interface to access all these agents.
Avinash Tiwari
So that forms the agentic systems for us and that's the testing galaxy I wanted to present. you can you can pick and choose what to start with and it's difficult to kind of build everything together. So you can think of starting in smaller pieces maybe build one agent integrate with a foundation layer tool build orchestrator connect to some LLM and then expand it. I mean that's the approach we also took in the beginning. Now that we have so many agents we can say that we have this stack but it was all very gradual for us. So with that stack in place what can it achieve right? I mean this is like now okay this is the stack in place what's the picture that we can paint and this is what we have been working on and to a large extent we have been able to achieve this and we are going to release this for our general customer as general availability as well. So it's a it's a agent collaboration workflow where once your code is pushed till your testing happens you can you can see that there are multi- aent workflow which is getting executed here starting from code analysis to your impact analysis test case generation environment setup then execution in parallel finally depending on the failures triaging u autofixing escalation and then overnight regression. So this is one of the flow which we have been kind of executing internally for our own development purpose using the agentic system that we have built and soon it's going to be available for our customers as well and you'll see a glimpse of it once Jessica demonstrate but this is not very far I would say of course at different layer we have also used human in the loop approach though in the diagram you see everything autonomous but there are human in the loop approach where you can take a decision whether to proceed to the next stage or you should you want to stop it or provide feedback to the to the next agent so that it can customize it.
Avinash Tiwari
So some of those human in the loop systems are in place because I personally feel that it the agentic systems can work best if we don't think of it fully autonomous. It has to be a combination of agents and human in the loop. so with that said with that said I think this gives probably a good insight into what kind of workflow we can think of in terms of agent systems and I feel within six to 6 months to one years all these will be very commonly used across different organizations. Now with that I think I will pass this on to Jessica to give a small demonstration of it. Jessica over to you.
Jessica Selvin
Sure. Thank you. Thanks a stop sharing. >> Yeah. I hope my screen is visible. >> yes Jessica. Okay.
Jessica Selvin
Yeah. So firstly before I get into the demo what this is about our PL's agentic AI suite. right and firstly starting off with our test case generation agent right so here you would be able to give different types of inputs for example your PR documents or if you have any screenshots of your application or your Figma URL or you just want to give your user story or a feature description you can just give or you can give multiple types of inputs and once you have fed all those inputs and you go ahead and give analyze you So for example here in this example you see a screenshot being uploaded. You can see the different scenarios the test scenarios being generated and you can pick and choose which ones are relevant to your u for your use case and you will be able to shortlist or you can further refine with AI. You can ask it to maybe focus on a specific scenario or you can ask it to add more edge cases or negative testing scenarios right and other validations and accordingly refine it and once that's done you will see that the test cases are generated with the preconditions test steps priorities and all of this can be edited as well. So and once you are satisfied with the test cases that are generated you can either export it to an Excel or a CSV or directly automate it using our test automation agent which is our Qpilot agent. So here you can see that you can and we support be it a mobile or an iOS or Android as well as a web browser right you can launch it on our real device platform and then you can just go ahead and give those plain English language descriptions which you took from our QGEN agent or if you have it ready in your manual test cases or your Excel you can directly input those right and go ahead and give plain English language descriptions that you see here right and once that's done each step would be picked up and you can see that there's a step execution that's happening so what happens here is the AI agent goes through the entire DOM structure of your application right sees which is the closest fit based on the description right so for example here we have said add to card of the first item so it's just picked up the first item right and it will give you the reasoning as to why it selected that particular object or element and it'll also give you the locator information right so all of this will be done for each and every step and there'll be a stepwise execution that happens to ensure that your final automated test case is 100% accurate and you can simply go ahead and execute it a number of times after that right so you can also edit it at a test case level right so you can edit your steps you can or you can open it again and edit it or edit at a stepwise level or edit the X paths and also give validation steps as well. Right? So and once your test case is generated you can see this is how it will look like. You can go ahead and seamlessly now run it and scale it as a collection as well. So this is our Qpilot collections where you can run it across platform so across web mobile and API test cases. So here you can see that you can insert multiple test cases across web right and mobile as well as API. You can also enable selfhealing in your collection and also give the devices and browsers that you want it to run on. You can specify the device configurations. So with PD we have our real device infrastructure. So you can directly run it on our platform. So with that it would become an end toend automation solution for you right and then coming then it is our test orchestration agent right so here you will be able to the orchestration agent would be able to prioritize the test cases and also distribute it intelligently across the cloud right so it will be able to prioritize and also optimize in terms of minimizing any network latencies right placing the test cases in proximity to the real devices such that there are minimum network hops and you'll be able to run your regression suites much faster and you'll be able to get your results more seamlessly and also more prioritized across the different devices in the cloud. So once you are once you have run your test cases we have our test reporting and in all our reports we have a AI failure analysis agent as well that will give you an AI analysis as to why a particular test case failed.
Jessica Selvin
So here is how the visualization would look like where you can see your test suite and how many test cases have passed how many have failed. All right and you can deep dive into any particular test case. So here we can see that we've picked out a few failed test cases and it will highlight what exactly was the issue causing the test case to fail right so here it was an element not found issue and also it will go through the entire log the Appium or the Selenium logs to pinpoint all the different log lines where that particular issue was identified how many times it occurred and what is the time stamp and finally what was the reasoning right so it'll give this for you at the test case level and also at the test suite level if there are any patterns it has identified for example if there is one particular device where a test case seems to be failing it will highlight the device specific issue or if there is any specific error message that's recurring across different test cases it will be able to highlight those patterns and give you even test suite level analysis right and also not just from an error message standpoint even if there is even if your test case passed but if there is any performance issues or stability issues in your test case right it will be able to highlight those issues as well right so that's how AI failure analysis works and it will give you error distributions and different percentage distributions of your errors in your test cases and finally as a highlighted right you would also be able to have your own agent workflow agent builder specifically designed for QA teams. right. So here with all these AI workflows sorry AI agents working together right where you'll have multiple inputs you can pull in your input maybe we can have an integration from Jira where it can get the user story generate test cases and have your QGEN and then Qpilot right and any errors that need to be reported or any bugs that need to be logged directly log that into your buck tracking tools also having seamless CI/CD integration right or triggering from your pipeline itself. Right. So these kind of agent builder platforms directly designed for QA right which does not require much technical knowledge. So that is our agent builder platform where you will be able to have all these agents working seamlessly and in tandem to have high efficiency as a whole for your QA team. So that is the a quick overview of all the AI key AI agents in our PQR platform. So with that I'll hand it over back to Ain to conclude. Thank you. >> Thank you Jessica. good crisp and quick one for everyone. So hope you have got a sense of how different agents are working together. let me just give you a sense of where are we heading towards. so with all the all that we discussed in terms of agentic systems and agents kind of doing most of the workflows it definitely poses a question as to how are we seeing the QE discipline changing right and it's a question kind of everyone is grappling with and trying to figure out right ways of thinking about it of course we are seeing a lot of changes so one of the way to think about is u in terms of tasks AI would be able to do lot of tasks for us which currently let's say users are responsible for example let's say executing script right or creating an script so AI will become the tour if I have to just kind of define it in a simplistic language AI will become the tour it can it can check did it work. It can execute it can repair different kind of tests. It can find impacts analyzing the differences. it can also find out signals from your reports, logs, identify pattern and some of it you saw in Jessica's demonstration. For example, we have a self-healing agent where if the test fails, it will repair on its own. From the logs, we are generating patterns. All the regression execution is automatic. So AI is basically the executor and the doer in the current Q discipline that we foresee right but I think the judgment part of it is has to be owned by the humans and the QE leaders have to own that role right so if I have to build an agent what should be the objective of that agent who is going to define it is where the human judgment comes into place. Should it work like this? What should be the release conference decision? And it will vary from product to product, features to feature. So it's not that you define it once and it will be kind of used by agents all the time. So for every feature or a product release the release conference decisions would vary and that's where human humans have to come and play the role. what should be the strategy and governance all the guardrails around on around the agent executions risk framing and ownership. and then finally agents are good at executing what is already there or creating generic test cases and exe executing all those kind of tasks. But what about the edge cases? what about putting yourself into user shoes and thinking about different scenarios that the users can execute. I don't think AI will be able to execute it. so some of those kind of tasks has to be owned by the QE teams and they have to focus there and let AI take over the execution part of it. and in that context again continuing the same thought QE will become the harness builder right so QE will set up the context engineering I mean agents are as good as the context it has so like I talked about data fabric who is going to create the data fabric how do you ensure that it has the right context if agent is not able to pick up the context the human has to figure out that it's not working well and then provide the feedback and maybe make the changes do the fine tuning etc. all the architectural constraints like guard rails and when should human in the loop come into play when should agent take over all those kind of constraints and decisions has to be taken by the QE and the organization and then governance right like all the risks and everything related to oversight and governance. So QE will become harness of all the agents that we're talking about. So building agents, configuring it, fine-tuning it to making sure that these are executed in the right way will become the primary role of QE rather than thinking about how to execute the test cases. That's the kind of future we are looking at. so the question is where do we stand right now right? I mean I have seen lot of organizations who are very proactive and have taken initiatives building things ground up and have done some amazing work. But there are organizations who are still thinking about and then and kind of pushing back on and then still trying to work the old way. So the choice is between option A and B. Option A is where you wait and fall behind or and react to change. Option B is start taking initiatives, build something small and then add to it. Right? That's how probably I mean that's the startup world we all I mean I have been part of where we start with something small. We don't have all the time full picture of it. We build MVPs, we show it to the customers and then take feedback and then iterate and that's the approach enterprises also will have to take. in AI world it's all the more important because there is the predictability is not so high yet.
Jessica Selvin
So you have to figure out what is working what is not working and then iterate. So the best time to start was yesterday and the second best time is now. So if not started it start it now right and we as an organization would be more than happy to work with you engage and then collaborate on that part. so it's it's we are sitting at a moment where like I said in the beginning a time where the technology is truly transformative the power of technology is exponential and then it's in our hand to utilize and see how much benefit can we get from this from this moment. so are you already making the decision in terms of making your QE organization AI native AI ready? what it means for you to think about in terms of quality governance and not just test execution and are you thinking about future in terms of building a intelligence in built rather than thinking about individual agents is are some of the kind of points to ponder over and take back home from this talk. so platform is the enabler and in terms of leadership is the multiplier is what we have seen in the industry the it's the leadership which kind of inspires and make sure that the vision is executed. so with that I would kind of end it and take any questions if you have as I said in the beginning love to have your feedback and also feel free to share your experiences in terms of what you have done and what you have built. Would love to learn from you. Thank you so much. >> Well thank you Avin u. So right now like a mentioned we'll open up for some questions some Q&A. I do have a few questions to kick us off with. so while people are preparing maybe we can start with this question Ain like what is your view on local LLMs versus public LLM? You did mention something about custom LLMs that we were using on our platform. >> Yeah.
Jessica Selvin
So what about these local or public LLMs for testing systems? >> Yeah. No, I think I touched it briefly but to reemphasize that point for enterprise it's extremely important to figure out what's the right LLM strategy for your use case. we have seen organizations where they're very security conscious they're very concerned about their data. So in those cases custom opensource LLM deployed within the organization premise is an option and nowadays with the power open source model that we are seeing I mean you can pretty much achieve very complex use cases as well with the custom models. Of course, it requires some in-depth skills to be able to fine-tune the model, but there are more and more resources getting available for anyone to be able to do that. But in terms of strategy, you can think of a hybrid strategy. Certain use cases can be done by custom LLMs or open source LLMs that you can deploy internally. certain use cases you can rely on your enterprise strategy enterprise license that you might be having with the public LLMs. so I feel that future is multimodel but custom local LMS are going to have a big play and also considering the fact that the pricing strategies for these public LLMs are changing very fast. I mean I if you have heard the news everyone is experimenting with tokenbased pricing and we all know that with tokenbased pricing the cost of AI will shoot up like anything right so if you have to manage your AI cost probably you'll have to fall back to your own local deployed model to get better ROI so that's another perspective in terms of thinking about local versus public LLMs >> interesting can take there ain thank you for that here's another question that I'd like to share what is the ideal role of a human in the loop in such system like an agentic QA system where does the human come in between and where >> I think that's a very important decision when we are thinking about building any agentic system I in my personal view I think and lot of reports also kind of support it. Most of the agentic systems which have been able to build or generate ROI has been the hybrid system where a human in the loop has been implemented from the very beginning right and that's the approach we have also taken so in the demonstration you would have seen at every stage for example in the QGEN when you generate the test cases you still have an option to choose or reject the test cases before you finally kind of send it to your automation agent I mean you are always have an option of sending it directly but I mean user has an option of kind of intervening and then deciding what to send or maybe modify it before we send it to the automation agent so I feel that it's a the hybrid approach is going to work the best you can't replace human judgment so that's where I think human in the loop becomes extremely important regular execution for example generating hundreds of test cases is the is the primary skill of AI but then all those 100 test cases are correct is it in the right context do we need to add more thing has it missed something has it considered all scenarios that's where human jud judgment comes into play and humans are good at providing those feedbacks and we should use human in the loop in most I mean at right at right appropriate places within this within the workflow that we are going to build That's the thought process with which anyone should think about building any agentic systems. >> Absolutely agree. Well, a follow-up question to that could be, what are the skills then should people start to develop? Should QA engineers build to stay relevant in this in this model in the coming model or even in the future? What are some skills that they need to be kind of thinking about thinking through?
Jessica Selvin
I think that's that's that's like we think the nail on its head that's that's one of the primary shift we are seeing like I mentioned in my last slide also Q will become the harness rather than becoming rather than being the executor and that's an opportunity as well I mean one of the benefit of AI is after a long time there is a technology where every one can become a builder to give you an example earlier for example, if I had to build something, I was relying on my development team to I'll give the requirement and they will build it for me. Now I'm free. I can use cloud code or maybe lovable to create something myself, right? And that liberates me. That gives me so much power to create so much. in marketing we see that you're not dependent on maybe your designer to if you have content you can create the complete I mean your article without anyone's help right you can do it on your own so you all I mean we all have the opportunity to become creator and that's true in the case of QE as well I mean we are not bound to just time writing test cases and some test cases are so mundane so you would like to spend time thinking about your edge test cases rather than writing some general test cases for each of the feature which is getting developed. So just hand over that task to AI and focus on the edge cases. So in terms of skill I think and I mean working with AI and different tools is definitely important prompt engineering context engineering working with rag implementing rag systems understanding how fine-tuning happens fine-tuning has different ways it could be with rag it could be some other methods so understanding fine-tuning approach all that are definitely the key skills even the QE should understand and that will definitely improve the outcome and the efficiency that they are currently able to achieve. nowadays I mean writing an agent itself has become so easy. So if you can learn a little bit of Python or any programming language you can write your own agents and start kind of delivering value to the organization. No one is I mean you are you shouldn't be waiting for someone to come and ask to do it. If you see mundane task you can write an agent and improve your productivity. So in so many ways you can kind of contribute like I said you can become the builder by acquiring some of the skills that I just spoke about and contribute to your organization. So it's a exciting time for all of us in that context. >> Great. I think we'll wrap up with that on that note and also mention that also a quick mention of our resources on the pcloud website. Aminash did mention about PR engineering and many other skills that he talked about. We have some of these resources that are available on the website. We have a great guide a prompt engineering guide that you can get a hold of for free and go learn about how you can work around with AI agents and with different AI tools that are out there. with that note, let me thank Avin and Jessica for sharing, your valuable nuggets of information in the Aentic testing space and doing a deep dive into what Agentic quality platform is made up of and how it works as well. So, thank you so much.
Jessica Selvin
We also want to thank each one of our participants for joining in, taking the time to join the webinar. We're sure that some of these insights that was shared during the webinar might have sparked a few ideas. So my recommendation would be to mull over these ideas, discuss with us, talk to us. If you have any questions, you can reach out to us. just write to us at info Pcloudy.com and we'll respond to you as soon as possible. You can also reach out to us on our handles. We're active on Twitter, on LinkedIn and the other on our website. You can just click on the chat option and reach out to any one of us at PCloud. So, thank you once again for being an engaging audience and happy orchestrating. >> Thank you so much. Thanks everyone for joining.
