Dinakar
All right, great.
Dinakar
Uh, hello everybody and welcome to Pcloudy's webinar on the QA professionals guide to AI prompt engineering.
Dinakar
My name is Dinakar and I'll be the host for today.
Dinakar
Uh, before we get started, I would like to go over a few housekeeping rules.
Dinakar
Uh, you may have noticed that your line is currently muted.
Dinakar
If you do have any questions, you can submit them using the Q&A option available at the bottom of the screen.
Dinakar
We will answer your questions as part of the Q&A section at the end towards the end of the webinar.
Dinakar
And also please note that this webinar is being recorded and we will send out the link of the recording to share with your colleagues later or watch it a second time.
Dinakar
Uh we also have a special giveaway for all our attendees uh today.
Dinakar
Uh we will be sharing a prompt engineering guide that is specifically designed for QA professionals like you.
Dinakar
Uh so with that said, let me hand it to our speaker.
Dinakar
Let me introduce the speaker for today.
Dinakar
Avin Dvari.
Dinakar
Avin is a renowned thought leader, a recognized keynote speaker and the co-founder of Pcloudy.
Dinakar
He has 20 plus years experience in product development, testing and recently in the AI space as well.
Dinakar
He brings with him a passion for emerging technologies and quick adoption.
Dinakar
both of which have solidified his reputation as a leader in the AI testing space.
Dinakar
Without further ado, let me hand it to our speaker Avin Tavari.
Dinakar
Over to you.
Avinash
Thank you so much Dakar for introduction and setting up the rules.
Avinash
Uh welcome everyone.
Avinash
Uh great to have you all here for this webinar.
Avinash
uh thanks for taking out time at this hour.
Avinash
Maybe it's early morning for you, for some it might be late evening.
Avinash
So thank you so much for finding time.
Avinash
Um AI is changing our world and um number of signups and participation for this webinar uh kind of reflects the change in the direction of the wind.
Avinash
So I am super excited to show you some details of the world to come and also we'll try to give you a way to navigate in this new world.
Avinash
So let's get started.
Avinash
Uh back in 2023 when chat GPT was launched um I was curious what the future will hold for all of us and I was wondering how it's going to change the role of quality engineers in general testing and quality space but having spent la last two years building AI native tools ploudy is today fully agent authentic digital experience testing platform.
Avinash
I feel more confident and optimistic about uh the future of quality and testing and that's what we are going to talk about uh and and focus more on the specific area of prompt engineering.
Avinash
Uh but you will you'll see a glimpse of what the future could look like.
Avinash
But before we kind of get into the actual um specific of the session that we are planning to talk about, let's do a quick thought experiment.
Avinash
Of course, I can't see you.
Avinash
Uh so uh whether you close your eyes or you do it without closing your eyes, it's up to you.
Avinash
But I want you to just spend 10 minutes think 10 seconds, sorry.
Avinash
uh just 10 seconds thinking about what all announcements what all tools that you came across uh what news that you came across in last 30 days related to AI just last 30 days I'm not asking about one year or 6 months just think about whatever news or announcements or tool releases that you came across in last 30 days so just for 10 seconds so countdown begins […]
Avinash
Hope you you could think many I mean as far as I am concerned I could think of so many um announcements so many interesting news for example chat GPT uh 5 was released few days back Google VO a video generation platform was released uh almost a month back and it's already transforming the video industry.
Avinash
Uh Google also released Gemini AI preview in Google search and whole SEO industry scrambling for answers.
Avinash
uh lovable a prototype uh uh app development platform reached $und00 million of revenue in just 7 days just uh 7 months uh with handful of folks and I could go on and on and talk about uh all the changes which happened in just last 30 days.
Avinash
What I want to point out here is the pace of change that we are seeing around us is overwhelming and it's it's accelerating as we speak.
Avinash
So the transformation with AI isn't coming, it's already here.
Avinash
So with that u being said or with that as as fact uh I just want to take a step back and talk about why AI is different from some of the other technology wave that we have seen in past and I call it I mean I I I would like to term it as uh the leverage equation.
Avinash
AI is not just a tool it's a leverage.
Avinash
In past we have come across many technology waves.
Avinash
We saw many tools uh productivity enhancements but most of these were linear in nature.
Avinash
So you add a tool you get proportionate result.
Avinash
You add more people you could get proportionate results.
Avinash
But with AI what we are seeing is an exponential u leverage.
Avinash
So one person with AI can give you 100x output.
Avinash
And I'm not joking here.
Avinash
100x is something people are already talking about.
Avinash
If you have heard about Y Combinator, they have released a new thesis where one of the theme is 10 people 10 billion dollar company.
Avinash
So just imagine the scale and the exponential nature of AI world that people are already envisioning.
Avinash
Now we and we can see this in real world examples as well especially in marketing development velocity um in in in many industries for example in finance the fraud detection has become a task of few seconds which was unthinkable uh before and in medical science the diagnosis has become so easy with the help of AI so it it's happening all across in in different industries that uh we see around the question is what's happening to QE right what and that's the question mark I have it here that what kind of leverage that we can have in the field of QE of course I'm not uh going into detail of uh the the different aspects of change in the quality engineering because that's not focus of the session but you'll see a glimpse of what is needed for QE to transform and become relevant in in the in the coming world or the coming future Now um one of the uh other thing that I want to talk about before we get into the specifics is what is the paradigm shift here from and what is this leverage that we are talking about.
Avinash
One of the change that has happened with AI is English has become the new programming language.
Avinash
So it's no more about uh writing code or maybe in in in space of QE writing test cases.
Avinash
It's about describing what you want, right?
Avinash
For example, test your checkout process with edge cases.
Avinash
You give context and it can generate variety of test cases that you need.
Avinash
And that is so powerful.
Avinash
We we are feeling threatened about it.
Avinash
But that's so powerful.
Avinash
It's it's bringing it's elevating each of us to become a builder than a doer.
Avinash
So all of us and all of us today have an opportunity to become a builder from what we are right now today right I mean for example uh if you want to build something and if you're not if you don't know coding right in the in the previous era you were constrained you need to go and talk to develop some developer to help you today you can do w coding and build something and show it to the world it's it has become so easy for you so if you have an idea bringing that idea to uh execution has become so simple right so we all can become a builder that's the whole power of this leverage that I was talking about in the previous slide and if everyone can become a builder and become more creative you can generate so much more and that's the optimistic nature of this change that we are talking about so QE would not require to write tests in future in few years from now they will have to guide AI they they need to become a uh strategist uh and that's some of the glimpse of it is is what we're going to uh see today in the session uh as we move forward now u and and continuing from what uh I just talked about Andre Karpati one of the earlier co-founder of open AI he has been um he has been one of the most prominent figure in this space and he said that one of the most valuable skill won't be coding like I said English is the new programming language it's all about communicating with AI right so how do you communicate effectively with AI is one of the primary skill that everyone needs today in any field be development coding marketing sales we all need to learn how to communicate with AI.
Avinash
And when we talk about communicating with AI, that brings us to this whole error I mean the whole area of prompt engineering.
Avinash
Right?
Avinash
Now, how do you communicate with uh AI?
Avinash
Whether you are building an agent, agentic platform, building workflows or directly interacting with an LLM through a chat interface, you are in some form communicating with the AI and that form is your prompts.
Avinash
How do you define the prompts?
Avinash
How do you write the prompts?
Avinash
Defines the success of whatever you are trying to build beyond the other engineering skill.
Avinash
One of the key skill is to write those prompts, create those system prompts, etc.
Avinash
to be able to communicate with AI.
Avinash
Right?
Avinash
That's what prompt engineering is all about in in in a nutshell.
Avinash
How we are going to communicate with AI.
Avinash
So to understand this uh whole prompt engineering skill set it's also important to understand what's under the hood and if we understand that I think the the rest of the part becomes much easier to um grasp and uh to to build uh something on top of what we'll understand what's under the hood will become much easier.
Avinash
So I'll spend some time explaining what's under the hood.
Avinash
So just be with me for another five 10 minutes before we jump into the uh prompt engineering specifics.
Avinash
So some basics we all know AI is not new.
Avinash
It has been there in the market for long we had computer vision machine learning traditional artificial intelligence from quite some time from last at least 20 years.
Avinash
A genative AI has been there in the market for quite quite some time.
Avinash
We all have used Google translate and Google translate is an example of generative AI which is there from last 10 12 years using NLP.
Avinash
We have seen you you enter a text in English and you can generate a text in French or Spanish or whatever language that you want.
Avinash
So it's generating a content.
Avinash
So generative AI has been there in some form.
Avinash
But what has been new uh is this whole large language models or language models which has kind of changed the game and taken generative AI to another level where we all can interact so easily and it has democratized the access of AI and within the LLM space there are two key concepts one is foundation model and using foundation model building a language model.
Avinash
So language model is all about data architecture and training.
Avinash
So I'm not getting into the architecture piece of it today but we'll talk a little bit about data and training because that is the basis of how uh a prompt can be created right so if you understand that part I think rest of the concept becomes very very easy to understand.
Avinash
So diving a bit uh deeper into language modeling of the concept of language modeling is all about predicting what comes next.
Avinash
Right?
Avinash
So for example, if you give a sentence to a language model, for example, continuous testing shifts testing two, it might give you left or early, right?
Avinash
So given the sequence of words so far, predicting what comes next is the core of any language model and it has been there for some time.
Avinash
Now the new architecture that came as transformer model if you have heard about that changed the game in making the prediction much more accurate at a very large scale in any format.
Avinash
So that's where we are talking about language models here.
Avinash
So again you feed the context and the sequence of words and it predicts it creates the probability of the next words and whatever is the highest probability you get as the next word.
Avinash
That's the fundamental of uh any language modeling.
Avinash
Now on language modeling uh there are other steps that happens so that you you are able to interact uh in a much more instruction format the way we have we are used to uh talking to chat GPD and all now where how do we reach that stage is critical to understand I'll try to explain it quickly and that happens through a process called training and that is divided into three parts the first part is what we call it pre-training uh the first section that you see pre-training stage where the model is fed a huge corpus of data trillions of parameters or trillions of data sets uh depending on what you tune it could be smaller data set as well but if you consider chat GPT they have kind of scanned probably the whole internet trillions of uh data uh sets and you can go and look at open crawl uh data sets on hugging face and all and you get some understanding of what's the volume of data set that are talking about.
Avinash
So model is fed in a raw format all that data which exists uh in in those kind of data sets I was talking about for example open crawl right you get a foundation model and the foundation model can only predict next word it can't predict a complete uh structural structured output that we uh see in in interact interaction with chpt or claude that happens in the second stage which is called superi supervised fine-tuning where you teach model how to interact and this is another massive exercise where millions of question answer pairs are created to teach the model that if a particular type of question is asked you need to answer in a particular format and when that sft happens model understands that okay if someone asks a question I need to respond it in a this in a particular way Right?
Avinash
That is called tuned model where the model is ready to interact with the user and answer a much more structured responses.
Avinash
I want to focus most uh I want to focus a bit on this whole supervised fine-tuning concept because that is important to understand for building right kind of prompts.
Avinash
And the last part is reinforcement learning where you teach the model what is right answer, what is wrong answer.
Avinash
You generate multiple responses from an LLM and then tell the model through human feedback or now we have reward models to tell that this particular response is better.
Avinash
This particular response is maybe less uh um less accurate.
Avinash
So model understand that the these kind of responses are better.
Avinash
I we are not going into this part in detail.
Avinash
Just understand that it gives a model a sense of what is good and bad.
Avinash
So these are the three steps that happens and again there are multiple approaches but largely these are the three steps that you will see in any kind of model training.
Avinash
Once these three steps are done you get something like chat GPT.
Avinash
So you can use a open source uh model add your data do supervised training and then create your own opensource LLM or SLM.
Avinash
So this is the stage where you get something like chat GPT.
Avinash
Now I want to focus specifically on this uh SFT.
Avinash
What is SFT?
Avinash
Right?
Avinash
Like I said you create question and answer pair.
Avinash
So for example this is one of the math question right?
Avinash
So where you create input output pair uh and like I said it will be millions of pairs that you'll create across depending on what kind of model you're training.
Avinash
It's a generic model.
Avinash
You'll have maths questions.
Avinash
You will have um general knowledge questions and all sorts of Q&A pairs.
Avinash
So for example, this is a simple math question that Julie is reading a 120 page book.
Avinash
Yesterday she was able to read 12.
Avinash
So how much um if she wants to read half of remaining pages tomorrow, how many pages would should she read?
Avinash
Now output could be just the answer of this question.
Avinash
Right?
Avinash
So you can create input output pair in a format where you have a question and you directly give output as the answer as one of the uh uh Q&A pair for training.
Avinash
That's one way to train the model and it was one of the early way of training the models where you just created these kind of pairs and did your SFT.
Avinash
So model understands what should be the answer.
Avinash
Now but model here doesn't understand how you are calculating the answer right and that is where the next research came where in the output pair you started explaining how the answer has been achieved how the 42 was achieved.
Avinash
So like you solve a math paper you are writing the answer explaining how you achieve 42 right and this was the first time done by deepseek in a much more uh detailed way and that was start of reasoning models where when you are training you're not just providing a simple Q&A pair you are talking about how to reach that response and it's not just about math for example I it could be an input output pair how what is gravity on a moon right so you can say moon gravity is one sixth of earth's gravity or you can explain more uh the reason of that part before you talk of the final answer in the in the output pair so that is how now the reasoning models are being trained right and it's important just keep that in And because this uh this kind of dis uh will help you understand why we are writing prompts in a particular way.
Avinash
Right?
Avinash
So nowadays uh I want to repeat this again.
Avinash
This is extremely important that most of the training happens by providing the reasoning traces as we call it in the uh model language as part of the input output pair when the model training is happening and millions of such uh Q&A pairs will be created.
Avinash
Now another aspect that we need to keep in mind uh under the hood uh model settings.
Avinash
Now you might not be looking at these settings because you are interacting with chat GPD.
Avinash
They've already done lot of uh fine-tuning and they have arrived at parameters.
Avinash
But if you're interacting with the APIs of for example OpenAI, you might be interacting with these uh settings or if you're training open source model, you'll have to set that and these impacts your output.
Avinash
So you need to understand what these parameters are.
Avinash
So for example, there is one important parameter of a model is temperature.
Avinash
Right?
Avinash
Now a temperature closer to zero means a more focused concentrated response.
Avinash
It's good for fact-based prompts.
Avinash
But if you want to write for example a blog or a creative article or a poem, you want model to be much more flexible, broaden its area.
Avinash
So you can set the temperature high.
Avinash
So temperature low means more focused response.
Avinash
Temperature high means more wider response.
Avinash
So depending on the kind of uh work that you are doing you can set the temperature max tokens.
Avinash
Max tokens is all about how much maximum tokens you want in your response.
Avinash
This is also important from the costing perspective because more tokens cost you more.
Avinash
If you are using let's say APIs and all right so depending on what kind of uh task that you are doing you can set the number of tokens this uh aspect is u another important parameter uh top p right so top p value again is le to focused and confident response due to limited choice of tokens available.
Avinash
So if you want a very focused fact-based task, you can set this value as low.
Avinash
If you want a more creative kind of response, you can set higher top.
Avinash
So these are certain model parameters that we need to keep in mind to make sure that we know that okay, if you're not getting right output, this could be also another area to look at, right?
Avinash
So just keep these uh under the hood concepts in mind.
Avinash
With that I think we are all set to discuss how do we write good prompts or what is prompt engineering.
Avinash
I'll skip this part.
Avinash
Um we have discussed it.
Avinash
Um but what are the good prompt engineering techniques that you should be looking at and for that you need to understand the anatomy of a good prompt right uh or prompt.
Avinash
So if you look at all the recommendations given by open AI or other models, a prompt is typically divided into five different components.
Avinash
Uh role, context, uh instruction, uh success criteria and input data.
Avinash
Now let's look at this as with an example.
Avinash
So for example this is one of the prompt and the prompt says that as a QA engineer which is a role with expertise in web accessibility and security testing you are giving the context that I am I'm going to talk more about accessibility and security testing uh uh in in the task right then the actual instruction what do you want the prompt to uh uh do for you so generate comprehensive test cases is uh or a financial service uh registration form right and then that must comply with which is more like a success criteria and the your input data that across these areas I want to generate my test cases.
Avinash
Now I can write a simple prompt that give me test cases for a financial service registration form for uh accessibility testing.
Avinash
That could be another prompt and that's how probably we have been uh trying that's not the most optimum way because the output that you'll get will differ significantly from the prompt if you try this prompt because this follow the structure a model expects right because model has been trained in this way when we talked about SFT in SFT all this uh is part of the uh input output pairs that we were talking about that for what is the role what is the input what is the output there are context I mean it's a it's a broad uh area that we can get into but this is how a ideal prompt should be right you need to have all the component in place to make sure that the model understands it well and then you get a better outcome let's dive uh bit deeper around different techniques And there are three main techniques that we have seen uh in the prompt engineering space.
Avinash
The first uh is the most simplest one which we all have been using u for for any task that comes to our mind.
Avinash
We go and just try a zeroshot prompt right.
Avinash
So zeros prompting is the most basic kind of uh prompt engineering where you ask the LLM to perform a task without providing any kind of example just straightforward and it's it's more suited for simple well- definfined task for example I can go and ask for a definition give me I mean what is continuous testing right or tell me how do I create different set of data for a password field with with these many characters.
Avinash
Very simple tasks.
Avinash
You need quick results.
Avinash
You don't require too much of domain knowledge.
Avinash
The model doesn't require too much domain knowledge to perform that task.
Avinash
So for those kind of examples, zeroot prompting is uh is is relevant and you can go ahead and use it.
Avinash
Just to kind of uh get into more uh detail of what a zeros prompting is here.
Avinash
Uh it's about no example, no input.
Avinash
Um and this is one simple example that classify this statement to a positive or negative statement.
Avinash
The cookie doesn't test good.
Avinash
The model output is negative.
Avinash
So it's a very simple straightforward fact-based uh task and response that we're talking about.
Avinash
So it it is good for like I said simple task not suitable for complex task struggles with specialized task which requires uh domain understanding or reasoning um uh to be done uh not good for long form content.
Avinash
For example, if you want to generate a blog or a technical article, you just say that generate an article on on a particular topic is not good enough.
Avinash
um the outcome will not be uh great and and you will have to do again prompting etc to make sure that it it meets your expectation.
Avinash
So we can avoid zeros prompting if we are talking about complex and long form uh content kind of tasks.
Avinash
And uh another I mean example of a simple zeros short prompt in in specifically in case of a testing scenario is list all the test cases needed for a password strength validator that requires minimum eight character this and that.
Avinash
So model is able to generate this without any context or without other details to be added in this prompt engineering.
Avinash
So simple zeroot prompting works for it.
Avinash
But uh moving on to the other technique which is used more frequently and I think u we all have learned to use it you might be using it already few short prompting um and the key aspect of the few short prompting is providing multiple examples to establish pattern.
Avinash
It's good for complex test scenarios.
Avinash
Um and let's see one example of it.
Avinash
So this is how the the zero shot sorry the few short prompting is u structured.
Avinash
Uh you provide some examples and then your input uh the task that you want to do.
Avinash
So for example you are you are giving an example to the prompt.
Avinash
Best product ever means a positive uh sentiment.
Avinash
It didn't work for me not happy negative sentiment.
Avinash
What a scam don't buy negative sentiment.
Avinash
And then you say this is my input.
Avinash
This is so helpful.
Avinash
Love it.
Avinash
Give me the sentiment.
Avinash
Based on the examples that we have given the model understands what do you mean by positive and negative sentiment in your context and then response you give your response as positive.
Avinash
So by giving you by providing examples to the model you are uh allowing model to understand the pattern that you are looking at and based on that pattern model gives you answer in your context.
Avinash
So you can you can get much better outcomes uh with a few short short prompting uh examples.
Avinash
For example, I think the slide is slightly misplaced here.
Avinash
So this is one example.
Avinash
uh for example I'm I'm trying to create a validation test for credit card number field which is the bottom uh input I'm trying to uh provide as part of the prompting but I have given examples of what are the uh validation tests what are the valid inputs what are the invalid inputs uh for email for phone numbers and then asking that okay create validation test for credit card number field so now the model understand what exactly are are looking at from valid input kind valid or invalid kind of uh examples and based on that the model will generate the output.
Avinash
So this is called fshot.
Avinash
So whatever is your task that you want the model to uh generate output for you give some examples which you think are right results for you and this is used extensively in any kind of for example agentic uh systems that you build.
Avinash
So you tell agents that okay this is what the task is and these are some few short examples that you can refer to to provide the output and it need not be limited to two three you can have more few short examples.
Avinash
It depends on the context length and again it adds to the cost because more context length more tokens and it adds to the cost.
Avinash
So you can have an optimum number of few short examples depending on the kind of task that you are uh looking to perform.
Avinash
So with that with that I think we are we are coming to the most important and and the recent uh development the chain of thought prompting right and this approach is again um you can go back and remember what I talked about in the supervised uh fine-tuning that we can create an input output pair where input uh is a question and the output is the direct answer versus The output has a reasoning trace as to how did you achieve that answer that is nothing but a chain of thought.
Avinash
You are asking model to have a chain of thought before reaching an answer.
Avinash
And you can apply the same principle in your prompting as well.
Avinash
You are forcing the model to think in a certain way.
Avinash
That's what is chain of thought uh prompting is all about.
Avinash
And this is the most effective way of getting the right outcome.
Avinash
Of course, it's required for a complex task.
Avinash
You don't require for a simple like we saw other examples.
Avinash
But most of the work that we are doing today uh requires this kind of thinking.
Avinash
How do I create the right chain of thought kind of prompt to achieve the best outcome from the model?
Avinash
And also when you are writing agents etc.
Avinash
chain of thought becomes the most important uh and effective way of uh writing the prompts.
Avinash
Let's uh see an example here.
Avinash
Again, this is like a math uh example where uh you have model input and model output straightforward.
Avinash
But here what I'm doing is uh with the question I am giving an example of an answer and how to think about it.
Avinash
So this is an example of a kind of few short prompting one question or answer I am already uh adding in the uh prompt and then I'm asking the other question when I do that the model understands that I need to think in this way I need to apply this reason before I answer the question that is exactly what chain of thought prompting is and we can look at more examples of this for example in this case let's think step by step about testing an e-commerce checkout process.
Avinash
I can just say that and stop it.
Avinash
That becomes my zero shot prompting.
Avinash
The model will generate some outcome.
Avinash
But here what I'm doing is my I'm asking model to think step by step and I'm giving an example of what is step by step.
Avinash
So first because what happens is if you just say it might generate for few components and it might miss certain components.
Avinash
Then again you have to see what it has missed and then add uh uh your clarifications and then try to generate it again again in instead I can ask the model to think about all the components involved.
Avinash
It's a simple example.
Avinash
I can add more in this chain of thought reasoning that okay think about component think about testing types.
Avinash
You can also add what are the testing type you should think about think about these kind of use cases.
Avinash
So you can give a stepbystep way of thinking to the model and the model will perform the task in that particular fashion itself.
Avinash
And that is is is proven to be the most effective way of generating outcomes through a prompt engineering uh uh process.
Avinash
Other examples for example and you can also nowadays you can use think tokens but okay you can just ignore this for for the time being you can say consider a multi-step registration form with these steps right and then ask what are the possible uh user paths I can just say that okay there is multi-step registration form with username and and password go ahead and create the edge cases for me and rather than that I'm asking the uh prompt to go step by step and then create the test cases for right so again simple examples we'll we'll look at some complex ones as well another example in in case of let's say API security you're again asking the model to think about all these aspects uh in a step-by-step way in a sequence before answering the question so that is typically a chain of thought um u way of generating prompts […]
Avinash
Now uh we have discussed most of it.
Avinash
There are some optimization techniques again links linked to what we discussed in terms of f short in terms of zeros and chain of thought.
Avinash
But to give you more specific way of thinking uh there are three optimization technique enhance the context.
Avinash
Um again you can link it to what we just discussed.
Avinash
So for example you can say generate a test case for a form right this is poor prompt because you you're not providing any context I am adding my role as a Q engineer generate test case for registration form I am providing better context it's better than the previous one but we can we can reach a much better state by adding more context like I said you are a senior Q engineer with expertise generate comprehensive test cases across these uh compliance requirements ments consider both functional and non-functional requirements.
Avinash
So I'm adding more context uh to the prompt which provides better output.
Avinash
So enhancing the context is one way of optimizing the prompt format the output and again this is a few short you can link it to fuse short that what happens a lot of times is we generate and then the output is in not the right format that we are looking at.
Avinash
Okay, generate the test case and it generates the test case in a bullet point format.
Avinash
That's not good enough good enough for you, right?
Avinash
You want test case in a t table format with these five fields, serial number, steps, expected output and things like that.
Avinash
You can provide an example of the output that you are looking at.
Avinash
Right?
Avinash
So, generate the test cases for a login functionality in this format.
Avinash
test case ID, title, step one, step two and things like that.
Avinash
And again, you can add more context to it.
Avinash
Uh so formatting the output is another way uh uh to improve and optimize the prompt.
Avinash
And the third way is third way is defining the constraint because what happens is when you write a prompt, the model can go absolutely broad and give you so much result that you are overwhelmed.
Avinash
you can restrict by giving it uh the constraints and and that is how we think in QA right we look at okay what are the uh data sets that we should be testing for positive negative data sets so you can give those constraints the model as well so for example generate test cases for a file upload feature with these output constraints maximum file size allowed formats and this and that I mean you can have that list so defining the constraints um give you much more uh accurate result and and uh much more usable output if I can say it correctly.
Avinash
So with all that I mean we saw different prompting techniques we saw optimization techniques what would be an ideal prompt and this guideline was released by chat GPT um few days back uh chat GPT 5 um few days back this is one of the ideal prompt which I generated as for the for a QA professional right taking one example of let's say the the core task is create a comprehensive QA task plan test plan for two factor authentication for login feature in a mobile banking app.
Avinash
I can just say go ahead and create a QA test plan for me for this feature and leave it and let the model generate.
Avinash
I'll get output but how do I improve this to make it the ideal prompt where I get the best output from the model.
Avinash
Right?
Avinash
And we are using all the techniques that we discussed uh so far.
Avinash
One is we are using all the components of a prompt.
Avinash
So it starts with a role right.
Avinash
So you can read here.
Avinash
I'm not sure the font is visible.
Avinash
I hope it is.
Avinash
Uh but just to give you a quick summary in role we are specifying that act as a senior quality assurance special with with 15 years of expertise in functional non-functional and digital experience testing of web, mobile etc.
Avinash
Right.
Avinash
Uh then we talk about task and in task we are talking about different areas to include functional, negative, boundary, non-functional scenarios.
Avinash
So we are asking the the model to think in those stepwise u rather than just going very broad.
Avinash
Then we are providing context.
Avinash
Okay, what kind of application it is?
Avinash
What kind of context is present for this for for the model to generate this QA test plans?
Avinash
For example, what is application type?
Avinash
What kind of environment are we talking?
Avinash
What kind of tech stack compliance other constraints dependencies and you can add more right and that is where the uh new skill of a QE is expected.
Avinash
Rather than writing individual test cases, you need to write these ideal prompts and as whoever writes in a better format, whoever writes better context, better task description, we'll get a better output.
Avinash
The third uh the fourth uh block is reasoning where you are asking model to reason it in certain way.
Avinash
So plan must validate that uh two-actor authentication is secure, reliable, user friendly, performant.
Avinash
you can add more include positive, negative and edge desk cases.
Avinash
So, so these are the four component that we discussed in the beginning as well and we we saw a glimpse of it in in different techniques that we talked about.
Avinash
Finally, the output format, right?
Avinash
I mean this continuation of the prompt because it's so big that I couldn't fit it in one slide.
Avinash
So, continuation of the previous prompts, I'm also giving the output format that I need the uh test plan in this format with these fields.
Avinash
And then finally constraint that okay when this condition meets you can stop when all the functional negative boundary condition is listed you have added risk mitigation strategy you have added automation plan compliance checks etc.
Avinash
This combined thing is one ideal problem.
Avinash
So we are moving away from just write asking models to generate with single line questions to giving it in a way that the models can generate right optimum output for you because models are designed to think in this particular way.
Avinash
Right?
Avinash
So if you understand those components and use it to your advantage, you can generate amazing outputs.
Avinash
Right?
Avinash
And with that I think you can also um think about how the roles are changing right you you might not be required to write actual test cases in some cases very complex domain inensive applications where the models are not having that kind of knowledge you you might require but in in most of the scenarios that we come across probably the manual test case writing or even the automation Code generation will not be so much needed but you will require to write right kind of prompts train your agents to perform the task in a in a accurate way is the expectation uh we are looking at from all quality engineers or uh sds or whatever roles that exist today in the quality world now there are few don'ts I think we have covered most of it in our previous techniques as well.
Avinash
So no v prompts we we saw that without context uh and without success criterias or some of the other components that we talked about uh these are don'ts but I think we have covered it in pretty much detail.
Avinash
So with that I think we have come to the end of this session.
Avinash
I want to just conclude um with that thought again that we the the AI is changing our role from uh doers to builders right you can see the kind of creativity uh that you can apply using some of the techniques that we are talking about we all can become uh strategist here right you don't have to be let's say in today's world of 15 years experience to write a test plan you if you can write a good prompt you can generate a test plan even if you have let's say few years of experience.
Avinash
So it's it's allowing all us to be more creative become builder and that will be so powerful in in in the future to come.
Avinash
So with that thought I think we have a giveaway as well.
Avinash
You can uh whatever we discussed is also part of the prompt engineering guide that we have created.
Avinash
So you can download and and look at it in your uh uh free time and also provide your feedbacks on that.
Avinash
Um thank you so much for listening.
Avinash
I hope it was useful and uh you got some insights into how to be more effective and how to communicate effectively with AI because that's the skill set we all need.
Avinash
I I don't think we can say that um why should I do that?
Avinash
It's a choice.
Avinash
It's not a choice now.
Avinash
It's a mandate that we all need to start behaving like AI engineers.
Avinash
That's the expectation which is there in front of us.
Avinash
So, thank you so much for listening and we are open to question and answer if you have any um if you're not not able to answer all the questions we will respond on emails.
Avinash
Uh I'll be happy to take up as much as I can in the time which is left for us.
Avinash
So, Dak, over to you.
Dinakar
All right, great.
Dinakar
Uh, thank you, Abinash.
Dinakar
Uh, thank you so much.
Dinakar
Well, uh, we do have a few questions that are pouring in.
Dinakar
I'm just going to give it a few minutes while that's happening.
Dinakar
Also um just so that all of our participants are aware uh we have posted the you know the the engineers guide the QA engineers guide for prompt engineering.
Dinakar
So you can download that from the chat box um you know using the link or you can just click on the download button the file is right there for you to use and go through.
Dinakar
Uh with that let me just quickly look at our questions.
Dinakar
One of the questions that we have is a van, how do you balance AI generated test coverage with human review uh to ensure quality?
Dinakar
I think this is more like the post using the prompt but yes.
Avinash
Yeah.
Avinash
No, that's I mean um I kind of uh briefly touched upon that part that we all are becoming strategist.
Avinash
Um and in that role our job is about uh reviewing looking at deeper insights than uh executing routine tasks.
Avinash
Right?
Avinash
Um and my uh feeling is that human in the loop is the best approach which will work in the future.
Avinash
Um uh that that is in front of us.
Avinash
uh whether we are directly interacting with LLMs or whe whether we are becoming whether we are writing agentic workflows they'll be human in the loop approach where whatever is generated by the u the model will be the humans will have an oversight of that and final approval will be done by humans before it moves to the ne next stage so that is extremely important so you are right I think we all need to use the skills that we have to review and then give a final go ahead to the next stage of whatever that we are generating.
Avinash
It could be test coverage, it could be test plan, it could be automation scripts, execution workflows.
Avinash
I mean for example once the automation report is generated AI can also log bugs but you can always have human in the loop to review the bugs finalize uh whether it has to be tried it has to be sent to devs that that human insight will be needed uh at all stages.
Avinash
So I think hope I answered the hope I have answered that question.
Dinakar
Absolutely.
Dinakar
Uh just another question here.
Dinakar
Um are there any generic prompts for testers that one can use to get the best results?
Avinash
I I I don't believe in generic prompts.
Avinash
uh u and that's the reason I kind of focused on one of the ideal prompt uh where uh you can follow the strategy to create your ideal template right once you are you have been able to generate your ideal template you can use it for various tasks right but that ideal template requires certain aspects which uh which is contextual to you right for example the kind of application in the I mean the context of the application the constraints the type of testing that you want to do or task type of task that you want to perform that is very specific to your environment your context.
Avinash
So you need to learn the uh strategies and the components to be included and based on that you need to build.
Avinash
I would not suggest generic prompts um to anyone.
Avinash
That's the reason we focused on kind of talking about the strategies.
Avinash
How do we write the good prompts and once you know the strategy it becomes much more simpler for you to use that in your uh task or in your context.
Avinash
So I would suggest to go and look deeper into these strategies than look at generic prompts.
Dinakar
We have another question.
Dinakar
How do you see prompt engineering changing the role of a QA tester in next two years?
Avinash
Yeah, I think I have kind of answered that uh uh as part of my discussion itself.
Avinash
We are we are becoming more QE strategist, QE um um reviewers than uh executors.
Avinash
Um and this requires a very strong prompt engineering skills.
Avinash
Um and this this is not just for interaction with models, right?
Avinash
Where you are talking about specific task.
Avinash
For example, you might be required to write agents.
Avinash
And when you write agents, the first thing that you need to do is to write the system from to the agent.
Avinash
And some of the skills that we talked about is what you need to write the system from, right?
Avinash
So whether you are directly interacting with the model, writing agents, creating workflow, this is one of the core skill that you need right and this has become essential part of our jobs.
Avinash
I think if someone is not adopting this there is definitely a threat.
Avinash
Um but if we adopt it as I said we all have an opportunity to become uh builders here and provide much more output to our teams to to the to the organization create our I mean and do our jobs much faster with an scale which is unprecedented so it's an it's one of the core skills that we all need to learn whether in testing or in any other area Right.
Avinash
So as I said, we all have to become AI engineers in that sense by understanding how do we communicate with LM through prompt engineering and then some of the other skill sets uh which are needed.
Avinash
Okay.
Avinash
Uh we have another question from Anoj is prompt engineering skill role will be fading in upcoming years as the new AI models are getting better at understanding vague or incomplete statements.
Avinash
I mean this is this is evolving.
Avinash
Um but there will be some I mean uh the maybe the format of the prompts will change but I mean without context without right inputs whatever uh new model comes to the market I don't think you will be able to get the outcome that you're looking at right um and even after reasoning models came into play right we were we were dealing with pre-reasoning era uh direct chat GPT even after pre-reing the comp the prompting has become much more complex because we are expected to write those steps um uh the context otherwise you will get the output but you will struggle to find the right output you will end up spending more time um answering clarification questions or having multi-shot prompting to reach your outcome rather If you write the prompts in a in a particular way and the prompt engineering skill set will evolve as someone is pointing out rightly that the new models will come but I feel that uh a way of communication will always exist until we reach a stage where maybe AGI comes into play and that that's a threat to humanity itself but uh until that happens I think we will require to learn and communicate to the model that communication process will change prompting skills will change like in any other skill set we keep learning new things but we the reality is that we need to learn to communicate with the models and there is more work uh there are more use cases coming up with this skill set so I don't see that uh we'll not be able to utilize this skill in in next at least uh five years or
Dinakar
Yeah, agree.
Dinakar
Agree.
Dinakar
Yeah.
Dinakar
And able to hear me.
Dinakar
Uh, can you guys hear me?
Dinakar
Okay.
Avinash
Yep.
Dinakar
All right.
Dinakar
Great.
Avinash
Um, for some reason, I mean, some technical glitches are going on and my video is off.
Avinash
Not able to get that back on.
Avinash
Uh but uh the questions I mean Rohit uh Roit were you able to kind of finish up or were there a few more questions that we need to answer?
Dinakar
Yeah, one last question we got from Ganesh.
Dinakar
It's which LLM can be used for this for better results.
Avinash
I mean it's a it's a choice of the enterprise that you are working with.
Avinash
I mean at an individual level I mean um claude or chat GPT are the two best models u that exist but there are so many open-source ones which are extremely good like deepseama when so depending on what's your u uh what's the context that you have can you use open source models are you allowed to use public models.
Avinash
So depending on the organization policy and the task that you're looking at, you can choose.
Avinash
But if if you don't have a preference, open API chart GPT or Enthropic cloud are the two options that you can look at to start.
Avinash
Uh very easy to use and and yeah, you can use some of the skills that we talked about to get started on.
Dinakar
Great.
Dinakar
So I think we are we are out of the time.
Dinakar
Um uh we will conclude the session for today and we'll look at uh we'll answer some of the questions.
Dinakar
If we haven't been able to answer it right now, we can send it through email.
Dinakar
We can reach out to uh the specific individual on email.
Dinakar
Uh with that I would like to conclude.
Dinakar
Thank you so much for being so patient and asking relevant questions.
Dinakar
Um yeah, thank you.
