Week 3 Exercise - Refining Prompt for Testing Task

Thanks ​@parwalrahul for the assignment.

Before GenAI, we used to hear more about it, like, We need your prompt attention to it, prompt response matters, sometimes etc.; 

Usually, a prompt is something quick.

In the GenAI version, it's kind of a request we are submitting for getting a response from any LLM like GPT-4o, Gemini 2.o Flash, Sonnet 3.7, etc.

The prompt is a combination of instructions, context, input, and output format(examples as well).

When GPT was launched, it hallucinated more (even now it does, but there are ways to reduce it), and it's similar to a baby but with vast knowledge. 

If we give just a prompt, I need information about Apple. Based on its training data, it usually thinks of Apple as a company rather than hoping that it's a fruit and responds like that.

Like how humans were trained, I'm not sure about all of you till I remember there is only A for Apple and B for Ball :)

Back to the assignment, a good prompt should be readable and understandable by humans as well as by any LLM.

"Give me some test data for this app." It has some context, one instruction, which makes us hallucinate more than LLM, as new GenAI models are trained to respond to something they think might work for this.

Let us make it a better one—
Adding context:
As a functional tester, I'm looking to generate test data for an e-commerce website for the search field. This website sells only apparel, accessories, cosmetics, and footwear from different brands.

Sounds good? Then the context is clear, but we didn't provide any specific instructions: provide what the search field accepts, the length, whether the website is only for Indians (it matters as it'll impact the test data), etc.

This search field will accept only characters and numbers (to search any product ID), has of maximum length of 30, and the website is only for Indians.

I hope we are good with the prompt. Combine both and send it to LLM. You'll definitely get a good and definite response that's better than the first one.

Nowhere did we ask a question here or what exactly we are looking for; we will add that too—I'm looking for your help on generating 10 samples of test data for each fashion category above.

We are on track. We made LLM think and work like a junior you. We will add some touch-ups here.

I want this data to be created in table format for different fashion categories.

Here is the final prompt:
As a functional tester, I'm looking to generate test data for an e-commerce website for the search field. This website sells only apparel, accessories, cosmetics, and footwear from different brands.
This search field will accept only characters and numbers (to search any product ID), has a maximum length of 30, and the website is only for Indians.
I'm looking for your help in generating 10 samples of test data for apparel, accessories, cosmetics, and footwear.
I want this data to be created in 4 different table formats for different fashion categories. You can get started by analyzing this step by step.

Things I've considered here:
1. Didn't mention what not to generate; rather, focused on what only to generate
2. Tried to be specific about what I want rather than thinking that LLM could handle everything.
3. I have added "step-by-step" at last—it's a zero-shot chain-of-thought. Simple version of deep thinking—mostly for complex tasks, not for this though, but it's nice.
4. LLMs are different and non-deterministic. Even the prompt can add more columns and provide test data in one session and fewer columns in another, but the task can be accomplished.

5.Add the persona as a tester

That's all.

​@Dinesh_Gujarathi wow! this is by far the best explanation and a mini guide on how to create good prompts for specific and context dependent testing activities and task.

I would suggest you publish this as a LinkedIn article or on your own blog / GitHub page.

this is a good reference that you have created. wonderful work. see you tonight :)

​@Charmi07 ok, that explains it. Cool! Thanks for answering.

Hi ​@parwalrahul 

Regarding the given prompt - "Give me some test data for this app." 

I have listed a few issues with the given prompt - ​​​​​

1. Lack of Context: It doesn’t specify what "this app" is—its purpose, functionality, or domain (e.g., a banking app, game, or e-commerce platform).
2. Undefined Scope: It’s unclear what kind of testing the data is for (e.g., unit testing, UI testing, performance testing, or edge cases).
3. No Expected Output: It doesn’t indicate how the test data should be used or what outcomes are desired (e.g., validating inputs, triggering errors).
4. Missing Format: There’s no guidance on how the data should be presented (e.g., a list, table, JSON, or raw values).
5. Ambiguity in Quantity/Complexity: It doesn’t specify how much data is needed or whether it should include typical, boundary, or invalid cases.

Refined Prompt for Better Testing Data Generation

As a functional tester, I'm looking to generate test data or a mobile banking app’s login feature to support functional and security testing. I'm looking for your help to generating 5 valid username-password pairs (e.g., typical user credentials), 3 invalid pairs (e.g., wrong password, locked account, SQL injection attempt), and 2 edge cases (e.g., 50-character username, Unicode symbols). I want this data to  be in JSON object with 'valid', 'invalid', and 'edge' keys, where each entry lists the username, password, and a note explaining its test purpose. Ensure the data aligns with common security standards like OWASP recommendations. You can get started by analyzing this step by step.

Why This Works:

• Context: Specifies the app (mobile banking) and feature (login).
• Scope: Clarifies the testing type (functional testing).
• Expectations: Defines the types of data (valid, invalid, edge) and quantity (5, 3, 2).
• Usage: Implies the data will test login success, failure, and boundaries.
• Format: Requests JSON with categorized entries for clarity and usability

Key Takeaways

1. Be Specific – Always mention the app type, modules, and test scope.
2. Define Expected Output – Mention the format like JSON, CSV, or database entries.
3. Include Test Coverage – Specify valid, invalid, boundary, and edge cases.
4. Tailor for the Use Case – Differentiate between functional, API, performance, or security testing needs.

Prompt :Give me some test data for this app

Issues - This prompt is vague and does not clearly type of application, what  are likely input fields, any specific type of data needed, any specific data format required, quantity of test data , if data is required for specific type of testing like BVA, EP , etc.

Improved prompt  1:  Please provide test data to perform boundary value analysis tests on a bank login form

Improved prompt 2 :Please provide test data to perform boundary value analysis tests on a bank login form with account number or user name 9 char/digits and password with 10 chars. Also password needs to be a combination of letter, digits, symbols

In both cases , Chat GPT4.0 replied with quite good detailing about how BVA is done and with some assumptions on each fields provided data in excel format as well as JSON format, with valid and invalid data.

Hello   @parwalrahul  Rahul,

Honestly, I thought if I give just application name its able to give me details, I have tried act as tester. But not got much details.

So Identified Issues:

• No app description:  What does the app do?  What is its purpose?  What kind of data does it process?
• No data structure: How is the data organized?  What are the fields or columns?  What are their data types?
• No data constraints: Are there any limitations on the data?
• No desired format:  What format do you want the data in? 

Improved the Prompt

Give me some test data for my movie review app.  The app stores movie titles (string), user IDs (integer), user ratings (integer from 1 to 5), and review text (string, max length 500 characters).  I need about 50 data points in CSV format to test the functionality of the rating system and the review display.:

​@Bharat2609 really smart one. you invoked the chain of thought prompting at the end.

it’s like a power hack to get it to generate stuff on all the diverse details that you have asked it to work on.

atomic prompting / chain of thought prompting is really not well used. you have shared a good example.

cheers!

​@KajalS good one. you mapped test data to the test design technique. nice way to get targeted test data.

your approach is also a unique one. cheers!

​@sarika77 did you got good results with your improved prompt? 

did you noticed some benefits or did it still hallucinated?

Yes ​@parwalrahul  after improving prompts, it’s giving good results I got test data exactly what I want. 

Improved the Prompt

Give me some test data for movie review web application.  It stores movie titles (string), user IDs (integer), user ratings (integer from 1 to 5), and review text (string, max length 500 characters).  I need about 50 data points in CSV format to test the functionality of the rating system and the review display.

Movie Title,User ID,User Rating,Review Text
Inception,101,5,An absolutely mind-blowing movie with incredible visuals and a complex, thought-provoking plot. Nolan at his best!
Titanic,102,4,A timeless love story set against the tragic sinking of the Titanic. Excellent performances and cinematography.
The Dark Knight,103,5,Heath Ledger's Joker steals the show in this dark and thrilling superhero masterpiece.
Avengers: Endgame,104,5,A satisfying conclusion to an epic saga. Emotional, action-packed, and visually stunning.
The Godfather,105,5,A masterclass in filmmaking. Perfectly executed in every way – acting, direction, and score.
The Shawshank Redemption,106,5,A deeply moving story about hope, friendship, and resilience. Tim Robbins and Morgan Freeman are outstanding.
Pulp Fiction,107,4,Quentin Tarantino's signature style shines through in this unpredictable, witty, and violent crime film.
Forrest Gump,108,5,A heartwarming tale that blends humor, history, and drama. Tom Hanks delivers a brilliant performance.
The Matrix,109,5,A groundbreaking sci-fi film with revolutionary visual effects and a captivating story.
Interstellar,110,4,An ambitious space odyssey exploring love, time, and survival. Visually mesmerizing but a bit dense at times.
Fight Club,111,5,A dark, satirical, and subversive film that challenges consumer culture. Truly unforgettable.
Jurassic Park,112,5,Steven Spielberg's thrilling adventure with lifelike dinosaurs and groundbreaking special effects.
The Lion King,113,4,A beautifully animated Disney classic with memorable songs and a timeless story.
Star Wars: A New Hope,114,5,The start of an iconic space saga. A landmark in sci-fi and film history.
The Lord of the Rings: The Fellowship of the Ring,115,5,A stunning adaptation of Tolkien's epic tale. Captivating from start to finish.
The Silence of the Lambs,116,5,A chilling psychological thriller with unforgettable performances from Anthony Hopkins and Jodie Foster.
Schindler's List,117,5,A harrowing and deeply impactful film about the Holocaust. A true masterpiece.
Goodfellas,118,5,Martin Scorsese's gripping tale of the rise and fall of a gangster. Powerful and brilliantly acted.
Saving Private Ryan,119,5,An intense, realistic portrayal of World War II. The opening D-Day scene is unforgettable.
The Sixth Sense,120,5,A suspenseful and emotional thriller with a shocking twist ending that will leave you speechless.
Gladiator,121,4,A gripping historical drama with a commanding performance by Russell Crowe.
The Prestige,122,5,An intricate, mind-bending tale of obsession, deception, and magic. Christopher Nolan excels again.
The Departed,123,5,An intense and unpredictable crime thriller with a stellar cast and gripping plot.
La La Land,124,4,A visually stunning and charming modern musical with captivating performances and music.
Whiplash,125,5,A tense and exhilarating exploration of ambition, obsession, and sacrifice. J.K. Simmons is phenomenal.
Blade Runner 2049,126,4,Visually stunning and thought-provoking, though slower-paced than its predecessor.
Black Panther,127,5,A culturally significant superhero film with great action and a compelling story.
Spider-Man: Into the Spider-Verse,128,5,A visually dazzling and creatively told animated Spider-Man film.
Coco,129,5,A beautifully animated and emotionally resonant film celebrating family and Mexican culture.
Inside Out,130,5,A clever and heartfelt exploration of emotions, beautifully animated by Pixar.
Frozen,131,4,A magical and visually stunning Disney hit with catchy songs and a heartfelt message.
Zootopia,132,5,An entertaining and socially relevant animated film with humor, heart, and a clever mystery.
Up,133,5,The opening sequence alone makes this Pixar classic unforgettable. Full of heart and adventure.
The Incredibles,134,4,A fun, action-packed superhero family film with great humor and animation.
Toy Story,135,5,The film that started it all for Pixar. Funny, heartwarming, and groundbreaking.
Finding Nemo,136,5,A visually stunning underwater adventure with humor, heart, and unforgettable characters.
WALL-E,137,5,A visually stunning and touching story about love, loneliness, and the environment.
The Grand Budapest Hotel,138,5,Wes Anderson's quirky and visually rich film, full of humor, heart, and eccentric characters.
The Social Network,139,5,An engrossing look at the creation of Facebook and the personal drama behind it.
The Wolf of Wall Street,140,5,An outrageous and darkly funny tale of greed, excess, and corruption.
Mad Max: Fury Road,141,5,A relentless, high-octane thrill ride with incredible practical effects.
Get Out,142,5,A brilliantly crafted and socially conscious thriller. Jordan Peele's debut is a knockout.
Parasite,143,5,A darkly comedic and gripping social satire with unexpected twists and masterful direction.
The Irishman,144,4,A contemplative and richly detailed gangster epic. Long, but rewarding.
Knives Out,145,5,A cleverly plotted and immensely entertaining modern whodunit with a stellar cast.
A Quiet Place,146,5,A tense, atmospheric horror film with minimal dialogue but maximum impact.
Her,147,4,A unique and poignant love story exploring technology, loneliness, and connection.
Moonlight,148,5,A beautifully crafted and emotionally resonant coming-of-age story.
The Revenant,149,5,A brutal, visually stunning survival tale with an intense performance by Leonardo DiCaprio.
Dunkirk,150,5,A tense and immersive World War II thriller with minimal dialogue and stunning visuals.
 

​@sarika77 cool and wonderful!

​@parwalrahul 

Thankyou

Given Prompt for Review: “ Give me some test data for this app.”


Issues identified with the Prompt:

Lack of Context: The prompt lacks clarity about the context, the type
of data required, and the app's functionality.
Vague Requirements: The requirement is vague; the only clear aspect is
that data is needed.
Unclear Testing Objective: The prompt does not specify what type of
testing the data is for, such as Functional, Security, or Performance
testing.
Unspecified Persona: It is unclear which user persona or role the data
is intended for or whether data is required for all personas.
Ineffective for AI: AI may not generate the expected dataset
accurately due to the generic nature of the prompt.

Prompts I came up with:

1. Provide test data for a Sales user when a Customer is logged into
Amazon and using Gift Vouchers.
2. Generate functional test data for an Amazon customer using Gift
Vouchers to complete a purchase.
 

ChatGPT Response: 

Here’s a sample test data for a Sales User when a customer is logged into Amazon and using Gift Vouchers:

Test Data for Sales User - Amazon Gift Vouchers

The refined AI Prompt impacted my response with below benefits:

• Increased specificity: Defined user roles and actions.
• Provided relevant data context: Specified the scenario.
• Clarified platform/application: Identified "Amazon" as the context.
• Specified functionality (Gift Vouchers): Narrowed the scope of data.
• Improved data relevance: Focused on the needed data.
• Enhanced data reliability: Led to more consistent outputs.
• Reduced AI bias: Avoided vague terms, preventing skewed results.

1. The given prompt is not clear because it doesn’t mention the type of app, what kind of test data is needed, or the format of the output.

2. Improved prompt: "Generate test data for a mobile banking app login. Include valid and invalid cases like correct login, wrong password, and locked account. Provide the data in a table format with input values and expected results."

3. This makes the prompt more specific and helps get useful test data.

4. A detailed prompt gives better and more accurate results for testing.

​@Kusumketu : giving user persona is always a great hack for extracting relevant outputs. I am glad that you did that.

All the best!

​@japankansara good one. clear mention of your specific cases and scenarios helps llms generate specific and curated answers. cheers!

Step 1 – Reviewing the Given Prompt
The original prompt is:
"Give me some test data for this app."

Step 2 – Identifying Issues
This prompt is too vague and lacks important details. Here’s why it might not yield useful results:

• No app details: The AI doesn’t know what kind of app it is (e-commerce, banking, social media, etc.).

• No testing scope: It doesn’t specify whether the test data is for functional testing, performance testing, security testing, or something else.

• No expected data format: Should the data be in JSON, CSV, or plain text?

• No specific scenarios: It doesn’t mention if the data is for valid cases, invalid inputs, edge cases, or boundary testing.

• No indication of volume: Does the tester need just a few data samples or a large dataset for stress testing?

Because of these issues, the AI might generate random or generic data that isn’t useful for the actual testing needs.

Step 3 – Improving the Prompt
To make the request more precise, I refined the prompt as follows:

"Generate test data for an e-commerce web application to test the checkout process. The data should include valid and invalid customer details, different payment methods, and various shipping addresses. Provide the output in a structured JSON format with 10 unique test cases."

Step 4 – Reasoning Behind the Refinement

• Added app details: Specified that it’s an e-commerce web application.

• Defined the testing scope: Focused on the checkout process.

• Included different test scenarios: Asked for valid and invalid customer details, payment methods, and shipping addresses.

• Specified output format: Requested JSON format to ensure structured data.

• Defined quantity: Mentioned 10 unique test cases to ensure enough variety.

Step 5 – Final Reflection
This refined prompt is much clearer and more actionable. AI will now provide meaningful, structured test data that aligns with the actual testing needs. By specifying details like app type, testing focus, format, and data variety, testers can get more relevant and usable results from AI.

this is the piece of the puzzle that makes your response really clear.

everyone should focus on these factors.

• Added app details: Specified that it’s an e-commerce web application.

• Defined the testing scope: Focused on the checkout process.

• Included different test scenarios: Asked for valid and invalid customer details, payment methods, and shipping addresses.

• Specified output format: Requested JSON format to ensure structured data.

• Defined quantity: Mentioned 10 unique test cases to ensure enough variety.

Exercise 3:

Steps:

Review the Given Prompt: Here’s a poor prompt for a testing task:

"Give me some test data for this app."
 
Identify Issues: Consider why this prompt might not yield useful results. Think about missing details like the type of app, testing scope, expected tests, usage of the output produced, output format, or any specific testing techniques.
 
Improve the Prompt: Rewrite the prompt to make it more precise and useful.

Refined Prompt: 

"As a functional tester for financial application banking app. Please provide me some positive and negative scenarios and test data to check whether application accept correct value. Please provide me data in .csv and and xls format which will later I will use for automation testing scripts."

​@parwalrahul  I missed the deadline for this assignment and am completing it now. My apologies.

On providing this prompt to ChatGPT -” Give me some test data for this app”,I received the following response: ‘Could you specify the app's functionality or what kind of test data you need? For example, do you need user login credentials, transaction data, form inputs, or something else? Let me know, and I'll generate relevant test data for you!

The prompt is vague and does not specify the requirement clearly. In other cases, this may also result in getting irrelevant test data.

I rewrote the prompt :

Prompt: Give me test data with the country name starting with C followed by the capital city in uppercase and currency with the symbol

Output : Received following output .

Learning: You simply need to tell AI exactly what you want, and it will provide you with precise results.

1.

​@parwalrahul ​@Mustafa 

Hello Rahul,

Here is my response below 🙂

1.Original Prompt (Not Ideal):

"Give me some test data for this app."

 2. What’s wrong with This Prompt?

This prompt is far too vague to be useful. Here are the main issues:

• Which app are we talking about? There’s no context or details about the application or what it does.
• What kind of testing is this for? It’s unclear whether it’s functional testing, regression, performance, or something else entirely.
• What’s the purpose of the test data? Is it meant for manual testing, automation, edge case analysis?
• What format should the data be in? Should it be a CSV file, JSON, or just plain text?
• Are there any data boundaries to consider? Should it include valid and invalid inputs, boundary cases, or security-related scenarios?

Without these details, it’s really difficult to produce meaningful or usable test data.

3. Improved Prompt:

Please generate test data for the login module of a mobile banking application. Users need to enter a valid email address and password to log in. I need at least 5 valid and 5 invalid input combinations, including cases like empty fields, wrongly formatted emails, SQL injection attempts, and overly long inputs. The output should be in a table format with the following columns: Email, Password, and Expected Result (e.g., ‘Login Successful’ or a specific error message). This data will be used for both manual testing and automation scripts.

Thanks,

Ramanan

​@NitinMore : nice. thanks for doing the exercise.

​@Yastho no worries. thanks for taking part in the exercise.

​@parwalrahul  week 3 exercise

Regarding the given prompt - "Give me some test data for this app." 

I have listed a few issues with the given prompt - ​​​​​

1. Lack of Context: It doesn’t specify what "this app" is—its purpose, functionality, or domain (e.g., a banking app, game, or e-commerce platform).
2. Undefined Scope: It’s unclear what kind of testing the data is for (e.g UI testing, performance testing, or edge cases).
3. No Expected Output: It doesn’t indicate how the test data should be used or what outcomes are desired (e.g., validating inputs, triggering errors).
4. Missing Format: There’s no guidance on how the data should be presented (e.g., a list, table, JSON, or raw values).

Improved prompt: 

Develop test data for a web-based ecommerce app that includes user registration, product search, and checkout features. The test cases should consider all relevant scenarios, including typical user inputs, erroneous inputs, and edge cases. The data should be delivered in a CSV file with clearly labeled columns 

Nice ​@DamiS