Week 2 Exercise - Head-to-Head: Evaluating AI Models

Hello ​@parwalrahul ,

Both models are strong in their own ways. ChatGPT 4.0 excels at generating well-structured, in-depth responses that feel natural and insightful. Gemini, on the other hand, focuses more on straightforward, factual delivery. If the task requires detailed analysis or a human-like tone, ChatGPT 4.0 is the better choice. If brevity and direct accuracy are the priority, Gemini performs well.

Ultimately, the best model depends on the specific use case.

Thanks,

Ramanan 

Hello

ChatGPT 4.0 and Gemini make a powerful impression but have different strengths. 
ChatGPT 4.0 offers strong text-based capabilities and delivers creative results. Sometimes somewhat biased answers. You have to be able to read and evaluate the answers correctly. Provides answers even when ChatGPT doesn't know - creative.
Gemini has a multimodal design and the integration with Google services provides a versatile and comprehensive user experience. 
I think the choice between the two models depends on the specific user needs, such as the need for multimodal processing, integration with existing tools or the focus on unbiased answers.

Frank

​@Ramanan nice takeaway.. my own takeaway is pretty similar to that.

One clear area where gemini excels is image recognition (OCR), especially if we go into native (local) languages… Google really has some edge there.

​@Frank Kokoska yeah, for people in the google ecosystem or services, Gemini could really stand out.

I feel similar possibilities will become a reality with copilot (backed by ChatGPT). already Microsoft is integrating it with office applications.

Interesting times ahead 🤞

Hello,

So both Chat GPT 4.0 and Gemini are good and powerful in their own ways. However I found Gemini provides more precise and detailed information when it comes to technical details on application, tools, languages.

Can we get the recordings or minutes of meeting for  week2 session?

​@parwalrahul 

I used a prompt to generate a test strategy document from both ChatGPT4.0 and the Gemini model and obtained some results from it.

ChatGPT 4.0 and Gemini make a powerful impression but have different strengths. 

1.ChatGPT 4.0  provides a more detailed, structured, and technical approach to the test strategy document, making it ideal for in-depth understanding. In contrast, Gemini offers a more narrative and engaging overview, which may be better for stakeholders seeking a broader understanding without delving into technical specifics. 

2.ChatGPT: Presents a well-organized structure with distinct sections and subsections, making it easy to follow and navigate. It offers a methodical layout that covers all key aspects of the testing strategy.Gemini: While organized, it follows a more narrative style. The sections are present but not as clearly separated, resulting in a more fluid but less formal structure.

3.ChatGPT: Uses formal and technical language suited for a professional audience. The tone is clear, direct, and focused on delivering precise information. Gemini: Adopts a conversational and engaging tone, which may appeal to a broader audience but could lack the technical depth expected in formal documentation.

>>>I believe the choice between the two models depends on the user's specific needs, such as the requirement for multimodal processing, compatibility with existing tools, or a focus on providing unbiased answers.

Attachment- testing document

​@KajalS “technical details on application, tools, languages.”

what specific differences did you noticed? interested to know more about this.

Sure Rahul. One of my question was around how can I integrate API tests written in postman with a pipeline on circleCI.

Gemini responded with almost similar steps but along with few examples with detailing like -

1. .circleci/config.yml  looks like and what each keyword specifies with in the file (for e.g. -e environment.json,  --reporter cli,juni).
2. Tips like - Important Security Considerations:
3. Format of environment.json file to store environment variables.

My question was also around Cypress tool, how to start with it.  It replied with basic information like how to install it, key cypress commands, assertions and examples.

Where as Chat GPT provides more theoretical answers(unless specified more accurately) like key features, advantages, etc.

Both ChatGPT and Gemini are powerful AI tools that provide in-depth insights, though they take different approaches.

• ChatGPT offers a well-rounded perspective by highlighting free AI tools for accessibility testing, detailing how each tool targets specific functional areas. It also outlines key components of accessibility testing, covering both manual and automated approaches.

• Gemini focuses on the role of AI in accessibility testing, explaining how AI is integrated into the process. It provides a detailed analysis of the advantages and disadvantages of using AI for accessibility testing.

Conclusion:
Both models deliver comprehensive information, but the choice depends on specific needs. ChatGPT is ideal for those looking for practical tools and a balanced approach, while Gemini is better suited for exploring AI-driven accessibility testing in depth.

​@Bharat2609 nice try.

Multimodal processing is indeed an amazing possibility and would be the thing that would get more prominence.

We might be finetuning the responses from public LLMs through our custom models and vice versa.

thanks for sharing this possibility. Have a nice day!

​@KajalS nice observation. thanks for the details.

even I have a feeling that gemini is relatively less used compared to the quality it offers.

it is at part with GPT models and sometimes even better in specific tech tasks.

Let’s see where this race of models will end. as of now, there is no clear winner.

thanks for sharing your response and observations, ​@Darshana :)

GPT-4-mini provides a superior & clear structure

Gemini Flash 1.5 offers valuable context but lacks the precision

nice comparison and to the point. great job, ​@satishracherla 

Hello ​@parwalrahul 

I have tried to use two examples in the attached document to create a user story with identical prompt across both the LLM's Chat GPT and Gemini

Comparison Summary:

Response Quality:

1. Structure:

   • ChatGPT: Provides a more traditional Jira ticket structure with clear sections.
   • Gemini: Offers a more detailed and comprehensive Jira ticket format.

2. Clarity:

   • ChatGPT: Concise and straightforward, but lacks some detail.
   • Gemini: More thorough and descriptive, providing clearer context.

3. Detail Level:

   • ChatGPT: Offers basic information, somewhat generic.
   • Gemini: Provides more specific details and scenarios.

4. BDD Format:

   • ChatGPT: Does not strictly adhere to BDD format in acceptance criteria.
   • Gemini: Correctly uses Given-When-Then format for scenarios.

Accuracy:

1. User Story:

   • ChatGPT: Presents a basic user story format.
   • Gemini: Provides a more comprehensive user story with clear benefit.

2. Acceptance Criteria:

   • ChatGPT: Lists criteria but doesn't follow BDD format.
   • Gemini: Correctly uses BDD format with specific scenarios.

3. Definition of Ready (DoR):

   • ChatGPT: Includes relevant points but is somewhat generic.
   • Gemini: More specific to the task, includes technical considerations.

4. Definition of Done (DoD):

   • ChatGPT: Covers essential points but lacks some technical specifics.
   • Gemini: More comprehensive, includes testing and deployment steps.

5. Additional Elements:

   • ChatGPT: Includes priority and labels.
   • Gemini: Adds project name, issue type, assignee, and epic link.

Unique Features:

• ChatGPT: Includes a section for user feedback mechanisms.
• Gemini: Provides negative case scenarios and more technical details in DoD.

Overall Evaluation:

While both outputs provide valuable information, Gemini's response appears to better meet the requirements of the prompt. It offers a more structured, detailed, and technically accurate representation of a Jira ticket for a user story. The use of proper BDD format in the acceptance criteria and the inclusion of both positive and negative scenarios demonstrate a more comprehensive understanding of the task.

ChatGPT's response, while clear and concise, lacks some of the technical depth and specificity that would be expected in a real-world Jira ticket for this type of task.

In terms of accuracy and adherence to best practices in Agile development and Jira ticket creation, Gemini's output is superior in this instance.

​@ameet213 true. even i have felt this.

your observations also matches the observations by a lot of other members who attended this exercise.

Also, with gemini, I have noticed that it has some guards or mechanism to stop answering if user puts in excessive debug logs or stack trace and asks for more information.

this was an interesting observation that I had. rest, it’s similar to yours. thanks :)

Hi Rahul,

Couple of things I want to highlight:

AICamp is definitely worth the try and its going to be the camp I can refer to check different models or to save the prompts from different models at a place. 

1. Utilized GPT 4o-mini and Gemini - Both are good.
2. Created an assistant but it can do better still there is a lot of improvement and it’s from the instructions from user side and also the template which assistant is following.
3. We can share the workspace with any one - this feature people won’t expect.
4. These models were not that powerful when compared to accessing them in their own websites like I asked it to generate an image but it doesn’t(both models) - Text generation is what these models are for. And we can also chat with a document.

Here are my comparisons with an assistant:

1. Clarity: Created some test cases by using GPT 4o(openAI), 1.5 pro(Gemini) but the response provided from Gemini is good when compared to GPT 4o
2. Correctness: Can’t comment on this both are not connected to the internet as I can see their trained data is limited to OCT 2023 but till this point, everything is ok.
3. Consistency: In multiple test runs with the same prompt, Gemini provided more consistent and predictable responses compared to GPT 4o-mini. GPT 4o-mini exhibited greater variability in its outputs, sometimes deviating significantly from previous responses.

4. 

    

Found one thing interesting while testing it: That's great! I'm an AI assistant, created by AICamp.

So I believe AICamp is having it’s own template and using the API’s from OpenAI and Google.

If you find the pros of unified access and prompt management within AICamp more interesting, it's the platform for you. If the limitations (cons) outweigh the benefits, then the respective GenAI websites are the best to use.

Here we go on Week 2 assignment:

 

Comparison and Analysis: I gave Prompt to write Test Case for Air Ticket Booking System

GPT-4o-mini: Detailed Structured Test Case

Strengths:

Provides a clear and detailed structure using a step-by-step approach.

Includes well-defined sections like Test Case ID, Description, Preconditions, Test Data, Steps, and Expected Results.

Ensures traceability and ease of execution for testers.

Suitable for manual testing and easy to maintain.

Limitations:

Limited coverage compared to Gemini; it focuses on one scenario.

Lacks broader test coverage including negative and security test cases.

Not scalable for complex systems with multiple scenarios.
 

Gemini: Categorized Test Cases

Strengths:

Offers a comprehensive test coverage across functional, negative, usability, and security aspects.

Organized by categories, making it easy to prioritize and manage testing efforts.

Ideal for large-scale applications where multiple components are involved.

Encourages collaboration with QA, Dev, and Product teams.

Limitations:

Test steps are not elaborated, which may require additional context for testers.

Requires separate detailed steps and data for each test case.

May introduce inconsistencies if not standardized.

 

Response Quality Comparison between GPT-4o-mini and Gemini

GPT-4o-mini provides a clear and detailed test scenario with actionable steps. It’s ideal for step-by-step validation.

Gemini offers a holistic view, ensuring better test coverage. It’s suitable for teams aiming to cover edge cases and ensure overall system stability.

Accuracy Analysis

GPT-4o-mini is more accurate for functional validation of a specific scenario.

Gemini is more accurate for identifying defects in various parts of the system, including performance, usability, and security.

Final Reflection

For simple scenarios or when precision is needed, GPT-4o-mini is more appropriate.

For complex systems requiring end-to-end coverage, Gemini is recommended as it ensures a broader and more thorough testing approach.

A hybrid approach, using GPT-4o-mini for critical test scenarios and Gemini for comprehensive coverage, would offer the most effective results.

** Attached testcase docs for reference

My prompt to ChatGPT and Gemini was, "Teach me Playwright." I noticed that Gemini focused more on the theoretical aspects, while ChatGPT provided results that were more technically oriented. One interesting thing I observed about ChatGPT, which I didn't see in Gemini, was that ChatGPT's responses included prompts for me to answer at the end. It felt like it was trying to interact with you.

​@Dinesh_Gujarathi thanks for your detailed submission brother :)

you really did cover the exercise from all the aspects including setting up stuff via AICamp.

Because you tried that, I can see how you could instantly realize it’s potential and amazing capabilities.

I use it regularly and I know of the image generation bug too. they are working on it :D (bug reported by me already).

I have also setup a couple of assistants via it and I am excited to expand them more as the knowledge feature for it gets enabled.

See you in the event today. have a nice evening!

​@Kusumketu loved your experimentation and insight.

For simple scenarios or when precision is needed, GPT-4o-mini is more appropriate.

For complex systems requiring end-to-end coverage, Gemini is recommended as it ensures a broader and more thorough testing approach.

A hybrid approach, using GPT-4o-mini for critical test scenarios and Gemini for comprehensive coverage, would offer the most effective results.

this nutshell summary after experimentations is something that we all would have to make and reach to at some point.

models are slowly becoming commodities. 
 

Evaluating models is going to become something similar to evaluating libraries (eg. selenium vs playwright).

Experimentation with an open mind is what will help us all :)

thanks for doing it and submitting your strong response :)

​@Yastho : did you try via AI Camp or native chatgpt?

It may also be due to the system prompt being injected by aicamp.

Hello ​@parwalrahul 
 

Key Takeaways from ChatGPT 4.0 vs. Gemini Evaluation for QA Tasks

1. Response Quality & Accuracy

   • ChatGPT 4.0 provided detailed and structured responses, often including examples and best practices.
   • Gemini demonstrated strong contextual awareness, particularly excelling in CI/CD and automation-related discussions.

2. Completeness & Clarity

   • ChatGPT 4.0's responses were more comprehensive and well-organized, making them easier to use for test documentation.
   • Gemini occasionally provided concise answers, which were useful for quick insights but sometimes lacked depth.

3. Adaptability & Reasoning

   • ChatGPT 4.0 adapted well to variations in prompt wording and provided logical reasoning for test case creation and bug analysis.
   • Gemini was effective in breaking down complex requirements but sometimes lacked detailed step-by-step explanations.

4. Test Automation & Code Review

   • ChatGPT 4.0 generated clearer test cases and test data, making it better suited for structured QA workflows.
   • Gemini performed well in CI/CD pipeline discussions and automation strategies, making it useful for DevOps-oriented QA teams.

5. Integration & Usability

   • ChatGPT 4.0 is better suited for structured QA tasks like test planning, requirement analysis, and detailed documentation.
   • Gemini is stronger in AI-assisted debugging, automation insights, and CI/CD optimization.
      

Final Conclusion :

ChatGPT 4.0 is preferable for structured test case creation, requirement analysis, and bug reporting. Gemini is beneficial for DevOps-focused teams needing quick insights on CI/CD and automation strategies

Example my prompt given was: I am a tester, Without coding knowledge how to learn AI agents in a simple way. Suggest an easy way

Final Reflection: Key Takeaways

Recommendation:
 

• Use ChatGPT’s approach for foundational, broad learning about AI agents without code, suitable if you seek comprehensive conceptual grounding.
• Use Gemini Pro 1.5’s approach if your primary goal is learning AI specifically from a testing perspective, emphasizing practical testing scenarios and hands-on application tailored explicitly for testers.

ChatGPT 4.0 and Gemini both make a powerful impression but have different strengths.

• ChatGPT 4.0 excels in text-based capabilities, delivering structured and creative results. However, it may sometimes provide biased responses, so users need to critically evaluate its answers. It is also capable of generating responses even when it lacks complete knowledge, making it highly creative.

• Gemini, with its multimodal design and integration with Google services, offers a more versatile and comprehensive user experience.

The choice between these two models depends on specific user needs, such as multimodal processing, integration with existing tools, or a focus on unbiased responses.

Comparison Between ChatGPT 4.0 and Gemini

1. Depth and Approach

   • ChatGPT 4.0 provides a more detailed, structured, and technical approach, making it ideal for in-depth understanding, such as in test strategy documents.
   • Gemini offers a more narrative and engaging overview, which may be better suited for stakeholders who need a broader understanding without diving into technical specifics.

2. Structure and Organization

   • ChatGPT 4.0 presents a well-organized structure with distinct sections and subsections, making it easy to navigate. It follows a methodical layout that covers all key aspects comprehensively.
   • Gemini, while organized, follows a more fluid and narrative style. Although the sections are present, they are not as distinctly separated, resulting in a less formal but more engaging structure.

3. Tone and Language

   • ChatGPT 4.0 uses formal and technical language, making it suitable for a professional audience. Its tone is clear, direct, and focused on delivering precise information.
   • Gemini adopts a more conversational and engaging tone, appealing to a broader audience but potentially lacking the technical depth required for formal documentation.