What kind of applications do you think companies are building with the power of AI and how do you think you can test them?

Companies are building AI-powered apps like chat bots, recommendation systems, predictive analytics, autonomous vehicles, and medical diagnostics.

Test with real-world data, edge cases, performance metrics, user feedback, and ethical audits to ensure accuracy and reliability.

Companies are building AI apps like emotion-aware chatbots, predictive engines, and adaptive supply chains.

Testing involves handling unpredictable behavior through real-time data and edge cases.

Key focus areas: bias detection, explainability, and ethical responses.

Goal: Ensure AI is not just intelligent, but also fair, safe, and reliable.

Note: AI testing isn’t about confirming performance—it's about discovering the unknown. The goal is to ensure AI behaves like a wise apprentice: smart, reliable, accountable—and always learning from its mistakes.

Answer Karthik KK’s Question for a chance to win a ShiftSync Giftbox

 

 

At my company, we're building crop monitoring and seed recommendation systems that help farmers make data-driven decisions about crop selection and field management.

How I test our crop monitoring and seed recommendation systems:

Domain Understanding: I collaborate with our agricultural team and partner farmers to understand soil science, weather patterns, and regional growing conditions, which guides my testing approach.

Automated Testing: We use Tricentis Testim to automate key user flows - farmer onboarding, data input validation, recommendation generation, and dashboard interactions. This ensures our core workflows remain stable as we iterate on our AI models.

AI-Specific Testing: I validate our recommendation engine with historical crop data, test various soil and weather scenarios, and ensure our system handles missing sensor data gracefully.

Real-World Validation: We partner with pilot farms to test recommendations in actual growing conditions, tracking performance against traditional farming methods over complete seasons.

Continuous Monitoring: I monitor recommendation accuracy, farmer adoption rates, and actual crop yield outcomes in production to ensure our AI genuinely improves farming results.

Safety Testing: I ensure our system never recommends crops that could fail catastrophically or damage soil health, with proper fallbacks when data is incomplete.

Success is measured not just by technical accuracy, but by whether we're actually helping farmers improve their harvests and livelihoods

Picture this: You order coffee through an app, and AI predicts your usual order before you even think about it. Netflix knows you'll binge-watch that new series before you do. Your bank flags suspicious transactions faster than you can say "fraud". These aren't science fiction anymore—they're Monday morning reality.

Companies are going all-in on conversational AI (those chatbots that actually understand you), recommendation engines that feel like mind-readers, and automated decision systems that process loans, resumes, and insurance claims while you sleep.
 

Traditional testing is like "input A, expect output B." But AI? It's more like "input A, get output B, C, or maybe something completely unexpected that's still somehow correct."

Smart Testing Strategies:

• Behavioral testing instead of exact matching—does the chatbot stay helpful even when users get creative with their questions?
• Bias hunting—making sure your AI doesn't accidentally become prejudiced
• Load testing with a twist—AI models can be computationally hungry beasts
• Good old UI testing—Playwright still rocks for testing how users interact with AI features

The Bottom Line

Testing AI applications is like being a detective, data scientist, and quality guardian all rolled into one. You're not just checking if it works—you're ensuring it works fairly, consistently, and doesn't go rogue at 3 AM.

AI apps companies are building:
     Generative AI Applications
     Predictive Analytics
     Computer Vision Applications
     Natural Language Processing (NLP)
     AI in Automation and RPA

How to Test AI Applications:
     Data Testing
     Model Testing
     Functional Testing
     Self-healing and Resilience Testing

AI is being used almost everywhere now with the intention of providing better personalised experiences to the users with minimal human intervention, as quickly as possible, with more accurate information or the action that the user intends to do.

Testing AI applications is in no way similar to testing traditional UI or APIs using ’n’ number of automation tools or manual testing.

Testing AI applications needs validation and verification of a large number of parameters, functional testing, non-functional testing, data accuracy, context, and so on. Hence, as you mentioned, we need to use LLM as a judge to do the verification part while still using our human intelligence (which obviously trained the AI models 😄) to think and come up with highly efficient test cases, which are like edge cases in today’s non-AI testing world. 

Answer Karthik KK’s Question for a chance to win a ShiftSync Giftbox

 

 

Companies are building all sorts of cool stuff with AI right now. Think chatbots that can actually hold smart conversations, tools that recommend products or movies like they know you personally, and apps that can summarize documents or extract info from invoices without anyone lifting a finger.

Then there’s AI in coding — tools that suggest or even write code for you, plus image recognition for things like defect detection in factories or even diagnosing medical scans. It’s everywhere.

Now, testing these kinds of apps? That’s a different game compared to traditional testing.

You’re not just checking if A leads to B — because AI outputs aren’t always the same. So you look at how accurate or useful the results are, not just if they "work." You test the prompts, the edge cases, check for bias, and use metrics like BLEU or ROUGE to measure output quality. Some people even use one AI to judge another (yep — LLM as a judge!).

It’s less about pass/fail and more about “is this good enough, safe, and consistent?” You also want to make sure it doesn't go rogue or give biased answers.

Companies across industries are using AI to build applications that can automate, predict, understand, and interact in human-like ways. Here are some popular types:

1. Customer Support & Chatbots

• AI-powered virtual assistants (e.g., in banking, e-commerce, airlines)
• Chatbots that understand user intent and provide quick responses

2. Recommendation Engines

• Suggesting products, videos, or content (used by Amazon, Netflix, Spotify)
   

3. Fraud Detection & Risk Analysis

• AI models monitor transactions for suspicious behavior (used in finance, insurance)
   

4. Image & Speech Recognition

• Used in healthcare (scanning X-rays), security (face recognition), or voice assistants (Alexa, Siri)
   

5. Predictive Analytics

• Forecasting customer behavior, stock trends, supply chain needs
   

6. Personalized Experiences

• AI adjusts the UI or marketing content based on user profiles and preferences
   

7. Process Automation (RPA + AI)

AI bots that handle emails, data entry, and approvals in HR, finance, or customer service

At my company, we’re building a next-generation Learning Management System (LMS) and blended academy platform that offers both live instructor-led courses and on-demand recorded content.

AI-Specific Testing:
Currently we are validating our recommendation engine using anonymized historical learner data, simulate various student personas, and systematically test “edge” cases such as low engagement or unusual learning paths. We rigorously check that course and path recommendations adapt appropriately, and that auto-generated feedback is helpful, fair, and bias-free.

Task in pipeline:

Shadow testing in production: silently monitoring real users to spot unseen patterns, drift, and even emotional cues—then letting AI itself flag anything ‘weird’ that humans might miss.

Counterfactuals: to test AI’s reasoning, not just results.

Answer Karthik KK’s Question for a chance to win a ShiftSync Giftbox

 

 

​@executeautomation Wow nice challenge!!

Here is my response,

With the power of AI, companies are creating applications that are not only smart but adaptive, personalized, and predictive reshaping industries in the process.

Intelligent Automation Tools:

From self-healing IT systems to robotic process automation, AI tools are eliminating repetitive tasks and driving efficiency.

Predictive & Personalized Experiences:

E-commerce and OTT platforms leverage AI for hyper-personalized recommendations, dynamic pricing, and predicting user behavior.

Conversational AI & Chatbots:

Virtual assistants and intelligent chatbots are redefining customer engagement by offering 24/7, human-like support.

Generative AI Applications:

Companies are accelerating creativity with tools for content generation, design, and even AI-assisted coding.

AI-Powered Analytics & Decision Support:

Industries like healthcare, finance, and logistics use AI to detect anomalies, deliver real-time insights, and support strategic decisions.

Testing these applications requires a shift from traditional methods to intelligent, data-driven testing approaches:

Data Validation & Bias Testing: Ensure training and inference data is clean, diverse, and free from bias.

Functional & Accuracy Testing: Validate model outputs across real-world and edge-case scenarios for consistent reliability.

Performance & Scalability Testing: Measure how the system handles massive data volumes, concurrent users, and low-latency requirements.

Explainability & Ethical Testing: Confirm that AI decisions are transparent, explainable, and aligned with ethical standards.

Continuous Learning Validation: As AI models evolve, regression testing with versioned datasets ensures stable and trustworthy performance.

In essence, AI applications are transforming the way businesses operate, and as testers, our role is no longer just to ask “Does it work?” but “Is it fair, reliable, and intelligent?”

By combining domain expertise, automation, and AI-driven testing techniques, we can guarantee that these solutions deliver trustworthy, high-impact user experience

Thanks,

Ramanan

Question: What kind of applications do you think companies are building with the power of AI and how do you think you can test them?

Here is my observation ​@executeautomation  ​@Mustafa 

After years of testing traditional applications and now diving deep into AI/LLM testing, I'm seeing fascinating patterns in how companies are leveraging AI - and the unique testing challenges that follow.

1.Intelligent Customer Support-

-Chatbots that understand context, sentiment, and complex queries

-Virtual assistants that handle multi-step conversations

-Email response generators that maintain brand voice

--Apart from that I have already started to creating chatbot, I ‘,m using mistral llm model,sqllite database, FAISS database (vector database) , working totally on RAG concept (still more in POC)

2.Code Generation & Development

-Automated code completion and bug fixing

-Documentation generators

-Test script creation from requirements

---I have already created this kind of agent and what tech stack I am using like:

Microsoft Autogen -ai agent for automating complex tasks

Streamlit-  A framework for building interactive web application in python

python - programming language

pandas- for data manipulation

Pydantic ai -data extraction

fpdf - generating pdf report

simplejsson - for json parsing

ollama -hosting llm model

llama 3.2 - llm model

tavily python -search library

python dotenv - python library

langgraph
langchain_groq

langchain_core

groq

.

.

.

But I will try also for paid subscription to take azure open ai gpt4 to run more fast response.

3.Data Analysis & Decision Support

-Business intelligence tools with natural language queries

-Predictive analytics for forecasting

-Anomaly detection in complex systems

4,.Process Automation (RPA_AI)

5.Model Testing

What you’re testing:

-Accuracy, precision, recall (for classification)

-Hallucinations (does it invent facts?)

-Robustness (how does it handle weird inputs?)

Guardrails: Use frameworks to block toxic/off-topic outputs.

Resolution (When it breaks):

-Prompt engineering: Tweak inputs to reduce hallucinations 

-Human-in-the-loop (HITL): Route low-confidence outputs to human reviewers.

-Model fine-tuning: Retrain on failure cases 

What you’re testing:

-Accuracy, precision, recall (for classification)

-Hallucinations 

-Robustness 

6.Self Healing and Resilence Testing

What you’re testing:

-Auto-recovery from failures 

-Graceful degradation 

-Scalability 

Mitigation (Prevention)

-Chaos engineering

-Circuit breakers

-Load testing

Resolution (When it breaks):

-Auto-scaling: Spin up more instances during traffic surges.

-Fallback models: If GPT-4 fails, switch to a lighter model like Llama 3.

-Health checks: Monitor latency/error rates → auto-restart unhealthy services.

𝗛𝗼𝘄 𝘄𝗲 𝗰𝗮𝗻 𝘁𝗲𝘀𝘁 𝘁𝗵𝗲𝘀𝗲 𝗔𝗜-𝗽𝗼𝘄𝗲𝗿𝗲𝗱 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀:

1.Functional Testing

Traditional test cases won't work! We need to test ranges of acceptable outputs

Validate that responses are contextually appropriate, not just "correct"

Test edge cases that might trigger hallucinations or inappropriate responses

2.Performance Testing

Measure response times under varying loads

Evaluate resource consumption (AI models can be resource-intensive!)

Test scalability as user base grows

3.Bias & Fairness Testing

Check for demographic, cultural, or gender biases in outputs

Ensure equitable treatment across different user groups

Validate against harmful content generation

4.Security Testing

Test for prompt injection vulnerabilities

Evaluate data privacy protection

Assess resistance to adversarial attacks

5.User Experience Testing

Evaluate the naturalness and helpfulness of interactions

Test error handling when AI doesn't understand

Measure user satisfaction with AI-generated content

--Happy Testing!

Bharat Varshney

1. Customer Support Automation- 
   How to Test:
   Automated Dialog Testing: Tools like Botium, Rasa Test, Chatette.
   NLU Evaluation: Confusion matrix, F1 scores for intents/entities. 
   Edge Case Simulation: Garbled text, sarcasm, incomplete queries.Sentiment Analysis Testing: Ensure correct emotional tone detection
2. HR Chatbots
   How to Test:
   Fairness & Bias Audits: Tools like Fairlearn, AIF360.
   PII Redaction Testing: Ensure sensitive fields (SSN, salary) aren’t leaked.
   User Role Testing: Different permissions for candidates, employees, recruiters.
   Data Retention & Consent: Verify audit logs and opt-ins.
3. Document Understanding & Automation 
   How to Test
   Document Variants: Test across vendors, formats (scanned vs digital).
   Gold Dataset Evaluation: Use labelled docs to compute precision/recall.
   Error Injection: Blur images, add noise, change font size to simulate poor scans.
   Regression Testing: After model updates, compare against baseline outputs.

Thank you everyone for participating in this challenge!  And special thank you to ​@executeautomation for hosting it.

🎉We’re happy to announce the winner of this challenge: ​@Kunal2027

​@Kunal2027 Please keep an eye on your email, we will reach out shortly to arrange your giftbox delivery, or A Udemy course of your choice, as well as a shareable certificate.

Stay tuned to ShiftSync for more events!👀

As have seen and involved in testing, companies are mostly developing:
1. AI powered chatbots/assistants
2. Agents to automate the tasks

I think these are the low hanging fruits. Easy to start and quick to implement.
No model training/fine-tuning is required for these use cases. 

Mostly RAG based applications are most used scenarios.
As the time passes, we may see more AI adoption based on the success.