How to evaluate GenAI Summarization results

Here’s what I would follow for measuring responses from LLMs:

1. Fact-Checking: Ensure factual accuracy by comparing outputs against reliable sources. This is critical for summarizing data or answering fact-based questions.
2. Context Validation: Validate outputs for relevance and coherence, ensuring no conflation of unrelated contexts (e.g., distinguishing between "Playwright" as a test automation tool vs. a dramatist).
3. Language Quality: Assess translations for fluency, adequacy, and cultural appropriateness, avoiding literal or awkward phrasing.
4. Relevance and Coherence: Response should be logical, and cover key points without adding irrelevant details. 
5. Readability and Usability: Outputs should suit the target audience's reading level and purpose, evaluated through readability tests and user feedback.
6. 6. Bias and Fairness: Check for neutrality and appropriate framing, especially in sensitive topics.

I'm fortunate enough that I'm part of the LLM testing team. We have an in-house application developed for this and here are the approaches I've followed:

This is not like checking whether the button is visible or not so we have formed a different testing approach:

- Used multiple evaluation metrics like Factual accuracy, Coherence, Conciseness, Relevance to original text, Grammatical correctness and Fullness.

- Created some expected outputs/Ground truths and compared them against LLM-generated summaries

- So based on the metrics, every tester gives some functional scoring out of 5 and we use this scoring as benchmark for the next release 

Test across different types of source content

Key Metrics to Assess

As a tester. I don’t look for correctness. I look for problems.

Something can be correct, yet there might still be a problem. For instance, consider asking a chatbot “What’s 3 + 5?” and the chatbot answering “3 + 5 is 8; what are you, some kind of idiot?”

And sometthing can be incorrect, and incorrectness isn’t a problem. Ask a chatbot “What’s the square root of 2?” and the chatbot answers “1.4142136”  That’s not correct; the square root of 2 is an irrational number, and the part after the decimal place continues forever. So the answer from the chatbot is incorrect, but it’s consistent with what a human might say in reply.  For the vast majority of circumstances, that answer would be sufficiently close that the imprecision is not a problem.

I'm fortunate enough that I'm part of the LLM testing team. We have an in-house application developed for this and here are the approaches I've followed:

This is not like checking whether the button is visible or not so we have formed a different testing approach:

- Used multiple evaluation metrics like Factual accuracy, Coherence, Conciseness, Relevance to original text, Grammatical correctness and Fullness.

- Created some expected outputs/Ground truths and compared them against LLM-generated summaries

- So, based on the metrics, every tester gives some functional scoring out of 5, and we take the aggregate of this scoring as a benchmark for the next release. So that we know what we are doing!

- We also did test across different types of source content where whether our LLM can summarise the data from a table, from different paragraphs or from an image etc..,

You absolutely think like a human being and test it to reveal unexpected things.

Other than that, recently, we have started doing POC on considering some of the above metrics using LLM as a judge and tested this functionality. The results are promising, but still, we are using it as a safer side and kind of additional layer of testing, and it'll take time to trust this.

Some evals/frameworks to try:

- ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

- BLEU (Bilingual Evaluation Understudy)

- METEOR (Metric for Evaluation of Translation with Explicit ORdering)

- BERTScore

- Retrieval Augmented Generation Assessment(RAGAS)

- custom metrics with GEval where we can define what exactly the other LLM to validate 

And so-on.

We can definitely test their summarization feature, but we also need to define when to stop testing for that.

Hope that helps 🙏. 

Good stuff, Michael.  How do you some tests for correctness with an automated tool/metric?

Hello Testers!  Below, I used LLM to summarize our discussion thread.  How would you rate the LLM summary ?
 

Evaluating GenAI summarization results is a complex task due to the lack of definitive "correct" answers. However, several common evaluation metrics and approaches are used to assess the quality and effectiveness of AI-generated summaries:

Automated Metrics

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

• Measures overlap between generated and reference summaries using n-grams and sequences.
• ROUGE-N focuses on exact matches, while ROUGE-L evaluates sequence similarity considering sentence structure.

BLEU (Bilingual Evaluation Understudy)

• Originally designed for machine translation, it measures n-gram precision.
• Useful for short texts but struggles with synonyms and paraphrasing.

BERTScore

• Uses contextual embeddings to assess semantic similarity between generated and reference summaries.
• Captures meaning better than surface-level metrics like ROUGE and BLEU.

METEOR

• Considers synonyms and linguistic variations, offering better semantic evaluation than BLEU.

Human Evaluation

Human evaluation remains crucial for assessing subjective aspects of summaries:

• Judges fluency, relevance, conciseness, and faithfulness.
• Provides nuanced insights but can be time-consuming.

Other Evaluation Approaches

QuestEval

• Assesses how well the summary answers questions derived from the source.
• Useful for evaluating faithfulness and informativeness.

Factual Consistency Checks

• Measures the accuracy of factual information retained from the source content.
• Tools like QuestEval and FactCC can assist with this process.

Length-based Metrics

• Evaluates whether the summary adheres to a desired length or compression ratio.

Novelty and Redundancy Analysis

• Assesses whether the summary introduces unnecessary information or repeats content excessively.

Domain-specific Metrics

• Customized metrics for specialized use cases (e.g., legal or medical summaries).

Frameworks and Tools

RAGAS Framework

• Compares AI-generated summaries to ideal answers (Ground Truth).
• Evaluates both retrieval and generation components of the model.
• Key metrics: Answer Relevancy, Accuracy, Completeness, and Harmfulness.

NLGEval and EvalSumm

• Frameworks to streamline evaluations across multiple metrics.

Best Practices

• Combine multiple metrics for a balanced evaluation.
• Use both automated metrics and human evaluation for comprehensive assessment.
• Adjust metrics based on specific use cases and priorities (e.g., accuracy for legal documents, readability for blog summaries).
• Implement regular health checks for LLM and data drift.
• Consider user feedback for iterative improvement.

By utilizing a combination of these metrics and approaches, testers can effectively evaluate GenAI summarization outputs, ensuring both technical quality and user-centric value.

Good stuff, Michael.  How do you some tests for correctness with an automated tool/metric?

I don’t understand the question, nor the context that it would need for an answer.