How to evaluate GenAI Summarization results

• ROUGE: Measures overlap between the generated and reference summaries using n-grams and sequences. Widely used but focuses on surface-level matches.

• BLEU: Evaluates precision of n-gram overlap. Good for short texts but struggles with synonyms and paraphrasing.

• METEOR: Considers synonyms and linguistic variations, offering better semantic evaluation than BLEU.

• BERTScore: Uses embeddings to measure semantic similarity. More accurate for meaning but computationally intensive.

• QuestEval: Assesses how well the summary answers questions derived from the source. Great for faithfulness and informativeness.

• Human Evaluation: Judges fluency, relevance, conciseness, and faithfulness. Essential for nuanced insights but time-consuming.

1.  Define expected type of results - Range 
2.  there is no 100% test coverage - apply risk based testing - human need to compensate for missing test coverage 
3. Implement Health check  as LLM and Data drifts

Evaluating GenAI summarization involves automatic and human metrics. Common automatic metrics include ROUGE (word/phrase overlap), BLEU (n-gram precision), BERTScore (semantic similarity), and SummaC (factual consistency). These methods are fast but may overlook nuances. Human evaluation focuses on informativeness (key points captured), coherence (logical flow), fluency (grammar/readability), relevance (aligned with source), and factual accuracy (truthfulness). Combining both is common: automatic metrics for scale, human reviews for depth. Emerging trends include task-specific metrics and user feedback for iterative improvement, providing a balanced and comprehensive evaluation strategy for GenAI summarization.

Evaluate as a ratio of the correct answers or nearly correct answer to the sum of correct and incorrect answers. 

Look for true positive, false positive, true negative and false negative responses and evaluate the recall and precision of the answers. 

This will allow the tester irrespective of AI expertise to evaluate the LLM

It is difficult to evaluate the Gen AI responses, there are no pre written conditions for the responses to be pass/fail. But it is important to ensure the accuracy, reliability of the responses.

We can perform health check regularly to ensure the reliability, performance of the LLM and monitor the LLM’s. Many API’s are available to perform the health checks on the LLM’s.

Also, there is no fixed output or expected output, we have to compare the actual results with the facts.

Human Evaluation is the perfect way to assess the Gen AI responses, human evaluators can assess the responses with the factual data, relevant data.

The responses should be easy to read and understandable. Apply some automation techniques to assess the responses. ROUGE, BLEU and TER.

By using RAGAS Framework( Retrieval Augmented Generation Assessment Evaluation Framework)

we can evaluate AI models that generate answers by retrieving relevant information. It works by comparing the AI's response to an ideal answer(Ground Truth) based on the context and additional retrieved information.
RAG Assessment employs a set of predefined metrics to evaluate both the retrieval and generation components of the model. These metrics help ensure that the responses generated by the GenAI are of high quality, relevant, accurate, coherent, contextually appropriate, complete, and safe. Below are the key metrics used in RAG Assessment and how they are applied:
1. Answer Relevancy,2. Answer Accuracy,3. Answer Completeness,4. Answer Harmfulness

Evaluating summarization outputs in Generative AI (GenAI) use cases can be complex due to the absence of fixed "expected results." Instead, the evaluation relies on comparing generated summaries against acceptable benchmarks.

While doing this we can face lot of challenges which would be on following lines and i think we have to adopt optimum approach to address them, for e.g.

1. Outputs can be diverse, to handle this we can have a human intervention to evaluate the model flexibility and accurateness.
2. Adjust metrics to prioritize different qualities (e.g., accuracy for legal documents, readability for blog summaries).
3. Use frameworks like NLGEval or EvalSumm to streamline evaluations across metrics.

By combining metrics like Weighted scoring (for fleuncy, accuracy, flexibility), Evaluating different dimesions, Benchmarking Against Multiple References summarization results can be holistically assessed to ensure they meet user needs and context-specific requirements.

Human intervention is ofcourse needed.Using APIs is one of the way.

I’d like to go a bit deeper on the question, rather than simply providing an answer. A metric is a measurement function, a mathematical operation by which we hang a number onto an observation. One of the important questions is to ask: how do you count to one? In qualitative research, there’s a (clunky) term for this: “operationalization”.  That is, what is the operation by which you put a number on “relevancy”, “accuracy”, “completeness”, or “harmfulness”?  

A description or an assessment makes sense to me in this context; numbers less so.