How to evaluate GenAI Summarization results

It is challenging to evaluate the results of GenAI applications because there are no expected results, only results from an acceptable range.  For summarization GenAI use cases, what are the most common evaluation metrics?

By using RAGAS Framework( Retrieval Augmented Generation Assessment Evaluation Framework)

we can evaluate AI models that generate answers by retrieving relevant information. It works by comparing the AI's response to an ideal answer(Ground Truth) based on the context and additional retrieved information.
RAG Assessment employs a set of predefined metrics to evaluate both the retrieval and generation components of the model. These metrics help ensure that the responses generated by the GenAI are of high quality, relevant, accurate, coherent, contextually appropriate, complete, and safe. Below are the key metrics used in RAG Assessment and how they are applied:
1. Answer Relevancy,2. Answer Accuracy,3. Answer Completeness,4. Answer Harmfulness

There is no quantitative answer to this . But we can evaluate them based on the user needs. 
Weighted scoring (for fluency, accuracy, flexibility,not repetitive), 
Evaluating dimensions  
Benchmarking Against Multiple References(citations, OCR) summarization results can be assessed to ensure they meet user needs and context-specific requirements.

In summary we need to focus on the following components to cover all the use cases : 
Model health checks, Chatbot checks, Doc processing platforms

There is no quantitative answer to this . But we can evaluate them based on the user needs. 
Weighted scoring (for fluency, accuracy, flexibility,not repetitive), 
Evaluating dimensions  
Benchmarking Against Multiple References(citations, OCR) summarization results can be assessed to ensure they meet user needs and context-specific requirements.

In summary we need to focus on the following components to cover all the use cases : 
Model health checks, Chatbot checks, Doc processing platforms

Evaluating summarization outputs in Generative AI (GenAI) use cases can be complex due to the absence of fixed "expected results." Instead, the evaluation relies on comparing generated summaries against acceptable benchmarks.

While doing this we can face lot of challenges which would be on following lines and i think we have to adopt optimum approach to address them, for e.g.

1. Outputs can be diverse, to handle this we can have a human intervention to evaluate the model flexibility and accurateness.
2. Adjust metrics to prioritize different qualities (e.g., accuracy for legal documents, readability for blog summaries).
3. Use frameworks like NLGEval or EvalSumm to streamline evaluations across metrics.

By combining metrics like Weighted scoring (for fleuncy, accuracy, flexibility), Evaluating different dimesions, Benchmarking Against Multiple References summarization results can be holistically assessed to ensure they meet user needs and context-specific requirements.

There is no quantitative answer to this . But we can evaluate them based on the user needs. 
Weighted scoring (for fluency, accuracy, flexibility,not repetitive), 
Evaluating dimensions  
Benchmarking Against Multiple References(citations, OCR) summarization results can be assessed to ensure they meet user needs and context-specific requirements.

In summary we need to focus on the following components to cover all the use cases : 
Model health checks, Chatbot checks, Doc processing platforms

Evaluating summarization outputs in Generative AI (GenAI) use cases can be complex due to the absence of fixed "expected results." Instead, the evaluation relies on comparing generated summaries against acceptable benchmarks.

While doing this we can face lot of challenges which would be on following lines and i think we have to adopt optimum approach to address them, for e.g.

1. Outputs can be diverse, to handle this we can have a human intervention to evaluate the model flexibility and accurateness.
2. Adjust metrics to prioritize different qualities (e.g., accuracy for legal documents, readability for blog summaries).
3. Use frameworks like NLGEval or EvalSumm to streamline evaluations across metrics.

By combining metrics like Weighted scoring (for fleuncy, accuracy, flexibility), Evaluating different dimesions, Benchmarking Against Multiple References summarization results can be holistically assessed to ensure they meet user needs and context-specific requirements.

Evaluating GenAI summarization results requires a combination of objective metrics and human judgment since the outputs often lack a definitive "correct" answer. Here are the most common evaluation metrics used:  

1. ROUGE (Recall-Oriented Understudy for Gisting Evaluation):

   - Measures the overlap of n-grams (e.g., unigrams, bigrams) between the generated summary and a reference summary.  

   - ROUGE-N focuses on exact matches, while ROUGE-L evaluates sequence similarity considering sentence structure.  

2. BLEU (Bilingual Evaluation Understudy): 

   - Originally designed for machine translation, BLEU can also be used to measure the precision of n-grams in the generated summary compared to a reference.  

   - While useful, it often undervalues summaries with varied but accurate phrasing.  

3. BERTScore:  

   - Uses contextual embeddings from models like BERT to assess semantic similarity between the generated summary and reference summaries.  

   - It captures meaning better than surface-level metrics like ROUGE and BLEU.  

4. Length-based Metrics:

   - Evaluates whether the summary adheres to a desired length or compression ratio without losing essential information.  

5. Factual Consistency Checks:

   - Measures the accuracy of factual information retained from the source content using automated tools or manual verification.  

   - Tools like QuestEval and FactCC can assist with this.  

6. Human Evaluation:

   - Involves assessing summaries based on criteria such as coherence, relevance, fluency, and informativeness.  

   - This step ensures the summary aligns with user expectations and captures subjective nuances.  

7. Novelty and Redundancy Analysis:

   - Evaluates whether the summary introduces unnecessary information (novelty) or repeats content excessively (redundancy).  

8. Domain-specific Metrics:

   - For specialized use cases (e.g., legal or medical summaries), customized metrics are designed to assess adherence to domain-specific requirements.  

Using a combination of these metrics provides a balanced evaluation, ensuring both technical and user-centric quality in GenAI summarization outputs.