AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization

Anum Afzal, Ribin Chalumattu, Florian Matthes, Laura Mascarell
7/16/2024
cs.CL

No official code found. Best implementation: anum94/AdaptEval (1 star, last commit 2026-04-10).

Abstract

Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.

View on arXivView PDF

Code Implementations(1)

anum94/AdaptEvalCommunity100%

This is the Framework for Domain Adaptation Suite introduced in the paper AdaptEval https://www.arxiv.org/abs/2407.11591

10PythonJul 22, 20246 months ago

Similar Papers

Word Matters: What Influences Domain Adaptation in Summarization?

90% match1 repo2024

Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization

87% match2025

DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization

86% match1 repo22024

Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings

86% match1 repo2025

Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning

84% match1 repo2026

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study

84% match2023

Cite this paper

@article{afzal2024adapteval,
  title  = {AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization},
  author = {Anum Afzal and Ribin Chalumattu and Florian Matthes and Laura Mascarell},
  year   = {2024},
  eprint = {2407.11591},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.11591v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.11591.svg)](https://codeofpaper.com/paper/2407.11591)

Discussion