AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Jonas Groschwitz, Shay B. Cohen, Lucia Donatelli, Meaghan Fowlie
12/6/2023
cs.CL

No official code found. Best implementation: jgroschwitz/GrAPES (0 stars, GPL-3.0, last commit 2025-07-23).

Abstract

We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

View on arXivView PDF

Code Implementations(1)

jgroschwitz/GrAPESCommunity100%
GPL-3.0
01TeX, ShellOct 17, 20231 years agoGPL-3.0

Similar Papers

Holistic Exploration on Universal Decompositional Semantic Parsing: Architecture, Data Augmentation, and LLM Paradigm

82% match1 repo42023

DOCR-Inspector: Fine-Grained and Automated Evaluation of Document Parsing with VLM

82% match1 repo82025

AMR Parsing with Causal Hierarchical Attention and Pointers

81% match1 repo42023

On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

81% match2023

MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios

81% match1 repo8932026

"You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation

81% match2023

Cite this paper

@article{groschwitz2023parsing,
  title  = {AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite},
  author = {Jonas Groschwitz and Shay B. Cohen and Lucia Donatelli and Meaghan Fowlie},
  year   = {2023},
  eprint = {2312.03480},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.03480v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.03480.svg)](https://codeofpaper.com/paper/2312.03480)

Discussion