DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translation

Rachel Bawden, Sophie Rosset, Thomas Lavergne, Eric Bilinski
5/30/2019
cs.CL

Official code: rbawden/DiaBLa-dataset (17 stars, CC-BY-SA-4.0, last commit 2022-04-29).

Abstract

We present a new English-French test set for the evaluation of Machine Translation (MT) for informal, written bilingual dialogue. The test set contains 144 spontaneous dialogues (5,700+ sentences) between native English and French speakers, mediated by one of two neural MT systems in a range of role-play settings. The dialogues are accompanied by fine-grained sentence-level judgments of MT quality, produced by the dialogue participants themselves, as well as by manually normalised versions and reference translations produced a posteriori. The motivation for the corpus is two-fold: to provide (i) a unique resource for evaluating MT models, and (ii) a corpus for the analysis of MT-mediated communication. We provide a preliminary analysis of the corpus to confirm that the participants' judgments reveal perceptible differences in MT quality between the two MT systems used.

View on arXivView PDF

Code Implementations(1)

rbawden/DiaBLa-datasetOfficial100%
CC-BY-SA-4.0

English-French MT dialogue dataset

172Shell, PythonMar 3, 20194 years agoCC-BY-SA-4.0

Similar Papers

A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators

84% match1 repo52023

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

83% match1 repo1.4k2017

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

83% match2026

Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models

83% match1 repo122026

BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues

83% match1 repo1632023

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

83% match2026

Cite this paper

@article{bawden2019diabla,
  title  = {DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translation},
  author = {Rachel Bawden and Sophie Rosset and Thomas Lavergne and Eric Bilinski},
  year   = {2019},
  eprint = {1905.13354},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1905.13354}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1905.13354.svg)](https://codeofpaper.com/paper/1905.13354)

Discussion