mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

David Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo
5/18/2023
cs.CL

No official code found. Best implementation: google-research/longt5 (183 stars, Apache-2.0, last commit 2023-05-26).

Abstract

We present our work on developing a multilingual, efficient text-to-text transformer that is suitable for handling long inputs. This model, called mLongT5, builds upon the architecture of LongT5, while leveraging the multilingual datasets used for pretraining mT5 and the pretraining tasks of UL2. We evaluate this model on a variety of multilingual summarization and question-answering tasks, and the results show stronger performance for mLongT5 when compared to existing multilingual models such as mBART or M-BERT.

View on arXivView PDF

Code Implementations(1)

google-research/longt5Community100%
Apache-2.0
18317PythonFeb 24, 20223 years agoApache-2.0

Similar Papers

A General-Purpose Multilingual Document Encoder

87% match1 repo52023

Efficient Long-Text Understanding with Short-Text Models

86% match1 repo702022

Pretraining and Benchmarking Modern Encoders for Latvian

86% match2026

Octopus: A Multitask Model and Toolkit for Arabic Natural Language Generation

85% match2023

Advancing Text-to-GLOSS Neural Translation Using a Novel Hyper-parameter Optimization Technique

85% match2023

idT5: Indonesian Version of Multilingual T5 Transformer

85% match2023

Cite this paper

@article{uthus2023mlongt,
  title  = {mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences},
  author = {David Uthus and Santiago Ontañón and Joshua Ainslie and Mandy Guo},
  year   = {2023},
  eprint = {2305.11129},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.11129v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.11129.svg)](https://codeofpaper.com/paper/2305.11129)

Discussion