Token-Level Fitting Issues of Seq2seq Models

Guangsheng Bao, Zhiyang Teng, Yue Zhang
5/8/2023
cs.CL

No confident code implementation found for this paper.

Abstract

Sequence-to-sequence (seq2seq) models have been widely used for natural language processing, computer vision, and other deep learning tasks. We find that seq2seq models trained with early-stopping suffer from issues at the token level. In particular, while some tokens in the vocabulary demonstrate overfitting, others underfit when training is stopped. Experiments show that the phenomena are pervasive in different models, even in fine-tuned large pretrained-models. We identify three major factors that influence token-level fitting, which include token frequency, parts-of-speech, and prediction discrepancy. Further, we find that external factors such as language, model size, domain, data scale, and pretraining can also influence the fitting of tokens.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models

85% match1 repo292023

Ablate-to-Validate: Are Vision-Language Models Really Using Continuous Thought Tokens?

83% match2026

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

83% match1 repo2023

Optimal Splitting of Language Models from Mixtures to Specialized Domains

83% match2026

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

83% match2026

Deep Reinforcement Learning For Sequence to Sequence Models

82% match1 repo7672018

Cite this paper

@article{bao2023tokenlevel,
  title  = {Token-Level Fitting Issues of Seq2seq Models},
  author = {Guangsheng Bao and Zhiyang Teng and Yue Zhang},
  year   = {2023},
  eprint = {2305.04493},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.04493v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.04493.svg)](https://codeofpaper.com/paper/2305.04493)

Discussion