An Information Extraction Study: Take In Mind the Tokenization!

Christos Theodoropoulos, Marie-Francine Moens
3/27/2023
cs.CL

No official code found. Best implementation: christos42/inductive_bias_IE (7 stars, MIT, last commit 2024-10-30).

Abstract

Current research on the advantages and trade-offs of using characters, instead of tokenized text, as input for deep learning models, has evolved substantially. New token-free models remove the traditional tokenization step; however, their efficiency remains unclear. Moreover, the effect of tokenization is relatively unexplored in sequence tagging tasks. To this end, we investigate the impact of tokenization when extracting information from documents and present a comparative study and analysis of subword-based and character-based models. Specifically, we study Information Extraction (IE) from biomedical texts. The main outcome is twofold: tokenization patterns can introduce inductive bias that results in state-of-the-art performance, and the character-based models produce promising results; thus, transitioning to token-free IE models is feasible.

View on arXivView PDF

Code Implementations(1)

christos42/inductive_bias_IECommunity100%
MIT

An Information Extraction Study: Take In Mind the Tokenization! (official repository of the paper)

71Shell, PythonJun 15, 20221 years agoMIT
biomedical-textdeep-learninginductive-biasesinformation-extractionlanguage-models+5 more

Similar Papers

Comparing Variation in Tokenizer Outputs Using a Series of Problematic and Challenging Biomedical Sentences

84% match2023

Learn Your Tokens: Word-Pooled Tokenization for Language Modeling

84% match1 repo72023

MorphPiece : A Linguistic Tokenizer for Large Language Models

83% match2023

On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi

83% match2023

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT

83% match2023

ByT5: Towards a token-free future with pre-trained byte-to-byte models

82% match1 repo5482021

Cite this paper

@article{theodoropoulos2023information,
  title  = {An Information Extraction Study: Take In Mind the Tokenization!},
  author = {Christos Theodoropoulos and Marie-Francine Moens},
  year   = {2023},
  eprint = {2303.15100},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.15100v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.15100.svg)](https://codeofpaper.com/paper/2303.15100)

Discussion