Examining Language Modeling Assumptions Using an Annotated Literary Dialect Corpus

Craig Messner, Tom Lippincott
10/3/2024
cs.CL

No official code found. Best implementation: comp-int-hum/orthography-embedding-clustering (0 stars, MIT, last commit 2024-11-16).

Abstract

We present a dataset of 19th century American literary orthovariant tokens with a novel layer of human-annotated dialect group tags designed to serve as the basis for computational experiments exploring literarily meaningful orthographic variation. We perform an initial broad set of experiments over this dataset using both token (BERT) and character (CANINE)-level contextual language models. We find indications that the "dialect effect" produced by intentional orthographic variation employs multiple linguistic channels, and that these channels are able to be surfaced to varied degrees given particular language modelling assumptions. Specifically, we find evidence showing that choice of tokenization scheme meaningfully impact the type of orthographic information a model is able to surface.

View on arXivView PDF

Code Implementations(1)

comp-int-hum/orthography-embedding-clusteringCommunity100%
MIT

Experiments clustering orthovariant embeddings/embedding differences over character and wordpiece models

00PythonMar 27, 20241 years agoMIT

Similar Papers

Modeling Orthographic Variation in Occitan's Dialects

84% match2024

ALDi: Quantifying the Arabic Level of Dialectness of Text

81% match1 repo42023

How Do Language Models Acquire Character-Level Information?

81% match2026

Tokenization is Sensitive to Language Variation

81% match2025

Pairing Orthographically Variant Literary Words to Standard Equivalents Using Neural Edit Distance Models

80% match2024

The Arabic Generality Score: Another Dimension of Modeling Arabic Dialectness

80% match2025

Cite this paper

@article{messner2024examining,
  title  = {Examining Language Modeling Assumptions Using an Annotated Literary Dialect Corpus},
  author = {Craig Messner and Tom Lippincott},
  year   = {2024},
  eprint = {2410.02674},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.02674v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.02674.svg)](https://codeofpaper.com/paper/2410.02674)

Discussion