Correlation Dimension of Natural Language in a Statistical Manifold

Xin Du, Kumiko Tanaka-Ishii
5/10/2024
cs.CLcond-mat.stat-mechcs.AI

No confident code implementation found for this paper.

Abstract

The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barabási-Albert process. Long memory is the key to producing self-similarity. Our method is applicable to any probabilistic model of real-world discrete sequences, and we show an application to music data.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Correlation Dimension of Auto-Regressive Large Language Models

86% match2025

Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations

81% match1 repo62024

Complex systems approach to natural language

80% match2024

Sequences of Logits Reveal the Low Rank Structure of Language Models

80% match2025

A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?

80% match2026

Towards a theory of how the structure of language is acquired by deep neural networks

80% match2024

Cite this paper

@article{du2024correlation,
  title  = {Correlation Dimension of Natural Language in a Statistical Manifold},
  author = {Xin Du and Kumiko Tanaka-Ishii},
  year   = {2024},
  eprint = {2405.06321},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.06321v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.06321.svg)](https://codeofpaper.com/paper/2405.06321)

Discussion