A Crucial Parameter for Rank-Frequency Relation in Natural Languages

Chenchen Ding
2/1/2024
cs.CL

No confident code implementation found for this paper.

Abstract

$f \propto r^{-α} \cdot (r+γ)^{-β}$ has been empirically shown more precise than a naïve power law $f\propto r^{-α}$ to model the rank-frequency ($r$-$f$) relation of words in natural languages. This work shows that the only crucial parameter in the formulation is $γ$, which depicts the resistance to vocabulary growth on a corpus. A method of parameter estimation by searching an optimal $γ$ is proposed, where a ``zeroth word'' is introduced technically for the calculation. The formulation and parameters are further discussed with several case studies.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

The Distribution of Phoneme Frequencies across the World's Languages: Macroscopic and Microscopic Information-Theoretic Models

78% match2026

Random Text, Zipf's Law, Critical Length,and Implications for Large Language Models

77% match2025

Non-Zipfian Distribution of Stopwords and Subset Selection Models

76% match2026

Statistical Patterns in the Equations of Physics and the Emergence of a Meta-Law of Nature

76% match2024

Relative-Based Scaling Law for Neural Language Models

75% match2025

The exponential distribution of the order of demonstrative, numeral, adjective and noun

75% match2025

Cite this paper

@article{ding2024crucial,
  title  = {A Crucial Parameter for Rank-Frequency Relation in Natural Languages},
  author = {Chenchen Ding},
  year   = {2024},
  eprint = {2402.00271},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.00271v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.00271.svg)](https://codeofpaper.com/paper/2402.00271)

Discussion