An Analysis of Tokenization: Transformers under Markov Data

Nived Rajaraman, Jiantao Jiao, K. Ramchandran
2/3/2026

No confident code implementation found for this paper. 2 possible matches can be shown below.

DOISemantic Scholar

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{rajaraman2026analysis,
  title  = {An Analysis of Tokenization: Transformers under Markov Data},
  author = {Nived Rajaraman and Jiantao Jiao and K. Ramchandran},
  year   = {2026},
  doi    = {10.52202/079017-1999},
  url    = {https://doi.org/10.52202/079017-1999},
  journal = {NEURIPS 2024 2024}
}

Discussion