Language Modeling Using Tensor Trains

Zhan Su, Yuqin Zhou, Fengran Mo, Jakob Grue Simonsen
5/7/2024
cs.CLcs.IR

Official code: shuishen112/tensortrainlm (6 stars, last commit 2026-03-20).

Abstract

We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by the tensor product of words, but computing the probabilities of sentences in a low-dimensional fashion. We demonstrate that the architectures of Second-order RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs are, essentially, special cases of TTLM. Experimental evaluations on real language modeling tasks show that the proposed variants of TTLM (i.e., TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks (RNNs) with low-scale of hidden units. (The code is available at https://github.com/shuishen112/tensortrainlm.)

View on arXivView PDF

Code Implementations(1)

shuishen112/tensortrainlmOfficial100%
60PythonJul 11, 20237 months ago

Similar Papers

Tensor Networks Meet Neural Networks: A Survey and Future Perspectives

83% match1 repo1562023

Lower Bounds on the Expressivity of Recurrent Neural Language Models

81% match2024

Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models

81% match2025

On Efficiently Representing Regular Languages as RNNs

81% match2024

Graph Tensor Networks: An Intuitive Framework for Designing Large-Scale Neural Learning Systems on Multiple Domains

81% match2023

A Tensor Decomposition Perspective on Second-order RNNs

80% match1 repo22024

Cite this paper

@article{su2024language,
  title  = {Language Modeling Using Tensor Trains},
  author = {Zhan Su and Yuqin Zhou and Fengran Mo and Jakob Grue Simonsen},
  year   = {2024},
  eprint = {2405.04590},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.04590v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.04590.svg)](https://codeofpaper.com/paper/2405.04590)

Discussion