On the Dimensionality of Word Embedding

Zi Yin, Yuanyuan Shen
12/11/2018
cs.LGcs.CLstat.ML

Official code: ziyin-dl/word-embedding-dimensionality-selection (327 stars, MIT, last commit 2020-06-17). 1 more implementation below.

Abstract

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

View on arXivView PDF

Code Implementations(2)

ziyin-dl/word-embedding-dimensionality-selectionOfficial100%
MIT

On the Dimensionality of Word Embedding

32745PythonNov 16, 20186 years agoMIT
ziyin-dl/PIP-experimentsOfficial100%
MIT
72C, RMar 2, 20187 years agoMIT

Similar Papers

On Dimensional Linguistic Properties of the Word Embedding Space

83% match1 repo82019

Factors Influencing the Surprising Instability of Word Embeddings

82% match1 repo82018

Contrastive Loss is All You Need to Recover Analogies as Parallel Lines

82% match2023

A Latent Variable Model Approach to PMI-based Word Embeddings

81% match1 repo552015

Multimodal Word Distributions

81% match1 repo2842017

Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms

80% match2026

Cite this paper

@article{yin2018dimensionality,
  title  = {On the Dimensionality of Word Embedding},
  author = {Zi Yin and Yuanyuan Shen},
  year   = {2018},
  eprint = {1812.04224},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1812.04224}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1812.04224.svg)](https://codeofpaper.com/paper/1812.04224)

Discussion