Discovering Universal Geometry in Embeddings with ICA

Hiroaki Yamagiwa, Momose Oyama, Hidetoshi Shimodaira
5/22/2023
cs.CL

No official code found. Best implementation: shimo-lab/Universal-Geometry-with-ICA (22 stars, last commit 2025-06-17).

Abstract

This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.

View on arXivView PDF

Code Implementations(1)

shimo-lab/Universal-Geometry-with-ICACommunity100%

Discovering Universal Geometry in Embeddings with ICA (Published in EMNLP 2023)

220Shell, PythonOct 18, 20231 years ago
cross-lingualembeddingsemnlpemnlp2023ica+6 more

Similar Papers

Linear Spaces of Meanings: Compositional Structures in Vision-Language Models

85% match2023

Imparting Interpretability to Word Embeddings while Preserving Semantic Structure

83% match2018

ITEm: Unsupervised Image-Text Embedding Learning for eCommerce

83% match2023

Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models

82% match2026

Geometric Patterns of Meaning: A PHATE Manifold Analysis of Multi-lingual Embeddings

82% match2025

ComAlign: Compositional Alignment in Vision-Language Models

81% match2024

Cite this paper

@article{yamagiwa2023discovering,
  title  = {Discovering Universal Geometry in Embeddings with ICA},
  author = {Hiroaki Yamagiwa and Momose Oyama and Hidetoshi Shimodaira},
  year   = {2023},
  eprint = {2305.13175},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.13175v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.13175.svg)](https://codeofpaper.com/paper/2305.13175)

Discussion