Parts-of-Speech as Emergent Categories in SAE Latent Space

Alessandro Bondielli, Lucia Passaro, Serena Auriemma, Alessandro Lenci
9/24/2026
cs.CL

No official code found. Best implementation: colinglab/pos-sae-latents (0 stars, MIT, last commit 2026-08-26).

Abstract

Sparse AutoEncoders (SAEs) offer a promising way to inspect language model representations, but it is still unclear what kind of linguistic structure their latents expose. We use part-of-speech (PoS) categories as a controlled test case to study whether morpho-syntactic information is encoded by individual latents or by structured groups of features. We find that PoS distinctions are highly recoverable from SAE activations, but do not align with one-to-one latent / category mappings. This recoverability is not reducible to lexical memorisation, and Open and Closed PoS classes differ substantially. Categories are supported by compact groups of sparse latents, with substantial variation across tags. These groups remain stable on held-out data, while also showing overlap between related categories. Our results show that SAEs localise morpho-syntactic information in a distributed and category-dependent form rather than through atomic grammatical features.

View on arXivView PDF

Code Implementations(1)

colinglab/pos-sae-latentsCommunity65%
MIT

Code for the EMNLP 2026 paper "Parts-of-Speech as Emergent Categories in SAE Latent Space": probing, salience/coverage, and held-out validation analyses testing whether PoS categories are encoded by individual Sparse Autoencoder latents or by structured groups of latents in LLaMA-3-8B activations on the UD English GUM treebank.

00Aug 26, 20261 months agoMIT

Similar Papers

Wave to Syntax: Probing spoken language models for syntax

83% match1 repo32023

Review of Unsupervised POS Tagging and Its Implications on Language Acquisition

83% match2023

JCoLA: Japanese Corpus of Linguistic Acceptability

82% match1 repo192023

A Joint Matrix Factorization Analysis of Multilingual Representations

82% match1 repo32023

Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models

82% match2026

Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models

82% match2026

Cite this paper

@article{bondielli2026partsofspeech,
  title  = {Parts-of-Speech as Emergent Categories in SAE Latent Space},
  author = {Alessandro Bondielli and Lucia Passaro and Serena Auriemma and Alessandro Lenci},
  year   = {2026},
  eprint = {2609.29362},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2609.29362v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2609.29362.svg)](https://codeofpaper.com/paper/2609.29362)

Discussion