Detecting Conceptual Abstraction in LLMs

Michaela Regneri, Alhassan Abdelhalim, Sören Laue
4/24/2024
cs.CLcs.LG

No confident code implementation found for this paper.

Abstract

We present a novel approach to detecting noun abstraction within a large language model (LLM). Starting from a psychologically motivated set of noun pairs in taxonomic relationships, we instantiate surface patterns indicating hypernymy and analyze the attention matrices produced by BERT. We compare the results to two sets of counterfactuals and show that we can detect hypernymy in the abstraction mechanism, which cannot solely be related to the distributional similarity of noun pairs. Our findings are a first step towards the explainability of conceptual abstraction in LLMs.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph

83% match1 repo132023

LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models

83% match2026

Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction

82% match1 repo22025

Ranking Entities along Conceptual Space Dimensions with LLMs: An Analysis of Fine-Tuning Strategies

80% match1 repo2024

Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Models

80% match2025

Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?

80% match1 repo822024

Cite this paper

@article{regneri2024detecting,
  title  = {Detecting Conceptual Abstraction in LLMs},
  author = {Michaela Regneri and Alhassan Abdelhalim and Sören Laue},
  year   = {2024},
  eprint = {2404.15848},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.15848v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.15848.svg)](https://codeofpaper.com/paper/2404.15848)

Discussion