Extracting Paragraphs from LLM Token Activations

Nicholas Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, Lucile Ter-Minassian
9/10/2024
cs.CL

No official code found. Best implementation: nickypro/extracting-paragraphs (2 stars, last commit 2024-11-20).

Abstract

Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.

View on arXivView PDF

Code Implementations(1)

nickypro/extracting-paragraphsCommunity66%

Extracting Paragraphs from LLM Token Activations

21PythonSep 5, 20241 years ago

Similar Papers

Unraveling Token Prediction Refinement and Identifying Essential Layers in Language Models

84% match2025

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

83% match2024

ParaScopes: What do Language Models Activations Encode About Future Text?

83% match2025

Beyond Tokens in Language Models: Interpreting Activations through Text Genre Chunks

82% match2025

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

82% match2025

Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

82% match2024

Cite this paper

@article{pochinkov2024extracting,
  title  = {Extracting Paragraphs from LLM Token Activations},
  author = {Nicholas Pochinkov and Angelo Benoit and Lovkush Agarwal and Zainab Ali Majid and Lucile Ter-Minassian},
  year   = {2024},
  eprint = {2409.06328},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.06328v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.06328.svg)](https://codeofpaper.com/paper/2409.06328)

Discussion