Towards Measuring Representational Similarity of Large Language Models

Max Klabunde, Mehdi Ben Amor, Michael Granitzer, Florian Lemmerich
12/5/2023
cs.LGcs.CL

No official code found. Best implementation: mklabunde/llm_repsim (5 stars, MIT, last commit 2023-12-15).

Abstract

Understanding the similarity of the numerous released large language models (LLMs) has many uses, e.g., simplifying model selection, detecting illegal model reuse, and advancing our understanding of what makes LLMs perform well. In this work, we measure the similarity of representations of a set of LLMs with 7B parameters. Our results suggest that some LLMs are substantially different from others. We identify challenges of using representational similarity measures that suggest the need of careful study of similarity scores to avoid false conclusions.

View on arXivView PDF

Code Implementations(1)

mklabunde/llm_repsimCommunity100%
MIT
50Python, Jupyter NotebookNov 14, 20232 years agoMIT

Similar Papers

A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models

88% match1 repo32025

Data Similarity is Not Enough to Explain Language Model Performance

85% match1 repo42023

Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models

84% match1 repo72024

On the Impact of Language Selection for Training and Evaluating Programming Language Models

84% match2023

Rethinking STS and NLI in Large Language Models

84% match2023

Understanding the Inner Workings of Language Models Through Representation Dissimilarity

84% match2023

Cite this paper

@article{klabunde2023towards,
  title  = {Towards Measuring Representational Similarity of Large Language Models},
  author = {Max Klabunde and Mehdi Ben Amor and Michael Granitzer and Florian Lemmerich},
  year   = {2023},
  eprint = {2312.02730},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2312.02730v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2312.02730.svg)](https://codeofpaper.com/paper/2312.02730)

Discussion