BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

Taj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles
1/1/2026
cs.CLcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

85% match1 repo12025

Why and How Auxiliary Tasks Improve JEPA Representations

82% match2025

M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture

81% match2 repos322024

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

80% match2025

Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures

80% match1 repo62026

Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning

80% match2024

Cite this paper

@article{gillin2026bertjepa,
  title  = {BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics},
  author = {Taj Gillin and Adam Lalani and Kenneth Zhang and Marcel Mateos Salles},
  year   = {2026},
  eprint = {2601.00366},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2601.00366v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2601.00366.svg)](https://codeofpaper.com/paper/2601.00366)

Discussion