FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models

Konstantin Dobler, Gerard de Melo
5/23/2023
cs.CL

No official code found. Best implementation: konstantinjdobler/focus (37 stars, MIT, last commit 2025-06-07).

Abstract

Using model weights pretrained on a high-resource language as a warm start can reduce the need for data and compute to obtain high-quality language models for other, especially low-resource, languages. However, if we want to use a new tokenizer specialized for the target language, we cannot transfer the source model's embedding matrix. In this paper, we propose FOCUS - Fast Overlapping Token Combinations Using Sparsemax, a novel embedding initialization method that initializes the embedding matrix effectively for a new tokenizer based on information in the source model's embedding matrix. FOCUS represents newly added tokens as combinations of tokens in the overlap of the source and target vocabularies. The overlapping tokens are selected based on semantic similarity in an auxiliary static token embedding space. We focus our study on using the multilingual XLM-R as a source model and empirically show that FOCUS outperforms random initialization and previous work in language modeling and on a range of downstream tasks (NLI, QA, and NER).

View on arXivView PDF

Code Implementations(1)

konstantinjdobler/focusCommunity100%
MIT

[EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"

376PythonJan 20, 20231 years agoMIT

Similar Papers

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

86% match2023

Model-Aware Tokenizer Transfer

86% match2025

Word Embeddings Are Steers for Language Models

86% match1 repo1472023

Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages

85% match2023

Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages

85% match1 repo62023

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

85% match2026

Cite this paper

@article{dobler2023focus,
  title  = {FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models},
  author = {Konstantin Dobler and Gerard de Melo},
  year   = {2023},
  eprint = {2305.14481},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.14481v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.14481.svg)](https://codeofpaper.com/paper/2305.14481)

Discussion