Zero-shot Cross-lingual Transfer is Under-specified Optimization

Shijie Wu, Benjamin Van Durme, Mark Dredze
7/12/2022
cs.CL

No official code found. Best implementation: shijie-wu/crosslingual-nlp (92 stars, MIT, last commit 2023-09-13).

Abstract

Pretrained multilingual encoders enable zero-shot cross-lingual transfer, but often produce unreliable models that exhibit high performance variance on the target language. We postulate that this high variance results from zero-shot cross-lingual transfer solving an under-specified optimization problem. We show that any linear-interpolated model between the source language monolingual model and source + target bilingual model has equally low source language generalization error, yet the target language generalization error reduces smoothly and linearly as we move from the monolingual to bilingual model, suggesting that the model struggles to identify good solutions for both source and target languages using the source language alone. Additionally, we show that zero-shot solution lies in non-flat region of target language error generalization surface, causing the high variance.

View on arXivView PDF

Code Implementations(1)

shijie-wu/crosslingual-nlpCommunity100%
MIT

This repo supports various cross-lingual transfer learning & multilingual NLP models.

926Shell, PythonAug 31, 20193 years agoMIT
crosslingual-transfermultilingual-nlpnatural-language-processing

Similar Papers

DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer

89% match2023

Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution

89% match2023

Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization

88% match1 repo102023

Empirical study of pretrained multilingual language models for zero-shot cross-lingual knowledge transfer in generation

86% match2023

When English Isn't the Best Teacher: Source Language Effects in Cross-Lingual In-Context Learning

86% match2026

Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

86% match2024

Cite this paper

@article{wu2022zeroshot,
  title  = {Zero-shot Cross-lingual Transfer is Under-specified Optimization},
  author = {Shijie Wu and Benjamin Van Durme and Mark Dredze},
  year   = {2022},
  eprint = {2207.05666},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2207.05666v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2207.05666.svg)](https://codeofpaper.com/paper/2207.05666)

Discussion