Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu +3 more
7/1/2020
cs.IRcs.CLcs.LG

Official code: microsoft/ANCE (385 stars, MIT, last commit 2026-01-06).

Abstract

Conducting text retrieval in a dense learned representation space has many intriguing advantages over sparse retrieval. Yet the effectiveness of dense retrieval (DR) often requires combination with sparse retrieval. In this paper, we identify that the main bottleneck is in the training mechanisms, where the negative instances used in training are not representative of the irrelevant documents in testing. This paper presents Approximate nearest neighbor Negative Contrastive Estimation (ANCE), a training mechanism that constructs negatives from an Approximate Nearest Neighbor (ANN) index of the corpus, which is parallelly updated with the learning process to select more realistic negative training instances. This fundamentally resolves the discrepancy between the data distribution used in the training and testing of DR. In our experiments, ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and sparse retrieval baselines. It nearly matches the accuracy of sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned representation space and provides almost 100x speed-up.

View on arXivView PDF

Code Implementations(1)

microsoft/ANCEOfficial100%
MIT

A novel embedding training algorithm leveraging ANN search and achieved SOTA retrieval on Trec DL 2019 and OpenQA benchmarks

38551Shell, PythonJun 4, 20209 months agoMIT

Similar Papers

Non-negative Elastic Net Decoding for Information Retrieval

87% match2026

Optimizing Dense Retrieval Model Training with Hard Negatives

86% match1 repo1272021

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

85% match1 repo132024

Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling

85% match1 repo602021

Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization

85% match1 repo132023

LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval

84% match1 repo392023

Cite this paper

@article{xiong2020approximate,
  title  = {Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval},
  author = {Lee Xiong and Chenyan Xiong and Ye Li and Kwok-Fung Tang and Jialin Liu and Paul Bennett and Junaid Ahmed and Arnold Overwijk},
  year   = {2020},
  eprint = {2007.00808},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2007.00808}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2007.00808.svg)](https://codeofpaper.com/paper/2007.00808)

Discussion