RETSim: Resilient and Efficient Text Similarity

Marina Zhang, Owen Vallis, Aysegul Bumin, Tanay Vakharia, Elie Bursztein
11/1/2023

No official code found. Best implementation: unum-cloud/USearch (4,295 stars, Apache-2.0, last commit 2026-05-02). 1 more below.

Abstract

This paper introduces RETSim (Resilient and Efficient Text Similarity), a lightweight, multilingual deep learning model trained to produce robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. We demonstrate that RETSim is significantly more robust and accurate than MinHash and neural text embeddings, achieving new state-of-the-art performance on dataset deduplication, adversarial text retrieval benchmarks, and spam clustering tasks. We also introduce the W4NT3D benchmark (Wiki-40B 4dversarial Near-T3xt Dataset) for evaluating multilingual, near-duplicate text retrieval capabilities under adversarial settings. RETSim and the W4NT3D benchmark are open-sourced under the MIT License at https://github.com/google/unisim.

View on arXivView PDF

Code Implementations(2)

unum-cloud/USearchCommunity100%
Apache-2.0

Fast Open-Source Search & Clustering engine × for Vectors & Arbitrary Objects × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍

4,295344C, C#Feb 22, 20235 months agoApache-2.0
approximate-nearest-neighbor-searchclusteringdatabasefaissfull-text-search+13 more
google/unisimCommunity100%
ONNXMIT

UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.

1499Python, PureBasicAug 1, 20231 years agoMIT
deduplicationfuzzy-matchingnlponnxsimilarity+2 more

Similar Papers

Multilingual De-Duplication Strategies: Applying scalable similarity search with monolingual & multilingual embedding models

83% match2024

Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages

82% match2026

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

82% match1 repo22026

MTEB: Massive Text Embedding Benchmark

81% match1 repo3.5k2022

IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios

81% match1 repo12024

On the Robustness of Multilingual Text Embedding Rankings Across Learning Tasks, Languages, and Benchmark Datasets

81% match2026

Cite this paper

@article{zhang2023retsim,
  title  = {RETSim: Resilient and Efficient Text Similarity},
  author = {Marina Zhang and Owen Vallis and Aysegul Bumin and Tanay Vakharia and Elie Bursztein},
  year   = {2023},
  eprint = {2311.17264},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2311.17264},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.17264.svg)](https://codeofpaper.com/paper/2311.17264)

Discussion