Contrastive Learning in Distilled Models

Valerie Lim, Kai Wen Ng, Kenneth Lim
1/23/2024
cs.CL

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

Natural Language Processing models like BERT can provide state-of-the-art word embeddings for downstream NLP tasks. However, these models yet to perform well on Semantic Textual Similarity, and may be too large to be deployed as lightweight edge applications. We seek to apply a suitable contrastive learning method based on the SimCSE paper, to a model architecture adapted from a knowledge distillation based model, DistilBERT, to address these two issues. Our final lightweight model DistilFace achieves an average of 72.1 in Spearman's correlation on STS tasks, a 34.2 percent improvement over BERT base.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Distillation Contrastive Decoding: Improving LLMs Reasoning with Contrastive Decoding and Distillation

83% match1 repo352024

Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matching

82% match2025

Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective

82% match2023

2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings

82% match2025

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

82% match2024

Basic Reading Distillation

81% match2025

Cite this paper

@article{lim2024contrastive,
  title  = {Contrastive Learning in Distilled Models},
  author = {Valerie Lim and Kai Wen Ng and Kenneth Lim},
  year   = {2024},
  eprint = {2401.12472},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.12472v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.12472.svg)](https://codeofpaper.com/paper/2401.12472)

Discussion