Towards No.1 in CLUE Semantic Matching Challenge: Pre-trained Language Model Erlangshen with Propensity-Corrected Loss

Junjie Wang, Yuxiang Zhang, Ping Yang, Ruyi Gan
8/5/2022
cs.CL

Official code: IDEA-CCNL/Fengshenbang-LM (4,124 stars, Apache-2.0, PyTorch, last commit 2024-08-13).

Abstract

This report describes a pre-trained language model Erlangshen with propensity-corrected loss, the No.1 in CLUE Semantic Matching Challenge. In the pre-training stage, we construct a dynamic masking strategy based on knowledge in Masked Language Modeling (MLM) with whole word masking. Furthermore, by observing the specific structure of the dataset, the pre-trained Erlangshen applies propensity-corrected loss (PCL) in the fine-tuning phase. Overall, we achieve 72.54 points in F1 Score and 78.90 points in Accuracy on the test set. Our code is publicly available at: https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/hf-ds/fengshen/examples/clue_sim.

View on arXivView PDF

Code Implementations(1)

IDEA-CCNL/Fengshenbang-LMOfficial100%
PyTorchApache-2.0

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。

4,124374C, C++Oct 28, 20212 years agoApache-2.0
aigcchinese-nlpdistributed-trainingmultimodalpretrained-models+2 more

Similar Papers

Pre-Training with Whole Word Masking for Chinese BERT

86% match1 repo10.2k2019

Weighted Sampling for Masked Language Modeling

85% match2023

Revisiting Pre-Trained Models for Chinese Natural Language Processing

85% match1 repo7222020

Self-Evolution Learning for Discriminative Language Model Pretraining

85% match2023

Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

83% match1 repo22024

Parameter-Efficient Fine-Tuning without Introducing New Latency

83% match2023

Cite this paper

@article{wang2022towards,
  title  = {Towards No.1 in CLUE Semantic Matching Challenge: Pre-trained Language Model Erlangshen with Propensity-Corrected Loss},
  author = {Junjie Wang and Yuxiang Zhang and Ping Yang and Ruyi Gan},
  year   = {2022},
  eprint = {2208.02959},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2208.02959v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2208.02959.svg)](https://codeofpaper.com/paper/2208.02959)

Discussion