Mixture of Small and Large Models for Chinese Spelling Check

Ziheng Qiao, Houquan Zhou, Zhenghua Li
6/7/2025
cs.CL

Official code: zhqiao-nlp/MSLLM (6 stars, last commit 2025-06-16).

Abstract

In the era of large language models (LLMs), the Chinese Spelling Check (CSC) task has seen various LLM methods developed, yet their performance remains unsatisfactory. In contrast, fine-tuned BERT-based models, relying on high-quality in-domain data, show excellent performance but suffer from edit pattern overfitting. This paper proposes a novel dynamic mixture approach that effectively combines the probability distributions of small models and LLMs during the beam search decoding phase, achieving a balanced enhancement of precise corrections from small models and the fluency of LLMs. This approach also eliminates the need for fine-tuning LLMs, saving significant time and resources, and facilitating domain adaptation. Comprehensive experiments demonstrate that our mixture approach significantly boosts error correction capabilities, achieving state-of-the-art results across multiple datasets. Our code is available at https://github.com/zhqiao-nlp/MSLLM.

View on arXivView PDF

Code Implementations(1)

zhqiao-nlp/MSLLMOfficial100%

Mixture of Small and Large Models for Chinese Spelling Check

60Shell, PythonMay 28, 20251 years ago

Similar Papers

A Simple yet Effective Training-free Prompt-free Approach to Chinese Spelling Correction Based on Large Language Models

85% match1 repo892024

Unveiling the Impact of Multimodal Features on Chinese Spelling Correction: From Analysis to Design

85% match1 repo62025

C-LLM: Learn to Check Chinese Spelling Errors Character by Character

84% match1 repo302024

Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking

84% match2024

EdaCSC: Two Easy Data Augmentation Methods for Chinese Spelling Correction

84% match2024

Refining Corpora from a Model Calibration Perspective for Chinese Spelling Correction

82% match2024

Cite this paper

@article{qiao2025mixture,
  title  = {Mixture of Small and Large Models for Chinese Spelling Check},
  author = {Ziheng Qiao and Houquan Zhou and Zhenghua Li},
  year   = {2025},
  eprint = {2506.06887},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.06887v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.06887.svg)](https://codeofpaper.com/paper/2506.06887)

Discussion