A Survey of Pre-trained Language Models for Processing Scientific Text

Xanh Ho, Anh Khoa Duong Nguyen, An Tuan Dao, Junfeng Jiang, Yuki Chida +4 more
1/31/2024
cs.CL

No official code found. Best implementation: Alab-NII/Awesome-SciLM (46 stars, last commit 2024-02-22).

Abstract

The number of Language Models (LMs) dedicated to processing scientific text is on the rise. Keeping pace with the rapid growth of scientific LMs (SciLMs) has become a daunting task for researchers. To date, no comprehensive surveys on SciLMs have been undertaken, leaving this issue unaddressed. Given the constant stream of new SciLMs, appraising the state-of-the-art and how they compare to each other remain largely unknown. This work fills that gap and provides a comprehensive review of SciLMs, including an extensive analysis of their effectiveness across different domains, tasks and datasets, and a discussion on the challenges that lie ahead.

View on arXivView PDF

Code Implementations(1)

Alab-NII/Awesome-SciLMCommunity100%

Pre-trained Language Model for Scientific Text

469Sep 13, 20232 years ago
bio-lmschem-lmsgeoscience-lmlanguage-modelslarge-language-models+3 more

Similar Papers

Towards Efficient Large Language Models for Scientific Text: A Review

88% match2024

SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models

85% match1 repo282024

History, Development, and Principles of Large Language Models-An Introductory Survey

85% match2024

Scientific Large Language Models: A Survey on Biological & Chemical Domains

85% match1 repo3582024

The Empowerment of Science of Science by Large Language Models: New Tools and Methods

85% match2025

Mastering the Craft of Data Synthesis for CodeLLMs

84% match1 repo82024

Cite this paper

@article{ho2024survey,
  title  = {A Survey of Pre-trained Language Models for Processing Scientific Text},
  author = {Xanh Ho and Anh Khoa Duong Nguyen and An Tuan Dao and Junfeng Jiang and Yuki Chida and Kaito Sugimoto and Huy Quoc To and Florian Boudin and Akiko Aizawa},
  year   = {2024},
  eprint = {2401.17824},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.17824v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.17824.svg)](https://codeofpaper.com/paper/2401.17824)

Discussion