BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba

Ling Yue, Sixue Xing, Yingzhou Lu, Tianfan Fu
8/5/2024
cs.CL

No official code found. Best implementation: LeoYML/BioMamba (14 stars, MIT, last commit 2026-04-13).

Abstract

The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.

View on arXivView PDF

Code Implementations(1)

LeoYML/BioMambaCommunity100%
MIT

BioMamba is a pre-trained Mamba architecture model for the biomedical domain.

143Shell, PythonAug 4, 20246 months agoMIT

Similar Papers

BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text

84% match1 repo6442024

Benchmarking large language models for biomedical natural language processing applications and recommendations

84% match1 repo182023

BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine

83% match1 repo102024

MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model

83% match2025

BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment

83% match2025

BioMNER: A Dataset for Biomedical Method Entity Recognition

83% match2024

Cite this paper

@article{yue2024biomamba,
  title  = {BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba},
  author = {Ling Yue and Sixue Xing and Yingzhou Lu and Tianfan Fu},
  year   = {2024},
  eprint = {2408.02600},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.02600v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.02600.svg)](https://codeofpaper.com/paper/2408.02600)

Discussion