PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training

Rongjie Yi, Xiang Li, Weikai Xie, Zhenyan Lu, Chenghua Wang +4 more
11/7/2024
cs.CLcs.AIcs.LG

Official code: UbiquitousLearning/PhoneLM (70 stars, Apache-2.0, last commit 2024-11-16).

Abstract

The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at https://github.com/UbiquitousLearning/PhoneLM.

View on arXivView PDF

Code Implementations(1)

UbiquitousLearning/PhoneLMOfficial100%
Apache-2.0
703PythonNov 2, 20241 years agoApache-2.0

Similar Papers

SlimLM: An Efficient Small Language Model for On-Device Document Assistance

85% match2024

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

83% match1 repo1.5k2024

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

83% match2026

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

83% match1 repo6672024

GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices

83% match2025

Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection

82% match2025

Cite this paper

@article{yi2024phonelman,
  title  = {PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training},
  author = {Rongjie Yi and Xiang Li and Weikai Xie and Zhenyan Lu and Chenghua Wang and Ao Zhou and Shangguang Wang and Xiwen Zhang and Mengwei Xu},
  year   = {2024},
  eprint = {2411.05046},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.05046v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.05046.svg)](https://codeofpaper.com/paper/2411.05046)

Discussion