TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese

Nicholas Kluge Corrêa, Sophia Falk, Shiza Fatimah, Aniket Sen, Nythamar de Oliveira
1/30/2024
cs.CLcs.LG

Official code: Nkluge-correa/TeenyTinyLlama (46 stars, Apache-2.0, last commit 2026-02-24).

Abstract

Large language models (LLMs) have significantly advanced natural language processing, but their progress has yet to be equal across languages. While most LLMs are trained in high-resource languages like English, multilingual models generally underperform monolingual ones. Additionally, aspects of their multilingual foundation sometimes restrict the byproducts they produce, like computational demands and licensing regimes. In this study, we document the development of open-foundation models tailored for use in low-resource settings, their limitations, and their benefits. This is the TeenyTinyLlama pair: two compact models for Brazilian Portuguese text generation. We release them under the permissive Apache 2.0 license on GitHub and Hugging Face for community use and further development. See https://github.com/Nkluge-correa/TeenyTinyLlama

View on arXivView PDF

Code Implementations(1)

Nkluge-correa/TeenyTinyLlamaOfficial100%
Apache-2.0

A pair of tiny foundational models trained in Brazilian Portuguese.🦙🦙

469Python, Jupyter NotebookJan 27, 20247 months agoApache-2.0
language-modellargellamallama2portuguese-brazilian

Similar Papers

Foundation Models for Low-Resource Language Education (Vision Paper)

84% match2024

MiniLingua: A Small Open-Source LLM for European Languages

83% match2025

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

83% match1 repo2612024

Tucano 2 Cool: Better Open Source LLMs for Portuguese

83% match2026

Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?

82% match2025

The Open Source Advantage in Large Language Models (LLMs)

82% match2024

Cite this paper

@article{corra2024teenytinyllama,
  title  = {TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese},
  author = {Nicholas Kluge Corrêa and Sophia Falk and Shiza Fatimah and Aniket Sen and Nythamar de Oliveira},
  year   = {2024},
  eprint = {2401.16640},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.16640v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.16640.svg)](https://codeofpaper.com/paper/2401.16640)

Discussion