SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish

Dmytro Kalpakchi, Johan Boye
4/27/2023
cs.CL

No confident code implementation found for this paper.

Abstract

We present SweCTRL-Mini, a large Swedish language model that can be used for inference and fine-tuning on a single consumer-grade GPU. The model is based on the CTRL architecture by Keskar, McCann, Varshney, Xiong, and Socher (2019), which means that users of the SweCTRL-Mini model can control the genre of the generated text by inserting special tokens in the generation prompts. SweCTRL-Mini is trained on a subset of the Swedish part of the mC4 corpus and a set of Swedish novels. In this article, we provide (1) a detailed account of the utilized training data and text pre-processing steps, to the extent that it is possible to check whether a specific phrase/source was a part of the training data, and (2) an evaluation of the model on both discriminative tasks, using automatic evaluation methods, and generative tasks, using human referees. We also compare the generative capabilities of the model with those of GPT-3. SweCTRL-Mini is fully open and available for download.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Personalized Text Generation with Fine-Grained Linguistic Control

86% match1 repo72024

Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation

86% match2024

Controllable Text Generation for Large Language Models: A Survey

86% match1 repo2042024

Controlled Text Generation for Black-box Language Models via Score-based Progressive Editor

86% match1 repo92023

GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content

86% match1 repo92023

LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning

85% match1 repo112023

Cite this paper

@article{kalpakchi2023swectrlmini,
  title  = {SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish},
  author = {Dmytro Kalpakchi and Johan Boye},
  year   = {2023},
  eprint = {2304.13994},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.13994v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.13994.svg)](https://codeofpaper.com/paper/2304.13994)

Discussion