Character-Aware Neural Language Models

Yoon Kim, Yacine Jernite, David Sontag, Alexander M. Rush
8/26/2015
cs.CLcs.NEstat.ML

Official code: yoonkim/lstm-char-cnn (837 stars, MIT, last commit 2016-08-24).

Abstract

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

View on arXivView PDF

Code Implementations(1)

yoonkim/lstm-char-cnnOfficial100%
MIT

LSTM language model with CNN over characters

837222Lua, ShellJun 13, 201510 years agoMIT

Similar Papers

Gated Word-Character Recurrent Language Model

90% match1 repo252016

Fully Character-Level Neural Machine Translation without Explicit Segmentation

85% match1 repo1432016

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

85% match1 repo32023

Visual Grounding Helps Learn Word Meanings in Low-Data Regimes

85% match1 repo102023

Evaluating Transformer Models and Human Behaviors on Chinese Character Naming

84% match2023

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

84% match1 repo1.8k2021

Cite this paper

@article{kim2015characteraware,
  title  = {Character-Aware Neural Language Models},
  author = {Yoon Kim and Yacine Jernite and David Sontag and Alexander M. Rush},
  year   = {2015},
  eprint = {1508.06615},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1508.06615}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1508.06615.svg)](https://codeofpaper.com/paper/1508.06615)

Discussion