Introduction to Transformers: an NLP Perspective

Tong Xiao, Jingbo Zhu
11/29/2023
cs.CLcs.AIcs.LG

Official code: NiuTrans/Introduction-to-Transformers (53 stars, CC0-1.0, last commit 2023-12-21).

Abstract

Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.

View on arXivView PDF

Code Implementations(1)

NiuTrans/Introduction-to-TransformersOfficial100%
CC0-1.0

An introduction to basic concepts of Transformers and key techniques of their recent advances.

534Nov 23, 20232 years agoCC0-1.0
deep-learningnatural-language-processingtransformers

Similar Papers

An Introduction to Transformers

90% match2023

Learning Transformer Programs

89% match1 repo1672023

A Survey on Transformers in Reinforcement Learning

88% match2023

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

88% match2023

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures

87% match1 repo12024

Why transformers are obviously good models of language

87% match2024

Cite this paper

@article{xiao2023introduction,
  title  = {Introduction to Transformers: an NLP Perspective},
  author = {Tong Xiao and Jingbo Zhu},
  year   = {2023},
  eprint = {2311.17633},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.17633v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.17633.svg)](https://codeofpaper.com/paper/2311.17633)

Discussion