Enhanced Transformer Architecture for Natural Language Processing

Woohyeon Moon, Taeyoung Kim, Bumgeun Park, Dongsoo Har
10/17/2023
cs.CLcs.AI

No confident code implementation found for this paper. 8 possible matches can be shown below.

Abstract

Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires a lot of training resources such as computing capacity. In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset. As a result, the Enhanced Transformer achieves 202.96% higher BLEU score as compared to the original transformer with the translation dataset.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 8 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering

86% match2023

Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment

86% match2026

Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey

86% match2023

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

86% match2023

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

86% match1 repo2402023

idT5: Indonesian Version of Multilingual T5 Transformer

85% match2023

Cite this paper

@article{moon2023enhanced,
  title  = {Enhanced Transformer Architecture for Natural Language Processing},
  author = {Woohyeon Moon and Taeyoung Kim and Bumgeun Park and Dongsoo Har},
  year   = {2023},
  eprint = {2310.10930},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.10930v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.10930.svg)](https://codeofpaper.com/paper/2310.10930)

Discussion