Improving Transformers using Faithful Positional Encoding

Tsuyoshi Idé, Jokin Labaien, Pin-Yu Chen
5/15/2024
cs.LG

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Alternative positional encoding functions for neural transformers

89% match2025

Positional Encoding in Transformer-Based Time Series Models: A Survey

88% match1 repo252025

Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary

86% match1 repo12024

Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization

85% match2025

Input Compression with Positional Consistency for Efficient Training and Inference of Transformer Neural Networks

84% match1 repo42023

U-shaped Transformer: Retain High Frequency Context in Time Series Analysis

82% match2023

Cite this paper

@article{id2024improving,
  title  = {Improving Transformers using Faithful Positional Encoding},
  author = {Tsuyoshi Idé and Jokin Labaien and Pin-Yu Chen},
  year   = {2024},
  eprint = {2405.09061},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.09061v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.09061.svg)](https://codeofpaper.com/paper/2405.09061)

Discussion