Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers

Romain Menegaux, Emmanuel Jehanno, Margot Selosse, Julien Mairal
4/21/2023
cs.LG

No official code found. Best implementation: inria-thoth/csa (10 stars, MIT, last commit 2023-06-14).

Abstract

We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology.

View on arXivView PDF

Code Implementations(1)

inria-thoth/csaCommunity100%
MIT

Official Pytorch implementation of Chromatic Graph Transformers

102Shell, PythonApr 20, 20233 years agoMIT

Similar Papers

Towards Mechanistic Interpretability of Graph Transformers via Attention Graphs

87% match1 repo302025

An end-to-end attention-based approach for learning on graphs

87% match1 repo862024

SignGT: Signed Attention-based Graph Transformer for Graph Representation Learning

86% match2023

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

86% match1 repo132025

AnchorGT: Efficient and Flexible Attention Architecture for Scalable Graph Transformers

85% match2024

Graph-Aware Isomorphic Attention for Adaptive Dynamics in Transformers

85% match1 repo702025

Cite this paper

@article{menegaux2023selfattention,
  title  = {Self-Attention in Colors: Another Take on Encoding Graph Structure in Transformers},
  author = {Romain Menegaux and Emmanuel Jehanno and Margot Selosse and Julien Mairal},
  year   = {2023},
  eprint = {2304.10933},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.10933v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.10933.svg)](https://codeofpaper.com/paper/2304.10933)

Discussion