Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Sergey Zagoruyko, Nikos Komodakis
12/12/2016
cs.CV

Official code: szagoruyko/attention-transfer (1,465 stars, PyTorch, last commit 2018-07-11).

Abstract

Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer

View on arXivView PDF

Code Implementations(1)

szagoruyko/attention-transferOfficial100%
PyTorch

Improving Convolutional Networks via Attention Transfer (ICLR 2017)

1,465273Python, Jupyter NotebookJan 17, 20178 years ago
attentiondeep-learningknowledge-distillationpytorch

Similar Papers

ELA: Efficient Local Attention for Deep Convolutional Neural Networks

87% match2024

BA-Net: Bridge Attention in Deep Neural Networks

86% match2024

Partial Convolution Meets Visual Attention

86% match2025

Attention mechanisms in neural networks

85% match2026

RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks

85% match1 repo1902023

AttZoom: Attention Zoom for Better Visual Features

85% match2025

Cite this paper

@article{zagoruyko2016paying,
  title  = {Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer},
  author = {Sergey Zagoruyko and Nikos Komodakis},
  year   = {2016},
  eprint = {1612.03928},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1612.03928}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1612.03928.svg)](https://codeofpaper.com/paper/1612.03928)

Discussion