Differentiable Hierarchical Visual Tokenization

Marius Aasan, Martine Hjelkrem-Tan, Nico Catalano, Changkyu Choi, Adín Ramírez Rivera
11/4/2025
cs.CV

No official code found. Best implementation: dsb-ifi/dHT (45 stars, MIT, last commit 2025-11-26). 1 more below.

Abstract

Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

View on arXivView PDF

Code Implementations(2)

dsb-ifi/dHTCommunity67%
MIT

Differentiable Hierarchical Visual Tokenization

451PythonMay 16, 202510 months agoMIT
mariuaas/diffHTCommunity61%
MIT

Differentiable Hierarchical Visual Tokenization

00Cuda, PythonMay 16, 20251 years agoMIT

Similar Papers

A Spitting Image: Modular Superpixel Tokenization in Vision Transformers

84% match1 repo232024

Factorized Visual Tokenization and Generation

82% match1 repo592024

Wavelet-Based Image Tokenizer for Vision Transformers

82% match2024

Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment

82% match1 repo482026

Vision Transformers with Natural Language Semantics

82% match2024

CAT: Content-Adaptive Image Tokenization

81% match2025

Cite this paper

@article{aasan2025differentiable,
  title  = {Differentiable Hierarchical Visual Tokenization},
  author = {Marius Aasan and Martine Hjelkrem-Tan and Nico Catalano and Changkyu Choi and Adín Ramírez Rivera},
  year   = {2025},
  eprint = {2511.02652},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2511.02652v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2511.02652.svg)](https://codeofpaper.com/paper/2511.02652)

Discussion