Translating Visual Art into Music

Maximilian Müller-Eberstein, Nanne van Noord
9/3/2019
cs.CVcs.HCcs.LG

Official code: personads/synvae (34 stars, MIT, last commit 2019-07-28).

Abstract

The Synesthetic Variational Autoencoder (SynVAE) introduced in this research is able to learn a consistent mapping between visual and auditive sensory modalities in the absence of paired datasets. A quantitative evaluation on MNIST as well as the Behance Artistic Media dataset (BAM) shows that SynVAE is capable of retaining sufficient information content during the translation while maintaining cross-modal latent space consistency. In a qualitative evaluation trial, human evaluators were furthermore able to match musical samples with the images which generated them with accuracies of up to 73%.

View on arXivView PDF

Code Implementations(1)

personads/synvaeOfficial100%
MIT

Translating Visual Works of Art into Music (ICCVW 2019)

348Shell, PythonJan 21, 20197 years agoMIT

Similar Papers

Multimodal Action Quality Assessment

83% match2 repos392024

V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models

83% match1 repo292023

MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing

82% match1 repo262024

VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning

82% match2025

X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning

81% match2 repos11.3k2023

Detached and Interactive Multimodal Learning

80% match1 repo62024

Cite this paper

@article{mllereberstein2019translating,
  title  = {Translating Visual Art into Music},
  author = {Maximilian Müller-Eberstein and Nanne van Noord},
  year   = {2019},
  eprint = {1909.01218},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1909.01218}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1909.01218.svg)](https://codeofpaper.com/paper/1909.01218)

Discussion