Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

Reuben Dorent, Nazim Haouchine, Fryderyk Kögl, Samuel Joutard, Parikshit Juvekar +7 more
9/15/2023
cs.CV

Official code: ReubenDo/MHVAE (16 stars, MIT, last commit 2023-10-11).

Abstract

We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available \url{https://github.com/ReubenDo/MHVAE}.

View on arXivView PDF

Code Implementations(1)

ReubenDo/MHVAEOfficial100%
MIT
160PythonJul 20, 20233 years agoMIT

Similar Papers

Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts

90% match1 repo162024

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

82% match1 repo12023

Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation

80% match2022

High Fidelity Image Synthesis With Deep VAEs In Latent Space

80% match2 repos292023

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

79% match1 repo1042020

A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion

79% match2022

Cite this paper

@article{dorent2023unified,
  title  = {Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations},
  author = {Reuben Dorent and Nazim Haouchine and Fryderyk Kögl and Samuel Joutard and Parikshit Juvekar and Erickson Torio and Alexandra Golby and Sebastien Ourselin and Sarah Frisken and Tom Vercauteren and Tina Kapur and William M. Wells},
  year   = {2023},
  eprint = {2309.08747},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.08747v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.08747.svg)](https://codeofpaper.com/paper/2309.08747)

Discussion