Multimodal Benchmarking and Recommendation of Text-to-Image Generation Models

Kapil Wanaskar, Gaytri Jena, Magdalini Eirinaki
5/6/2025
cs.GRcs.AIcs.IR

No official code found. Best implementation: kapilw25/Evaluation_generated_images (1 star, last commit 2025-05-21).

Abstract

This work presents an open-source unified benchmarking and evaluation framework for text-to-image generation models, with a particular focus on the impact of metadata augmented prompts. Leveraging the DeepFashion-MultiModal dataset, we assess generated outputs through a comprehensive set of quantitative metrics, including Weighted Score, CLIP (Contrastive Language Image Pre-training)-based similarity, LPIPS (Learned Perceptual Image Patch Similarity), FID (Frechet Inception Distance), and retrieval-based measures, as well as qualitative analysis. Our results demonstrate that structured metadata enrichments greatly enhance visual realism, semantic fidelity, and model robustness across diverse text-to-image architectures. While not a traditional recommender system, our framework enables task-specific recommendations for model selection and prompt design based on evaluation metrics.

View on arXivView PDF

Code Implementations(1)

kapilw25/Evaluation_generated_imagesCommunity100%
10PythonMar 18, 20251 years ago

Similar Papers

UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation

86% match1 repo1392025

Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark

85% match1 repo202024

MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective

85% match1 repo1192024

LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

84% match1 repo202025

Multi-Modal Language Models as Text-to-Image Model Evaluators

84% match2025

EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models

84% match1 repo192024

Cite this paper

@article{wanaskar2025multimodal,
  title  = {Multimodal Benchmarking and Recommendation of Text-to-Image Generation Models},
  author = {Kapil Wanaskar and Gaytri Jena and Magdalini Eirinaki},
  year   = {2025},
  eprint = {2505.04650},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2505.04650v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2505.04650.svg)](https://codeofpaper.com/paper/2505.04650)

Discussion