Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models

Averi Bates, Ryan Vavricka, Shane Carleton, Ruosi Shao, Chongle Pan
3/15/2025
cs.SEcs.LG

No official code found. Best implementation: haotian-liu/LLaVA (24,897 stars, Apache-2.0, last commit 2024-08-12).

Abstract

The Unified Modeling Language is a standardized visual language widely used for modeling and documenting the design of software systems. Although many tools generate UML diagrams from UML code, generating executable UML code from image-based UML diagrams remains challenging. This paper proposes a new approach to generate UML code using a large multimodal language model automatically. Synthetic UML activity and sequence diagram datasets were created to train and test the model. We compared standard fine-tuning with LoRA techniques to optimize base models. The experiments measured code generation accuracy across different model sizes and training strategies. These results demonstrated that domain-adapted MM-LLMs perform for UML code generation automation, whereby, at the best model, it achieved BLEU and SSIM scores of 0.779 and 0.942 on sequence diagrams. This will enable the modernization of legacy systems and decrease the manual effort in software development workflows.

View on arXivView PDF

Code Implementations(1)

haotian-liu/LLaVACommunity100%
Apache-2.0

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

24,8972,774CSS, HTMLApr 17, 20232 years agoApache-2.0
chatbotchatgptfoundation-modelsgpt-4instruction-tuning+8 more

Similar Papers

Multilingual Multimodal Software Developer for Code Generation

85% match2025

Model Generation with LLMs: From Requirements to UML Sequence Diagrams

80% match2024

Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction

78% match2025

Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs

78% match2025

How an unintended Side Effect of a Research Project led to Boosting the Power of UML

77% match2025

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

77% match2026

Cite this paper

@article{bates2025unified,
  title  = {Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models},
  author = {Averi Bates and Ryan Vavricka and Shane Carleton and Ruosi Shao and Chongle Pan},
  year   = {2025},
  eprint = {2503.12293},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2503.12293v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2503.12293.svg)](https://codeofpaper.com/paper/2503.12293)

Discussion