VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments

Bufang Yang, Lixing He, Kaiwei Liu, Zhenyu Yan
4/3/2024
cs.CVcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

Individuals with visual impairments, encompassing both partial and total difficulties in visual perception, are referred to as visually impaired (VI) people. An estimated 2.2 billion individuals worldwide are affected by visual impairments. Recent advancements in multi-modal large language models (MLLMs) have showcased their extraordinary capabilities across various domains. It is desirable to help VI individuals with MLLMs' great capabilities of visual understanding and reasoning. However, it is challenging for VI people to use MLLMs due to the difficulties in capturing the desirable images to fulfill their daily requests. For example, the target object is not fully or partially placed in the image. This paper explores how to leverage MLLMs for VI individuals to provide visual-question answers. VIAssist can identify undesired images and provide detailed actions. Finally, VIAssist can provide reliable answers to users' queries based on the images. Our results show that VIAssist provides +0.21 and +0.31 higher BERTScore and ROUGE scores than the baseline, respectively.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People

88% match2026

Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users

87% match2025

VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Models

87% match2024

Vision-Language Models under Cultural and Inclusive Considerations

83% match2024

BLINK: Multimodal Large Language Models Can See but Not Perceive

83% match1 repo1702024

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

82% match1 repo442026

Cite this paper

@article{yang2024viassist,
  title  = {VIAssist: Adapting Multi-modal Large Language Models for Users with Visual Impairments},
  author = {Bufang Yang and Lixing He and Kaiwei Liu and Zhenyu Yan},
  year   = {2024},
  eprint = {2404.02508},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2404.02508v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2404.02508.svg)](https://codeofpaper.com/paper/2404.02508)

Discussion