Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs
Joshua Owusu Ansah, Anuj Kapoor, Ayush Khanna, Manvika Vinod, Precious Njeck + 1 more
TLDR
Insight uses LLMs to provide natural language interaction and real-time screen summarization, significantly improving mobile accessibility for BVI users.
Key contributions
- Introduces Insight, an Android accessibility service leveraging LLMs for blind and visually impaired users.
- Enables natural language interaction and real-time screen summarization for mobile devices.
- Experimental study shows Insight reduces mental effort and task time over traditional methods.
- Demonstrates LLM-based interfaces significantly improve mobile accessibility and suggests hybrid solutions.
Why it matters
This paper introduces Insight, an LLM-powered mobile accessibility service that significantly improves user experience for BVI individuals. It demonstrates how natural language interaction can reduce mental effort and task time compared to traditional gesture-based systems. The findings pave the way for more intuitive and inclusive mobile interfaces.
Original Abstract
This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.