Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
Jiateng Liu, Rushi Wang, Bingxuan Li, Kunlun Zhu, Yifan Shen + 4 more
TLDR
This paper augments Nielsen's usability heuristics to improve the reliability and generalization of computer-use agents on diverse interfaces.
Key contributions
- Reimagines Nielsen's 10 usability heuristics to improve computer-use agent reliability.
- Identifies agent-specific interface design failures and proposes additive augmentations.
- Introduces UI-Verse, a new benchmark with varied interfaces applying different heuristics.
- Demonstrates augmented heuristics boost agent task completion and efficiency without human usability regressions.
Why it matters
This work offers a novel approach to enhance computer-use agent reliability by focusing on interface design rather than just agent capability. It provides practical guidelines and a benchmark for creating more agent-compatible UIs, leading to more robust and generalizable agents.
Original Abstract
Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent compatible interface design offers a complementary path by aligning interaction semantics with agent prior knowledge. In this paper, we revisit Nielsen 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics consistently improve task completion and modestly improve efficiency, with combined heuristics yielding further gains. Human studies further show that these designs preserve the original interaction workflow without observable usability regressions. Overall, our findings highlight interface design as a practical complementary avenue for improving the reliability and generalization of computer use agents.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.