Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models

Blind individuals, who by necessity depend on screen readers to interact with computers, face considerable challenges in navigating the diverse and complex graphical user interfaces of diferent computer applications. The heterogeneity of various application interfaces often requires blind users to remember diferent keyboard combinations and navigation methods to use each application efectively. To alleviate this signifcant interaction burden imposed by heterogeneous application interfaces, we present Savant, a novel assistive technology powered by large language models (LLMs) that allows blind screen reader users to interact uniformly with any application interface through natural language. Novelly, Savant can automate a series of tedious screen reader actions on the control elements of the application when prompted by a natural language command from the user. These commands can be fexible in the sense that the user is not strictly required to specify the exact names of the control elements in the command. A user study evaluation of Savant with 11 blind participants demonstrated signifcant improvements in interaction efciency and usability compared to current practices.