by A groundbreaking software named WorldScribe, developed by researchers at the University of Michigan, is set to transform the lives of visually impaired individuals by narrating the surroundings in real time as captured by a camera. This innovative technology will be showcased at the 2024 ACM Symposium on User Interface Software and Technology in Pittsburgh.
The study, titled “WorldScribe: Towards Context-Aware Live Visual Descriptions,” was published on the arXiv preprint server.
WorldScribe harnesses the power of generative AI (GenAI) language models to interpret camera images and generate text and audio descriptions instantaneously. The tool can customize the level of detail based on user commands or the duration of an object’s presence in the camera frame. It also adapts to noisy environments by automatically adjusting the volume.
During the trial study, Sam Rau, a visually impaired participant, shared his excitement about the tool: “As a blind person, I don’t have any concept of sight, but when I tried the tool, I felt like I was getting a picture of the real world. I was thrilled to experience the colors and textures that I wouldn’t have access to otherwise.”
Rau further explained, “As a blind person, we’re constantly piecing together what’s going on around us, which can take a lot of mental effort. This tool provides us with the information right away, allowing us to focus on being human rather than figuring out our surroundings.”
WorldScribe uses three different AI language models to generate descriptions: the YOLO World model for quick, simple descriptions of objects that briefly appear in the camera frame; GPT-4, the model behind ChatGPT, for detailed descriptions of objects that remain in the frame for an extended period; and Moondream for an intermediate level of detail.
Anhong Guo, an assistant professor of computer science and engineering and a corresponding author of the study, explained, “Most existing assistive technologies that leverage AI focus on specific tasks or require turn-by-turn interaction. We saw an opportunity to use increasingly capable AI models to create automated and adaptive descriptions in real-time.”
WorldScribe can also respond to user-provided tasks or queries, such as prioritizing descriptions of specific objects. However, some study participants noted that the tool had difficulty detecting certain objects, like an eyedropper bottle.
Rau believes the tool still needs refinement for everyday use but would use it daily if it could be integrated into smart glasses or another wearable device. The researchers have applied for patent protection and are seeking partners to help develop and bring the technology to market. Guo is also an assistant professor of information within U-M’s School of Information.
*Note:
1. Source: Coherent Market Insights, Public sources, Desk research
2. We have leveraged AI tools to mine information and compile it
