Metrics
16 Record Views
Conference proceeding
AXNav: Replaying Accessibility Tests from Natural Language
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 962
ACM Conferences
CHI '24: Conference on Human Factors in Computing Systems (Honolulu, Hawaii, USA, 05/11/2024–05/16/2024)
05/11/2024
Web of Science ID: WOS:001266059701032
Abstract
Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.
Files and links (1)
Related links
Details
- Title
- AXNav
- Publication Details
- CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 962
- Resource Type
- Conference proceeding
- Conference
- CHI '24: Conference on Human Factors in Computing Systems (Honolulu, Hawaii, USA, 05/11/2024–05/16/2024)
- Publisher
- Association for Computing Machinery
- Series
- ACM Conferences
- Format
- link
- Number of pages
- 16
- Copyright
- © 2024 Copyright held by the owner/author(s).
- Identifiers
- WOS:001266059701032; 99380586995206600
- Academic Unit
- Cybersecurity and Information Technology; Hal Marcus College of Science and Engineering
- Language
- English