Science-Based Methodologies for Evaluating the Human-Centeredness of AI Systems
Jia Zhou, Martin Schrepp, Helmut Degen, Vincent G. Duffy, Qin Gao, George Margetis, Margaret McKay, Fiona Nah, Pei-Luen Patrick Rau, Dylan Schmorrow, …
International journal of human-computer interaction, pp.1-46
Human-Centered AI (HCAI) Human-Centered AI Scale (HuCAS) subjective perception objective assessment scale maturity Artificial Intelligence or Cybernetics Business Administration
Existing approaches to implement Human-Centered AI (HCAI) are often ad hoc, compliance-driven, or focused on isolated user outcomes. This paper proposes a dual-assessment framework encompassing a subjective questionnaire for end-users and an objective assessment tool for organizations. To measure end-users' subjective impressions of various HCAI qualities, a questionnaire was designed and distributed among 153 participants from 17 countries. The survey results contribute to the modular, seven-factor Human-Centered AI Scale (HuCAS), which measures Trust, Transparency, Explainability, Human Control, Adaptability, Formal Fairness, and Substantive Fairness. To assess organizational HCAI capabilities, Human-Centered Maturity Assessment Instrument for AI (HuMAI) was developed and validated with experts. The instrument operationalizes HCAI across ten topics and assesses each topic separately for pre-deployment and post-deployment using explicitly cumulative maturity levels (0-5). By integrating the HuCAS and HuMAI, practitioners can identify impact gaps between organizational design intent and end-user perception, track lifecycle drift from pre-deployment to post-deployment, and diagnose the extent of alignment with critical HCAI socio-technical dimensions. This paper describes the concepts behind the HuCAS questionnaire and the HuMAI maturity assessment method as well as their construction. Knowledge concerning reliability, validity and practical applicability of these instrument must be collected in real industrial contexts and are no part of this paper.
Files and links (1)
url
Science-Based Methodologies for Evaluating the Human-Centeredness of AI SystemsView