39 - Assessing User Experience in a Home-grown IRB Application System Through Comprehensive Usability Testing: Mixed Methods

Background: : IRBs utilize a variety of commercial and home-grown software application systems; however, the features of these systems related to usability are generally not examined. We analyzed the home-grown system; the methods used in this analysis may be useful by other institutions in directing modification to improve efficiency and usability.

Methods: : A sequential exploratory mixed-methods design was used to assess the usability of IAS. Participants included IRB stakeholders such as analysts, investigators, and research coordinators, with a sample size of 40, which exceeds what is deemed adequate for system usability testing. Participants had 1-15 years of experience with this IAS and included persons with diverse roles and varying interaction frequencies with the system. This diversity aims to gather varied perspectives on usability issues. Usability was measured quantitatively using the System Usability Scale (SUS) and the National Aeronautics and Space Administration Task Load Index (NASA-TLX), demonstrating good reliability with Cronbach’s alpha of 0.76 based on five items. Qualitative feedback was collected through scenario-task-based usability testing. Usability testing scripts were created for analysts and investigators based on different interfaces, reviewed and validated by experts, and analyzed thematically to understand navigation, workflow, system design issues, and suggestions for design improvements. Descriptive statistics and correlation analyses were performed to explore associations between usability scores and years of experience.

Results: : Most participants identified system lag, a lack of user control and freedom, poor clarity and accessibility of information, navigation complexity, inadequate guidance, slow document uploads, a lack of customizable interfaces- especially for investigators and coordinators- and limited messaging functionality as significant usability issues. Analysts reported greater ease of use with the system than coordinators and investigators. The overall SUS score averaged 56 (median = 54), which is below the acceptable threshold of 68. NASA-TLX scores indicated that mental demands, performance, physical demands, and frustration were key workload burdens, particularly among less experienced users. A negative correlation was found between years of experience and workload scores, along with a slight positive correlation between SUS and user experience, suggesting that familiarity reduces cognitive strain and improves ease of use.

Limitations:: This study was limited to a single IAS, which certainly affects the generalizability of the findings; however, the mixed method can be replicated, and feedback can be implemented in any IAS. Self-selection and self-administered survey bias may have influenced participant feedback. Despite the workload burdens expressed by users, most users rated “performance” highly, and cross-role comparisons are based on unequal group sizes.

Discussion: : Workload perception varied significantly across roles, with frustration and mental and physical demands being the primary contributors to task difficulty. The negative correlation between years of experience and NASA-TLX scores suggests that experienced individuals may develop adaptive strategies to manage workload more effectively, although systemic inefficiencies persist. Similarly, the average SUS score is below 68, which is the acceptable benchmark. The IAS presents significant usability and workload challenges, particularly for newer users. Improvements in navigation, help and user-controlled features, and customizable workflows are recommended. Findings underscore the value of continuous user-centered evaluations to guide IRB system redesign.