The impact of forms of AI feedback and image quality on reporting radiographers trust and decision switching when interpreting plain radiographic images of the appendicular skeleton

Rainey, Clare; McConnell, Jonathan; Hughes, Ciara; McLaughlin, Laura; Bond, Raymond; McFadden, Sonyia L.

The impact of forms of AI feedback and image quality on reporting radiographers trust and decision switching when interpreting plain radiographic images of the appendicular skeleton

cbd

Files

EPOS™.pdf(247.3 KB)

Published Version

Date

2024

Authors

Rainey, Clare

McConnell, Jonathan

Hughes, Ciara

McLaughlin, Laura

Bond, Raymond

McFadden, Sonyia L.

Publisher

European Society of Radiology

Published Version

https://doi.org/10.26044/ecr2024/C-23132

Abstract

Purpose or Learning Objective Reported accuracies and workforce shortages have increased integration of AI into clinical environments. Furthermore, radiographer reporting helps ease the burden of image reporting. ‘System trust’ is identified as a challenge to clinical AI integration. To the authors’ knowledge, no research has been conducted on the factors impacting reporting radiographers’ trust and decision making when using different forms of AI feedback. Methods or Background Twelve reporting radiographers, three from each region of the UK, participated in this study. The Qualtrics® platform was used to randomly allocate 18 radiographic examinations to each participant. Participants were asked to locate any pathology and indicate their agreement with the AI localisation, represented by GradCAM heatmaps and the AI binary diagnosis. Spearman’s rho and Kendall’s tau were used to investigate any correlation between trust and agreement with various forms of AI feedback and initial image quality. Results or Findings Participants disagreed with the AI heatmaps for the abnormal examinations 45.8% (n=66 of 144 individual images) of the time and agreed with binary feedback on 86.7% of examinations (26 of 30 cases). 0.7% (n=2) indicated that they would decision switch following AI feedback. 22.2% (n=32) agreed with the localisation of pathology from the heatmap. Agreement with AI feedback was correlated with trust (-.515; -.584, significant large negative correlation (p=<.01) and -.309; -.369, significant medium negative correlation (p=<.01) for GradCAM and binary diagnosis respectively). Conclusion The extent of agreement with both AI binary diagnosis and heatmap is correlated with trust in AI, where greater agreement with AI feedback is associated with greater trust, with a large effect size in agreement with GradCAM feedback. Limitations The Qualtrics® platform may not allow for an accurate simulation of the clinical setting. This will be further investigated in subsequent studies.

Keywords

Artificial intelligence , AI , Decision hygiene , Trust , Radiography

Citation

Rainey, C., McConnell, J., Hughes, C., Mc Laughlin, L., Bond, R.R. and McFadden, S. (2024), 'The impact of forms of AI feedback and image quality on reporting radiographers trust and decision switching when interpreting plain radiographic images of the appendicular skeleton', European Congress of Radiology, C-23132, Vienna, Austria, 28 February - 3 March. https://doi.org/10.26044/ecr2024/C-23132