The influence of experience and training in a group of novice
observers : A jackknife alternative free-response receiver operating characteristic analysis

Buissink, V; Thompson, JD; Voet, M; Hogg, P

Abstract

Purpose: The study evaluates the pre- and post-training lesion localisation ability of a group of novice
observers. Parallels are drawn with the performance of inexperienced radiographers taking part in
preliminary clinical evaluation (PCE) and ‘red-dot’ systems, operating within radiography practice.
Materials and methods: Thirty-four novice observers searched 92 images for simulated lesions. Pretraining
and post-training evaluations were completed following the free-response the receiver operating
characteristic (FROC) method. Training consisted of observer performance methodology, the
characteristics of the simulated lesions and information on lesion frequency. Jackknife alternative FROC
(JAFROC) and highest rating inferred ROC analyses were performed to evaluate performance difference
on lesion-based and case-based decisions. The significance level of the test was set at 0.05 to control the
probability of Type I error.
Results: JAFROC analysis (F(3,33) ¼ 26.34, p < 0.0001) and highest-rating inferred ROC analysis
(F(3,33) ¼ 10.65, p ¼ 0.0026) revealed a statistically significant difference in lesion detection performance.
The JAFROC figure-of-merit was 0.563 (95% CI 0.512,0.614) pre-training and 0.677 (95% CI
0.639,0.715) post-training. Highest rating inferred ROC figure-of-merit was 0.728 (95% CI 0.701,0.755)
pre-training and 0.772 (95% CI 0.750,0.793) post-training.
Conclusions: This study has demonstrated that novice observer performance can improve significantly.
This study design may have relevance in the assessment of inexperienced radiographers taking part in
PCE or commenting scheme for trauma.

Citation

observers : A jackknife alternative free-response receiver operating characteristic analysis. Radiography, 20(4), 300-305. https://doi.org/10.1016/j.radi.2014.06.016

• Journal Article Type: Article
• Acceptance Date: Jun 30, 2014
• Online Publication Date: Jul 17, 2014
• Publication Date: Nov 1, 2014
• Deposit Date: Jan 27, 2015
• Publicly Available Date: Apr 5, 2016
• Journal: Radiography
• Print ISSN: 1078-8174
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 20
• Issue: 4
• Pages: 300-305
• DOI: https://doi.org/10.1016/j.radi.2014.06.016
• Publisher URL: http://dx.doi.org/10.1016/j.radi.2014.06.016
• Related Public URLs: http://www.journals.elsevier.com/radiography/