Objective Deploying an automatic segmentation model in practice should require rigorous quality assurance (QA) and continuous monitoring of the model's use and performance, particularly in high-stakes scenarios such as healthcare. Currently, however, tools to assist with QA for such models are not available to AI researchers. In this work, we build a deep learning model that estimates the quality of automatically generated contours. Methods The model was trained to predict the segmentation quality by outputting an estimate of the Dice similarity coefficient given an image contour pair as input. Our dataset contained 60 axial T2-weighted MRI images of prostates with ground truth segmentations along with 80 automatically generated segmentation masks. The model we used was a 3D version of the EfficientDet architecture with a custom regression head. For validation, we used a fivefold cross-validation. To counteract the limitation of the small dataset, we used an extensive data augmentation scheme capable of producing virtually infinite training samples from a single ground truth label mask. In addition, we compared the results against a baseline model that only uses clinical variables for its predictions. Results Our model achieved a mean absolute error of 0.020 +/- 0.026 (2.2% mean percentage error) in estimating the Dice score, with a rank correlation of 0.42. Furthermore, the model managed to correctly identify incorrect segmentations (defined in terms of acceptable/unacceptable) 99.6% of the time. Conclusion We believe that the trained model can be used alongside automatic segmentation tools to ensure quality and thus allow intervention to prevent undesired segmentation behavior.

Quality assurance for automatically generated contours with additional deep learning / L. Isaksson, P. Summers, A. Bhalerao, S. Gandini, S. Raimondi, M. Pepa, M. Zaffaroni, G. Corrao, G. Mazzola, M. Rotondi, G. Lo Presti, Z. Haron, S. Alessi, P. Pricolo, F. Mistretta, S. Luzzago, F. Cattani, G. Musi, O. De Cobelli, M. Cremonesi, R. Orecchia, G. Marvaso, G. Petralia, B. Jereczek-Fossa. - In: INSIGHTS INTO IMAGING. - ISSN 1869-4101. - 13:(2022), pp. 137.1-137.10. [10.1186/s13244-022-01276-7]

Quality assurance for automatically generated contours with additional deep learning

L. Isaksson
Primo
;
F. Mistretta;S. Luzzago;G. Musi;O. De Cobelli;G. Marvaso;G. Petralia;B. Jereczek-Fossa
Ultimo
2022

Abstract

Objective Deploying an automatic segmentation model in practice should require rigorous quality assurance (QA) and continuous monitoring of the model's use and performance, particularly in high-stakes scenarios such as healthcare. Currently, however, tools to assist with QA for such models are not available to AI researchers. In this work, we build a deep learning model that estimates the quality of automatically generated contours. Methods The model was trained to predict the segmentation quality by outputting an estimate of the Dice similarity coefficient given an image contour pair as input. Our dataset contained 60 axial T2-weighted MRI images of prostates with ground truth segmentations along with 80 automatically generated segmentation masks. The model we used was a 3D version of the EfficientDet architecture with a custom regression head. For validation, we used a fivefold cross-validation. To counteract the limitation of the small dataset, we used an extensive data augmentation scheme capable of producing virtually infinite training samples from a single ground truth label mask. In addition, we compared the results against a baseline model that only uses clinical variables for its predictions. Results Our model achieved a mean absolute error of 0.020 +/- 0.026 (2.2% mean percentage error) in estimating the Dice score, with a rank correlation of 0.42. Furthermore, the model managed to correctly identify incorrect segmentations (defined in terms of acceptable/unacceptable) 99.6% of the time. Conclusion We believe that the trained model can be used alongside automatic segmentation tools to ensure quality and thus allow intervention to prevent undesired segmentation behavior.
Quality assurance (Health care); Confidence calibration; Diagnostic imaging; Prostate; Magnetic resonance imaging
Settore MED/24 - Urologia
Settore MED/36 - Diagnostica per Immagini e Radioterapia
17-ago-2022
Article (author)
File in questo prodotto:
File Dimensione Formato  
s13244-022-01276-7.pdf

accesso aperto

Tipologia: Publisher's version/PDF
Dimensione 1.35 MB
Formato Adobe PDF
1.35 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/2434/948548
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact