Machine learning-based diagnostic test accuracy research: measurement indicators_Chinese Journal of Evidence-Based Medicine

Authors：

XIONG Yutao ¹ , ZHONG Chenglan ² , ZENG Wei ¹ , GUO Jixiang ² ,  ZHANG Tao ³ , HUANG Yan ⁴ , TANG Wei ¹ ,  LIU Chang ^1,4

1. State Key Laboratory of Oral Diseases & National Clinical Research Center for Oral Diseases & Department of Oral and Maxillofacial Surgery & Department of Evidence-Based Dentistry & "Medicine + Manufacturing" Center, West China Hospital of Stomatology, Sichuan University, Chengdu 610041, P. R. China;
2. Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China;
3. Department of Epidemiology and Health Statistics, West China School of Public Health, Sichuan University, Chengdu 610041, P. R. China;
4. Department of Information Management & Department of Dental Informatics, West China Hospital of Stomatology, Sichuan University, Chengdu 610041, P. R. China;

Corresponding?author：

ZHANG Tao, Email: scdxzhangtao@163.com; LIU Chang, Email: liu_chang_92@sina.com

Keywords：

Diagnostic test accuracy study; Artificial intelligence; Machine learning; Deep learning; Descriptive statistics; Inferential statistics

DOI：

10.7507/1672-2531.202302048

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Machine learning-based diagnostic tests have certain differences of measurement indicators with traditional diagnostic tests. In this paper, we elaborate the definitions, calculation methods and statistical inferences of common measurement indicators of machine learning-based diagnosis models in detail. We hope that this paper will be helpful for clinical researchers to better evaluate machine learning diagnostic models.

Citation： XIONG Yutao, ZHONG Chenglan, ZENG Wei, GUO Jixiang, ZHANG Tao, HUANG Yan, TANG Wei, LIU Chang. Machine learning-based diagnostic test accuracy research: measurement indicators. Chinese Journal of Evidence-Based Medicine, 2023, 23(8): 963-969. doi: 10.7507/1672-2531.202302048 Copy

1.	吳景玲, 葛龍, 張俊華, 等. 多個診斷性試驗準確性的比較: 網狀Meta分析方法介紹. 中國循證醫學雜志, 2017, 17(8): 987-992.
2.	Bahar RC, Merkaj S, Cassinelli Petersen GI, et al. Machine learning models for classifying high- and low-grade gliomas: a systematic review and quality of reporting analysis. Front Oncol, 2022, 12: 856231.
3.	Majanga V, Viriri S. Dental images' segmentation using threshold connected component analysis. Comput Intell Neurosci, 2021, 2021: 2921508.
4.	Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol, 2004, 11(2): 178-189.
5.	Thiel-Egenter C, Alvarez N, Holderegger R, et al. Break zones in the distributions of alleles and species in alpine plants. J Biogeogr, 2011, 38(4): 772-782.
6.	Yuan Y, Chao M, Lo YC. Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging, 2017, 36(9): 1876-1886.
7.	Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans Pattern Anal Mach Intell, 2015, 37(11): 2153-2163.
8.	Yao J, Zeng W, He T, et al. Automatic localization of cephalometric landmarks based on convolutional neural network. Am J Orthod Dentofacial Orthop, 2022, 161(3): 250-259.
9.	B?hning D, Holling H, Patilea V. A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Stat Methods Med Res, 2011, 20(5): 541-550.
10.	Lian L, Zhu T, Zhu F, et al. Deep learning for caries detection and classification. Diagnostics (Basel), 2021, 11(9): 1672.
11.	Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 1989, 16(4): 561-568.
12.	Everingham M, van Gool L, Williams C, et al. The PASCAL visual object classes challenge 2007.
13.	Everingham M, van Gool L, Williams C, et al. Visual object classes challenge 2010 (VOC2010).
14.	Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977, 33(1): 159-174.
15.	紀守領, 杜天宇, 鄧水光, 等. 深度學習模型魯棒性研究綜述. 計算機學報, 2022, 45(1): 190-206.
16.	Castelvecchi D. Can we open the black box of AI. Nature, 2016, 538(7623): 20-23.
17.	楊桂元. 中心極限定理及其在統計分析中的應用. 統計與信息論壇, 2000, (3): 13-15.

1. 吳景玲, 葛龍, 張俊華, 等. 多個診斷性試驗準確性的比較: 網狀Meta分析方法介紹. 中國循證醫學雜志, 2017, 17(8): 987-992.
2. Bahar RC, Merkaj S, Cassinelli Petersen GI, et al. Machine learning models for classifying high- and low-grade gliomas: a systematic review and quality of reporting analysis. Front Oncol, 2022, 12: 856231.
3. Majanga V, Viriri S. Dental images' segmentation using threshold connected component analysis. Comput Intell Neurosci, 2021, 2021: 2921508.
4. Zou KH, Warfield SK, Bharatha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol, 2004, 11(2): 178-189.
5. Thiel-Egenter C, Alvarez N, Holderegger R, et al. Break zones in the distributions of alleles and species in alpine plants. J Biogeogr, 2011, 38(4): 772-782.
6. Yuan Y, Chao M, Lo YC. Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging, 2017, 36(9): 1876-1886.
7. Taha AA, Hanbury A. An efficient algorithm for calculating the exact Hausdorff distance. IEEE Trans Pattern Anal Mach Intell, 2015, 37(11): 2153-2163.
8. Yao J, Zeng W, He T, et al. Automatic localization of cephalometric landmarks based on convolutional neural network. Am J Orthod Dentofacial Orthop, 2022, 161(3): 250-259.
9. B?hning D, Holling H, Patilea V. A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test. Stat Methods Med Res, 2011, 20(5): 541-550.
10. Lian L, Zhu T, Zhu F, et al. Deep learning for caries detection and classification. Diagnostics (Basel), 2021, 11(9): 1672.
11. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 1989, 16(4): 561-568.
12. Everingham M, van Gool L, Williams C, et al. The PASCAL visual object classes challenge 2007.
13. Everingham M, van Gool L, Williams C, et al. Visual object classes challenge 2010 (VOC2010).
14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977, 33(1): 159-174.
15. 紀守領, 杜天宇, 鄧水光, 等. 深度學習模型魯棒性研究綜述. 計算機學報, 2022, 45(1): 190-206.
16. Castelvecchi D. Can we open the black box of AI. Nature, 2016, 538(7623): 20-23.
17. 楊桂元. 中心極限定理及其在統計分析中的應用. 統計與信息論壇, 2000, (3): 13-15.

Previous Article
Evaluation of clinical guidelines and consensus on the quality of central venous catheters
Next Article
Current status and challenges of living guidelines

Chinese Journal of Evidence-Based Medicine

Machine learning-based diagnostic test accuracy research: measurement indicators

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content