Performance of ChatGPT and Radiology Residents on Ultrasonography Board-Style Questions

doi:10.37015/AUDT.2024.240002

Abstract

Abstract:

Objective: This study aims to assess the performance of the Chat Generative Pre-Trained Transformer (ChatGPT), specifically versions GPT-3.5 and GPT-4, on ultrasonography board-style questions, and subsequently compare it with the performance of third-year radiology residents on the identical set of questions.
Methods: The study, conducted from May 19 to May 30, 2023, utilized a selection of 134 multiple-choice questions sourced from a commercial question bank for American Registry for Diagnostic Medical Sonography (ARDMS) examinations and imported into the ChatGPT model (encompassing GPT-3.5 and GPT-4 versions). ChatGPT’s responses were evaluated overall, by topic, and by GPT version. An identical question set was assigned to three third-year radiology residents, enabling a direct comparison of performances with ChatGPT.
Results: GPT-4 correctly responded to 82.1% of questions (110 of 134), significantly surpassing the performance of GPT-3.5 (P = 0.003), which correctly answered 66.4% of questions (89 of 134). Although GPT-3.5’s performance was statistically indistinguishable from the average performance of the radiology residents (66.7%, 89.3 of 134) (P = 0.969), there was a notable difference in the accuracy in question-answering accuracy between GPT-4 and the residents (P = 0.004).
Conclusions: ChatGPT demonstrated significant competency in responding to ultrasonography board-style questions, with the GPT-4 version markedly surpassing both its predecessor GPT-3.5 and the radiology residents.

Key words: Artificial intelligence; Ultrasonography; Accuracy; Medical education

Xu Jiale, Xia Shujun, Hua Qing, Mei Zihan, Hou Yiqing, Wei Minyan, Lai Limei, Yang Yixuan, Zhou Jianqiao. Performance of ChatGPT and Radiology Residents on Ultrasonography Board-Style Questions. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(4): 250-254.

Figures/Tables 3

Figure 1

Table 1

Table 2

References 9

[1]	OpenAI. Introducing ChatGPT. https://openai.com/blog/chatgpt/, Nov 30, 2022. Accessed May 19, 2023.
[2]	Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT. Radiology 2023; 307:e230424.
[3]	Mihalache A, Popovic MM, Muni RH. Performance of an artificial intelligence Chatbot in ophthalmic knowledge assessment. JAMA Ophthalmol 2023; 141:589-597. doi: 10.1001/jamaophthalmol.2023.1144 pmid: 37103928
[4]	Adams LC, Truhn D, Busch F, Karder A, Niehues SM, Makowski MR, et al. Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: A multilingual feasibility study. Radiology 2023; 307:e230725.
[5]	Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology Board-style examination: Insights into current strengths and limitations. Radiology 2023; 307:e230582.
[6]	OpenAI. GPT-4 Technical Report. https://ui.adsabs.harvard.edu/abs/2023arXiv230308774O May 8, 2023. Accessed May 10, 2023.
[7]	Bhayana R, Bleakney RR, Krishna S. GPT-4 in radiology: Improvements in advanced reasoning. Radiology 2023; 307:e230987.
[8]	Ultrasound Registry Review Question Bank. https://www.prepry.com/ Accessed May 19, 2023.
[9]	Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, et al. ChatGPT and other large language models are double-edged swords. Radiology 2023; 307:e230163.

Question topics	Number of questions	Number of correct answers		P value
Question topics	Number of questions	GPT-3.5	GPT-4	P value
Physics	20	17(85)	18(90)	> 0.99
Clinic	114	72(63.2)	92(80.7)	0.003
Vascular technology	20	11(55)	19(95)	0.004
Abdomen	20	10(50)	15(75)	0.102
Obstetrics and gynecology	20	15(75)	18(90)	0.405
Pediatric sonography	20	10(50)	12(60)	0.525
Breast	20	14(70)	17(85)	0.449
Adult echocardiography	14	12(85.7)	11(78.6)	>0.99
Total	134	89(66.4)	110(82.1)	0.003

Question topics	Number of questions	Number of correct answers
Question topics	Number of questions	Resident A	Resident B	Resident C
Physics	20	14(70)	13(65)	12(60)
Clinic	114	72(63.2)	80(70.2)	77(67.5)
Vascular technology	20	14(70)	16(80)	15(75)
Abdomen	20	13(65)	12(60)	16(80)
Obstetrics and gynecology	20	15(75)	16(80)	15(75)
Pediatric sonography	20	9(45)	10(50)	9(45)
Breast	20	15(75)	13(65)	13(65)
Adult echocardiography	14	6(42.8)	13(92.8)	9(64.3)
Total	134	86(64.2)	93(69.4)	89(66.4)

[1]	Ge Xifeng, Liu Wenzheng, Chen Wen, Mei Fang, Cui Ligang. Gastroduodenal Intussusception Due to a Gastric Gastrointestinal Stromal Tumor in Adult from Sonographer's Perspective [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(4): 255-258.
[2]	Shanqing Li, MM, Rong Hu, MM, Xijing Liu, MD, Fan Yang, MD. Update on the Genetics and Prenatal Ultrasound Features of Williams-Beuren Syndrome [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(3): 79-85.
[3]	Guiwu Chen, MS, Zhizhong He, BS, Wenqin Liu, MS, Junjun Chen, BS, Xiaomin Liao, MS, Yuhuan Xie, BS. Multimodal Sonographic Findings of Embryonal Carcinoma in the Testis: A Case Report and Literature Review [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(2): 74-77.
[4]	Siqi Zheng, MM, Min Bai, MM. Application Progress of Ultrasound Elastography in the Evaluation of Diabetic Peripheral Neuropathy [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(1): 1-6.
[5]	Guoying Zhang, MD, Ying Tang, BS, Mingyang Wang, MD, Weina Kong, MD. Diagnostic Utility of Superb Microvascular Imaging of ultrasound Examinations to Evaluate Hepatic Ischemia-reperfusion Injury [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(1): 15-21.
[6]	Hui Li, MD, Nan Zheng, MD, Penglin Zou, MD, Chao Jia, MD, Long Liu, MD, Gang Li, MD, Ziqi Wang, MD, Rong Wu, MD, Lianfang Du, MD, Qiusheng Shi, MD. The Role of Ultrasonography in the Diagnosis of Systemic Sarcoidosis: a Case Report and Literature Review [J]. Advanced Ultrasound in Diagnosis and Therapy, 2024, 8(1): 32-38.
[7]	Yang Qi, MD, Dengsheng Sun, MD, Linyao Wang, MD, Jie Yu, MD, Ping Liang, MD. State-of-the-Art and Development Trend of Interventional Ultrasound in China [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 313-320.
[8]	Huihui Chai, MS, Xiaowan Bo, MD, Lehang Guo, MD, Chengzhong Peng, MD. Experience and Enlightenment of Handheld Ultrasound Applications in Multiple Scenarios Based on 5G Technology [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 356-365.
[9]	Sebastián Eustaquio Martín Pérez, MSc, Raúl Hernández García, PT, Alberto Brito Lorenzo, PT, Carlos Daniel Sabater Cruz, PT, Mario Herrera Pérez, PhD, Fidel Rodríguez Hernández, PhD, Kristin Briem, PhD, Isidro Miguel Martín Pérez, MD. Ultrasonographic Identification of Muscle Atrophy in Hamstring Muscles after Anterior Cruciate Ligament Repair among Soccer Players: A Case-control Study [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 381-389.
[10]	Lujing Li, MD, Zuofeng Xu, MD. The Value of VTTQ Combined with B-mode US for Distinguishing Benign from Malignant Breast Masses by Comparing with SE: A Clinical Research [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 394-400.
[11]	Rodanthi Sfakiotaki, MS, Sergia Liasi, BM, Eleni Papaiakovou, BM, Irene Vraka, PhD, Marina Vakaki, PhD, Chrysoula Koumanidou, PhD. Juvenile Granulosa Cell Tumor of the Testis: A Preoperative Approach of the Diagnosis with Ultrasound [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 409-411.
[12]	Yiming Li, BM, Jing Xiao, MD, Fang Xie, MD, Yu Lin, BM, Mingbo Zhang, MD, Yukun Luo, MD. The Value of CEUS in the Diagnosis and Treatment of Thyroid Primary Squamous Cell Carcinoma: A Case Report [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 412-415.
[13]	Nianyu Xue, MM, Shengmin Zhang, BS. Appendiceal Mucinous Neoplasms Involving the Testis: A Case Report [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 420-422.
[14]	Shuangyu Wu, MM , Xinling Zhang, MD . Advances and Applications of Transperineal Ultrasound Imaging in Female Pelvic Floor Dysfunction [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(3): 235-247.
[15]	Bo Jiang, MD, Yiman Du, MD, Xiang Fei, MD, Jianing Zhu, MD, Lianhua Zhu, MD, Qiuyang Li, MD, Yukun Luo, MD, PhD. Ultrasound-Guided Attenuation Parameter May Replace B-mode Ultrasound in Diagnosing Nonalcoholic Fatty Liver Disease [J]. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(3): 260-266.