Current Status, Prospect and Bottleneck of Ultrasound AI Development: A Systemic Review

doi:10.37015/AUDT.2023.230020

Abstract

Abstract:

In recent years, ultrasound imaging has become an important means of medical diagnosis because of its safety and radiation-free advantages. With the continuous progress of deep learning, Artificial Intelligence (AI) models can process large amounts of ultrasound data quickly and accurately, providing decision support for clinicians in diagnosis. From the perspective of ultrasound image classification, detection and segmentation, this paper systemically introduces the latest progress of AI technology in ultrasound imaging, and summarizes the recent high-level related work. At the same time, we also discuss the development prospect and bottleneck of AI in ultrasound imaging processing, which provides the future research directions for researchers in related fields.

Key words: Artificial intelligence; Ultrasound imaging; Deep learning; Diagnosis

Siyi Xun, MA, Wei Ke, PhD, Mingfu Jiang, MA, Huachao Chen, BA, Haoming Chen, BA, Chantong Lam, PhD, Ligang Cui, MD, Tao Tan, PhD. Current Status, Prospect and Bottleneck of Ultrasound AI Development: A Systemic Review. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(2): 61-72.

Figures/Tables 6

Figure 1

Figure 2

Figure 3

Table 1

Summary the areas of interest, methods, data sets and performance of major researches on AI in the field of medical ultrasound classification in recent years"

Paper	Region	Method	Dataset	Performance
Cao et al. [7]	Breast lesions	DenseNet	Private dataset Benign = 579 Malignant = 464	APR = 0.9689 ARR = 0.6723 F1 = 0.7938
Al-Dhabyani et al. [8]	Breast lesions	CNN Transfer learning	BUSI dataset Total = 780 Normal = 133 Benign = 437 Malignant = 210	Acc = 0.94
Han et al. [9]	Breast lesions	DDSTN	Private dataset Total = 106 Benign = 54 Malignant = 51	Acc = 0.8679 ± 0.0154 Sen = 0.8645 ± 0.0144 Spe = 0.8731 ± 0.0437
Zhou et al. [10]	Breast lesions	Multi-task learning	Private dataset Total = 170	Acc = 0.741 Rec = 0.798 Pre = 0.826 FPR = 0.392 F1 = 0.811
Badawy et al. [11]	Breast lesions	FCM U-Net	Private dataset Total = 1200	Acc = 0.9544 F1 = 0.6807
Bourouis et al. [12]	Breast lesions	GWO-WNN	Private dataset Total = 346 Benign = 97 Malignant = 249	Acc = 0.98 Sen = 0.988 Spe = 0.959
Jabeen et al. [13]	Breast lesions	CNN DarkNet53	BUSI dataset Total = 780 Normal = 133 Benign = 437 Malignant = 210	Acc = 0.991
Ragab et al. [14]	Breast lesions	VGG-16 VGG-19 SqueezeNet	BUSI dataset Total = 780 Normal = 133 Benign = 437 Malignant = 210	Acc = 0.9709
Gheflati et al. [15]	Breast lesions	ViT	BUSI dataset Total = 780 Normal = 133 Benign = 437 Malignant = 210	Acc = 0.867 AUC = 0.95
Ayana et al. [16]	Breast lesions	MSTL	Mendeley dataset Total = 250 Benign = 100 Malignant = 150	Acc = 0.999 Sen = 1 Spe = 0.98 AUC = 0.999 F1 = 0.989
Liu et al. [17]	Thyroid nodules	Multi-branch classification network	Private dataset1 Benign = 2551 Malignant = 5139 Private dataset2 Benign = 128 Malignant = 322	Acc = 0.971 Sen = 0.982 Spe = 0.951
Kuo et al. [18]	Kidney	ResNet	Private dataset Total = 4505	Acc = 0.856
Roy et al. [19]	Lung	CNN	ICLUS-DB video Total = 277	F1 = 0.61 ± 0.12 Pre = 0.70 ± 0.19 Rec = 0.60 ± 0.07
Xie et al. [20]	Brain	CNN	Private dataset Standard = 15372 Abnormal = 14047	Acc = 0.963 Sen = 0.969 Spe = 0.959
Sanagala et al. [21]	Carotid	DCNN Transfer learning	Private dataset Total = Unknown	AUC = 0.8333, 0.9566

Table 1

Table 2

Table 3

Summary the region of interest, method, dataset, and performance of the major researches on AI in medical ultrasound detection in recent years"

Paper	Region	Method	Dataset	Performance
Kim et al. [37]	Breast cancer	Weakly-supervised deep learning method	Private dataset Train = 1000 Test = 400	AUC = 0.86-0.96
Shen et al. [38]	Breast cancer	Deep learning model	NYU Breast Ultrasound Dataset Train = 3265744 Test = 1632872 Validation = 544291	AUC = 0.976 Sen = 0.918 PPV = 0.38
Niu et al. [39]	Breast lesions	Grey level gradient cooccurrence Matrix analysis	Private dataset Total = 206	BI-RADS 4A: Acc > 0.9
Zhang et al. [40]	Breast lesions	Lightweight neural network	Private dataset Train = 5030 Test = Unknown Validation = 1830	Sen = 0.8925 Spe = 0.9633 Average Pre = 0.85
Qian et al. [41]	Breast cancer	Deep-learning system	Private dataset Train = 10815 Test = 912 Validation = Unknown	Bimodal US images: AUC= 0.880 MultimodalUS images: AUC = 0.920
Baloescu et al. [42]	Lung lesions	Deep learning automated algorithm	Private dataset Train = 1847 Test = 100 Validation = 468	Sen = 0.93 Spe = 0. 96
Diaz-Escobar et al. [43]	Lung lesions, COVID-19	Deep learning architectures	POCUS dataset Train = 2661 Test = Unknown Validation = 665	Average Acc = 0.891 Balanced Acc = 0.893 COVID-19 detection AUC = 0.971
Fang et al. [44]	Lung lesions	CNN architectures based on transfer learning	Private dataset Train = 916 Test = Unknown Validation = Unknown	Clinical diagnosis, CXR, Chest CT: Kappa values = 0.943, 0.837, 0.835
Kulhare et al. [45]	Lung lesions	Single Shot CNN Model	Private dataset Train = 18713 Test = 444 Validation = Unknown	Acc = 0.89
Abdel-Basset et al. [46]	Lung lesions, COVID-19	CNN	POCUS dataset Total = 3234	Acc = 0. 934 F1 = 0.931 AUC = 0.97
Choi et al. [47]	Thyroid nodule	CAD system using AI	Private dataset Train = Unknown Test = Unknown Validation = Unknown	Sen = 0.907
Wei et al. [48]	Thyroid nodule	CNN	Private dataset Train = 5000 Test = 2214	Acc = 0.92
Wang et al. [49]	Thyroid nodule	YOLOv2 neural network	Private dataset Train = 5007 Test = Unknown Validation = 351	ROC = 0.902 Sen = 0.905 PPV = 0.9522 NPV = 0.8099 Acc = 0.9031 Spe = 0.8991
Jassal et al. [50]	Thyroid nodule	AI model	Private dataset Train = 857 Test = 198 Validation = Unknown	Acc = 0.89 Sen = 0.89 Spe = 0.83 F1 = 0.94 AUC = 0.86
Deng et al. [51]	Thyroid nodule	ResNet50 Random forest	Private dataset Train = 366 Test = 122 Validation = 122	Sen = 0.8587 Spe = 0.9718 Acc = 0.9377 AUC = 0.982

Table 3

References 55

[1]	Reddy UM, Filly RA, Copel JA. Prenatal imaging: ultrasonography and magnetic resonance imaging. Obstet Gynecol 2008; 112:145-157. doi: 10.1097/01.AOG.0000318871.95090.d9 pmid: 18591320
[2]	Anas EMA, Seitel A, Rasoulian A, John PS, Pichora D, Darras K, et al. Bone enhancement in ultrasound using local spectrum variations for guiding percutaneous scaphoid fracture fixation procedures. Int J CARS 2015; 10:959-969. doi: 10.1007/s11548-015-1181-6
[3]	Noble JA, Boukerroui D. Ultrasound image segmentation: a survey. IEEE Trans Med Imaging 2006; 25:987-1010. doi: 10.1109/TMI.2006.877092
[4]	Pereira F, Bueno A, Rodriguez A, Perrin D, Marx G, Cardinale M, et al. Automated detection of coarctation of aorta in neonates from two dimensional echocardiograms. J Med Imaging 2017; 4:014502. doi: 10.1117/1.JMI.4.1.014502
[5]	Chen C. CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature. Journal of the American Society for Information Science and Technology 2006; 57:359-377. doi: 10.1002/(ISSN)1532-2890
[6]	Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A. Dataset of breast ultrasound images. Data Brief 2020; 28:104863. doi: 10.1016/j.dib.2019.104863
[7]	Cao Z, Duan L, Yang G, Yue T, Chen Q. An experimental study on breast lesion detection and classification from ultrasound images using deep learning architectures. BMC medical imaging 2019; 19:1-9. doi: 10.1186/s12880-018-0301-5
[8]	Al-Dhabyani W, Gomaa M, Khaled H, Fahmy A. Deep learning approaches for data augmentation and classification of breast masses using ultrasound images. Int. J. Adv. Comput. Sci. Appl 2019; 10:1-11. doi: 10.3390/app10010001
[9]	Han X, Wang J, Zhou W, Chang C, Ying S, Shi J. Deep doubly supervised transfer network for diagnosis of breast cancer with imbalanced ultrasound imaging modalities//Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part VI 23. Springer International Publishing; 2020:141-149.
[10]	Zhou Y, Chen H, Li Y, Liu Q, Xu X, Wang S, et al. Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images. Medical Image Analysis 2021; 70:101918. doi: 10.1016/j.media.2020.101918
[11]	Badawy SM, Mohamed AENA, Hefnawy AA, Zidan HE, GadAllah MT, El-Banby GM. Automatic semantic segmentation of breast tumors in ultrasound images based on combining fuzzy logic and deep learning—a feasibility study. PLoS One 2021; 16:e0251899.
[12]	Bourouis S, Band SS, Mosavi A, Agrawal S, Ahila A, Poongodi M, et al. Meta-heuristic algorithm-tuned neural network for breast cancer diagnosis using ultrasound images. Frontiers in Oncology 2022; 12:834028. doi: 10.3389/fonc.2022.834028
[13]	Jabeen K, Khan MA, Alhaisoni M, Tariq U, Zhang YD, Hamza A, et al. Breast cancer classification from ultrasound images using probability-based optimal deep learning feature fusion. Sensors 2022; 22:807. doi: 10.3390/s22030807
[14]	Ragab M, Albukhari A, Alyami J, Mansour RF. Ensemble deep-learning-enabled clinical decision support system for breast cancer diagnosis and classification on ultrasound images. Biology 2022; 11:439. doi: 10.3390/biology11030439
[15]	Gheflati B, Rivaz H. Vision transformers for classification of breast ultrasound images. Annu Int Conf IEEE Eng Med Biol Soc 2022:480-483.
[16]	Ayana G, Park J, Jeong JW, Choe S. A novel multistage transfer learning for ultrasound breast cancer image classification. Diagnostics 2022; 12:135. doi: 10.3390/diagnostics12010135
[17]	Liu T, Guo Q, Lian C, Ren X, Liang S, Yu J, et al. Automated detection and classification of thyroid nodules in ultrasound images using clinical-knowledge-guided convolutional neural networks. Medical image analysis 2019; 58:101555. doi: 10.1016/j.media.2019.101555
[18]	Kuo CC, Chang CM, Liu KT, Lin WK, Chiang HY, Chung CW, et al. Automation of the kidney function prediction and classification through ultrasound-based kidney imaging using deep learning. NPJ digital medicine 2019; 2:29. doi: 10.1038/s41746-019-0104-2
[19]	Roy S, Menapace W, Oei S, Luijten B, Fini E, Saltori. Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound. IEEE Trans Med Imaging 2020; 39:2676-2687. doi: 10.1109/TMI.42
[20]	Xie HN, Wang N, He M, Zhang LH, Cai HM, Xian B J, et al. Using deep-learning algorithms to classify fetal brain ultrasound images as normal or abnormal. Ultrasound Obstet Gynecol 2020; 56:579-587. doi: 10.1002/uog.21967 pmid: 31909548
[21]	Skandha SS, Gupta SK, Saba L, Koppula VK, Johri AM, Khanna NN, et al. 3-D optimized classification and characterization artificial intelligence paradigm for cardiovascular/stroke risk stratification using carotid ultrasound-based delineated plaque: Atheromatic^TM 2.0. Comput Biol Med 2020;125103958.
[22]	Chen H, Zheng Y, Park JH, Heng PA, Zhou SK, et al. Iterative multi-domain regularized deep learning for anatomical structure detection and segmentation from ultrasound images. IEEE Transactions on Medical Imaging 2019; 38:2514-2524.
[23]	Cui W, Meng D, Lu K, Wang L, Zhou X, Li Y, et al. Automatic segmentation of ultrasound images using SegNet and local Nakagami distribution fitting model. Biomedical Signal Processing and Control 2023; 81:104431. doi: 10.1016/j.bspc.2022.104431
[24]	Dangoury S, Sadik M, Alali A, Fail A, Khoury R, Nassar J. V-net performances for 2D ultrasound image segmentation. IEEE 18th International Colloquium on Signal Processing & Applications (CSPA) 2022:96-100.
[25]	Yap MH, Pons G, Martí J, Ganau S, Sentís M, Zwiggelaar R, et al. Automated breast ultrasound lesions detection using convolutional neural networks. IIEEE J Biomed Health Inform 2018; 22:1218-1226.
[26]	Sharifzadeh M, Benali H, Rivaz H. Shift-invariant segmentation in breast ultrasound images. IEEE International Ultrasonics Symposium (IUS) 2021:1-4.
[27]	Gare GR, Li J, Joshi R, Patel S, Kumar V, Shah P, et al. W-Net: Dense and diagnostic semantic segmentation of subcutaneous and breast tissue in ultrasound images by incorporating ultrasound RF waveform data. Medical Image Analysis 2022; 76:102326. doi: 10.1016/j.media.2021.102326
[28]	Yin S, Peng Q, Li H, Gao X, Liu L, Feng S, et al. Automatic kidney segmentation in ultrasound images using subsequent boundary distance regression and pixelwise classification networks. Medical Image Analysis 2020; 60:101602. doi: 10.1016/j.media.2019.101602
[29]	Torres HR, Queirós S, Morais P, Fonseca JC, Vilaça JL, Oliveira C, et al. Kidney segmentation in 3-D ultrasound images using a fast phase-based approach. IEEE Trans Ultrason Ferroelectr Freq Control 2020;68:1521-1531.
[30]	Chen G, Dai Y, Li R, Wang J, Wu X, Zhang L, et al. SDFNet: Automatic segmentation of kidney ultrasound images using multi-scale low-level structural feature. Expert Systems with Applications 2021; 185:115619. doi: 10.1016/j.eswa.2021.115619
[31]	Valente S, Ferreira HA, Silva M, Oliveira R, Santos J, Gomes P, et al. A deep learning method for kidney segmentation in 2D ultrasound images. Annu Int Conf IEEE Eng Med Biol Soc 2022: 2022:3911-3914. doi: 10.1109/EMBC48229.2022.9871748 pmid: 36086291
[32]	Leclerc S, Smistad E, Pedrosa J, Østvik A, Cervenansky F, Espinosa F, et al. Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Trans Med Imaging 2019;38:2198-2210.
[33]	Pu B, Wang Q, Zhang Y, Zhao H, Liu Y, Chen X, et al. MobileUNet-FPN: A semantic segmentation model for fetal ultrasound four-chamber segmentation in edge computing environments. IEEE J Biomed Health Inform 2022; 26:5540-5550. doi: 10.1109/JBHI.2022.3182722
[34]	Ma J, Wu F, Jiang T, Zhao Q, Kong D. Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int J Comput Assist Radiol Surg 2018; 13:1685-1697. doi: 10.1007/s11548-018-1815-6
[35]	Li H, Fang J, Liu S, Liang X, Yang X, Mai Z, et al. Cr-UNet: a composite network for ovary and follicle segmentation in ultrasound images. IEEE J Biomed Health Inform 2019; 24:974-983. doi: 10.1109/JBHI.6221020
[36]	Qiu Z, Qin Y, Duan X, Shi Y, Yang X, Qin J, et al. Automatic mouse embryo brain ventricle & body segmentation and mutant classification from ultrasound data using deep learning. IEEE International Ultrasonics Symposium (IUS) 2019:12-15.
[37]	Kim J, Kim HJ, Kim C, Lee JH, Kim KW, Park YM, et al. Weakly-supervised deep learning for ultrasound diagnosis of breast cancer. Scientific Reports 2021; 11:24382. doi: 10.1038/s41598-021-03806-7 pmid: 34934144
[38]	Shen Y, Shamout FE, Oliver JR, Witowski J, Kannan K, Park J, et al. Artificial intelligence system reduces false-positive findings in the interpretation of breast ultrasound exams. Nature communications 2021; 12:5645. doi: 10.1038/s41467-021-26023-2 pmid: 34561440
[39]	Niu S, Huang J, Li J, Liu X, Wang D, Zhang RF, et al. Application of ultrasound artificial intelligence in the differential diagnosis between benign and malignant breast lesions of BI-RADS 4A. BMC cancer 2020; 20:1-7. doi: 10.1186/s12885-019-6169-0
[40]	Zhang X, Lin X, Zhang Z, Dong LC, Sun XL, Sun DS, et al. Artificial intelligence medical ultrasound equipment: application of breast lesions detection. Ultrasonic Imaging 2020; 42:191-202. doi: 10.1177/0161734620928453
[41]	Qian X, Pei J, Zheng H, Xie X, Yan L, Zhang H, et al. Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning. Nat Biomed Eng 2021; 5:522-532. doi: 10.1038/s41551-021-00711-2 pmid: 33875840
[42]	Baloescu C, Toporek G, Kim S, McNamara K, Liu R, Shaw M, et al. Automated lung ultrasound B-line assessment using a deep learning algorithm. IEEE Trans Ultrason Ferroelectr Freq Control 2020; 67:2312-2320. doi: 10.1109/TUFFC.58
[43]	Diaz-Escobar J, Ordóñez-Guillén NE, Villarreal-Reyes S, Galaviz-Mosqueda A, Kober V, Rivera-Rodriguez R, et al. Deep-learning based detection of COVID-19 using lung ultrasound imagery. Plos one 2021; 16:e0255886.
[44]	Fang X, Li W, Huang J, Li W, Feng Q, Han Y, et al. Ultrasound image intelligent diagnosis in community-acquired pneumonia of children using convolutional neural network based transfer learning. Front Pediatr 2022; 10:1063587. doi: 10.3389/fped.2022.1063587
[45]	Kulhare S, Zheng X, Mehanian C, Gregory C, Zhu M, Gregory K, et al. Ultrasound-based detection of lung abnormalities using single shot detection convolutional neural networks. In Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation: International Workshops 2018;pp:65-73.
[46]	Abdel-Basset M, Hawash H, Alnowibet KA, Mohamed AW, Sallam KM. Interpretable deep learning for discriminating pneumonia from lung ultrasounds. Mathematics 2022; 10:4153. doi: 10.3390/math10214153
[47]	Choi YJ, Baek JH, Park HS, Shim WH, Kim TY, Shong YK, et al. A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid 2017; 27:546-552. doi: 10.1089/thy.2016.0372 pmid: 28071987
[48]	Wei X, Zhu J, Zhang H, Gao H, Yu R, Liu Z, et al. Visual interpretability in computer-assisted diagnosis of thyroid nodules using ultrasound images. Med Sci Monit 2020; 26:e927007.
[49]	Wang L, Yang S, Yang S, Zhao C, Tian G, Gao Y, et al. Automatic thyroid nodule recognition and diagnosis in ultrasound imaging with the YOLOv2 neural network. World J Surg Oncol 2019; 17:1-9. doi: 10.1186/s12957-018-1541-0
[50]	Jassal K, Koohestani A, Kiu A, Strong A, Ravintharan N, Yeung M, et al. Artificial intelligence for pre-operative diagnosis of malignant thyroid nodules based on sonographic features and cytology category. World J Surg 2023; 47:330-339. doi: 10.1007/s00268-022-06798-1
[51]	Deng C, Han D, Feng M, Lv Z, Li D. Differential diagnostic value of the ResNet50, random forest, and DS ensemble models for papillary thyroid carcinoma and other thyroid nodules. J Int Med Res 2022;50.
[52]	Fujioka T, Mori M, Kubota K, Kikuchi Y, Katsuta L, Adachi M, et al. Breast ultrasound image synthesis using deep convolutional generative adversarial networks. Diagnostics 2019; 9:176. doi: 10.3390/diagnostics9040176
[53]	Jarosik P, Lewandowski M, Klimonda Z, Byra M. Pixel-wise deep reinforcement learning approach for ultrasound image denoising. 2021 IEEE International Ultrasounds Symposium (IUS) 2021;1-4.
[54]	Czajkowska J, Juszczyk J, Piejko L, Glenc-Ambroży M. High-frequency ultrasound dataset for deep learning-based image quality assessment. Sensors 2022; 22:1478. doi: 10.3390/s22041478
[55]	Wang J, Yang X, Zhou B, Sohn JJ, Zhou J, Jacob JT, et al. Review of machine learning in lung ultrasound in COVID-19 pandemic. Journal of Imaging 2022; 8:65. doi: 10.3390/jimaging8030065

Paper	Region	Method	Dataset	Performance
Chen et al. [22]	Anatomical structures	Iterative multi-domain regularized deep learning	Private dataset Train = Unknown Test = Unknown	DSC = 0.927
Cui et al. [23]	Ultrasound	SegNet	Private dataset Train = Unknown Test = Unknown	Unknown
Dangoury et al. [24]	Ultrasound	V-net	Private dataset Train = 5635 Test = 5508	DSC = 0.8501 Sen = 0.8556 Spe = 0.9987 Acc = 0.9992
Yap et al. [25]	Breast	CNN	Inbreast Total = 410	AUC = 0.94
Sharifzadeh et al. [26]	Breast	Shift-Invariant Segmentation	US breast images dataset Train = 100 Validation = 30 Test = 33	DSC = 0.94 JI = 0.91
Gare et al. [27]	Subcutaneous, Breast	W-Net	Private dataset Train = 450 Test = 50	DSC = 0.883
Yin et al. [28]	Kidney	Pixelwise Classification Networks	Private dataset Total = 918	DSC = 0.959
Torres et al. [29]	Kidney	BEAS framework	Private dataset Total = 45	DSC = 0.93
Chen et al. [30]	Kidney	SDFNet	Private dataset Train = 450 Test = 50	DSC = 0.941
Valente et al. [31]	Kidney	Deep learning method	Private dataset Training = 2166 Validation = 193 Testing = 358	DSC = 0.94
Leclerc et al. [32]	heart	Deep learning	CAMUS Total = 2000	DSC = 0.92
Pu et al. [33]	Fetal heart	MobileUNet-FPN	Private dataset Train = 575 Validation = 102 Test = 207	DSC = 0.935
Ma et al. [34]	Thyroid	CNN	Private dataset Total = 352	DSC = 0.901
Li et al. [35]	Ovary Follicle	Cr-UNet	Private dataset Train = 2509 Test = 695	DSC = 0.9601
Qiu et al. [36]	Mouse Embryo	Deep Learning	Private dataset Train = Unknown Test = Unknown	ACC = 0.98