Semi-supervised Learning for Real-time Segmentation of Ultrasound Video Objects: A Review

doi:10.37015/AUDT.2023.230016

Abstract

Abstract:

Real-time intelligent segmentation of ultrasound video object is a demanding task in the field of medical image processing and serves as an essential and critical step in image-guided clinical procedures. However, obtaining reliable and accurate medical image annotations often necessitates expert guidance, making the acquisition of large-scale annotated datasets challenging and costly. This presents obstacles for traditional supervised learning methods. Consequently, semi-supervised learning (SSL) has emerged as a promising solution, capable of utilizing unlabeled data to enhance model performance and has been widely adopted in medical image segmentation tasks. However, striking a balance between segmentation accuracy and inference speed remains a challenge for real-time segmentation. This paper provides a comprehensive review of research progress in real-time intelligent semi-supervised ultrasound video object segmentation (SUVOS) and offers insights into future developments in this area.

Key words: Ultrasound video segmentation; Semi-supervised learning; Real-time segmentation; Video object segmentation

Jin Guo, MD, Zhaojun Li, PhD, Yanping Lin, PhD. Semi-supervised Learning for Real-time Segmentation of Ultrasound Video Objects: A Review. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 333-347.

Figures/Tables 10

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Table 1

Table 2

References 74

[1]	Klibanov AL, Hossack JA. Ultrasound in radiology: from anatomic, functional, molecular imaging to drug delivery and image-guided therapy. Invest Radiol 2015; 50:657-670. doi: 10.1097/RLI.0000000000000188 pmid: 26200224
[2]	Tajbakhsh N, Jeyaseelan L, Li Q, Chiang JN, Wu Z, Ding X. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Med Image Anal 2020; 63:101693. doi: 10.1016/j.media.2020.101693
[3]	Chapelle O, Scholkopf B, Zien A. Semi-supervised learning. IEEE Transactions on Neural Networks 2009; 20:542-542.
[4]	Yao Q, Xiao L, Liu P, Zhou SK. Label-free segmentation of COVID-19 lesions in lung CT. IEEE Trans Med Imaging 2021; 40: 2808-2819. doi: 10.1109/TMI.2021.3066161
[5]	Zhang L, Wang X, Yang D, Sanford T, Harmon S, Turkbey B, et al. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans Med Imaging 2020; 39:2531-2540. doi: 10.1109/TMI.42
[6]	Jiao R, Zhang Y, Ding L, Cai R, Zhang J. Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation. arXiv preprint arXiv 2207.14191, 2022.
[7]	Zou Y, Yu Z, Liu X, Kumar B, Wang J. Confidence regularized self-training. IEEE/CVF International Conference on Computer Vision (ICCV) 2019;5981-5990.
[8]	Bai W, Oktay O, Sinclair M, Suzuki H, Rajchl M, Tarroni G, et al. Semi-supervised learning for network-based cardiac MR image segmentation.Medical Image Computing and Computer Assisted Intervention-MICCAI 2017:253-260.
[9]	Chaitanya K, Erdil E, Karani N, Konukoglu E. Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation. Medical Image Analysis 2023:102792.
[10]	He R, Yang J, Qi X. Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation.Proceedings of the IEEE/CVF International Conference on Computer Vision(CVPR) 2021: 6930-6940.
[11]	Yang L, Zhuo W, Qi L, Shi Y, Gao Y. St++:Make self-training work better for semi-supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2022:4268-4277.
[12]	Van Engelen J E, Hoos H H. A survey on semi-supervised learning. Machine learning 2020; 109:373-440. doi: 10.1007/s10994-019-05855-6
[13]	Yao H, Hu X, Li X. Enhancing pseudo label quality for semi-supervised domain-generalized medical image segmentation. Proceedings of the AAAI Conference on Artificial Intelligence 2022; 36:3099-3107.
[14]	Lee D H. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning ICML 2013; 3:896.
[15]	Xia Y, Yang D, Yu Z, Liu F, Cai J, Yu L, et al. Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation. Med Image Anal 2020; 65:101766. doi: 10.1016/j.media.2020.101766
[16]	Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 2017,30.
[17]	Laine S, Aila T. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv 1610.02242, 2016.
[18]	Li X, Yu L, Chen H, Fu CW, Xing L, Heng PA. Transformation-consistent self-ensembling model for semisupervised medical image segmentation. IEEE Trans Neural Netw Learn Syst 2020; 32:523-534. doi: 10.1109/TNNLS.5962385
[19]	Li X, Yu L, Chen H, Fu CW, Xing L, Heng PA. Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model. arXiv preprint arXiv 1808.03887,2018.
[20]	Bortsova G, Dubost F, Hogeweg L, Katramados L & De Bruijine M. Semi-supervised medical image segmentation via learning consistency under transformations. Medical Image Computing and Computer Assisted Intervention-MICCAI 2019; 22:810-818.
[21]	French G, Laine S, Aila T, Mackiewicz M, Finlayson G. Semi-supervised semantic segmentation needs strong, varied perturbations. arXiv preprint arXiv 1906.01916,2019.
[22]	Olsson V, Tranheden W, Pinto J, Svensson L. Classmix: segmentation-based data augmentation for semi-supervised learning.Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2021: 1369-1378.
[23]	Ouali Y, Hudelot C, Tami M. Semi-supervised semantic segmentation with cross-consistency training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2020:12674-12684.
[24]	Chen X, Yuan Y, Zeng G, Wang J. Semi-supervised semantic segmentation with cross pseudo supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2021:2613-2622.
[25]	Alonso I, Sabater A, Ferstl D, Montesano L, Murillo AC. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. Proceedings of the IEEE/CVF International Conference on Computer Vision(ICCV) 2021:8219-8228.
[26]	Lai X, Tian Z, Jiang L, Liu S, Zhao H, Wang L, et al. Semi-supervised semantic segmentation with directional context-aware consistency. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2021:1205-1214.
[27]	Caelles S, Maninis K K, Pont-Tuset J, Leal-Taixe L, Cremers D, Van Gool L. One-shot video object segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR) 2017:221-230.
[28]	Voigtlaender P, Leibe B. Online adaptation of convolutional neural networks for video object segmentation.arXiv preprint arXiv 1706.09364,2017.
[29]	Maninis KK, Caelles S, Chen Y, Pont-Tuset J, Leal-Taixe L, Cremers D, et al. Video object segmentation without temporal information. IEEE Trans Pattern Anal Mach Intell 2019; 41:1515-1530. doi: 10.1109/TPAMI.34
[30]	Perazzi F, Khoreva A, Benenson R, Schiele B, Sorkine-Hornung A. Learning video object segmentation from static images.Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR)2017:2663-2672.
[31]	Hu Y T, Huang J B, Schwing A. Maskrnn: instance level video object segmentation. Advances in neural information processing systems 2017;30.
[32]	Luiten J, Voigtlaender P, Leibe B. Premvos:proposal-generation, refinement and merging for video object segmentation. Computer Vision -ACCV2018: 14th Asian Conference on Computer Vision 2019:565-580.
[33]	Cheng J, Tsai Y H, Hung W C, Wang S, Yang M. Fast and accurate online video object segmentation via tracking parts.Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR)2018:7415-7424.
[34]	Chen X, Li Z, Yuan Y, Yu G, Shen J, Qi D. State-aware tracker for real-time video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2020:9384-9393.
[35]	Wang Q, Zhang L, Bertinetto L, Hu W, Torr P H. Fast online object tracking and segmentation: A unifying approach. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition(CVPR) 2019:1328-1338.
[36]	Li B, Yan J, Wu W, Zhu Z, Hu X. High performance visual tracking with Siamese region proposal network.Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2018:8971-8980.
[37]	Shin Yoon J, Rameau F, Kim J, Lee S, Shin S, Kweon I. Pixel-level matching for video object segmentation using convolutional neural networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2017:2167-2176.
[38]	Hu Y T, Huang J B, Schwing AG. Videomatch: Matching based video object segmentation.Proceedings of the European Conference on Computer Vision (ECCV) 2018: 54-70.
[39]	Oh S W, Lee J Y, Xu N, Kim S J. Video object segmentation using space-time memory networks. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2019:9226-9235.
[40]	Lu X, Wang W, Danelljan M, Zhou T, Shen J& Van Gool L. Video object segmentation with episodic graph memory networks. Computer Vision-ECCV 2020:661-679.
[41]	Li Y, Xu N, Peng J, See J, Lin W. Delving into the cyclic mechanism in semi-supervised video object segmentation. Advances in Neural Information Processing Systems 2020; 33:1218-1228.
[42]	Wang H, Jiang X, Ren H, Hu Y, Bai S. Swiftnet:Real-time video object segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) 2021:1296-1305.
[43]	Wang Q, Zheng J, Yu H, Zhang J Q, Zhang J. Automatic detection of thyroid nodules with ultrasound images: Basing on semi-supervised learning. Journal of Physics 2021; 1976:012012.
[44]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer- Assisted Intervention-MICCAI 2015; 18:234-241.
[45]	Li Y, Liu Y, Huang L, Wang Z, Luo J. Deep weakly-supervised breast tumor segmentation in ultrasound images with explicit anatomical constraints. Med Image Anal 2022; 76:102315. doi: 10.1016/j.media.2021.102315
[46]	Cao X, Chen H, Li Y, Peng Y, Wang S, Cheng L. Uncertainty aware temporal-ensembling model for semi-supervised abus mass segmentation. IEEE Trans Med Imaging 2020; 40:431-443. doi: 10.1109/TMI.42
[47]	Li Y, Liu Y, Wang Z, Luo J. Semi-supervised deep learning for breast anatomy decomposition in ultrasound images. IEEE International Ultrasonics Symposium (IUS) 2021:1-4.
[48]	Zhai D, Hu B, Gong X, Zou H, Luo J. ASS-GAN: Asymmetric semi-supervised GAN for breast ultrasound image segmentation. Neurocomputing 2022; 493:204-216. doi: 10.1016/j.neucom.2022.04.021
[49]	Huang K, Zhang Y, Cheng HD, Xing P. MSF-GAN: Multi-scale fuzzy generative adversarial network for breast ultrasound image segmentation.Annu Int Conf IEEE Eng Med Biol Soc 2021:3193-3196.
[50]	Iqbal, Ahmed, and Muhammad Sharif. PDF-UNet: a semi-supervised method for segmentation of breast tumor images using a U-shaped pyramid-dilated network.Expert Systems with Applications 2023;221.
[51]	Pang T, Wong JHD, Ng WL, Chan CS. Semi-supervised GAN-based radiomics model for data augmentation in breast ultrasound mass classification. Comput Methods Programs Biomed 2021; 203:106018. doi: 10.1016/j.cmpb.2021.106018
[52]	Han L, Huang Y, Dou H, Wang S, Ahamad S, Luo H, et al. Semi-supervised segmentation of lesion from breast ultrasound images with attentional generative adversarial network. Comput Methods Programs Biomed 2020; 189:105275. doi: 10.1016/j.cmpb.2019.105275
[53]	Xu X, Sanford T, Turkbey B, Xu S, Wood BJ, Yan P. Shadow-consistent semi-supervised learning for prostate ultrasound segmentation. IEEE Trans Med Imaging 2022; 41:1331-1345. doi: 10.1109/TMI.2021.3139999
[54]	Chen Y, Wang Y, Lai B, Chen Z, Cao X, Ye N, et al. VeniBot: towards autonomous venipuncture with semi-supervised vein segmentation from ultrasound images. arXiv preprint arXiv 2105.12945,2021.
[55]	Xie X, Niu J, Liu X, Li Q, Wang Y, Tang S. DK-Consistency: a domain knowledge guided consistency regularization method for semi-supervised breast cancer diagnosis.IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2021: 3435-3442.
[56]	Mendel R, Rauber D, de Souza LA Jr, Papa JP, Palm C. Error-correcting mean-teacher: corrections instead of consistency-targets applied to semi-supervised medical image segmentation. Comput Biol Med 2023; 154:106585. doi: 10.1016/j.compbiomed.2023.106585
[57]	Wang R. Ultrasound video object segmentation of renal parenchyma. International Core Journal of Engineering 2022; 8:101-106.
[58]	Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, et al. Attention is all you need. Advances in neural information processing systems 2017;30.
[59]	Chen C H, Lee C S, Ahmad S, Hartley-Brown M. A single center review evaluating daratumumab outcomes in multiple myeloma patients at Monter Cancer Center/Northwell Health Cancer Institute. Blood 2020; 136:25.
[60]	Wu H, Liu J, Xiao F, Wen Z, Cheng L, Qin J. Semi-supervised segmentation of echocardiography videos via noise-resilient spatiotemporal semantic calibration and fusion. Med Image Anal 2022; 78:102397. doi: 10.1016/j.media.2022.102397
[61]	Ta K, Ahn S S, Lu A, Stendahl J C, Sinusas A J, Duncan J S. A semi-supervised joint learning approach to left ventricular segmentation and motion tracking in echocardiography. IEEE 17th International Symposium on Biomedical Imaging (ISBI) 2020:1734-1737.
[62]	El Rai M C, Darweesh M, Al-Saad M. Semi-supervised segmentation of echocardiography videos using graph signal processing. Electronics 2022; 11:3462. doi: 10.3390/electronics11213462
[63]	Chen Y, Zhang C, Liu L, Dong C, Luo Y, Wan X. USCL: pretraining deep ultrasound image diagnosis model through video contrastive representation learning. Medical Image Computing and Computer Assisted Intervention-MICCAI 2021; 24:627-637.
[64]	Sirjani N, Moradi S, Oghli MG, Hosseinsabet A, Alizadehasl A, Yadollahi M, et al. Automatic cardiac evaluations using a deep video object segmentation network. Insights Imaging 2022; 13:69. doi: 10.1186/s13244-022-01212-9
[65]	Song XF, Wang YN, Feng QJ, Wang Q. Improved graph cut model with features of superpixels and neighborhood patches for myocardium segmentation from ultrasound image. Math Biosci Eng 2019; 16:1115-1137. doi: 10.3934/mbe.2019053 pmid: 30947411
[66]	Shin S Y, Lee S, Yun I D, Kim S M, Lee K M. Joint weakly and semi-supervised deep learning for localization and classification of masses in breast ultrasound images. IEEE Trans Med Imaging 2018; 38:762-774. doi: 10.1109/TMI.2018.2872031
[67]	Dai F, Li Y, Zhang G, Shi Q, Xing W, Liu X, et al. An automatic ultrasonic segmentation method by two-stage semi-supervised learning strategy.IEEE International Ultrasonics Symposium (IUS) 2022:1-4.
[68]	Judge T, Judge A, Jodoin P. Anatomically constrained semi-supervised learning for echocardiography segmentation. Medical Imaging with Deep Learning 2022.
[69]	Yang H, Shan C, Kolen AF, Peter HN. Deep Q-network-driven catheter segmentation in 3D US by hybrid constrained semi-supervised learning and dual-UNet. Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 646-655.
[70]	Dai W, Li X, Ding X, Cheng K. Cyclical self-supervision for semi-supervised ejection fraction prediction from echocardiogram videos. IEEE Transactions on Medical Imaging 2022.
[71]	Wang P, Vives M, Patel VM, Hacihaliloglu I. Robust bone shadow segmentation from 2D ultrasound through task decomposition.Medical Image Computing and Computer Assisted Intervention-MICCAI 2020:805-814.
[72]	Leclerc S, Smistad E, Pedrosa J, Ostvik A, Cervenansky F, Espinosa F, et al. Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Trans Med Imaging 2019; 38:2198-2210. doi: 10.1109/TMI.42
[73]	Ouyang D, He B, Ghorbani A, Yuan N, Ebinger J, Langlotz CP, et al. Video-based AI for beat-to-beat assessment of cardiac function. Nature 2020; 580:252-256. doi: 10.1038/s41586-020-2145-8
[74]	Sun Y, Chen G, Zhou T, Zhang Y, Liu N. Context-aware cross-level fusion network for camouflaged object detection. arXiv preprint arXiv 2105.12555,2021.

True value	Predicted value
True value	Prospects	Background
Prospects	TP	FN
Background	FP	TN

Author	Method (Based on)	Dice/mIou	FPS	Device	Dataset
Zhai D et al. [48]	Generative adversarial networks	0.8319/0.7123	17.48	1080Ti	DBUS (15% labeled)
Pang T et al. [51]	Generative adversarial networks	0.94	33.2	2060Super	Mendeley US
Wang P et al. [71]	Generative adversarial networks	0.962/0.9297	33.33	TiTan Xp	Bone US
Cao X et al. [46]	Consistency regularization	0.7287		2x2080Ti	ABUS (300 labeled)
Wu H et al. [60]	Consistency regularization	0.9379	31.25	2080Ti	CAMUS
Xie X et al. [55]	Consistency regularization	0.7951		V100	BUSI (10% labeled)
Sirjani N et al. [64]	Detection-based	0.7220	12.5	2060-A8G	Echocardiography series
Dai F et al. [67]	Detection-based	0.7594/0.6271			BUS
Wang R et al. [57]	Matching-based	0.8387	Real-time (Not verified)	2080Ti	Renal Parenchyma
B Li Y et al. [47]	Simple linear iterative clustering super-pixel	0.8823/0.8495			BUS (100 labeled)
Song X et al. [65]	Super-Pixel	0.900		CPU	MUS
El Rai M C et al. [62]	Graph Signal Processing	0.9270	Real-time (Not verified)		CAMUS (5% labeled)
Yang H et al. [69]	Uncertainty and contextual constraint loss	0.65	50	1080Ti	Heart US