Research on three-dimensional skull repair by combining residual and informer attention_Journal of Biomedical Engineering

Authors：

QIN Chuanbo ¹ , ZENG Junbo ^1,2 , ZHENG Bin ¹ ,  ZENG Junying ¹ , ZHAI Yikui ¹ , ZHANG Wenguang ³ , YAN Jingwen ⁴

1. Faculty of Intelligent Manufacturing, Wuyi University, Jiangmen, Guangdong 529020, P.R.China;
2. Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian 361102, P.R.China;
3. Department of Neurosurgery, Jiangmen Central Hospital, Jiangmen, Guangdong 529020, P.R.China;
4. College of Engineering, Shantou University, Shantou, Guangdong 515063, P.R.China;

Corresponding?author：

ZENG Junying, Email: zengjunying@126.com

Keywords：

Cranial implant design; Informer attention; Residual block

DOI：

10.7507/1001-5515.202202047

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Cranial defects may result from clinical brain tumor surgery or accidental trauma. The defect skulls require hand-designed skull implants to repair. The edge of the skull implant needs to be accurately matched to the boundary of the skull wound with various defects. For the manual design of cranial implants, it is time-consuming and technically demanding, and the accuracy is low. Therefore, an informer residual attention U-Net (IRA-Unet) for the automatic design of three-dimensional (3D) skull implants was proposed in this paper. Informer was applied from the field of natural language processing to the field of computer vision for attention extraction. Informer attention can extract attention and make the model focus more on the location of the skull defect. Informer attention can also reduce the computation and parameter count from N² to log(N). Furthermore,the informer residual attention is constructed. The informer attention and the residual are combined and placed in the position of the model close to the output layer. Thus, the model can select and synthesize the global receptive field and local information to improve the model accuracy and speed up the model convergence. In this paper, the open data set of the AutoImplant 2020 was used for training and testing, and the effects of direct and indirect acquisition of skull implants on the results were compared and analyzed in the experimental part. The experimental results show that the performance of the model is robust on the test set of 110 cases fromAutoImplant 2020. The Dice coefficient and Hausdorff distance are 0.940 4 and 3.686 6, respectively. The proposed model reduces the resources required to run the model while maintaining the accuracy of the cranial implant shape, and effectively assists the surgeon in automating the design of efficient cranial repair, thereby improving the quality of the patient’s postoperative recovery.

Citation： QIN Chuanbo, ZENG Junbo, ZHENG Bin, ZENG Junying, ZHAI Yikui, ZHANG Wenguang, YAN Jingwen. Research on three-dimensional skull repair by combining residual and informer attention. Journal of Biomedical Engineering, 2022, 39(5): 897-908. doi: 10.7507/1001-5515.202202047 Copy

1.	Gall M, Li X, Chen X, et al. Computer-aided planning and reconstruction of cranial 3D implants//IEEE engineering in medicine and biology society, Orlando: IEEE, 2016: 1179-1183.
2.	Di A L, Di S P, Governi L, et al. A robust and automatic method for the best symmetry plane detection of craniofacial skeletons. Symmetry, 2019, 11(2): 245.
3.	Li J, Pepe A, Gsaxner C, et al. A baseline approach for autoimplant: the MICCAI 2020 cranial implant design challenge//Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures. Cham: Springer, 2020: 75-84.
4.	冀宏, 錢旭升, 周志勇, 等. 基于級聯多尺度卷積網絡的計算機斷層掃描圖像腎腫瘤自動分割方法. 生物醫學工程學雜志, 2021, 38(4): 722-731.
5.	張恒良, 李鏘, 關欣. 一種改進的三維雙路徑腦腫瘤圖像分割網絡. 光學學報, 2021, 41(3): 60-67.
6.	汪豪, 吉邦寧, 何剛, 等. 一種提高直腸癌診斷精度的基于U型網絡和殘差塊的電子計算機斷層掃描圖像分割算法. 生物醫學工程學雜志, 2022, 39(1): 166-174.
7.	Wang Bomin, Liu Zhi, Li Yujun, et al. Cranial implant design using a deep learning method with anatomical regularization//Cranial Implant Design Challenge, Cham: Springer, 2020: 85-93.
8.	Matzkin F, Newcombe V, Glocker B, et al. Cranial implant design via virtual craniectomy with shape priors//Cranial Implant Design Challenge, Cham: Springer, 2020: 37-46.
9.	Jin Yuan, Li Jianning, Egger J. High-resolution cranial implant prediction via patch-wise training. Lecture Notes in Computer Science, Cham: Springer, 2020, 12439: 94-103.
10.	Ellis D G, Aizenberg M R. Deep learning using augmentation via registration: 1st place solution to the autoimplant 2020 challenge. Lecture Notes in Computer Science, Cham: Springer, 2020, 12439: 47-55.
11.	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. arXiv: 1505.04597, 2015. https: //doi.org/10.48550/arXiv.1505.04597.
12.	Li J, Pimentel P, Szengel A, et al. AutoImplant 2020-First MICCAI Challenge on Automatic Cranial Implant Design. IEEE Transactions on Medical Imaging, 2021, 40(9): 2329-2342.
13.	Song X, Guo H, Xu X, et al. Cross-modal attention for MRI and ultrasound volume registration//24th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2021), Strasbourg: MICCAI, 2021. https: //doi.org/10.1007/978-3-030-87202-1_7.
14.	Sinha A, Dolz J. Multi-scale self-guided attention for medical image segmentation. IEEE J Biomed Health Inform, 2021, 25(1): 121-130.
15.	Beltagy I, Peters M E, Cohan A. Longformer: the long-document transformer. arXiv: 2004.05150, 2020. https: //doi.org/10.48550/arXiv.2004.05150.
16.	Li Shiyang, Jin Xiaoyong, Xuan Yao, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. arXiv: 1907.00235, 2019. https: //doi.org/10.48550/arXiv.1907.00235.
17.	Child R, Gray S, Radford A, et al. Generating long sequences with sparse transformers. arXiv: 1904.10509, 2019. https: //doi.org/10.48550/arXiv.1904.10509.
18.	Zhou H, Zhang S, Peng J, et al. Informer: Beyond efficient transformer for long sequence time-series forecasting. arXiv: 2012.07436, 2020. https: //doi.org/10.48550/arXiv.2012.07436.
19.	Srinivas A, Lin T Y, Parmar N, et al. Bottleneck transformers for visual recognition. arXiv: 2101.11605, 2021. https: //doi.org/10.48550/arXiv.2101.11605.
20.	Han Jun, Moraga C. The influence of the sigmoid function parameters on the speed of backpropagation learning. The International Workshop on Artificial Neural Networks: From Natural to Artificial Neural Computation(IWANN 1996), Berlin: Springer, 1995: 195-201.
21.	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016. DOI: 10.1109/CVPR.2016.90.
22.	Tsai Y H H, Bai Shaojie, Yamada M, et al. Transformer dissection: a unified understanding of transformer's attention via the lens of kernel. arXiv: 1908.11775, 2019. https: //doi.org/10.48550/arXiv.1908.11775.
23.	Memisevic R, Zach C, Pollefeys M, et al. Gated softmax classification//Advances in Neural Information Processing Systems, Vancouver: NIPS, 2010, 23: 1603-1611.
24.	Kingad A. A method for stochastic optimization. Anon//International Conference on Learning Representations, SanDego: ICLR, 2015.
25.	Milletari F, Navab N, Ahmadi S A. V-net: fully convolutional neural networks for volumetric medical image segmentation//2016 fourth international conference on 3D vision, IEEE, 2016: 565-571.
26.	Montavon G, Orr G B, Müller K R. Neural networks: tricks of the trade. Springer, 2012. DOI: 10.1007/978-3-642-35289-8.
27.	Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
28.	Huttenlocher D P, Klanderman G A, Rucklidge W J. Comparing images using the Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1993, 15(9): 850-863.
29.	Shi Haochen, Chen Xiaojun. Cranial implant design through multiaxial slice inpainting using deep learning//Cranial Implant Design Challenge, Cham: Springer, 2020: 28-36.
30.	Kodym O, Španel M, Herout A. Cranial defect reconstruction using cascaded CNN with alignment//Cranial Implant Design Challenge, Cham: Springer, 2020: 56-64.
31.	Pimentel P, Szengel A, Ehlke M, et al. Automated virtual reconstruction of large skull defects using statistical shape models and generative adversarial networks//Cranial Implant Design Challenge, Cham: Springer, 2020: 16-27.
32.	Mainprize J G, Fishman Z, Hardisty M R. Shape completion by U-Net: an approach to the autoimplant MICCAI cranial implant design challenge//Cranial Implant Design Challenge, Cham: Springer, 2020: 65-76.
33.	Bayat A, Shit S, Kilian A, et al. Cranial implant prediction using low-resolution 3D shape completion and high-resolution 2D refinement//Cranial Implant Design Challenge, Cham: Springer, 2020: 77-84.

1. Gall M, Li X, Chen X, et al. Computer-aided planning and reconstruction of cranial 3D implants//IEEE engineering in medicine and biology society, Orlando: IEEE, 2016: 1179-1183.
2. Di A L, Di S P, Governi L, et al. A robust and automatic method for the best symmetry plane detection of craniofacial skeletons. Symmetry, 2019, 11(2): 245.
3. Li J, Pepe A, Gsaxner C, et al. A baseline approach for autoimplant: the MICCAI 2020 cranial implant design challenge//Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures. Cham: Springer, 2020: 75-84.
4. 冀宏, 錢旭升, 周志勇, 等. 基于級聯多尺度卷積網絡的計算機斷層掃描圖像腎腫瘤自動分割方法. 生物醫學工程學雜志, 2021, 38(4): 722-731.
5. 張恒良, 李鏘, 關欣. 一種改進的三維雙路徑腦腫瘤圖像分割網絡. 光學學報, 2021, 41(3): 60-67.
6. 汪豪, 吉邦寧, 何剛, 等. 一種提高直腸癌診斷精度的基于U型網絡和殘差塊的電子計算機斷層掃描圖像分割算法. 生物醫學工程學雜志, 2022, 39(1): 166-174.
7. Wang Bomin, Liu Zhi, Li Yujun, et al. Cranial implant design using a deep learning method with anatomical regularization//Cranial Implant Design Challenge, Cham: Springer, 2020: 85-93.
8. Matzkin F, Newcombe V, Glocker B, et al. Cranial implant design via virtual craniectomy with shape priors//Cranial Implant Design Challenge, Cham: Springer, 2020: 37-46.
9. Jin Yuan, Li Jianning, Egger J. High-resolution cranial implant prediction via patch-wise training. Lecture Notes in Computer Science, Cham: Springer, 2020, 12439: 94-103.
10. Ellis D G, Aizenberg M R. Deep learning using augmentation via registration: 1st place solution to the autoimplant 2020 challenge. Lecture Notes in Computer Science, Cham: Springer, 2020, 12439: 47-55.
11. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. arXiv: 1505.04597, 2015. https: //doi.org/10.48550/arXiv.1505.04597.
12. Li J, Pimentel P, Szengel A, et al. AutoImplant 2020-First MICCAI Challenge on Automatic Cranial Implant Design. IEEE Transactions on Medical Imaging, 2021, 40(9): 2329-2342.
13. Song X, Guo H, Xu X, et al. Cross-modal attention for MRI and ultrasound volume registration//24th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2021), Strasbourg: MICCAI, 2021. https: //doi.org/10.1007/978-3-030-87202-1_7.
14. Sinha A, Dolz J. Multi-scale self-guided attention for medical image segmentation. IEEE J Biomed Health Inform, 2021, 25(1): 121-130.
15. Beltagy I, Peters M E, Cohan A. Longformer: the long-document transformer. arXiv: 2004.05150, 2020. https: //doi.org/10.48550/arXiv.2004.05150.
16. Li Shiyang, Jin Xiaoyong, Xuan Yao, et al. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. arXiv: 1907.00235, 2019. https: //doi.org/10.48550/arXiv.1907.00235.
17. Child R, Gray S, Radford A, et al. Generating long sequences with sparse transformers. arXiv: 1904.10509, 2019. https: //doi.org/10.48550/arXiv.1904.10509.
18. Zhou H, Zhang S, Peng J, et al. Informer: Beyond efficient transformer for long sequence time-series forecasting. arXiv: 2012.07436, 2020. https: //doi.org/10.48550/arXiv.2012.07436.
19. Srinivas A, Lin T Y, Parmar N, et al. Bottleneck transformers for visual recognition. arXiv: 2101.11605, 2021. https: //doi.org/10.48550/arXiv.2101.11605.
20. Han Jun, Moraga C. The influence of the sigmoid function parameters on the speed of backpropagation learning. The International Workshop on Artificial Neural Networks: From Natural to Artificial Neural Computation(IWANN 1996), Berlin: Springer, 1995: 195-201.
21. He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas: IEEE, 2016. DOI: 10.1109/CVPR.2016.90.
22. Tsai Y H H, Bai Shaojie, Yamada M, et al. Transformer dissection: a unified understanding of transformer's attention via the lens of kernel. arXiv: 1908.11775, 2019. https: //doi.org/10.48550/arXiv.1908.11775.
23. Memisevic R, Zach C, Pollefeys M, et al. Gated softmax classification//Advances in Neural Information Processing Systems, Vancouver: NIPS, 2010, 23: 1603-1611.
24. Kingad A. A method for stochastic optimization. Anon//International Conference on Learning Representations, SanDego: ICLR, 2015.
25. Milletari F, Navab N, Ahmadi S A. V-net: fully convolutional neural networks for volumetric medical image segmentation//2016 fourth international conference on 3D vision, IEEE, 2016: 565-571.
26. Montavon G, Orr G B, Müller K R. Neural networks: tricks of the trade. Springer, 2012. DOI: 10.1007/978-3-642-35289-8.
27. Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
28. Huttenlocher D P, Klanderman G A, Rucklidge W J. Comparing images using the Hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1993, 15(9): 850-863.
29. Shi Haochen, Chen Xiaojun. Cranial implant design through multiaxial slice inpainting using deep learning//Cranial Implant Design Challenge, Cham: Springer, 2020: 28-36.
30. Kodym O, Španel M, Herout A. Cranial defect reconstruction using cascaded CNN with alignment//Cranial Implant Design Challenge, Cham: Springer, 2020: 56-64.
31. Pimentel P, Szengel A, Ehlke M, et al. Automated virtual reconstruction of large skull defects using statistical shape models and generative adversarial networks//Cranial Implant Design Challenge, Cham: Springer, 2020: 16-27.
32. Mainprize J G, Fishman Z, Hardisty M R. Shape completion by U-Net: an approach to the autoimplant MICCAI cranial implant design challenge//Cranial Implant Design Challenge, Cham: Springer, 2020: 65-76.
33. Bayat A, Shit S, Kilian A, et al. Cranial implant prediction using low-resolution 3D shape completion and high-resolution 2D refinement//Cranial Implant Design Challenge, Cham: Springer, 2020: 77-84.

Journal of Biomedical Engineering

Research on three-dimensional skull repair by combining residual and informer attention

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content