Full-size diffusion model for adaptive feature medical image fusion_Journal of Biomedical Engineering

Authors：

 DI Jing , SHI Shuhui , WANG Heran , LIANG Chan , ZHU Yunlong

School of Electronic & Information Engineering, Lanzhou Jiaotong University, Lanzhou 730070, P. R. China;

Corresponding?author：

DI Jing, Email: 46891771@qq.com

Keywords：

Medical image fusion; Full-scale diffusion model; Multi-scale denoising U-network; Multiscale feature integration module; Adaptive fusion rules

DOI：

10.7507/1001-5515.202412050

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

To address issues such as loss of detailed information, blurred target boundaries, and unclear structural hierarchy in medical image fusion, this paper proposes an adaptive feature medical image fusion network based on a full-scale diffusion model. First, a region-level feature map is generated using a kernel-based saliency map to enhance local features and boundary details. Then, a full-scale diffusion feature extraction network is employed for global feature extraction, alongside a multi-scale denoising U-shaped network designed to fully capture cross-layer information. A multi-scale feature integration module is introduced to reinforce texture details and structural information extracted by the encoder. Finally, an adaptive fusion scheme is applied to progressively fuse region-level features, global features, and source images layer by layer, enhancing the preservation of detail information. To validate the effectiveness of the proposed method, this paper validates the proposed model on the publicly available Harvard dataset and an abdominal dataset. By comparing with nine other representative image fusion methods, the proposed approach achieved improvements across seven evaluation metrics. The results demonstrate that the proposed method effectively extracts both global and local features of medical images, enhances texture details and target boundary clarity, and generates fusion image with high contrast and rich information, providing more reliable support for subsequent clinical diagnosis.

1.	Khan S U, Khan M A, Azhar M, et al. Multimodal medical image fusion towards future research: A review. Journal of King Saud University-Computer and Information Sciences, 2023, 35(8): 101733.
2.	劉云鵬, 李瑾, 王宇, 等. 圖像增強下基于生成對抗網絡和卷積神經網絡的CT與MRI融合方法. 生物醫學工程學雜志, 2023, 40(2): 208-216.
3.	Kalpana V, Vijaya Kishore V, Satyanarayana R V S. MRI and SPECT brain image analysis using image fusion//Proceedings of the Third Mobile Radio Communications and 5G Networks (MRCN), Singapore: Springer Nature Singapore, 2023: 571-586.
4.	羅文斌, 王沛, 張一偉, 等. 基于影像融合診斷前列腺癌的研究進展. 生物醫學工程學雜志, 2024, 41(5): 1078-1084.
5.	Zhang H, Xu H, Xiao Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity//Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), New York: AAAI Press, 2020, 34(7): 12797-12804.
6.	Xie X, Cui Y, Ieong C I, et al. Fusionmamba: dynamic feature enhancement for multimodal image fusion with Mamba. arXiv preprint, 2024, arXiv: 2404.09498.
7.	Qiu X, Li M, Zhang L, et al. Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication, 2019, 72: 35-46.
8.	Ganasala P, Prasad A D. Medical image fusion based on laws of texture energy measures in stationary wavelet transform domain. International Journal of Imaging Systems and Technology, 2020, 30(3): 544-557.
9.	Ma J, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion. Information Fusion, 2019, 48: 11-26.
10.	Li J, Li B, Jiang Y, et al. MSAt-GAN: a generative adversarial network based on multi-scale and deep attention mechanism for infrared and visible light image fusion. Complex & Intelligent Systems, 2022, 8(6): 4753-4781.
11.	Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
12.	Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
13.	Yue J, Fang L, Xia S, et al. Dif-fusion: towards high color fidelity in infrared and visible image fusion with diffusion models. IEEE Transactions on Image Processing, 2023, 32: 5705-5720.
14.	Zhao Z, Bai H, Zhu Y, et al. DDFM: denoising diffusion model for multi-modality image fusion//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris: IEEE, 2023: 8082-8093.
15.	劉慧, 朱積成, 王欣雨, 等. 面向醫學圖像融合的多尺度特征頻域分解濾波. 軟件學報, 2024, 35(12): 5687-5709.
16.	Tan W, Thit?n W, Xiang P, et al. Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomedical Signal Processing and Control, 2021, 64: 102280.
17.	Tang W, He F, Liu Y, et al. MATR: multimodal medical image fusion via multiscale adaptive transformer. IEEE Transactions on Image Processing, 2022, 31: 5134-5149.
18.	Xu H, Ma J, Jiang J, et al. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
19.	Zhang H, Ma J. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. International Journal of Computer Vision, 2021, 129(10): 2761-2785.
20.	Cheng C, Xu T, Wu X J. MUFusion: a general unsupervised image fusion network based on memory unit. Information Fusion, 2023, 92: 80-92.
21.	Ma J, Tang L, Fan F, et al. SwinFusion: cross-domain long-range learning for general image fusion via Swin Transformer. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.
22.	Wang W, Deng L J, Vivone G. A general image fusion framework using multi-task semi-supervised learning. Information Fusion, 2024, 108: 102414.
23.	Wen J, Khan A, Chen A, et al. High-quality fusion and visualization for MR-PET brain tumor images via multi-dimensional features. IEEE Transactions on Image Processing, 2024, 33: 3550-3563.
24.	Roberts J W, Aardt J A V, Ahmed F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2008, 2(1): 023522.
25.	Qiu D, Hu X, Liang P, et al. A deep progressive infrared and visible image fusion network. Journal of Image and Graphics, 2023, 28(1): 156-165.
26.	Singh S, Singh H, Bueno G, et al. A review of image fusion: methods, applications and performance metrics. Digital Signal Processing, 2023, 137: 104020.
27.	唐霖峰, 張浩, 徐涵, 等. 基于深度學習的圖像融合方法綜述. 中國圖象圖形學報, 2023, 28(1): 3-36.
28.	Zhao W, Wang D, Lu H. Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(4): 1102-1115.
29.	孫彬, 高云翔, 諸葛吳為, 等. 可見光與紅外圖像融合質量評價指標分析. 中國圖象圖形學報, 2023, 28(1): 144-155.
30.	Zhang X, Demiris Y. Visible and infrared image fusion using deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10535-10554.

1. Khan S U, Khan M A, Azhar M, et al. Multimodal medical image fusion towards future research: A review. Journal of King Saud University-Computer and Information Sciences, 2023, 35(8): 101733.
2. 劉云鵬, 李瑾, 王宇, 等. 圖像增強下基于生成對抗網絡和卷積神經網絡的CT與MRI融合方法. 生物醫學工程學雜志, 2023, 40(2): 208-216.
3. Kalpana V, Vijaya Kishore V, Satyanarayana R V S. MRI and SPECT brain image analysis using image fusion//Proceedings of the Third Mobile Radio Communications and 5G Networks (MRCN), Singapore: Springer Nature Singapore, 2023: 571-586.
4. 羅文斌, 王沛, 張一偉, 等. 基于影像融合診斷前列腺癌的研究進展. 生物醫學工程學雜志, 2024, 41(5): 1078-1084.
5. Zhang H, Xu H, Xiao Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity//Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI), New York: AAAI Press, 2020, 34(7): 12797-12804.
6. Xie X, Cui Y, Ieong C I, et al. Fusionmamba: dynamic feature enhancement for multimodal image fusion with Mamba. arXiv preprint, 2024, arXiv: 2404.09498.
7. Qiu X, Li M, Zhang L, et al. Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication, 2019, 72: 35-46.
8. Ganasala P, Prasad A D. Medical image fusion based on laws of texture energy measures in stationary wavelet transform domain. International Journal of Imaging Systems and Technology, 2020, 30(3): 544-557.
9. Ma J, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion. Information Fusion, 2019, 48: 11-26.
10. Li J, Li B, Jiang Y, et al. MSAt-GAN: a generative adversarial network based on multi-scale and deep attention mechanism for infrared and visible light image fusion. Complex & Intelligent Systems, 2022, 8(6): 4753-4781.
11. Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
12. Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.
13. Yue J, Fang L, Xia S, et al. Dif-fusion: towards high color fidelity in infrared and visible image fusion with diffusion models. IEEE Transactions on Image Processing, 2023, 32: 5705-5720.
14. Zhao Z, Bai H, Zhu Y, et al. DDFM: denoising diffusion model for multi-modality image fusion//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris: IEEE, 2023: 8082-8093.
15. 劉慧, 朱積成, 王欣雨, 等. 面向醫學圖像融合的多尺度特征頻域分解濾波. 軟件學報, 2024, 35(12): 5687-5709.
16. Tan W, Thit?n W, Xiang P, et al. Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomedical Signal Processing and Control, 2021, 64: 102280.
17. Tang W, He F, Liu Y, et al. MATR: multimodal medical image fusion via multiscale adaptive transformer. IEEE Transactions on Image Processing, 2022, 31: 5134-5149.
18. Xu H, Ma J, Jiang J, et al. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
19. Zhang H, Ma J. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. International Journal of Computer Vision, 2021, 129(10): 2761-2785.
20. Cheng C, Xu T, Wu X J. MUFusion: a general unsupervised image fusion network based on memory unit. Information Fusion, 2023, 92: 80-92.
21. Ma J, Tang L, Fan F, et al. SwinFusion: cross-domain long-range learning for general image fusion via Swin Transformer. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.
22. Wang W, Deng L J, Vivone G. A general image fusion framework using multi-task semi-supervised learning. Information Fusion, 2024, 108: 102414.
23. Wen J, Khan A, Chen A, et al. High-quality fusion and visualization for MR-PET brain tumor images via multi-dimensional features. IEEE Transactions on Image Processing, 2024, 33: 3550-3563.
24. Roberts J W, Aardt J A V, Ahmed F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2008, 2(1): 023522.
25. Qiu D, Hu X, Liang P, et al. A deep progressive infrared and visible image fusion network. Journal of Image and Graphics, 2023, 28(1): 156-165.
26. Singh S, Singh H, Bueno G, et al. A review of image fusion: methods, applications and performance metrics. Digital Signal Processing, 2023, 137: 104020.
27. 唐霖峰, 張浩, 徐涵, 等. 基于深度學習的圖像融合方法綜述. 中國圖象圖形學報, 2023, 28(1): 3-36.
28. Zhao W, Wang D, Lu H. Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 29(4): 1102-1115.
29. 孫彬, 高云翔, 諸葛吳為, 等. 可見光與紅外圖像融合質量評價指標分析. 中國圖象圖形學報, 2023, 28(1): 144-155.
30. Zhang X, Demiris Y. Visible and infrared image fusion using deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10535-10554.

Journal of Biomedical Engineering

Latest ArticlesFull-size diffusion model for adaptive feature medical image fusion

Abstract Full text Figures/Tables Video References Cited by

Format

Content