Research on emotion recognition methods based on multi-modal physiological signal feature fusion_Journal of Biomedical Engineering

Authors：

ZHANG Zhiwen ^1,2 ,  YU Naigong ^1,2 , BIAN Yan ^3,4 , YAN Jinhan ^1,2

1. School of Information Science and Technology, Beijing University of Technology, Beijing 100124, P. R. China;
2. Beijing Key Laboratory of Computational Intelligence and Intelligent Systems, Beijing 100124, P. R. China;
3. School of Automation and Electrical Engineering, Tianjin Polytechnic Normal University, Tianjin 300222, P. R. China;
4. Tianjin Key Laboratory of Information Sensing and Intelligent Control, Tianjin 300222, P. R. China;

Corresponding?author：

YU Naigong, Email: yunaigong@bjut.edu.cn

Keywords：

Emotion; Multi-modality; Feature fusion; Electroencephalogram; Electromyogram; Electrodermal activity

DOI：

10.7507/1001-5515.202401020

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Emotion classification and recognition is a crucial area in emotional computing. Physiological signals, such as electroencephalogram (EEG), provide an accurate reflection of emotions and are difficult to disguise. However, emotion recognition still faces challenges in single-modal signal feature extraction and multi-modal signal integration. This study collected EEG, electromyogram (EMG), and electrodermal activity (EDA) signals from participants under three emotional states: happiness, sadness, and fear. A feature-weighted fusion method was applied for integrating the signals, and both support vector machine (SVM) and extreme learning machine (ELM) were used for classification. The results showed that the classification accuracy was highest when the fusion weights were set to EEG 0.7, EMG 0.15, and EDA 0.15, achieving accuracy rates of 80.19% and 82.48% for SVM and ELM, respectively. These rates represented an improvement of 5.81% and 2.95% compared to using EEG alone. This study offers methodological support for emotion classification and recognition using multi-modal physiological signals.

Citation： ZHANG Zhiwen, YU Naigong, BIAN Yan, YAN Jinhan. Research on emotion recognition methods based on multi-modal physiological signal feature fusion. Journal of Biomedical Engineering, 2025, 42(1): 17-23. doi: 10.7507/1001-5515.202401020 Copy

1.	張迪, 萬柏坤, 明東. 基于生理信號的情緒識別研究進展. 生物醫學工程學雜志, 2015, 32(1): 229-234.
2.	楊捷鴻, 焦學軍, 曹勇, 等. 多生理信號信息融合技術的情緒識別發展. 生物醫學工程研究, 2021, 40(4): 420-427.
3.	Illendula A, Sheth A. Multimodal emotion classification. CoRR, 2019(3): 120-125.
4.	王雪霏, 嚴欣, 周藝. 基于生理數據測量的景觀情緒體驗研究綜述. 城市建筑, 2023, 20(9): 144-147,159.
5.	陳恬, 狄海波. 皮膚電在情緒識別研究中的應用進展. 健康研究, 2019, 39(1): 47-49.
6.	Oh S J, Kim D K. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203-1215.
7.	Liu Y J, Yu M, Zhao G, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE T Affect Comput, 2017, PP(99): 1.
8.	Arya R, Singh J, Kumar A. A survey of multidisciplinary domains contributing to affective computing. Comput Sci Rev, 2021, 40(3): 100399.
9.	Maxime M, Estefania B, Kim C, et al. Multimodal emotion perception in young and elderly patients with multiple sclerosis. Mult Scler Relat Dis, 2022, 58(1): 230-238.
10.	Sousa A, d'Aquin M, Zarrouk M, et al. Person-independent multimodal emotion detection for children with high-functioning autism. Ceur Workshop Proceedings, 2020, 2760(13): 1598-1610.
11.	Morency L P, Mihalcea R, Doshi P. Towards multi-modal sentiment analysis: harvesting opinions from the web// Proceedings of the 13th International Con-ference on Multimodal Interfaces. Alicante: IEEE, 2011: 169-176.
12.	Zadeh A, Zellers R, Pincus E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intel Sys, 2016, 31(6): 82-88.
13.	Pérez-Rosas V, Mihalcea R, Morency L P. Utterance-level multimodal sentiment analysis// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia: IEEE, 2013: 973-982.
14.	牛曉偉, 劉光遠. 用于生理信號情感識別的自適應遺傳算法. 計算機工程與設計, 2008(14): 3726-3728,3731.
15.	Hassan M M, Alam M G R, Uddin M Z, et al. Human emotion recognition using deep belief network architecture. Inform Fusion, 2019, 51: 10-18.
16.	李芳, 朱昭紅, 白學軍. 高興和悲傷電影片段誘發情緒的有效性和時間進程. 心理與行為研究, 2009, 7(1): 32-38.
17.	Mert A, Akan A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl, 2018, 21(1): 1-9.
18.	Chen Z. Research on cognitive differences on cerebral left and right cognition in the processing of emotional vocabulary. AP, 2016, 6(3): 290-298.
19.	Liu J, Su Y, Liu Y. Multi-modal emotion recognition with temporal-band attention based on LSTM-RNN. Cham: Springer, 2017.
20.	凌文芬, 陳思含, 彭勇, 等. 基于3D分層卷積融合的多模態生理信號情緒識別. 智能科學與技術學報, 2021, 3(1): 76-84.
21.	Guo K, Candra H, Yu H, et al. EEG-based emotion classification using innovative features and combined SVM and HMM classifier// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju Island: IEEE, 2017: 489-492.
22.	汪佳衡, 王躍明, 姚林. 基于濾波器組長短時記憶網絡的腦電信號情緒識別. 生物醫學工程學雜志, 2021, 38(3): 447-454.
23.	陳東偉, 陳俊杰. 情感識別中腦電信號Lempel-Ziv復雜度的研究. 太原理工大學學報, 2014, 45(6): 758-763.
24.	陳沙利. 基于外周生理信號的情緒識別研究. 杭州: 浙江大學, 2021.
25.	鄧麗娜, 王曉亮. 基于生理信號的情緒識別研究綜述. 物聯網技術, 2021, 11(7): 33-41.
26.	趙運基, 范存良, 張新良. 融合多特征和通道感知的目標跟蹤算法. 計算機科學與探索, 2022, 16(6): 1417-1428.
27.	Alanazi S A, Alruwaili M, Ahmad F, et al. Estimation of organizational competitiveness by a hybrid of one-dimensional convolutional neural networks and self-organizing maps using physiological signals for emotional analysis of employees. Sensors (Basel), 2021, 21(11): 3760.
28.	杜思清. 基于皮膚電信號的心理應激建模的研究. 南京: 東南大學, 2019.
29.	李佳澤, 梅紅巖, 賈麗云, 等. 動態時間序列建模的多模態情感識別方法. 計算機工程與應用, 2025, 61(1): 196-205.
30.	Domínguez-Jiménez A, Campo-Landines K C, Martínez-Santos J C, et al. A machine learning model for emotion recognition from physiological signals. Biomed Signal Proces, 2020, 55(C): 101646.
31.	Shu L, Xie J, Yang M, et al. A review of emotion recognition using physiological signals. Sensors, 2018, 18(7): 2074.

1. 張迪, 萬柏坤, 明東. 基于生理信號的情緒識別研究進展. 生物醫學工程學雜志, 2015, 32(1): 229-234.
2. 楊捷鴻, 焦學軍, 曹勇, 等. 多生理信號信息融合技術的情緒識別發展. 生物醫學工程研究, 2021, 40(4): 420-427.
3. Illendula A, Sheth A. Multimodal emotion classification. CoRR, 2019(3): 120-125.
4. 王雪霏, 嚴欣, 周藝. 基于生理數據測量的景觀情緒體驗研究綜述. 城市建筑, 2023, 20(9): 144-147,159.
5. 陳恬, 狄海波. 皮膚電在情緒識別研究中的應用進展. 健康研究, 2019, 39(1): 47-49.
6. Oh S J, Kim D K. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203-1215.
7. Liu Y J, Yu M, Zhao G, et al. Real-time movie-induced discrete emotion recognition from EEG signals. IEEE T Affect Comput, 2017, PP(99): 1.
8. Arya R, Singh J, Kumar A. A survey of multidisciplinary domains contributing to affective computing. Comput Sci Rev, 2021, 40(3): 100399.
9. Maxime M, Estefania B, Kim C, et al. Multimodal emotion perception in young and elderly patients with multiple sclerosis. Mult Scler Relat Dis, 2022, 58(1): 230-238.
10. Sousa A, d'Aquin M, Zarrouk M, et al. Person-independent multimodal emotion detection for children with high-functioning autism. Ceur Workshop Proceedings, 2020, 2760(13): 1598-1610.
11. Morency L P, Mihalcea R, Doshi P. Towards multi-modal sentiment analysis: harvesting opinions from the web// Proceedings of the 13th International Con-ference on Multimodal Interfaces. Alicante: IEEE, 2011: 169-176.
12. Zadeh A, Zellers R, Pincus E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intel Sys, 2016, 31(6): 82-88.
13. Pérez-Rosas V, Mihalcea R, Morency L P. Utterance-level multimodal sentiment analysis// Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Sofia: IEEE, 2013: 973-982.
14. 牛曉偉, 劉光遠. 用于生理信號情感識別的自適應遺傳算法. 計算機工程與設計, 2008(14): 3726-3728,3731.
15. Hassan M M, Alam M G R, Uddin M Z, et al. Human emotion recognition using deep belief network architecture. Inform Fusion, 2019, 51: 10-18.
16. 李芳, 朱昭紅, 白學軍. 高興和悲傷電影片段誘發情緒的有效性和時間進程. 心理與行為研究, 2009, 7(1): 32-38.
17. Mert A, Akan A. Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl, 2018, 21(1): 1-9.
18. Chen Z. Research on cognitive differences on cerebral left and right cognition in the processing of emotional vocabulary. AP, 2016, 6(3): 290-298.
19. Liu J, Su Y, Liu Y. Multi-modal emotion recognition with temporal-band attention based on LSTM-RNN. Cham: Springer, 2017.
20. 凌文芬, 陳思含, 彭勇, 等. 基于3D分層卷積融合的多模態生理信號情緒識別. 智能科學與技術學報, 2021, 3(1): 76-84.
21. Guo K, Candra H, Yu H, et al. EEG-based emotion classification using innovative features and combined SVM and HMM classifier// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju Island: IEEE, 2017: 489-492.
22. 汪佳衡, 王躍明, 姚林. 基于濾波器組長短時記憶網絡的腦電信號情緒識別. 生物醫學工程學雜志, 2021, 38(3): 447-454.
23. 陳東偉, 陳俊杰. 情感識別中腦電信號Lempel-Ziv復雜度的研究. 太原理工大學學報, 2014, 45(6): 758-763.
24. 陳沙利. 基于外周生理信號的情緒識別研究. 杭州: 浙江大學, 2021.
25. 鄧麗娜, 王曉亮. 基于生理信號的情緒識別研究綜述. 物聯網技術, 2021, 11(7): 33-41.
26. 趙運基, 范存良, 張新良. 融合多特征和通道感知的目標跟蹤算法. 計算機科學與探索, 2022, 16(6): 1417-1428.
27. Alanazi S A, Alruwaili M, Ahmad F, et al. Estimation of organizational competitiveness by a hybrid of one-dimensional convolutional neural networks and self-organizing maps using physiological signals for emotional analysis of employees. Sensors (Basel), 2021, 21(11): 3760.
28. 杜思清. 基于皮膚電信號的心理應激建模的研究. 南京: 東南大學, 2019.
29. 李佳澤, 梅紅巖, 賈麗云, 等. 動態時間序列建模的多模態情感識別方法. 計算機工程與應用, 2025, 61(1): 196-205.
30. Domínguez-Jiménez A, Campo-Landines K C, Martínez-Santos J C, et al. A machine learning model for emotion recognition from physiological signals. Biomed Signal Proces, 2020, 55(C): 101646.
31. Shu L, Xie J, Yang M, et al. A review of emotion recognition using physiological signals. Sensors, 2018, 18(7): 2074.

Journal of Biomedical Engineering

Research on emotion recognition methods based on multi-modal physiological signal feature fusion

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content