A method for emotion transition recognition using cross-modal feature fusion and global perception_Journal of Biomedical Engineering

Authors：

 JIE Lilin ¹ , ZOU Yangmeng ¹ , LI Zhengxiu ¹ , LYU Baoliang ² ,  ZHENG Weilong ² , LI Ming ¹

1. Key Laboratory of Jiangxi Province for Image Processing and Pattern Recognition, Nanchang Hangkong University, Nanchang 330063, P. R. China;
2. The Center for Brain-Like Computing and Machine Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, P. R. China;

Corresponding?author：

JIE Lilin, Email: jielilin@nchu.edu.cn; ZHENG Weilong, Email: weilong@sjtu.edu.cn

Keywords：

Emotion recognition; Emotion transition; Cross-modal feature fusion; Deep canonical correlation analysis; Deep learning model

DOI：

10.7507/1001-5515.202504040

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Current studies on electroencephalogram (EEG) emotion recognition primarily concentrate on discrete stimulus paradigms under controlled laboratory settings, which cannot adequately represent the dynamic transition characteristics of emotional states during multi-context interactions. To address this issue, this paper proposes a novel method for emotion transition recognition that leverages a cross-modal feature fusion and global perception network (CFGPN). Firstly, an experimental paradigm encompassing six types of emotion transition scenarios was designed, and EEG and eye movement data were simultaneously collected from 20 participants, each annotated with dynamic continuous emotion labels. Subsequently, deep canonical correlation analysis integrated with a cross-modal attention mechanism was employed to fuse features from EEG and eye movement signals, resulting in multimodal feature vectors enriched with highly discriminative emotional information. These vectors are then input into a parallel hybrid architecture that combines convolutional neural networks (CNNs) and Transformers. The CNN is employed to capture local time-series features, whereas the Transformer leverages its robust global perception capabilities to effectively model long-range temporal dependencies, enabling accurate dynamic emotion transition recognition. The results demonstrate that the proposed method achieves the lowest mean square error in both valence and arousal recognition tasks on the dynamic emotion transition dataset and a classic multimodal emotion dataset. It exhibits superior recognition accuracy and stability when compared with five existing unimodal and six multimodal deep learning models. The approach enhances both adaptability and robustness in recognizing emotional state transitions in real-world scenarios, showing promising potential for applications in the field of biomedical engineering.

1.	Li K, Huang W, Gao W, et al. An electroencephalography-based brain-computer interface for emotion regulation with virtual reality neurofeedback. IEEE Trans Cogn Dev Syst, 2024, 16(4): 1405-1417.
2.	Houssein E H, Hammad A, Ali A A, et al. Human emotion recognition from EEG-based brain-computer interface using machine learning: a comprehensive review. Neural Comput Appl, 2022, 34(15): 12527-12557.
3.	Xu J, Hu Z, Zou J, et al. Intelligent emotion detection method based on deep learning in medical and health data. IEEE Access, 2020, 8: 3802-3811.
4.	Yu M, Xiao S, Hua M, et al. EEG-based emotion recognition in an immersive virtual reality environment: from local activity to brain network features. Biomedical Signal Processing and Control, 2022, 72(Part A): 103349.
5.	張天恒,王磊,郭苗苗,等. 虛擬現實視覺體驗對腦功能網絡的影響. 生物醫學工程學雜志, 2020, 37(2): 251-261.
6.	Liu D, Dai W, Zhang H, et al. Brain-machine coupled learning method for facial emotion recognition. IEEE Trans Pattern Anal Mach Intell, 2023, 45(9): 10703-10717.
7.	Li W, Xue J, Tan R, et al. Global-local-feature-fused driver speech emotion detection for intelligent cockpit in automated driving. IEEE Trans Intell Veh, 2023, 8(4): 2684-2697.
8.	Huang Y, Wen H, Qing L, et al. Emotion recognition based on body and context fusion in the wild//2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal: IEEE, 2021: 3602-3610.
9.	Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
10.	Zhang T, El Ali A, Wang C, et al. Weakly-supervised learning for fine-grained emotion recognition using physiological signals. IEEE Trans Affect Comput, 2023, 14(3): 2304-2322.
11.	Singh U, Shaw R, Patra B K, et al. A data augmentation and channel selection technique for grading human emotions on DEAP dataset. Biomed Signal Process Control, 2023, 79: 104060.
12.	Gu Y, Zhong X, Qu C, et al. A domain generative graph network for EEG-based emotion recognition. IEEE J Biomed Health Inform, 2023, 27(5): 2377-2386.
13.	Yang K, Yao Z, Zhang K, et al. Automatically extracting and utilizing EEG channel importance based on graph convolutional network for emotion recognition. IEEE J Biomed Health Inform, 2024, 28(8): 4588-4598.
14.	Zhang Y, Liu H, Wang D, et al. Cross-modal credibility modelling for EEG-based multimodal emotion recognition. J Neural Eng, 2024, 21(2): 1-15.
15.	Hou K, Zhang X, Yang Y, et al. Emotion recognition from multimodal physiological signals via discriminative correlation fusion with a temporal alignment mechanism. IEEE Trans Cybern, 2024, 54(4): 3079-3092.
16.	Zhang X, Liu J, Shen J, et al. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern, 2021, 51(9): 4386-4399.
17.	Skaramagkas V, Giannakakis G, Ktistakis E, et al. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng, 2023, 16: 260-277.
18.	Lu Y, Zheng W L, Li B, et al. Combining eye movements and EEG to enhance emotion recognition//Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI). Buenos Aires: AAAI Press, 2015: 1170-1176.
19.	Wu X, Zheng W L, Li Z Y, et al. Investigating EEG-based functional connectivity patterns for multimodal emotion recognition. J Neural Eng, 2022, 19(1): 016012.
20.	Eysenck S B G, Eysenck H J, Barrett P. A revised version of the psychoticism scale. Pers Individ Differ, 1985, 6(1): 21-29.
21.	Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 2015, 7(3): 162-175.
22.	Yan Y, Wu X, Li C, et al. Topological EEG nonlinear dynamics analysis for emotion recognition. IEEE Trans Cogn Dev Syst, 2023, 15(2): 625-638.
23.	Hatipoglu Yilmaz B, Kose C. A novel signal to image transformation and feature level fusion for multimodal emotion recognition. Biomed Tech, 2021, 66(4): 353-362.
24.	Huang Z T, Ma Y H, Wang R R, et al. A model for EEG-based emotion recognition: CNN-Bi-LSTM with attention mechanism. Electronics, 2023, 12(14): 3188.
25.	Li G, Yuan B, Ouyang D, et al. Emotion recognition based on selected EEG signals by common spatial pattern. IEEE Sens J, 2024, 24(6): 8414-8426.
26.	Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
27.	Zhong P, Wang D, Miao C. EEG-based emotion recognition using regularized graph neural networks. IEEE Trans Affect Comput, 2022, 13(3): 1290-1301.
28.	Ma R, Yu T, Zhong X, et al. Capsule network for ERP detection in brain-computer interface. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 718-730.
29.	Liu Q, Wu H, Zhang H, et al. Spatial-temporal Transformers for EEG emotion recognition//Proceedings of the 6th International Conference on Advances in Artificial Intelligence (ICAAI 2022). Xi’an: Northwestern Polytechnical University Press, 2022: 116-120.
30.	Song Y, Zheng Q, Liu B, et al. EEG conformer: convolutional transformer for EEG decoding and visualization. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 710-719.
31.	Qiu J L, Liu H, Lu B L. Multi-view emotion recognition using deep canonical correlation analysis//Proceedings of the 25th International Conference on Neural Information Processing (ICONIP 2018). Siem Reap: Springer, 2018: 221-231.
32.	Lan Y T, Liu W, Lu B L. Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Virtual Conference: IEEE, 2021: 1-6.
33.	Zhao Z W, Liu W, Lu B L. Multimodal emotion recognition using a modified dense co-attention symmetric network//Proceedings of the International IEEE/EMBS Conference on Neural Engineering (CNE). Virtual Conference: IEEE, 2021: 73-76.
34.	Zhou S, Huang D, Liu C, et al. Objectivity meets subjectivity: a subjective and objective feature fused neural network for emotion recognition. Appl Soft Comput, 2022, 122: 108849.
35.	Fu B L, Gu C R, Fu M, et al. A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Front Neurosci, 2023, 17: 1234162.
36.	Yin J L, Wu M C, Yang Y, et al. Research on multimodal emotion recognition based on fusion of electroencephalogram and electrooculography. IEEE Trans Instrum Meas, 2024, 73: 1-12.

1. Li K, Huang W, Gao W, et al. An electroencephalography-based brain-computer interface for emotion regulation with virtual reality neurofeedback. IEEE Trans Cogn Dev Syst, 2024, 16(4): 1405-1417.
2. Houssein E H, Hammad A, Ali A A, et al. Human emotion recognition from EEG-based brain-computer interface using machine learning: a comprehensive review. Neural Comput Appl, 2022, 34(15): 12527-12557.
3. Xu J, Hu Z, Zou J, et al. Intelligent emotion detection method based on deep learning in medical and health data. IEEE Access, 2020, 8: 3802-3811.
4. Yu M, Xiao S, Hua M, et al. EEG-based emotion recognition in an immersive virtual reality environment: from local activity to brain network features. Biomedical Signal Processing and Control, 2022, 72(Part A): 103349.
5. 張天恒,王磊,郭苗苗,等. 虛擬現實視覺體驗對腦功能網絡的影響. 生物醫學工程學雜志, 2020, 37(2): 251-261.
6. Liu D, Dai W, Zhang H, et al. Brain-machine coupled learning method for facial emotion recognition. IEEE Trans Pattern Anal Mach Intell, 2023, 45(9): 10703-10717.
7. Li W, Xue J, Tan R, et al. Global-local-feature-fused driver speech emotion detection for intelligent cockpit in automated driving. IEEE Trans Intell Veh, 2023, 8(4): 2684-2697.
8. Huang Y, Wen H, Qing L, et al. Emotion recognition based on body and context fusion in the wild//2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal: IEEE, 2021: 3602-3610.
9. Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
10. Zhang T, El Ali A, Wang C, et al. Weakly-supervised learning for fine-grained emotion recognition using physiological signals. IEEE Trans Affect Comput, 2023, 14(3): 2304-2322.
11. Singh U, Shaw R, Patra B K, et al. A data augmentation and channel selection technique for grading human emotions on DEAP dataset. Biomed Signal Process Control, 2023, 79: 104060.
12. Gu Y, Zhong X, Qu C, et al. A domain generative graph network for EEG-based emotion recognition. IEEE J Biomed Health Inform, 2023, 27(5): 2377-2386.
13. Yang K, Yao Z, Zhang K, et al. Automatically extracting and utilizing EEG channel importance based on graph convolutional network for emotion recognition. IEEE J Biomed Health Inform, 2024, 28(8): 4588-4598.
14. Zhang Y, Liu H, Wang D, et al. Cross-modal credibility modelling for EEG-based multimodal emotion recognition. J Neural Eng, 2024, 21(2): 1-15.
15. Hou K, Zhang X, Yang Y, et al. Emotion recognition from multimodal physiological signals via discriminative correlation fusion with a temporal alignment mechanism. IEEE Trans Cybern, 2024, 54(4): 3079-3092.
16. Zhang X, Liu J, Shen J, et al. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern, 2021, 51(9): 4386-4399.
17. Skaramagkas V, Giannakakis G, Ktistakis E, et al. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev Biomed Eng, 2023, 16: 260-277.
18. Lu Y, Zheng W L, Li B, et al. Combining eye movements and EEG to enhance emotion recognition//Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI). Buenos Aires: AAAI Press, 2015: 1170-1176.
19. Wu X, Zheng W L, Li Z Y, et al. Investigating EEG-based functional connectivity patterns for multimodal emotion recognition. J Neural Eng, 2022, 19(1): 016012.
20. Eysenck S B G, Eysenck H J, Barrett P. A revised version of the psychoticism scale. Pers Individ Differ, 1985, 6(1): 21-29.
21. Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 2015, 7(3): 162-175.
22. Yan Y, Wu X, Li C, et al. Topological EEG nonlinear dynamics analysis for emotion recognition. IEEE Trans Cogn Dev Syst, 2023, 15(2): 625-638.
23. Hatipoglu Yilmaz B, Kose C. A novel signal to image transformation and feature level fusion for multimodal emotion recognition. Biomed Tech, 2021, 66(4): 353-362.
24. Huang Z T, Ma Y H, Wang R R, et al. A model for EEG-based emotion recognition: CNN-Bi-LSTM with attention mechanism. Electronics, 2023, 12(14): 3188.
25. Li G, Yuan B, Ouyang D, et al. Emotion recognition based on selected EEG signals by common spatial pattern. IEEE Sens J, 2024, 24(6): 8414-8426.
26. Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
27. Zhong P, Wang D, Miao C. EEG-based emotion recognition using regularized graph neural networks. IEEE Trans Affect Comput, 2022, 13(3): 1290-1301.
28. Ma R, Yu T, Zhong X, et al. Capsule network for ERP detection in brain-computer interface. IEEE Trans Neural Syst Rehabil Eng, 2021, 29: 718-730.
29. Liu Q, Wu H, Zhang H, et al. Spatial-temporal Transformers for EEG emotion recognition//Proceedings of the 6th International Conference on Advances in Artificial Intelligence (ICAAI 2022). Xi’an: Northwestern Polytechnical University Press, 2022: 116-120.
30. Song Y, Zheng Q, Liu B, et al. EEG conformer: convolutional transformer for EEG decoding and visualization. IEEE Trans Neural Syst Rehabil Eng, 2023, 31: 710-719.
31. Qiu J L, Liu H, Lu B L. Multi-view emotion recognition using deep canonical correlation analysis//Proceedings of the 25th International Conference on Neural Information Processing (ICONIP 2018). Siem Reap: Springer, 2018: 221-231.
32. Lan Y T, Liu W, Lu B L. Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Virtual Conference: IEEE, 2021: 1-6.
33. Zhao Z W, Liu W, Lu B L. Multimodal emotion recognition using a modified dense co-attention symmetric network//Proceedings of the International IEEE/EMBS Conference on Neural Engineering (CNE). Virtual Conference: IEEE, 2021: 73-76.
34. Zhou S, Huang D, Liu C, et al. Objectivity meets subjectivity: a subjective and objective feature fused neural network for emotion recognition. Appl Soft Comput, 2022, 122: 108849.
35. Fu B L, Gu C R, Fu M, et al. A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Front Neurosci, 2023, 17: 1234162.
36. Yin J L, Wu M C, Yang Y, et al. Research on multimodal emotion recognition based on fusion of electroencephalogram and electrooculography. IEEE Trans Instrum Meas, 2024, 73: 1-12.

Journal of Biomedical Engineering

Latest ArticlesA method for emotion transition recognition using cross-modal feature fusion and global perception

Abstract Full text Figures/Tables Video References Cited by

Format

Content