Electroencephalogram emotion recognition based on state-space models combined with spatio-temporal feature_Journal of Biomedical Engineering

Authors：

 WANG Chunli , ZHANG Jiahao , DOU Jiarong , WANG Chenming , CHANG Wenwen

School of Electronic and Information Engineering, Lanzhou Jiao tong University, Lanzhou 730070, P. R. China;

Corresponding?author：

WANG Chunli, Email: 1093353186@qq.com

Keywords：

Electroencephalogram; Emotion recognition; Mamba; Deep learning; Electroencephalogram signal processing

DOI：

10.7507/1001-5515.202508024

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

To address the challenges of spatiotemporal feature heterogeneity, insufficient utilization of frequency band information, and weak cross-subject generalization in electroencephalogram (EEG)-based emotion recognition, this paper proposes a hierarchical spatiotemporal feature learning architecture named spatio-temporal mamba (ST-Mamba) based on state space models. Firstly, the proposed conv-spatio-temporal (CST) dual-branch collaborative module integrates the local feature extraction capability of convolutional neural network (CNN) with the global modeling ability of state space models. Through adaptive weighted fusion, it effectively mitigates the issue of inadequate modeling of inter-channel relationships in EEG signals. Secondly, the designed multi-band spatio-temporal feature pyramid (MBSTP) module adaptively weights features from different frequency bands via a frequency-band attention mechanism, while capturing spatial topological dependencies across brain regions through a hierarchical fusion strategy. Additionally, a data augmentation framework efficiently enhances the model’s cross-subject generalization by applying augmentations in the frequency, temporal, and spatial domains. The proposed model achieves average accuracies of 95.56% and 84.47% on the Shanghai Jiao Tong University emotion EEG dataset (SEED), version III (SEED-III) and version IV (SEED-IV), respectively. Experiments demonstrate that the state space model effectively alleviates the over-smoothing issue in deep networks, offering a novel solution to spatiotemporal heterogeneity and cross-subject generalization challenges in EEG-based emotion recognition.

Citation： WANG Chunli, ZHANG Jiahao, DOU Jiarong, WANG Chenming, CHANG Wenwen. Electroencephalogram emotion recognition based on state-space models combined with spatio-temporal feature. Journal of Biomedical Engineering, 2026, 43(2): 311-318. doi: 10.7507/1001-5515.202508024 Copy

1.	Yang K, Li M, Wang J, et al. A comprehensive review of circumplex model-based affect analysis: from representation to applications. IEEE Trans Affect Comput, 2024, 15(2): 1432-1450.
2.	Wang L, Zhang Y, Chen H, et al. EEG-based emotion recognition: a review of methods and applications. IEEE Trans Cogn Dev Syst, 2023, 15(3): 789-805.
3.	Russell J A. A circumplex model of affect. J Pers Soc Psychol, 1980, 39(6): 1161-1178.
4.	Wolpaw J R, Birbaumer N, McFarland D J, et al. Brain-computer interfaces for communication and control. Clin Neurophysiol, 2002, 113(6): 767-791.
5.	Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
6.	Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Cogn Dev Syst, 2017, 9(3): 281-293.
7.	Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
8.	Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
9.	Zhang T, Zheng W, Cui Z, et al. Spatial-temporal graph convolutional network for EEG-based emotion recognition//International Conference on Neural Information Processing. Bangkok: Springer, 2020: 413-424.
10.	Li Y, Huang J, Zhou H, et al. Graph neural network for EEG-based emotion recognition: a survey. IEEE Trans Neural Netw Learn Syst, 2023, 34(8): 4321-4335.
11.	Song T, Zheng W, Song P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 2020, 11(3): 532-541.
12.	Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9(8): 1735-1780.
13.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Advances in Neural Information Processing Systems. Long Beach: Curran Associates Inc, 2017: 5998-6008.
14.	Gu A, Dao T. Mamba: linear-time sequence modeling with selective state spaces. arXiv, 2023: 2312.00752.
15.	Yang Y, Zhang X, Zhang X, et al. MI-Mamba: a hybrid motor imagery electroencephalograph classification model with Mamba's global scanning. Ann N Y Acad Sci, 2025, 1546(1): 15288.
16.	Zhu L, Liao B, Zhang Q, et al. Vision Mamba: efficient visual representation learning with bidirectional state space model. arXiv, 2024: 2401.09417.
17.	Wang Q, Wu B, Zhu P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 11534-11542.
18.	Gu A, Johnson I, Goel K, et al. Combining recurrent, convolutional, and continuous-time models with linear state space layers//Advances in Neural Information Processing Systems. Virtual: NeurIPS, 2021: 572-585.
19.	Elfwing S, Uchibe E, Doya K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw, 2018, 107: 3-11.
20.	Hendrycks D, Gimpel K. Gaussian error linear units (GELUs). arXiv, 2016: 1606.08415.
21.	Dao T, Fu D, Ermon S, et al. FlashAttention: fast and memory-efficient exact attention with IO-awareness. J Mach Learn Res, 2023, 24(120): 1-42.
22.	Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module//Proceedings of the European Conference on Computer Vision (ECCV). Munich: Springer, 2018: 3-19.
23.	Chen H, Wang Y, Zhang J, et al. Cross-subject EEG emotion recognition using multi-source domain manifold feature selection. Comput Biol Med, 2023, 153: 106860.
24.	Li X, Zhang Y, Wang K, et al. Adaptive frequency-band analysis for EEG-based emotion recognition with deep learning. IEEE Trans Neural Syst Rehabil Eng, 2024, 32: 1024-1034.
25.	Cheng J, Liu G, Li M, et al. Data augmentation for EEG-based emotion recognition with deep learning. J Neurosci Methods, 2022, 364: 109367.
26.	Zhang R, Li P, Wang Y, et al. A novel multi-scale spatial-temporal feature learning method for EEG-based emotion recognition. IEEE Trans Instrum Meas, 2023, 72: 1-12.
27.	Dauphin Y N, Fan A, Auli M, et al. Language modeling with gated convolutional networks//Proceedings of the International Conference on Machine Learning (ICML). Sydney: PMLR, 2017: 933-941.
28.	Chollet F. Xception: deep learning with depthwise separable convolutions//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 1251-1258.
29.	Liu Y, Huang J, Wang Y, et al. Temporal feature extraction from EEG signals for emotion recognition. J Neural Eng, 2021, 18(4): 046033.
30.	Chen R T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations//Advances in Neural Information Processing Systems. Montreal: Curran Associates Inc, 2018: 6571-6583.
31.	Tang Y, Li M, Wang J, et al. Spatio-temporal attention mechanism for EEG-based emotion recognition. Biomed Signal Process Control, 2024, 87: 105419.
32.	Oh S, Lee J, Kim Y. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203.
33.	Yang J, Wang H, Li S, et al. Development of emotion recognition using multi-physiological signal information fusion technology. Biomed Eng Res, 2021, 40(4): 420-427.
34.	Lee S, Kim H, Park J, et al. Compact and efficient deep learning models for EEG-based brain-computer interfaces: a survey and benchmark study. IEEE Rev Biomed Eng, 2024, 17: 345-360.
35.	Rahmani S, Nasiri M, Saeedi P, et al. EEG-based emotion recognition using Bayesian graph convolutional neural networks with adaptive node weighting. Sci Rep, 2024, 14(1): 8921.
36.	Illendula A, Yedida R, Sharma S. Multimodal emotion classification using physiological signals//Proceedings of the ACM International Conference on Multimodal Interaction. New York: ACM, 2019: 120-125.
37.	Cortes C, Vapnik V. Support-vector networks. Mach Learn, 1995, 20(3): 273-297.
38.	Sengar S S, Hasan A B, Kumar S, et al. Generative artificial intelligence: a systematic review and applications. Multimed Tools Appl, 2025, 84(21): 23661-23700.
39.	Hochreiter S, Bengio Y, Frasconi P, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies//A field guide to dynamical recurrent networks. New York: IEEE, 2001: 237-244.
40.	Liu Z, Lin Y, Cao Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal: IEEE, 2021: 10012-10022.
41.	Li J, Zhang Z, Li X, et al. EEG-based emotion recognition via channel-wise attention and self-attention mechanisms. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1532-1542.
42.	Shao W, Xiang S, Zhong Z, et al. Hyper-graph based sparse canonical correlation analysis for the diagnosis of Alzheimer’s disease from multi-dimensional genomic data. Methods, 2021, 189: 86-94.
43.	Karim F, Majumdar S, Darabi H, et al. Multivariate LSTM-FCNs for time series classification. Neural Netw, 2023, 167: 213-228.
44.	Vasu P K A, Gabriel J, Zhu J, et al. MobileOne: an improved one millisecond mobile backbone//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE, 2023: 7907-7917.
45.	Liu Y, Tian Y, Zhao Y, et al. VMamba: visual state space model. arXiv, 2024: 2401.10166.
46.	Hatamizadeh A, Kautz J. MambaVision: a hybrid Mamba-Transformer vision backbone//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2025.

1. Yang K, Li M, Wang J, et al. A comprehensive review of circumplex model-based affect analysis: from representation to applications. IEEE Trans Affect Comput, 2024, 15(2): 1432-1450.
2. Wang L, Zhang Y, Chen H, et al. EEG-based emotion recognition: a review of methods and applications. IEEE Trans Cogn Dev Syst, 2023, 15(3): 789-805.
3. Russell J A. A circumplex model of affect. J Pers Soc Psychol, 1980, 39(6): 1161-1178.
4. Wolpaw J R, Birbaumer N, McFarland D J, et al. Brain-computer interfaces for communication and control. Clin Neurophysiol, 2002, 113(6): 767-791.
5. Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 2012, 3(1): 18-31.
6. Zheng W L, Lu B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Cogn Dev Syst, 2017, 9(3): 281-293.
7. Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
8. Schirrmeister R T, Springenberg J T, Fiederer L D J, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 2017, 38(11): 5391-5420.
9. Zhang T, Zheng W, Cui Z, et al. Spatial-temporal graph convolutional network for EEG-based emotion recognition//International Conference on Neural Information Processing. Bangkok: Springer, 2020: 413-424.
10. Li Y, Huang J, Zhou H, et al. Graph neural network for EEG-based emotion recognition: a survey. IEEE Trans Neural Netw Learn Syst, 2023, 34(8): 4321-4335.
11. Song T, Zheng W, Song P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 2020, 11(3): 532-541.
12. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput, 1997, 9(8): 1735-1780.
13. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Advances in Neural Information Processing Systems. Long Beach: Curran Associates Inc, 2017: 5998-6008.
14. Gu A, Dao T. Mamba: linear-time sequence modeling with selective state spaces. arXiv, 2023: 2312.00752.
15. Yang Y, Zhang X, Zhang X, et al. MI-Mamba: a hybrid motor imagery electroencephalograph classification model with Mamba's global scanning. Ann N Y Acad Sci, 2025, 1546(1): 15288.
16. Zhu L, Liao B, Zhang Q, et al. Vision Mamba: efficient visual representation learning with bidirectional state space model. arXiv, 2024: 2401.09417.
17. Wang Q, Wu B, Zhu P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 11534-11542.
18. Gu A, Johnson I, Goel K, et al. Combining recurrent, convolutional, and continuous-time models with linear state space layers//Advances in Neural Information Processing Systems. Virtual: NeurIPS, 2021: 572-585.
19. Elfwing S, Uchibe E, Doya K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw, 2018, 107: 3-11.
20. Hendrycks D, Gimpel K. Gaussian error linear units (GELUs). arXiv, 2016: 1606.08415.
21. Dao T, Fu D, Ermon S, et al. FlashAttention: fast and memory-efficient exact attention with IO-awareness. J Mach Learn Res, 2023, 24(120): 1-42.
22. Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module//Proceedings of the European Conference on Computer Vision (ECCV). Munich: Springer, 2018: 3-19.
23. Chen H, Wang Y, Zhang J, et al. Cross-subject EEG emotion recognition using multi-source domain manifold feature selection. Comput Biol Med, 2023, 153: 106860.
24. Li X, Zhang Y, Wang K, et al. Adaptive frequency-band analysis for EEG-based emotion recognition with deep learning. IEEE Trans Neural Syst Rehabil Eng, 2024, 32: 1024-1034.
25. Cheng J, Liu G, Li M, et al. Data augmentation for EEG-based emotion recognition with deep learning. J Neurosci Methods, 2022, 364: 109367.
26. Zhang R, Li P, Wang Y, et al. A novel multi-scale spatial-temporal feature learning method for EEG-based emotion recognition. IEEE Trans Instrum Meas, 2023, 72: 1-12.
27. Dauphin Y N, Fan A, Auli M, et al. Language modeling with gated convolutional networks//Proceedings of the International Conference on Machine Learning (ICML). Sydney: PMLR, 2017: 933-941.
28. Chollet F. Xception: deep learning with depthwise separable convolutions//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 1251-1258.
29. Liu Y, Huang J, Wang Y, et al. Temporal feature extraction from EEG signals for emotion recognition. J Neural Eng, 2021, 18(4): 046033.
30. Chen R T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations//Advances in Neural Information Processing Systems. Montreal: Curran Associates Inc, 2018: 6571-6583.
31. Tang Y, Li M, Wang J, et al. Spatio-temporal attention mechanism for EEG-based emotion recognition. Biomed Signal Process Control, 2024, 87: 105419.
32. Oh S, Lee J, Kim Y. Comparative analysis of emotion classification based on facial expression and physiological signals using deep learning. Appl Sci, 2022, 12(3): 1203.
33. Yang J, Wang H, Li S, et al. Development of emotion recognition using multi-physiological signal information fusion technology. Biomed Eng Res, 2021, 40(4): 420-427.
34. Lee S, Kim H, Park J, et al. Compact and efficient deep learning models for EEG-based brain-computer interfaces: a survey and benchmark study. IEEE Rev Biomed Eng, 2024, 17: 345-360.
35. Rahmani S, Nasiri M, Saeedi P, et al. EEG-based emotion recognition using Bayesian graph convolutional neural networks with adaptive node weighting. Sci Rep, 2024, 14(1): 8921.
36. Illendula A, Yedida R, Sharma S. Multimodal emotion classification using physiological signals//Proceedings of the ACM International Conference on Multimodal Interaction. New York: ACM, 2019: 120-125.
37. Cortes C, Vapnik V. Support-vector networks. Mach Learn, 1995, 20(3): 273-297.
38. Sengar S S, Hasan A B, Kumar S, et al. Generative artificial intelligence: a systematic review and applications. Multimed Tools Appl, 2025, 84(21): 23661-23700.
39. Hochreiter S, Bengio Y, Frasconi P, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies//A field guide to dynamical recurrent networks. New York: IEEE, 2001: 237-244.
40. Liu Z, Lin Y, Cao Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal: IEEE, 2021: 10012-10022.
41. Li J, Zhang Z, Li X, et al. EEG-based emotion recognition via channel-wise attention and self-attention mechanisms. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1532-1542.
42. Shao W, Xiang S, Zhong Z, et al. Hyper-graph based sparse canonical correlation analysis for the diagnosis of Alzheimer’s disease from multi-dimensional genomic data. Methods, 2021, 189: 86-94.
43. Karim F, Majumdar S, Darabi H, et al. Multivariate LSTM-FCNs for time series classification. Neural Netw, 2023, 167: 213-228.
44. Vasu P K A, Gabriel J, Zhu J, et al. MobileOne: an improved one millisecond mobile backbone//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE, 2023: 7907-7917.
45. Liu Y, Tian Y, Zhao Y, et al. VMamba: visual state space model. arXiv, 2024: 2401.10166.
46. Hatamizadeh A, Kautz J. MambaVision: a hybrid Mamba-Transformer vision backbone//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville: IEEE, 2025.

Journal of Biomedical Engineering

Electroencephalogram emotion recognition based on state-space models combined with spatio-temporal feature

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content