Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals_Journal of Biomedical Engineering

Authors：

 YANG Wenyang ^1,2 , XU Kexin ¹

1. College of Computer, Xi'an Shiyou University, Xi’an 710065, P. R. China;
2. Key Laboratory of Modern Teaching Technology, Ministry of Education, Xi’an 710062, P. R. China;

Corresponding?author：

Keywords：

Emotion recognition; Electroencephalogram signal; Virtual reality technology; Machine learning

DOI：

10.7507/1001-5515.202310045

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Emotion recognition refers to the process of determining and identifying an individual's current emotional state by analyzing various signals such as voice, facial expressions, and physiological indicators etc. Using electroencephalogram (EEG) signals and virtual reality (VR) technology for emotion recognition research helps to better understand human emotional changes, enabling applications in areas such as psychological therapy, education, and training to enhance people’s quality of life. However, there is a lack of comprehensive review literature summarizing the combined researches of EEG signals and VR environments for emotion recognition. Therefore, this paper summarizes and synthesizes relevant research from the past five years. Firstly, it introduces the relevant theories of VR and EEG signal emotion recognition. Secondly, it focuses on the analysis of emotion induction, feature extraction, and classification methods in emotion recognition using EEG signals within VR environments. The article concludes by summarizing the research’s application directions and providing an outlook on future development trends, aiming to serve as a reference for researchers in related fields.

Citation： YANG Wenyang, XU Kexin. Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals. Journal of Biomedical Engineering, 2024, 41(2): 389-397. doi: 10.7507/1001-5515.202310045 Copy

1.	Jafari M, Shoeibi A, Khodatars M, et al. Emotion recognition in EEG signals using deep learning methods: a review. Computers in Biology and Medicine, 2023, 165: 107450.
2.	Valentina M, Francesca B, Chiara S, et al. How do emotions elicited in virtual reality affect our memory? a systematic review. Computers in Human Behavior, 2023, 146: 107812.
3.	楊俊峰. 虛擬現實產業邁入快車道. 人民日報海外版, 2022. [2022-11-23(8)]. DOI: 10.28656/n.cnki.nrmrh.2022.004035.
4.	Hofmann S M, Klotzsche F, Mariola A, et al. Decoding subjective emotional arousal from EEG during an immersive virtual reality experience. Elife, 2021, 10: e64812.
5.	Ekman, P. Basic emotions. Handbook of Cognition and Emotion, 1999, 98(16): 45-60.
6.	Xie J, Lan P, Wang S, et al. Brain activation differences of six basic emotions between 2D screen and virtual reality modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022. DOI: 10.1109/TNSRE.2022.3229389.
7.	Russell J A. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161-1178.
8.	Ricci G, De Crescenzio F, Santhosh S, et al. Relationship between electroencephalographic data and comfort perception captured in a virtual reality design environment of an aircraft cabin. Scientific Reports, 2022, 12(1): 10938.
9.	Piacrd R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Ma chine Intelligence, 2001, 23(10): 1175-1191.
10.	Zhang J, Xu Z, Zhou Y, et al. An empirical comparative study on the two methods of eliciting singers’ emotions in singing: self-imagination and VR training. Frontiers in Neuroscience, 2021, 15: 693468.
11.	Sofian N S, James M, Jason T. A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets. Big Data and Cognitive Computing, 2022, 6(1): 16.
12.	聶瑋, 賈江旭, 王敉敉, 等. 基于腦電實驗的虛擬現實環境全景綠視率對人體愉悅度的影響研究. 景觀設計學(中英文), 2022, 10(2): 36-51.
13.	Namazi H, Babini M H, Kuca K, et al. Information and memory-based analysis for decoding of the human learning between normal and virtual reality (VR) conditions. Fractals, 2021, 29(3): 2150163.
14.	王晨. 不同沉浸度的虛擬自然場景對人情緒和恢復性效益的影響研究. 北京: 北京建筑大學, 2023.
15.	Peng Y, Jin F, Kong W, et al. OGSSL: a semi-supervised classification model coupled with optimal graph learning for EEG emotion recognition. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1288-1297.
16.	Raheel A, Majid M, Anwar S M, et al. Emotion classification in response to tactile enhanced multimedia using frequency domain features of brain signals//2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin: IEEE, 2019: 1201-1204.
17.	Cao X, Zhao K, Xu D. Emotion recognition of single-electrode EEG based on multi-feature combination in time-frequency domain. Journal of Physics: Conference Series, 2021, 1827: 012031.
18.	毛玲, 魏士松, 張靈維, 等. 面向飛行模擬器的立體視覺刺激腦機接口系統. 中南大學學報(自然科學版), 2022, 53(8): 2946-2954.
19.	莫云. 基于CSP變換和濾波器組的對數帶通功率特征提取方法. 電子測量技術, 2021, 44(10): 33-38.
20.	Hou H R, Zhang X N, Meng Q H. Odor-induced emotion recognition based on average frequency band division of EEG signals. Journal of Neuroscience Methods, 2020, 334: 108599.
21.	Suhaimi N S, Teo J, Mountstephens J. Emotional state classification in virtual reality using wearable electroencephalography. IOP Conference Series: Materials Science and Engineering, 2018, 341: 012016.
22.	王發旺, 陳睿, 伏云發. 基于DBN和RF的跨被試情緒識別研究. 南京大學學報(自然科學), 2021, 57(4): 617-626.
23.	?mer T, Sira? M ?. The convolutional neural network approach from electroencephalogram signals in emotional detection. Concurrency and Computation: Practice and Experience, 2021, 33(20): e6356.
24.	Algarni M, Saeed F, Al-Hadhrami T, et al. Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors, 2022, 22(8): 2976.
25.	Kalpana M C, J. A, Jude D H. Emotion recognition from EEG signals using recurrent neural networks. Electronics, 2022, 11(15): 2387.
26.	趙丹丹, 趙倩, 董宜先, 等. 基于EEG和DE-CNN-GRU的情緒識別. 計算機系統應用, 2023, 32(4): 206-213.
27.	Tan W, Xu Y, Liu P, et al. A method of VR-EEG scene cognitive rehabilitation training. Health Information Science and Systems, 2021, 9(1): 4.
28.	Liu C, Zhang Y L, Sun L M, et al. The effect of classroom wall color on learning performance: a virtual reality experiment. Building Simulation, 2022, 15(12): 2019-2030.
29.	楊曉哲, 任友群. 虛擬現實與腦電聯動系統的開發及其教育研究功能探索. 遠程教育雜志, 2019, 37(1): 45-52.
30.	Yokota Y, Naruse Y. Temporal fluctuation of mood in gaming task modulates feedback negativity: EEG study with virtual reality. Frontiers in Human Neuroscience, 2021, 15: 536288.
31.	陸鈺鑫. VR游戲類型、交互方式以及網絡時延對玩家認知情緒體驗和眩暈感的影響研究. 天津: 天津師范大學, 2021.
32.	Armin M, Cruz-Garza J G, Saleh K. Enhancing lighting design through the investigation of illuminance and correlated color temperature's effects on brain activity: an EEG-VR approach. Journal of Building Engineering, 2023, 75: 106776.
33.	Schnack A, Wright J M, Holdershaw L J. Does the locomotion technique matter in an immersive virtual store environment?-Comparing motion-tracked walking and instant teleportation. Journal of Retailing and Consumer Services, 2021, 58: 102266.
34.	Byoungho J, Gwia K, Marguerite M, et al. Consumer store experience through virtual reality: its effect on emotional states and perceived store attractiveness. Fashion and Textiles, 2021, 8(1): 1-21.
35.	Li J P, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Transactions on Cybernetics, 2019, 50(7): 3281-3293.
36.	郭苗苗, 陳昕彤, 王磊, 等. 子域自適應網絡跨被試情緒識別算法. 信號處理, 2022, 38(10): 2211-2220.
37.	楊文陽, 張文瑄. 基于腦電信號的注意力水平評價研究進展. 生物醫學工程學雜志, 2023, 40(4): 820-828.
38.	Cai J, Xiao R, Cui W, et al. Application of electroencephalography-based machine learning in emotion recognition: a review. Frontiers in Systems Neuroscience, 2021, 15: 729707.

1. Jafari M, Shoeibi A, Khodatars M, et al. Emotion recognition in EEG signals using deep learning methods: a review. Computers in Biology and Medicine, 2023, 165: 107450.
2. Valentina M, Francesca B, Chiara S, et al. How do emotions elicited in virtual reality affect our memory? a systematic review. Computers in Human Behavior, 2023, 146: 107812.
3. 楊俊峰. 虛擬現實產業邁入快車道. 人民日報海外版, 2022. [2022-11-23(8)]. DOI: 10.28656/n.cnki.nrmrh.2022.004035.
4. Hofmann S M, Klotzsche F, Mariola A, et al. Decoding subjective emotional arousal from EEG during an immersive virtual reality experience. Elife, 2021, 10: e64812.
5. Ekman, P. Basic emotions. Handbook of Cognition and Emotion, 1999, 98(16): 45-60.
6. Xie J, Lan P, Wang S, et al. Brain activation differences of six basic emotions between 2D screen and virtual reality modalities. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022. DOI: 10.1109/TNSRE.2022.3229389.
7. Russell J A. A circumplex model of affect. Journal of Personality and Social Psychology, 1980, 39(6): 1161-1178.
8. Ricci G, De Crescenzio F, Santhosh S, et al. Relationship between electroencephalographic data and comfort perception captured in a virtual reality design environment of an aircraft cabin. Scientific Reports, 2022, 12(1): 10938.
9. Piacrd R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Ma chine Intelligence, 2001, 23(10): 1175-1191.
10. Zhang J, Xu Z, Zhou Y, et al. An empirical comparative study on the two methods of eliciting singers’ emotions in singing: self-imagination and VR training. Frontiers in Neuroscience, 2021, 15: 693468.
11. Sofian N S, James M, Jason T. A dataset for emotion recognition using virtual reality and EEG (DER-VREEG): emotional state classification using low-cost wearable VR-EEG headsets. Big Data and Cognitive Computing, 2022, 6(1): 16.
12. 聶瑋, 賈江旭, 王敉敉, 等. 基于腦電實驗的虛擬現實環境全景綠視率對人體愉悅度的影響研究. 景觀設計學(中英文), 2022, 10(2): 36-51.
13. Namazi H, Babini M H, Kuca K, et al. Information and memory-based analysis for decoding of the human learning between normal and virtual reality (VR) conditions. Fractals, 2021, 29(3): 2150163.
14. 王晨. 不同沉浸度的虛擬自然場景對人情緒和恢復性效益的影響研究. 北京: 北京建筑大學, 2023.
15. Peng Y, Jin F, Kong W, et al. OGSSL: a semi-supervised classification model coupled with optimal graph learning for EEG emotion recognition. IEEE Trans Neural Syst Rehabil Eng, 2022, 30: 1288-1297.
16. Raheel A, Majid M, Anwar S M, et al. Emotion classification in response to tactile enhanced multimedia using frequency domain features of brain signals//2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin: IEEE, 2019: 1201-1204.
17. Cao X, Zhao K, Xu D. Emotion recognition of single-electrode EEG based on multi-feature combination in time-frequency domain. Journal of Physics: Conference Series, 2021, 1827: 012031.
18. 毛玲, 魏士松, 張靈維, 等. 面向飛行模擬器的立體視覺刺激腦機接口系統. 中南大學學報(自然科學版), 2022, 53(8): 2946-2954.
19. 莫云. 基于CSP變換和濾波器組的對數帶通功率特征提取方法. 電子測量技術, 2021, 44(10): 33-38.
20. Hou H R, Zhang X N, Meng Q H. Odor-induced emotion recognition based on average frequency band division of EEG signals. Journal of Neuroscience Methods, 2020, 334: 108599.
21. Suhaimi N S, Teo J, Mountstephens J. Emotional state classification in virtual reality using wearable electroencephalography. IOP Conference Series: Materials Science and Engineering, 2018, 341: 012016.
22. 王發旺, 陳睿, 伏云發. 基于DBN和RF的跨被試情緒識別研究. 南京大學學報(自然科學), 2021, 57(4): 617-626.
23. ?mer T, Sira? M ?. The convolutional neural network approach from electroencephalogram signals in emotional detection. Concurrency and Computation: Practice and Experience, 2021, 33(20): e6356.
24. Algarni M, Saeed F, Al-Hadhrami T, et al. Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM). Sensors, 2022, 22(8): 2976.
25. Kalpana M C, J. A, Jude D H. Emotion recognition from EEG signals using recurrent neural networks. Electronics, 2022, 11(15): 2387.
26. 趙丹丹, 趙倩, 董宜先, 等. 基于EEG和DE-CNN-GRU的情緒識別. 計算機系統應用, 2023, 32(4): 206-213.
27. Tan W, Xu Y, Liu P, et al. A method of VR-EEG scene cognitive rehabilitation training. Health Information Science and Systems, 2021, 9(1): 4.
28. Liu C, Zhang Y L, Sun L M, et al. The effect of classroom wall color on learning performance: a virtual reality experiment. Building Simulation, 2022, 15(12): 2019-2030.
29. 楊曉哲, 任友群. 虛擬現實與腦電聯動系統的開發及其教育研究功能探索. 遠程教育雜志, 2019, 37(1): 45-52.
30. Yokota Y, Naruse Y. Temporal fluctuation of mood in gaming task modulates feedback negativity: EEG study with virtual reality. Frontiers in Human Neuroscience, 2021, 15: 536288.
31. 陸鈺鑫. VR游戲類型、交互方式以及網絡時延對玩家認知情緒體驗和眩暈感的影響研究. 天津: 天津師范大學, 2021.
32. Armin M, Cruz-Garza J G, Saleh K. Enhancing lighting design through the investigation of illuminance and correlated color temperature's effects on brain activity: an EEG-VR approach. Journal of Building Engineering, 2023, 75: 106776.
33. Schnack A, Wright J M, Holdershaw L J. Does the locomotion technique matter in an immersive virtual store environment?-Comparing motion-tracked walking and instant teleportation. Journal of Retailing and Consumer Services, 2021, 58: 102266.
34. Byoungho J, Gwia K, Marguerite M, et al. Consumer store experience through virtual reality: its effect on emotional states and perceived store attractiveness. Fashion and Textiles, 2021, 8(1): 1-21.
35. Li J P, Qiu S, Shen Y Y, et al. Multisource transfer learning for cross-subject EEG emotion recognition. IEEE Transactions on Cybernetics, 2019, 50(7): 3281-3293.
36. 郭苗苗, 陳昕彤, 王磊, 等. 子域自適應網絡跨被試情緒識別算法. 信號處理, 2022, 38(10): 2211-2220.
37. 楊文陽, 張文瑄. 基于腦電信號的注意力水平評價研究進展. 生物醫學工程學雜志, 2023, 40(4): 820-828.
38. Cai J, Xiao R, Cui W, et al. Application of electroencephalography-based machine learning in emotion recognition: a review. Frontiers in Systems Neuroscience, 2021, 15: 729707.

Previous Article
Progress in the analysis of hemolysis and coagulation models for interventional micro-axial flow blood pumps
Next Article
A review on electroencephalogram based channel selection

Journal of Biomedical Engineering

Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content