Determination of stress obtained from sensor-based wearable measurements using VGG16 deep learning model
Abstract
There are many researches carried out for different purposes on human-computer interactions. One of them is related to stress-related activity detection. Today, in people with disorders whose activity is not understood or misunderstood, the correct detection of the relevant movement can be vital in some cases. It may be more advantageous to use physiological signals in the body in determining the type of activity. Due to these important situations, two different research applications were carried out within the scope of this study in order to automatically detect four different types of stress, namely Neutral, Emotional, Mental, and Physical. For both applications, the data was first converted to images as a preprocessing. In the first stage of the research, the images of the standard dataset were presented to the VGG16 deep learning model. As a result, the highest accuracy rate was obtained as 67% for class 1 Neutral activation. In the second part of the study, an application was performed using the Isolation Forest Algorithm on the existing image data to remove outliers. The new dataset obtained were presented to the same model and detailed analyses were made. Accordingly, the maximum accuracy value was 97% in Physical activity. In the same application, the average rate for all activities was 82.5%. Briefly, the research contributes to the literature by demonstrating the significant impact of outliers on system performance through image transformations of existing time series physiological signals.
Keywords
Full Text:
PDFReferences
1. Oxford University Press. Available online: https://www.oed.com/ (accessed on 25 July 2023).
2. Chen L, Nugent CD. Sensor-Based Activity Recognition Review. Human Activity Recognition and Behaviour Analysis. Published online 2019: 23-47. doi: 10.1007/978-3-030-19408-6_2
3. Kim E, Helal S, Cook D. Human Activity Recognition and Pattern Discovery. IEEE Pervasive Computing. 2010, 9(1): 48-53. doi: 10.1109/mprv.2010.7
4. Bao L, Intille SS. Activity Recognition from User-Annotated Acceleration Data. Pervasive Computing. Published online 2004: 1-17. doi: 10.1007/978-3-540-24646-6_1
5. Handley TE, Lewin TJ, Perkins D, et al. Self‐recognition of mental health problems in a rural Australian sample. Australian Journal of Rural Health. 2018, 26(3): 173-180. doi: 10.1111/ajr.12406
6. del R Millan J, Mourino J, Franze M, et al. A local neural classifier for the recognition of EEG patterns associated to mental tasks. IEEE Transactions on Neural Networks. 2002, 13(3): 678-686. doi: 10.1109/tnn.2002.1000132
7. Horlings R, Datcu D, Rothkrantz LJM. Emotion recognition using brain activity. Proceedings of the 9th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing -CompSysTech’08. Published online 2008. doi: 10.1145/1500879.1500888
8. Agrafioti F, Hatzinakos D, Anderson AK. ECG Pattern Analysis for Emotion Detection. IEEE Transactions on Affective Computing. 2012, 3(1): 102-115. doi: 10.1109/t-affc.2011.28
9. Chen Y, Yang Z, Wang J. Eyebrow emotional expression recognition using surface EMG signals. Neurocomputing. 2015, 168: 871-879. doi: 10.1016/j.neucom.2015.05.037
10. Tapia EM, Intille SS, Larson K. Activity Recognition in the Home Using Simple and Ubiquitous Sensors. Pervasive Computing. Published online 2004: 158-175. doi: 10.1007/978-3-540-24646-6_10
11. Mshali H, Lemlouma T, Moloney M, et al. A survey on health monitoring systems for health smart homes. International Journal of Industrial Ergonomics. 2018, 66: 26-56. doi: 10.1016/j.ergon.2018.02.002
12. Lymberis A, Olsson S. Intelligent Biomedical Clothing for Personal Health and Disease Management: State of the Art and Future Vision. Telemedicine Journal and e-Health. 2003, 9(4): 379-386. doi: 10.1089/153056203772744716
13. Wei D, Nagai Y, Jing L, et al. Designing comfortable smart clothing: for infants’ health monitoring. International Journal of Design Creativity and Innovation. 2018, 7(1-2): 116-128. doi: 10.1080/21650349.2018.1428690
14. Jerritta S, Murugappan M, Nagarajan R, et al. Physiological signals based human emotion Recognition: a review. 2011 IEEE 7th International Colloquium on Signal Processing and its Applications. Published online March 2011. doi: 10.1109/cspa.2011.5759912
15. Mohino-Herranz I, Gil-Pita R, Rosa-Zurera M, et al. Activity Recognition Using Wearable Physiological Measurements: Selection of Features from a Comprehensive Literature Study. Sensors. 2019, 19(24): 5524. doi: 10.3390/s19245524
16. Mohino-Herranz I, Gil-Pita R, Ferreira J, et al. Assessment of Mental, Emotional and Physical Stress through Analysis of Physiological Signals Using Smartphones. Sensors. 2015, 15(10): 25607-25627. doi: 10.3390/s151025607
17. Rattanyu K, Mizukawa M. Emotion Recognition Based on ECG Signals for Service Robots in the Intelligent Space During Daily Life. Journal of Advanced Computational Intelligence and Intelligent Informatics. 2011, 15(5): 582-591. doi: 10.20965/jaciii.2011.p0582
18. Lara OD, Labrador MA. A Survey on Human Activity Recognition using Wearable Sensors. IEEE Communications Surveys & Tutorials. 2013, 15(3): 1192-1209. doi: 10.1109/surv.2012.110112.00192
19. McCraty R, Atkinson M, Tiller WA, et al. The effects of emotions on short-term power spectrum analysis of heart rate variability. The American Journal of Cardiology. 1995, 76(14): 1089-1093. doi: 10.1016/s0002-9149(99)80309-9
20. Jemioło P, Storman D, Mamica M, et al. Datasets for Automated Affect and Emotion Recognition from Cardiovascular Signals Using Artificial Intelligence— A Systematic Review. Sensors. 2022, 22(7): 2538. doi: 10.3390/s22072538
21. Aristizabal S, Byun K, Wood N, et al. The Feasibility of Wearable and Self-Report Stress Detection Measures in a Semi-Controlled Lab Environment. IEEE Access. 2021, 9: 102053-102068. doi: 10.1109/access.2021.3097038
22. Chen S, Jiang K, Hu H, et al. Emotion Recognition Based on Skin Potential Signals with a Portable Wireless Device. Sensors. 2021, 21(3): 1018. doi: 10.3390/s21031018
23. Kyamakya K, Al-Machot F, Haj Mosa A, et al. Emotion and Stress Recognition Related Sensors and Machine Learning Technologies. Sensors. 2021, 21(7): 2273. doi: 10.3390/s21072273
24. Yusup NB. Hybrid Feature Selection Technique for Classification of Human Activity Recognition. Universiti Teknologi Malaysia; 2021.
25. Iqbal T, Redon-Lurbe P, Simpkin AJ, et al. A Sensitivity Analysis of Biophysiological Responses of Stress for Wearable Sensors in Connected Health. IEEE Access. 2021, 9: 93567-93579. doi: 10.1109/access.2021.3082423
26. Lu K, Yang L, Seoane F, et al. Fusion of Heart Rate, Respiration and Motion Measurements from a Wearable Sensor System to Enhance Energy Expenditure Estimation. Sensors. 2018, 18(9): 3092. doi: 10.3390/s18093092
27. Lotfan S, Shahyad S, Khosrowabadi R, et al. Support vector machine classification of brain states exposed to social stress test using EEG-based brain network measures. Biocybernetics and Biomedical Engineering. 2019, 39(1): 199-213. doi: 10.1016/j.bbe.2018.10.008
28. Zhang B, Morère Y, Sieler L, et al. Reaction time and physiological signals for stress recognition. Biomedical Signal Processing and Control. 2017, 38: 100-107. doi: 10.1016/j.bspc.2017.05.003
29. Berbano AEU, Pengson HNV, Razon CGV, et al. Classification of stress into emotional, mental, physical and no stress using electroencephalogram signal analysis. 2017 IEEE International Conference on Signal and Image Processing Applications (ICSIPA). Published online September 2017. doi: 10.1109/icsipa.2017.8120571
30. Lee J, Yoo SK. Design of User-Customized Negative Emotion Classifier Based on Feature Selection Using Physiological Signal Sensors. Sensors. 2018, 18(12): 4253. doi: 10.3390/s18124253
31. Mohino-Herranz I, Gil-Pita R, García-Gómez J, et al. A Wrapper Feature Selection Algorithm: An Emotional Assessment Using Physiological Recordings from Wearable Sensors. Sensors. 2020, 20(1): 309. doi: 10.3390/s20010309
32. Saini SK, Gupta R. Mental Stress Assessment using Wavelet Transform Features of Electrocardiogram Signals. 2021 International Conference on Industrial Electronics Research and Applications (ICIERA). Published online December 22, 2021. doi: 10.1109/iciera53202.2021.9726532
33. Kim N, Kim J, Park J, et al. Effects of Measurement Positions and Emotional Changes on ECG Indices: A Preliminary Study. ICIC express letters Part B. Applications: an international journal of research surveys 2020, 11: 297-303.
34. Activity recognition using wearable physiological measurements. UCI Machine Learning Repository. https://doi.org/10.24432/C5RK6V
35. Yang H, Ni J, Gao J, et al. A novel method for peanut variety identification and classification by Improved VGG16. Scientific Reports. 2021, 11(1). doi: 10.1038/s41598-021-95240-y
36. Zhang RQea. Convolutional neural network for peanut pod grade image recognition based on transfer learning. Trans Chin Soc Agric Eng. 2020, 36(23): 171-180.
37. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
38. Gopalakrishnan K, Khaitan SK, Choudhary A, et al. Deep Convolutional Neural Networks with transfer learning for computer vision-based data-driven pavement distress detection. Construction and Building Materials. 2017, 157: 322-330. doi: 10.1016/j.conbuildmat.2017.09.110
39. Theckedath D, Sedamkar RR. Detecting Affect States Using VGG16, ResNet50 and SE-ResNet50 Networks. SN Computer Science. 2020, 1(2). doi: 10.1007/s42979-020-0114-9
40. Xu H, Pang G, Wang Y, et al. Deep Isolation Forest for Anomaly Detection. IEEE Transactions on Knowledge and Data Engineering. 2023, 35(12): 12591-12604. doi: 10.1109/tkde.2023.3270293
41. Ding Z, Fei M. An Anomaly Detection Approach Based on Isolation Forest Algorithm for Streaming Data using Sliding Window. IFAC Proceedings Volumes. 2013, 46(20): 12-17. doi: 10.3182/20130902-3-cn-3020.00044
42. Liu FT, Ting KM, Zhou ZH. Isolation Forest. 2008 Eighth IEEE International Conference on Data Mining. Published online December 2008. doi: 10.1109/icdm.2008.17
43. Lesouple J, Baudoin C, Spigai M, et al. Generalized isolation forest for anomaly detection. Pattern Recognition Letters. 2021, 149: 109-119. doi: 10.1016/j.patrec.2021.05.022
44. Wikipedia contributors, “Confusion matrix,” Wikipedia, The Free Encyclopedia. Available online: https://en.wikipedia.org/w/index.php?title=Confusion_matrix&oldid=1171174671 (accessed on 25 July 2023).
45. Vieira SM, Kaymak U, Sousa JMC. Cohen’s kappa coefficient as a performance measure for feature selection. International Conference on Fuzzy Systems. Published online July 2010. doi: 10.1109/fuzzy.2010.5584447
46. Shchur O, Mumme M, Bojchevski A, et al. Pitfalls of graph neural network evaluation. arXiv preprint. 2018. arXiv:05868.
47. Rong Y, Huang W, Xu T, et al. Dropedge: Towards deep graph convolutional networks on node classification. arXiv. 2019. arXiv:10903.
48. Batzner S, Musaelian A, Sun L, et al. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications. 2022, 13(1). doi: 10.1038/s41467-022-29939-5
DOI: https://doi.org/10.32629/jai.v7i4.1222
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Cüneyt Yücelbaş, Şule Yücelbaş
License URL: https://creativecommons.org/licenses/by-nc/4.0/