SleepyWheels: An ensemble model for drowsiness detection leading to accident prevention
Abstract
Approximately 30% of traffic accidents in the world are attributed to driver drowsiness. Various approaches have been suggested by different research teams to detect drowsiness, but their methods have drawbacks. Some involve invasive techniques that cause driver discomfort, some involve too many false positives that cause distractions while driving while others rely on complex models that are too resource-intensive. In this paper, we present SleepyWheels, a novel drowsiness detection system. Our key insight is the combined use of two lightweight neural networks: a binary classifier and a facial landmark detector. This innovative approach minimizes false positives and proves resilient across diverse testing scenarios, such as different camera positions, variations in skin tone and when there is obscurement of facial features by objects like eyewear. Research outcomes include a working prototype of the system, a custom dataset for training the classifier and a trained model, which attains an impressive 97% accuracy rate. Deployment and testing were performed on Windows 10 but the system can be deployed on edge devices like Raspberry Pi. The lightweight nature of the models unlocks possibilities of deployment on mobile and embedded devices for use in vehicles.
Keywords
Full Text:
PDFReferences
1. Saleem S. Risk assessment of road traffic accidents related to sleepiness during driving: a systematic review. Eastern Mediterranean Health Journal. 2022, 28(9): 695–700. doi: 10.26719/emhj.22.055
2. Drowsy Driving: | NHTSA. Available online: https://www.nhtsa.gov/book/countermeasures/countermeasures-work/drowsy-driving (accessed on 7 September 2023).
3. Sleep testing can reduce sleep disorder related road accidents in India | HT Auto. Available online: https://auto.hindustantimes.com/auto/news/sleep-testing-can-reduce-sleep-disorder-related-road-accidents-in-india-41620010214153.html (accessed on 7 September 2023).
4. SleepHealthFoundation | Drowsy Driving. Available online: https://www.sleephealthfoundation.org.au/drowsy-driving.html (accessed on 7 September 2023).
5. PIARC Road Safety Knowledge 2023 Vulnerable Road Users Factsheets. Available online: https://www.piarc.org/ressources/documents/source/Road-Safety-Knowledge-2023/eef5dda-41087-PIARC-Road-Safety-Knowledge-2023-Vulnerable-Road-Users-Factsheets.pdf (accessed on 7 September 2023).
6. Sewalkar P, Seitz J. Vehicle-to-Pedestrian Communication for Vulnerable Road Users: Survey, Design Considerations, and Challenges. Sensors. 2019, 19(2): 358. doi: 10.3390/s19020358
7. Sun Coast Traffic School—VRU Handout. Available online: https://www.suncoasttrafficschool.com/wp-content/uploads/2015/11/VRU-Handout_%C2%A9.pdf (accessed on 7 September 2023).
8. Distracted Driving | Transportation Safety | Injury Center | CDC. Available online: https://www.cdc.gov/transportationsafety/Distracted_Driving/index.html (accessed on 7 September 2023).
9. Harith SH, Mahmud N, Doulatabadi M. Environmental factor and road accident: A review paper. In: Proceedings of the International Conference on Industrial Engineering and Operations Management, 2019(MAR), 3409–3418.
10. Hammad HM, Ashraf M, Abbas F, et al. Environmental factors affecting the frequency of road traffic accidents: a case study of sub-urban area of Pakistan. Environmental Science and Pollution Research. 2019, 26(12): 11674–11685. doi: 10.1007/s11356-019-04752-8
11. Topolšek D, Babić D, Fiolić M. The effect of road safety education on the relationship between Driver’s errors, violations and accidents: Slovenian case study. European Transport Research Review. 2019, 11(1). doi: 10.1186/s12544-019-0351-y
12. Wicaksana A, Rachman T. On the redesign of accident liability for the world of autonomous vehicles. Angewandte Chemie International Edition, 2018, 6(11), 951–952.
13. Drowsy Driving—Facts, Causes and Effects. Available online: https://www.medindia.net/patientinfo/drowsy-driving.htm (accessed on 8 August 2022).
14. Wang J, Chai W, Venkatachalapathy A, et al. A Survey on Driver Behavior Analysis from In-Vehicle Cameras. IEEE Transactions on Intelligent Transportation Systems. 2022, 23(8): 10186–10209. doi: 10.1109/tits.2021.3126231
15. Chui KT, Tsang KF, Chi HR, et al. An Accurate ECG-Based Transportation Safety Drowsiness Detection Scheme. IEEE Transactions on Industrial Informatics. 2016, 12(4): 1438–1452. doi: 10.1109/tii.2016.2573259
16. Chai M, Li S, Sun W, et al. Drowsiness monitoring based on steering wheel status. Transportation Research Part D: Transport and Environment. 2019, 66: 95–103. doi: 10.1016/j.trd.2018.07.007
17. Altameem A, Kumar A, Poonia RC, et al. Early Identification and Detection of Driver Drowsiness by Hybrid Machine Learning. IEEE Access. 2021, 9: 162805-162819. doi: 10.1109/access.2021.3131601
18. Paulo JR, Pires G, Nunes UJ. Cross-Subject Zero Calibration Driver’s Drowsiness Detection: Exploring Spatiotemporal Image Encoding of EEG Signals for Convolutional Neural Network Classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021, 29: 905–915. doi: 10.1109/tnsre.2021.3079505
19. Reddy TK, Arora V, Gupta V, et al. EEG-Based Drowsiness Detection with Fuzzy Independent Phase-Locking Value Representations Using Lagrangian-Based Deep Neural Networks. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 2022, 52(1): 101–111. doi: 10.1109/tsmc.2021.3113823
20. Schmidt J, Laarousi R, Stolzmann W, et al. Eye blink detection for different driver states in conditionally automated driving and manual driving using EOG and a driver camera. Behavior Research Methods. 2017, 50(3): 1088–1101. doi: 10.3758/s13428-017-0928-0
21. Jiao Y, Deng Y, Luo Y, et al. Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks. Neurocomputing. 2020, 408: 100–111. doi: 10.1016/j.neucom.2019.05.108
22. Satti AT, Kim J, Yi E, et al. Microneedle Array Electrode-Based Wearable EMG System for Detection of Driver Drowsiness through Steering Wheel Grip. Sensors. 2021, 21(15): 5091. doi: 10.3390/s21155091
23. Deng W, Wu R. Real-Time Driver-Drowsiness Detection System Using Facial Features. IEEE Access. 2019, 7: 118727–118738. doi: 10.1109/access.2019.2936663
24. Verma B, Choudhary A. Deep Learning Based Real-Time Driver Emotion Monitoring. 2018 IEEE International Conference on Vehicular Electronics and Safety (ICVES); September 2018. doi: 10.1109/icves.2018.8519595
25. Xing Y, Lv C, Zhang Z, et al. Identification and Analysis of Driver Postures for In-Vehicle Driving Activities and Secondary Tasks Recognition. IEEE Transactions on Computational Social Systems. 2018, 5(1): 95–108. doi: 10.1109/tcss.2017.2766884
26. Abtahi S, Hariri B, Shirmohammadi S. Driver drowsiness monitoring based on yawning detection. 2011 IEEE International Instrumentation and Measurement Technology Conference; May 2011. doi: 10.1109/imtc.2011.5944101
27. Albadawi Y, Takruri M, Awad M. A Review of Recent Developments in Driver Drowsiness Detection Systems. Sensors. 2022, 22(5): 2069. doi: 10.3390/s22052069
28. Arakawa T. Trends and Future Prospects of the Drowsiness Detection and Estimation Technology. Sensors. 2021, 21(23): 7921. doi: 10.3390/s21237921
29. Krajewski J, Sommer D, Trutschel U, et al. Steering Wheel Behavior Based Estimation of Fatigue. Proceedings of the 5th International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design: Driving Assessment 2009. 2009. doi: 10.17077/drivingassessment.1311
30. Ma J, Murphey YL, Zhao H. Real Time Drowsiness Detection Based on Lateral Distance Using Wavelet Transform and Neural Network. 2015 IEEE Symposium Series on Computational Intelligence; December 2015. doi: 10.1109/ssci.2015.68
31. Arefnezhad S, Samiee S, Eichberger A, et al. Driver Drowsiness Detection Based on Steering Wheel Data Applying Adaptive Neuro-Fuzzy Feature Selection. Sensors. 2019, 19(4): 943. doi: 10.3390/s19040943
32. Jeon Y, Kim B, Baek Y. Ensemble CNN to Detect Drowsy Driving with In-Vehicle Sensor Data. Sensors. 2021, 21(7): 2372. doi: 10.3390/s21072372
33. Li G, Lee BL, Chung WY. Smartwatch-Based Wearable EEG System for Driver Drowsiness Detection. IEEE Sensors Journal. 2015, 15(12): 7169–7180. doi: 10.1109/jsen.2015.2473679
34. Taran S, Bajaj V. Drowsiness Detection Using Adaptive Hermite Decomposition and Extreme Learning Machine for Electroencephalogram Signals. IEEE Sensors Journal. 2018, 18(21): 8855–8862. doi: 10.1109/jsen.2018.2869775
35. Koh S, Cho BR, Lee J il, et al. Driver drowsiness detection via PPG biosignals by using multimodal head support. 2017 4th International Conference on Control, Decision and Information Technologies (CoDIT); April 2017. doi: 10.1109/codit.2017.8102622
36. Danisman T, Bilasco IM, Djeraba C, et al. Drowsy driver detection system using eye blink patterns. 2010 International Conference on Machine and Web Intelligence; October 2010. doi: 10.1109/icmwi.2010.5648121
37. You F, Gong Y, Tu H, et al. A Fatigue Driving Detection Algorithm Based on Facial Motion Information Entropy. Journal of Advanced Transportation. 2020, 2020: 1–17. doi: 10.1155/2020/8851485
38. Knapik M, Cyganek B. Driver’s fatigue recognition based on yawn detection in thermal images. Neurocomputing. 2019, 338: 274–292. doi: 10.1016/j.neucom.2019.02.014
39. Dua M, Shakshi, Singla R, et al. Deep CNN models-based ensemble approach to driver drowsiness detection. Neural Computing and Applications. 2020, 33(8): 3155–3168. doi: 10.1007/s00521-020-05209-7
40. Dwivedi K, Biswaranjan K, Sethi A. Drowsy driver detection using representation learning. 2014 IEEE International Advance Computing Conference (IACC); February 2014. doi: 10.1109/iadcc.2014.6779459
41. Park S, Pan F, Kang S, Yoo CD. Driver drowsiness detection system based on feature representation learning using various deep networks. In: Proceedings of Computer vision—ACCV 2016 workshops; 20–24 November 2016; Taipei, Taiwan.
42. Iandola N, Han S, Moskewicz MW, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. Feb. 2016, doi: 10.48550/arxiv.1602.07360.
43. Parkhi OM, Vedaldi A, Zisserman A. Deep Face Recognition. Procedings of the British Machine Vision Conference 2015. 2015. doi: 10.5244/c.29.41
44. Donahue J, Hendricks LA, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); June 2015. doi: 10.1109/cvpr.2015.7298878
45. Saito Y, Itoh M, Inagaki T. Driver Assistance System with a Dual Control Scheme: Effectiveness of Identifying Driver Drowsiness and Preventing Lane Departure Accidents. IEEE Transactions on Human-Machine Systems. 2016, 46(5): 660–671. doi: 10.1109/thms.2016.2549032
46. Mehreen A, Anwar SM, Haseeb M, et al. A Hybrid Scheme for Drowsiness Detection Using Wearable Sensors. IEEE Sensors Journal. 2019, 19(13): 5119–5126. doi: 10.1109/jsen.2019.2904222
47. Lee Boon Leng, Lee Boon Giin, Wan-Young Chung. Wearable driver drowsiness detection system based on biomedical and motion sensors. 2015 IEEE SENSORS; November 2015. doi: 10.1109/icsens.2015.7370355
48. Rajkumarsingh B, Totah D. Drowsiness Detection using Android Application and Mobile Vision Face API. R&D Journal. 2021, 37. doi: 10.17159/2309-8988/2021/v37a4
49. Zhang C, Wu X, Zheng X, et al. Driver Drowsiness Detection Using Multi-Channel Second Order Blind Identifications. IEEE Access. 2019, 7: 11829–11843. doi: 10.1109/access.2019.2891971
50. Dasgupta A, Rahman D, Routray A. A Smartphone-Based Drowsiness Detection and Warning System for Automotive Drivers. IEEE Transactions on Intelligent Transportation Systems. 2019, 20(11): 4045–4054. doi: 10.1109/tits.2018.2879609
51. Nguyen T, Ahn S, Jang H, et al. Utilization of a combined EEG/NIRS system to predict driver drowsiness. Scientific Reports. 2017, 7(1). doi: 10.1038/srep43933
52. Ji Q, Zhu Z, Lan P. Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue. IEEE Transactions on Vehicular Technology. 2004, 53(4): 1052–1068. doi: 10.1109/tvt.2004.830974
53. Liang Y, Reyes ML, Lee JD. Real-Time Detection of Driver Cognitive Distraction Using Support Vector Machines. IEEE Transactions on Intelligent Transportation Systems. 2007, 8(2): 340–350. doi: 10.1109/tits.2007.895298
54. Gwak J, Hirao A, Shino M. An Investigation of Early Detection of Driver Drowsiness Using Ensemble Machine Learning Based on Hybrid Sensing. Applied Sciences. 2020, 10(8): 2890. doi: 10.3390/app10082890
55. Guede-Fernandez F, Fernandez-Chimeno M, Ramos-Castro J, et al. Driver Drowsiness Detection Based on Respiratory Signal Analysis. IEEE Access. 2019, 7: 81826–81838. doi: 10.1109/access.2019.2924481
56. Bazarevsky V, Kartynnik Y, Vakunov A, et al. BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs. Jul. 2019, doi: 10.48550/arxiv.1907.05047.
57. MediaPipe. Available online: https://mediapipe.dev/ (accessed on 8 August 2022).
58. Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the International conference on machine learning, 2019. pp. 6105–6114.
59. Tan M, Le Q. Efficientnetv2: Smaller models and faster training. In: Proceedings of the International Conference on Machine Learning; 2021. pp. 10096–10106.
60. Sandler M, Howard A, Zhu M, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition; June 2018. doi: 10.1109/cvpr.2018.00474
61. Kartynnik Y, Ablavatski A, Grishchenko I, Grundmann M. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. Jul. 2019, doi: 10.48550/arxiv.1907.06724.
62. Soukupova T, Cech J. Real-Time Eye Blink Detection using Facial Landmarks. Center for Machine Perception, Department of Cybernetics Faculty of Electrical Engineering, Czech Technical University in Prague; 2016. pp. 1–8.
63. Gaur A, Kinage A, Rekhawar N, et al. Cursor Control Using Face Gestures. Proceedings of the 11th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2019). Published online August 1, 2020: 31–40. doi: 10.1007/978-3-030-49345-5_4
64. Dewi C, Chen RC, Jiang X, et al. Adjusting eye aspect ratio for strong eye blink detection based on facial landmarks. PeerJ Computer Science. 2022, 8: e943. doi: 10.7717/peerj-cs.943
65. George A, Routray A. Design and implementation of real-time algorithms for eye tracking and perclos measurement for on board estimation of alertness of drivers. Available online: http://arxiv.org/abs/1505.06162 (accessed on 7 September 2023).
66. SleepyWheels dataset. Available online: https://drive.google.com/drive/folders/1bhrgY8RcUFuD675oxcSLJkmtxY3Wxfg9 (accessed on 8 August 2022).
DOI: https://doi.org/10.32629/jai.v7i3.1117
Refbacks
- There are currently no refbacks.
Copyright (c) 2024 Jomin Jose, Andrew J, Kumudha Raimond, Shweta Vincent, Jennifer Eunice R
License URL: https://creativecommons.org/licenses/by-nc/4.0/