An extensive study of facial expression recognition using artificial intelligence techniques with different datasets
Abstract
Machine and deep learning (DL) algorithms have advanced to a point where a wide range of crucial real-world computer vision problems can be solved. Facial Expression Recognition (FER) is one of these applications; it is the foremost non-verbal intentions and a fascinating study of symmetry. A prevalent application of deep learning has become the area of vision, where facial expression recognition has emerged as one of the most promising new frontiers. Latterly deep learning-based FER models have been plagued by technical problems, including under-fitting and over-fitting. Probably inadequate information is used for training and expressing ideas. With these considerations in mind, this article gives a systematic and complete survey of the most cutting-edge AI strategies and gives a conclusion to address the aforementioned problems. It is also a scheme of classification for existing facial proposals in compact. This survey analyses the structure of the usual FER method and discusses the feasible technologies that may be used in its respective elements. In addition, this study provides a summary of seventeen widely-used FER datasets that reviews functioning novel machine and DL networks suggested by academics and outline their benefits and liability in the context of facial expression acknowledgment based on static replicas. Finally, this study discusses the research obstacles and open consequences of that well-conditioned face expression recognition scheme.
Keywords
Full Text:
PDFReferences
1. Kumari A, Tanwar S, Tyagi S, Kumar N. Fog computing for healthcare 4.0 environment: Opportunities and challenges. Computers and Electrical Engineering 2018; 72: 1–13. doi: 10.1016/j.compeleceng.2018.08.015
2. Hathaliya J, Sharma P, Tanwar S, Gupta R. Blockchain-based remote patient monitoring in healthcare 4.0. In: Proceedings of 2019 IEEE 9th International Conference on Advanced Computing (IACC); 13–14 December 2019; Tiruchirappalli, India.
3. Vora J, DevMurari P, Tanwar S, et al. Blind signatures based secured e-healthcare system. In: Proceedings of 2018 International Conference on Computer, Information and Telecommunication Systems (CITS); 11–13 July 2018; Alsace, France.
4. Zhang L, Verma B, Tjondronegoro D, Chandran V. Facial expression analysis under partial occlusion: A survey. ACM Computing Surveys 2018; 51(2): 1–49. doi: 10.1145/3158369
5. Maheswari UV, Aluvalu R, Chennam KK. Application of machine learning algorithms for facial expression analysis. Machine Learning for Sustainable Development 2021; 9: 77–96. doi: 10.1515/9783110702514-005
6. Swapna M, Viswanadhula UM, Aluvalu R, et al. Bio-Signals in medical applications and challenges using artificial intelligence. Journal of Sensor and Actuator Networks 2022; 11(1): 17. doi: 10.3390/jsan11010017
7. Spezialetti M, Placidi G, Rossi S. Emotion recognition for humanrobot interaction: Recent advances and future perspectives. Journal of Sensor and Actuator Networks 2020; 7: 145. doi: 10.3389/FROBT.2020.532279
8. Ramis S, Buades JM, Perales FJ. Using a social robot to evaluate facial expressions in the wild. Sensors 2020; 20(23): 1–24. doi: 10.3390/s20236716
9. Bhatti YK, Jamil A, Nida N, et al. Facial expression recognition of instructor using deep features and extreme learning machine. Computational Intelligence and Neuroscience 2021; 2021: 5570870. doi: 10.1155/2021/5570870
10. Li S, Deng W. Deep facial expression recognition: A survey. arXiv 2018; arXiv:1804.08348. doi: 10.1109/TAFFC.2020.2981446
11. Pulmamidi N, Aluvalu R, Maheswari VU. Intelligent travel route suggestion system based on pattern of travel and difficulties. IOP Conference Series: Materials Science and Engineering 2021; 1042: 012010. doi: 10.1088/1757-899X/1042/1/012010
12. Hemalatha G, Sumathi C. A study of techniques for facial detection and expression classification. International Journal of Computer Science and Engineering Survey 2014; 5(2): 27. doi: 10.5121/ijcses.2014.5203
13. Deodhare D. Facial Expressions to Emotions: A Study of Computational Paradigms for Facial Emotion Recognition. Springe; 2015. pp. 173–198.
14. Chengeta K, Viriri S. Facial expression recognition: A survey on local binary and local directional patterns. Lecture Notes in Computer Science 2018; 11055: 513–522. doi: 10.1007/978-3-319-98443-8_47
15. Baskar A, Kumar TG. Facial expression classification using machine learning approach: A review. Data Engineering and Intelligent Computing: Proceedings of IC3T 2016; 2018: 337–345.
16. Sariyanidi E, Gunes H, Cavallaro A. Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 2014; 37(6): 1113–1133. doi: 10.1109/TPAMI.2014.2366127
17. Tian Y-I, Kanade T, Cohn JF. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001; 23: 97–115. doi: 10.1109/34.908962
18. Yan H. Transfer subspace learning for cross-dataset facial expression recognition. Neurocomputing 2016; 208: 165–173. doi: 10.1016/j.neucom.2015.11.113
19. Benini S, Khan K, Leonardi R, et al. Face analysis through semantic face segmentation. Signal Processing: Image Communication 2019; 74: 21–31. doi: 10.1016/j.image.2019.01.005
20. Verma VK, Srivastava S, Jain T, Jain A. Local invariant feature-based gender recognition from facial images. In: Soft Computing for Problem Solving. Springer; 2019. pp. 869–878.
21. Lyons M, Akamatsu S, Kamachi M, Gyoba J. Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition; 14–16 April 1998; Nara, Japan.
22. Lucey P, Cohn JF, Kanade T, et al. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops; 13–18 June 2010; San Francisco, USA.
23. Du S, Tao Y, Martinez AM. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences 2014; 111(15): E1454–E1462. doi: 10.1073/pnas.1322355111
24. Mavadati SM, Mahoor MH, Bartlett K, et al. Disfa: A spontaneous facial action intensity database. IEEE Transactions Affective Computing 2013; 4(2): 151–160. doi: 10.1109/T-AFFC.2013.4
25. Pantic M, Valstar M, Rademaker R, Maat L. Web-based database for facial expression analysis. In: Proceedings of 2005 IEEE International Conference on Multimedia and Expo; 6–6 July 2005; Amsterdam, Netherlands.
26. Yin L, Wei X, Sun Y, et al. A 3D facial expression database for facial behavior research. In: Proceedings of 7th International Conference on Automatic Face and Gesture Recognition (FGR06); 10–12 April 2006, Southampton, UK.
27. Zhang X, Yin L, Cohn JF, et al. BP4D-spontaneous: A high-resolution spontaneous 3D dynamic facial expression database. Image Vision Computing 2014; 32: 692–706. doi: 10.1016/j.imavis.2014.06.002
28. Kaulard K, Cunningham DW, Bülthoff HH, Wallraven C. The MPI facial expression database—A validated database of emotional and conversational facial expressions. PLoS One 2012; 7(3): e32321. doi: 10.1371/journal.pone.0032321
29. Lundqvist D, Flykt A, Öhman A. The Karolinska directed emotional faces (KDEF). CD ROM Dep 1998; 91: 630. doi: 10.1037/t27732-000
30. Wang S, Liu Z, Lv S, et al. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Transactions on Multimedia 2010; 12(7): 682–691. doi: 10.1109/TMM.2010.2060716
31. Gross R, Matthews I, Cohn J, et al. Multi-PIE. Image Vision Computing 2010; 28(5): 807–813. doi: 10.1016/j.imavis.2009.08.002
32. Zhao G, Huang X, Taini M, et al. Facial expression recognition from near-infrared videos. Image Vision Computing 2011; 29(9): 607–619. doi: 10.1016/j.imavis.2011.07.002
33. Carrier PL, Courville A, Goodfellow IJ, et al. FER-2013 Face Database. Universit de Montral; 2013.
34. Valstar MF, Mehu M, Jiang B, et al. Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 2012; 42(4): 966–979. doi: 10.1109/TSMCB.2012.2200675
35. Dhall A, Goecke R, Lucey S, Gedeon T. Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 2012; 19(3): 34–41. doi: 10.1109/MMUL.2012.26
36. Dhall A, Goecke R, Lucey S, Gedeon T. Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: Proceedings of 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops); 6–13 November 2011; Barcelona, Spain.
37. Li S, Deng W. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Transions Image Processing 2019; 28: 356–370. doi: 10.1109/TIP.2018.2868382
38. Li S, Deng W. Blended emotion in-the-wild: Multi-label facial expression recognition using crowdsourced annotations and deep locality feature learning. International Journal of Computer Vision 2018; 127: 884–906. doi: 0.1007/s11263-018-1131-1
39. Whitehill J, Littlewort G, Fasel I, et al. Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 2009; 31(11): 2106–2111. doi: 10.1109/TPAMI.2009.42
40. Lucey P, Cohn JF, Prkachin KM, et al. Painful data: The UNBC-McMaster shoulder pain expression archive database. In: Proceedings of 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG); 21–25 March 2011; Santa Barbara, USA.
41. Viola P, Jones MJ. Robust real-time face detection. International Journal of Computer Vision 2004; 57: 137–154. doi: 10.1023/B:VISI.0000013087.49260.fb
42. Kazemi V, Sullivan J. One millisecond face alignment with an ensemble of regression trees. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition; 23–28 June 2014; Columbus, USA.
43. Zhang K, Zhang Z, Li Z, Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 2016; 23(10): 1499–1503. doi: 10.1109/LSP.2016.2603342
44. Tomasi C, Manduchi R. Bilateral filtering for gray and color images. In: Proceedings of Sixth International Conference on Computer Vision; 7–7 January 1998; Bombay, India.
45. Lindenbaum M, Fischer M, Bruckstein A. On Gabor’s contribution to image enhancement. Pattern Recognition 1994; 27(1): 1–8. doi: 10.1016/0031-3203(94)90013-2
46. Garg P, Jain T. A comparative study on histogram equalization and cumulative histogram equalization. International Journal of New Technology and Research 2017; 3: 41–43.
47. Hawkins DM. The problem of overfitting. Journal of Chemical Information and Computer Sciences 2004; 44: 1–12. doi: 10.1021/ci0342472
48. Jolliffe I. Principal Component Analysis. Springer; 2011.
49. Ojala T, Pietikäinen M, Mäenpää T. Multiresolution Gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002; 24(7): 971–987. doi: 10.1109/TPAMI.2002.1017623
50. Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001; 23(6): 681–685. doi: 10.1109/34.927467
51. Pakstas A, Forchheimer R, Pandzic IS. MPEG-4 Facial Animation: The Standard, Implementation and Applications. John Wiley & Sons; 2002.
52. Ravi R, Yadhukrishna SV, Prithviraj R. A face expression recognition using CNN and LBP. In: Proceedings of 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC); 11–13 March 2020; Erode, India.
53. Divya M, Reddy ROK, Raghavendra C. Effective facial emotion recognition using convolutional neural network algorithm. International Journal of Recent Technology and Engineering (IJRTE) 2019; 8(4): 2277–3878.
54. Shah ANB, Patel N, Dave JA, et al. Role of Artificial Intelligence and Neural Network in the Health-Care Sector: An Important Guide for Health Prominence. CRC Press; 2023. pp. 239–263.
55. Pisupati S, Ismail BM. Image registeration method for satellite image sensing using feature based techniques. International Journal of Advanced Trends in Computer Science and Engineering 2020; 9(1):490–593. doi: 10.30534/ijatcse/2020/82912020
56. Anjum G, Reddy TB, Ismail BM, et al. Variable block size hybrid fractal technique for image compression. In: Proceedings of 2020 6th International Conference on Advanced Computing and Communication Systems; 6–7 March 2020; Coimbatore, India.
57. Xue M, Mian A, Duan X, Liu W. Learning interpretable expression-sensitive features for 3D dynamic facial expression recognition. In: Proceedings of 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019); 14–18 May 2019; Lille, France.
58. Maheswari VU, Prasad GV, Raju SV. Facial expression analysis using local directional stigma mean patterns and convolutional neural networks. International Journal of Knowledge-Based and Intelligent Engineering Systems 2021; 25(1): 119–128. doi: 10.3233/KES-210057
59. Lakshmi KN, Reddy YK, Kireeti M, et al. Design and implementation of student chat bot using AIML and LSA. International Journal of Innovative Technology and Exploring Engineering (IJITEE) 2019; 8(6): 1742–1746.
60. Ismail M, Vardhan VH, Mounika VA, Padmini KS. An effective heart disease prediction method using artificial neural network. International Journal of Innovative Technology and Exploring Engineering 2019; 8(8): 1529–1532.
61. Mahmood MR, Abdulrazzaq MB, Zeebaree S, et al. Classification techniques performance evaluation for facial expression recognition. Indonesian Journal of Electrical Engineering and Computer Science 2021; 21:1176–1184.
62. Abdulrazaq MB, Mahmood MR, Zeebaree SR, et al. An analytical appraisal for supervised classifiers’ performance on facial expression recognition based on relief-f feature selection. Journal of Physics: Conference Series 2021; 1804: 012055. doi: 10.1088/1742-6596/1804/1/012055
63. Dino HI, Abdulrazzaq MB. A comparison of four classification algorithms for facial expression recognition. Polytechnic Journal 2020; 10: 74–80. doi: 10.25156/ptj.v10n1y2020.pp74-80
64. Le TTQ, Tran TK, Rege M. Dynamic image for micro-expression recognition on region-based framework. In: Proceedings of 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI); 11–13 August 2020, Las Vegas, USA.
65. Liu D, Ouyang X, Xu S, et al. SAANet: Siamese action-units attention network for improving dynamic facial expression recognition. Neurocomputing 2020; 413: 145–157. doi: 10.1016/j.neucom.2020.06.062
66. Chen L, Ouyang Y, Zeng Y, Li Y. Dynamic facial expression recognition model based on BiLSTM-Attention. In: Proceedings of 2020 15th International Conference on Computer Science & Education (ICCSE); 18–22 August 2020; Delft, Netherlands.
67. Chen W, Zhang D, Li M, Lee DJ. STCAM: Spatial-temporal and channel attention module for dynamic facial expression recognition. IEEE Transactions on Affective Computing 2020; 14(1): 800–810. doi: 10.1109/TAFFC.2020.3027340
68. Perveen N, Roy D, Chalavadi KM. Facial expression recognition in videos using dynamic kernels. IEEE Transactions on Image Processing 2020; 29: 8316–8325. doi: 10.1109/TIP.2020.3011846
69. Ni H, Liu J. 3D face dynamic expression synthesis system based on DFFD. In: Proceedings of 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC); 15–17 March 2019; Chengdu, China.
70. Alazrai R, Yousef KWA, Daoud MI. Emotion recognition based on decoupling the spatial context from the temporal dynamics of facial expressions. In: Proceedings of 2019 International Symposium on Networks, Computers and Communications (ISNCC); 18–20 June 2019; Istanbul, Turkey.
71. Verma M, Vipparthi SK, Singh G, Murala S. LEARNet: Dynamic imaging network for micro expression recognition. IEEE Transactions on Image Processing 2019; 29: 1618–1627. doi: 10.1109/TIP.2019.2912358
72. Dong J, Zheng H, Lian L. Dynamic facial expression recognition based on convolutional neural networks with dense connections. In: Proceedings of 2018 24th International Conference on Pattern Recognition (ICPR); 20–24 August 2018; Beijing, China.
73. Maheswari VU, Varaprasad G, Viswanadharaju S. Local double directional stride maximum patterns for facial expression retrieval. International Journal of Biometrics 2022; 14(3–4): 439–452. doi: 10.1504/ijbm.2022.124682
74. Lai YH, Lai SH. Emotion-preserving representation learning via generative adversarial network for multiview facial expression recognition. In: Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); 15–19 May 2018; Xi’an, China.
75. Ramakrishnan S, El Emary IMM. Speech emotion recognition approaches in human computer interaction. Telecommuniaction Systems 2013; 52: 1467–1478. doi: 10.1007/s11235-011-9624-z
76. Chang J, Scherer S. Learning representations of emotional speech with deep convolutional generative adversarial networks. In: Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 5–9 March 2017; New Orleans, USA.
77. Vo TH, Lee GS, Yang HJ, Kim SH. Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 2020; 8: 131988–132001. doi: 10.1109/ACCESS.2020.3010018
78. Yang Q. An Introduction to Transfer Learning. Springer; 2008.
79. Maheswari VU, Aluvalu R, Kantipudi MP, et al. Driver drowsiness prediction based on multiple aspects using image processing techniques. IEEE Access 2022; 10: 54980–54990. doi: 10.1109/ACCESS.2022.3176451
80. Narula V, Wang ZY, Chaspari T. An adversarial learning framework for preserving users’ anonymity in face-based emotion recognition. arXiv 2020; arXiv:2001.06103. doi: 10.48550/arXiv.2001.06103
81. Soysal OA, Guzel MS. An introduction to zero-shot learning: An essential review. In: Proceedings of 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA); 26–28 June 2020; Ankara, Turkey.
82. Wu J, Zhang Y, Zhao X, Gao W. A generalized zero-shot framework for emotion recognition from body gestures. arXiv 2020; arXiv:2010.06362. doi: 10.48550/arXiv.2010.06362
83. McMahan HB, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics 2017; 54: 1273–1282. doi: 10.48550/arXiv.1602.05629
84. Longo L, Goebel R, Lecue F, et al. Explainable artificial intelligence: Concepts, applications, research challenges and visions. International Cross-Domain Conference for Machine Learning and Knowledge Extraction. In: Lecture Notes in Computer Science: Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics 2020; 12279: 1–16. doi: 10.1007/978-3-030-57321-8_1
85. Generosi A, Ceccacci S, Mengoni M. A deep learning-based system to track and analyze customer behavior in retail store. In: Proceedings of 2018 IEEE 8th International Conference on Consumer Electronics-Berlin (ICCE-Berlin); 2–5 September 2018; Berlin, Germany.
86. Haque MIU, Valles D. A facial expression recognition approach using DCNN for autistic children to identify emotions. In: Proceedings of 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON); 1–3 November 2018; Vancouver, Canada.
87. Liu X, Lee K. Optimized facial emotion recognition technique for assessing user experience. In: Proceedings of 2018 IEEE Games, Entertainment, Media Conference (GEM); 15–17 August 2018; Galway, Ireland.
DOI: https://doi.org/10.32629/jai.v6i2.631
Refbacks
- There are currently no refbacks.
Copyright (c) 2023 Sridhar Reddy Karra, Arun L. Kakhandki
License URL: https://creativecommons.org/licenses/by-nc/4.0