Abstract
In this paper, a story-telling social robot is proposed. The robot is able to modify the evolution of the story considering the emotions the audience is feeling. To do that, the robot uses the user’s emotion from his/her face. We have used a deep learning-based model to identify the emotion. This model was trained and tested on state of the art dataset. We also demonstrate that involving generated, realistic samples in the training process as a way of data augmentation does not benefit the model at all. The mentioned samples are generated by a state of the art GAN, which is able to translate neutral faces to a range of emotions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Antoniou, A., Storkey, A., Edwards, H.: Augmenting image classifiers using data augmentation generative adversarial networks. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) Artificial Neural Networks and Machine Learning - ICANN 2018, pp. 594–603. Springer, Cham (2018)
Benitez-Quiroz, C.F., Srinivasan, R., Feng, Q., Wang, Y., Martínez, A.M.: EmotioNet challenge: recognition of facial expressions of emotion in the wild. CoRR. http://arxiv.org/abs/1703.01210 (2017)
Chatzara, K., Karagiannidis, C., Mavropoulou, S., Stamatis, D.: Digital storytelling for children with autism: software development and pilot application, pp. 287–300. Springer New York (2014). https://doi.org/10.1007/978-1-4614-6501-0_19
Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (June 2009)
Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., Bengio, Y.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015). http://www.sciencedirect.com/science/article/pii/S0893608014002159, special Issue on “Deep Learning of Representations”
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition supplementary materials (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, vol. 1, pp. 1097–1105. NIPS 2012, Curran Associates Inc., USA (2012). http://dl.acm.org/citation.cfm?id=2999134.2999257
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (June 2010)
Mariani, G., Scheidegger, F., Istrate, R., Bekas, C., Malossi, A.C.I.: Bagan: data augmentation with balancing GAN. arXiv:1803.09655 (2018)
Mikołajczyk, A., Grochowski, M.: Data augmentation for improving deep learning in image classification problem. In: 2018 International Interdisciplinary PhD Workshop (IIPhDW), pp. 117–122 (May 2018)
Quinn, M.-A., Grant Sivesind, G.R.: Real-time emotion recognition from facial expressions. CS229 - Stanford University. http://cs229.stanford.edu/proj2017/final-posters/5165034.pdf
Mollahosseini, A., Hassani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. CoRR (2017). http://arxiv.org/abs/1708.03985
Parsons, S., Guldberg, K., Porayska-Pomsta, K., Lee, R.: Digital stories as a method for evidence-based practice and knowledge co-creation in technology-enhanced learning for children with autism. Int. J. Res. Meth. Educ. 38(3), 247–271 (2015). https://doi.org/10.1080/1743727X.2015.1019852
Perez, L., Wang, J.: The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621 (2017)
Pinto Costa, A., Steffgen, G., Rodriguez Lera, F.J., Nazarikhorram, A., Ziafati, P.: Socially assistive robots for teaching emotional abilities to children with autism spectrum disorder. In: 3rd Workshop on Child-Robot Interaction at HRI 2017 (2017)
Pramerdorfer, C., Kampel, M.: Facial expression recognition using convolutional neural networks: state of the art. arXiv:1612.02903 (2016)
Rouhi, A., Catania, F., Cosentino, G., Gelsomini, M., Spitale, M.: Emotify: emotional game for children with autism spectrum disorder based-on machine learning (February 2019)
Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: database and results. Image Vis. Comput. 47, 3–18 (2016). http://www.sciencedirect.com/science/article/pii/S0262885616000147, 300-W, the First Automatic Facial Landmark Detection in-the-Wild Challenge
dos Santos Tanaka, F.H.K., Aranha, C.: Data augmentation using GANs. CoRR (2019). http://arxiv.org/abs/1904.09135
Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 958–963 (August 2003)
Stamatis, D., Karagiannidis, C., Mavropoulou, S.: Digital storytelling for children with autism: software development and pilot application, pp. 287–300 (September 2014)
Taylor, L., Nitschke, G.: Improving deep learning using generic data augmentation. CoRR (2017). http://arxiv.org/abs/1708.06020
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)
Acknowledgements
This work has been supported by the Spanish Government TIN2016-76515R Grant, supported with Feder funds. This work has also been supported by a Spanish grant for PhD studies ACIF/2017/243 and FPU16/00887. Thanks to Nvidia for the generous donation of a Titan Xp and a Quadro P6000.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Azuar, D., Gallud, G., Escalona, F., Gomez-Donoso, F., Cazorla, M. (2020). A Story-Telling Social Robot with Emotion Recognition Capabilities for the Intellectually Challenged. In: Silva, M., Luís Lima, J., Reis, L., Sanfeliu, A., Tardioli, D. (eds) Robot 2019: Fourth Iberian Robotics Conference. ROBOT 2019. Advances in Intelligent Systems and Computing, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-030-36150-1_49
Download citation
DOI: https://doi.org/10.1007/978-3-030-36150-1_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36149-5
Online ISBN: 978-3-030-36150-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)