Xi Yang
ABSTRACT
While previous studies of Conversational Agents (e.g. Siri, Google Assistant, Alexa and Cortana) have focused on evaluating usability and exploring capabilities of these systems, little work has examined users' affective experiences. In this paper we present a survey study with 171 participants to examine CA users' affective experiences. Specifically, we present four major usage scenarios, users' affective responses in these scenarios, and the factors which influenced the affective responses. We found that users' overall experience was positive with interest being the most salient positive emotion. Affective responses differed depending on the scenarios. Both pragmatic and hedonic qualities influenced affect. The factors underlying pragmatic quality are: helpfulness, proactivity, fluidity, seamlessness and responsiveness. The factors underlying hedonic quality are: comfort in human-machine conversation, pride of using cutting-edge technology, fun during use, perception of having a human-like assistant, concern about privacy and fear of causing distraction.
Supplemental Material
- Sandeep Avula. 2017. Searchbots: Using Chatbots in Collaborative Information-seeking Tasks (SIGIR '17). ACM, New York, NY, USA, 1375--1375. Google Scholar
Digital Library
- C. Baber. 1993. Developing interactive speech technology, Interactive speech technology: human factors issues in the application of speech input/output to computers. Taylor & Francis, Inc., Bristol, PA. Google ScholarDigital Library
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.Google Scholar
- Susan E. Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. (1990). http://www.citeulike.org/group/890/article/Google Scholar
- Justine Cassell. 2000. Embodied conversational agents. MIT press.Google ScholarDigital Library
- Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone. 1994. Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents. ACM, 413--420. Google ScholarDigital Library
- Justine Cassell and Kristinn R. Thorisson. 1999. The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence 13, 4--5 (1999), 519--538.Google ScholarCross Ref
- Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37--46.Google Scholar
- Phil Cohen, Adam Cheyer, Eric Horvitz, Rana El Kaliouby, and Steve Whittaker. 2016. On the future of personal assistants. ACM, 1032--1037. http://dl.acm.org/citation.cfm?id=2886425 Google ScholarDigital Library
- Benjamin R. Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. What can i help you with?: infrequent users' experiences of intelligent personal assistants. ACM, 43. http://dl.acm.org/citation. cfm?id=3098539 Google ScholarDigital Library
- Pieter M.A. Desmet. 2012. Faces of Product Pleasure: 25 Positive Emotions in Human-Product Interactions. International Journal of Dsign 6, 2 (2012), 1--29. http://www.ijdesign.org/ojs/index.php/IJDesign/ article/view/1190/459Google Scholar
- Boris Egloff, Stefan C. Schmukle, Lawrence R. Burns, CarlWalter Kohlmann, and Michael Hock. 2003. Facets of dynamic positive affect: Differentiating joy, interest, and activation in the positive and negative affect schedule (PANAS). Journal of personality and social psychology 85, 3 (2003), 528--540. https://www.researchgate.net/profile/Boris_Egloff/publication/ 5995270_Facets_of_Dynamic_Positive_Affect_Differentiating_Joy_ Interest_and_Activation_in_the_Positive_and_Negative_Affect_ Schedule_PANAS/links/540982830cf2822fb738e692.pdfGoogle ScholarCross Ref
- Malin Eiband, Mohamed Khamis, Emanuel von Zezschwitz, Heinrich Hussmann, and Florian Alt. 2017. Understanding shoulder surfing in the wild: Stories from users and observers. ACM, 4254--4265. http: //dl.acm.org/citation.cfm?id=3025636 Google ScholarDigital Library
- John C. Flanagan. 1954. The critical incident technique. Psychological bulletin 51, 4 (1954), 327. http://psycnet.apa.org/journals/bul/51/4/327/Google Scholar
- Jodi Forlizzi and Katja Battarbee. 2004. Understanding experience in interactive systems. ACM, 261--268. http://dl.acm.org/citation.cfm? id=1013152 Google ScholarDigital Library
- Barbara L. Fredrickson, Michael A. Cohn, Kimberly A. Coffey, Jolynn Pek, and Sandra M. Finkel. 2008. Open hearts build lives: positive emotions, induced through loving-kindness meditation, build consequential personal resources. Journal of personality and social psychology 95, 5 (2008), 1045.Google ScholarCross Ref
- James Glass. 1999. Challenges for spoken dialogue systems. http://www.cs.cmu.edu/afs/cs.cmu.edu/Web/People/dod/papers/ glass99.pdfGoogle Scholar
- Arthur C. Graesser, Nia Dowell, and Danielle Clewley. 2017. Assessing Collaborative Problem Solving Through Conversational Agents. Springer, 65--80.Google Scholar
- Bettina Graf, Maike Krüger, Felix Müller, Alexander Ruhland, and Andrea Zech. 2015. Nombot: Simplify Food Tracking (MUM '15). ACM, New York, NY, USA, 360--363. Google ScholarDigital Library
- Marc Hassenzahl. 2003. The thing and I: understanding the relationship between user and product. Springer, 31--42. https://link.springer.com/ content/pdf/10.1007/1--4020--2967--5_4.pdfGoogle Scholar
- Marc Hassenzahl. 2010. Experience Design: Technology for All the Right Reasons. Morgan & Claypool Publishers. Google ScholarCross Ref
- Marc Hassenzahl, Andreas Beu, and Michael Burmester. 2001. Engineering joy. Ieee Software 18, 1 (2001), 70. http://search.proquest. com/openview/5d220f6a29326527ad73de88fe581782/1?pq-origsite= gscholar Google ScholarDigital Library
- Marc Hassenzahl, Sarah Diefenbach, and Anja Göritz. 2010. Needs, affect, and interactive products--Facets of user experience. Interacting with computers 22, 5 (2010), 353--362. http://www.sciencedirect.com/ science/article/pii/S0953543810000366 Google ScholarDigital Library
- Marc Hassenzahl and Andrew Monk. 2010. The inference of perceived usability from beauty. Human--Computer Interaction 25, 3 (2010), 235--260.Google ScholarCross Ref
- Marc Hassenzahl and Noam Tractinsky. 2006. User experience - a research agenda. Behaviour & Information Technology 25, 2 (March 2006), 91--97.Google ScholarCross Ref
- Marc Hassenzahl, Annika Wiklund-Engblom, Anette Bengs, Susanne Hägglund, and Sarah Diefenbach. 2015. Experience-Oriented and Product-Oriented Evaluation: Psychological Need Fulfillment, Positive Affect, and Product Perception. International Journal of HumanComputer Interaction 31, 8 (Aug. 2015), 530--544.Google ScholarCross Ref
- Ronald J. Hunt. 1986. Percent agreement, Pearson's correlation, and kappa as measures of inter-examiner reliability. Journal of Dental Research 65, 2 (1986), 128--130.Google ScholarCross Ref
- Patrick W. Jordan. 2002. Designing Pleasurable Products: An Introduction to the New Human Factors. CRC Press.Google ScholarCross Ref
- Malte F. Jung. 2017. Affective Grounding in Human-Robot Interaction. ACM, 263--273. Google ScholarDigital Library
- Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Understanding user satisfaction with intelligent assistants. ACM, 121--130. http://dl.acm.org/citation.cfm?id=2854961 Google ScholarDigital Library
- Hannu Korhonen, Juha Arrasvuori, and Kaisa Väänänen-VainioMattila. 2010. Analysing user experience of personal mobile products through contextual factors. ACM, 11. http://dl.acm.org/citation.cfm? id=1899486 Google ScholarDigital Library
- Klaus Krippendorff. 2007. Computing Krippendorff's alpha reliability. Departmental papers (ASC) (2007), 43. http://repository.upenn.edu/ cgi/viewcontent.cgi?article=1043&context=asc_papersGoogle Scholar
- Ewa Luger and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. ACM, 5286--5297. http://dl.acm.org/citation.cfm?id=2858288 Google ScholarDigital Library
- Elisa D. Mekler and Kasper Hornbæk. 2016. Momentary Pleasure or Lasting Meaning?: Distinguishing Eudaimonic and Hedonic User Experiences (CHI '16). ACM, New York, NY, USA, 4509--4520. Google ScholarDigital Library
- Robert J. Moore, Raphael Arar, Guang-Jie Ren, and Margaret H. Szymanski. 2017. Conversational UX Design (CHI EA '17). ACM, New York, NY, USA, 492--497. Google ScholarDigital Library
- Livia J. Müller, Elisa D. Mekler, and Klaus Opwis. 2015. Facets In HCI: Towards Understanding Eudaimonic UX--Preliminary Findings. ACM, 2283--2288. http://dl.acm.org/citation.cfm?id=2732836 Google ScholarDigital Library
- Quynh N. Nguyen and Anna Sidorova. 2017. AI capabilities and user experiences: a comparative study of user reviews for assistant and non-assistant mobile apps. (2017). http://aisel.aisnet.org/amcis2017/ SemanticsIS/Presentations/5/Google Scholar
- Donald A. Norman. 2005. Emotional design: Why we love (or hate) everyday things. Basic books. https://books.google.co.uk/books? hl=en&lr=&id=h_wAbnGlOC4C&oi=fnd&pg=PA3&dq=emotional+ design+&ots=efVGQfn5m9&sig=LCB3ieNHbdf1X_Wf4egX_3fa5WYGoogle Scholar
- Sabine Payr. 2013. Virtual Butlers and Real People: Styles and Practices in Long-Term Use of a Companion. Springer. http://link.springer.com/ content/pdf/10.1007/978--3--642--37346--6.pdf#page=146 Google ScholarDigital Library
- Martin Porcheron, Joel E. Fischer, Moira McGregor, Barry Brown, Ewa Luger, Heloisa Candello, and Kenton O'Hara. 2017. Talking with Conversational Agents in Collaborative Action (CSCW '17 Companion). ACM, New York, NY, USA, 431--436. Google ScholarDigital Library
- Simon Provoost, Ho Ming Lau, Jeroen Ruwaard, and Heleen Riper. 2017. Embodied conversational agents in clinical psychology: A scoping review. Journal of medical Internet research 19, 5 (2017).Google ScholarCross Ref
- Amanda Purington, Jessie G. Taft, Shruti Sannon, Natalya N. Bazarova, and Samuel Hardman Taylor. 2017. Alexa is my new BFF: Social Roles, User Satisfaction, and Personification of the Amazon Echo. ACM, 2853-- 2859. http://dl.acm.org/citation.cfm?id=3053246 Google ScholarDigital Library
- Rajmohan Rammohan, Nisha Dhanabalsamy, Ves Dimov, and Frank J. Eidelman. 2017. Smartphone Conversational Agents (Apple Siri, Google, Windows Cortana) and Questions about Allergy and Asthma Emergencies. Journal of Allergy and Clinical Immunology 139, 2 (2017), AB250.Google ScholarCross Ref
- William E. Scott Jr. 1966. Activation theory and task design. Organizational behavior and human performance 1, 1 (1966), 3--30.Google Scholar
- Joel Sebastian and Deborah Richards. 2017. Changing stigmatizing attitudes to mental health via education and contact with embodied conversational agents. Computers in Human Behavior 73 (2017), 479-- 488. Google ScholarDigital Library
- Kennon M. Sheldon, Andrew J. Elliot, Youngmee Kim, and Tim Kasser. 2001. What is satisfying about satisfying events? Testing 10 candidate psychological needs. Journal of personality and social psychology 80, 2 (2001), 325. http://psycnet.apa.org/journals/psp/80/2/325/Google ScholarCross Ref
- Alexandre N. Tuch, Rune Trusell, and Kasper Hornbæk. 2013. Analyzing users' narratives to understand experience with interactive products. ACM, 2079--2088. http://dl.acm.org/citation.cfm?id=2481285 Google ScholarDigital Library
- Bert Vandenberghe. 2017. Bot Personas as Off-The-Shelf Users. ACM, 782--789. http://dl.acm.org/citation.cfm?id=3052767 Google ScholarDigital Library
- Roel Vertegaal, Robert Slagter, Gerrit Van der Veer, and Anton Nijholt. 2001. Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. ACM, 301--308. Google ScholarDigital Library
- Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles LA Clarke. 2017. Exploring Conversational Search With Humans, Assistants, and Wizards. ACM, 2187--2193. http://dl.acm.org/ citation.cfm?id=3053175 Google ScholarDigital Library
- David Watson, Lee A. Clark, and Auke Tellegen. 1988. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology 54, 6 (1988), 1063. http://psycnet.apa.org/psycinfo/1988--31508-001Google ScholarCross Ref
- JungKyoon Yoon, Pieter MA Desmet, and Aadjan van der Helm. 2012. Design for interest: Exploratory study on a distinct positive emotion in human-product interaction. International Journal of Design 6, 2 (2012).Google Scholar
- Jennifer Zamora. 2017. Rise of the Chatbots: Finding A Place for Artificial Intelligence in India and US (IUI '17 Companion). ACM, New York, NY, USA, 109--112. Google ScholarDigital Library
Index Terms
- Understanding Affective Experiences with Conversational Agents
Recommendations
Affective guidance of intelligent agents: How emotion controls cognition
How do emotions and moods color cognition? In this article, we examine how such reactions influence both judgments and cognitive performance. We argue that many affective influences are due, not to affective reactions themselves, but to the information ...
The Impact of Expectations on User Experience: Surprising the User
BCS-HCI '14: Proceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014 - Sand, Sea and Sky - Holiday HCIExciting product design and great user experience (UX) have become one of the most important selling points for digital interactive products. Strong competition fosters innovative user experience design and the development of new design techniques. ...
Modeling Multimodal Expression of User's Affective Subjective Experience
With the growing importance of information technology in our everyday life, new types of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major ...
Comments