Joongyum Kim
Taesik Gong
Sung-Ju Lee
ABSTRACT
While people primarily communicate with text in mobile chat applications, they are increasingly using visual elements such as images, emojis, and memes. Using such visual elements could help users communicate clearly and make chatting experience enjoyable. However, finding and inserting contextually appropriate images during the chat can be both tedious and distracting. We introduce MilliCat, a real-time image suggestion system that recommends images that match the chat content within a mobile chat application (i.e., autocomplete with images). MilliCat combines natural language processing (e.g., keyword extraction, dependency parsing) and mobile computing (e.g., resource and energy-efficiency) techniques to autonomously make image suggestions when users might want to use images. Through multiple user studies, we investigated the effectiveness of our design choices, the frequency and motivation of image usage by the participants, and the impact of MilliCat on mobile chat experiences. Our results indicate that MilliCat’s real-time image suggestion enables users to quickly and conveniently select and display images on mobile chat by significantly reducing the latency in the image selection process (3.19 × improvement) and consequently more frequent image usage (1.8 ×) than existing solutions. Our study participants reported that they used images more often with MilliCat as the images helped them convey information more effectively, emphasize their opinion, express emotions, and have fun chatting experience.
Supplemental Material
- Paul Rayson A, Dawn Archer B, Scott Piao B, and Tony Mcenery B. 2004. The UCREL Semantic Analysis System. In In proceedings of the workshop on Beyond Named Entity Recognition Semantic labelling for NLP tasks in association with 4th International Conference on Language Resources and Evaluation (LREC 2004). European Language Resources Association (ELRA), Paris, France, 7–12.Google Scholar
- Salvatore Andolina, Valeria Orso, Hendrik Schneider, Khalil Klouche, Tuukka Ruotsalo, Luciano Gamberini, and Giulio Jacucci. 2018. Investigating Proactive Search Support in Conversations. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 1295–1307. https://doi.org/10.1145/3196709.3196734Google ScholarDigital Library
- Ron Baecker. 2003. A Principled Design for Scalable Internet Visual Communications with Rich Media, Interactivity, and Structured Archives. In Proceedings of the 2003 Conference of the Centre for Advanced Studies on Collaborative Research(CASCON ’03). IBM Press, Toronto, Ontario, Canada, 16–29.Google ScholarDigital Library
- Craig Baehr. 2012. Incorporating user appropriation, media richness, and collaborative knowledge sharing into blended e-learning training tutorial. IEEE Transactions on Professional Communication 55, 2(2012), 175–184.Google ScholarCross Ref
- Saeideh Bakhshi, David A. Shamma, Lyndon Kennedy, Yale Song, Paloma de Juan, and Joseph ’Jofish’ Kaye. 2016. Fast, Cheap, and Good: Why Animated GIFs Engage Us. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 575–586. https://doi.org/10.1145/2858036.2858532Google ScholarDigital Library
- Frank Biocca. 2000. New media technology and youth: Trends in the evolution of new media. Journal of Adolescent Health 27, 2 (2000), 22–29.Google ScholarCross Ref
- David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993–1022.Google ScholarDigital Library
- Daniel Buschek, Mariam Hassib, and Florian Alt. 2018. Personal Mobile Messaging in Context: Chat Augmentations for Expressiveness and Awareness. ACM Trans. Comput.-Hum. Interact. 25, 4, Article 23 (Aug. 2018), 33 pages. https://doi.org/10.1145/3201404Google ScholarDigital Library
- Shana K Carpenter and Kellie M Olson. 2012. Are pictures good for learning new vocabulary in a foreign language? Only if you think they are not.Journal of experimental psychology: learning, memory, and cognition 38, 1(2012), 92.Google Scholar
- David Chen and Chengzheng Sun. 2002. Real-time text chat via collaborative editing systems. In Proceedings. of the ACM CSCW 2002 Workshop on Collaborative Editing. ACM, New York, NY, USA.Google Scholar
- Fanglin Chen, Kewei Xia, Karan Dhabalia, and Jason I. Hong. 2019. MessageOnTap: A Suggestive Interface to Facilitate Messaging-Related Tasks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300805Google ScholarDigital Library
- Tao Chen and Min-Yen Kan. 2013. Creating a live, public short message service corpus: the NUS SMS corpus. Language Resources and Evaluation 47, 2 (2013), 299–335.Google Scholar
- Zhenpeng Chen, Xuan Lu, Wei Ai, Huoran Li, Qiaozhu Mei, and Xuanzhe Liu. 2018. Through a Gender Lens: Learning Usage Patterns of Emojis from Large-Scale Android Users. In Proceedings of the 2018 World Wide Web Conference(Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 763–772. https://doi.org/10.1145/3178876.3186157Google ScholarDigital Library
- Henriette Cramer, Paloma de Juan, and Joel Tetreault. 2016. Sender-intended Functions of Emojis in US Messaging. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (Florence, Italy) (MobileHCI ’16). ACM, New York, NY, USA, 504–509. https://doi.org/10.1145/2935334.2935370Google ScholarDigital Library
- Dango. 2020. Dango.https://getdango.com/.Google Scholar
- Google Gboard. 2020. Google Gboard.https://play.google.com/store/apps/details?id=com.google.android.inputmethod.latin.Google Scholar
- GIPHY. 2020. GIPHY.http://giphy.com/.Google Scholar
- Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient Natural Language Response Suggestion for Smart Reply. arxiv:1705.00652 [cs.CL]Google Scholar
- Peter S Houts, Cecilia C Doak, Leonard G Doak, and Matthew J Loscalzo. 2006. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient education and counseling 61, 2 (2006), 173–190.Google Scholar
- Y. Huang, Y. L. Murphey, and Y. Ge. 2013. Automotive diagnosis typo correction using domain knowledge and machine learning. In 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM). IEEE, New York, NY, USA, 267–274.Google Scholar
- Yinghao Huang, Yi Lu Murphey, and Yao Ge. 2015. Intelligent typo correction for text mining through machine learning. International Journal of Knowledge Engineering and Data Mining 3, 2(2015), 115–142.Google ScholarDigital Library
- Jialun ”Aaron” Jiang, Jed R. Brubaker, and Casey Fiesler. 2017. Understanding Diverse Interpretations of Animated GIFs. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). ACM, New York, NY, USA, 1726–1732. https://doi.org/10.1145/3027063.3053139Google ScholarDigital Library
- Jialun ”Aaron” Jiang, Casey Fiesler, and Jed R. Brubaker. 2018. ’The Perfect One’: Understanding Communication Practices and Challenges with Animated GIFs. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 80 (Nov. 2018), 20 pages. https://doi.org/10.1145/3274349Google ScholarDigital Library
- Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, and Vivek Ramavajjala. 2016. Smart Reply: Automated Response Suggestion for Email. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Association for Computing Machinery, New York, NY, USA, 955–964. https://doi.org/10.1145/2939672.2939801Google ScholarDigital Library
- Mark D Kernighan, Kenneth W Church, and William A Gale. 1990. A spelling correction program based on a noisy channel model. In Proceedings of the 13th conference on Computational linguistics-Volume 2. Association for Computational Linguistics, New York, NY, USA, 205–210.Google ScholarDigital Library
- Sujay Khandekar, Joseph Higg, Yuanzhe Bian, Chae Won Ryu, Jerry O. Talton Iii, and Ranjitha Kumar. 2019. Opico: A Study of Emoji-first Communication in a Mobile Social App. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW ’19). ACM, New York, NY, USA, 450–458. https://doi.org/10.1145/3308560.3316547Google ScholarDigital Library
- Joongyum Kim, Taesik Gong, Bogoan Kim, Jaeyeon Park, Woojeong Kim, Evey Huang, Kyungsik Han, Juho Kim, Jeonggil Ko, and Sung-Ju Lee. 2020. No More One Liners: Bringing Context into Emoji Recommendations. Trans. Soc. Comput. 3, 2, Article 9 (April 2020), 25 pages. https://doi.org/10.1145/3373146Google ScholarDigital Library
- Joon-Gyum Kim, Chia-Wei Wu, Alvin Chiang, JeongGil Ko, and Sung-Ju Lee. 2016. A Picture is Worth a Thousand Words: Improving Mobile Messaging with Real-Time Autonomous Image Suggestion. In Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications (St. Augustine, Florida, USA) (HotMobile ’16). Association for Computing Machinery, New York, NY, USA, 51–56. https://doi.org/10.1145/2873587.2873602Google ScholarDigital Library
- K. Kim, W. Ryu, J. Park, and S. Lee. 2019. meChat: In-Device Personal Assistant for Conversational Photo Sharing. IEEE Internet Computing 23, 2 (March 2019), 23–30. https://doi.org/10.1109/MIC.2018.2883059Google ScholarDigital Library
- Philippe Kimura-Thollander and Neha Kumar. 2019. Examining the Global Language of Emojis: Designing for Cultural Representation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 495, 14 pages. https://doi.org/10.1145/3290605.3300725Google ScholarDigital Library
- Miki Liu, Austin Wong, Ruhi Pudipeddi, Betty Hou, David Wang, and Gary Hsieh. 2018. ReactionBot: Exploring the Effects of Expression-Triggered Emoji in Text Messages. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 110 (Nov. 2018), 16 pages. https://doi.org/10.1145/3274379Google ScholarDigital Library
- Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei. 2016. Learning from the Ubiquitous Language: An Empirical Analysis of Emoji Usage of Smartphone Users. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). ACM, New York, NY, USA, 770–780. https://doi.org/10.1145/2971648.2971724Google ScholarDigital Library
- Richard E Mayer. 2003. The promise of multimedia learning: using the same instructional design methods across different media. Learning and instruction 13, 2 (2003), 125–139.Google Scholar
- Richard E Mayer and Roxana Moreno. 2002. Aids to computer-based multimedia learning. Learning and instruction 12, 1 (2002), 107–119.Google Scholar
- Qiaozhu Mei. 2019. Decoding the New World Language: Analyzing the Popularity, Roles, and Utility of Emojis. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW ’19). ACM, New York, NY, USA, 417–418. https://doi.org/10.1145/3308560.3316541Google ScholarDigital Library
- Facebook Messenger. 2020. Facebook Messenger.https://www.messenger.com/.Google Scholar
- David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes 30, 1 (2007), 3–26.Google ScholarCross Ref
- Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proc. of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013). Association for Computational Linguistics, New York, NY, USA, 380–390.Google Scholar
- Larry Percy and John R Rossiter. 1997. A theory-based approach to pretesting advertising. Psychology Press, Mortimer Street, London. 267–281 pages.Google Scholar
- Matthew Pittman and Brandon Reich. 2016. Social media and loneliness: Why an Instagram picture may be worth more than a thousand Twitter words. Computers in Human Behavior 62 (2016), 155–167.Google ScholarDigital Library
- Henning Pohl, Christian Domin, and Michael Rohs. 2017. Beyond Just Text: Semantic Emoji Similarity Modeling to Support Expressive Communication. ACM Trans. Comput.-Hum. Interact. 24, 1, Article 6 (March 2017), 42 pages. https://doi.org/10.1145/3039685Google ScholarDigital Library
- Anabel Quan-Haase and Alyson L Young. 2010. Uses and gratifications of social media: A comparison of Facebook and instant messaging. Bulletin of Science, Technology & Society 30, 5 (2010), 350–361.Google ScholarCross Ref
- Paul Rayson. 2008. From key words to key semantic domains. International journal of corpus linguistics 13, 4 (2008), 519–549.Google ScholarCross Ref
- Paul Rayson, Roger Garside, and Pete Sawyer. 2000. Assisting Requirements Engineering with Semantic Document Analysis. In Content-Based Multimedia Information Access - Volume 2 (Paris, France) (RIAO ’00). LE CENTRE DE HAUTES ETUDES INTERNATIONALES D’INFORMATIQUE DOCUMENTAIRE, Paris, FRA, 1363–1371.Google Scholar
- Bradley Rhodes and Thad Starner. 1996. Remembrance Agent: A continuously running automated information retrieval system. In The Proceedings of The First International Conference on The Practical Application Of Intelligent Agents and Multi Agent Technology. AAAI Technical Report SS-96-02, Palo Alto, California, 487–495.Google Scholar
- Ladislao Salmerón, Thierry Baccino, Jose J Cañas, Rafael I Madrid, and Inmaculada Fajardo. 2009. Do graphical overviews facilitate or hinder comprehension in hypertext?Computers & Education 53, 4 (2009), 1308–1319.Google Scholar
- Andreas Schmeil, Martin Eppler, and Mattia Gubler. 2009. An Experimental Comparison of 3D Virtual Environments and Text Chat as Collaboration Tools.Electronic Journal of Knowledge Management 7, 5 (2009), 637–646.Google Scholar
- Sebastian Schuster and Christopher D Manning. 2016. Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). European Language Resources Association (ELRA), Paris, France, 2371–2378.Google Scholar
- Aaron Smith. 2015. U.S. Smartphone Use in 2015. PEW Research Center Report. Available at: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/.Google Scholar
- S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. MacArthur Foundation Digital Media and Learning Initiative, Chicago, IL, USA. 73–100 pages.Google Scholar
- Kakao Talk. 2020. Kakao Talk.http://www.kakao.com/talk/en.Google Scholar
- Garreth W. Tigwell and David R. Flatla. 2016. Oh That’s What You Meant!: Reducing Emoji Misunderstanding. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (Florence, Italy) (MobileHCI ’16). ACM, New York, NY, USA, 859–866. https://doi.org/10.1145/2957265.2961844Google ScholarDigital Library
- Tomoko Watanabe Traphagan, Yueh-hui Vanessa Chiang, Hyeseung Maria Chang, Benjaporn Wattanawaha, Haekyung Lee, Michael Charles Mayrath, Jeongwon Woo, Hyo-Jin Yoon, Min Jung Jee, and Paul E Resta. 2010. Cognitive, social and teaching presence in a virtual world and a text chat. Computers & Education 55, 3 (2010), 923–936.Google ScholarDigital Library
- M. Uther. 2002. Mobile Internet usability: what can ’mobile learning’ learn from the past?. In Proceedings. IEEE International Workshop on Wireless and Mobile Technologies in Education. IEEE, New York, NY, USA, 174–176.Google ScholarCross Ref
- Zheng Wang, Prabu David, Jatin Srivastava, Stacie Powers, Christine Brady, Jonathan D’Angelo, and Jennifer Moreland. 2012. Behavioral performance and visual attention in communication multitasking: A comparison between instant messaging and online voice chat. Computers in Human Behavior 28, 3 (2012), 968–975.Google ScholarDigital Library
- Sarah Wiseman and Sandy J. J. Gould. 2018. Repurposing Emoji for Personalised Communication: Why (PIZZA_EMOJI) Means “I Love You”. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 152, 10 pages. https://doi.org/10.1145/3173574.3173726Google ScholarDigital Library
- Ning Ye, Ariel Fuxman, Vivek Ramavajjala, Sergey Nazarov, J. McGregor, and Sujith Ravi. 2018. PhotoReply: Automatically Suggesting Conversational Responses to Photos. In WWW ’18: Proceedings of the 2018 World Wide Web Conference. ACM, New York, NY, USA, 1893–1899. https://doi.org/10.1145/3178876.3186164Google ScholarDigital Library
- Peijun Zhao, Jia Jia, Yongsheng An, Jie Liang, Lexing Xie, and Jiebo Luo. 2018. Analyzing and Predicting Emoji Usages in Social Media. In Companion Proceedings of the The Web Conference 2018 (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 327–334. https://doi.org/10.1145/3184558.3186344Google ScholarDigital Library
- Rui Zhou, Jasmine Hentschel, and Neha Kumar. 2017. Goodbye Text, Hello Emoji: Mobile Communication on WeChat in China. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 748–759. https://doi.org/10.1145/3025453.3025800Google ScholarDigital Library
Recommendations
Real-time vehicle detection using equi-height mosaicking image
RACS '13: Proceedings of the 2013 Research in Adaptive and Convergent SystemsIn this paper, we present a real-time forward vehicle detection warning system using a novel image representation called an equi-height mosaicking image. The proposed system uses a GPU (graphic processing unit) based approach for the real-time ...
A real - time image restoration
CompSysTech '03: Proceedings of the 4th international conference conference on Computer systems and technologies: e-LearningAn approach for image enhancement in case of smoothed images with a Known kernel is suggested, it is assumed t'hat the original image could be successfully restored if a correction proportional to the difference between it and the smoothed image is ...
Optimized contrast enhancement for real-time image and video dehazing
A fast and optimized dehazing algorithm for hazy images and videos is proposed in this work. Based on the observation that a hazy image exhibits low contrast in general, we restore the hazy image by enhancing its contrast. However, the overcompensation ...
Comments