Drini Cami
Daniel Vogel
ABSTRACT
We introduce a new pen input space by forming postures with the same hand that also grips the pen while writing, drawing, or selecting. The postures contact the multitouch surface around the pen to enable detection without special sensors. A formative study investigates the effectiveness, accuracy, and comfort of 33 candidate postures in controlled tasks. The results indicate a useful subset of postures. Using raw capacitive sensor data captured in the study, a convolutional neural network is trained to recognize 10 postures in real time. This recognizer is used to create application demonstrations for pen-based document annotation and vector drawing. A small usability study shows the approach is feasible.
Supplemental Material
- Chainer: A flexible framework for neural networks. https://chainer.org/Google Scholar
- Xiaojun Bi, Tomer Moscovich, Gonzalo Ramos, Ravin Balakrishnan, and Ken Hinckley. 2008. An exploration of pen rolling for pen-based interaction. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, Monterey, CA, USA, 191--200. Google ScholarDigital Library
- Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. In Proceedings of the working conference on Advanced visual interfaces. ACM, Napoli, Italy, 154--161. Google ScholarDigital Library
- William Buxton. 1995. Touch, Gesture & Marking. In Readings in Human Computer Interaction: Toward the Year 2000. San Francisco: Morgan Kaufmann Publishers, Chapter 7, 469--482.Google Scholar
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248--255.Google ScholarCross Ref
- John M. 1 Elliott and K. J Connolly. 1984. A classification of manipulative hand movements. Developmental Medicine & Child Neurology. Vol 26(3) (1984).Google Scholar
- Nicholas Fellion, Thomas Pietrzak, and Audrey Girouard. 2017. FlexStylus: Leveraging Bend Input for Pen Interaction. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 375--385. Google ScholarDigital Library
- George Fitzmaurice, Azam Khan, Robert Pieké, William A. S. Buxton, and Gordon Kurtenbach. 2003. Tracking menus. In Proceedings of the 16th annual ACM symposium on User interface software and technology. ACM, Vancouver, Canada, 71--79. Google ScholarDigital Library
- Clifton Forlines, Daniel Vogel, and Ravin Balakrishnan. 2006. HybridPointing: fluid switching between absolute and relative pointing with a direct input device. In Proceedings of the 19th annual ACM symposium on User interface software and technology. ACM, Montreux, Switzerland, 211--220. Google ScholarDigital Library
- Dustin Freeman, Hrvoje Benko, Meredith Ringel Morris, and Daniel Wigdor. 2009. ShadowGuides: Visualizations for In-situ Learning of Multi-touch and Whole-hand Gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 165--172. Google ScholarDigital Library
- Tovi Grossman, Ken Hinckley, Patrick Baudisch, Maneesh Agrawala, and Ravin Balakrishnan. 2006. Hover widgets: using the tracking state to extend the capabilities of pen-operated devices. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, Montréal, Québec, Canada, 861--870. Google ScholarDigital Library
- Chris Harrison, Robert Xiao, Julia Schwarz, and Scott E. Hudson. 2014. TouchTools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2913--2916. Google ScholarDigital Library
- Ken Hinckley, Patrick Baudisch, Gonzalo Ramos, and Francois Guimbretiere. 2005. Design and Analysis of Delimiters for Selection-action Pen Gesture Phrases in Scriboli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 451--460. Google ScholarDigital Library
- Ken Hinckley, François Guimbretière, Patrick Baudisch, Raman Sarin, Maneesh Agrawala, and Edward Cutrell. 2006. The springboard: multiple modes in one spring-loaded control. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, Montréal, Québec, Canada, 181--190. Google ScholarDigital Library
- Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen+touch = new tools . Uist '10 (2010), 27. Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).Google Scholar
- Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). ACM, New York, NY, USA, 258--264. Google ScholarDigital Library
- Yang Li, Ken Hinckley, Zhiwei Guan, and James a Landay. 2005. Experimental Analysis of Mode Switching Techniques in Pen-based User Interfaces. CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems (2005), 461--470. Google ScholarDigital Library
- Christine L. MacKenzie and T. Iberall. 1994. The Grasping Hand (Advances in Psychology) (1 ed.). North Holland.Google Scholar
- Fabrice Matulic and Moira Norrie. 2012. Empirical Evaluation of Uni- and Bimodal Pen and Touch Interaction Properties on Digital Tabletops. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces (ITS '12). ACM, New York, NY, USA, 143--152. Google ScholarDigital Library
- Fabrice Matulic and Moira C. Norrie. 2013. Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13). ACM, New York, NY, USA, 41--50. Google ScholarDigital Library
- Fabrice Matulic, Daniel Vogel, and Raimund Dachselt. 2017. Hand Contact Shape Recognition for Posture-Based Tabletop Widgets and Interaction. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 3--11. Google ScholarDigital Library
- Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 220--229. Google ScholarDigital Library
- Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML'10). Omnipress, USA, 807--814. http://dl.acm.org/citation.cfm?id=3104322.3104425 Google ScholarDigital Library
- John R. Napier. 1956. The prehensile movements of the human hand. The Journal of Bone and Joint Surgery. British Volume 38-B, 4 (Nov. 1956), 902--913. http://www.ncbi.nlm.nih.gov/pubmed/13376678Google Scholar
- Ken Pfeuffer, Ken Hinckley, Michel Pahud, Bill Buxton, and Interactive Systems. 2017. Thumb+Pen Interaction on Tablets . Acm Chi (2017), 3254--3266. Google ScholarDigital Library
- Gonzalo Ramos, Matthew Boulos, and Ravin Balakrishnan. 2004. Pressure widgets. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, Vienna, Austria, 487--494. Google ScholarDigital Library
- Rosemary Sassoon. 1993. The Art and Science of Handwriting. Intellect Books.Google Scholar
- Dominik Schmidt, Ming Ki Chong, and Hans Gellersen. 2010. HandsDown: Hand-contour-based User Identification for Interactive Surfaces. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI '10). ACM, New York, NY, USA, 432--441. Google ScholarDigital Library
- Julia Schwarz, Robert Xiao, Jennifer Mankoff, Scott E. Hudson, and Chris Harrison. 2014. Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 2009--2012. Google ScholarDigital Library
- Ann-Sofie Selin. 2003. Pencil grip: A descriptive model and four empirical studies . 140 pages.Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).Google Scholar
- Hyunyoung Song, Hrvoje Benko, Francois Guimbretiere, Shahram Izadi, Xiang Cao, and Ken Hinckley. 2011. Grips and gestures on a multi-touch pen. Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11 (2011), 1323. Google ScholarDigital Library
- Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. 15, 1 (Jan. 2014), 1929--1958. http://dl.acm.org/citation.cfm?id=2627435.2670313 Google ScholarDigital Library
- Hemant Bhaskar Surale, Fabrice Matulic, and Daniel Vogel. 2017. Experimental Analysis of Mode Switching Techniques in Touch-based User Interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3267--3280. Google ScholarDigital Library
- Daniel Vogel and Ravin Balakrishnan. 2010. Direct pen interaction with a conventional graphical user interface. Human-Computer Interaction 25, 4 (2010), 324--388.Google ScholarCross Ref
- Daniel Vogel and Gé ry Casiez. 2011. Conté: Multimodal input inspired by an artist's crayon. Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11 c (2011), 357. Google ScholarDigital Library
- Daniel Wigdor, Hrvoje Benko, John Pella, Jarrod Lombardo, and Sarah Williams. 2011. Rock & Rails: Extending Multi-touch Interactions with Shape Gestures to Enable Precise Spatial Manipulations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1581--1590. Google ScholarDigital Library
- Fong-Gong Wu and Shuyi Luo. 2006. Design and evaluation approach for increasing stability and performance of touch pens in screen handwriting tasks. Applied Ergonomics {Kidlington} 37, 3 (2006).Google Scholar
- Mike Wu and Ravin Balakrishnan. 2003. Multi-finger and Whole Hand Gestural Interaction Techniques for Multi-user Tabletop Displays. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). ACM, New York, NY, USA, 193--202. Google ScholarDigital Library
- Mike Wu, Chia Shen, Kathy Ryall, Clifton Forlines, and Ravin Balakrishnan. 2006. Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces. In Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems. IEEE Computer Society, Washington, DC, USA, 185--192. ACM ID: 1110635. Google ScholarDigital Library
- Robert Xiao, Julia Schwarz, and Chris Harrison. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (ITS '15). ACM, New York, NY, USA, 47--50. Google ScholarDigital Library
- Yizhong Xin, Xiaojun Bi, and Xiangshi Ren. 2011. Acquiring and Pointing: An Empirical Study of Pen-tilt-based Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 849--858. Google ScholarDigital Library
- Ka-Ping Yee. 2004. Two-handed interaction on a tablet display. In CHI '04 extended abstracts on Human factors in computing systems. ACM, Vienna, Austria, 1493--1496. Google ScholarDigital Library
Index Terms
- Unimanual Pen+Touch Input Using Variations of Precision Grip Postures
Recommendations
Get a Grip: Evaluating Grip Gestures for VR Input using a Lightweight Pen
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsThe use of Virtual Reality (VR) in applications such as data analysis, artistic creation, and clinical settings requires high precision input. However, the current design of handheld controllers, where wrist rotation is the primary input approach, does ...
Touch+Finger: Extending Touch-based User Interface Capabilities with "Idle" Finger Gestures in the Air
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyIn this paper, we present Touch+Finger, a new interaction technique that augments touch input with multi-finger gestures for rich and expressive interaction. The main idea is that while one finger is engaged in a touch event, a user can leverage the ...
Air+touch: interweaving touch & in-air gestures
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technologyWe present Air+Touch, a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: ...
Comments