PinchPad

Abstract

• Interactions for discrete, continuous and combined gesture command without hand-eye control
• PinchPad: 2 iPads back-to-back
• Gestures performed on back of grasped device

Introduction

• Recurring problem: grasping hand hides part of the visual content on-screen
• Study of human-artifact interaction, like newspapers, for transfer to grasped devices

Configuration of grasped devices

• B: Discrete commands (skipping forward/backward)
• C: Continuous commands (scrolling pages)
• D: Combined commands (modifying a value)

 Applying body schema to interaction design

(proprioception = sense of positioning and action of own body parts)

• Body schema: 
• what are fingers doing and where are they positioned, even when hidden
• internal, globally consistent model of a multisensory representation of 
• spatial positioning and ownership of human body parts
• construction of peripersonal space (immediately around the body)
• Users weight the use of vision and propriosception according to situational requirements

 Related work

• [25] Widgor et al: exploration of bimanual input on a double sided touch table
• [24] Widgor et al: Lucid Touch: prototype with back-side touch detection through camera
• [12] Kim: grip pattern recognition
• [27] Wimmer: handSense, for classifying graps while holding a phone size device
• [3] Essl et al: mobile device prototype with front and back interaction
• [8 & 15] Holman & Schwesig: grasp-based user input
• [9] Holz: touch performance and guidance feedback
• [26] Wimmer: situation sensitive parameters when grasping objects
• [28] Wobbrock: performance of certain fingers for touch-based interactions on front and back
• [16] Shen: double-sided multi touch with a see-through vision of finger position

 Design

• Guidance feedback (in this context)
• any perceivable information changes that result from human-computer interaction and which help to make the interaction more transparent to the user
• position of the fingers in relation to the interface that gathers the gestural data
• movements around and touch-based actions on the grasped device

Interaction design

• Used finger movements
• tap & drag gestures (identified as most practical [29])
• thumb movements (more degrees of freedom for movement)
• Thumb as point of reference
• for guidance feedback
• more comfortable and accurate (assumption)
• pinch-through action: push on opposite side of  a rested fingertip
• Interaction techniques
• Discrete (A&B): targeting rested fingers on the back side with the thumb
• Continuous (C): point to one finger with the thumb and drag to another
• Combined (D): circling fingers with the thumb

 Experimental work

 Measurements

•  Parameters
• finger addressability
• front and back screen mapping
• positional pointing accuracy (local finger closeness to thumb)
• pointing precision (finger-thumb offset)
• temporal pointing accuracy (only initial pinching)
• distance to middle (only circle pinching)
•  start-end distance (only slide pinching)

 PinchPad

• 2 stacked iPads back-to-back
• multitouch tracking using TUIO protocol

 Procedure

•  Test
• 4 interaction techniques
• with all fingers
• each hand seperately
• all repeated 5 times
•  Sample instructions
• initial pinch: pinch device between each finger and the thumb for a second
  (both thumb and finger start from a floating state, hence the possible time difference)
• rested pinch: point thumb to each of the fingers
  (the fingers are in a rested state from the start, so already touching the screen)
• circled pinch: draw a circle with the thumb around each finger
• slided pinch: slide the thumb from one finger to another and back, for each combination
•  Questionnaires: demographic data, experience at manual tasks

Participants

• 10 between 29 and 64 years old
• 6 female, 4 male
• all right handed, mostly experienced in manual and computer skills

 Results

 Pinch-through metaphor

• In all of the tasks, all participants were consistently able to locate their occluded fingers correctly when they were asked to move or point at them independently with the thumb

Pinch-through performance

• Initial pinching
• poor positional performance
• high time accuracy (synchronized touch from both thumb and finger)
• Errors were produced by 3 subjects, all others performed perfectly
•  Other gestures
• Over 84% accuracy
• Errors include offsets towards the hand palm, scaled circles and arcs
• No overall patterns
• No differences between dominant & non-dominant hand
• Performance differences between subjects 

Perceived performance and frustration

• Marginally significant difference in rated performance between gestures
• Overall low level of frustration for all gestures
• General sense of limited performance due to the form factor of the device
• weight, shape, depth
•  Task confidence varied widely: comfortable to  'blind and clumsy'
• All subjects felt unable to rate their own accuracy (users were to told if gestures were successful or not)

Discussion

• Gestures that start with a rested state, lack accuracy
• Users seem to be better at moving finger simultaneously than statically positioning them with a device in between
• Increase accuracy of other gestures
• Using initial pinching?
• Error patterns?
• User based offsets?
• Pro
• design of situation-aware interfaces
• more dynamic
•  Con
• lower accuracy
• harder analysis
• calibration due to differences in grasp

Conclusion

• Users can perform gestures without seeing them
• with high precision: pointing thumb at hidden fingers
• with high time accuracy: tapping thumb and finger simultaneously
• feedback from the own body (haptic & proprioception) can guide invisible gestures adequately, especially with an embodied reference
• own body feedback cannot replace end-of-gesture feedback

References

•  Double sided touch
• Essl, G., Rohs, M., Kratz, S. 2009. Squeezing the Sandwich: A Mobile Pressure-Sensitive Two-Sided Multi-Touch Prototype, In Proc. UIST 2009, Demo.
• Shen, E. E., Tsai, S. D., Chu, H., Hsu, Y. J., Chen, C. E. 2009. Double-side multi-touch input for mobile devices. In Proc. CHI 2009,4339-4344.
• Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., Shen, C. 2007. LucidTouch: A See-Through Mobile Device, In Proc. UIST 2007, 269-278.
• Wobbrock, J. O. et al. The performance of hand postures in front-and back-of-device interaction for mobile computing, International Journal of Human-Computer Studies, Volume 66, Issue 12, December 2008, 857-875.
• Device grasp
• Kim, K. et al. 2006. Hand Grip Pattern Recognition for Mobile User Interfaces, American Association for Artificial Intelligence 2006, 1789-1794.
• Wimmer, R. 2011. Grasp Sensing for Human-Computer Interaction, In Proc.TEI 2011, 221-228.
• Wimmer, R., Boring, S. HandSense - Discriminating Different Ways of Grasping and Holding a Tangible User Interface, In Proc. TEI 2009, 359-362.
• Invisible gestures
• Van Beers, R. J., Wolpert, D. M., Haggart, P. 2002. When Feeling Is More Important Than Seeing in Sensory Adaption, Current Biology, Vol. 12, May 14, 2002, 834–837.
• Wigdor, D., Leigh1, D., Forlines, C.,Samuel Shipman, S., John Barnwell, J., Balakrishnan, R., Shen, C.Under the table interaction. In Proc. UIST 2006, 259-268.