1. TAUCHI – Tampere Unit for Computer-Human Interaction Adaptive Software Button: Blind Interaction Techniques for Touchscreens Georgios Yfantidis Georgios.Yfantidis@uta.fi Project for the course New Interaction Techniques Fall 2003

2. TAUCHI – Tampere Unit for Computer-Human Interaction Introduction :1 New interaction techniques and usability research engage in constant experimentation, to provide solutions for effective computer based applications. Especial interest is for techniques concerning physically challenged user groups, more test cases are being developed and evaluated

3. TAUCHI – Tampere Unit for Computer-Human Interaction Introduction :2 Touch screen can take advantage of gesture recognition. Applications can be useful both in large terminals and in personal device assistants that have limited size of displays. Blind people have several difficulties in learning how to use keyboards. Blind people have difficulties pointing with the use of mouse or other devices. Blind people need constant feedback in the form of markers (audio, tactile, etc) to comprehend system’s functionality and state. 1

4. TAUCHI – Tampere Unit for Computer-Human Interaction The main objectives Create a new technique for gesture interaction based on software button metaphor. The button has to be adaptive. Simple text entry functionality should be implemented. Simplify the interaction for blind people. - only one button - appears anywhere they click, no need to search Facilitate the interaction for blind people by recognizing simple finger gestures as input mode.

5. TAUCHI – Tampere Unit for Computer-Human Interaction What is an adaptive button? By an adaptive button we mean a virtual button (as opposed to physical keys), which resembles a normal button in a wide sense but has a variable function :“caption’s layout”. The layout of the button changes accordingly, during the interaction. In our project the button properties should be adaptive regarding position and functionality.

6. TAUCHI – Tampere Unit for Computer-Human Interaction Why use an adaptive button? In general, the button is useful for blind people, or in situations where the user can use only one finger to enter text. The advantages of an adaptive button include - Flexibility since the button appears where the user touches. - Easy navigation with auditory feedback, so the user doesn’t need to look at the screen in order to complete the text entry. - Simplicity: Only one button with the ability to manipulate with only one finger (or stylus) which leaves a free hand to the user.

7. TAUCHI – Tampere Unit for Computer-Human Interaction Which letters do the layouts contain? The first layout contains the most commonly used letters in English language. These letters are E R A S T I O N The second layout includes characters that are used less frequently C L P D H F G M The third layout contains characters that are more rarely used Q U B K X V W Y English letter frequencies a b c d e f g h i j k l m n o p q r s t u v w x y z 73 9 30 44 130 28 16 35 74 2 3 35 25 78 74 27 3 77 63 93 27 13 16 5 19 1

8. TAUCHI – Tampere Unit for Computer-Human Interaction Succession and layout of layers E N I O RA T S C M F G LP D H Q Y V W UB K X LAYER 1 LAYER 3 LAYER 2 Time interval

9. TAUCHI – Tampere Unit for Computer-Human Interaction Software : Main structure The initial screen is comprised of a text box and a square shaped adaptive button. Button has 3 different layouts. One layout appears at a time. Each layout contains 8 characters in an array that follows the basic arrow directions, up down left right and the intermediate positions between them The layouts change cyclically in time.

10. TAUCHI – Tampere Unit for Computer-Human Interaction Software : Main structure 2 Characters are entered by moving the finger (or stylus) in their direction and lifting the finger (or stylus). While user does this movement there is sound feedback informative of the character that the user is about to select. Since there are three layouts with 8 letters the total number of letters and characters that can be entered in this way are only 24.

11. TAUCHI – Tampere Unit for Computer-Human Interaction Change of characters After pointing towards some characters and waiting, there is substitution of character with another character, symbol or special function. The user hears a sound signal that indicates the change. That means that if the user performs a “mouse up” action the character of the initial layout will be the substituted by new character.

12. TAUCHI – Tampere Unit for Computer-Human Interaction Character substitutions: 1rst Layer E N I O RA T S Becomes “space” after dwelling Enters “Next Line” after dwelling

13. TAUCHI – Tampere Unit for Computer-Human Interaction Character substitutions: 2nd Layer C M F G LP D H Becomes “backspace” function after dwelling Becomes coma “,” after dwelling Becomes “J” after dwelling Becomes point “.” after dwelling

14. TAUCHI – Tampere Unit for Computer-Human Interaction Character substitutions: 3d Layer Q Y V W UB K X Becomes question mark“?” after dwelling Changes between Up case/low case after dwelling Becomes “z”after dwelling

15. TAUCHI – Tampere Unit for Computer-Human Interaction Layer edit feature There is a special feature in the software that initiates an editing mode of the layers. User has the chance to change the order of the letters in the layers according to his needs or his preferences. By double clicking within the textual box user can call editing mode and alter the layout of the layers by removing, repositioning and adding new letters and characters. The new layer can be saved in order for the user to have a customized interaction with the button.

16. TAUCHI – Tampere Unit for Computer-Human Interaction Manipulation through time In this software the dimension of time plays an important role that guides the interaction. The adaptation of the button is happening by changing position and functionality, but the main factor that determines how this adaptation and interchange is happening, is time. Layers change cyclically according to time. There is a certain interval between the layers which in future versions of the software could be configurable or adaptive. Dwelling time after a movement (an action of the user) determines which characters within the layers are going to be selected (in case there is a character with a substitution feature).

17. TAUCHI – Tampere Unit for Computer-Human Interaction Movements and gestures The user can move his finger in 8 different directions. Each direction returns a sound feedback, and the user can perform an input entry by lifting the finger (or stylus) after pointing. In future versions of the software more gesture recognition could be applied in order for extra functionality to be added. An example could be to have a different feedback if the finger was backtracking in the same path. It could perhaps enter a second character if the user moved towards the director of the symbols and then returned to the center. Another option which was realized is cancel.

18. TAUCHI – Tampere Unit for Computer-Human Interaction Future investigation Future investigation could deal with the following matters Implementing a built-in statistical data monitor feature that could use the results to gain insight into the data and deal with usability issues. How to make the interchange of the layers more effective (perhaps some adaptive feature concerning the timer that would change the time interval according to user performance) Further investigating input techniques and how gestures can be more intuitive and perhaps fast according to layout and character selection techniques. Possibilities for multiple character entering by this method: e.g., according to dwelling time user could be presented with more options for succeeding letters based on prediction or statistics, then user could choose this option based on certain pointing upping or clicking with finger

19. TAUCHI – Tampere Unit for Computer-Human Interaction Conclusions The project in this initial phase seemed to fulfill most of the basic functionality and features an adaptive button for a touch screen could have. Gestures and time manipulation were tried out in a basic level and that revealed that single button manipulation by one finger could be really used as an alternative to traditional text entry methods (e.g., full-keyboard emulation unavailable for blind text entry on touchscreen without special matrixes) This method of text entry, can be used by blind people because it allows them to hit almost anywhere in the screen presented with this virtual button.. Sound feedback during the process provides a good and understandable guidance that makes the use of a display unnecessary.

20. TAUCHI – Tampere Unit for Computer-Human Interaction References Vanderheiden, G., Law, C. Ergonomics of a a non-visual touchscreen interface: a case study. Trace R&D Center, University of Wisconsin-Madison Vanderheiden. G. Use of audio-haptic interface techniques to allow nonvisual access to touchscreen appliances. Trace R&D Center, University of Wisconsin-Madison Evreinov, G. and Raisamo, R. Information Kiosks for All: Issues of Tactile Access. 2002 Interaction Design Guide for Touchscreen Applications, http://www.sapdesignguild.org/resources/TSDesignGL/Index.htm http://www.sapdesignguild.org/resources/TSDesignGL/Index.htm Lesher, G.W., Moulton, B.J., Higginbotham, D.J. Techniques for augmenting scanning communication. 1998 Dunlop, H., Jones, M., Cunningham, S.J. A digital library of conversational expressions: a communication aid for people with profound physical disabilities. Dept. of Computer Science University of Waikato Gnatenko, V. Multidirectional input keypad TM- text input solution for mobile devices, www.vitgn.com www.vitgn.com Zhai, S., Smith, B.A. Alphabetically Biased Virtual Keyboards Are Easier to Use– Layout Does Matter. IBM Almaden Research Center, San Jose, CA, USA Wobbrock, J.O., Myers, B.A., Hudson, S.E. Exploring Edge-Based Input Techniques for Handheld Text Entry. HCI Institute, School of Computer Science, Carnegie Mellon University Geißler, J. Gedrics: the next generation of icons. German National Research Center for Computer Science (GMD)