HCI The Computer

Interacting with computers To understand Human-Computer Interaction … need to understand computers! what goes in and out of computer devices, paper, sensors, etc what can the computer do? memory, processing, etc

HCI challenge Traditional I/O techniques may not be suitable Sensory/motor requirements may not be present in the disabled user New interaction methods and techniques are required

Von Neumann Architecture

Timeline Vacuum Tube Triode (1908) Transistor 1947 (silicon, germanium) Very Large Scale Integrated (VLSI) Circuits; 1970s-- (> 1,000 transistors per chip) Intel Itanium (Tukwila) 2008: 2 billion transistors

The Computer A computer system is made up of various elements Each of these elements affects the interaction input devices – text entry and pointing output devices – screen (small&large), digital paper virtual reality – special interaction and display devices physical interaction – e.g. sound, haptic, bio-sensing paper – as output (print) and input (scan) memory and processing– memory, access, speed of processing, networks

How many computers … in your house? in your pockets? PC DVD, HiFi, TV microwave, cooker, washing machine, fridge Air Condition security system can you think of more? in your pockets? PDA smartphone digital camera smart card electronic car key USB memory try your pockets and bags

Richer interaction sensors and devices everywhere

text entry devices keyboards (QWERTY et al.) chord keyboards, phone pads handwriting, speech

Input (1) Batch Large amounts Pre-prepared information Standardised data loaded quickly using automated routines Used a lot in the past (punch cards)

Keyboards Most common text input device Allows rapid entry of text by experienced users Keypress closes connection, causing a character code to be sent Usually connected by cable, but can be wireless

Keyboards QWERTY Keyboard (1878) the official standard of computer keyboards [ISO9995] was designed for two finger typing prevents stuck keys (on a typewriter) the position of the keys the hands are held close together helping to cause Carpal Tunnel Syndrome can be used without training is cheap and easy to find Any other keyboard?

Keyboards (ctd) - layout Alphabetic layout keys arranged in alphabetic order not faster for trained typists not faster for beginners either! Dvorak layout common letters under dominant fingers biased towards right hand common combinations of letters alternate between hands 10-15% improvement in speed and reduction in fatigue But no significant adoption, when compared to QWERTY

Keyboards Over the years, attempts were made to physically improve keyboards … Tilting Curving Splitting

Keyboards (ctd) - special keyboards Maltron left-handed keyboard Virtual keyboard Projection keyboard

Chord keyboards only a few keys - four or 5 letters typed as combination of keypresses compact size – ideal for portable applications short learning time – keypresses reflect letter shape fast – once you have trained BUT - social resistance, plus fatigue after extended use NEW – niche market for some wearables

Keyboards Chord Keyboards Don’t use an alphabet Use 4/5 keys Rely on combination of keys Ideal for one hand or mobile devices Can suffer fatigue after use New application in mobile devices?

Phone keypad and T9 entry use numeric keys with multiple presses 2 – a b c 6 - m n o 3 - d e f 7 - p q r s 4 - g h i 8 - t u v 5 - j k l 9 - w x y z hello = 4433555[pause]555666 surprisingly fast! T9 predictive entry type as if single key for each letter use dictionary to guess the right word hello = 43556 … ‘hello’ not ‘gdjjm’ note 26 could mean ‘am’ or ‘an’; phone gives those options

Soft or on-screen keyboards Keys are arranged in rows and columns Operated by eye tracking scanning input methods Text entry rate 0.5-5 wpm 6-8 wpm with rate enhancement techniques (prediction, ambiguity, abbreviation expansion)

Our Focus- Scanning Keyboards Soft keyboards operated with scanning input methods Q W E R T Y U I O P A S D F G H J K L Enter Z X C V B N M __ <-- Shift

Performance Models The RG model by Rosen and Goodenough- Trepagnier (1981) Based on three components L -- average no of language units per word A -- average no of motor acts required to input each language unit T – average time required to carry out each motor act TW = average time to compose an word = L*A*T

Input Devices

Skinput: Body as an Input Surface CHI’2010 Use a tiny projector on body to show menus Microphones to listen to taps on hand/arm Signal processing and machine learning to differentiate positions

Handwriting recognition Text can be input into the computer, using a pen and a digesting tablet natural interaction Technical problems: capturing all useful information (path, pressure, etc) in a natural manner segmenting joined up writing into individual letters coping with different styles of handwriting etc Used in PDAs, tablet computers, smart white boards … … leave the keyboard on the desk!

Speech recognition Most successful when: single user – initial training and learns peculiarities limited vocabulary systems Improving, but challenges such as: external noise interfering imprecision of pronunciation/articulation e.g. vagueness, pauses, clearing throat, repetition, etc large vocabularies different speakers

positioning, pointing and drawing mouse, touchpad trackballs, joysticks etc. touch screens, tablets eyegaze, cursors

Mouse Since the mouse sits on a desk Disadvantage Movement of mouse moves screen cursor mouse movement in (x, z) plane … Since the mouse sits on a desk moving it about is easy cursor is small and can thus be manipulated without obscuring the display Disadvantage hand-eye coordination problems for novice users

Mouse (ctd) Two methods for detecting motion Mechanical Optical ball on underside of mouse turns/rotates as mouse is moved can be used on almost any flat surface Optical light emitting diode on underside of mouse detects fluctuating alterations in reflected light intensity to calculate relative motion less susceptible to dust and dirt

Footmouse Motivation Relevant Challenges foot pedals are used heavily in musical instruments (pianos, organs, drums) and in mechanical equipment (cars, cranes, sewing machines) Relevant hands-occupied situations users with disabilities or with high-back or neck problems Challenges feet tend to be less precise than hands practicality of having foot controls in the work environment (e.g. pedals under a desk may be operated accidentally, etc)

Touchpad a pointing device featuring a specialized surface that can translate the motion and position of a user's fingers to a relative position on screen used mainly in laptop computers good ‘acceleration’ settings important

Trackball and thumbwheels ball is rotated inside static housing like an upside down mouse! separate buttons for picking/selection very fast for gaming used in some portable and notebook computers Thumbwheels two dials for X-Y cursor position for fast scrolling – single dial on mouse

keyboard nipple Keyboard nipple or pointing stick a pointing device typically mounted in a keyboard on laptops pointing stick operates by sensing applied force the velocity of the pointer depends on the applied force force applied on the pointing stick is echoed on the screen by movements of the pointer

Joystick

Joystick Joystick pressure of stick translates to velocity of movement buttons for selection e.g. on top or on front like a trigger often used for computer games, aircraft controls and 3D navigation

Haptic devices - example BMW iDrive for selecting and controlling menus one feels little bumps along the way as one maneuvers through options and menus makes it easier to select options by feel uses haptic technology from Immersion Corp

Idrive controller The iDrive system gives the driver complete control over many of the vehicle’s functions while allowing him to concentrate on the road.

Touch screen

Touch-sensitive screen Detect the presence of finger or stylus on the screen. mobile devices e.g. smartphones and tablets more natural gestures through multitouch (i.e. ability to recognize the presence of two or more points of contact with the screen surface, and therefore supporting more natural gestures such as pinching and reverse pinching)

Pointing Devices Integrated gaze and face tracking system. A camera and IR light source tracks gaze by computing the angle between the corneal reflection and the centroid of the pupil.

Discrete positioning controls in phones, TV controls etc mini-joysticks (mobile phones) central cursor pads & discrete left-right, up- down buttons (TV remote controls) e.g. +/- for controlling volume & navigating channels mainly for menu selection

Pointing Devices

Smart Systems

virtual reality Virtual reality is an artificial environment that is created with software and presented to the user in such a way that the user suspends belief and accepts it as a real environment. On a computer, virtual reality is primarily experienced through two of the five senses: sight and sound.

Positioning and viewing in 3D Positioning in 3D space 3D mouse: six-degrees of movement: x, y, z + roll, pitch, yaw data glove: to detect finger position VR helmets: to detect head motion and possibly eye gaze whole body tracking: accelerometers, image/video processing, etc Viewing in 3D often use stereoscopic vision ( two 2D images are presented separately to the left and right eye of the viewer, which are then combined in the brain to give the perception of depth), but also possible to use real 3D display (displaying an image in three full dimensions)

Simulators and VR caves scenes projected on walls realistic environment real controls other people

bitmap screens (CRT & LCD) large & situated displays digital paper display devices bitmap screens (CRT & LCD) large & situated displays digital paper

Display

Output Devices Permanent output Impermanent output Paper Microfiche Film Impermanent output Screen Sound Tactile

Bitmap displays bitmap display or raster scan the whole screen is scanned sequentially and horizontally screen is vast number of coloured dots

Visual Output Cathode Ray Tube (CRT) Liquid Crystal Display (LCD) Text based to high resolution Produce minor health hazards Since they are radiation emitting devices! Liquid Crystal Display (LCD) Alternative systems might not use a screen … Lights Gauges Dials Can you give examples?

Large and situated displays Large displays Used for meetings, lectures, etc Where very large screen areas are required, several smaller screens, either LCD or CRT, can be placed together in a video wall where each screen displays a portion of the whole (or separate sub-image) and the result is an enormous image Situated displays As well as for lectures and meetings, display screens can be used in various public places to display information; they could even be interactive

paper: printing and scanning print technology fonts, page description, WYSIWYG scanning, OCR

Permanent Output On paper using printers Impact Printers (Noisy) Dot Matrix (Slow + Low Quality) Daisy Wheel (Very Slow + 1 Font) Chain and Band printers (Fast + Expensive) Non-Impact Printers (Silent, High Quality, Fast) Inkjet Thermal (Expensive) Laser (Cost effective + Colour expensive)

Permanent Output Printers Speed measured in Pages Per Minutes Resolution measured in Dots Per Inch (dpi) 80 dpi (dot matrix) = Near Letter Quality 300 dpi – 1200 dpi (non-impact) = Near Book Quality Store information in firmware such as Typeface (Fonts) Character set Size More complex printers use Page Description Language (PDL) Communicates complex commands directly to printer Used in pdf and postscript

Fonts and this is twenty-four point Font – the particular style of text Courier font Helvetica font Palatino font Times Roman font §´µº¿Â Ä¿~  (special symbol) Size of a font This is ten point Helvetica This is twelve point This is fourteen point This is eighteen point and this is twenty-four point

Fonts (ctd) Pitch Serif or Sans-serif fixed-pitch – every character has the same width e.g. Courier variable-pitched – some characters wider e.g. compare the “i” and the “m” in Times New Roman Serif or Sans-serif sans-serif – square-ended strokes e.g. Helvetica serif – with splayed ends (such as) e.g. Times Roman

Readability of text lowercase UPPERCASE All lowercase text considered to be easier to read than all uppercase UPPERCASE better for individual letters and non- words e.g. flight numbers: BA793 vs. ba793

Scanners Take paper and convert it into a digital document Useful in desktop publishing for incorporating photographs and other images document storage and retrieval systems, instead of physical paper storage

Optical character recognition OCR converts bitmap back into text different fonts create problems for simple “template matching” algorithms more complex systems segment text, decompose it into lines and arcs, and decipher characters that way page format columns, pictures, headers and footers

physical controls, sensors etc. special displays and gauges sound, touch, feel, smell physical controls environmental and bio-sensing

Sounds can take many forms used for beeps bongs (deep ringing sounds) clonks (loud thudding sounds) whistles whirrs (like the sound of rapidly vibrating wings) used for error indication (e.g. incorrect command/input) or risky situations (e.g. when deleting files) confirmation of actions e.g. keyclick notification of events/status e.g. a new email has arrived, download has been completed

multi-function control Physical controls specialist controls needed … industrial controls, consumer products, etc. easy-clean smooth buttons multi-function control large buttons clear dials tiny buttons

Environment and bio-sensing sensors all around us car courtesy light small switch on door ultrasound detectors security, washbasins RFID security tags in shops temperature, weight, location

MediaCup Han's purpose to drink coffee cup has sensors heat, movement, pressure broadcasts state (IR) used for awareness user is moving, drinking, … Han's purpose to drink coffee incidentally colleagues are aware

OTHER ASPECTS

Computer Memory Short Term Long Term RAM Secondary Backing Storage Hard disk Disk drive CD’s Memory Sticks

Permanent Storage Cheaper Slower Permanent (Non-volatile) Several types Magnetic Optical Different capacities Floppies 1.4 Mb Memory Stick 64 Mb + CD 640 Mb DVD 4 Gb – 18 Gb Blue Ray 15 Gb – 30 Gb Harddisk 120 Gb + … Tera Bytes Two writing types Rewritable (Hard disk) Write Once Read Many WORM (CD, punch cards)

What computers just cannot do.

M.C. Escher Print Gallery

How do you represent music? Score: Audio samples Spectrogram Q: How do we represent music? Show: score vs. audio file Ask aloud “is score a faithful representation?” Q: What is the relationship between the score representation and what you hear? Explain spectrum - Show adding waves in time Adding waves in frequency Show spectogram - 440, 880, both. (with sound)

Creating music: Synthesis Examples: Stevie Wonder Tina Turner (FM) Kanye West/ Daft Punk (vocoder)

Review Questions What are the most common interaction devices? What are their limitations? How can some of them be overcome by novel devices? Suggest ideas for an interface which uses speech and sound effectively. How can sound be used to convey information?

Review Questions What are some of the factors you might consider when designing systems like a public access interactive kiosk? Identify the strengths and weaknesses of multimedia and hypermedia systems. Discuss how the speed of access to memory can affect the speed of an interactive computer system. Which other factors can affect such speed?

Revision