1. Physical User Interfaces What they are and how to build them Saul Greenberg University of Calgary

2. Physical User Interfaces
Special purpose computer-controlled devices that can be situated in a real-world setting.
typically designed for particular contexts and uses.

3. Physical User Interfaces
Foundations for
Ubiquitous Computing
Tangible Media
Context aware computing
Styles of use
foreground interaction
ambient displays
information appliance
collaborative interaction
physical controls

4. Ubiquitous Computing Mark Weiser Xerox PARC
A less-traveled path I call the invisible; its highest ideal is to make a computer so imbedded, so fitting, so natural, that we use it without even thinking about it.
Provide hundreds of wireless computing devices per person per office, of all scales … It is invisible, everywhere computing that does not live on a personal device of any sort, but is in the woodwork everywhere.
Mark Weiser Xerox PARC
Source: Mark Weiser’s UbiqCom web site

5. Ubiquitous Computing Invisible Everywhere Computing
tiny, embedded, attachable…
everywhere:
wireless, dynamically configurable, remote access, adapting…
aka Pervasive Computing
Mark Weiser Xerox PARC
Source: Mark Weiser’s UbiqCom web site

6. Ubicomp is Situated Computing
Makes use of simple shared context
space
time
proximity
affordances
Participation in the context
is physical
is out here with us
is in many small and large places, including trivial ones
Source: Mark Weiser’s UbiqCom web site

7. Ubicomp Technology Trends
Source: Mark Weiser’s UbiqCom web site

8. Ubicomp Technology Trends
Processors:
cheap, small, dedicated, microprocessors
Low Power
small batteries, solar (?)
Wireless
802.11, Bluetooth, infrared, mobile telephony, …
Displays
very small (inches) to very large (walls)
Peripherals
sensors, actuators, motors, …
Run-time systems
Linux on a chip, Windows CE, downloadable executables…
Source: Mark Weiser’s UbiqCom web site

9. ParcTab Mobile hardware A portable GUI device infrared
room-sized cells
location information
A portable GUI device
small case with belt clip, ½ size of PDAs
touch sensitive 128x64 pixels display
3 finger-operated mechanical buttons (chorded)
piezo-electric speaker
low power needs (~ 1 week between charges )
can be used in either hand by rotating display
Source: Mark Weiser’s UbiqCom web site

10. Tangible User Interfaces
From ‘painted bits’ to ‘tangible bits’
genre change
give physical form to digital information
physical objects, surfaces, and spaces become tangible embodiments of digital information.
seamlessly couple the dual worlds of bits and atoms
input: grasp and manipulate
output: change of physical properties of object
Hiroshi Ishii MIT
Source: Tangible Media Group web site

11. Context-Aware Computing
Context as information
… characterizes a situation of a person, place or object relevant to the interaction between a user and an application
location
identity
state and activites of people, groups
state of computational and physical objects
Context-aware computing
uses contextual information to
selectively present information and services
automatically execute a service
attach context information for later retrieval
Source: Dey, Abowd and Salber, HCI Journal 2001

12. Styles of use Foreground interactions
Conscious intentional interactions
graspable objects
augmented surfaces
link between physical and virtual actions
exploit human senses of touch and kinesthesia.

13. MusicalBottles - controlling sound, light
Interaction
One opens and closes bottles to control digital contents e.g., opening it tells a story
Music bottles
movement and uncorking of the bottles controls the different sound tracks and the patterns of colored light rear-projected onto the table’s translucent surface.
Source: Tangible Media Group web site

14. MusicalBottles - controlling sound, light
Ishii, MIT
Source: Tangible Media Group web site

15. IAMASCOPE

16. Wooden Mirror - wooden pixels, image
Daniel Rozin, NYU
Ishii, MIT

17. Wooden Mirror - wooden pixels, image
Ishii, MIT
Daniel Rozin, NYU

18. Triangles – building a digital story
When triangles connect together, they trigger digital events. These events influence the progress of a non-linear story, or allow users to organize media elements in order to create their own story space.
Extracted from Tangible Media Group web site

19. Triangles – building a digital story
Extracted from Tangible Media Group web site

20. Ambient displays Background information
communicate digitally-mediated senses of activity and presence at the periphery of human awareness.
ambient light sound, airflow, water movement, object motion…
peripheral displays

21. Extracted from Mark Weiser’s UbiqCom web site

22. Extracted from Mark Weiser’s UbiqCom web site

23. Pinwheels - ambient digital wind
Extracted from Tangible Media Group web site

24. Pinwheels - ambient digital wind
Extracted from Tangible Media Group web site

25. Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka

26. Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka

27. Ambient Light Display
light reflection from water onto ceiling

28. Ambient Room
ACM CHI’98, Tangible Media Group

29. Personal Ambient Display
Small, physical devices worn to display information to a person in a subtle, persistent, and private manner.
Ambient information is displayed solely through tactile modalities such as heating and cooling, movement and vibration, and change of shape.
Extracted from Tangible Media Group web site

30. Physical controls
Physically-mediated computer controlled interactivity

31. Marble Answering Machine
Incoming voice messages are physically instantiated as marbles.
The user can grasp the message (marble) and drop it into an indentation in the machine to play the message.
The user can also place the marble onto an augmented telephone, thus dialing the caller automatically.
Durrell Bishop

32. TouchCounters Computational tags track the usage of physical objects.
TouchCounters sense activity through magnetic, acceleration, and infrared sensors, and indicate their status on bright LED displays.
TouchCounters can be networked to a web server that generates use histograms for each object.
Extracted from Tangible Media Group web site

33. TouchCounters
Extracted from Tangible Media Group web site

34. Tagged Objects
ACM CHI’99, Xerox Parc

35. mediaBlocks
Extracted from Tangible Media Group web site

36. SenseTable
A system for tracking the positions and states of multiple objects wirelessly on a flat surface.
Objects can be equipped with various controls -- dials or buttons -- which can be monitored in real-time.
When coupled with a projector, the system can display information about the objects on or near the objects themselves.
Extracted from Tangible Media Group web site

37. HandScape
A vectorizing digital tape measure for digitizing field measurements, and visualizing the volume of the resulting vectors with computer graphics.
Using embedded orientation-sensing hardware, it captures relevant vectors on each linear measurements and transmits this data wirelessly to a remote computer in real-time.
Extracted from Tangible Media Group web site

38. Manipulative User Interfaces

39. Customizable Physical Interfaces
ACM UIST 2002, Greenberg & Boyle

40. Collaborative interactions
Physical objects connecting collaborators
collaborative interaction
awareness
Adapted from Tangible Media Group web site

41. PingPong Plus
features a "reactive table" that incorporates sensing, sound, and projection technologies. Projectors display patterns of light and shadow on the table; bouncing balls leave images of rippling water; and the rhythm of play drives accompanying music and visuals.
Extracted from Tangible Media Group web site

42. PingPong Plus
Extracted from Tangible Media Group web site

43. InTouch – collaborative haptics
Force-feedback technology is employed to create the illusion that people, separated by distance, are interacting with a shared physical object.
When one of the rollers is rotated, the corresponding roller on the other distant object rotates in the same way. Two people separated by distance can then play…
Extracted from Tangible Media Group web site

44. InTouch – collaborative haptics
Extracted from Tangible Media Group web site

45. Bench …two cold steel benches located in different cities.
When a person sits on one of these benches, a corresponding position on the other bench warms, and a bi-directional sound channel is opened.
At the other location, after feeling the bench for "body heat," another person can decide to make contact by sitting near the warmth.
Initially the sound channel is distorted, but as the second party lingers, the audio channel clears.
--summarized by Ishii and Ullmer
Anthony Dunne and Fiona Raby at the RCA

46. Physical but Digital Surrogates
Rotating figurine servo motor
Tippable figurine light sensors
Proximity detector ultrasonic sensor
Hydra unit video, camera, speakers, microphone
Several years ago, Hideaki Kuzuoka (an engineer) and I built a ubiquitous media space.
It was a reactive environment, and included proximity and light sensors as well as a servo motor to move one of the figurines.
He built the hardware, and I concentrated on the software side.
It tooks several months of full time effort for Hideaki to build this, but we ended up using the system for around 3 months.
After Hideaki returned to Japan, we found the system impossible to extend or maintain.
<animate>

47. Physical but Digital Surrogates
Several years ago, Hideaki Kuzuoka (an engineer) and I built a ubiquitous media space.
It was a reactive environment, and included proximity and light sensors as well as a servo motor to move one of the figurines.
He built the hardware, and I concentrated on the software side.
It tooks several months of full time effort for Hideaki to build this, but we ended up using the system for around 3 months.
After Hideaki returned to Japan, we found the system impossible to extend or maintain.
<animate>

48. Privacy preserving media space

49. Roomware i-land Computer-augmented room elements
like doors, walls, furniture (e.g. tables and chairs) with integrated information and communication technology.
From the GMD Darmstadt web site on I-Land

50. Roomware i-land
Dynawall
From the GMD Darmstadt web site on I-Land

51. Roomware i-land
CommChair
From the GMD Darmstadt web site on I-Land

52. Roomware i-land ConnecTable
By moving multiple ConnecTables together, they can be arranged to form a large display area. Integrated sensors measure the distance between the ConnecTables and initiate the automatic coupling of the displays
From the GMD Darmstadt web site on I-Land

53. Roomware i-land InteracTable
touch-sensitive plasma-display (PDP) is integrated into the table top
Border for leaning…
From the GMD Darmstadt web site on I-Land

54. Roomware i-land

55. Phillips – Intelligent Information Surfaces
Tokens
From the Philips Lime Video CD

56. Phillips – Intelligent Information Surfaces
From the Philips Lime Video CD

57. Designing out of the box
The problem:
programming / designing with physical devices is hard
circuit design (electrical engineering)
microprocessor interface to digital/analog devices
‘wire’ interface (serial, USB, wireless, IR…)
wire protocol
connection/disconnection/intermittent connectivity
software to use devices
maintenance and extensibility
simple things take a long time to do
most people don’t bother

58. Solution 1: Interdisciplinary team
Works, but…
Still takes time
When one of the team leaves, knowledge is lost
systems hard to maintain & extend
Hydra unit video, camera, speakers, microphone
Rotating figurine servo motor
Tippable figurine light sensors
Proximity detector ultrasonic sensor

60. Solution 2: Hack existing devices
Programmable Embodied Agents (Kaminsky et al)
hacked Microsoft Actimates

61. Solution 2: Hack existing devices
Programmable Embodied Agents (Kaminsky et al)
Arm position -> quantity
Squeezing hand/leg -> counting
Movement-> task progress
Proxy for other person
squeeze hand, other’s hand goes up)
Event monitoring
Signal document is printing, then complete
Barney biff

62. Solution 3: Phidgets Physical Widgets
simple, easy to program devices and component-based software with well-defined API
building blocks for physical interfaces
analogous to GUI widgets

63. Phidget Examples PhidgetServo: PhidgetPowerbar PhidgetInterfaceKit
Control 1 or 2 servo motors
PhidgetPowerbar
Control power state of outlets on a power bar
PhidgetInterfaceKit
8 simple input and outputs plus 2 sensors
A constructor kit
PhidgetProximitySensor
Returns how close something is to it
PhidgetMotionDetector
Periodically returns the amount of motion in a space

64. Related areas Mobile Computing Augmented Reality
Context-aware computing
Reactive Environments
Ubiquitous Media
Cooperative Buildings

65. Digital Inputs - switches
Rocker
Toggle
Push button
Push-Pull
Rotary
Slide
Tactile
Keylock

66. Analog inputs: Environmental Sensors
Motion
Distance
Light
Temperature
Pressure
rangefinder
proximity
Voltage
Weight
Distance

67. Analog inputs: Input Sensors
Force
Mini-joystick
Capacitive
Accelerometer
single-turn
multi-turn
encoder
slider
Potentiometers

68. Analog inputs: Input Sensors
Bend
Force
Camera
Tilt

69. Custom Input: Identification
RFID Tags and Antenna
Bar Code Scanner

70. Custom inputs
Camera

71. Digital Outputs – low power
lamps
LEDs
Lights
Relays
Solenoids

72. Output: Motors Servo Stepper DC Motor Position: 0-180o
Rotate by steps: +/- xo
Stepper
Speed
DC Motor

73. Output: Displays
Graphics (not yet)
Text LCD
Numeric-alpha

74. Approaches: PIC Micro-controller
Single programmable chip computer with:
CPU, RAM, ROM, I/O, serial/parallel ports, A/D and D/A converters
Need to know:
basic circuit design
basic electronics
resistor, capacitor, diodes, transistors…
low level programming
PIC hardware details …
Powerful, flexible, but
learning curve
excessive time in low level details
complex things are hard
serial
e.g.

75. Approaches: Basic Stamp
Pre-built boards
Pic microcontroler
pre-wired circuits and connectors
boards designed for different uses
Still need to know
electronic components
electronic circuitry
PBasic language: stamp-specific instruction set
Still flexible, but
learning curve still there
time in low level details
complex things are hard
serial

76. Approaches: HW as SW components
Phidgets and Making Things
dedicated devices
some plug and play electronics
under direct computer control
well-defined component-based software
Interface via APIs, objects, and/or widgets
Need to know
conceptual knowledge of device
high level programming language
software component documentation
High design flexibility, low electronic flexibility
very low learning curve
design by combining and varying
time in conceptual design, not electronics

77. Approaches: Programmable Brick
Lego Mindstorms
robotics (downloadable code)
computer control as well
proprietary RCX microcontroller
infrared communication
3 input ports
light/touch sensors…
3 output ports
motors, lights…
Lego building blocks
children’s programming language for downloading but
well-defined SDK
3rd party access from standard languages
Need to know
SDK / language
High design potential for robotics
limited input/output (3+3)
expensive for basic set, plus add-ons
lego/images/grabberarm1.jpg
e.g.
e.g.