1. History of HCI Key systems, people and ideas Matthias Rauterberg
Technical University Eindhoven (TU/e)
The Netherlands

2. History of Computer Technology
Digital computer grounded in ideas from 1700’s & 1800’s
Computer technology became available in the 1940’s and 1950’s
see further: History of Computing History of HCI
© M. Rauterberg, TU/e

3. Konrad Zuse ( )
In 1936 Konrad Zuse, absolutely set apart from the academic world, started constructing an automatic machine to solve calculation problems for designing plane wings; these analyses forcing him to long and repetitive calculations. In his living-room and with the mere aid of few tools, he first produced a binary mechanical memory, to which he soon connected a mechanical calculation unit as well as a programming unit controlled from old movie films punched by hand. He called this model V1 (Versuchsmodell 1), but subsequently changed this name into Z1 in order not to confuse it with the flying bombs having the same name! Having become aware of the poor liability and slowness of this machine, in 1939, Zuse prepared a second one, called Z2, characterized from a still mechanical memory but with a relay-operated electromechanical calculating unit.
In the following years, Zuse accomplished a real electromechanical working computer, Z3, which was submitted in 1941 to an audience of engineers and scientists, raising great interest. Not yet satisfied, living in a Berlin continuously bombed and in which it was difficult to find even food, Zuse constructed Z4 with a mechanical memory (relays were now unfindable): the machine was ended in Zuse arrived in protecting Z4 from the destructions of war and from the hands of Allies, hiding it in a cellar in the small Bavarian village of Hindelang. Once tranquillity returned again, Zuse transferred in to the Swiss Federal Institute of Technology (ETH), in Zurich, were it remained working for 15 years. Up to 1951, this machine remained the only working computer in continental Europe.
The history of Zuse is emblematic for at least five reasons: 1) his contribution was completely original, as he was very isolated from the rest of the world but even from the German research activity; 2) the fact that he has conceived a binary representation of figures which is that adopted from all the modern computers; 3) the fact that he has independently achieved an architecture which was already suggested by Babbage; the invention of the first programming language (Plankalkul, ); 5) the extremely practical and simple way of facing the problem: the estimated cost of Z2 is US $, only.
© M. Rauterberg, TU/e

4. Konrad Zuse in front of his reconstructed Z3
Zuse’s Z3 was the world’s first reliable working machine for very complicated arithmetic calculations, which was freely programmable and was based on a binary floating point number and switching system.
Konrad Zuse in front of his reconstructed Z3
© M. Rauterberg, TU/e

5. Eniac (1943)
A general view of the ENIAC, the first all electronic numerical integrator and computer in USA.
From IBM Archives.
© M. Rauterberg, TU/e

6. The Mark I paper tape readers.
© M. Rauterberg, TU/e
From Harvard University Cruft Photo Laboratory.

7. von Neuman Architecture (1946)
Instructions and data are stored in the same memory for which there is a single link (the von Neumann bottleneck) to the CPU which decodes and executes instructions.
The CPU can have multiple functional units.
The memory access can be enhanced by use of caches made from faster memory to allow greater bandwidth and lower latency.
Johann (John) von Neumann ( )
J. Presper Eckert Jr. and John Mauchly were the first to develop the von Neuman architecture. John von Neumann wrote "First Draft of a Report to the EDVAC" describing the ideas of a stored memory computer. The complicated story is described in the wonder history of computers "Engines of the Mind" by Joel Shurkin.
© M. Rauterberg, TU/e

8. Princeton Architecture (1946)
A course on computer design based on the von Neuman concept was run by the Moore School in 1946, and was attended by British and American professionals. Also in 1946, von Neuman, Herman Goldstine and Arthur Burks published a comprehensive report of their work at Princeton's Institute of Advanced Study Electronic Computer Project where they had established themselves following their involvement with the Moore School. The report detailed the operation and architecture of their work on digital computers, and has been described as the blue print for ‘modern' digital computing.
© M. Rauterberg, TU/e

9. Stretch (1961) A close-up of the Stretch technical control panel
Mainframe Computers
Stretch (1961) A close-up of the Stretch technical control panel
IBM SSEC (1948)
© M. Rauterberg, TU/e
From IBM Archives.

10. Enabling Technology
1904 Sir John Ambrose Fleming invents the vacuum tube and diode.
1947 the first Transistor was invented by Bardeen, Brattain and Shockley in the Bell Labs in the USA.
1958 Jack Kilby invented the integrated circuit at Texas Instruments and got the Noble Prize. Comprised of only a transistor and other components on a slice of germanium, Kilby's invention, 7/16-by-1/16-inches in size, revolutionized the electronics industry.
Intel's 4004 IC chip, generally acknowledged as the world's first "microcomputer on a chip," was originally designed in for the Busicom (Japan) 141-PF desktop calculator.
© M. Rauterberg, TU/e

11. Historical overview 1983 Apple Lisa 1984 Apple Macintosh
1945 Memex
1969 Flex
1973 Alto
1974 Bravo
1974 IBM portable computer
1983 Apple Lisa
1984 Apple Macintosh
1987 MicroSoft Windows
© M. Rauterberg, TU/e

12. Vannevar Bush ( )
“As We May Think” Atlantic Monthly “…publication has been extended far beyond our present ability to make real use of the record.”
© M. Rauterberg, TU/e

13. Vannevar Bush (1945) Postulated Memex device
Can store all records/articles/communications
Large memory
Items retrieved by indexing, keywords, cross references
Can make a trail of links through material
etc.
Envisioned as microfilm, not computer
© M. Rauterberg, TU/e

14. Memex design sketch (1945)
© M. Rauterberg, TU/e

15. J.R. Licklider (1915-1990) 1960 - Postulated “man-computer symbiosis”
Couple human brains and computing machines tightly to revolutionize information handling “The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today.”
An MIT psychoacoustrician named J.C.R. Licklider took and immediate and intense interest in computer after Clark demonstrated the TX-0 to him. Licklider applied his background in psychology to research how people interacted with computers, and he became known as an expert in human-computer interaction. People at ARPA took notice and offered Licklider the job of director for their new Information Processing Techniques Office (IPTO). He accepted the position as the founding director and continued his research in human-computer interaction.
© M. Rauterberg, TU/e

16. Vision/Goals (1945-1995) Immediate Intermediate Long-term Time sharing
Electronic I/O
Interactive, real- time system
Large scale information storage and retrieval
Combined speech recognition, character recognition, light- pen editing
Natural language understanding
Speech recognition of arbitrary users
Heuristic programming
© M. Rauterberg, TU/e

17. Mid 1960’s Computers too expensive for individuals -> timesharing
increased accessibility
interactive systems, not jobs
text processing, editing
, shared file system
© M. Rauterberg, TU/e

18. DEC PDP-1 (1961)
As the world's first commercial interactive computer, the PDP-1 was used by its purchasers to pioneer timesharing systems, making it possible for smaller businesses and laboratories to have access to much more computing power than ever before.
© DEC Inc.
© M. Rauterberg, TU/e

19. Ivan Sutherland (1938-) SketchPad: 1963 PhD thesis at MIT
Hierarchy - pictures & subpictures
Master picture with instances
Constraints
Icons
Copying
Light pen as input device
Recursive operations
© M. Rauterberg, TU/e

20. Ivan Sutherland using the console of the TX-2 at MIT
Sketchpad (1963)
Input device: Light pen used on cathode ray tube.
Graphical objects could be drawn and modified through constraints.
Object oriented modell.
Ivan Sutherland using the console of the TX-2 at MIT
Copy and paste.
© MIT Lincoln Lab
© M. Rauterberg, TU/e

21. Douglas C. Engelbart (1925 - )
Engelbart invented the mouse at Stanford Research Labs in 1964.
Landmark system/demo:
hierarchical hypertext, multimedia, mouse, high-resolution display, windows, shared files, electronic messaging, CSCW, teleconferencing, …
Augment/NLS system [NLS: oN Line System]
© M. Rauterberg, TU/e

22. The First Mouse (1964) Knee control Douglas Engelbart
Years before personal computers and desktop information processing became commonplace or even practicable, Douglas Engelbart had invented a number of interactive, user-friendly information access systems that we take for granted today: the computer mouse, windows, shared-screen teleconferencing, hypermedia, groupware, and more.
© M. Rauterberg, TU/e

23. Augment/NLS (1968) Augment/NLS Features:
2-dimensional display text editting, by two persons from different consoles, at the same time.
Links.
Video-conferencing.
Mouse.
© M. Rauterberg, TU/e

24. Alan C. Kay (1940-)
Dynabook - Notebook sized computer loaded with multimedia and can store everything
Personal Computing
Desktop Interface
The FLEX software
© M. Rauterberg, TU/e

25. FLEX & Dynabook (1969)
Computer should function like a living organism.
Kay‘s Ph.D was about FLEX, an early object orientated language.
First idea of a book-sized Computer.
Use of graphic rather than text.
Alan Kay developed the Dynabook at Xerox PARC.
© Xerox PARC
© M. Rauterberg, TU/e

26. Theodor (Ted) H. Nelson (1937-)
Computers can help people, not just business
Coined term “hypertext”
Ted Nelson originally invented the word "hypertext" for "non- sequential writing". His long-standing interest in all things related to HT became the Xanadu project. The Xanadu Operating Company was owned for a while by Autodesk, but later dropped.
© M. Rauterberg, TU/e

27. Nicholas Negroponte (ca 1938-)
MIT machine architecture & AI group ( s)
Ideas:
wall-sized displays, video disks
AI in interfaces (agents), speech recognition, multimedia with hypertext
© MIT MediaLab, Boston
© M. Rauterberg, TU/e

28. Personal Computers (PC)
Late ‘70’s
Apple II
Z-80 CP/M
IBM PC
Text and command based
Word processing
Spreadsheets
© M. Rauterberg, TU/e

29. Input/output devices Input Output
Early days connecting wires lights on display
paper tape & punch cards paper
keyboard teletype
Past keyboard scrolling glass teletype
+ cursor keys character terminal
+ mouse bit-mapped screen
+ microphone audio
Today data gloves + suits head-mounted displays
computer jewelry ubiquitous computing
natural language autonomous agents
© M. Rauterberg, TU/e

30. IBM (1974)
Mark-8
The Mark-8 was an Intel 8008 based machine with 256 bytes RAM. It was introduced in July 1974, and were produced. It was the first portable computer to really be marketed, and had no ROM. The market value of a Mark-8 was a round $12, The machine pictured right was the precursor to the IBM 5100 machine. It was introduced in 1975, and was very costly. It was IBM's first entry into the microcomputer market.
© IBM
© M. Rauterberg, TU/e

31. From http://www.blinkenlights.com/pc.shtml
IBM (1975)
IBM 5100
IBM 5100, introduced in September , was IBM's first portable computer.
The 5100 was just one of several portable computers IBM made before the Personal Computer (PC).
The 5100 model was followed by the , the 5120, the Datamaster, and then finally the 5150 PC.
© IBM
From
© M. Rauterberg, TU/e

32. From http://www.blinkenlights.com/pc.shtml
IBM (1981)
IBM PC
Product shot of IBM Personal Computer (5150), introduced in 1981;
Features: monitor, keyboard, and pin-feed printer; b/w. The operating system (OS) was by Microsoft, who licensed it to IBM as PC-DOS.
Although not necessarily the best machine by technological standards, IBM's expertise and the fact that the IBM PC actually looks and feels like a professional computer system made the IBM PC and the numerous PC clones extremely popular. They have evolved into today's so-called Wintel (Windows + Intel) computer systems, used world-wide.
© IBM
From
© M. Rauterberg, TU/e

33. DOS (history)
1980, April: Tim Patterson begins writing an operating system for use with Seattle Computer Products' 8086-based computer. Seattle Computer Products decides to make their own disk operating system (DOS), due to delays by Digital Research in releasing a CP/M-86 OS , August: QDOS 0.10 (Quick and Dirty OS) is shipped by Seattle Computer Products. Even though it had been created in only two man-months, the DOS worked surprisingly well , September: Tim Patterson shows Microsoft his 86-DOS, written for the 8086 chip , October: Microsoft's Paul Allen contacts Seattle Computer Products' Tim Patterson, asking for the rights to sell SCP's DOS to an unnamed client (IBM). Microsoft pays less than US$100,000 for the right , December: Seattle Computer Products renames QDOS to 86-DOS, releasing it as version 0.3. Microsoft then bought non-exclusive rights to market 86-DOS.
1981, February: MS-DOS 1.0 runs for the first time on IBM's prototype microcomputer , July: Microsoft buys all rights to DOS from Seattle Computer Products, and the name MS-DOS is adopted , August: IBM announces the IBM 5150 PC Personal Computer, featuring a 4.77-MHz Intel 8088 CPU, 64KB RAM, 40KB ROM, one 5.25-inch floppy drive, and PC-DOS 1.0 (Microsoft's MS-DOS), for US$3000.
1982, May: Microsoft releases MS-DOS 1.1 to IBM, for the IBM PC. It supports 320KB double-sided floppy disk drives. Microsoft also releases MS-DOS 1.25, similar to 1.1 but for IBM-compatible computers.
1988, June: Microsoft releases MS-DOS 4.0, including a graphical/mouse interface.
© M. Rauterberg, TU/e

34. Bill Gates (1955-)
William (Bill) H. Gates is chairman and chief software architect of Microsoft Corporation, the worldwide leader in software, services and Internet technologies for personal and business computing. Microsoft had revenues of $25.3 billion for the fiscal year ending June 2001, and employs more than 40,000 people in 60 countries. In his junior year, Gates left Harvard to devote his energies to Microsoft, a company he had begun in 1975 with his childhood friend Paul Allen. Guided by a belief that the computer would be a valuable tool on every office desktop and in every home, they began developing software for personal computers. Gates' foresight and his vision for personal computing have been central to the success of Microsoft and the software industry.
Under Gates' leadership, Microsoft's mission has been to continually advance and improve software technology, and to make it easier, more cost-effective and more enjoyable for people to use computers. The company is committed to a long-term view, reflected in its investment of more than $4 billion on research and development in the current fiscal year 2001.
© M. Rauterberg, TU/e
© source

35. MS DOS (1981)
Microsoft DOS (Disk Operating System) is a command line user interface.
Microsoft releases MS-DOS 1.0 to IBM, for the original IBM PC in 1981.
In 1982 MS-DOS 1.1 supports 320KB double-sided floppy disk drives.
Microsoft also releases MS-DOS 1.25, similar to 1.1 but for IBM-compatible computers with 720 KB floppy disk drives.
© Microsoft Inc.
© M. Rauterberg, TU/e

36. PCs with GUIs Xerox PARC - mid 1970’s invention of the first PC: ALTO
Local processor, Bitmap display, Mouse
Precursor to modern GUI
LAN - Ethernet
© M. Rauterberg, TU/e

37. Xerox Alto (precursor to the Star)
Alto applications:
Bravo WYSIWYG text editor.
BravoX Mesa implementation of Bravo, an ancestor of Microsoft Word.
Laurel Electronic mail program.
Neptune Disk file manipulation program, sort of like sweep.
Works like the Font DA mover program on a Macintosh.
Press Document printing program.
Sil Drawing program.
© M. Rauterberg, TU/e
From Xerox Alto Archive

38. Xerox Star - ‘81
First commercial PC designed for “business professionals”
Desktop metaphor, pointing, WYSIWYG
First system based on usability engineering
© M. Rauterberg, TU/e

39. Xerox Star (1981)
© M. Rauterberg, TU/e

40. Xerox Star (GUI)
© M. Rauterberg, TU/e

41. Xerox Star (history) Commercial flop $15k cost closed architecture
lacking key functionality (spreadsheet)
© M. Rauterberg, TU/e

42. Apple II (1977)
Built in 1977, the Apple II was based on Wozniak's Apple I design, but with several additions. The first was the design of a plastic case--a rarity at the time-- which was painted beige. The second was the ability to display color graphics--a holy grail in the industry. The Apple II also included a larger ROM, more expandable RAM (4K to start), and 8 expansion slots. It had integer BASIC hard-coded on the ROM for easier programming, and included two game paddles and a demo cassette for $1,298. In early 1978 Apple also released a disk drive for the machine, one of the most inexpensive available. The Apple II remained on the Apple product list until 1980.
Steve Wozniak (1950-)
© Apple Inc.
1979, February Apple Computer releases DOS , July Apple Computer releases DOS 3.2.1
© M. Rauterberg, TU/e

43. Steve Jobs (1955-) co-founder Apple Computer Corporation
Apple Lisa - ‘82
Based on ideas of Star
More personal rather than office tool
Still expensive!
Conceptual success, but commercial failure
Steve Jobs (1955-) co-founder Apple Computer Corporation
© M. Rauterberg, TU/e

44. Apple Lisa (1983)
Named for one of its designer's daughters, the Lisa (pictured left) was supposed to be the Next Big Thing. It was the first personal computer to use a Graphical User Interface (GUI). Aimed mainly at large businesses, Apple said the Lisa would increase productivity by making computers easier to work with. The Lisa had a Motorola Processor running at 5 Mhz, 1 MB of RAM two " 871k floppy drives, an external 5 MB hard drive, and a built in 12" 720 x 360 monochrome monitor.
© Apple Inc.
© M. Rauterberg, TU/e

45. Apple Lisa (applications)
LisaWrite: word processor
LisaCalc: spread sheet
LisaGraph: charts
LisaList: an outline builder, idea manager
LisaProject: project scheduler
LisaDraw: drawing program (predecessor to Mac Draw)
LisaTerminal: modem communications software.
© Apple Inc.
© M. Rauterberg, TU/e

46. Apple Macintosh - ‘84 Aggressive pricing - $2500
Not trailblazer, smart copier
Good interface guidelines
3rd party applications
High quality graphics and laser printer
© M. Rauterberg, TU/e

47. Apple Macintosh (1984)
Released in January of 1984, the Macintosh was the first affordable computer to include a Graphical User Interface (GUI). It was built around the new Motorola chip, which was significantly faster than previous processors, running at 8 MHz. The Mac came in a small beige case with a black and white monitor built in. It came with a keyboard and mouse, and had a floppy drive that took 400k 3.5" disks--the first personal computer to do so.
© Apple Inc.
The early Mac team members (1979) consisted of Jeff Raskin, Brian Howard, Marc LeBrun, Burrell Smith, Joanna Hoffman and Bud Tribble.
© M. Rauterberg, TU/e

48. Apple Macintosh (GUI)
© Apple Inc.
© M. Rauterberg, TU/e

49. Apple Macintosh (history)
Problem of the Mac: Only 128k main memory
Well written applications: MacWrite and MacDraw
Mac 512k, Mac512ke and Mac Plus were introduced to save the Mac
New applications: Pagemaker, Word Excel
© M. Rauterberg, TU/e

50. MS Windows (1987)
Release of the IBM PC AT in August 1984 running at 6MHz
Windows planed 1983 was released on August 11, 1987
Windows 1.01 was a large disappointment!
© Microsoft Inc.
© M. Rauterberg, TU/e

51. Microsoft Windows (history)
Steve Jobs from Apple Inc. complained about the stolen Mac OS’s interface design.
Bill Gates from Microsoft Inc. replied: “Hey Steve, just because you broke into Xerox’s house before I did and took the TV doesn’t mean I can’t go in later and take the stereo.”
Microsoft Windows 2.03 was released January 1988.
Microsoft Windows 3.1 was released in 1992.
© M. Rauterberg, TU/e

52. Oberon at ETH (1985)
Oberon is the name of a modern integrated software environment. It is a single-user, multi-tasking system that runs on bare hardware or on top of a host operating system. Oberon is also the name of a programming language in the Pascal/Modula tradition. The Oberon project was launched in 1985 by Niklaus Wirth and Jürg Gutknecht at the ETH in Zurich. Although the project was originally targeted towards in-house hardware, the language and system have now been ported to many computer platforms. Oberon is the first worldwide modeless software system: completely modeless graphical user interface with support for graphical primitives; both overlapping and tiled windowing systems supported concurrently; possibility of configuring the working environment.
Niklaus Wirth (1934-)
Jürg Gutknecht
© M. Rauterberg, TU/e

53. Ben Shneiderman (1947-)
Dr. Shneiderman is the author of ‘Software Psychology: Human Factors in Computer and Information Systems’ (1980) in which he coined the term direct manipulation. Later he wrote the influential text book ‘Designing the User Interface: Strategies for Effective Human-Computer Interaction’ (1987, third edition 1998).
Ben Shneiderman has written over 200 articles and published several books, including Elements of FORTRAN Style: Techniques for Effective Programming (with Charles Kreitzberg, 1972); and Hypertext Hands-On! An Introduction to a New Way of Organizing and Accessing Information (with Greg Kearsley, 1989). He has also edited numerous articles and several books, including Directions in Human/Computer Interaction (1982) and Sparks of Innovation in Human-Computer Interaction (1993).
Ben Shneiderman was a Professor in the Department of Computer Science, Founding Director ( ) of the Human-Computer Interaction Laboratory, and Member of the Institutes for Advanced Computer Studies and for Systems Research, all at the University of Maryland at College Park.
© M. Rauterberg, TU/e

54. Historical Overview (1945-1995)
[source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp ]
© M. Rauterberg, TU/e

55. Mobile Phone (1954) Some important years in the mobile history,
Growing up in Chicago, Martin Cooper earned a degree in electrical engineering at the Illinois Institute of Technology. Later hired by Motorola in 1954, he lead a group of research team to develop the first ever portable phone, the Motorola Dyna Tac, which stands for Dynamic Adaptive Total Coverage. Weighing 1089 gram, the first commercially viable version of the Dyna Tac was released in It measures 9 x 5 x 1.75 inches in size with 30 circuit boards. Unlike handsets today, there was no display available and the only feature was call, dial and listen (what else would you expect?). The heavy batteries can only withstand 35 minutes of talk time and need a long 10 hour recharge.
Martin Cooper
Some important years in the mobile history,
1955 introducing the worlds first whole automatic mobile-phone system.
1972 A global system is presented. Covers all the oceans of the world.
1978 introducing the worlds first person searching-system with a number-display.
1981 The world’s first automatic and boundless mobile-phone system.
1986 First time when you can transfer computer services via a mobile system.
1988 The pocket-phone is introduced.
Dyna Tac
© M. Rauterberg, TU/e

56. What is Virtual Reality (VR)?
In 1989, Jaron Lanier, CEO of VPL, coined the term virtual reality to bring all of the virtual projects under a single rubric. The term therefore typically refers to three- dimensional realities implemented with stereo viewing goggles and reality gloves.
Myron Krueger (1991):
….The term (virtual worlds) typically refers to three-dimensional realities implemented with stereo viewing goggles and reality gloves.
George Coates (1992):
Virtual Reality is electronic simulations of environments experienced via head mounted eye goggles and wired clothing enabling the end user to interact in realistic three-dimensional situations.
Myron Krueger
P. Greenbaum (1992):
Virtual Reality is an alternate world filled with computer-generated images that respond to human movements. These simulated environments are usually visited with the aid of an expensive data suit which features stereophonic video goggles and fiber-optic gloves.
© M. Rauterberg, TU/e

57. Dimensions to define VR
Vividness
(richness of an environments representation)
• breadth (visibility, audibility, touch, smell)
• depth (quality, fidelity)
Interactivity
(extend to which a user can modify form and content of a mediated environment)
• speed (update rates, time lag)
• mapping (text, speech, gestures, gaze, complex behavior patterns)
© M. Rauterberg, TU/e

58. Classification of VR and other Media
From Jonathan Steuer
© M. Rauterberg, TU/e

59. History of VR (technological milestones)
1956 Sensorama (Morton Heilig)
3D visuals, vibration, stereo sound, wind, smell, little interaction.
1961 Headsight System (Philco Corp.)
Head Mounted Display (HMD), head tracking, remote video camera,
telepresence.
1965 The Ultimate Display (Ivan Sutherland)
Stereoscopic HMD, computer generated images, tracking, visually
coupled system.
1967 Grope (University of North Carolina)
6 degree of freedom force feedback.
1977 The Sayre Glove (Sandin, Sayre, DeFanti Univ. Illinois)
Gesture recognition.
1987 Virtual Cockpit (British Aerospace)
head and hand tracking, eye tracking, 3d visuals, 3D audio,
speech recognition, vibro tactile feedback.
© M. Rauterberg, TU/e

60. Morton Heilig
In the late '50s, a quiet man named Morton Heilig began designing the first multisensory virtual experiences. He developed something called the Sensorama. Resembling one of today's arcade machines, the Sensorama combined projected film, audio, vibration, wind, and even prepackaged odors, all designed to make the users feel as if they were actually in the film rather than simply watching it. Since real-time computer graphics were many years away, the entire experience was prerecorded, and played back for the user.
Although he was a gifted and visionary inventor, Heilig was less successful as a businessman. He was unable to get funding for his Sensorama machines, and they were never manufactured. Fortunately, he didn't give up there;
Heilig had an idea that would later prove to be the basis for an entire industry: the first Head-Mounted Display (HMD), which he patented in 1962.
© M. Rauterberg, TU/e

61. Sensorama (1956)
Sensorama is a simulator that gives one person at a time an illusion of reality. It is a semi-portable automatic machine that can be plugged in anywhere electricity is available. The illusion of reality is achieved by providing the viewer with a wide range of sensory information. All of this information is perfectly synchronized with the picture (aromas, wind, and vibrations change instantly). All the control information is on one piece of film. Sensorama is completely automatic. The viewer activates it by depositing a coin or pushing a button (depending on application).
© M. Rauterberg, TU/e

62. Sutherland & Sproull (Harvard, 1967)
Headsight System (1961)
(© nVision)
first HMD (1961)
Sutherland & Sproull (Harvard, 1967)
Datavisor 80
© M. Rauterberg, TU/e

63. © NASA Ames Research Center
Ultimate Display (1965)
Ivan Sutherland is a pioneer in the field of computer graphics and in 1965 he described 'The Ultimate Display', which included interactive graphics and force-feedback devices.
In 1968, he described a prototype virtual reality system in his paper 'A head-mounted three- dimensional display'.
But it was a team at the NASA Ames Research Center who really opened up the possibilities of virtual reality worlds with their Virtual Interface Environmental Workstation (VIEW), developed during the 1980s as a training system for future astronauts.
© NASA Ames Research Center
© M. Rauterberg, TU/e

64. Grope (1967) Haptic Display Grope III
The Force-Feedback Project, which began in 1967, first focused on the development of a system to support scientific visualization in the area of molecular docking, the Docker application. This application provides graphic (wire-frame) representations of molecules and their inter-atomic forces to allow a user to adjust the relative position and orientation of molecules while searching for minimum energy binding sites.
A series of systems have been developed, evolving from a 2-D system, through a 3-D system and a 6-D system for a simple docking task, to a full 6-D molecular docking system called GROPE-III. These later systems have employed a modified Model E-3 Argonne Remote Manipulator (ARM).
© Fred Brooks, University of North Carolina
© M. Rauterberg, TU/e

65. Sayre Glove (1977)
Early contributions to computer graphics included performances with real-time graphics accompanied by music and the use of its hardware and software for creating the computer animation of the Death Star "schematic" for the first Star Wars.
In 1976, based on an idea by colleague Rich Sayre, DeFanti and Sandin developed an inexpensive, lightweight glove to monitor hand movements; the Sayre Glove provided an effective method for multidimensional control, such as mimicking a set of sliders.
Projects in the 1970s through mid-1980s centered on video game technology, real- time computer animation on microcomputers, and interactive multimedia installations.
© University of Illinois at Chicago
© M. Rauterberg, TU/e

66. Digital Data Entry Glove (1985)
Digital Data Entry Glove, T. G. Zimmerman & Y. Harvill
first glove-like device (cloth) onto which numerous touch, bend, and inertial sensors were sewn.
measured finger flexure, hand-orientation and wrist-position, and had tactile sensors at fingertips.
orientation of hand tracked by video camera; required clear line-of-sight observation for the glove to function.
designed as alternative to keyboard; matched recognized gestures/hand orientations to specific characters, specifically to recognize the Single Hand Manual Alphabet for the American Deaf; circuitry hard-wired to recognize 80 unique combinations of sensor readings to output a subset of the 96 printable ASCII characters; a tool to “finger-spell” words.
finger flex sensors, tactile sensors at the fingertips, orientation sensing and wrist-positioning sensors; positions of sensors were changeable.
US Patent 4,542,291: Zimmerman & Harvill, Optical Flex Sensor, September 17, 1985 [ACM CHI paper 1987]
© VPL Research, Inc.
© M. Rauterberg, TU/e

67. VIEW (ca.1985)
The VIEW configuration included a head-mounted display, head and hand tracking, speech recognition, three- dimensional audio output, and a tracked and instrumented glove.
The glove was the interface through which the user could interact with the virtual world. A graphical representation of the glove moved around the virtual world in response to the user's hand movements. The glove had fibreoptics embedded in it and these detected changes in finger positions, while a separate motion sensor detected the position of the hand.
The computer recalculated the coordinates of the glove's image based its movements
© NASA Aerospace Lab.
© M. Rauterberg, TU/e

68. © British Aerospace Lab.
Virtual Cockpit (1987)
© British Aerospace Lab.
© M. Rauterberg, TU/e

69. Input Devices (overview)
Sensor Devices
1. Spatial Position/Orientation Sensors
• 2DOF (Mouse)
• 3DOF (Microscribe, FreeD Joystick)
• 6DOF (Polhemus Fastrack)
2. Directional Force Sensors
• 5 DOF (Spacemouse)
• 2 DOF (Joystick)
3. Gesture Recognition
• Data Gloves
4. Eye Tracking
5. Speech Recognition Systems
© M. Rauterberg, TU/e

70. Directional Force Sensors
Input Devices (1)
Directional Force Sensors
SpaceMaster
© M. Rauterberg, TU/e

71. Input Devices (2) Gesture Recognition Dextrous Hand Master, Exos
SUPERGLOVE, Nissho
Cyberglove , 5th Dimension
© M. Rauterberg, TU/e

72. Spatial Position/Orientation Sensors
Input Devices (3)
Spatial Position/Orientation Sensors
Polhemus InsideTrack
(Magnetic Tracking)
MicroScribe
(Mechanical Tracking)
FreeD Joystick
(UltraSonic Tracking)
© M. Rauterberg, TU/e

73. Visual Haptic Workbench
Input Devices (4)
Visual Haptic Workbench
The Visual Haptic Workbench consists of five hardware components.
The dominant hand of the user experiences haptic feedback from the PHANToM, and the subdominant hand navigates through a menu interface via Pinch glove contact gestures. Head tracking is done with a Polhemus Fastrak receiver mounted on a pair of Stereographics CrystalEyes LCD shutter glasses. The subdominant hand can also be tracked with a separate receiver to facilitate more complex interaction paradigms. The audio subsystem gives the user additional reinforcement cues to clarify
[see also
© M. Rauterberg, TU/e

74. Output Devices (1)
NASA HMD (ca 1985)
© M. Rauterberg, TU/e

75. An Immersive VR Environment
Output Devices (2)
Four-sided CAVE (1992)
An Immersive VR Environment
© Electronic Visualization Laboratory
University of Illinois at Chicago
© M. Rauterberg, TU/e

76. The KTH Six-sided CAVE built by TAN GmbH.
Output Devices (3)
Six-sided CAVE (1998)
The KTH Six-sided CAVE built by TAN GmbH.
In October 1998 TAN finished the world's first 6-sided TAN VR-CUBE™ for the Royal Institute PDC/KTH in Stockholm/Sweden. This type of VR-CUBE™ complete encloses the user from all sides.
© KTH, Stockholm
© M. Rauterberg, TU/e

77. © VR-Systems, United Kingdom
Output Devices (4)
CyberSphere (1998)
The scientists Eyre and Eureka in VR-Systems UK have been researching a CyberSphere, a device, which consists of a large, translucent sphere containing the user.
The images are distortion-corrected and then projected on the surface of the sphere, allowing the user a full 360 degree field of view.
It also allows the user to move around in the world, by walking inside the ball, which will move in response to the users movements.
© VR-Systems, United Kingdom
© M. Rauterberg, TU/e

78. Ubiquitous Computing (1991)
Ubiquitous computing is just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives.
Mark Weiser is the father of ubiquitous computing (1991).
[Mark Weiser, "The Computer for the Twenty-First Century”, Scientific American, pp , Sept. 1991]
Mark Weiser ( )
What Ubiquitous Computing Isn't
Ubiquitous computing is roughly the opposite of virtual reality. Where virtual reality puts people inside a computer-generated world, ubiquitous computing forces the computer to live out here in the world with people. Virtual reality is primarily a horse power problem; ubiquitous computing is a very difficult integration of human factors, computer science, engineering, and social sciences.
© M. Rauterberg, TU/e

79. Apple’s Newton (1992)
Apple Computer's Newton line of personal digital assistants (PDA) began as CEO John Scully's pet project in Since then, seven different Newton models were released before "iCEO" Steve Jobs came back to Apple: in 1998 he quickly axed the newly formed subsidiary responsible for the Newton line, Newton, Inc.
A small, hand-held device with pen-input, personal organizational functions, and communication capabilities becomes the first shipped Newton product by Apple Inc.
Although intelligent software is an essential element, this product is viewed as a smaller, cheaper, stripped-down version of the initial Newton concept that can be mass produced.
© Apple Computing Inc.
© M. Rauterberg, TU/e

80. PARCtab (1993)
The PARCtab is most easily operated with two hands: one to hold the tab, the other to use a passive stylus or a finger to touch the screen.
But since office workers often seem to have their hands full, we designed the tab so that three mechanical buttons fall beneath the fingers of the same hand that holds the tab, allowing one-handed use.
The device also includes a piezo-electric speaker so that applications can generate audio feedback
© Xerox PARC
© M. Rauterberg, TU/e

81. Unistroke alphabet
Techniques for handwriting recognition have improved in recent years, and are used on some PDAs for text entry.
But they are still far from ideal since they respond differently to the unique writing characteristics of each operator.
Xerox PARC have experimented on the PARCTAB with Unistrokes, which depart from the traditional approach in that they require the user to learn a new alphabet---one designed specifically to make handwriting easier to recognize
© Xerox PARC
© M. Rauterberg, TU/e

82. © Matias, MacKenzie & Buxton, Toronto
Half-QWERTY (1993)
Typing With One Hand Using Your Two-handed Skills
Half-QWERTY is a one-handed typing technique, designed to facilitate the transfer of two- handed typing skill to the one- handed condition
© Matias, MacKenzie & Buxton, Toronto
Bill Buxton
© M. Rauterberg, TU/e

83. PalmPilot (1996) The PalmPilot has a lot functionality.
This device fits with its pocket size into one hand.
There is a communication channel via IR to the PC.
Small, and a reasonable price
© Palm Computing Inc.
© M. Rauterberg, TU/e

84. PalmPilot alphabet Input similar to “natural” alphabet
not user specific
minimize the user’s learning and adapting costs
© Palm Computing Inc.
© M. Rauterberg, TU/e

85. Newton MessagePad 2x00 (1997)
The final incarnation of the Newton OS came packaged in the MessagePad 2100, an internet- ready, heap-increased version of the MessagePad The 161.9Mhz StrongArm processor, which Palm is just beginning to port PalmOS to in 2002, is the true reason the Newton 2X00 line is still one of the more functional PDAs of these days.
When the MP2K was first released in 1997, unlike the PalmPilot, they were meant to work to laptop functionality rather than act as a digitized datebook. This means that you can browse the Web in full grayscale (and perhaps Javascipt), number crunch on spreadsheets, use graphing calculators, wordsmith in a functional word processor, and so on. Movies have been made to run on Newtons and Mp3 syncing with iTunes has been implemented.
© Apple and Newton, Inc.
© M. Rauterberg, TU/e

86. Brian Shackel (ca 1920-)
Brian Shackel is Emeritus Professor of Ergonomics at Loughborough University and the founder of the HUSAT Research Institute. He is one of the pioneers in applying human factors and ergonomics to computer systems. After graduating from Cambridge and following military service, he set up EMI’s human factors group and worked on many systems and product user interfaces. He joined Loughborough University of Technology and set up HUSAT in He has been an advocate international standards committees and publishing widely. He is the founder and first chairman of the IFIP Technical Committee “Human-Computer Interaction” in 1989.
The prestigious BRIAN SHACKEL AWARD is associated with each IFIP TC13 INTERACT Conference, usually biennial, and is to recognise the most outstanding contribution in the form of a refereed paper submitted to and delivered at the INTERACT conference.
“From early days (cf Licklider & Clark, 1962) the need for larger displays has been emphasised; but just when it seemed, in the late s, that full page and larger displays would come with lower prices, the focus in the industry turned to portability and we moved backward to smaller screens. While there was some improvement, larger screens (eg 21 inch CRT displays) are still not available at an acceptable price; as long ago as 1977 Kay & Goldberg (1977) in their Dynabook concept specified a display the size of a full paper page, but I know of no portable laptop, let alone notebook, which has an A4 page size screen.” (Brian Shackel, 2000)
© M. Rauterberg, TU/e

87. Tim Berners-Lee (1955-) BORN June 8, 1955, in London
1976 Graduates from Queen's College, Oxford
1980 While at CERN, writes "Enquire"
1989 Proposes global hypertext project called "WorldWideWeb"
1991 The Web debuts on the Internet
1993 University of Illinois releases Mosaic browser
1994 Joins M.I.T. to direct the W3 consortium
1999 Today nearly 150 million people log on to the Internet via WWW
Tim Berners-Lee is considered to be the founder of the World Wide Web.
“First of all, let's get clear the difference. The internet is a collection of computers, which was put together during the 1970's. When I proposed the Web in 1989, the internet had been around for 15 years. You could use , you could store files on ftp servers, and people could access them, but it was very complicated. The web was the step to make accessing a remote document just one click. The internet spread really quite slowly. It started in research, moved into universities, and many people only heard about it when the web became available as an easy way to use it. “ (T. Berners-Lee, 1999)
© M. Rauterberg, TU/e

88. World Wide Web (1990)
In Europe, researchers at CERN (the European Laboratory for Particle Physics) were struggling with their own computer networking problems. Throughout the system people used different techniques, protocols, and equipment, making communication between computers very complex. In 1980, Tim Berners-Lee, a consultant at CERN, wrote a program called "Enquire-Within-Upon-Everything," enabling links to be made between any point in the system. Nine years later Berners-Lee wrote "Information Management: A Proposal:" Instead of standardizing the equipment or software, they created standards for data, and a universal addressing system. That way any document on the Internet could be retrieved and viewed. In 1990, CERN was the largest Internet site in Europe. Over the next year or two, the proposal was circulated and revised, resulting in an initial program being developed that was dubbed the World Wide Web. At least one expert has called the Web a "side effect of CERN's scientific agenda." In 1992, the World Wide Web was demonstrated and distributed, and browser software was released throughout and beyond CERN. That November there were about 26 reliable Web servers.
All you needed to use the Web was a browser. The early browsers were functional but not especially "user-friendly." A young programmer at the National Center for Supercomputing Applications (NCSA) named Marc Andreesen created a new graphical Web browser. This was pleasing to the eye and easy to use -- just point-and-click. Users didn't need to know any programming or even any Internet addresses. It also made it fairly simple for users to add their own material to the Web. Andreesen and his coworkers called this browser Mosaic, and released free versions for Windows and Macintosh in August of Interest in the Web -- especially commercial interest -- exploded with the arrival of Mosaic. By October there were more than 200 Web servers, and at the end of 1993, Mosaic was being downloaded from NCSA at a rate of 1,000 copies per day. By June 1994, there were 1,500 Web servers.
In July 1993, there were 1,776,000 hosts in 26,000 domains; by July 1996, there were 12,881,000 hosts in 488,000 domains. In July , there were 3,054 Internet service providers and projections of Web user sessions rising to billion in the year 2000.
© M. Rauterberg, TU/e

89. Historical Overview: Robots
1818 Mary Shelley, Frankenstein Joseph Atterly, A Voyage to the Moon Jules Verne, A Journey to the Center of the Earth Edward S. Ellis, The Steam Man of the Prairies Jules Verne, 20,000 Leagues Under the Sea H.G. Wells, The Time Machine Karel Capek, R.U.R. (first use of the word "robot")
The term "robot" was first used in a play called "R.U.R." or "Rossum's Universal Robots" by the Czech writer Karel Capek.
Science fiction writer Isaac Asimov first used the word "robotics" to describe the technology of robots and predicted the rise of a powerful robot industry.
"Cybernetics", an influence on artificial intelligence research was published by Norbert Wiener.
George Devol and Joseph Engelberger formed the world's first robot company.
Computer-assisted manufacturingg was demonstrated at the Servomechanisms Lab at MIT.
The first industrial robot was online in a General Motors automobile factory in New Jersey. It was called UNIMATE.
The first artificial robotic arm to be controlled by a computer was designed. The Rancho Arm was designed as a tool for the handicapped and it's six joints gave it the flexibility of a human arm
© M. Rauterberg, TU/e

90. UNIMATE (1961)
UNIMATE, the first industrial robot, began work at General Motors. Obeying step-by-step commands stored on a magnetic drum, the 4,000-pound arm sequenced and stacked hot pieces of die- cast metal. The brainchild of Joe Engelberger and George Devol, UNIMATE originally automated the manufacture of TV picture tubes.
© M. Rauterberg, TU/e

91. Wabot-1 (1973)
The WABOT-1 was the first fun-scale anthropomorphic robot developed in the world at Tokyo's Waseda University under Ichiro Kato.
It consisted of a limb-control system, a vision system and a conversation system. The WABOT-1 was able to communicate- with a person in Japanese and to measure distances and directions to the objects using external receptors, artificial ears and eyes, and an artificial mouth.
The WABOT-1 walked with his lower limbs and was able to grip and transport objects with hands that used tactile-sensors. It was estimated that the WABOT-1 has the mental faculty of a one-and-half-year-old child. WABOT-1 consisted of the WAM- 4 (as its artificial hands) and the WL-5 (Its artificial legs).
Ichiro Kato
WABOT-1
© M. Rauterberg, TU/e

92. SONY’s AIBO (1999)
1999 marked a turning point for the world of entertainment when Sony introduced the electronic robot AIBO in Japan. AIBO, referred to as the Entertainment Robot means “companion” and is an acronym for Artificial Intelligence RoBOt.
Sony has developed two versions of AIBO, the first run ERS-110 and the newer ERS-111. Both of the models have proven to be a large success. When the first 5,000 ERS-110 models were introduced 3,000 sold in Japan within twenty minutes on the Internet. The 2,000 models that remained were made available exclusively for the United States and were sold within four days on the Internet. Demand for the new AIBO ERS-111, which has a 64-bit microprocessor and 32 bytes of memory, rose immensely. Ten thousand ERS-111 AIBO models were made available despite receiving 135,000 orders. “The digital ‘bot is more dexterous than its predecessor: In addition to all the usual tricks—heeling or chasing a ball—it does a little dance and waves a front paw on hearing its name. Speech-recognition software lets it learn up to 50 commands” (International Business, # 3709, page 170). The cost for AIBO in the United States has remained thus far steady at $2,500 with the option to purchase a $500 performer kit. AIBO is attempting to become popular all over the world, however, 90% of AIBO purchases still come from Japan.
© M. Rauterberg, TU/e

93. SDR-3X (2000)
Much like its predecessors Aibo and Aibo II, Sony's 50cm-tall, 50kg prototype SDR-3X has been designed to entertain. It can perform a variety of relatively high-speed, autonomous movements, including walking at a speed of 15 meters a minute and dancing to a tune with a quick tempo. It's equipped with speech- and image- recognition functions. The SDR-3X gave a demonstration by walking at a high speed, moving its body like a gymnast, dancing to a disco tune with a quick tempo and kicking a ball into a goal net as instructed by voice. The SDR-3X model employs the same OPEN-R architecture as Sony's autonomous entertainment canine robot "AIBO." Similar to AIBO, the SDR-3X model recognizes human voices and images. It has an 18,000-pixel CCD color camera in its head area. The SDR-3X model can maintains its balance in the upper half of the body by twisting its body and moving its arms, thereby realizing stable walking movements with alternating feet. The SDR-3X model has 24 joints, with actuators for each joint that help move the joints. It has two joints in the neck, two in the trunk, four in each arm and six in each leg. Two 64-bit RISC processors enable real-time control of the joints to realize autonomous movement. Sony's original real-time operating system called "Aperios" is used for the SDR-3X. The biggest challenge for a humanoid robot is to keep its body in full balance. The SDR-3X keeps balance by moving its arms and twisting at the waist to counteract the yaw-axis moment, the force needed to turn the body right and left, which is generated from the lower half of the body every time the robot takes one step forward in the high-speed walking movement. And the robot's posture is controlled in real-time, to prevent it from falling over. It uses a variety of information, such as the angle of the floor, gathered from contact sensors in the torso section as well as from a "dual-axis accelerometer" and a "dual-axis angular rate sensor" in the waist section, whenever the robot walks up slopes and moves its whole body. It can (1) move forward or backward and walk sideways at a speed of up to 15 meters a minute; (2) turn left or right when walking (with the maximum 90 degrees of freedom for each step forward); (3) get up from the position of lying on its stomach or its back; (4) stand on one leg (possible even on inclined ground); (5) walk on a bumpy road; (6) kick a ball; and (7) dance to a wide range of tunes.
© M. Rauterberg, TU/e

94. Augmented Reality (AR)
Core aspects:
User sees real environment; combines virtual with real.
Supplements reality, instead of completely replacing it.
Photo-realism not necessarily a goal.
© M. Rauterberg, TU/e

95. AR (historical overview)
•Early 1990’s: Boeing coined the term “AR”.
Wire harness assembly application begun.
•Early to mid 1990’s: UNC ultrasound visualization project
•1994: Motion stabilized display [Azuma]
•1994: Fiducial tracking in video see-through [Bajura / Neumann]
© M. Rauterberg, TU/e

96. AR at Boeing (1990)
The term "augmented reality" was coined at Boeing in 1990 by researcher Tom Caudell. He and a colleague, David Mizell, were asked to come up with an alternative to the expensive diagrams and marking devices then used to guide workers on the factory floor. They proposed replacing the large plywood boards, which contained individually designed wiring instructions for each plane, with a head-mounted apparatus that would display a plane's specific schematics through high-tech eyeware and project them onto multipurpose, reusable boards. Instead of reconfiguring each plywood board manually in each step of the manufacturing process, the customized wiring instructions would essentially be worn by the worker and altered quickly and efficiently through a computer system.
Tom Caudell
© Boeing Inc., USA
© M. Rauterberg, TU/e

97. Motion-stabilized AR Display (1994)
This system is the first motion-stabilized AR display that works outdoors and achieves tighter registration than any previous outdoor AR system.
© Ronald Azuma, HRL Laboratories
© M. Rauterberg, TU/e

98. © University of Southern California
Fiducial tracking (1994)
The AR tracking system of Bajura and Neumann consists of a robust vision landmark tracker, inertial gyro sensors, and the complementary fusion filter described above. The sensor module contains a CCD video camera (Sony XC-999 with 6mm lens), and three orthogonal rate gyroscopes (GyroChip II QRS , from Systron Donner), which are tightly covered by a foam block to provide shock protection and a stable temperature environment from the sensors. The video camera provides a 30 Hz video stream, while the three gyroscopes are sampled at 1kHz via a 16-bit A/D converter (National Instruments DAQPCI-AI-16XE-20).
© M. Rauterberg, TU/e
© University of Southern California

99. MIT Wearables (1996-) What's a Wearable?
To date, personal computers have not lived up to their name. Most machines sit on the desk and interact with their owners for only a small fraction of the day. Smaller and faster notebook computers have made mobility less of an issue, but the same staid user paradigm persists. Wearable computing hopes to shatter this myth of how a computer should be used. A person's computer should be worn, much as eyeglasses or clothing are worn, and interact with the user based on the context of the situation. With heads-up displays, unobtrusive input devices, personal wireless local area networks, and a host of other context sensing and communication tools, the wearable computer can act as an intelligent assistant, whether it be through a Remembrance Agent, augmented reality, or intellectual collectives.
© M. Rauterberg, TU/e

100. Digital Desk (1991) © Xerox EuroPARC, UK
Pierre Wellner
The DigitalDesk is built around an ordinary physical desk and can be used as such, but it has extra capabilities. A video camera is mounted above the desk, pointing down at the work surface. This camera's output is fed through a system that can detect where the user is pointing (using an LED-tipped pen) and it can recognise the documents that are placed on it. The more advanced version also has a computer-driven projector mounted above the desk enabling electronic objects to be projected onto real paper documents -- removing the burden of having to switch attention between screen and paper and allowing additional user- interaction techniques. [invented and built by Pierre Wellner, Xerox EuroPARC, ACM CHI paper]
© Xerox EuroPARC, UK
© M. Rauterberg, TU/e

101. © University of Illinois at Chicago
ImmersaDesk (1994)
The ImmersaDesk is a drafting-table format virtual prototyping device. Using stereo glasses and sonic head and hand tracking, this projection- based system offers a type of virtual reality that is semi-immersive.
The ImmersaDesk features a 4x5-foot rear-projected screen at a 45-degree angle. The size and position of the screen give a sufficiently wide-angle view and the ability to look down as well as forward. The resolution is x 768 at 96Hz.
© University of Illinois at Chicago
© M. Rauterberg, TU/e

102. Build-It (1996)
design team with experts from different disciplines are planning together
integration of different expertise
intuitive interaction style with a tangible object ‘brick’
an example for a natural user interface
© ETH, Zurich
© M. Rauterberg, TU/e

103. Cubby (1997)
Cubby, a desktop VR system, addresses the problem ‘ unification of the display and manipulation spaces’. With a pen-like input device, the user can operate on virtual objects where they appear Unlike many VR systems, Cubby is well-suited to precision manipulation tasks. Possible areas of application include surgical simulation and computer aided modeling.
Invented and built by Tom Djajadiningrat
© Technical University Delft, The Netherlands
© M. Rauterberg, TU/e

104. Virtual Workbench (1998) Kent Ridge Digital Labs (KRDL), Singapore
© M. Rauterberg, TU/e

105. DynaWall & CommChairs (1998)
The size of the DynaWall opens a new set of human-computer interactions.
It is possible that information objects can be taken at one position and put somewhere else on the display or thrown from one side to the opposite side.
Dialog boxes always appear in front of the current user(s).
User interface components are always at hand, etc.
© GMD IPSI, Germany
© M. Rauterberg, TU/e

106. Horizontal 3D Displays (1998)
© CMD, Uppsala University
© M. Rauterberg, TU/e

107. Speech and Voice in User Interface
Application areas
• Control and Data input in “hands-busy” environments
• Feedback in visually limited environments
• System control over telephone-line
• Control for impaired or disabled people
© M. Rauterberg, TU/e

108. Speech Signal Processing (1949)
J. Dreyfus-Graf (1949). “Sonograph and Sound Mechanics”. The Journal of the Acoustic Society of America, 22, pp
This was not a real speech recognizer but an oscilloscope that would put the beam at a different spot depending of the content of the speech.
Raymond Kurzweil: "[Bell's] insights into separating the speech signal into different frequency components and rendering those components as visible traces were not successfully implemented until Potter, Kopp, and Green designed the spectrogram and Dreyfus-Graf developed the steno-sonograph in the late 1940s. These devices generated interest in the possibility of automatically recognizing speech because they made the invariant features of speech visible for all to see."
© M. Rauterberg, TU/e

109. Speech Recognizer (1952)
The first real pattern matcher was developed at AT&T Bell Labs:
Davis, K., Biddulph, R. & Balashek, S. (1952). “Automatic recognition of spoken digits”. The Journal of the Acoustic Society of America, 24(6),
In 1952, as the US government-funded research began to gain momentum, Bell Laboratories developed an automatic speech recognition system that successfully identified the digits 0 to 9 spoken to it over the telephone. Major developments at MIT followed. In 1959, a system successfully identified vowel sounds with 93% accuracy. Then seven years later, a system that had a vocabulary of 50 words was successfully tested. In the early 1970’s, the SUR program yielded its first substantial results. The HARPY system, at Carnegie Mellon University, could recognize complete sentences that consisted of a limited of range of grammar structures. But the computing power it required was prodigious; it took 50 contemporary computers to process a recognition channel. [source
© M. Rauterberg, TU/e

110. TX-0 (1959) MIT Speech group with TX-0, c1959.
Speech, handwriting recognition, neuro data analysis, etc.
Interactive editors, debuggers, etc.
The TX-0 computer was the world's first high speed transistorized computer. The TX-0 was the most powerful system of its day (1957), representing a quantum leap in computer technology at the time.
© MIT Lincoln Lab
© M. Rauterberg, TU/e

111. History of Speech Recognition Technologya n e o u s w o r d s p t i n g v c e m a y f l b 2 - u k
& s p e e c h F l u e n t s p e e c h e t l y m e s s a g i n g t s g n R e a d i k s p e e c h a e p S C o n n e c t e d s p e e c h I s o l a t e d w o r d s 2 2 2 2 2 U n r e s t r i c t e d
© M. Rauterberg, TU/e V o c a b u l a r y s i z e ( n u m b e r o f w o r d s )

112. History of DARPA speech recognition benchmark
100%
Switchboard
Conversational
Speech
foreign
Read
Speech
WSJ
Broadcast
Speech
Spontaneous
Speech
Varied
Microphone 
20k
foreign  
ATIS
NAB
WORD ERROR RATE
10%  5k 
Noisy 1k     
Resource
Management
Courtesy NIST 1999 DARPA
HUB-4 Report, Pallett et al. 1%
© M. Rauterberg, TU/e

113. Speech controlled application
A voice activated dental record system (ORATEL, ORQUEST)
(200 different words, relatively simple grammar)
1. Dragon
2. L&H
3. IBM Pure Speech
4. Kurzweil
© M. Rauterberg, TU/e

114. International Standards
Based on the path-breaking work of Dzida, Herda and Itzfeld (1978) twenty years ago, the German national standard DIN 66234, part 8 was developed and published in 1988.
Dzida, W., Herda,S. and Itzfeld, W.D. 1978, User-perceived quality of interactive systems. IEEE Transactions on Software Engineering, SE-4, 270 – 276.
Its definitions of usability principles for software user interfaces for office work became the basis for the international and European standard ISO EN (ISO 1996). This standard serves as the reference for the European Community directive 90/270/EEC for minimum safety and health requirements to be guaranteed by an employer for his staff working at computer workstations.
: DIN : Grundsätze ergonomischer Dialoggestaltung.
1990: European Community Directive 90/270/EEC for minimum safety and health requirements.
1995: ISO 9241: Ergonomic requirements for office work with display terminals (VDTs).
© M. Rauterberg, TU/e