1. 1 Direct Manipulation Interfaces  Positive acceptance of an application  Mastery of the interface  Competence in performing tasks  Ease in learning originally and in assimilating advanced features  Confidence in the capacity to retain mastery over time  Enjoyment in using the interface  Eagerness to show off interfaces to novices  Desire to explore more powerful aspects

2. 2 Direct Manipulation Interfaces  Attributes of direct-manipulation interfaces  Visibility of the objects and actions of interest Example: Driving an automobile  Rapid, reversible, incremental actions  Replacement of types commands by a pointing action on the object of interest  Example: Dragging a file to a trash can versus “rm file.doc” What about: “rm file*.doc”? What about “rm *.*” with no undo?

3. 3 Direct Manipulation Interfaces  Extensions of direct manipulation  Virtual reality – users are in an immersed environment Reality is blocked out via a head-mounted display Hand gestures (via a data glove) allow users to point, select, grasp and navigate  Augmented reality – user remains in normal surroundings, but adds a transparent overlay. Examples – labeled buildings, hidden plumbing Google: Project Glass http://www.youtube.com/watch?v=W2E2zcFt9Xo http://www.youtube.com/watch?v=tnRJaHZH9lo&feature=related  Tangible user interfaces – users manipulate physical objects Example – putting several plastic blocks near each other to create an office floor plan

4. 4 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  The Tubeless Interface

5. 5 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors (e.g., IBM MVS, VM, TSO, JCL)

6. 6 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors versus WYSIWYG (what you see is what you get) editors (e.g., Microsoft Word – early 1990s)  Example: Three basic modes of vi Command mode (Telling the computer what to do: Low level commands, e.g., move the cursor to the right one character)  Default when you enter vi.  Most letters, or short sequences of letters, that you type will be interpreted as commands  Pressing Esc when you're in command mode, your terminal will beep at you. This is a very good way to tell when you're in command mode Insert mode (Entering the content)  Whatever you type is inserted in the file at the cursor position  Press Esc to end insert mode, and return to command mode. Line mode (Telling the computer what to: High level commands, e.g. Save)  To enter line mode from command mode, type a colon ( : )  Your cursor moves to the bottom of the screen, by a colon prompt.  Type a line mode command, then press Enter.

7. 7 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors (e.g., vi) Starting vi and Saving Files Starting vi: vi filename (start editing filename, create it if necessary) Saving the file you're working on and/or leaving vi: :wq (write the file to disk and quit) Quit without saving any changes: :q! :w! newfile (write all lines from the entire current file into the file 'newfile', overwriting any existing newfile) :n,m w! newfile (write the lines from n to m, inclusive, into the file newfile, overwriting any existing newfile)

8. 8 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors TypeTo Move To hone space to the left (also try left arrow) jone line down (also try down arrow) kone line up (also try up arrow) lone space to the right (also try right arrow) $end of current line ^beginning of current line Enterbeginning first word on the next line Gend of file :nline n; use :0 to move the beginning of the file wbeginning of next word; 5w moves to the beginning of the 5th word to the right eend of next word bbeginning of previous word Ctrl-bone page up Ctrl-fone page down %the matching (, ), [, ], {, or } (Press % with your cursor on one of these characters to move your cursor its mate.)

9. 9 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors Searching for text Inserting text TypeTo /stringsearch down for string ?stringsearch up for string nrepeat last search from present position TypeTo aappend starting right of cursor Aappend at the end of the current line iinsert starting left of cursor Iinsert at beginning of the current line oopen line below cursor, then enter insert mode Oopen line above cursor, then enter insert mode :r newfile add the contents of the file newfile starting below the current line

10. 10 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors Deleting text TypeTo xdelete single character; 5x deletes 5 characters dwdelete word; 5dw deletes 5 words dddelete line; 5dd deletes 5 rows cwdelete word, leaves you in insert mode (i.e. change word) ccchange line -- delete line and start insert mode schange character -- delete character and start insert mode Ddelete from cursor to end of line Cchange from cursor to end of line -- delete and start insert mode uundo last change Uundo all changes to current line Jjoin current line with line that follows (press Enter in insert mode to split line)

11. 11 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Single-line and Multi-line Editors Cutting and Pasting Miscellaneous Commands TypeTo xptranspose two characters (two commands, x followed by p) yyyank (i.e. copy) one line into a general buffer (5yy to yank 5 lines) "ayyyank into the buffer named a Pput the general buffer back before the current line "aPput from buffer a before current line pput the general buffer back after the current line "apput from buffer a after the current line TypeTo Ctrl-gshow line number of current line Ctrl-lredraw the entire display

12. 12 Direct Manipulation Interfaces  Command-line versus display editors versus word processors  Early 1980s – Text editing was done with line-oriented command languages  Nroff/troff: Unix based word processor.nf = no-fill, you use for graphs or text that you don't want spaces to be ignored.ce # = centers by the # of sentences you enter.ti # =.ti indents but only for one line, so if you have one sentence that needs to be indented 8 spaces but the rest of the page is indented 3 you can use.ti 8 for that single sentence.fi = fill-in, extra space will be ignored and text that is entered like this, the fill-in command will continue until you enter a.nf command and vice-versa.in # = # is the number of spaces you want the text indented, this command will be in place until you put in a new.in command...if you want something to not be indented you put in.in 0

13. 13 Direct Manipulation Interfaces  Command-line editors  Who would use these interfaces?  Why?

14. 14 Direct Manipulation Interfaces  Command-line editors  Who would use these interfaces?  Why? Mastery of the interface Competence in performing tasks Ease in learning originally and in assimilating advanced features Confidence in the capacity to retain mastery over time Enjoyment in using the interface Eagerness to show off interfaces to novices Desire to explore more powerful aspects

15. 15 Direct Manipulation Interfaces  Command-line editors  Who would use these interfaces?  Why? BETTER THAN A TYPEWRITER BETTER THAN A SLIDE RULE BETTER THAN A CALCULATOR

16. 16 Direct Manipulation Interfaces  Advantages of WYSIWYG Editors  Users see a full page of text 20 to 60 lines provides a context for each sentence  The document is seen as it will appear when printed Eliminating the clutter of formatting commands  Cursor action is visible Indicates where to focus attention and apply action  Cursor motion is natural Arrow keys or mouse provide natural physical mechanisms for moving the cursor  Labeled icons make frequent actions rapid Toolbar for frequent actions  Immediate display of the results of an action Example: Clicking a button to center text provides immediate result  Rapid response and display Full page of text in a fraction of a second  Easily reversible actions Example: Undo, backspace

17. 17 Direct Manipulation Interfaces  Technology advancements evolving from word processing  Integration of graphics, spreadsheets, animations, photographs, etc.  Desktop publishing software  Presentation software  Hypermedia environments and the World Wide Web (hyperlinks to documents)  Improved macro facilities (e.g., construct, save and edit sequences of frequently used actions)  Spell checkers and thesauri  Grammar checkers Use of passive voice Excessive use of certain words Lack of parallel construction  Document assemblers Contracts Wills

18. 18 Direct Manipulation Interfaces  Spreadsheets  1979 – VisiCalc from a Harvard Business School student  254 rows and 63 columns  Functions within a cell as it relates to other cells  Simulation of an accountants spreadsheet  Lotus 1-2-3 dominated the market in the 1980s  Today Excel dominates Graphics displays Multiple windows Statistical routines Database access (e.g., Price List to Service Catalog)

19. 19 Direct Manipulation Interfaces  Spatial Data Management  Geographic applications – Nicholas Negroponte at MITNicholas Negroponte at MIT  Display of the world and able to zoom in on the Pacific Ocean to see markers for convoys of military ship

20. 20 Direct Manipulation Interfaces  Spatial Data Management  ArcView – ESRI ArcView – ESRI Global Information Systems (Demo)Demo Select type of information to display (roads, population, rainfall, topography, political boundaries

21. 21 Direct Manipulation Interfaces  Video Games  The most exciting, well-engineered, commercially successful application of direct-manipulation concepts?  Pong Pong  Pacman (http://www.activitypad.com/online-games/pacman/)http://www.activitypad.com/online-games/pacman/  Field of action is visual and compelling  Button presses, joystick motions and knob rotations produce rapid response on the screen  No syntax to remember  Error messages are rare – the results of the action are obvious and easily reversed  Often there is continuous display of the score (competition between others and the player themselves) Positive reinforcement that encourages mastery

22. 22 Direct Manipulation Interfaces  Video Games  Educational Video Games – direct manipulation SimCity – education on urban planning The Sims – stronger attraction to women then men

23. 23 Direct Manipulation Interfaces  Video Games  Computer Role Playing Games (CRPGs) Players assume the role of a fictional character Activity takes place in a fictional world User’s control many of their character's actions Myst  Massively Multiplayer On-Line Role Playing Games (MMORPGs) A large number of players interact with one another in a virtual world Interaction is in a persistent world –hosted by the game's publisher –continues to evolve while the player is away from the game Worldwide MMORPGs revenues exceeded half a billion dollars in 2005 World of Warcraft

24. 24 Direct Manipulation Interfaces  Video Games versus Business Applications  Game players Engaged in competition with a system or other players Seek entertainment and focus on challenge May prefer random events  Application users Prefer a strong internal locus of control Focus on their tasks and may resent too many playful distractions Do not prefer random events

25. 25 Direct Manipulation Interfaces  Computer-aided Design  Automobiles, electronic circuitry, aircraft, mechanical engineering  Structural engineering, floorplans, interiors, landscaping, plumbing, electrical installation, etc.  When the design is complete, the program can provide information regarding: Current Voltage drops Fabrication costs Manufacturing problems

26. 26 Direct Manipulation Interfaces  Computer – aided Manufacturing and Process Control  Honeywell’s Experion Process Knowledge System Provides the manager of a oil refinery or power utility plant with a colorized schematic of the plant Can indicate with a red line a sensor value that is out of range With a single click the operator can get a more detailed view of the troubling component A second click can provide more detailed information the sensor, or reset a value or circuit Basic strategy: eliminate the need for complex commands that the operator might only need to recall during a once-a-year emergency

27. 27 Direct Manipulation Interfaces  Direct Manipulation in Office Automation  Xerox Star Sophisticated Text Formatting Graphics Multiple Fonts High Resolution Cursor Based Interface  Apple Lisa Precursor to the Macintosh Hardware and software designs supported –Pull-down menus –Multi-window manipulation –Editing of graphics and text –Dragging of icons

28. 28 Direct Manipulation Interfaces  Direct Manipulation in Office Automation  MS-DOS Commands vs. Macintosh Direct Manipulation Tasks: Creating, copying, renaming, erasing files After training and practice, average task times: –MS-DOS is 5.8 minutes –Macintosh is 4.8 minutes After training and practice, average errors: –MS-DOS is 2.0 –Macintosh is 0.8

29. 29 Direct Manipulation Interfaces  Continuing evolution of direct manipulation  Quicken  Home Automation Direct manipulation on a floor plan of: –Burglar alarms –Heat sensors –Smoke detectors –Opening/closing curtains or screens –Air conditioning and heating –Audio/video speakers or screens »E.g., users can route sound from a MP3 player located in the living room to the kitchen by dragging the MP3 icon into the kitchen  Virtual Worlds Travel through the human body Ride an electron cloud as it spins around a nucleus

30. 30 Direct Manipulation Interfaces  Continuing evolution of direct manipulation  Problems with direct manipulation Spatial or visual representation are not necessarily an improvement over text –Especially for blind or visually impaired users Direct manipulation designs may consume considerable screen space –May result in scrolling or multiple actions Users must learn the meanings of visual representations –Icon interpretation For experienced typists, taking a hand off the keyboard to move a mouse may take more time then typing the relevant command Users may not share the same understanding of the metaphor, analogy, or conceptual model with the designer (testing is required) Browser based applications limit direct manipulation (e.g., drag & drop) –Require Dynamic HTML, Java or Flash

31. 31 Direct Manipulation Interfaces  Advantages of direct manipulation  Continuous representation of the objects and actions of interest with meaningful visual metaphors  Physical actions or presses of labeled buttons, instead of complex syntax  Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

32. 32 Direct Manipulation Interfaces  Advantages of direct manipulation  Design systems with the following benefits Novices can learn basic functionality quickly Experts can work rapidly to carry out a wide range of tasks, even defining new functions and features Knowledgeable intermittent users can retain operational concepts Error messages are rarely needed Users can immediately see whether their actions are furthering their goals, and if the actions are counterproductive, they can change the direction of their activity Users experience less anxiety because the interface is comprehensible and because actions are easily reversed Users gain confidence and mastery because they are the initiators of action, they feel in control, and they predict the interface’s responses

33. 33 Direct Manipulation Interfaces  The OAI Model and Direct Manipulation  The object of interest is displayed so that interface actions are close to the high-level task domain  Little need for mental decomposition of tasks into multiple interface commands with complex syntactic forms (e.g., vi: go to line, go to word, go to character)  Each action produces a comprehensible result in the task domain that is visible in the interface immediately  The closeness of the task domain to the interface domain reduces operator problem-solving load and stress  Compared to textual descriptors, visual representations of objects may be more natural and closer to human innate capabilities Action and visual skills emerged well before language in human evolution

34. 34 Direct Manipulation Interfaces  Visual Thinking and Icons  Semiotics – the study of signs and symbols  Icon – an image, picture or symbol representing a concept In computer systems usually less than one inch square (64x64 pixels) Smaller icons are often integrated with a window border or toolbar  Task dependency When working on a visual task (e.g., painting program), icons may be useful When working on a text-based task, it may be better to stay text based  Icons with words (or mouse-overs) are useful

35. 35 Direct Manipulation Interfaces  Icon related guidelines  Represent the object or action in a familiar and recognizable manner  Limit the number of different icons  Make the icon standout from its background  3-d icons can be visually distracting  Ensure that a single selected icon is clearly visible when surrounded by unselected icons  Make each icon distinctive from every other icon  Ensure harmoniousness of each icon as a member of a family of icons  Design the movement animation (e.g., grayed-out ghost image on a drag)  Detail information Larger shadowing for a larger file Color to show the age of a document Animation to show how much of a file has printed (document icon absorbed progressively into the printer icon)  Explore use of combination of icons (E.g., drag a document to a printer icon)

36. 36 Direct Manipulation Interfaces  Direct Manipulation Programming  Example: programming a radio to a set of stations by pressing/holding a channel selection button  Phone Services

37. 37 Direct Manipulation Interfaces  Direct Manipulation Programming  Programming in the User Interface Sufficient computational generality –Conditionals (if, then, else) –Iteration (repeat/while) Access to appropriate data structures and operators –File structures for directories –Addition, subtraction, etc. Ease in programming –By specification or by demonstration (Flash animation) –Argument passing Simplicity in invocation and assignment of arguments Low risk –High probability of bug free programs –Halt and resume

38. 38 Direct Manipulation Interfaces  Direct Manipulation Programming  Viscosity – the difficulty of making changes to a program  Progressive evaluation – the capacity for execution of partial programs

39. 39 Direct Manipulation Interfaces  3-Dimensional Interfaces  Some applications are designed as 2-D to be simpler than real-world systems Constrain movement Limit interface actions Ensure visibility of interface objects  Enhanced 3D may be better than 3D Flying through objects Multiple simultaneous views of objects X-ray vision Shrink/expand objects Group/ungroup components Going back in time  Less than successful 3D interfaces Air-traffic control Showing altitude by perspective drawing only adds clutter when compared to an overview from directly above http://www.youtube.com/watch?v=Hw-O4zX8qRY

40. 40 Direct Manipulation Interfaces  4-Dimensional Interfaces  4-D Anyone? 4-D Anyone? http://www.youtube.com/watch?v=bIj0oW- tTF4&feature=fvwrel Tesseract

41. 41 Direct Manipulation Interfaces  Second Life Second Life  Multi-user environment where users interact  Users can choose avatars (fantasy images, desirable characteristics)

42. 42 Direct Manipulation Interfaces  Relationship to:  Personality Theory  Social Psychology  Spatial Cognition  Applicable to business meetings, community discussion groups, political forums?  Blaxxun envision Blaxxun envision

43. 43 Direct Manipulation Interfaces  3D Desktops and Workplaces  Microsoft’s Task Gallery Microsoft’s Task Gallery  Intel’s Grand Canyon  Xerox PARC’s Information Visualizer  No successful products yet

44. 44 Direct Manipulation Interfaces  Tips for effective 3D interfaces  Use occlusion, shadows, perspective and other 3D techniques carefully  Minimize the number of navigation steps for users to accomplish their tasks  Keep text readable (better rendering, good contrast with background, an no more than 30-degree tilt)  Avoid unnecessary visual clutter, distraction, contrast shifts and reflections  Simplify user movement (keep movements planar, avoid surprises like going through walls)  Organize groups of items in aligned structures to allow rapid visual search  Enable users to construct visual groups to support spatial recall (e.g., placing items in corners)

45. 45 Direct Manipulation Interfaces  Guidelines for inclusion of 3D features  Provide users overviews so they can see the big picture  Allow teleportation (rapid context shifts by selecting destination in an overview)  Offer x-ray vision  Provide history keeping (recording, undoing, replaying, editing)  Permit rich user actions on objects (save, copy, annotate, share, send)  Give users control over explanatory text (pop-up, floating, screen tips)  Offer tools to select, mark and measure  Implement dynamic queries to rapidly filter out unneeded items  Enable landmarks to show themselves even at a distance  Allow multiple coordinated views (users can be in more than one place at a time)

46. 46 Direct Manipulation Interfaces  Teleoperation  Derived from direct manipulation and process control  Physical processes taking place in a remote location Clean-up in a nuclear reactor  Need adequate feedback in sufficient time to permit effective decision making Manufacturing Medicine (consultation, radiology) Military operations (drones)  Home automation Answering machines Security systems Energy control Appliances Mahru Humanoid Robot Real-Time Teleoperation http://www.youtube.com/watch?v=TJmQqC1nHTU&feature=fvwrel

47. 47 Direct Manipulation Interfaces  Design to accommodate teleoperation issues  Slow response times and time delays Transmission delay (time for command to reach the microscope) Operation delay (time until the microscope responds)  Incomplete feedback The microscope can transmit its current position, but operates so slowly it cannot indicate the exact current position  Unanticipated interferences The slide is accidentally moved by a person at the local site  May be better for the user to specify a destination (rather than a motion) and wait until the action is completed http://www.youtube.com/watch?v=aFPQfhUArjI&feature=fvw rel

48. 48 Direct Manipulation Interfaces  Telemedicine  Remote examination  Remote surgery  Telepathology Magnification Focus Illumination Position

49. 49 Direct Manipulation Interfaces  Virtual and Augmented Reality  “Being in” as opposed to “Looking at”  Architectural applications Wall-sized image to give perspective Animation to simulate movement (left to right) Treadmill to simulate walking toward, walk through doors, stairs Replace projector with a head-mounted display  Some applications are better when “looked at” Air-traffic control To surgeons want to “Be in” the patients body?  Training using virtual reality Fifth Dimension Technologies www.5DT.comwww.5DT.com

50. 50 Direct Manipulation Interfaces  Virtual and Augmented Reality  The CAVE (National Center for Supercomputing Applications)  An immersive virtual reality facility designed for the exploration of and interaction with spatially engaging environments.  The stereoscopic capabilities, coupled with its uniquely immersive design, enable scientists and researchers to interact with their data An atmospheric scientist can actually "climb inside" of a hurricane and visualize its complex and chaotic elements from any angle or visual perspective A biological researcher, examining a tightly coiled strand of DNA, can virtually "unravel" this strand and manipulate it in an environment that preserves the critical depth information of the data. Teach a child to cross a street http://www.youtube.com/watch?v=C5jSSKwBbVM

51. 51 Direct Manipulation Interfaces  Artificial Reality  VideoPlace – Myron Krueger Surround the user with an artificial reality which responded to their movements and actions. The users were able to visually see the results of their actions on screen, through the use of colored silhouettes. The users had a sense of presence while interacting with onscreen objects and other users. The sense of presence was enough that users pulled away when their silhouettes intersected with those of other users. http://www.youtube.com/watch?v=0MsySesdisE eHarmony 2.0?

52. 52 Direct Manipulation Interfaces  Applications of virtual environments  Phobia treatment Acrophobia  Pain Control Immersive environments provide distractions for patients  Interior Design OAI model –Click, drag, enlarge objects –Room painting tool http://www.youtube.com/watch?v=jNIqyyypojg

53. 53 Direct Manipulation Interfaces  Augmented Reality  See the real world with an overlay of additional information See wires or plumbing behind walls Tourist glasses – label buildings in a historic town Molecular biology http://www.youtube.com/watch?v=ZczX6qleV4Q

54. 54 Direct Manipulation Interfaces  Virtual environments dependent on integration of multiple technologies  Visual Display Normal Display –12 to 17 inches diagonally at a normal viewing distance of 70 cm subtends a visual angle of about 5-degrees Large Screen –17 to 30 inches can cover 20 to 30-degrees Head Mounted Displays –100 degrees horizontally and 60-degrees vertically –Head motion produces new images so users perceive 360-degrees –Displays must approach 100-millisecond delay in presenting images to approach real time

55. 55 Direct Manipulation Interfaces  Virtual environments dependent on integration of multiple technologies  Head-position sensing Head-mounted displays can provide differing views depending on head position  Hand-position sensing DataGlove

56. 56 Direct Manipulation Interfaces  Virtual environments dependent on integration of multiple technologies  Force feedback and haptics Hand-operated remote-control devices for performing chemistry experiments or for handling nuclear materials Gives users a sense of grasp  Sound input and output Training of Army tank crews while using realistic sounds of battle resulted in: –Elevated heart rates, more rapid breathing, and increased perspiration Speech recognition for initiating actions and making menu selection –Keyboard and mouse use is restricted  Other sensations Tilting and vibration of flight simulators  Collaborative and competitive virtual environments Two people at remote sites working together while seeing each others actions and the object of interest

57. 57 Direct Manipulation Interfaces  Definition, benefits, and drawbacks of direct manipulation  Definition Visual representation (metaphor) of the “world of action” Objects are always shown Rapid, incremental, and reversible actions Replacement of typing with pointing/selecting Immediate visibility of results of actions  Benefits over commands Less syntax reduces error rates Errors are more preventable Faster learning and higher retention Encourages exploration

58. 58 Direct Manipulation Interfaces  Definition, benefits, and drawbacks of direct manipulation  Concerns Increased system resources (possibly) Some actions may be cumbersome (e.g., form fillin forcing use of mouse) History and other tracing may be difficult Visually impaired users may have more difficulty

59. 59 Direct Manipulation Interfaces  Piaget’s Four Stages of Development  Sensorimotor (birth to 2 years)  Preoperational (2 to 7 years)  Concrete operational (7 to 11 years) Physical actions on an object are comprehensible Children acquire the concept of conservation or invariance Direct manipulation brings activity to this stage  Formal operational (begins at 11 years) Symbol manipulation to represent actions on objects