1. Direct Manipulation Shneiderman and Plaisant, Chapter 6
(from slides at
Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct Manipulation interfaces. Human-Computer Interaction, 1,

2. Overview Introduction Examples of Direct-Manipulation systems
Big picture first, details and a theoretical account later
Examples of Direct-Manipulation systems
Explanations of Direct Manipulation
Hutchins et al. paper
Virtual Environments, or Immersive Interfaces

3. Recall, Interaction Styles from Shneiderman
Styles for “giving commands” to the system
5 main types (p. 67):
Direct Manipulation
More detail later
Menu selection
Form fillin
Command language
Natural language
May blend, especially when users are diverse

4. Paradigm Example: File System Direct Manipulation vs. Command Line
User operates on model world to change task world
Will sharpen this up with more detailed accounts
Dix et al. and Hutchins, Hollan, & Norman (reading)

5. “Direct Manipulation” Interface to Car No Computer, but “Direct” – “an ultimate example”
Task: Maneuver the car from Point A to Point B (within certain constraints)
Interface: Steering wheel, pedals, etc.
Geographic area through which user (driver) maneuvers car is displayed directly in front of user
(and in high resolution, and in large field of view!)
Manipulating steering wheel top to left (and right) causes car front to move left (and right)
visual and kinetic feedback is immediate
Pushing brake pedal results in car slowing
arbitrary mapping of user action to car maneuver (automatized)
Definition, again:
User operates on a visually displayed “world” to perform tasks in a domain
Here, the world on which the user operates, is the real world
For computing tasks, there is the task world and an interface model world
The user operates on the model world to change the task world

7. Recall (or not), from 1st class Interaction Frameworks (Dix)
Communication between user and system
Why have a framework?
Allows “precision” in accounting for differences
E.g., gulfs of execution and evaluation
Presents global view
All elements receive attention
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”

8. Background: Models in UI Design From Dix et al.
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”
and its representation
Models of UI design consider:
System
Interface
User
Will “rewrite” Dix’s account to incorporate standard account of models

9. Background: Models in UI Design From Dix et al.
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”
and its representation
Models of UI design consider:
System
Interface
User
Will “rewrite” Dix’s account to incorporate standard account of models
First, it is the System that directly provides feedback about the task
Task
“work on task”
“gives commands”
User
feedback
and its representation
System

10. Models in UI Design Task User System Task User System
“work on task”
“gives commands”
User
feedback
and its representation
System
Next, just change notation a bit
And …
Task
“work on task”
“gives commands”
User
System

11. Models in UI Design Task System User User System Interface Model Model
“work on task”
“gives commands”
User
System
For our purposes, Dix’s System (which also represents task) comprised of two components
1. System model
How system actually works
Implementation model
2. Interface model
Model system presents to user
Manifest model
System
Model
Interface
Model
User
Model
Finally, 3. User model
How the user thinks system works
Conceptual model

12. User/Conceptual/Mental … Model
How the user thinks system works
Conceptual model
May or may not correspond to the system!
Ideally, should match closely, … if (and only if) system reflects task
Mental model of the system (and task)

13. Example of DM: Word Processing
People used to use script files for formatting, until 1980’s
E.g., nroff
“word processors” allowed direct manipulation of document (text and images) output
Albeit with a fair amount of pointing, selecting, and menu use

14. Example of DM: VisiCalc and Later
People used to use paper and pencil for accounting, until 80’s
Shneiderman
The VisiCalc spreadsheet and its descendants
Imagine erasing a thousand numbers …
“VisiCalc users delighted in watching the program propagate changes across the screen.”
In some cases, spatial representations provide a better model of reality
Successful spatial data-management systems depend on choosing appropriate:
Icons
Graphical representations
Natural and comprehensible data layouts

15. Example of DM: WIMP File System
Copy, move, rename, delete
But, is it a file cabinet?

16. Example of DM: Drawings
Autocad, etc.
Spatial

17. Example of DM: Statistical Analysis
Specify statistical operations by manipulating icons
Similar to “visual programming”

18. Example of DM: Statistical Analysis

19. Example of DM: Games
D & D with voice chat
Doom, etc.
.. And better

20. Basic Interaction Paradigm In context of direct manipulation and command language
Task
“work on task”
“ gives commands”
System
User
feedback
“feedback”
and its representation
(possibly manipulable)
Historically, “commands” differ by paradigm
Switches, cards
Command line
W-Icons-M-P
Relatively, direct or indirect manipulation of system
“giving commands to system”
Cmd lang: Conversational
User has a conversation with interface, telling it what to do
3rd person
User tells system what to do in task
Then system does it
Direct Manipulation: Direct
No conversation
User interacts directly with (representation) of task
1st person
Operates directly on model worlds

21. General Comments about DM Schneiderman’s Historical Account … and Definition
Shneiderman described in 1974 – will see sharper description
Positive feelings associated with “good” user interfaces often attributed to “direct manipulation” interfaces:
Mastery of the interface
Competence in performing tasks
Ease in learning the system originally and in assimilating advanced features
Confidence in the capacity to retain mastery over time
Enjoyment in using the system
“Eagerness to show the system off to novices” <- !
Desire to explore more powerful aspects of the system
Definition:
User operates on a visually displayed “world” to perform tasks in a domain 
All in all easier, to see examples, then analyze

22. Direct (and Indirect) Manipulation Interfaces
“Direct manipulation” user interfaces contrasted with,
well, “indirect manipulation” interface
Say, command line systems, or card systems, or switch systems, or …
Recall, discussion of history of computing … relative costs, technology available
But, that was a long time ago …
Paradigm example: File system manipulation
Bottom line, file system in a computer is just a collection of storage elements orientations
Even idea of “files” is an abstraction
Organization of files in a hierarchy is a further abstraction
Command line
cp /usr/fowler/presentations/colloquium_98/hypertextnodes.gif /usr/fowler/lectures/6362/dir_manip/
Tell the system to take one file and make a copy of it and place that copy in the designated place
mv hypertextgraph.gif temp_hypertextgraph.gif
Tell the system to “move” one file to another file with a different name (rename)

23. Conversations and Model Worlds… In context of direct manipulation and command language
Interface as conversation:
Interface is a language medium - user and computer have a conversation about an assumed, but not explicitly represented, world
Interface is intermediary between user and assumed world about which things are said (and in which commands are issued)
Interface as model world: 
Interface is itself a world where tuser can act and which changes state in response to user actions
World (domain) of interest is explicitly represented - there is no intermediary between user and world
Well done, this can create a sense of acting upon the object of the task domain themselves
feeling of direct engagement
Task
“work on task”
“ gives commands”
System
User
feedback
“feedback”
and its representation
(possibly manipulable)

24. Direct Manipulation “Characterized” Again, … history and fyi
An orientation
Really, direct manipulation is perhaps best characterized as an “orientation” to interface style
“Definition/orientation”, yet again:
User operates on a visually displayed “world” to perform tasks in a domain
Among the earliest characterizations (Schneiderman, 1982):
1. Continuous representation of the object of interest
2. Physical actions or labeled button presses instead of complex syntax
3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible
Benefits (Schneiderman, 1982):
1. Novices can learn basic functionality quickly, usually through a demonstration by a more experienced user
2. Experts can work extremely rapidly to carry out a wide range of tasks, even defining new functions and features
3. Knowledgeable intermittent users can retain operational concepts
4. Error message are rarely needed (why?)
5. Users can see immediately if their actions are furthering their goals, and if not, they can simply change the direction of their activity

25. Direct Manipulation “Characterized” Hutchins et al. orientation
Typical characteristics of direct manipulation interfaces:
1. System (task) objects have visual representations
2. User interacts with a model world
3. No “conversation” about what to do
the user directly manipulates the model world
4. Representations that match the user model facilitate feelings of directness
5. Output from one system action is input for another
Hutchins et al.’s account
Delineates range of interface phenomena that can contribute to feeling of directness
Assumption is that the feeling of directness results from the commitment of fewer cognitive resources
Or (put the other way around), the need to commit additional of the user’s cognitive resources leads to feelings of indirectness
And, so, account is in essence a cognitive account of direct manipulation interfaces

26. Hutchins et al. Paper Overview
Provides a detailed analysis of the elements of direct manipulation
Again, conversation metaphor and model world metaphor for interfaces
Two aspects of directness in direct manipulation:
Distance and Engagement
directness = f(distance, engagement)
Cognitive effort results from gulfs of execution and evaluation
Gulf = f (Mismatch of system elements and user’s task element)
gulf of execution
match of commands and mechanisms of system with thoughts and goals of user
gulf of evaluation
extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated
Model worlds and inter-referential input/output
Two forms of distance: Semantic and articulatory
A space of interfaces:
Engagement x Distance from user goals

27. directness = f(distance, engagement)
Two Aspects of Directness in Direct Manipulation Distance and Engagement
directness = f(distance, engagement)
Directness
The feeling that results from interaction with an interface (direct vs. indirect)
Distance
“Distance” between user’s thoughts and system’s requirements for using it to perform task
(e.g., g32, s/edtor/editor - large distance)
Short distance means translation between user’s thoughts and system is simple and straightforward
(e.g., insert ‘i’, or type over “edtor” with “editor”)
Describes factors which underlie the generation of the feeling of directness

28. directness = f(distance, engagement)
Two Aspects of Directness in Direct Manipulation Distance and Engagement
directness = f(distance, engagement)
Engagement
Feeling that the user is directly manipulating the objects of interest
e.g., dragging a subdirectory to another subdirectory vs. mv /usr/bob/*.* /usr/jo/
Again, conversation metaphor and model world metaphor for interfaces
talking about and describing an operation
vs.
performing operation in a (visual) model world (representing the task domain)

29. directness = f(distance, engagement) Gulfs of Execution and Evaluation
Distance: Relationship between user’s task and way task can be accomplished via interface
To enhance directness,
need to minimize cognitive effort required to map interface actions to task actions
i.e., “bridge the gulf of user’s goals and way they are specified in the system”
“gulfs” exist to the extent of the mismatch of user’s task intentions and how system used to carry out those task intentions
Gulfs are about 1. making things happen and 2. knowing if happened
1. gulf of execution
match of commands and mechanisms of system with thoughts and goals of user
2. gulf of evaluation
extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated

30. Directness and Model World Metaphor 1 - Cognitive Effort
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”
directness = (inversely related to, or 1/) amt of cognitive effort to manipulate system
Less effort, more directness
More effort, less directness
Leading to feelings of “direct engagement” with the task
Cognitive effort results from gulf of execution and evaluation
Explains why systems are relatively easy or hard to use, and suggests how to minimize effort, or, make system easier to learn and use

31. Directness and Model World Metaphor 2 – Controlling world vs. interface
Yet, “direct engagement” describes a qualitative feeling that user is directly engaged with control of objects (of task), rather than the computer or program which controls the task
e.g., really drawing the drawing vs. moving a cursor which, then, effects a line or fill
Characterized as feeling of “first-personess” vs. “third-personess”
User is operating on objects of task directly vs. telling someone (the system) to operate on objects
Latter is surely case in “conversation metaphor” model of user interfaces
User operates with linguistic structures (i.e., telling interface what to do)
vs.
User him/herself operating directly on the task objects (with no intermediary)
*** Central metaphor of system is the model world itself - the key to direct manipulation interfaces

32. Model Worlds and Inter-referential Input/Output
Operating in a model world, of course, requires that representations of task (model world) objects be manipulable in a direct, visual way
Additionally, if user is to carry out a series of operations (effect a series of instructions/commands which change state of system)
Then necessary that results of one operation be available for further modification
That is, the output of one operation serve as input for another operation
Term “inter-referential input/output” refers to this characteristic of interface objects
e.g., files and folders are moved and more moved again, etc.
words and paragraphs in editor undergo further modification (of course)
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”

33. Two Forms of Distance: Semantic & Articulatory directness = f(distance, engagement)
Term “language” is quite general
Refers in general to mapping (often arbitrary) of sign to meaning
Interface language - elements which cause change in system state
Two “dialects”
Language for input (causing change to occur)
Language for output (evaluating ensuing state of system),
e.g., in dm are files in new loc vs. result of mv ... (in fact none unless cd and ls)
Semantic Distance
Relationship of an expression in interface language to what user wants to say
Articulatory Distance
Relationship between meaning of expression and its physical form
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”

34. Two Forms of Distance: Semantic & Articulatory directness = f(distance, engagement)
Semantic Distance
Relationship of an expression in interface language to what user wants to say
Articulatory Distance
Relationship between meaning of expression and its physical form
E.g., dragging a file icon to another location in a file structure is small – has both small semantic (goals) and articulatory (expression (=visual form) distances
Both relatively large for command langs
Menu interface might have intermediate or less
Goals can be easy or hard to semantically map (good vs. not menu structure)
Likely, form of expression (~command) may have large dist.
Task
“work on task”
“commands”
System
User
“gives”
“performs”
“feedback”

35. Both Articulatory and Semantic Distances in Gulfs of Execution and Evaluation
Relationships among:
semantic distance
articulatory distance and
gulfs of execution and evaluation
Recall, menu example

36. Direct Engagement
Summing up, “direct engagement” occurs when user experiences direct interactions with the objects in a domain
Again, “first-personness”
no intermediary or conversation about what to do
Model world represented in interface is the task
Interface becomes “invisible” or transparent or disappears
Not clear-cut how to create
And these feeling in fact can come from automization (learning) for many (or any?) interface
Laurel (1986) suggests direct engagement requires:
Execution and evaluation should exhibit both semantic and articulatory directness
Input and output languages should be inter-referential
allows to feel as if manipulating objects of concern
Rapid system response (ideally, 0 or time appropriate for task)
“Unobtrusive” interface
interface is not to be noticed, if it is, leads to third-person relation

37. A Space of Interfaces
Direct manipulation interfaces minimize semantic and articulatory distances and maximize engagement

38. Problems with DM Interfaces
Graphical interfaces, generally, have universal usability challenges
E.g., for visually impaired
Uses much screen space for graphical representations of objects
Detail through text may be displaced
May still have to learn meaning of visual representations
Not all visual representations have “straightforward” meaning
Visual representation may be misleading
Even with analogy grasped, may still have to learn limitations
Requires switching between keyboard and mouse
Small screens with touch interfaces limit interaction

39. Challenges for DM Interfaces
Choosing “right”, or useful, interface metaphor (that has a natural visual representation) a challenge …
What is a good metaphor for air traffic control?
What is the “best” metaphor
… and for whom
E.g., file cabinet for secretaries, directed acyclic graph for computer scientists
“Rapid (<100 ms), incremental, reversable” actions using graphic representations can have high processing demands, even by current standards
E.g., high processing load db systems
Doing this web-based currently a challenge

40. Summary – Principles of DM (from Shneiderman)
Continuous representations of the objects and actions of interest with meaningful visual metaphors
Physical actions or presses of labeled buttons, instead of complex syntax
Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

41. End .

42. Fyi - Visual Thinking and Icons
Exploiting visual nature of computers
Can challenge the first generation of hackers
Individual cognitive styles vary:
Verbal vs. visual
Linear vs. holistic
“Left vs. right - brain”
Icon defined
An image, picture, or symbol representing a concept
(or a small religious object)
Guidelines for icon design
Represent the object or action in a familiar manner
Limit the number of different icons
Make icons stand out from the background
Consider three-dimensional icons
Ensure a selected icon is visible from unselected icons
Design the movement animation
Add detailed information
Explore combinations of icons to create new objects or actions
Five levels of icon design:
Lexical qualities
Syntactics
Semantics
Pragmatics
Dynamics

43. Fyi - Direct-Manipulation Programming
Visual representations of information make direct-manipulation programming possible in other domains
Example
Demonstrational programming is when users create macros by simply doing their tasks
The five challenges of programming in the user interface:
Sufficient computational generality
Access to the appropriate data structures and operators
Ease in programming and editing programs
Simplicity in invocation and assignment of arguments
Low risk
Cognitive dimensions framework may help analyzing design issues of visual programming environments.