1. A Sketch Interface for Mobile Robots
Marjorie Skubic
Craig Bailey
George Chronis
Computational Intelligence Research Lab
University of Missouri-Columbia

2. Outline Motivation and context Route maps The PDA sketch interface
Experimental study and results
Conclusions and future work

3. Spatial Reasoning with Guinness
References
Acknowledgements

4. Route Maps Tversky’s work Michon and Denis Depictions vs. Descriptions
Extraction of route descriptions
1 to 1 correlation
Michon and Denis
Landmarks and critical nodes

5. The Sketch Interface
Objects
Labels
Paths
Delete
Start
Move
Undo
Send

6. Objects Closed Polygons Any shape or size
Thresholds to determine gap closure
Feedback on recognition
Sound
Color

7. Labels Default numbering for object labels Tap on screen to edit
Can use Palm OS Graffiti recognition or a software keyboard

8. Paths
Limit of one
A minimum length required
Color Feedback

9. Path Direction Default direction is the direction the path is drawn
User can specify the direction with a sketched “blob” to denote the start of the path
Recognized by
Number of points
Average distance of all points
Proximity to path endpoint

10. Delete An intuitive delete: cross out an object Recognized by
Two consecutive strokes
Both lengths shorter than a path
The strokes cross
Color feedback
Search for closest object or path

11. Determining Crossed Marks
Use the slope equations of lines
Endpoints of strokes determine the line
A pair of decision parameters can be computed
If both parameters lie between 0 and 1, then the two strokes must have an intersection
ua= (X4-X3)(Y1-Y3) - (Y4-Y3)(X4-X3)
(Y4-Y3)(X2-X1) - (X4-X3)(Y2-Y1)
ub= (X2-X1)(Y1-Y3) - (Y2-Y1)(X1-X3)
IF (0 < ua < 1) AND (0 < ub < 1) THEN
the lines intersect
(X1,Y1)
(X3,Y3)
(X4,Y4)
(X2,Y2)

12. Menu Commands Also accessible through graffiti m  Move u  Undo
c  Clear
t  Transmit
f  Configure

13. “Digitizing” the Sketch

14. User Evaluation
Tested how well the interface performed with real users
Pre-experimental questionnaire
Tasks
Sketch tasks
Re-sketch tasks
Task scores
Post-experimental questionnaire
Questionnaires contain Lickert style statements (Lickert, 1932) along with several open-ended questions

15. Statistical Analysis 2 groups, 2 scenes: Compared by scene sketched
Compared by course level of participant
Means compared with the t test
Null Hypothesis: there are no differences when compared by sketched scene or course level

16. Participants 26 students from CS courses
One participant scores was not used
Only 5 owned a PDA
Students of Scene B rated themselves significantly better at giving directions (p = 0.02)
No differences when compared by course level

17. Scene A

18. Example Sketches of Scene A

19. Scene B

20. Example Sketches of Scene B

21. Post-Experimental Survey: Landmark Scores 1 = very difficult; 5 = very easy
Creating Landmarks
4.6 ± 0.6
Deleting Landmarks
4.2 ± 0.9
Usefulness of Deleting
4.7 ± 0.6
Usefulness of Labeling
4.8 ± 0.6

22. Post-Experimental Survey: Path Scores 1 = very difficult; 5 = very easy
Creating a path
4.4 ± 1.0
Deleting a path
Usefulness of deleting
4.7 ± 0.7
Usefulness of the starting point
4.2 ± 0.9

23. Post-Experimental Survey: Overall Scores
Usefulness of changing sketch
4.8 ± 0.4
Usefulness of deleting sketch
4.2 ± 1.0
How well sketch represents environment
83.6 ± 7.4
Overall ease of interface
4.4 ± 0.6

24. Usability Results
Only two significant differences (p<=0.05) were found among the scores
Usefulness of deleting by scene (p=0.0)
Final sketch rating by scene (p=0.05)
In both cases, students in scene B rated higher
Same group that rated themselves better at giving directions
Differences were not found when compared by course level
The Null Hypothesis is accepted

25. Task Score Results Collected sketches were scored
+1 for starting landmark
+1 for each correct turn
+1 for landmark at turn
+1 for each correct straight segment
+ 1 for ending landmark
-1 for extra turns or straight segments
No significant differences found (p=0.12)
Sketch Task Score = 0.91 ± 0.11
Re-sketchTask Score = 0.82 ± 0.26

26. Conclusions
Created a new sketch based interface on a handheld computer
Intuitive and little reliance on traditional menus and icons
User evaluation finds the interface as easy to use as pencil and paper by 2:1

27. Future Work Continue integration into the Guinness system
Recognition of more sketched symbols
Recognition of turning rate
Creation of 3D virtual environments with libraries of objects
Web:
funded by the Naval Research Lab

28. ARCHITECTURE PDA SRserver imageserver speech mapserver spatial
behaviors
obstacle
avoidance
pose
continuous
localization
speech
robot
Cortex
sensor data
robot cmds
corrections
oldest short term map
user commands
and responses
SR &
map info
robot pose
encoders
sensor info
query &
label
commands
sketch
directives
& feedback
robot commands
trulla
vfh
short
term
map
long
GUI(EUT)
gesture
PDA

29. SRserver User: How many objects do you see?
Behind the table
User: How many objects do you see?
Robot: I am sensing four objects.
User: Object 2 is a table.
User: Describe the scene.
Robot: There are objects on my front right. The object number 4 is mostly in front of me. The table is behind me.
User: Go behind the table.

30. between object 1 and object 2
using the midpoint between closest points
using the midpoint between centroids
using the CFMD

31. Image Server

32. Understanding Sketched Route Maps
PATH DESCRIPTION GENERATED FROM THE SKETCHED ROUTE MAP
1. When table is mostly on the right and door is mostly to the rear (and close) Then Move forward
2. When chair is in front or mostly in front Then Turn right
3. When table is mostly on the right and chair is to the left rear Then Move forward
4. When cabinet is mostly in front Then Turn left
5. When ATM is in front or mostly in front Then Move forward
6. When cabinet is mostly to the rear and tree is mostly on the left and ATM is mostly in front Then Stop

33. References
[1] M. Skubic, P. Matsakis, G. Chronis and J. Keller, "Generating Multi-Level Linguistic Spatial Descriptions from Range Sensor Readings Using the Histogram of Forces", Autonomous Robots, Vol. 14, No. 1, Jan., 2003, pp
[2] M. Skubic, D. Perzanowski, S. Blisard, A. Schultz, W. Adams, M. Bugajska and D. Brock “Spatial Language for Human-Robot Dialogs,” IEEE Transactions on SMC, Part C, to appear in the special issue on Human-Robot Interaction.
[3] M. Skubic, S. Blisard, C. Bailey, J.A. Adams and P. Matsakis, "Qualitative Analysis of Sketched Route Maps: Translating a Sketch into Linguistic Descriptions," IEEE Transactions on SMC Part B, to appear.
[4] G. Chronis and M. Skubic, “Sketch-Based Navigation for Mobile Robots,” In Proc. of the IEEE 2003 Intl. Conf. on Fuzzy Systems, May, 2003, St. Louis, MO.
[5] G. Scott, J.M. Keller, M. Skubic and R.H. Luke III, “Face Recognition for Homeland Security: A Computational Intelligence Approach,” In Proc. of the IEEE 2003 Intl. Conf. on Fuzzy Systems, May, 2003, St. Louis, MO.

34. Guinness and Gang From left to right
George Chronis, Grant Scott, Dr. Marge Skubic, Matt Williams,
Craig Bailey, Bob Luke, Charlie Huggard and Sam Blisard
Missing: Dr. Jim Keller

35. Sketch-Based Navigation
The robot traversing the sketched route
The sketched route map

36. Sketch-Based Navigation
The robot traversing the sketched route
The digitized sketched
route map

37. Acknowledgements
This work has been supported by ONR and the U.S. Naval Research Lab. Natural language understanding is accomplished using a system developed by NRL, called Nautilus [Wauchope, 2000]. We also want to acknowledge the help of Dr. Pascal Matsakis.

38. NRL’s Multimodal Robot Interface