1. Designing Systems to Support Document Triage Frank Shipman Center for the Study of Digital Libraries Texas A&M University

2. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions

3. Document Triage The practice of quickly determining the merit and disposition of relevant documents in a collection of documents Common aspects –Selection from information repository via querying and browsing interfaces –Extensive, hyper-extensive and intensive reading and viewing in reading interfaces –Collection and interpretation of resources in organization interface –Mode switching / attention shifting

4. Information Work Variety of information tasks –Short-term: Facts and references What is the escape velocity? –Long-term: Analysis and synthesis How to design a space craft? For longer-term information activities the work really begins after potentially relevant materials are located.

5. Information Life-Cycle Location: Searching and Browsing Comprehension: Skimming and Reading Located resources must be understood to be evaluated Understanding one document may require other documents or result in further information requests Modification: Annotation & Authoring Reading results in annotation, note taking, and writing Added content influences further access Modified from work on software libraries: [Fischer, Henninger, Redmiles 1991]

6. Document Triage We want to look at situations where people are reading more than one document at once Document triage places different demands on attention than single- document reading activities Continuum of types of reading: working in overview (metadata), reading at various levels of depth (skimming), reading intensively

7. Library Table as Success Model How people make use of library resources can give design goals. Characteristics of the library table: –Integration and easy differentiation of source materials and personal interpretation –Implicit and explicit expression via spatial layout and attached annotation –Patrons can collaborate using the materials on the table as a prop for their conversation Limitations: The library table and resources are shared/limited resources, so must be cleaned up after each work session.

8. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions

9. Spatial Hypertext and Document Triage Spatial hypertext – where inter-document relationships are expressed via visual and spatial cues rather than links. Earlier study compared use of two variations of the VIKI spatial hypertext system with paper [Marshall, Shipman 1997] Results showed that –people use the affordances of the medium provided –those working with paper read more –those working with VIKI organized more

10. Visual Knowledge Builder (VKB) VKB is a second generation spatial hypertext –greater support for collaborative and long-term tasks –navigable history –explicit (as well as implicit) links VKB provides: –A hierarchy of two-dimensional workspaces called collections for placing information –Easy manipulation of visual properties of information –Information objects pointing to external content –Attribute/value pairs for attaching metadata –Integrated search for Google and NSDL

12. Personal Collection Creation and Use Getting content in VKB –Embedded Search for NSDL and Google –Drag-and-drop file system folders –Metadata peeling for files, jpg, mp3, search results Comprehension and modification of content –Metadata visualization of NSDL search results –Metadata extraction and applicators –Mouse-based browsing of content (including mp3 collections)

17. Metadata Extraction and Application Goal: to allow easy and consistent metadata authoring. Select objects as source for extracting metadata attributes and values Menubar of applicators is updated to allow attaching same metadata to other objects.

18. Metadata Profiles Metadata applicators can be saved in profiles –Profiles stored in VKB datafile and in user’s VKB settings for reuse. –Profiles are easily swapped out.

19. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions

20. Study of VKB Use for Selecting and Organizing Materials Study designed to understand how spatial hypertext would change work practices when accessing a digital library. Decided to look at document triage –deciding what to keep –expressing an initial view of relationships

21. Task: 16 subjects placed in role of a reference librarian, selecting and organizing information on ethnomathematics for a teacher Setting: top 20 search results from NSDL & top 20 search results from Google 16 subjects were divided into two groups of 8: * Initial search done by us Subjects given as much time as they deemed necessary (after training for VKB users) Study Setup VKB (VKB/IE)Control (IE/Editor) Search *VKBIE ReadingIE OrganizationVKBEditor

23. Results Data collected –Demographic information –Questionnaire about experience –Videos of screen activity –VKB files (with history) for VKB users Analysis of activity –All subject organized links into labeled categories.

27. Perception of Activity and Results VKB/I E IE/Edit or (p) I was able to organize everything as I wanted 3.632.630.064 Easy for someone to understand my organization 4.133.250.132 Five point Likert scale where 1 is “strongly disagree” and 5 is “strongly agree” VKB group: –felt more able to organize the content as desired –that their organizations would be more understandable to others

28. Time, Selection, Organization VKB/IEIE/Editor(p) Time spent on the task in minutes52.8843.000.315 Number of links kept34.6318.380.003 Number of links kept from NSDL17.138.130.002 Number of links kept from Google17.5010.250.015 Number of collections9.635.000.062 Number of top level collections4.754.000.506 Number of levels of collections2.001.380.032 Little difference in time spent on task VKB participants –kept more links –created deeper organizations of categories

30. IE/Editor Authoring Activities VKB/IEIE/Editor(p) Percentage of subjects in group that added personal comments 0.0037.500.080 Percentage of subjects in group that copied and pasted text from web 12.5050.000.124 Percentage of subjects in group that processed links in the order presented 12.5062.500.043 Percentage of subjects in group that changed links or added new ones 25.0050.000.335 IE/editor participants more likely to: –Add comments about resource –Select parts of resource for teacher –Visit links in order

32. Discussion & Caveats Initial metadata visualization seems to cause users to avoid changing visual semantics –Compared to 1997 study of VIKI use, VKB users did not express interpretation via color Some effects may be due to experience –IE/editor participants were using their normal tools compared with novice VKB users. –Training did not show how to drag-and-drop portions of Web pages into VKB space. Study suggests value in spatial hypertext for collecting and organizing information resources

33. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions

34. Attention Switching in Document Triage Document Triage Recap: –different demands on attention than single- document reading activities –people are reading more than one document at once –people switch between reading and organizing –transitions generate potential for breakdown Question: Can a dedicated reading surface make a difference in how people engage with content during triage?

35. A second look at the earlier data… Subject ID1234567 Total time1:04:080:54:140:21:590:22:481:33:281:20:091:03:481:01:43 8 Number of transitions 134287881981068790 Overview in VKB/Content displayed in IE, and transitions between the two Average number of transitions between applications: 88 VKBIE Total time (seconds) 18,8747,596 71%29% Average time looking at window (seconds) 4720 Summary Average time on the task (minutes): 58 % total time (in app) more than 2/3 of the time is spent organizing references time spent reading unfamiliar material is very brief window management is extremely time-consuming!

36. Document Triage—Starting Point Given reduced representations of multiple relevant documents (e.g. a list of search results or email headers), people don’t spend much time reading (or even skimming) –Lots of time is spent managing screen space and windows (opening, closing, reshaping, etc.) – might people be trying to minimize that? When people are overwhelmed in this way, there’s a tendency to work from metadata instead of content, manipulating and organizing it –Think about how we handle our email (especially spam, but others as well) –Think about how we decide to follow a link from a list of search results (perhaps using poor or deceptive metadata) We’d like to give people a chance to read more, focus their attention, and spend less time managing windows: what happens if we give readers a dedicated reading surface like a tablet computer?

37. Duplicate task – subjects act as reference librarians, sifting through ethnomathematics material from the NSDL (National Science Digital Library) and Google Envisioned technology scenario: Associated technology prototyping – to develop infrastructure for the study and to investigate heuristic techniques for assessing interest through action Initial information triage study setup to answer our research question

38. Infrastructure Development Our envisioned technology scenario (tetherless tablet with extended display for overview and organizing) didn’t work –Controlling Windows on a second screen using a pen is not easy. –Pushed infrastructure development to create a case close enough to envisioned scenario –Extended desktop was sufficient for two cases that didn’t use a tablet computer Infrastructure: screen control of windows displayed on different computers –Push and pull selected windows between different computers Logging and instrumentation – capturing events of interest

39. Study Configurations Display Configuration Input DevicesAssignment of Activity Laptop and tabletop LCD display Extended desktop controlled via keyboard and mouse User controls which windows are on which display Laptop and projected display Extended desktop controlled via keyboard and mouse User controls which windows are on which display Tablet computer and projected display Projected display controlled via keyboard and mouse, tablet computer controlled via pen Software assigns document overview to projected display and IE to tablet

40. Data Sources Wide variety of data captured: video capture of environment (subject doing the task) continuous screen capture of both displays demographic profile of participants interviews and questionnaires about task, technology, and resources including identifying 5 most useful and 5 least useful documents activity logs (for IE and for VKB)

41. A First Look at the New Data Configuration Prior StudyCurrent Study Desktop PC Laptop & LCD Screen Laptop & Projected Display Tablet PC & Projected Display # of Displays1222 Avg. Total Time3,3093,5543,6424,234 Avg. Time (VKB)2,3592,4532,6273,005 Avg. Time (IE)9501,1021,0151,229 Avg. # of Transitions (shifts of focus) 97193168205

42. Time Spent in IE (glancing, skimming, reading) One DisplayTwo Display tendency: in the 2 display condition, there’s a greater number of brief encounters; might represent more glances, more checking more revisits?

43. Questionnaires & Interviews Subjects with laptop and extra screen felt most comfortable of the multiple display configurations Tablet computer was rated lowest in all questions concerning ease of use or enjoyment Preference for multiple displays at the same focal length Subjects found size of projected display and pen interactions annoying

44. Did Reading Actions Correlate with Document Preferences? Top five and bottom five documents were identified Log files recorded user actions in IE A number of user actions were significantly correlated with document preference

45. User Actions vs. User Interests (1) time spent (2) # of following embedded links (3) # of visits (4) text selections (5) clicks (6) scrolls 0.532 -0.334* 0.480 0.331 0.354 0.632 p < 0.0001 p = 0.034 p = 0.049 p = 0.003 p = 0.040 p = 0.001 Pearson Coefficient

46. Discussion Caveat: this data comes from a single domain and document set –Need to explore other document sets –Need to investigate effect of domain and subject matter expertise on task performance This is purely activity in reading interface, not organizing interface.

47. Display Configuration Summary Number of transitions between applications almost doubled for multiple display configurations No significant difference between display configurations although subjects did express a preference for two side-by-side displays Scrolling, time spent on document, and number of visits to document were correlated with document preference

48. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions

49. User Interest and Document Value from Reading and Organizing Activities Recognizing user interest & document value Representing user interest Recognizing documents of potential interest Visualizing interest information

50. User Interest Explicit interest indicators –Precise, easy to implement –Distraction, cognitive load, fewer result Implicit interest indicators –Reading activity –Annotation activity

51. Motivation People spend much time for documents that they finally evaluate as not useful Understanding of user interests on documents could be the basis for active supporting document triage User activity in reading & organizing implicitly represent user interests on documents

52. Motivation (cont.) Observed significant differences between individual styles in reading & organizing Observed no dominant factor to determine user interests Combining partially identified user interests from multiple applications could more accurately recognize user interests

53. Interest Profile Manager (1) Infrastructure for sharing information about user activity between multiple application Estimation of user interests from user activity

54. Interest Profile Manager (2) Flexible information structure to handle various user activity from multiple applications

55. System Architecture Interest Profile Manager VKB Instrumen ted Internet Explorer

56. Data Sources Document (Web page) attributes (3) Number of characters, number of links, … User events from reading activity (10) Reading time, scrolls, clicks… User events from organizing activity (14) Moving symbol, resizing symbol, … VKB document object attributes (6) Object location, object size, depth, …

57. User Model (1) Predict a general pattern of user interests on documents User activity vs. user interests Statistical & qualitative approach

58. User Model (2) Statistical model 1 (reading-activity) 0.877 + 0.133 * factor1 + 0.120 * factor2 Statistical model 2 (organizing activity) 0.877 + 0.185 * factor1 – 0.092 * factor2 Statistical model (combined) 0.877 + 0.125 * factor1 + 0.152 * factor2 + 0.0662 * factor3 + 0.0653 * factor4 * Factors of different models are different from each other

59. User Model (3) Factors for the combined activity model

60. User Model (4) Result when three models are used with the same data set ModelRR Square Adjusted R Square Reading0.6900.4770.444 Organizin g 0.7970.6360.613 Combined0.8410.7080.669

61. User Model (5) Qualitative model (14)

62. Comparison of Models Result when four models are used with different data set ModelAverage ErrorStandard Deviation Reading0.2580.192 Organizing0.2160.146 Combined0.1760.138 Qualitative0.1970.134

63. Results User activity in reading & organizing often corresponds to user interests and can be the basis for supporting document triage Combined activity model is better than all the other models Combining partially identified user interests from multiple applications can be the basis for more accurate estimation of user interests

64. Outline What is Document Triage? Spatial Hypertext and VKB VKB and Document Triage Effects of Display Configuration Recognizing User Interest / Document Value Current Directions