1. Evaluating Visual and Statistical Exploration of Scientific Literature Networks Robert Gove 1,3, Cody Dunne 1,3, Ben Shneiderman 1,3, Judith Klavans 2, and Bonnie Dorr 2,3 1 HCIL, 2 CLIP, 3 Computer Science University of Maryland, College Park, USA

2. How do we help users explore scientific literature domains?

3. Commonly available capabilities Document statistics Keyword summary Citation visualization Corpus statistics Full-text search

4. Three types of systems Search engines & digital libraries Reference managers Summarization & recommender systems

5. Taxonomy of System Functionalities

6. Action Science Explorer Visual and statistical exploration

9. Evaluation Corpus of 147 papers on Dependency Parsing Exploration-based usability testing – 4 domain experts, two hours each 1.Training 2.Predefined tasks 3.Exploration questions 4.Participant-defined tasks

10. 1. Training 15-30 minutes Showed videos demonstrating Action Science Explorer Answered participant questions Gave think-aloud instructions

11. 2. Predefined tasks Number of participants Capability Task 1: Identifying important authors and papers Task 2: Collecting evidence Attribute Ranking ███ 3██ 2 Find Communities █ 1 Plot Nodes In-Cite Text █ 1 In-Cite Summary Search █ 1 Sorting █ 1 Groups ██ 2█ 1

12. 3. Participants’ questions How has the field evolved? What are subtopics within the field? What are the most accessible papers? What are the relationships between authors? What are the methods and applications?

13. 4. Participant-defined tasks Number of participants Capability All participant-defined tasks Attribute Ranking ██ 2 Find Communities █ 1 Plot Nodes In-Cite Text █ 1 In-Cite Summary Search █ 1 Sorting █ 1 Groups █ 1

14. Answering participants’ questions 2/4 questions were answered – 1 unanswered question was uncompletable – 1 unanswered question not well supported

15. Design recommendations for exploration Confirmed design decisions – Provide metrics for ranking documents – Support user-created groups and document annotation – Give overview of corpus Suggested new capabilities – Allow control over corpus refinement – Incorporate undo/redo functionality

16. Conclusion A taxonomy of capabilities in current tools An evaluation of Action Science Explorer A list of literature exploration questions A list of system design recommendations Robert Gove - rpgove@gmail.com

17. Acknowledgements Supported by NSF grant IIS 0705832 and Booz Allen Hamilton Dragomir Radev and Vahed Qazvinian, University of Michigan David Zajic and Michael Whidby, University of Maryland