Instrumenting the Learning Commons Eugene Agichtein, Qi Guo and Ryan Kelly Intelligent Information Access Lab, Math & CS Department Arthur Murphy, Selden Deemer, Kyle Fenton Emory Libraries

2 Intelligent Information Access Lab Goals/Motivation  Evaluate effectiveness of search/discovery with behavioral metrics (task-specific) Perform aggregate, longitudinal studies  Tools for usability studies “in the wild” Scale (hundreds/thousands of “participants”) Realistic behavior and tasks On-demand playback of “interesting” sessions  Unified analysis/query framework for internal and external resource access and usage statistics Web-based query and statistics interface Access auditing, privacy, anonymity enforced

3 Intelligent Information Access Lab Approach: Client-side instrumentation  Implementation within the Emory Installation of the LibX Toolbar: (  Extended LibX to track UI events: JavaScript patch to sample the mouse movements and other events on pre-specified web search pages. Events are encoded into a string and buffered, and periodically sent to the server (on internal library network).

4 Intelligent Information Access Lab How it works  On login, firefox is started with  If user has previously opted in (or out) Redirect to Euclid homepage  If new user, show consent form  Store choice in database; if opted in, also store salted hash string for user log in Can track user behavior over “lifetime” No way to recover login id by dictionary attack Can be removed at any time by deleting mapping  LibX sends http requests to server with encoded event strings.

5 Intelligent Information Access Lab User Opt-in (new Learning Commons Users)

6 Intelligent Information Access Lab Events captured (v0.4, deployed)  Button/link clicks/Url changes Name of the button, link, other meta-info  Mouse movements (x,y) coordinates sampled ~every 10ms  Scrolling Start, stop position, ~ every 10ms  Text entry Query text, options changes  Keypress events  Menu item events Print, bookmark, save (all of them)

7 Intelligent Information Access Lab Example: Mouse Movement  “Cheap” proxy for eye- tracking: Capture physiological characteristics of the mouse trajectories  Segment Properties: Speed Acceleration Rotation Drift Hover 5 segments: initial, early, middle, late, and end. For each: avg. speed, avg. acceleration, rotation etc.

8 Intelligent Information Access Lab Example applications: Classifying Search Intent  Initial exploration: Standard supervised machine learning classification techniques WEKA implementation of SVM and decision trees.

9 Intelligent Information Access Lab Summary of Features Used

10 Intelligent Information Access Lab Example Informational query: “spanish wine”

11 Intelligent Information Access Lab Emory User Behavior Analysis System  EUBA: Client-side instrumentation, Data mining/machine learning (Qi Guo) Log DB parsing, indexing, web-based interface for querying, playback, annotation (Ryan Kelly)  Plan: to release the system to research/library community

12 Intelligent Information Access Lab Demo Prototype: user: test password: notsafe