UCI Expectation-Driven Event Monitoring (EDEM) David Hilbert, David Redmiles

UCI Expectation-Driven Event Monitoring (EDEM) Complex Systems Evolve from Use but Use Information is Difficult and Expensive to Collect –Requirements of end users and their work environments are difficult to anticipate. –Rapid and distributed deployment of systems (e.g., over the internet) reduces the opportunities for traditional user testing while increasing the variety of use situations and number of end users. Expectation-Driven Event Monitoring enables Developers to Easily and Cheaply Benefit from Use Information –Developers identify usability expectations, create agents to monitor user interactions, agents are deployed with applications. –Agents detect mismatches in expected versus actual systems use. –Agents monitor use passively or allow users to provide feedback. Need Solution

UCI Agents in Evolutionary Design

UCI Expectation Agents Approach Comparison of “actual use” with “expected use”: –Designers’ expectations are represented in agents –Agents observe users’ actions and recognize mismatches with the designers’ expectations –After a mismatch, an agent interacts with the user and returns the situation and the users comments to the developers –Agents may also allow authorized, unobtrusive observation of users. –Agents free developers from constantly observing users interacting with prototypes and therefore more users can be involved

UCI EDEM Approach

UCI Travel Voucher Scenario The end user violates the expectation that the ZIP field is entered before the City or State:

UCI Notification Action Agents may post messages to inform users of expectation violations; users may respond:

UCI End User Response The user suggests a potential design improvement; feedback is reported to developers:

UCI Repository for Review Users’ responses and other and documents are categorized for devlopers’ reviews:

UCI Unobtrusive Agent Monitoring Agents may monitor user patterns without intruding on their tasks:

UCI Developers Define Expectations Developer defines an agent to “fire” whenever a user edits the Street or City fields while the ZIP field is empty:

UCI Selecting Events Developer expresses interest in detecting when a user begins editing the Street field:

UCI Agent Global Properties Agents can be loaded automatically via URL and results sent to developers via

UCI How are agents represented and triggered? Agents are represented as instances of a Java class w/ the following members: (1) An EventPattern and list of Events (2) A ConditionPattern and list of Conditions (3) A set of Actions to perform Agents are triggered in the following way: (1) If the EventPattern occurs, then the ConditionPattern is checked. (2) If the ConditionPattern is true, then the Actions are performed.

UCI Event Patterns An EventPattern is one of the following: (1) "A or B or... " (2) "A and B and... " (3) "A then B then... " (4) "(A and B) with no interleaving C" (5) "(A then B) with no interleaving C" Where variables A,B,C are filled in by specifying: (1) a Component from the UI plus an Awt or EDEM event on that component (e.g. "TextField1:LOST_EDIT" which occurs when TextField1 is edited and then input focus shifts and editing begins in another component) (2) another Agent (e.g. "AddressDone" which occurs when another agent detects that the address section has been completed)

UCI Condition Patterns A ConditionPattern is one of the following: (1) "A or B or... " (2) "A and B and... " Where variables A,B are filled in by specifying: (1) a Component from the UI and some expression involving its properties (e.g. "TextField1:value='Married'" or "Button1:count>100") (2) another Agent and some expression involving its properties (e.g. "AddressStarted:enabled=true" or "AddressDone:count>1")

UCI EDEM Architecture EDEM server monitoring other instances of EDEM Event Repository (Oracle, Sybase, Knowledge Depot) EDEM monitoring an Applet (server machine) (auxiliary server machine) (client machines)

UCI Integration with Different Software Technologies Event Monitoring, Software Process, and Perpetual Testing Design Requirements, Rationale, and Organizational Memory Design Environments

UCI Critics: Opportunities for DR Use & Capture

UCI Summary Conclusions Software evolves with feedback Usability feedback is needed to reduce training costs, organizational costs, and risk of error. Expectation Agents reduce cost of and increase the amount of usability feedback. Agent reports combine with other sources of rationale in an organizational memory. Expectation Agents help integrate design, usage, and rationale.