Presentation on theme: "Recent Work in Model-Based User Interfaces"— Presentation transcript:
1 Recent Work in Model-Based User Interfaces Jeffrey NicholsLecture #1305-830: Advanced User Interface Software
2 Last time… Model-based User Interfaces Automatic generation of the user interface so the programmer won’t do a bad job.Dialog boxes are relatively easy to generateThe full application interface is hard to generateAbstract descriptions of the interface can be longer and harder to generate than implementing the interface itself.Interface builders turned out to be easier…
3 But work continued… Focus Changed A Couple Projects of Interest: Task models were leveraged moreDesign assistant aspect emphasizedA Couple Projects of Interest:TridentMecano & Mobi-DFUSEAIDE
4 TRIDENT An interface design assistant Interesting features: Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP , Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995.An interface design assistantInteresting features:Knowledge-based approach (i.e. expert system)Choosing WidgetsDoing LayoutUse of Task ModelsDecides where separate windows are needed
5 Choosing Widgets Used a decision tree Chose abstract interaction objects (AIO)Similar to Brad’s Interactor ModelLots of parametersContinuous?CapacityEtc.
6 Choosing Layout Uses Right/Bottom Strategy Next component is placed to the right or below the current componentDecision made by heuristics or designer
8 Windows from Task Models Basically used for constructing wizard-like interfacesWhat information should be on the first screen, etc.
9 What are task models, anyway? Description of the process a user takes to reach a goal in a specific domainTypically have hierarchical structureIntroduced by GOMSNumber of different task modeling languagesGOMSUANConcurTaskTrees
10 ConcurTaskTreesDeveloped by Fabio Paterno et al. for the design of user interfacesGoalsGraphical for easy interpretationConcurrent model for representing UI tasksDifferent task typesRepresent all tasks, including those performed by the system
11 Task Building Process Three phases Temporal Relationships Hierarchically decompose the tasksIdentify the temporal relationships among tasks at same levelIdentify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.Temporal RelationshipsT1  T2 - ChoiceT1 ||| T2 - InterleavingT1 || T2 - SynchronizationT1 >> T2 - EnablingT1 >> T2 - Enabling with Information PassingT1 [> T2 - DeactivationT1* - IterationT1(n) - Finite Iteration[T1] - OptionalT – Recursion
14 Building/Editing Task Models Tools are availableConcurTaskTrees Environmentor Google for “ConcurTaskTrees”
15 Recent Systems XIML – eXtensible Interface Markup Language XWeb Developed by the makers of Mecano/Mobi-D and TridentKitchen-sink language for modeling any part of the interface design processXWebNow known as ICE – Interactive Computing EverywhereICrafterA system for integrating user interfaces from multiple devicesPersonal Universal ControllerMy research…
16 XIML eXtensible Interface Markup Language Designed by RedWhale SoftwareIntended to support the full lifecycle of interface building
17 XIML Requirements Central Repository of Data For one user interface or manyComprehensive Lifecycle SupportAbstract and Concrete ElementsRelational SupportUnderlying TechnologyXIML must be independent of particular tools
18 Models in XIML An XIML document can contain any type of model Task DomainUserPresentationDialog
19 Example Use for XIMLMulti-platform interface development
20 Status of XIMLUsed by RedWhale Software to drive their interface consultant businessThey have developed many toolsmove interaction data to/from XIMLLeverage data in XIML to better understand various interfacesAutomate parts of the interface design process
21 Model-Based Interfaces for Control XWebICrafterPUC
22 XWeb Work by Dan Olsen and group at BYU Premise: “Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”The web comes close to solving this problem, but is interactively insufficient.
23 XWeb Protocols Based upon the architecture of the web XTP Interaction ProtocolServer-side data has a tree structureStructured Data in XMLURLs for location of objectsxweb://automate.home/lights/livingroom/xweb://automate.home/lights/familyroom/-1
24 XWeb & XTP CHANGE message (similar to GET in HTTP) Sequence of editing operations to apply to a sub-treeSet an attribute’s valueDelete an attributeChange some child object to a new valueInsert a new child objectMove a subtree to a new locationCopy a subtree to a new location
25 Platform Independent Interfaces Two models are specifiedDataView – The attributes of the serviceXView – A mapping of the attributes into high-level “interactors”Interactors are somewhat like abstract interaction objectsAtomicNumericTimeDateEnumerationTextLinksAggregationGroupList
29 Other XWeb DetailsHas simple approach for adjusting to different screen sizesShrink portions of the interfaceAdd additional columns of widgetsAlso capable of generating speech interfacesBased on a tree traversal approach like Universal Speech Interfaces
30 ICrafterPart of the Interactive Workspaces research project at StanfordMain objective:“to allow users of interactive workspaces to flexibly interact with services”ContributionAn intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.In practice, much of the interface generation is done by hand though automatic generation is supported.
32 How is aggregation accomplished? High-level service interfaces (programmatic)Data ProducerData ConsumerThe Interface Manager has pattern generatorsRecognize patterns in the services usedGenerate interfaces for these patternsThis means that unique functionality will not be available in the aggregate interface
35 Personal Universal Controller My work with BradProblem:Appliance interfaces are too complex and too idiosyncratic.Solution:Separate the interface from the appliance and use a device with a richer interface to control the appliance:PDA, mobile phone, etc.
36 Idea Control existing appliances Generate multi-modal interfaces SpecificationsControlFeedbackControl existing appliancesGenerate multi-modal interfaces
38 Language Design Approach Create reference interfaces AIWA Shelf StereoAT&T Telephone/Answering MachineTest interfaces with subjectsUsers twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfacesAnalyze interfaces for functional informationBe careful of saying “same manufacturer’s appliances”
39 Language Elements State Variables and Commands Label Information Represent functions of applianceState variables have typesBoolean, Enumeration, Integer, String, etc.Variables sufficient for most functions but not all“seek” button on a RadioLabel InformationOne label not suitable everywhereThe optimal label length changes with screen sizeSpeech interfaces may benefit from pronunciation and text-to-speech information
40 Language Elements, cont. Group TreeSpecify organization of functionsWe use n-ary tree with variables or commands at leavesNeed a place for things like “no specific layout information” related work or new high-level spec slide
41 Language Elements, cont. Dependency InformationFormulas that specify when a variable or command is active in terms of other state variablesEquals, Greater Than, Less ThanLinked with logical operators (AND, OR)For example,<and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals> </and>
42 Interface Generators Generators for Two Modalities Graphical Speech Implemented for PocketPC in Java 1.1Uses dependency information to generate panel structure of interfaceSpeechImplemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)
43 Graphical Interface Generator Focuses on panel structure of user interfaceSmall groups of controls have basic layoutsComplexity comes from structure of groupsStructure can be inferred from dependency info!decomposed
44 Inferring StructureFind sets of variables that are “mutually exclusive”Every variable in a set will never be active at the same time as a variable in another setCreate structure with sets, using overlapping panels
46 Making the Interface Concrete Finish conceptual layoutChoose controls (decision tree)Choose row layouts(one column, two column, etc.)Allocate spaceExamine panel contents and choose sizesInstantiate and place controls
47 Generating Speech Interfaces Automatically build USI tree from dependenciesAllows verbal navigation of functional groupsAutomatically generate grammar for parserPhrases for query and control“What is playmode?”“Set playmode to play”“play”Automatically generate language model and pronunciation for recognizer