Presentation on theme: "Recent Work in Model-Based User Interfaces"— Presentation transcript:
1Recent Work in Model-Based User Interfaces Jeffrey NicholsLecture #1305-830: Advanced User Interface Software
2Last time… Model-based User Interfaces Automatic generation of the user interface so the programmer won’t do a bad job.Dialog boxes are relatively easy to generateThe full application interface is hard to generateAbstract descriptions of the interface can be longer and harder to generate than implementing the interface itself.Interface builders turned out to be easier…
3But work continued… Focus Changed A Couple Projects of Interest: Task models were leveraged moreDesign assistant aspect emphasizedA Couple Projects of Interest:TridentMecano & Mobi-DFUSEAIDE
4TRIDENT An interface design assistant Interesting features: Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience, Technical Report RP , Facultes Universitaires Notre-Dame de la Paix, Institut d’Informatique, Namur, 1995.An interface design assistantInteresting features:Knowledge-based approach (i.e. expert system)Choosing WidgetsDoing LayoutUse of Task ModelsDecides where separate windows are needed
5Choosing Widgets Used a decision tree Chose abstract interaction objects (AIO)Similar to Brad’s Interactor ModelLots of parametersContinuous?CapacityEtc.
6Choosing Layout Uses Right/Bottom Strategy Next component is placed to the right or below the current componentDecision made by heuristics or designer
8Windows from Task Models Basically used for constructing wizard-like interfacesWhat information should be on the first screen, etc.
9What are task models, anyway? Description of the process a user takes to reach a goal in a specific domainTypically have hierarchical structureIntroduced by GOMSNumber of different task modeling languagesGOMSUANConcurTaskTrees
10ConcurTaskTreesDeveloped by Fabio Paterno et al. for the design of user interfacesGoalsGraphical for easy interpretationConcurrent model for representing UI tasksDifferent task typesRepresent all tasks, including those performed by the system
11Task Building Process Three phases Temporal Relationships Hierarchically decompose the tasksIdentify the temporal relationships among tasks at same levelIdentify what objects are manipulated and what actions can be performed on them, and assign these to the tasks as appropriate.Temporal RelationshipsT1  T2 - ChoiceT1 ||| T2 - InterleavingT1 || T2 - SynchronizationT1 >> T2 - EnablingT1 >> T2 - Enabling with Information PassingT1 [> T2 - DeactivationT1* - IterationT1(n) - Finite Iteration[T1] - OptionalT – Recursion
14Building/Editing Task Models Tools are availableConcurTaskTrees Environmentor Google for “ConcurTaskTrees”
15Recent Systems XIML – eXtensible Interface Markup Language XWeb Developed by the makers of Mecano/Mobi-D and TridentKitchen-sink language for modeling any part of the interface design processXWebNow known as ICE – Interactive Computing EverywhereICrafterA system for integrating user interfaces from multiple devicesPersonal Universal ControllerMy research…
16XIML eXtensible Interface Markup Language Designed by RedWhale SoftwareIntended to support the full lifecycle of interface building
17XIML Requirements Central Repository of Data For one user interface or manyComprehensive Lifecycle SupportAbstract and Concrete ElementsRelational SupportUnderlying TechnologyXIML must be independent of particular tools
18Models in XIML An XIML document can contain any type of model Task DomainUserPresentationDialog
19Example Use for XIMLMulti-platform interface development
20Status of XIMLUsed by RedWhale Software to drive their interface consultant businessThey have developed many toolsmove interaction data to/from XIMLLeverage data in XIML to better understand various interfacesAutomate parts of the interface design process
21Model-Based Interfaces for Control XWebICrafterPUC
22XWeb Work by Dan Olsen and group at BYU Premise: “Pervasive computing cannot succeed if every device must be accompanied by its own interactive software and hardware…What is needed is a universal interactive service protocol to which any compliant interactive client can connect and access any service.”The web comes close to solving this problem, but is interactively insufficient.
23XWeb Protocols Based upon the architecture of the web XTP Interaction ProtocolServer-side data has a tree structureStructured Data in XMLURLs for location of objectsxweb://automate.home/lights/livingroom/xweb://automate.home/lights/familyroom/-1
24XWeb & XTP CHANGE message (similar to GET in HTTP) Sequence of editing operations to apply to a sub-treeSet an attribute’s valueDelete an attributeChange some child object to a new valueInsert a new child objectMove a subtree to a new locationCopy a subtree to a new location
25Platform Independent Interfaces Two models are specifiedDataView – The attributes of the serviceXView – A mapping of the attributes into high-level “interactors”Interactors are somewhat like abstract interaction objectsAtomicNumericTimeDateEnumerationTextLinksAggregationGroupList
29Other XWeb DetailsHas simple approach for adjusting to different screen sizesShrink portions of the interfaceAdd additional columns of widgetsAlso capable of generating speech interfacesBased on a tree traversal approach like Universal Speech Interfaces
30ICrafterPart of the Interactive Workspaces research project at StanfordMain objective:“to allow users of interactive workspaces to flexibly interact with services”ContributionAn intelligent infrastructure to find services, aggregate them into a single interface, and generate an interface for the aggregate service.In practice, much of the interface generation is done by hand though automatic generation is supported.
32How is aggregation accomplished? High-level service interfaces (programmatic)Data ProducerData ConsumerThe Interface Manager has pattern generatorsRecognize patterns in the services usedGenerate interfaces for these patternsThis means that unique functionality will not be available in the aggregate interface
35Personal Universal Controller My work with BradProblem:Appliance interfaces are too complex and too idiosyncratic.Solution:Separate the interface from the appliance and use a device with a richer interface to control the appliance:PDA, mobile phone, etc.
38Language Design Approach Create reference interfaces AIWA Shelf StereoAT&T Telephone/Answering MachineTest interfaces with subjectsUsers twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfacesAnalyze interfaces for functional informationBe careful of saying “same manufacturer’s appliances”
39Language Elements State Variables and Commands Label Information Represent functions of applianceState variables have typesBoolean, Enumeration, Integer, String, etc.Variables sufficient for most functions but not all“seek” button on a RadioLabel InformationOne label not suitable everywhereThe optimal label length changes with screen sizeSpeech interfaces may benefit from pronunciation and text-to-speech information
40Language Elements, cont. Group TreeSpecify organization of functionsWe use n-ary tree with variables or commands at leavesNeed a place for things like “no specific layout information” related work or new high-level spec slide
41Language Elements, cont. Dependency InformationFormulas that specify when a variable or command is active in terms of other state variablesEquals, Greater Than, Less ThanLinked with logical operators (AND, OR)For example,<and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals> </and>
42Interface Generators Generators for Two Modalities Graphical Speech Implemented for PocketPC in Java 1.1Uses dependency information to generate panel structure of interfaceSpeechImplemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)
43Graphical Interface Generator Focuses on panel structure of user interfaceSmall groups of controls have basic layoutsComplexity comes from structure of groupsStructure can be inferred from dependency info!decomposed
44Inferring StructureFind sets of variables that are “mutually exclusive”Every variable in a set will never be active at the same time as a variable in another setCreate structure with sets, using overlapping panels
46Making the Interface Concrete Finish conceptual layoutChoose controls (decision tree)Choose row layouts(one column, two column, etc.)Allocate spaceExamine panel contents and choose sizesInstantiate and place controls
47Generating Speech Interfaces Automatically build USI tree from dependenciesAllows verbal navigation of functional groupsAutomatically generate grammar for parserPhrases for query and control“What is playmode?”“Set playmode to play”“play”Automatically generate language model and pronunciation for recognizer