Faculté des sciences appliquées Département d’ingénierie informatique Presented by Transparent migration and adaptation in a Graphical User Interface toolkit Donatien Grolaux Thesis Public Defense September 4 h, 2007

Context Ubiquitous computing Ubiquitous computing UIs could take advantage of the ubiquity of devices UIs could take advantage of the ubiquity of devices –Migration –Adaptation User centric view where the application follows the user in its mobility User centric view where the application follows the user in its mobility

Demo

Purpose Main goal: find a way to provide migration and adaptation support to applications Main goal: find a way to provide migration and adaptation support to applications –Introduce as little extra complexity as possible How could that be achieved ? How could that be achieved ? –Ad hoc solution for each application –Do not touch the application, the migration/adaptation is provided externally –Introduce extra models for the migration/adaptation and have them executed at runtime –Provide the support for migration/adaptation in the toolbox that builds the UI to provide migration/adaptation Create a new toolbox from scratch ?Create a new toolbox from scratch ? Extend an existing toolboxExtend an existing toolbox No Yes No Yes

Approach of this thesis Give a tool to the developers that enables them to create multiplatform migratable & adaptable UIs. Give a tool to the developers that enables them to create multiplatform migratable & adaptable UIs. –While minimizing the impact on the interactive applications implementation This tool is a graphical toolkit This tool is a graphical toolkit –That presents itself as a usual graphical toolkit (AWT, GTk, Tcl/Tk…) Concepts of widgets, user events…Concepts of widgets, user events… –Multiplatform support –Migration support –Adaptation support –Some more features

Approach of this thesis Do not try to solve discovery problems, context awareness, meta-UI, tasks specification… Do not try to solve discovery problems, context awareness, meta-UI, tasks specification… –It is up to the application to solve these issues –However the tool allows for simple solutions

Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Overview Goal: create applications supporting migratable and adaptable user interfaces. Goal: create applications supporting migratable and adaptable user interfaces. Means: a graphical toolkit supporting these features, in an easy to use way. Means: a graphical toolkit supporting these features, in an easy to use way. –Using a toolkit-agnostic middleware designed to provide these extensions –By extending an existing toolkit Tools: the Oz programming language. Tools: the Oz programming language. –Support for distributed applications –Multi-paradigm programming language (O-O, symbolic programming, functional programming…)

Application This thesis allows writing applications whose UI: This thesis allows writing applications whose UI: –Is dynamically migratable –Is dynamically adaptable Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Application Low development cost Low development cost –Extend classical graphical toolkit approach Give control to the application itself Give control to the application itself Dynamicity Dynamicity Fault tolerance Fault tolerance Extensible Extensible Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Mozart Multi-paradigm programming language Multi-paradigm programming language –Strongly dynamically typed –Procedural, Object-Orientation –Functional, Logic and Symbolic programming –Transparent distribution support Distributed network protocols are attached automatically to some of the native data types of OzDistributed network protocols are attached automatically to some of the native data types of Oz Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

EBL/Tk Toolkit developped for this thesis Toolkit developped for this thesis –Complete graphical UI toolkit with widgets, fonts, windows, dialog boxes… –Use a mixed declarative/imperative approach Declarative approach for creating UIsDeclarative approach for creating UIs Imperative approach for running UIsImperative approach for running UIsUI={Build window( window( name:window name:window selector( selector( name:selector name:selector items:["Radiobuttons" "Listbox" "Menu"] items:["Radiobuttons" "Listbox" "Menu"] curselection:1 curselection:1 text:"Selector" text:"Selector" action:proc{$} action:proc{$} R={UI.selector get(curselection:$)} R={UI.selector get(curselection:$)} in in {UI.selector setContext((default#listbox#menu).R)} {UI.selector setContext((default#listbox#menu).R)} end))} end))} {UI.window show} Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

EBL/Tk Adaptation Adaptation is a special configuration parameter Adaptation is a special configuration parameterUI={Build window( window( name:window name:window selector( selector( name:selector name:selector items:["Radiobuttons" "Listbox" "Menu"] items:["Radiobuttons" "Listbox" "Menu"] curselection:1 curselection:1 text:"Selector" text:"Selector" action:proc{$} action:proc{$} R={UI.selector get(curselection:$)} R={UI.selector get(curselection:$)} in in {UI.selector setContext((default#listbox#menu).R)} {UI.selector setContext((default#listbox#menu).R)} end))} end))} {UI.window show} Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

EBL/Tk Migration Capability based migration Capability based migration –Capability represents the authority to migrate a widget –Capability contains the Internet address of the widget –All widgets (except toplevel widgets) have the migration capability {Offer {UI.selector getRef($)}} UI2={Build window(name:window)} UI2={Build window(name:window)} {UI2.window show} {UI2.window show} {UI2.window display({Obtain})} {UI2.window display({Obtain})} Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Transparent Migration and Adaptation Transparent migration Transparent migration –Widgets are used by the application indifferently of their actual situation (local/remote) as if they were always local. –The migration can happen at any time, and the protocol is executed independently of the concurrent running application. Transparent adaptation Transparent adaptation –Widgets are used by the application indifferently of their actual visual representation as if there was only one representation. –The adaptation can happen at any time, and the protocol is executed independently of the concurrent running application. Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Thesis main contribution: EBL Middleware that has to be interfaced to an existing toolkit Middleware that has to be interfaced to an existing toolkit –Toolkit agnostic, could be interfaced to any Infrastructure for migration  Distributed environment Infrastructure for migration  Distributed environment Device 1 Device 2 Device 3 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Distributed Toolkit Application is stationary, widget is migratory Application is stationary, widget is migratory –Two parts: proxy & renderer Site A Site B Site C Proxy A1 Proxy A2 Proxy C1 Renderer A1 Renderer A2 Renderer C1 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Fault Tolerance Network fault tolerance Network fault tolerance –Proxy has the complete state –Renderer eventually notified of state update, reflected to the widget Site A Site B Site C Proxy A1 Proxy A2 Proxy C1 Renderer A1 Renderer A2 Renderer C1 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Site D Renderer C2

Migration Dynamic creation of a renderer Dynamic creation of a renderer –Linked to a proxy –Using local resources Site A Proxy A1 Site B Renderer A1 Local Resources Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Geometry Management Geometry is managed by hierarchy of container widgets Geometry is managed by hierarchy of container widgets –Root widget: toplevel Toplevel is rooted at the application that creates itToplevel is rooted at the application that creates it –td and lr widgets provide compact syntax for tables Site A Display A Application A Toplevel Proxy Toplevel Renderer Site B Application B Widget Proxy Widget Renderer Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Migration Mechanism Migration based on capability Migration based on capability –Embeds the authority to migrate the widget (PULL) –Embeds the Internet address of the widget Migration triggered by giving the migration capability to a container widget Migration triggered by giving the migration capability to a container widget Application A Widget Proxy Ref Discovery Service Ref Application B Window Proxy get Ref Ref Window Renderer        Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Widget Renderer 

Adaptation Alternate compatible renderers Alternate compatible renderers Widget X definition Widget Y definition Proxy Y Renderer Y1 Renderer Y2 Renderer Y3 Proxy XRenderer X1 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Adaptation Adaptation is supported by dynamically switching the renderer Adaptation is supported by dynamically switching the renderer Switching the renderer is implemented by migrating the widget into the place it already occupies, but using a different renderer definition Switching the renderer is implemented by migrating the widget into the place it already occupies, but using a different renderer definition Application A Proxy Y Application B Renderer Y1 Application A Proxy Y Application B Renderer Y2 Adaptation of Y, from renderer Y1 to Y2 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Proxy-Renderer Collaboration implemented by shared memory abstraction: store Collaboration implemented by shared memory abstraction: store –StoreName X KeyName -> KeyValue –Type checking, transparent marshalling Application AApplication B Widget Proxy - public API EBL Manager EBL Store x EBL Store y EBL Store z Widget Renderer - EBL compatible API EBL Manager EBL Store x EBL Store y EBL Store z Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Delegation In an ideal situation, the functional core of the widget is reproduced at its proxy In an ideal situation, the functional core of the widget is reproduced at its proxy –Require development cost –Functional core typically already present at the renderer A delegation mechanism allows the proxy to rely on a renderer for a state update A delegation mechanism allows the proxy to rely on a renderer for a state update –The proxy sends the request to the renderer and waits for its response –The renderer uses the toolkit to apply the request –The renderer gets back the resulting state, and responds to the proxy –The proxy updates its state accordingly Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Delegation Example Proxy meth change(Something) OldState={Store get(main state $)} OldState={Store get(main state $)} NewState={Diff OldState Something} NewState={Diff OldState Something}in {Store set(main state NewState)} {Store set(main state NewState)}end meth change(Something) NewState={Manager ask(Something $)} NewState={Manager ask(Something $)}In {Store set(main state NewState)} {Store set(main state NewState)}end Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Renderer meth set(I K V) case I#K of main#state then case I#K of main#state then {Toolkit apply(state V)} {Toolkit apply(state V)} end endend meth ask(Q R) {Toolkit apply(state Q)} {Toolkit apply(state Q)} R={Toolkit getState($)} R={Toolkit getState($)}end No Delegation Delegation

StateT1T2 Proxy State | T1, T2, … Proxy State’ | T2, … Transaction protocol for Delegation Delegation is implemented by transactions Delegation is implemented by transactions –Transactions are resolved in the order they are submitted. –At the proxy, transaction requests blocks until they are committed. –For a transaction to commit, the renderer must survive long enough to give its answer. –When that is not the case, the transactions keeps on blocking until a new renderer comes in. The state of the renderer is first synchronized to the current state of the proxy.The state of the renderer is first synchronized to the current state of the proxy. Transactions are then retried, still respecting the order in which they are submittedTransactions are then retried, still respecting the order in which they are submitted Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Renderer State Resp T1Resp T2

Reflexivity Build function is also available at the renderer Build function is also available at the renderer –Allows the creation of compound widgets using the high expressivity level of EBL even when implementing the low level renderer. Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Frame={New Toolkit.frame tkInit(parent:Parent)} YesButton={New Toolkit.button tkInit(parent:Frame)} {YesButton tk(configure(text:“Yes”))} NoButton={New Toolkit.button tkInit(parent:Frame)} {NoButton tk(configure(text:”No”))} {Toolkit.pack YesButton} {Toolkit.pack NoButton} Widgets={Manager build(lr(name:frame button(text:”Yes”) button(text:”No”)) $)} button(text:”Yes”) button(text:”No”)) $)}Frame=Widgets.frame

Model based approach using EBL Let’s consider an application specific model Let’s consider an application specific model –Implemented by Oz data structures Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk data(name:"Roger" surname:"Rabbit" surname:"Rabbit" address1:"Rue des toons" address1:"Rue des toons" address2:"WB") address2:"WB")

fun{Transform2 D} fun{Loop P} fun{Loop P} case P of I#E|Xs then case P of I#E|Xs then label(text:I)| label(text:I)| label(text:E)| label(text:E)| newline| newline| {Loop Xs} {Loop Xs} else nil end else nil end end endin {List.toTuple lr {List.toTuple lr {Loop {Record.toListInd D}}} {Loop {Record.toListInd D}}}end Model based approach using EBL Let’s consider an application specific model Let’s consider an application specific model –Implemented by Oz data structures –Functional paradigm of Oz manipulates these data structures Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk data(name:"Roger" surname:"Rabbit" surname:"Rabbit" address1:"Rue des toons" address1:"Rue des toons" address2:"WB") address2:"WB") fun{Transform1 D} {List.toTuple td {List.toTuple td {List.map {List.map {Record.toListInd D} {Record.toListInd D} fun{$ I#E} fun{$ I#E} lr(label(text:I) lr(label(text:I) label(text:E)) label(text:E)) end}} end}}end td(lr(label(text:address1) label(text:"Rue des toons")) label(text:"Rue des toons")) lr(label(text:address2) lr(label(text:address2) label(text:"WB")) label(text:"WB")) lr(label(text:name) lr(label(text:name) label(text:"Roger")) label(text:"Roger")) lr(label(text:surname) lr(label(text:surname) label(text:"Rabbit"))) label(text:"Rabbit"))) + lr(label(text:address1) label(text:"Rue des toons") newline label(text:"Rue des toons") newline label(text:address2) label(text:address2) label(text:"WB") newline label(text:"WB") newline label(text:name) label(text:name) label(text:"Roger") newline label(text:"Roger") newline label(text:surname) label(text:surname) label(text:"Rabbit") newline) label(text:"Rabbit") newline)

Model based approach using EBL Let’s consider an application specific model Let’s consider an application specific model –Implemented by Oz data structures –Functional paradigm of Oz manipulates these data structures –End result is a valid EBL/Tk description record Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk lr(label(text:address1) label(text:"Rue des toons") newline label(text:"Rue des toons") newline label(text:address2) label(text:address2) label(text:"WB") newline label(text:"WB") newline label(text:name) label(text:name) label(text:"Roger") newline label(text:"Roger") newline label(text:surname) label(text:surname) label(text:"Rabbit") newline) label(text:"Rabbit") newline) td(lr(label(text:address1) label(text:"Rue des toons")) label(text:"Rue des toons")) lr(label(text:address2) lr(label(text:address2) label(text:"WB")) label(text:"WB")) lr(label(text:name) lr(label(text:name) label(text:"Roger")) label(text:"Roger")) lr(label(text:surname) lr(label(text:surname) label(text:"Rabbit"))) label(text:"Rabbit"))) lr(label(text:address1) entry(text:"Rue des toons") newline entry(text:"Rue des toons") newline label(text:address2) label(text:address2) entry(text:"WB") newline entry(text:"WB") newline label(text:name) label(text:name) entry(text:"Roger") newline entry(text:"Roger") newline label(text:surname) label(text:surname) entry(text:"Rabbit") newline) entry(text:"Rabbit") newline)

Model based approach using EBL Let’s consider an application specific model Let’s consider an application specific model –Implemented by Oz data structures –Functional paradigm of Oz manipulates these data structures –End result is a valid EBL/Tk description record –Alternate renderings are placed into alternate renderers Because of the reflexivity, this is a straightforward processBecause of the reflexivity, this is a straightforward process Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk Data=data(name:"Roger" surname:"Rabbit" surname:"Rabbit" address1:"Rue des toons" address1:"Rue des toons" address2:"WB") address2:"WB") All={Build window(form(data:Data name:form) name:top)} {All.top show} {All.form setContext(unalignedview)} {All.form setContext(alignedview)} {All.form setContext(alignededit)}

Evaluation Performance overhead Performance overhead –When no delegation is used, EBL runs in an asynchronous way 328 ms for operations328 ms for operations –When delegation is used, EBL runs in a synchronous way 1780 ms for operations1780 ms for operations –Network overhead Typically way greater than EBL overheadTypically way greater than EBL overhead Typically the throughput limitation can be felt only when sending huge amount of data in a small window of time, which does not happen often with EBLTypically the throughput limitation can be felt only when sending huge amount of data in a small window of time, which does not happen often with EBL For asynchronous operations: the application is not slowed down, the UI lags behind by the network delay timeFor asynchronous operations: the application is not slowed down, the UI lags behind by the network delay time For synchronous operations: the application is slowed down 2Xnetwork delayFor synchronous operations: the application is slowed down 2Xnetwork delay Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Evaluation Expressiveness Expressiveness –Full fledged graphical toolkit. –The declarative approach greatly reduces the size of the code required for building a UI (QTk experiment: down to 1/3 the size). –The mixing of declarative and imperative approaches allows the creation of adaptative UIs at reduced cost (FlexClock experiment: 300 lines of code for 17 different views). –The transparent migration approach allows migrating whole UIs with few lines of code. –The transparent adaptation approach allows adaptating whole UIs with few lines of code. –The delegation greatly lowers the actual toolkit binding development cost. –The extensibility of EBL allows reusability. Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Conclusion We have a GUI toolkit We have a GUI toolkit –That runs, is available and is documented –Which is expressively equivalent to usual GUI toolkits –Which is often more expressive and simpler than usual GUI toolkits (imperative/declarative approach) –Runs on all platforms supported by Mozart (Mac OS X, Windows, Linux, Unix…) –Supports dynamic migration –Supports dynamic adaptation –Is easily extensible

Shortcomings and Future work Tcl/Tk is aging Tcl/Tk is aging –Replace it by a more up to date toolkit (GTk) Limitations of the current implementation Limitations of the current implementation –Concurrent multi-toolkit support –Limited multi-user functionality Communications between proxy and renderer is restricted (store, delegation) Communications between proxy and renderer is restricted (store, delegation) –Expressivity is enough for EBL/Tk, but is it general enough? Address security issues Address security issues Explore multi-user aspects Explore multi-user aspects Explore usability aspects Explore usability aspects

The End

More slides just in case

Multi-paradigm & adaptation Functional, Logic and Symbolic programming Functional, Logic and Symbolic programmingdata(name:"Roger" surname:"Rabbit" surname:"Rabbit" address1:"Rue des toons" address1:"Rue des toons" address2:"WB") address2:"WB") td(lr(label(text:address1) label(text:"Rue des toons")) label(text:"Rue des toons")) lr(label(text:address2) lr(label(text:address2) label(text:"WB")) label(text:"WB")) lr(label(text:name) lr(label(text:name) label(text:"Roger")) label(text:"Roger")) lr(label(text:surname) lr(label(text:surname) label(text:"Rabbit"))) label(text:"Rabbit")))lr(label(text:address1) label(text:"Rue des toons") newline label(text:"Rue des toons") newline label(text:address2) label(text:address2) label(text:"WB") newline label(text:"WB") newline label(text:name) label(text:name) label(text:"Roger") newline label(text:"Roger") newline label(text:surname) label(text:surname) label(text:"Rabbit") newline) label(text:"Rabbit") newline) F1 F2 Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk

Adaptation Protocol RMPMRC setContext(N) replace destroy getClassDe f connectManager current state RC 2 apply current state classDef    Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk state update

Migration Protocol RCPCPM getRef PM ref importHere(Ref PI) getClassDef classDef create env create manager connectManager current state create RM RM state update apply current state          Mozart TCP/IP DistributionLayer Application EBL/Tk EBL Tcl/Tk