Microsoft Research Faculty Summit 2003 Brad A. Myers & Jeffrey Nichols The Personal Universal Controller and.NET CF Pebbles Research Project Human Computer Interaction Institute Carnegie Mellon University

Pebbles Project Seen in the plenary demo yesterday! Seen in the plenary demo yesterday! Use of multiple devices at the same time Use of multiple devices at the same time Pocket PC and desktop PC Pocket PC and desktop PC Mobile Phone and Smart Home Mobile Phone and Smart Home Multiple handhelds in a meeting Multiple handhelds in a meeting Pocket PC and appliances Pocket PC and appliances Multiple users with their devices Multiple users with their devices Single user with multiple devices Single user with multiple devices

Early Pebbles Work For business meetings: For business meetings: For group work For group work How laser pointers can be integrated How laser pointers can be integrated To augment desktop applications To augment desktop applications For individuals For individuals For classrooms For classrooms To make lectures more interactive To make lectures more interactive For military command posts For military command posts To facilitate communication and collaboration To facilitate communication and collaboration

Recent Project: Personal Universal Controller Using handhelds with appliances to improve the user interface Using handhelds with appliances to improve the user interface Key Features Key Features User interface-independent appliance specification User interface-independent appliance specification Automatic generation of GUI and speech interfaces Automatic generation of GUI and speech interfaces Specifications Control Feedback

Automatic Generation of UIs Benefits All interfaces consistent for the user All interfaces consistent for the user With conventions of handheld With conventions of handheld Even from multiple manufacturers Even from multiple manufacturers Multiple modalities (GUI + Speech UI) Multiple modalities (GUI + Speech UI)

Development History Visual C Extensions for WinCE Visual C Extensions for WinCE eMbedded Visual C++ eMbedded Visual C++ Early Pebbles Applications Early Pebbles Applications eMbedded Visual Basic 3.0 eMbedded Visual Basic 3.0 Early tests of Personal Universal Controller (PUC) concept Early tests of Personal Universal Controller (PUC) concept PersonalJava 1.1 PersonalJava 1.1 Initial PUC implementation Initial PUC implementation.NET Compact Framework.NET Compact Framework Current PUC implementation Current PUC implementation

Overview Introduction Introduction PUC Implementation PUC Implementation Advantages of.NET &.NET CF Advantages of.NET &.NET CF Limitations of.NET CF Limitations of.NET CF Short Demo Short Demo Resources Resources

PUC Implementation Implemented three interface generators Implemented three interface generators PocketPC PocketPC Desktop Desktop Smartphone Smartphone Implementation uses a large part of the.NET Compact Framework Implementation uses a large part of the.NET Compact Framework UI Toolkit UI Toolkit Graphics Graphics Networking & I/O Networking & I/O XML Parsing XML Parsing

PUC System Architecture Communication (802.11, Bluetooth, RF-Lite, etc.) PUC Devices (automatic user interface generation) Adaptors (publishes description + appliance state + controls appliance) XML-Based Protocol (two-way communication of specification and state) Appliances (media players, cameras, etc.) Communication (802.11, Bluetooth, RF-Lite, etc.) XML-Based Protocol (two-way communication of specification and state) Third-party components Multi-platform components.NET CF components Legend:

PUC Device Architecture Protocol XML Parsers <message> PlayMode PlayMode Network TCP/IP Sockets IPEndPoint end = new IPEndPoint(_ipAddress,_port); _socket.Connect(end); int len = _socket.Receive(num,4,SocketFlags.None); Concrete Interface glue to.NET CF Controls Model Analysis & Rule-Based Design analysis of models (appliance & device), rule-based design of abstract interface, choose concrete controls & layout

Advantages of.NET Managed Environment Managed Environment Garbage collection Garbage collection Modern Programming Languages Modern Programming Languages Strong types Strong types Consistent, Well-Designed API Consistent, Well-Designed API Object-oriented Object-oriented Better documentation Better documentation

Advantages of.NET CF For mobile application development For mobile application development Compatible with normal.NET Framework Compatible with normal.NET Framework Can develop for handheld and desktop almost simultaneously Can develop for handheld and desktop almost simultaneously.NET CF being ported to multiple handheld platforms.NET CF being ported to multiple handheld platforms PocketPC, Smartphone 2003, WinCE 4.0? PocketPC, Smartphone 2003, WinCE 4.0? Good performance on handhelds Good performance on handhelds

Advantages of.NET CF Code Portability Portablility makes re-use very easy 90% from PocketPC to TabletPC* 90% from PocketPC to TabletPC* 60% from PocketPC to Smartphone* 60% from PocketPC to Smartphone* Different input model on Smartphone requires different generator rules and several new widgets Different input model on Smartphone requires different generator rules and several new widgets * These numbers are estimates from development experience

Advantages of.NET CF Versus Java Solid, robust implementation on handhelds Solid, robust implementation on handhelds Consistent programming interface across mobile platforms Consistent programming interface across mobile platforms Java Micro Edition uses different APIs for Graphics and the UI toolkit from standard Java Java Micro Edition uses different APIs for Graphics and the UI toolkit from standard Java More features in the UI toolkit More features in the UI toolkit More controls (vs. AWT) More controls (vs. AWT)

Limitations Disclaimer I am not a Microsoft developer! So, these limitations may have solutions I am not aware of: I did not find the solutions I did not find the solutions They are fixed in more recent versions of the Compact Framework They are fixed in more recent versions of the Compact Framework

Limitations of.NET CF Compared to normal.NET Framework, CF has: Fewer controls Fewer controls Fewer features in each control Fewer features in each control Can sub-class built-in controls, but… Can sub-class built-in controls, but… No ability to owner draw No ability to owner draw No ability to handle low-level events No ability to handle low-level events Must sub-class base Control class to make custom controls Must sub-class base Control class to make custom controls General Gripe: Controls do not know their own preferred or minimum size.

Limitations of.NET CF continued Available CF controls are mostly adequate Some custom controls needed Some custom controls needed ImageButton ImageButton Time Picker Time Picker Scrolling Panel (Panels dont have Auto Scroll) Scrolling Panel (Panels dont have Auto Scroll) Smartphone List Control Smartphone List Control Smartphone Scrolling Panel Smartphone Scrolling Panel

Limitations of.NET CF continued Smartphone Still early in the design process Navigation on screen based on Z-order and enabledness of controls. Navigation on screen based on Z-order and enabledness of controls. Panels dont automatically scroll, as required by interface. Panels dont automatically scroll, as required by interface. Back button can only be handled through key-press events (which cant always be caught, and cant be fully worked around) Back button can only be handled through key-press events (which cant always be caught, and cant be fully worked around) Scrollbars do not accept user input (we worked around this) Scrollbars do not accept user input (we worked around this) Not many people working with phone Few resources when you have problems.

Summary Advantages and Limitations Like any new set of APIs, there are bugs Overall, the good outweighs the bad Rapid prototyping like Visual Basic Rapid prototyping like Visual Basic Power of modern language in C# Power of modern language in C# Easily able to share code across platforms (PocketPC, Desktop & Smartphone) Easily able to share code across platforms (PocketPC, Desktop & Smartphone) More than adequate performance on handhelds More than adequate performance on handhelds Little need to worry about memory or performance constraints Little need to worry about memory or performance constraints

PUC Implementation Jeffrey Nichols Human Computer Interaction Institute Carnegie Mellon University

Resources Usenet Groups Usenet Groupsmicrosoft.public.dotnet.framework.compactframework Books from Microsoft Press Books from Microsoft Press.NET Compact Framework Core Reference* Visual C#.NET Step-By-Step Microsoft Visual C#.NET Language Reference Developer Labs (?) Developer Labs (?) Thanks Microsoft! * Have not looked at this book.

PUC and the.NET CF Brad A. Myers & Jeffrey Nichols Carnegie Mellon University thanks!

Limitations of.NET CF continued Modifying any property of the UI from outside the UI thread causes problems. Example: Network thread receiving messages wants to set the value of a scrollbar Example: Network thread receiving messages wants to set the value of a scrollbar Application crashes 99% of the time Application crashes 99% of the time Control.BeginInvoke method not implemented (yet?) Control.BeginInvoke method not implemented (yet?) Must work-around by using Timers and an event queue. Must work-around by using Timers and an event queue.