T4G The Intelligent Application of Technology TM Natural User Interfaces & Microsoft Surface Development

T4G The Intelligent Application of Technology TM Why are we here? Obligatory corporate plug Obligatory personal plug The real content: – What is a Natural User Interface (NUI)? – What is the Microsoft Surface? – MS Surface development environment – Demos and code: Hello, Surface! Raw Images in WPF Contact Data Visualization Surface Controls Tag Visualizations – The Future: Windows Touch and Surface 2.0

T4G The Intelligent Application of Technology TM Our Company, Culture and Success Based on our philosophy of The Intelligent Application of Technology TM, we design, develop, and implement technology solutions that help our customers run their businesses better.

T4G The Intelligent Application of Technology TM T4G at a glance 1996 Year T4G was established 260+ # of employees in 7 locations across North America 80 % of customers in 2010 who invited us back for multiple projects 3 # of consecutive years on the Great Place to Work® list

T4G The Intelligent Application of Technology TM OUR VALUES Integrity Passion Communication Teamwork Adaptive Innovation Entrepreneurialism Value Conscious Social Responsibility

T4G The Intelligent Application of Technology TM Delivering leadership and expertise Travel and Hospitality Retail Energy Financial Services Professional Services Healthcare Government Communications Our Vertical Expertise Our Supporting Services Business Intelligence Infrastructure & Security Software Development & Integration Content Delivery Mobile Solutions Social Media Strategy & Measurement Strategic Planning Hosting & Application Management Portals & Collaboration User Experience & Design

T4G The Intelligent Application of Technology TM Who am I? Principal Consultant at T4G Focused on Portal Technologies (SharePoint) Also work on NUIs and Multi-touch technologies, among other things 25+ years on software and various technology things

T4G The Intelligent Application of Technology TM What is a Natural User Interface? A natural user interface is a user interface designed to use natural human behaviours for interacting directly with content.

T4G The Intelligent Application of Technology TM NUI is much more than Touch A New Way of Interacting Augmented Reality Speech Object Recognition Gestures Eye Tracking User eXperience Natural Learning Social Interaction

T4G The Intelligent Application of Technology TM A New Way of Interacting Social Immersive “The content is the application”

T4G The Intelligent Application of Technology TM Implications Pervasive UIs Don’t replace human interaction, enhance it Extend the real world, blur the lines between UI and real life

T4G The Intelligent Application of Technology TM What is the Surface? “Microsoft Surface is a revolutionary multi- touch computer that responds to natural hand gestures and real-world objects, helping people interact with digital content in a simple and intuitive way.”

T4G The Intelligent Application of Technology TM Surface is not just multi-touch Windows 7 & WPF 4 are a great baseline for touch Surface creates unique opportunities with specialized HW – Robust top – Horizontal form factor – Infrared camera array

T4G The Intelligent Application of Technology TM Surface: Beyond Multi-Touch Massive multi-touch Multi-user experiences Object recognition Optical engineering

T4G The Intelligent Application of Technology TM Surface: Multi-User Trend: Software continues to reduce the need for face-to-face human interactions Problem: Happiness and creativity depend on social interactions Surface brings people together – Industrial design encourages users to gather – Input capabilities allow simultaneous use – Apps facilitate collaboration / competition

T4G The Intelligent Application of Technology TM Surface: Massive Multi-Touch Surface can efficiently track 50+ simultaneous fingers and objects

T4G The Intelligent Application of Technology TM Surface: Objects Uniquely identify & react to billions of tagged objects Byte Tags 256 unique values Identity Tags >340,282,366,920,938,000,000,000,000,000,000,000,000 unique values Tags include orientation indictors dots

T4G The Intelligent Application of Technology TM Surface: Optics Determine the size and shape of every input See into and project onto physical objects

T4G The Intelligent Application of Technology TM What makes a GOOD Surface Application? The app must be right for the Surface paradigm It should be social Should not involve large amounts of “traditional” content Needs to be implemented thinking about touch

T4G The Intelligent Application of Technology TM Who is using it? Retailers (Microsoft, Wind Mobile, etc.) Hospitality (Hard Rock Café, Rio iBar, Sheraton, etc.) Museums (Smithsonian) Media (MSNBC) Healthcare, Education, Others

T4G The Intelligent Application of Technology TM Development Environment Architecture of the Surface Development Platform

T4G The Intelligent Application of Technology TM Development Environment Windows Vista 32-bit.NET 3.5 WPF or XNA All stuff you know…

T4G The Intelligent Application of Technology TM Development Environment On a unit, you can use the Microsoft Surface SDK to develop and test Microsoft Surface applications directly. You can run the unit in two modes. In administrator mode, you can access all of the Windows Vista functionality. User mode demonstrates how the unit appears in a commercial venue. That is, the unit suppresses the Windows Vista user interface, automatically starts required applications, and makes sure that those applications are restarted if they need to. Typically, you develop applications in administrator mode. Get your own Surface…

T4G The Intelligent Application of Technology TM Development Environment On a separate workstation, you can install the Microsoft Surface SDK and use the Surface Simulator tool to simulate a Microsoft Surface test environment. – Runs the same Surface Shell as the Surface does – Use 1 or more mice to simulate multi-touch – Tools for simulating Tags or Blobs – Record and Replay interactions to support testing and debugging Limitations: – No cameras (obviously) so if you use raw image data, cannot test it on simulator – Should have a pretty good sized screen – The SDK is only supported on Windows Vista Business, 32-bit! But, there are ways around that! ( surface-sdk-on-windows-7-and-or-x64.aspx) surface-sdk-on-windows-7-and-or-x64.aspx If you do not have a Surface or you share one)…

T4G The Intelligent Application of Technology TM Surface SDK Features for Multi-Touch Common controls optimized for Multi-Touch Controls design primarily for Multi-Touch Essential Multi-Touch UX Functionality SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton ScatterView ScatterViewItem LibraryContainer LibraryBar LibraryStack LibraryBarItem LibraryStackItem Rich async drag & dropInput visualizations

T4G The Intelligent Application of Technology TM Lessons Learned There is a lot to learn about NUIs There is just as much to unlearn Forget most of what you know about what makes a “good” UI Visuals/graphics/interaction design more important than code Less is more Development is easier on a workstation

T4G The Intelligent Application of Technology TM Hello, World! Plain Old WPF XAML <Window x:Class="WpfApplication1.MainWindow“ xmlns= xmlns:x= Title="MainWindow" Height="350" Width="525">

T4G The Intelligent Application of Technology TM Hello, World! Surface XAML <s:SurfaceWindow x:Class="HelloSurface.SurfaceWindow1“ xmlns= xmlns:x= xmlns:s= Title="Hello Surface">

T4G The Intelligent Application of Technology TM Code Sample 1 Basic Surface Application Responding to Touch Events Responding to multiple Touch Events Handling orientation information Hello, Surface!

T4G The Intelligent Application of Technology TM Code Sample 2 Types of Contact events Surface sees (fingers, tags, and blobs) Information about contacts Contact Data Visualization

T4G The Intelligent Application of Technology TM Code Sample 3 Surface Controls Common controls optimized for Multi-Touch Controls design primarily for Multi-Touch SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton ScatterView ScatterViewItem LibraryContainer LibraryBar LibraryStack LibraryBarItem LibraryStackItem

T4G The Intelligent Application of Technology TM Code Sample 3 Surface Controls Common controls optimized for Multi-Touch Controls design primarily for Multi-Touch SurfaceWindow SurfaceButton SurfaceSlider SurfaceScrollViewer SurfaceInkCanvas SurfaceCheckBox SurfaceRadioButton SurfaceListBox SurfaceListBoxItem SurfaceThumb SurfaceScrollBar SurfaceRepeatButton SurfaceToggleButton ScatterView ScatterViewItem LibraryContainer LibraryBar LibraryStack LibraryBarItem LibraryStackItem

T4G The Intelligent Application of Technology TM Code Sample 3 ScatterView Control Adding ScatterViewItems in XAML Adding ScatterViewItems in code (in response to Contact Events) ScatterViewItems through Binding (with simple item template) Handling contact events when ScatterViewItem content is “active” Surface Controls

T4G The Intelligent Application of Technology TM Surface: Objects Uniquely identify & react to billions of tagged objects Byte Tags 256 unique values Identity Tags >340,282,366,920,938,000,000,000,000,000,000,000,000 unique values Tags include orientation indictors dots

T4G The Intelligent Application of Technology TM Code Sample 4 Adding a Tag Visualizer to recognize a tag Extracting the information in the tag Displaying a Tag Visualization Tag Visualizations

T4G The Intelligent Application of Technology TM Touch Development Roadmap Windows 7 (2009) Multi-Touch Controls Multi-Touch API Multi-Touch Controls & API

T4G The Intelligent Application of Technology TM Touch Development Roadmap.NET 4.0 & Surface Toolkit (Q1 2010) Multi-Touch Controls Surface Multi-Touch Controls &API Multi-Touch API Multi-Touch Controls & API

T4G The Intelligent Application of Technology TM Touch Development Roadmap Surface 2.0 Multi-Touch Controls Surface Multi-Touch Controls &API Multi-Touch API Surface-specific Controls & API Surface-specific Controls & API

T4G The Intelligent Application of Technology TM Surface 2.0 Integration & Extension of WPF 4 All input is routed through the WPF InputManager Surface input provider can send to multiple windows Surface extension methods provide access to additional input data (tag values, orientation, shape, etc)

T4G The Intelligent Application of Technology TM Surface 2.0 Announced at CES 2011 Available later this year SDK Available April 2011 (?) Embedded Windows 7 Professional 64-bit.NET Framework 4.0 Windows Presentation Foundation (WPF) 4.0 Microsoft XNA® Framework 4.0 Windows PowerShell and DMTF DASH support, and enhanced administrator tools

T4G The Intelligent Application of Technology TM Resources On the web: Surface Web Site ( Deconstructing the NUI (Joshua Blake, Surface MVP) My Blog ( On – the official MS Surface twitter identity

T4G The Intelligent Application of Technology TM Questions?