HTML5 Haptics Standardization Mohamed Mansour hello@mohamedmansour.com HTML5 Haptics Standardization Haptics Chrome
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Vision Haptics Integration into HTML5 Rich JavaScript API to communicate to Haptics Cross platform Haptic device independent Standardized in W3C Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 1991 HTML 1994 HTML 2 1996 CSS 1 JavaScript 1997 HTML 4 1998 CSS 2 2000 XHTML 2002 Tableless 2005 AJAX 2009 HTML 5 INNOVATION Rough timeline of Web Technologies HTML – Tim Berners-Lee CSS – Separation of design from content by Hakon Wium Lie XHTML (HTML 4.01) – Has doctype many tags (ul, u, thead, table), proper nested, proper xml, closed, lowercase, one root element Tableless – With CSS (960 grid) AJAX – Jesse James Garret key player Mohamed Mansour
What is HTML5? HTML5 = HTML + CSS + JavaScript APIs Mohamed Mansour
HTML5 New features Storage Application Cache Web Workers Web Sockets Notifications Drag and Drop Geolocation Semantic Tags Link Relations, and Microdata ARIA New form fields Audio and Video Canvas and SVG Font Text Support, Opacity, Rounded Corners Shadows, Gradients Backgrounds, Transitions Transformations, Animations Device A lot more! Selectors - getElementsByClassName(‘foo’) - querySelectoryAll(‘ul li:nth-child(odd)’) WebStorage - localStorage Web SQL Database Var db = Window.openDatabase(‘database name’); db.transaction(function(tx) {tx.executeSql Application Cache - Looks in a manifest for <html manifest=“cache.manifest”> for cache status Web Workers Var workers = new Worker(‘some file.js’); worker.onmessage = function(event) {alert(event.data)); Self.onmessage = function(event) {do some work} Web Sockets var socket = new WebSocket(location); Socket.onopen Socket.onmessage Socket.onclose Drag and Drop Dragstart, dropend Semantic Tags (header, section, nav, article, aside, footer) Link Relationships pingback, prefetch, archives, external, license, nofollow, tag all under “rel” Microdata Rich for snippets and search engines, like itemscope, itemtype, itemprop ARIA Accessible Rich Internet Applications Role, aria-expanded, aria-labelledby New form fields - Placeholder for search, color, number, email, tell types, range, etc Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Haptics Technology that interfaces with the user through the sense of touch [definition] The process of recognizing objects through touch [another definition] Mohamed Mansour
Haptic Devices Omni Wrist band Novint Jacket Gloves Way more … Mohamed Mansour
How they work? Robotic Sensors Feedback Ex: You send forces to the device, and it magically moves. You request positional data from the device. Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Plugins Browser plugins is the only way you can allow the user to communicate to native code. Current state of HTML is still limited … look at Flash, why did it exist? Can do almost anything! But very very very Dangerous! Mohamed Mansour
Plugins Active X NPAPI (very low-level) Microsoft Internet Explorer Google Chrome Firefox Opera Safari Mohamed Mansour
NPAPI Very long name …. Netscape Plugin Application Programming Interface. Very powerful! It is just a single library (.dll, .so, .plugin) file that gets loaded. You just need at least 3 Library Exports: NP_GetEntryPoints NP_Initialize NP_Shutdown Mohamed Mansour
Scriptable to NPObject* NPAPI Architecture Plugin LOGIC <embed> 3 1 2 DLL EXPORTS Scriptable to NPObject* 5 4 Browser API NPN_* Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Google Chrome Multi Process Browser Very fast JavaScript and Rendering Tab, Plugins, Renders, Extensions, Browser Very fast JavaScript and Rendering GPU Accelerated Canvas and WebGL Secure Sandbox, for Plugins Very fast at implementing HTML5 features. Open Source browser Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Google Chrome Extensions Small software that can modify and enhance the functionality of Google Chrome. Using web technologies, HTML, JavaScript, and CSS. Easily expose NPAPI functionality. Separate Process Mohamed Mansour
Chrome Extension Basics Background page Lengthy process, long lived, singleton. Browser Actions / Page Actions Desktop Notifications Override Pages Browser Interaction (Bookmarks, Cookies, History, Events, Management, Windows, Tabs) Plugin Interaction. Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Prototype HTML5 Device Tag <device> Interface Type Mohamed Mansour
Device Tag Still in implementation (research phase) It can change! Add a Haptics “type” Dreamy Ex: <p> To start, Select a haptic device: <device type=haptic onchange="update(this.data)"> </p> <script> function update(stream) { console.log(stream.data); } </script> Mohamed Mansour
Prototype , Needed? Proof of concept needed to prove that Haptics needs love! And the future of computer human interactions! Rescue … Mohamed Mansour
Prototype, Chrome Extension How do we do it right now! HTML5 (Web Workers) CSS3, Canvas (WebGL Context Graphics) Chrome Extension Dedicated Thread (Background Page) Haptics Controller (Browser Action) Page Interactions (Page Action) Accessibility (ARIA) NPAPI Haptics Plugin communicates to Chrome Extension C++ extreme (scene graph, triangles) collision detection algorithms Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Architecture High Level C Plus Plus Novint Device HTML5 Mohamed Mansour
Architecture Plugin Plugin Module Plugin Gate Scripting Bridge Haptics Service NP_GetEntryPoints NP_Initialize NP_Shutdown Haptics Device Utils Haptics Signal Mohamed Mansour
Glimpse of Plugin Code Mohamed Mansour
Architecture, Extension Browser Action to act as the controller Background Page, to allow the plugin to live forever. JavaScript for Some Collision Detection, Haptic Loop, and Rendering Loop. Mohamed Mansour
Framework O3D JavaScript API for creating rich and interactive 3D application in the broswer using Canvas, WebGL context. Haptics API void startDevice(); void stopDevice(); void sendForce(double[3]); double[3] position; addEventListener(‘servo’, function(Haptic data)); Mohamed Mansour
Open Source’d Release it to the public! That is how I learned how to code, give back to the community! You can fork it from GitHub from here: http://github.com/mohamedmansour/haptics-chrome-extension Mohamed Mansour
Working Prototype Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
HTML5 Haptics Standardization 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour
Future Present this to the W3C Developer Group Help Google Chrome complete <device> tag implementation. Implement the Prototype again using the <device> tag. Implement webpage interactions, like annotated YouTube Videos to feel video touch. Mohamed Mansour
Done! Thank you, questions? 1 2 3 Vision HTML5 Haptics Plugins Google Chrome Extensions Prototype Architecture Future 4 5 6 7 Mohamed Mansour