1. 1 Clouds and Web2.0 II: Case Study and Tutorial CTS08 Tutorial Hyatt Regency Irvine California May 19 2008 Geoffrey Fox, Marlon Pierce Community Grids Laboratory, School of informatics Indiana University http://www.infomall.org/multicore gcf@indiana.edugcf@indiana.edu, http://www.infomall.orghttp://www.infomall.org

2. Polar Grid: A Web 2.0 Case Study Using Microformats, GeoRSS, REST, and KML Slides from Yu Ma

3. More Detailed Information See Marie’s blog: http://tethealla.blogspot.com/ http://tethealla.blogspot.com/ We use blogs in the lab extensively as lab books.

5. PolarGrid Datasets A typical processed dataset consists of: – One plain text header file describing the overall system and metadata of the particular chunk of measurements. – Multiple JPG images processed from the corresponding raw data. – Collection timestamp and fundamental properties such as waveform and transmitting (TX) /receiving (RX) antenna IDs are embedded in file names. Images are processed by Kansas University’s Matlab-based code, GOAP

6. Approaches Convert unstructured metadata description into microformats. Construct a relational database to enable fast and dynamic queries. Provide geographic annotation embedded into web feeds that can be plotted by Google Maps and Google Earth Technologies: – GeoRSS: RSS syndication with geo-location – Microformats: extensions to XHTML

7. Microformat Conversion Microformats capture “semantic” metadata information in XHTML format. Principals for converted microformats: – Simple: capture the most basic information unit. – Reusable: capture shared features among data. – Composable: when combined, can reflect more complex relationships among data. Most of our microformat classes are prefixed with “pg:” denoting the PolarGrid project namespace, except when existing generic ones such as the geo microformat are used.

8. How To Extend XHTML Use either or tags. Display of these tags can be controlled with style sheets You can also dynamically control these with JavaScript. Next generation of browsers will have built-in support for well-known microformats such as hCard.

9. Simple Example: Geo Format -2.193 52.686

10. Polar Grid Microformat Examples Simple descriptions: MCRDS Measurements with unit: 2.5 us

11. Microformat Examples Identification for cross-references: 1 Commonly shared features: (e.g. both TX and RX antennas record same kind of information) 0 TX/RX 0

12. Microformat Examples Structured collections/Arrays : 2 0 TX 0 1 TX 0

13. Microformat Examples Complex objects: (e.g. spatial and temporal information that uniquely defines a processed data chunk) Start/Stop 1202755351.892651 -2.193 52.686

14. GeoRSS-Simple Embedded Atom Feed <feed xmlns="http://www.w3.org/2005/Atom" xmlns:georss="http://www.georss.org/georss"> PolarGrid Expedition tag:pg3.ucs.indiana.edu,2008:polargrid-expedition-2008-05 2008-05-11T02:02:54Z Polar Grid

15. GeoRSS-Simple Embedded Atom Feed Data Chunk: 2008-03-11 11:55:41 tag:pg3.ucs.indiana.edu,2008:polargrid expedition-2008-05- 262 2008-03-11T11:55:41Z … (data chunk information described in microformats) … 77.585957 -52.448461 77.580976 -52.435148

16. Google RSS Feed Reader

17. Google Maps

18. Corresponding KML PolarGrid Expedition Data Chunk: 2008-03-11 11:55:41 <![CDATA[ … (data chunk information described in microformats) … ]]> -52.448461,77.585957 -52.435148,77.580976

19. Google Earth and KML

20. Web 2.0 Lessons Web 2.0 approaches can be applied to management of scientific information. With the right choices, you get a lot for free. Microformats: simple ways to encode name- value pairs, build up semantic descriptions. GeoRSS: orders data by both time and space – Works in all standard RSS/Atom readers. – Google Maps supports natively

21. Web 2.0 Tutorial Map

22. Browser + JavaScript Libraries Browser + JavaScript Libraries Browser + JavaScript Libraries Blogs, Calendars, Docs, etc Social Gadget Containers Gadgets, Gadget Aggregators Facebook Facebook Apps Server-Side Gdata Apps User Layer System Cloud Layer AJAX, JSON, REST, RSS SOAP, REST, RSS

23. Map Key Red blocks represent browsers and things that run in them (JavaScript). – This is the “user” level. – Client side mashups Green blocks represent Web servers and their applications. – This is the “developer” level. – Server-side mashups. – These can run on any hosting environment: your web server, Amazon EC2, Google GAE, etc. Blue blocks represent third party services. – This is the “system cloud” layer. Arrows represent network communications. – Everything goes over HTTP – REST, AJAX: communication patterns. – RSS, ATOM, JSON, SOAP: message format.

24. Using Google’s GData API How to Write Server-Side Mash-Ups

25. GData: the API to Google Services Google provides an extensive set of online services with well-known user interfaces. – Docs – Spreadsheet – YouTube – Blogger/Blogspot – Calendar – Picasso Photo Sharing – GMail These have programming interfaces as well as user interfaces. – Server side: Java,.NET, PHP, Python – Client side: JavaScript Client interfaces include visual components suitable for mash-ups Server-side APIs don’t include visual components but do allow you to make server-side mash-ups.

26. Server-Side Mash-Up Use Case Imagine you have a Web service that runs a large parallel finite element code. – Output can include images, movies, output files, metadata, etc. – Your web service manages the interaction with the queuing system (another topic…) With a server-side mash-up, your Web Service can – Post status information about your jobs (“Step 521 Completed!”) on your blog. – Post URLs pointing to your output files on your blog. – Upload your visualizations to Picasso and YouTube (and post links). – Generate RSS/Atom feeds of the above. – Post results to Google Calendar We’ll look at some Java examples.

27. Getting Started with GData Create a Blog, make sure you can login to YouTube, etc. You will need a clientID and associated developer key. – Get these from http://code.google.com/apis/base/signup.html.http://code.google.com/apis/base/signup.html Get the Java code – http://code.google.com/p/gdata-java-client/downloads/list http://code.google.com/p/gdata-java-client/downloads/list See any number of Google examples – http://code.google.com/apis/base/javadevguide.html http://code.google.com/apis/base/javadevguide.html One trick: You can get your blogger ID by examining the your Blog’s Atom feed. – It will be in the header and look something like this: tag:blogger.com,1999:blog-19457310 – The blog ID to use in the code below would be 19457310 in this example. The full working example is here – http://communitygrids.blogspot.com/2008/03/googles-gdata-java-api.html http://communitygrids.blogspot.com/2008/03/googles-gdata-java-api.html – We will simplify for pedagogical reasons.

28. Posting Results to Your Blogspot Blog String userName="me@gmail.com"; String password= "qwerty"; String content=“…”; GoogleService myService= new GoogleService("blogger",""); myService.setUserCredentials(userName,password); Entry myEntry = new Entry(); myEntry.setTitle(new PlainTextConstruct(title)); myEntry.setContent(new PlainTextConstruct(content)); URL postUrl = new URL("http://www.blogger.com/feeds/" + blogID + "/posts/default"); myService.insert(postUrl, myEntry); Create the entry to insert Login in Replace these with your values Insert the new entry using REST’s POST

29. Getting a YouTube Feed String feedUrl="http://gdata.youtube.com/feeds/api/videos/"; String scarFace5sec="http://gdata.youtube.com/feeds/api/videos/egwB7hVIIEc"; YouTubeService service = new YouTubeService(clientID, developer_key); service.setUserCredentials(userName,password); VideoFeed videoFeed = youTubeService.getFeed(new URL(feedUrl),VideoFeed.class); VideoEntry videoEntry = youTubeService.getEntry(new URL(scarFace5sec), VideoEntry.class); //Loop over feed entries and display. String feedContent= "Title: " + videoEntry.getTitle().getPlainText()+”"; feedContent+=videoEntry.getMediaGroup(). getDescription().getPlainTextContent()+”";

30. Posting To Google Calendar CalendarService calService=new CalendarService(clientID); calService.setUserCredentials(userName,password); URL postUrl = new URL("http://www.google.com/calendar/feeds/"+userName+"/private/full"); CalendarEventEntry myEntry = new CalendarEventEntry(); myEntry.setTitle(new PlainTextConstruct(titleOfEvent)); myEntry.setContent(new PlainTextConstruct(contentToPost)); DateTime startTime = new DateTime(new Date()); When eventTimes = new When(); eventTimes.setStartTime(startTime); myEntry.addTime(eventTimes); CalendarEventEntry insertedEntry = calendarService.insert(postUrl, myEntry); CalendarEventEntry myEntry2 = new CalendarEventEntry(); String now=(new Date()).toString(); myEntry2.setContent(new PlainTextConstruct("Test post at "+now)); myEntry2.setQuickAdd(true); // Send the request and receive the response: CalendarEventEntry insertedEntry2 =calendarService.insert(postUrl, myEntry2); Create the event time and insert Create a new Calendar entry Log in to the calendar service An alternative way to post the entry

31. Some Other Things You Can Do with GData Server Code YouTube – Search movies by keyword tag, “related”, categories – Upload and download movies. – Put movies into collections Calendar – Retrieve events as RSS feed Spreadsheet – Remotely retrieve and insert or change row and cell data. – Use structured queries to retrieve data ranges. – Remotely invoke batch operations

32. Making FaceBook Applications Using Facebook as a backend service.

33. Full Examples and More Detailed Notes PHP and JavaScript notes – http://communitygrids.blogspot.com/2008/02/qu ick-facebook-javascript-api-notes.html JavaScript example: – http://communitygrids.blogspot.com/2008/02/litt le-more-jsfacebook-hacking.html

34. Why Use FaceBook as a Portal? Because it has 10’s of millions of users. – The largest Web 2.0-style Web portal Easy to create Social Networks: groups of friends and share applications, communicate, share information, etc. – And more importantly, tools for building and managing these networks Can leverage many third party applications – For example, photo albums – Of course many of them are trivial, silly

35. FaceBook APIs These come in two flavors – Embedded: make your application available through FaceBook. PHP APIs – Embedding: use Facebook authentication and social network data in your application. JavaScript API In both cases, you run your application on your Web server. – You application do anything you can implement. – And you can query FaceBook for social network information, user metadata.

36. Using The JavaScript API Getting Started: – You just need a Web server to host your application. – Register as a developer – Place Facebook’s xd_receiver.htm on your web server Make sure the relative path used is correct. What will happen? – Your page will run outside of Facebook. – Anyone loading your application will be directed to Facebook to login. – User is then redirected back to your page. Your application can make calls back to Facebook. And you can do anything else you want. – So you could embed this application in a portlet, interact with a local database, – You can build your application with.NET or Java Server Faces, etc.

37. HTML Page Example var api = new FB.ApiClient(' ', '/xd_receiver.htm', null); …

38. Interact with FaceBook api.requireLogin(function(exception) { api.friends_get(null,getResults); var myinfo= 'last_name, first_name, hometown_location, work_history, pic_small'; api.users_getInfo(api.get_session().uid, myinfo, getInfo); });

39. Notes on Previous Slides The requireLogin, friends_get, and users_getInfo are FaceBook API calls. The getResults and getInfo methods are callback methods that I defined. – Shown in next slide The myinfo data structure is taken from the Facebook XML user profile. – http://wiki.developers.facebook.com/index.php/U sers.getInfo http://wiki.developers.facebook.com/index.php/U sers.getInfo

40. Developer Defined Call-Back Functions function getResults(result,exception) { Debug.dump(api.get_session().uid,’You'); Debug.dump(result,’Your Friends’); } function getInfo(result,exception) { Debug.dump(result,’Your info'); } These two methods receive the output returned from Facebook by the API’s friends_get() and users_getInfo() methods. The results are dumped to the text window but you can do more interesting things.

41. Output Info Dumped to Text Area you: 627774031 Your friends: {Array} [Deleted] Your Info: {Array} [0]: {Object} first_name: Marlon hometown_location: {Object} city: state: country: zip: last_name: Pierce pic_small: http://profile.ak.facebook.com/profile6/1797/98/t627774031_2463.jpg uid: 627774031 work_history: {Object} User’s Facebook ID and the ID numbers of his friends User’s profile metadata. Note some fields are strings, some are Arrays, and some are structured objects of strings.

42. Facebooks’ Query Language All of the FaceBook API is basically a set of wrappers around their SQL like FQL query language. So you can, if you prefer, make custom methods out of FQL query strings. var myQuery='SELECT name FROM user WHERE uid='+api.get_session().uid; api.fql_query(myQuery, getFQLResponse); getFQLResponse() is a developer-written callback function

43. Working with Facebook Groups To list information about groups you are a member of, use the following. – You can replace the value of uid with any UID you know. – 18629081888 is the Group ID for the Open Grid Forum’s uninteresting Facebook group api.groups_get(api.get_session().uid,null,getGroups); api.groups_getMembers(18629081888,getGroupMemb ers);

44. More Information The JavaScript documentation is available from here – http://wiki.developers.facebook.com/index.php/J avaScript_Client_Library http://wiki.developers.facebook.com/index.php/J avaScript_Client_Library It is pretty minimal, but it is essentially the same as the better documented PHP API. – Replace PHP’s "." with a "_". – So PHP’s users.getInfo becomes users_getInfo(...) in JavaScript.

45. Open Social API and Gadgets

46. What Is a Gadget? Simple gadgets for getting a Grid proxy credential and running remote commands. Both run on my own Web server.

47. Gadget Definition This XML page is web accessible via http://156.56.104.143:8080/GTLAB/MyProxyGadget.xml http://156.56.104.143:8080/GTLAB/MyProxyGadget.xml This URL points to the actual application (written in JSF) Google uses an HTML IFrame to load the page. You can also use content type=“html” for more integrated applications. – For example, you can send messages between gadgets in the client.

48. Gadgets and Open Social Open Social is a consortium of Facebook’s major competitors. – Orkut, MySpace, Hi5, LinkedIn, Friendster, etc. The Open Social API has two parts – JavaScript libraries for building social gadgets. Must run in a gadget container (server) – REST APIs for exchanging data between container servers. The ideas: – JS APIs let you make portable gadgets that will work in different containers (Hi5  LinkedIn) but data will be different. – REST will let you export/import data.

49. Getting Started You need a Gadget container to host your gadgets. – LinkedIn is notably selective about the gadgets it will host. Orkut provides a sandbox, but you have to upload your gadgets. Apache Shindig is an extremely simple Java-based container. – De facto reference implementation of Open Social – Not suitable for production by a long ways. – But it will work on an air plane. Shindig is something of a moving target. – My February notes were obsolete…

50. Getting Started with Shindig Visit the Web site for information – http://incubator.apache.org/shindig http://incubator.apache.org/shindig Check out the code with SVN – mkdir $HOME/shindig; cd shindig – svn co http://svn.apache.org/repos/asf/incubator/shindig/trunk/http://svn.apache.org/repos/asf/incubator/shindig/trunk/ Build and run it with Apache Maven – Run “mvn” from $HOME/shindig/ – Run “mvn -Prun” from $HOME/shindig/java/server Point browser to http://localhost:8080/gadgets/files/container/sample1.html and look through other samples. http://localhost:8080/gadgets/files/container/sample1.html – You can also load these HTML files using file:// if you don't want to run the Jetty server. Periodically check for updates and rebuild – svn update The READMEs have better/more up-to-date information than the website.

51. What Do You Get? Shindig actually has two major parts: – A container that can run social gadgets This must have access to your user and social network database. – A gadget aggregator that displays gadgets Runs the layout manager. Think of this as your own iGoogle server. Access the gadget container – http://localhost:8080/gadgets/files/samplecontai ner/samplecontainer.html

52. The gadget can be displayed in your http://localhost:8080 display container.http://localhost:8080 The gadget is actually served remotely. This is the module definition.

53. Import JS Libraries into Container Page

54. var my = {}; my.gadgetSpecUrls = [ 'http://www.google.com/ig/modules/horoscope.xml', 'http://www.google.com/ig/modules/aue07otr.xml', 'http://www.labpixies.com/campaigns/todo/todo.xml' ]; my.LayoutManager = function() { gadgets.LayoutManager.call(this); }; my.LayoutManager.inherits(gadgets.LayoutManager); my.LayoutManager.prototype.getGadgetChrome = function(gadget) { var chromeId = 'gadget-chrome-' + gadget.id; return chromeId ? document.getElementById(chromeId) : null; }; Create Your Layout Manager Note gadgets are loaded from remote host containers.

55. my.init = function() { gadgets.container.layoutManager = new my.LayoutManager(); }; my.renderGadgets = function() { for (var i = 0; i < my.gadgetSpecUrls.length; ++i) { var gadget = gadgets.container.createGadget( {specUrl: my.gadgetSpecUrls[i]}); gadgets.container.addGadget(gadget); gadgets.container.renderGadget(gadget); } }; Initialize and Render

56. Sample: Dynamic Height HTML Display Shows the 3 Gadgets. As we saw, this will use “flow” layout and “chrome” styling.

57. Hosting Your Own Social Gadget Place your widget code here: – $HOME/shindig/javascript/samplecontainer/examples After compilation, these will be located here: –./target/gadgets/files/samplecontainer/examples. These will be served up from – http://localhost:8080/gadgets/files/samplecontainer/ examples/myapp.xml http://localhost:8080/gadgets/files/samplecontainer/ examples/myapp.xml Myapp.xml is the gadget definition we saw earlier.

58. http://localhost:8080/gadgets/files/samplecontainer/samplecontainer.html Hello World Gadget in Container User and his network of friends provided by the local container. “Hello” chosen based friend’s preferred Uses flat XML file.

59. Shindig social network XML “database”. Real containers would use a real database. OpenSocial API essentially lets you retrieve this sort of data.

60. my.gadgetSpecUrls = [ 'http://www.google.com/ig/modules/horoscope.xml', 'http://www.google.com/ig/modules/aue07otr.xml', 'http://localhost:8080/gadgets/files/samplecontainer/examples/So cialHelloWorld.xml' ];

61. More Information See http://communitygrids.blogspot.com for more detailed information.http://communitygrids.blogspot.com Email: mpierce@cs.indiana.edumpierce@cs.indiana.edu