1. 1 Mashroom: End-User Mashup Programming Using Nested Tables Guiling Wang, Shaohua Yang, Yanbo Han Institute of Computing Technology (ICT) Chinese Academy of Sciences http://sigsit.ict.ac.cn Email: wangguiling@software.ict.ac.cn INSTITUTE OF COMPUTING TECHNOLOGY, CAS

2. 2 Outline Introduction Foundamentals Programming Model –Application Structure –Data Model –Script & Operators Trail Application Evaluation Related Works Conclusion

3. 3 Motivation HTML, RSS/Atom, Open API Mashup

4. 4 Motivation Seldom can a single website provide an integrated view of movies on demand –title, director, showtime, reviews, map, … Showtimes Website http://www.google.com/movies http://www.imdb.com … Movie reviews Website http://www.mrqe.com http://www.douban.comhttp://www.douban.com,...

5. 5 Motivation End-User Mashup Programming –Enable non-professional users to build Web applications by combining functionalities offered by more than one websites to deal with situational and ad-hoc problems Popular mashup editors adopt flow-chart-like formalisms like Yahoo! Pipes, MS Popfly, … –Some studies show that the concept of data flow is the main barrier for end users [Wong,07] Explicit control operators: branches, loops, … Explicit alignment of inputs and outputs at each step

6. 6 Problem Definition Spreadsheet programming can also been adopted in end-user mashup programming like C3W, SpreadMash, … –Excellent end user programming paradigm, but also face some challenges Goal –Enable non-professional users to build mashups –Adopt spreadsheet programming paradigm –Balance usability and expressiveness

7. 7 Outline Introduction Foundamentals Programming Model –Application Structure –Data Model –Script & Operators Trail Application Evaluation Related Works Conclusion

8. 8 Foundamentals of Mashroom Programming Learn from nested relation model and nested table –Nested relation model is one of the most adopted data model for representing web data. –Strong foundation both in query algebra and in query optimization –Nested table is understandable by end users Challenges –existing graphical query languages based on the nested table are low-level SQL-like, QSByE [Filha,01] and GXQL [Qin,04] Data query operators are not enough

9. 9 Foundamentals of Mashroom Programming Learn from Spreadsheet –Loop execution is described by selecting a range of cells intuitively –Allows users to manipulate data by flexibly editing a formula Challenges –Enable spreadsheet suitable for displaying nested table –Enable direct manipulation on the nested relational data model

10. 10 Foundamentals of Mashroom Programming- Learn From Programming by Example

11. 11 Outline Introduction Foundamentals Programming Model –Application Structure –Data Model –Script & Operators Trail Application Evaluation Related Works Conclusion

12. 12 Application Structure

13. 13 Data Model : MashSheet σ(Theaters(MoviesLabel=”Waiting in Beijing”)) MashSheet = {worksheet} a worksheet is a nested table “column” is the first-class object Mashroom formula is built from column references, operators, and constant values

14. 14 Data Mashup Script and Operators script = –vars : global input parameters op = –actor : computation that computes on nested tables from actorIn –actorIn : a set of input variables

15. 15 Operators in Mashroom Operators Operations on Nested Relation Model Worksheet CreationImport, CreateSheetNew, Insert Worksheet Data Manipulation Filter, Sort, HeaderTruncate, TailTruncate Selection(σ) Worksheet Schema Manipulation DeleteColumn, RenameColumn, Nest/Unnest Delete,Update, Nest(η),Unnest(μ) Worksheet CleaningAddFunction, Replace, MergeInstance Insert, Selection(σ) Worksheet Composition Merge, Fuse, LinkService Union( ∪ ),Join( ) Worksheet ExportSink －

16. 16 Worksheet Creation Operators (Import, CreateSheet) createsheet(sheet0:Theaters/Movies)

17. 17 Worksheet Data Manipulation Operators (Filter, Sort, HeadTruncate, TailTruncate) filter(sheet0:Book/Price, (sheet0:Book/Price 80))

18. 18 Worksheet Schema Manipulation Operators (DeleteColumn, RenameColumn, Nest/Unnest )

19. 19 Worksheet Cleaning (mergeInstance, addFunction, replace)

20. 20 Worksheet Composition (merge, fuse, linkService) linkservice(sid, mapping,…)

21. 21 Operator Visualization Nested table

22. 22 Operator Visualization : Drag and Drop create a new worksheet merge

23. 23 Operator Visualization Implicit control operators: loops, … Avoid alignment of inputs and outputs at each step

24. 24 Outline Introduction Foundamentals Programming Model –Application Structure –Data Model –Script & Operators Trail Application Evaluation Related Works Conclusion

25. 25 Trail Application IMDBDS (from imdb.com) –Output : (label, director, actor) GooglemovieDS (from shenghuo.google.cn) –Input: city –Output : (title, director, actor) MovieReviewsDS (from douban.com) –Input: subjectID –Output: reviewLink, summary How to get the integrated movie list with reviews ?

26. 26 Trail Application(1)

27. 27 Trail Application(2)

28. 28 Trail Application(3)

29. 29 Demo Video

30. 30 Evaluation Goal –expressivity and usability Method –10 popular mashups from Yahoo! Pipe community (those have high “clone” number), two mashup directories (programmableWeb.com and mashupAwards.com) and a set of other typical mashups. –“Mashup Patterns” –User study with 5 groups of people to measure the average time to build a mashup

31. 31 Aggregation Patterns Aggregation for Collection (same kind of Web Sources) –e.g. “Aggregated News Alerts”, search at Bloglines, Google Blog Search, Microsoft Live News etc. Aggregation for Comparison (same kind of Web Sources) –e.g. “Book Price Comparison”, compares the price of the same book from dangdang.com and amazon.cn Focus View or Data Analysis (single Web Source) –e.g. “Focus View of YouTube Video”, list YouTube videos of a certain category or tag Aggregation for Collection (different kinds of Web Sources with dependency) –e.g. “Aggregated Movie Reviews”

32. 32 “similarity aggregation without dependency” pattern

33. 33 “similarity aggregation with comparison” pattern

34. 34 “focus view or analysis” pattern

35. 35 “aggregation with dependency” pattern “search subjectID first” pattern

36. 36 Selected Mashups, Patterns and Experiment Results

37. 37 Related Works flow-chart-like programming –Microsoft Popfly, IBM Damia, Marmite Yahoo Pipes –Provide a set of flow-chart-based graphical operators and adopt a flow-style orchestration specification spreadsheet-like programming –SpreadMash and C3W –Data services are processed in a manner similar to spreadsheets tree-based programming –The old version of Intel MashMaker –Data services are combined based on a tree structure browser-centric programming –Ubiquity and d.mix –Provide mechanisms for users to trigger the mashup operations in the context of browsing

38. 38 Related Works flow-chart-like programming (Yahoo! Pipe) spreadsheet- like programming (C3W) Tree-based programming (old version MashMaker) Mashroom Knowledge required, learning curve Data flow modeling knowledge Spreadsheet using knowledge Specific tree- based programming knowledge a little spreadsheet knowledge Service dependency Alignment of Input and Outputs in data flow Cell reference in spreadsheet Parameter configuration Contextual menu and dialog Complex control operator Explicit control module Control expressions on cell reference Hide complex control operator

39. 39 Conclusion A programming model for building mashups by end users –the design philosophy –the abstraction of mashup applications, and the implementation –combine the nested table with the spreadsheet- like programming –Evaluation