2. June 1999 CHAIMS1 Neal Sample Stanford University Objective: Investigate new approaches to large-scale software composition. Approach: Develop and validate a composition-only language (CLAM), a protocol for large, distributed, heterogeneous and autonomous megamodules (CPAM), and a supporting system. CHAIMS: A Reuse and Composition System for Services

3. June 1999 CHAIMS2 Shift in Programming Tasks Coding Integration, Composition, Reuse 1970 1990 2010

4. June 1999 CHAIMS3 Composition of Processes... versus composition and integration of Components –reusing small components (copy/paste, shared library, local install, etc.) –large distributed components within same “domain” (e.g. within one bank or airline)

5. June 1999 CHAIMS4 Domain expert, Programmers local computer or local distributed system I/O Purchasing Local Components a b c d e e and copy Distributor purchase

6. June 1999 CHAIMS5 Composition of Processes... versus composition and integration of Components –reusing small components (copy/paste, shared library, local install, etc.) –large distributed components within same “domain” (e.g. within one bank or airline) versus composition and integration of Data –data-warehouses –wrapping data available on the web

7. June 1999 CHAIMS6 Accessing Remote Data Control & Computation Services Wrappers s e r v e r s c l i e n t I/O a b c d e Domain expert, Programmers local computer or local distributed system Data Resources

8. June 1999 CHAIMS7 Composition of Processes... versus composition and integration of Components –reusing small components (copy/paste, shared library, local install, etc.) –large distributed components within same “domain” (e.g. within one bank or airline) versus composition and integration of Data –data-warehouses –wrapping data available on the web CHAIMS/CLAM: –compose remote autonomous services

9. June 1999 CHAIMS8 Accessing Remote Services a b c d e s e r v e r s at provider's sites c l i e n t s control data IO module Domain expert Client workstation

10. June 1999 CHAIMS9 Accessing Remote Services s e r v e r s at provider's sites c l i e n t s control data IO module Domain expert Client workstation IO module Domain expert Client workstation a b c d e

11. June 1999 CHAIMS10 IO module Domain expert Client workstation Accessing Remote Services s e r v e r s at provider's sites c l i e n t s control data IO module Domain expert Client workstation a b c d e IO module Domain expert Client workstation

12. June 1999 CHAIMS11 Composing Computational Services Megamodules IO module a b c d e s e r v e r s at provider's sites c l i e n t CPAM and distribution systems control data IO module Domain expert Client workstation Remote Computation Services Data Resources

13. June 1999 CHAIMS12 Example of Remote Services Bio-informatics: »several research institutes develop algorithms for modeling and matching molecule structures »several knowledge bases available, many of them with additional computational services »a scientist has to combine several of these processing units to produce desired result

14. June 1999 CHAIMS13 Typical Scenario: Logistics A general has to ship troops and/or various material from Los Angeles to Washington DC: –different kind of material: criteria for preferred transport differ –not every airport equally suited –congestion, prices –actual weather –certain due or ready dates Today: calling different companies, looking up information on the web, reservations by hand » hand-coded systems Tomorrow: system proposes possibilities that take into account various conditions » composition of processes

15. June 1999 CHAIMS14 Advantages Knowledge »separation of concerns (provider - composer) »technical expertise (databases …) »maintenance, newest algorithms Availability »pay per usage instead of pay per instance »lower barrier for clients to try out new modules »choice of several providers even at run-time Resources »access to computation intensive modules for everybody »minimal local resources possible Yet there are also challenges... Advantages of reuse by accessing remote services:

16. June 1999 CHAIMS15 Challenge: Computation Intensive Services What we would like: ==> exploiting parallelism among services without complex coding on client side ==> partial extraction of results when they are ready ==> progressive extraction of results ==> monitoring progress of service invocation and results ==> cost estimation prior to invocation of a service for possibility to choose cheapest or fastest service Services are not free for a client: execution time of a service transfer time and fee for data fees for services

17. June 1999 CHAIMS16 Challenge: Autonomy Separation of control ==> client has no direct control over: availability of services resources provided for services Heterogeneity: heterogeneity concerning languages, platforms as well as distribution protocols legacy code, applications and libraries

18. June 1999 CHAIMS17 Challenge: Composition Task Services must somehow be composed on the client-side automated: generation of effective access code composition-oriented: composition only language or interface easy: simple for domain experts to use

19. June 1999 CHAIMS18 other language or interpreter or incremental programming server side client side CHAIMS Architecture CPAM protocol Distribution Systems (CORBA, RMI…) Composer interacts with e d a b c MEGA Modules CHAIMS Repository adds information to Megamodule Provider provides megamodules Wrapper Templates

20. June 1999 CHAIMS19 Implementation of CPAM On top of various protocols: »RMI, CORBA, DCE, plain TCP/IP, http »each primitive is a procedure call, or a pair of XML messages »or mimicked by other interfaces: SWAP, several methods Method parameters: »simple data element, or recursive structure of data elements »each data element has also type and descriptive name information »encoding: now ASN.1, future XML »either megamodule conformity or ontology mediator megamodules Wrappers for CPAM compliance: »asynchrony and state management (client-specific parameters, results) »progress information and cost estimation »data encoding

21. June 1999 CHAIMS20 server side client side CHAIMS Architecture CPAM protocol Distribution Systems (CORBA, RMI…) Composer writes Client Side Run Time CLAM program CHAIMS Compiler generates e d a b c MEGA Modules CHAIMS Repository adds information to Megamodule Provider provides megamodules Wrapper Templates

22. June 1999 CHAIMS21 Architecture: CLAM and CHAIMS-Protocols Megaprogram Megaprogrammer M e g a m o d u l e s CLAM CHAIMS-protocols CORBA-idlDCE-idlJava-class Compiler/Interpreter

23. June 1999 CHAIMS22 CLAM Primitives Pre-invocation: SETUP: set up the connection to a megamodule SET-, GETPARAM: in a megamodule ESTIMATE: for optimization Invocation and result gathering: INVOKE EXAMINE: test progress of an invoked method EXTRACT: extract results from an invoked method Termination: TERMINATE: terminate a method invocation/connection to a megamodule Control: WHILE, IF

24. June 1999 CHAIMS23 Pre-invocation: SETUP Establish communication between client CLAM program and a megamodule Returns a megamodule handle Maps logical name of service to provider ex: mm_handle = SETUP (“TransportationModule”) CLAM programmers avoid module location and transportation protocol issues Authentication occurs here (open question).

25. June 1999 CHAIMS24 Pre-invocation: ESTIMATE Returns cost of a method invocation in terms of fee, time, and data volume Considers values of method parameters ESTIMATE for runtime module selection Useful for scheduling and cost minimization.

26. June 1999 CHAIMS25 Cost Estimation for Selection Choosing megamodules: RouteChoose - Optimum - … BestPick - BestRoute -... CHAIMS clientCHAIMS client time SETUP () SETPARAM ( attributes essential for cost estimation ) (f1=fee, t1=time) = ESTIMATE (“Optimum”) SETUP () SETPARAM ( attributes essential for cost estimation ) (f2=fee, t2=time) = ESTIMATE (“BestRoute”) calculate cost function ==> today BestPick is better BestPick.INVOKE (“BestRoute”, …)

27. June 1999 CHAIMS26 Pre-invocation: SET-, GETPARAM GET-: returns current parameter values SET -: overrides current parameter values Operates on a megamodule handle ex: mm_handle.SETPARAM (city = “Amsterdam”) Utility –presetting for ESTIMATE –avoid resetting with repeated invocations –simple input perturbation.

28. June 1999 CHAIMS27 Example: parameters of method TripPlan general default values client-specific settings for client A method TripPlan, client A: invocation-specific settings client-specific settings for client B fromdest start- date end- date number 1JAN1998 6OCT19984OCT1998 7OCT19984OCT1998 LASBWI1 2 actual values used in invocation TripPlan1 SJOZRH SFO 1 1 SJO6OCT19984OCT1998ZRH actual values used in invocation TripPlan2 FRA 7OCT1998 9DEC19981DEC1998 9DEC19981DEC19982LASFRA actual values used in invocation TripPlan1 method TripPlan, client A: invocation-specific settings method TripPlan, client B: invocation-specific settings Hierarchical Setting of Parameters

29. June 1999 CHAIMS28 CLAM Primitives Pre-invocation: SETUP: set up the connection to a megamodule SET-, GETPARAM: in a megamodule ESTIMATE: for optimization Invocation and result gathering: INVOKE EXAMINE: test progress of an invoked method EXTRACT: extract results from an invoked method Termination: TERMINATE: terminate a method invocation/connection to a megamodule Control: WHILE, IF

30. June 1999 CHAIMS29 Invocation: INVOKE Starts execution of a method Operates on a megamodule handle Returns an invocation handle ex: in_handle = mm_handle.INVOKE (“GenericMethod”) in_handle = mm_handle.INVOKE (“RoutePlanMethod”, Pair_of_Cities = cities) Achieves asynchrony in sequential clients.

31. June 1999 CHAIMS30 Invocation: EXAMINE Determines the state of a invocation Operates on an invocation handle Returns: –(status, “progress information”) –status = {DONE, NOT_DONE, PARTIAL, ERROR} Utility –Progressive extractions –Partial extractions –Optimized module execution scheduling –Steering.

32. June 1999 CHAIMS31 Invocation: EXAMINE quick_route:found at time 30 optimal_route:found at time 100 Method Status: WHILE (in_handle.EXAMINE() != DONE) {} (routeVar = quick_route) = in_handle.EXTRACT() Attribute Status: WHILE (in_handle.EXAMINE(quick_route) != DONE) {} (routeVar = quick_route) = in_handle.EXTRACT() Steering: IF (in_handle.EXAMINE(quick_route) != PARTIAL) … ask user (or scheduler) to continue or halt…

33. June 1999 CHAIMS32 Invocation: EXTRACT Collects data from a megamodule method Operates on an invocation handle Returns: assignment of values to parameters ex: (routeVar = quick_route) = in_handle.EXTRACT() Allows partial data retrieval –saves volume: abandon uninteresting elements –saves time: termination of useless invocation Allows progressive data retrieval.

34. June 1999 CHAIMS33 CLAM Data Types Simple data types still have a role –opaque//exclusively used for user data –integer –string –Boolean –real No classes or amalgamated types Only be assigned and compared Used for control (not opaque)

35. June 1999 CHAIMS34 CLAM Primitives Pre-invocation: SETUP: set up the connection to a megamodule SET-, GETPARAM: in a megamodule ESTIMATE: for optimization Invocation and result gathering: INVOKE EXAMINE: test progress of an invoked method EXTRACT: extract results from an invoked method Termination: TERMINATE: terminate a method invocation/connection to a megamodule Control: WHILE, IF

36. June 1999 CHAIMS35 TERMINATE Ends method invocation or closes a connection to a megamodule Why necessary? –delayed extractions –repeated/progressive extractions –partial extractions –early termination ex: in_handle.TERMINATE () mm_handle.TERMINATE ()

37. June 1999 CHAIMS36 CLAM Composition Language Purely compositional: –no primitives for arithmetic –no primitives for input/output Splitting up CALL-statement: –parallelism by asynchrony in sequential program –novel possibilities for optimizations –reduction of complexity of invoke statements Higher-level language: –assembler => HLLs, –HLLs => composition/megamodule paradigm

38. June 1999 CHAIMS37 Ongoing & Upcoming Research Issues CPAM automated composition CLAM graphical composition incremental composition automated scheduling of invocations electronic marketplace of services exploiting XML web-services direct dataflows between megamodules optimization of compositions repository in XML,... directory services CHAIMS broker repository browser composition wizard sign-up and authentication cost estimation incremental result extraction c o m p o s i t i o n e - c o m m e r c ep r o t o c o l Repository

39. June 1999 CHAIMS38 Conclusion New reuse model »computational modules stay at provider’s site »reuse of computational modules by accessing computational services over the internet New issues »cost-estimation, progress monitoring »easy and efficient composition of remote, computation-intensive services

40. June 1999 CHAIMS39 Questions? http://www-db.stanford.edu/CHAIMS/

41. June 1999 CHAIMS40 Other Approaches Traditional Languages/Extensions –FX: Self-contained language for module building composition. Much finer grain than service composition. –Orca, Opus: Extensions to HPF. Programmers require intimate knowledge of all process data. Composition Languages –MANIFOLD: Similar IWIM model, with black box process –Myriad scripting languages, generally assume complete ontological control of services

42. June 1999 CHAIMS41 Megaprogram Example: Code io_mmh = SETUP ("InputOutput") route_mmh = SETUP ("RouteInfo")... best2_mmh.SETATTRIBUTES (criterion = "cost") cities_default = route_mmh.GETPARAM(Pair_of_Cities) input_cities_ih = io_mmh.INVOKE ("input”, cities_default) WHILE (input_cities_ih.EXAMINE() != DONE) {} cities = input_cities_ih.EXTRACT()... route_ih = route_mmh.INVOKE ("AllRoutes", Pair_of_Cities = cities) WHILE (route_ih.EXAMINE() != DONE) {} routes = route_ih.EXTRACT() … IF (best1_mmh.ESTIMATE("Best_Route") < best2_mmh.ESTIMATE("Optimum") ) THEN {best_ih = best1_mmh.INVOKE ("Best_Route", Goods = info_goods, Pair_of_Cities = cities, List_of_Routes = routes, Cost_Ground = cost_list_ground, Cost_Air = cost_list_air)} ELSE {best_ih = best2_mmh.INVOKE ("Optimum", Goods = info_goods, …... best2_mmh.TERMINATE() // Setup connections to megamodules. // Set global variables valid for all invocations // of this client. // Get information from the megaprogram user // about the goods to be transported and about // the two desired cities. // Get all routes between the two cities. //Get all city pairs in these routes. //Calculate the costs of all the routes. // Figure out the optimal megamodule for // picking the best route. //Pick the best route and display the result. // Terminate all invocations

43. June 1999 CHAIMS42 Name of Person Data objects: Gentype bags Minimal Typing within CHAIMS: Integer, Boolean only for control All else is placed into an ASN.1 bag, transparent to compiler : A Gentype is a triple of name, type and value, where value is either a simple type or a list of other gentypes (i.e. a complex type). Simple types: given by ASN.1, the ASN.1-conversion library for C++, our own conversion routines. Example: Person_Information complex First Name string Joe Last Name string Smith Personal Data complex Address Date of Birthdate 6/21/54 Soc.Sec.No string 345-34-3456

44. June 1999 CHAIMS43 Scheduling: Simple Example 1 cost_ground_ih = cost_mmh.INVOKE ("Cost_for_Ground", 1 List_of_City_Pairs = city_pairs,Goods = info_goods) 2 WHILE (cost_ground_ih.EXAMINE() != DONE) {} 3 cost_list_ground = cost_ground_ih.EXTRACT() 3 cost_air_ih = cost_mmh.INVOKE ("Cost_for_Air", 2 List_of_City_Pairs = city_pairs,Goods = info_good) 4 WHILE (cost_air_ih.EXAMINE() != DONE) {} 4 cost_list_air = cost_air_ih.EXTRACT() order in unscheduled megaprogram order in automatically prescheduled megaprogram