1. Slide 1 Metropolis Pat Helland Architect, Microsoft Metropolis: The Evolution of the IT Shop Into the World of Services June 7, 2004

2. Slide 2 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

3. Slide 3 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

4. Slide 4 The Metropolis Analogy IT Shops and Cities –Gradual Evolution –Developed in Isolation Railroads Arrive –People Travel –Stuff Travels Commodity versus Manufactured Internet Arrives –People Browse –Data Moves Commodity versus Structured We propose that this analogy shows us a lot about where we are heading!

5. Slide 5 Metropolis Cities IT Shops Factories and Buildings Apps Trans-portationCommun-ication Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance

6. Slide 6 Implications of Metropolis Heterogeneity Happens! Ongoing IT Investment –Infrastructure versus Business –Historic Monuments Standardization Is Nascent –Connection Largely by People –Efficiencies Still to Come Business Process Is Nascent –Still Mostly Ad-hoc –Growing to Become Dominant Force Loose Coupling Helps Investments

7. Slide 7 Cities IT Shops Factories and BuildingsApps Trans-portationCommun-ication Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance Cities IT Shops Metropolis

8. Slide 8 The Evolution of Cities Gradual Growth –Gather for Commerce and Manufacturing Independent Buildings –No Connections Independent Cities –Travel Too Hard –Cities Did Things Their Own Way

9. Slide 9 The Evolution of the IT Shop Gradual Growth –New Apps Gradually Built Applications Largely Independent –People Interact Separately with Apps B2B Still Limited –Largely via People –Independently Designed and Incompatible Apps

10. Slide 10 Cities and IT Shops Big Complex and Evolving Environments –Fed by Economics –Ongoing Investment New and Renovated Buildings (or Apps) Infrastructure for Connectivity –Both Have Historic Monuments to Consider!

11. Slide 11 Cities IT Shops Factories and BuildingsApps Trans-portationCommun-ication Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance Factories and Buildings Apps Metropolis

12. Slide 12 Factories and Buildings Early 1800s –Independent Structures Shipping Difficult –Many Different Types of Structures Housing, Barns, Churches, Inns, Basic Factories Late 1800s –Autonomous Yet Interconnected Located by Railroads Stuff Efficiently Flows Between Buildings City Infrastructure –Even More Types High-Density Housing, Sophisticated Factories, Offices, Retail, Train Stations –Economy Based on the Flow of Stuff

13. Slide 13 Applications Today –Independent Applications Raw Communication (Shipping Bytes) Easy Understanding Structured Data and Operations Hard –Many Different Types of Apps On-Line Transaction Processing, Web-Based, Scientific, CAD, Client-Server, Games Tomorrow –Autonomous Yet Interconnected Structured Data and Ops Flow Tapped into IT Infrastructure –Economy Based on the Flow of Structured Data and Ops No App Is an Island

14. Slide 14 Factories and Apps Independence Is Essential –Get Work Done –Decoupled Evolution Advantages to Interconnection –Leverage Others –Economy Based on Flow of Stuff (Data and Ops) Tap Into Infrastructure –Services from City or IT-Shop

15. Slide 15 Factories and Buildings Apps Trans-portationCommun-ication Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance Trans-portation Commun- ication Metropolis

16. Slide 16 Transportation Railroad Arrives! –Money Moving People & Raw Commodities –Speculations, Booms, & Busts… Movement of People Changed Things –Retail Expanded Movement of Stuff Changed Things –Stuff Had to Work Together –Retailers Could Gather Stuff for Sale

17. Slide 17 Communication Internet Arrives! –Money In Browsing & Moving Raw Data –Speculations, Booms, & Busts… People Browsing Changed Things –Directly Access Remote Apps –Driving Demand for Business Process Changes from Movement of Data Nascent –Data Still Doesn’t Work Together –Business Process Still Very Limited

18. Slide 18 Transportation and Communication Started Moving People and Commodities Drove New Changes –Standardization Stuff and Data –Retail and Business Process –Economic Consolidation Cities IT-Shops Using Structured Data

19. Slide 19 Trans-portation Commun- ication Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance Manu- factured Goods Structured Data and Operations Metropolis

20. Slide 20 Manufactured Goods Early 1800s –Hand-crafted Goods -- “Trim-and-Shim” –American System of Manufacturing –Eli Whitney – Interchangeable Parts Late 1800s –De Facto Standards Per Industry –Marketplace Demanded Compatibility –Retailing Demanded Compatibility Retooled or Went Under! See "Eli Whitney" by Constance McL. Green Addison Wesley Longman 1956

21. Slide 21 Structured Data Today –Non-Standard Data Structures –Mostly Human “Trim-and-Shim” to Integrate –Beginnings of Standardization Web Services Foundation Need Industry Standards Demand for Business Process Soon –Industry De Facto Standards –Increased Compatibility –More Sophisticated Business Process

22. Slide 22 Manufactured Goods and Structured Data Must Connect to Other Stuff –Can’t Live in Isolation –Manufacturing Retooled New Efficiencies and Markets Came –Applications Must Retool Data and Business Process Integration Tremendous Payoffs to Come

23. Slide 23 Retail & Distribution Manu- factured Goods Structured Data and Operations Retail & Distribution Business Process Manu- factured Assemblies Virtual Enterprises Urban Infra- structureIT Infra- structure City Government ITGovernance Business Process Urban Infra- structureIT Infra- structure ITGovernance Metropolis Manu- factured Assemblies Virtual Enterprises

24. Slide 24 Manufactured Assemblies Assemble Parts from Other Factories –Requires Detailed Standardization –Produces High-Value Goods –Leverage Other Companies’ Stuff Competition Drove Efficiencies –Better Parts –Better Prices

25. Slide 25 Virtual Enterprises Business Function Outsourcing –Acquire Business Functionality Outside –Produce More Sophisticated Business Value –Concentrate on Your Center of Excellence Competition Drives Efficiencies –Better Quality & Price Requires Standards –Data and Biz-Process

26. Slide 26 Manufactured Assemblies and Virtual Enterprises Standards Allow Composable Stuff (Data and Ops) Better Stuff (Data and Ops) is Created –Combining Efforts of Many Companies Increased Efficiencies –Lower Prices or Greater Profitability –Companies Focus on Specialty

27. Slide 27 Manu- factured Assemblies Virtual Enterprises Retail & Distribution Business Process Urban Infra- structureIT Infra- structure City Government ITGovernance Metropolis Retail & Distribution Business Process

28. Slide 28 Retail and Distribution Late 19th Century –Bring People to Stores Trains Brought Shoppers –Bring Standardized Stuff to Stores Department Stores Emerge –Send Standardized Stuff to People Mail Order 20th Century –Department Stores and Supermarkets –Wal-Mart Shift in Power to the Retailer!

29. Slide 29 Business Process Today: Looking Ahead: –Data and Operation Standards –Need “Interchangeability” Standardized Clothes Sizes Not as Specific and Detailed –Allows Pre-allocation –Makes Biz Process Efficient Swivel-Chair Integration Excellent for EAI across apps FAX and Pray Integration Most common form of B2B work ALT-TAB Integration Reduces errors via the clipboard

30. Slide 30 Retail and Business Process Amazing Transformation in Retail –People Cheerfully Accept Standard Stuff Customization Is Rare and Expensive Business Process Mostly Hand-Crafted –Poor Standards; Manual “Trim and Shim” –Poor “Interchangeability” Business Process Will Grow to Drive the Apps!

31. Slide 31 Retail & Distribution Business Process Urban Infra- structureIT Infra- structure City Government ITGovernance Metropolis Urban Infra- structure IT

32. Slide 32 Urban Infrastructure Urban Density  Urban Infrastructure –Water, Sewer, Gas, Electricity, Broadband… Requires Metropolitan Support Requires Local Hookups Retrofit Happens –Notre Dame has Toilets and Electricity –Conduits for Future Funding –Sometimes Private –Sometimes Public

33. Slide 33 IT Infrastructure Many Apps & Servers  IT Infrastructure –Easier with Single Mainframe Federated Infrastructure: –Identity, Security, Naming and Directory, etc. Requires IT Services Requires App Hookup Retrofit Happens –Web Services Offer Hope to Ease Retrofit Funding Competes with Apps

34. Slide 34 Urban And IT Infrastructure Infrastructure for Crowded Environments –Needs Supporting Services –Needs Hookup to Buildings or Apps Retrofit Happens –May Be Biggest Cost –Design for Future Extensions Funding Competition

35. Slide 35 City Government ITGovernance Urban Infra- structure IT CityGovernmentITGovernance Metropolis

36. Slide 36 City Government Decisions for Allocating Resources –City and Business Leaders –Infrastructure Usually Needs Cooperation Factories/Buildings Usually Business Driven –Cities Usually Constrain and Control Manufactured Goods Controlled by Business –Cities Have Little Say…

37. Slide 37 IT Governance Who Makes Decisions? –CEO? CIO? Business Unit Execs? Techies? Committees? What Are Priorities? –Asset Utilization? Cost? Flexibility? Growth? Metrics –What is Success? Who Is Accountable? What Are Our Goals? –Reduce Cost? –Better Information? –Competitive Advantage? “Don’t Just Lead, Govern! Governing IT for Different Performance Goal” Professor Peter Weill, CISR, MIT Sloan School of Managent; June 2003

38. Slide 38 City Government and IT Governance Similar Decisions Problems for Cities and IT-Shops –Cities Provide Inspiration Cities (Usually) Optimize for Growth –Businesses Drive Building/Factory Investment See: “Don’t Just Lead, Govern! Governing IT for Different Performance Goal”, IBID

39. Slide 39 CityGovernmentITGovernance Metropolis

40. Slide 40 Looking To The Future Equivalent to 1880 or so… –Communication and Browsing Well Established –Virtual Enterprises Getting Going –Business Process a Gleam in Our Eye Lots of Fun Ahead of Us –Biz-Process Won’t Take as Long as Retail Did to Mature18251850187519001925195019752000 Single Factory Interchange- ability RailroadGrows Manufactured Assemblies Department Stores Begin Wal-Mart Asserts Itself Shopping Excursions You Are Here!

41. Slide 41 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

42. Slide 42 Service PolicyPolicy Schema and Contract Schema and Contract Service-Oriented Architecture Service-Orientation –Independent Services Chunks of Code and Data –Interconnected via Messaging Four Basic Tenets: –Boundaries Are Explicit –Services Are Autonomous –Services Share Schema and Contract Not Implementation –Service Compatibility Is Based on Policy

43. Slide 43 The Mail-Order Business A Mail-Order Business is Asynchronous –Work Requests Arrive in Bags of Mail –Product Arrives in Shipments Each Message (Order) Is a Transaction –Goods Are Prepared and Packed –Payment Is Processed –Stuff is Shipped Standards and Interchangeability Required –Both Goods and Forms Mail-Order Is a Service- Oriented Architecture! –Autonomous –Explicit Boundaries –Interaction via Forms –Explicit Policies for Work

44. Slide 44 A Look at Today’s Apps People Walk in the Front Door –Perhaps Buy Standard Stuff –Frequently Do Customized Operations Work Is Usually Done “While You Wait” –This Is Synchronous and Specialized Processing –How Manufacturing Was Done Before Interchangeability! Difficult to Do Mass Production –Standards Rare –Efficiencies Are Difficult Not Service-Oriented! –Silos, not Services Rarely Interact Together –Muddled Resource Mgmt and Business Process

45. Slide 45 A Visit from the Salesperson Sales-People Arrive with Catalogs and Forms –Catalogs Are Needed to Fill Out Forms –Filling Out Forms Doesn’t Need Factory Access Sales-People Make It Easy –They May Represent Multiple Factories –They May Help Deliver and Get Started Using Stuff Smart Clients Are Like Sales-People –They Make It Easy –They Work Offline –They Accept Orders and Provide Results –Perhaps Multiple Apps

46. Slide 46 Watching a Business Grow Small Businesses Are Intimate –Everybody Knows What’s Going On –Process and Data Are Shared By Everyone Big Businesses Formalize Process –Cross-Department Work Requires Forms –Everything Gets Documented and Recorded Composite Services –One Company’s Work –Implemented As Separate Departments –Formal Interfaces –Separate Records –Asynchronous Processing –Looks Like One Company

47. Slide 47 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

48. Slide 48 Three Reasons for Using Services Autonomy of Applications –Independent Development and Evolution –Clarifies Encapsulation and Privacy of Data Separation of Biz-Process and App –Allow Separate Development of Biz-Process –Apps Gravitate to Managing Resources Enabling IT Infrastructure –Surround the App (Service) in Predictable Way –Connect the App (Service) into IT Infrastructure These Changes Empower New Business Models!

49. Slide 49 Three Categories of Service Use Building New Solutions With Services –Independent Pieces Within Apps –Independent Development and Maintenance –Scale-Out Cleaving Together Existing Apps –Connectivity B2B: Business-to-Business EAI: Enterprise Application Integration –Business Process –Tap Into IT Infrastructure Cleaving Apart Existing Apps –Disentangling the “Big Ball of Mud” See “Big Ball of Mud” by Brian Foote and Joseph Yoder Department of Computer Science University of Illinois, Urbana-Champaign

50. Slide 50 Service Contract Schema and Contracts –Expose the Message Formats and Sequences –The “Black-Box” Behavior of the Service Reject-Order Accept-Order One Of Tentative Place-Order Confirm Place-Order One Of Cancel Place-Order

51. Slide 51 Schema, Contracts, and Standardization Schema and Contracts Define Behavior –Messages and Sequences –Specify How to Interact with a Service De facto Standards Will Emerge –Both Horizontal and Vertical –Driven by Market Leader Standards Eliminate Semantic Chaos –Applications Adopt Semantics or Die –Industries Will Shake Out to Commonality –This Takes Years…

52. Slide 52 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

53. Slide 53 Messages Are Special The World Between Services is Special –Messages Are Sent and Float Around Between Messages Are a Different Kind of Data –Written Once and Never Changed –Independently Defined and Extensible Schemas XML and XML-Schema Are Important Innovations MSG MSG MSG MSG MSG MSG MSG MSG Service Service Service Service Service Service Service Service Service No Man’s Land!

54. Slide 54 XML Is Like Cardboard XML Surrounds and Protects Data and Ops –It Is Malleable and Self-Describing Cardboard Protects Stuff When It Is Shipped –It Is Very Malleable Many Different Shapes and Forms –It Is Self-Describing The Outside Is Labeled The Inside Has Instructions Cardboard Is One of the World’s Largest Industries –Almost Everything Gets Shipped in Cardboard –Started Around 1880… Sometimes the Cardboard Weighs More than the Stuff Being Protected… –But It’s Always More Valuable…

55. Slide 55 Messages Are Data! Enterprise-Class Services Record Messages –Work Happens by Changing the Database Stimulated by Messages (Recorded in DB) Changes Data Records Outgoing Responses Enterprise-Class Services Live in the Database! Transaction Service Logic Service-Private Data Business-Service

56. Slide 56 Cleaving Together Existing Apps App The Three Categories of Service Use Cleaving Apart Existing Apps App Building New Solutions With Services

57. Slide 57 Building New Solutions With Services A Service May Comprise Many Services –Perhaps on Many Machines –Essential for Scale-Out Looks Like a Single Service –Can’t Tell from the Outside Communication with the outside world Service

58. Slide 58 Business-Service Composite Services and Scale-Out Scale-Out Trick: –Load-Balanced Compute Engines –Partitioned Database Engines Multi-Message Conversation –Stored in Database –Find Right Partition Centralized Back-End Service –Manages Shared Resources Back-End Biz-Svc The Soul of the Biz-Service Is the SQL Database! The Soul of the Biz-Service Is the SQL Database! Biz Logic Biz-Svc Service-to Service Messaging DB A-J DB K-Q DB R-Z DB Logic

59. Slide 59 Goals For Wrapping an Existing App Most Apps Are Used by Humans –People Know How to Make Them Work –People Add Judgment to Fit the Circumstances Reduce the Need for People to Use the App –Increase the Times Machine-to-Machine Works OK –The Goal Is NOT to Eliminate the Need for People Wrap an App to Look Like a Service –Increase the Service-to-Service (App) Work

60. Slide 60 If you can tap into the App, you can pretend to be a human for some operations This can allow the App to look like a Service. Gaining Entrée to Existing Apps How Do People Get Into the App? –HTML, 3270s, 3-tier, 2-tier, EDI, MQ, etc… Surround and Interface to App –Screen Scraping, HTML Parsing, etc… Sometimes This Is Hard! –Client-Server Needs App Code Changes 2-Tier Especially Hard Service Services

61. Slide 61 Identifying Restartable Sequences of Operations Every App Deals with Failure –People Have Procedures –These Are Repeated Until Success Usually Ad-Hoc Find the Restartable Sequences –A Set of Operations Against the App –Each Op May Be Not Restartable –The Entire Sequence Is Restartable Deposit $1-Billion Crash! Restart Look to see if the deposit happened If not, deposit $1-Billion Shit! Restartable Sequence of Operations

62. Slide 62 Defining The Messages and Contracts Define Messages –Map to Business-Operations –Search for Restartability! Contracts Are Sequences of Messages –They Are Related and Perform Long-Running Work Service Restartable Sequence of Ops Look to see if the deposit happened If not, deposit $1-Billion

63. Slide 63 Inventory Service Cleaving Applications Together Surround the App with a Service –Decide the Functionality to Wrap Subset of Functionality Is OK –Create Messaging –Tap Into User Interface or Biz-Logic of App It is OK to partially surround the app. Some functionality is exposed through the service, some the old way… Application SQL Inventory Biz-Logic Inventory User-Interface Inventory

64. Slide 64 Application Accounting Service Inventory Service SQL Accounting Inventory Cleaving Applications Apart Cleaving Apps Apart Into Services –Disentangle the Data –Separate the Functionality –Add Messaging Between Hard to Disentangle and Separate –Find Natural Boundaries and Pray a Lot! Easier for Message Semantics –They Grew Up in the Same Neighborhood Biz-Logic Accounting Inventory Biz-Logic Accounting SQL Accounting User-Interface Accounting Inventory Biz-Logic Inventory SQL Inventory

65. Slide 65 Cleaving Together Existing Apps App The Three Categories of Service Use Cleaving Apart Existing Apps App Building New Solutions With Services

66. Slide 66 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

67. Slide 67 The New IT Shop Cities versus Towns –In Cities, Most People You Meet Are Strangers –Requires Mechanisms to Manage Trust of Strangers IT Shops Will Get More Connected –Connected to Apps (Services) in the Shop –Connected to Other Businesses Work Spreads Across Services –Some Services Surround Resources –Some Provide Biz-Process IT Infrastructure Hooks the Services Together IT Governance –Helped by Independence of Services See: “Passage to Union: How the Railroads Transformed American Life, 1829-1929” by Sarah H. Gordon, Elephant Paperbacks, 1997 See: “The Death and Life of Great American Cities” by Jane Jacobs, Vintage Books 1961

68. Slide 68 The New Application Remodeling Your Application: –Cleaving Apart Remodeling to Add Interior Partitions Allows for Efficiencies in Space Management Separate Spaces Sometimes Work Better –Cleaving Together Busting a Door Through a Wall Connection Provides Efficiencies Apps Will Comprise Services –Some Services Control Resources Analogous to Manufacturing –Others Will Focus on Biz-Process Analogous to Retail –Smart Clients Are Services Behaving Like Salespeople See: “How Buildings Learn What Happens After They’re Built?” by Stewart Brand, Penguin Books 1994 Apps Become Services Either by original design or remodeling…

69. Slide 69 Communication and Structured Data Sharing Structured Data –Immense Pain at First –Standards Will Emerge Vertical and Horizontal Semantic Differences Get Eliminated Over Time –Sometimes Killing Apps Due to Incompatibility Services Empower This –Difference Between Inside and Outside the Service

70. Slide 70 Business Process and Virtual Enterprises Business Process Will Grow Distinct and Strong –Today, Apps Are a Mixture of Resource Management and Biz-Process Soon, They Will Separate –Manufacturing and Retail Grew Apart Biz-Process Will Dominate –It Will Drive the Energy and Money –As Biz-Process Crosses Companies It Becomes “Virtual Enterprises” Corporate Boundaries Blur Tremendous Efficiencies and Economic Benefit

71. Slide 71 Empowering IT Infrastructure IT Infrastructure Is New and Growing –Cope with Many Different Machines… Connect IT-Wide Services with App Services –Coordinate with IT-Wide Services –Tap Into the App Services “Services” Technology Allows Connection! –Empowers Tapping In –Will Get Easier… Federated Infrastructure –Identity, Security, Mgmt, Monitoring, Naming, etc –Stuff We Haven’t Thought of Yet

72. Slide 72 Empowering IT Governance IT Governance Is Usually a Pendulum –Swing to Centralized Control to Rein in Costs Biz-Units Complain about Pace of App Delivery –Swing to Decentralized Control to Gain Function Chaotic Growth of Apps Becomes Overwhelming “Services” Empower a Balance –Can Decentralize While Constraining Integration –Must Tap into Infrastructure –Must Tap into Biz-Process –Resource Management Separate from Biz-Process

73. Slide 73 Outline Metropolis: The Analogy What the Heck Are Services? Why Use Services? Implementing Services for the Enterprise Metropolis As Guidance Conclusion

74. Slide 74 Implications Of Metropolis Heterogeneity Happens! Ongoing IT Investment –Infrastructure versus Business –Historic Monuments Standardization Is Nascent –Connection Largely by People –Efficiencies Still to Come Business Process Is Nascent –Still Mostly Ad-hoc –Growing to Become Dominant Force Loose Coupling Helps Investments

75. Slide 75 Envisioning The Service-Oriented Enterprise Services Move Us Forward –Lots of Islands of App Code –Services Are for Connecting the Islands! –Complements Object-Oriented Technology Independence Is Essential –Services Evolve Independently –Build versus Buy Inside Your Company and Across Companies –EAI: Enterprise Application Integration –B2B: Business to Business –It’s All About HST! Services Cleave Your Applications –They Cleave Them Apart Into Independent Pieces –They Cleave Them Together Allowing Interaction Service Schema and Contract Schema and Contract

76. Slide 76 And Now, a Word from Our Sponsor… There’s Some Cool Technology Coming! –Web Services Industry Agreed Upon Interoperability Standards –Indigo.NET Technologies for Building and Managing Service-Oriented Systems –BizTalk Message Transformation and Orchestration –Yukon SQL Service Broker Integrated Messaging into Databases Automatic Support for Conversational Messaging Transaction Protected Reliable Messaging Keep an Eye on This Technology!

77. Slide 77 Build Your Services Now! You Will Be Building and Connecting Services –The Economics Force: Pieces of Application (Services) Connected by Messaging You Can Build Service Now! –Look for “Low-Hanging Fruit” Find Where You Can Save Money for Small Investments Be Pragmatic –It’ll Be Easier with Longhorn but Don’t Wait! The Sooner Start, the More You Gain! –Guidance Available to Help: http://msdn.microsoft.com/architecture http://blogs.msdn.com/PatHelland -- My Blog Full of Miscellany

78. Slide 78 © 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.