1. Architecting Multi- Zone SIF Solutions Utilizing the new infrastructure options with existing Zone design patterns Ron Kleinman SIF 2011 Annual Meeting

2. Objectives Establish a common terminology to define the components of a zone and how they interoperate Understand the techniques available to scale up a Zone Apply those techniques to address a set of typical problems

3. Where to start?

4. Client Application Object Provider Application The Communicating Parties: Client, “Service” and … the Wire DB Request Response

5. The Point to Point Wire (1:1) How did the client discover the Object Provider? – Where is it? / What it can do? / Correct Version? What is the agreed upon Transport protocol? – HTTP? / HTTPS? / SOAP? / REST? How is Security Imposed? – Encryption / Authorization / Authentication What is the Quality of Service? – Best Effort / Once only Delivery / Guaranteed Delivery – … or Synchronous or Asynchronous Response? – “Push” or “Pull”?

6. Message Broker: The 3 rd Party on the wire Object Provider Application Client Application DB Request Response Persistent Buffers

7. The Message Broker Wire (N:1) How did the client discover the Object Provider? – Content based routing (request routed to provider) What is the supported Transport protocol? – HTTP/S or SOAP 1.1 (with SIF v2.5) How is Security Imposed? – Transport Layer Security (TLS) with X.509 Certificates – Enforced ACL Authentication Rules (Ex: Can this client application request data from this type of object?) What is the Quality of Service? – Guaranteed (Buffered) Message Delivery Synchronous or Asynchronous Response? – Synchronous ACK / Asynchronous Response – Push AND Pull Mode

8. Pub / Sub Message Exchange Paradigm Object Provider Application Client Application DB 3A. Publish 3B. Event Persistent Buffers Client Application 3B. Event 2. Subscribe 1. Register

9. Message Broker support for the Publish / Subscribe Wire (1:N) Sequence of Operations – Publisher Registers as “Object Type” Event Provider – Clients Subscribe for Events on that Object Type – Provider Publishes a Object Type Event to MB (CUD) – MB Routes Event to all Subscribers Advantages – Publisher never needs to know of subscribers (0, 100) – All Subscriber messages routed correctly guaranteed When Offline Subscribers return, events delivered

10. A few small additions still needed

11. The Message Broker alone does not solve everything Excess “Coupling” problem only deferred – Limited coupling between Applications Major coupling between MB and Application  – Common Transport – Shared Security Conventions – Topic List (Object Types that can be CRUD) – Application must “Register” for Events / As a Provider – Version specific conversions Need “Bridge” Architectural Component – A “mini-broker” between MB and Application

12. Enter the Agent Client Application Object Provider Application Message Broker DB Object Provider Application DB

13. Agent Provides whatever Application needs Wraps: – MB Transport Protocol details (HTTPS or SOAP) – Asynch responses behind Synch Method Calls Automatic Initial Application Registration to MB Automatic Event Subscriptions on MB “Topics” Application independent of Message Broker Application doesn’t know Message Broker is there Transparent Database Updates from Events – Application may not even know Agent is there!

14. “There ain’t no such thing as a free lunch” - Anon Applications are fully independent of wire! – But … Agent and Message Broker remain closely coupled Add another level of indirection? – Somewhere, somehow, two components will be coupled The Aha Moment! – “Interoperability is much easier if it’s you on both ends of the wire” - EduGuy Standardize the internal MB to Agent connection – Applications focus only on Data Object issues

15. What is the School Interoperability Framework (SIF) Standard? The SIF Object Hierarchy A set of XML Schemas defining the format of data transmitted between K-12 Applications – Student, Gradebook, Library, Transportation, … The SIF Zone The “on-the-wire” Infrastructure that supports such transfers. – Agents and the Zone Integration Server (ZIS)

16. The SIF Zone: It’s all about exchange of information

17. Copyright © SIF Association SIF Zone: The Formal Definition Publish and SubscribeRequest and Reply Message ChannelMessage BusGuaranteed Delivery Message Application AApplication B Message Endpoint SIF Agent SIF Zone Hohpe, G and Woolf, B. (2004) Enterprise integration patterns: Designing, building and deploying messaging solutions

18. What a SIF Zone is: A dedicated “Message Broker” centric intranet for message exchanged between two or more Agents. – Each Agent “fronts” for a different K-12 Application – The Zone Message Broker is the ZIS – All Agent Messages are sent to the ZIS and routed to their appropriate Destination(s) A collection of certified interoperable components – Any SIF-Certified ZIS on any platform is compatible with any SIF-Certified Agent on any platform. – Certification is for interoperability …not “correctness”

19. The SIF Zone: An informal view Agent Client Application Object Provider Application Zone Integration Server SIF DB Agent Client Application DB SIF

20. What a SIF Zone ain’t: A Centralized Data “Union” Zone supports an Object Hierarchy Object “Read Data” Requests  predetermined “Providers” Objects may contain references to other objects supplied by other providers (additional Requests needed) Each Application keeps its own data – Provider may “ignore” CUD Object Events from others A SIF Zone is a Distributed Data Confederacy!

21. The ZIS lies at the Heart of the Zone Guaranteed Message Delivery Content Based Request Routing Publish and Subscribe Application Registry and auto-Provider Discovery Security: Encryption, Authentication, Authorization ZIS

22. Simplified Connection Hierarchy Agents handle the SIF communication for the applications. Zones are connection points for agents. ZIS hosts one or more Isolated Zones ZIS Zone Agent Application Agent Application Zone Agent Application Agent Application

23. Two alternative Multi-Zone Architectures

24. Background: Establishing Zone Scope No standardized Zone to Zone message exchanges ZIS to ZIS conversations undefined No standard way to support them Only one Registered Provider for Any Object Type – ZIS needs to know where to route object requests – One authoritative source for any data element (MOM) – What if one size Zone doesn’t fit all cases?

25. “Multiple Zone” Use Cases School with “special needs” children has two SIS – Cafeteria, Library, Transportation Systems shared – Are two separate Zones needed? 4 School Districts share one Transportation System – Each school has a separate SIS, Library, Cafeteria System – Are 4 separate copies of the Transportation System needed?

26. 2 “Multiple Zone” Solution Techniques Define multiple Zones within the same ZIS – (Most ZIS vendors support this) – Transparent to all agents and applications Have Agent Register Application in multiple Zones – (Many Agent vendors support this) – Subscribe to same Object Type Events in all Zones – Publish own Events in all Zones – Respond to Object Data Requests from any Zone – ** Must determine which Zone to make own requests

27. Multiple Zone Design Pattern #1: Single Agent / App Spans “n” Zones School with “special needs” children has two SIS – Split school into two Zones within same ZIS – Agent Registers school-wide application in both Zones Subscribe to SIS Events from each School Zone Transparently combine SIS events for Application 4 School Districts share one Transportation System – Transportation Agent Registers in each School Zone Subscribe to SIS Events from each School Zone Respond to Data Requests from each School Zone Publish Events to each School Zone

28. #1: “Global” Agent Span Smaller Zones Agent Transportation SIS School Zone SIS Agent School Zone

29. Multiple Zone Design Pattern #1 Ex: “Strong” SEA access to LEA data Student data xfers directly from Districts to State – State Agent registers in all District Zones – Subscribes to SIS Events in each District Zone – Requests latest student data direct from owning District – Full SEA / LEA student data sharing State level Student Agent – Data updates are dynamic (as they happen at District) – Student Data formatted and provided to State LDS. – Can be written as a Web Service (SEA is “SOA”)

30. #1: “Global” SEA Agent Spans Smaller District Zones State SIS Agent District SIS Agent Student Info Gatherer Application District SIS District Y Zone District SIS Agent District X Zone State Student Database SQL & DB Updates School Y Zone SOAP / WSDL

31. Multiple Zone Design Pattern #2: Agents Register in Larger Zone School with “special needs” children has two SIS – Create a 3 rd school Zone spanning both, within same ZIS – SIS Agents Register in local and global school Zone Publish SIS Events to both School Zones Subscribers receive combined Events / Requests are Directed 4 School Districts share one Transportation System – Create a District Wide Zone – Transportation Agent registers there – Agents of all Applications needing Transportation Info register in District Zone as well as their School Zone

32. #2: Local Agents Register in “Global” Zone Agent Transportation SIS School Zone SIS Agent District Zone School Zone

33. Multiple Zone Design Pattern #2: Ex: “Controlled” SEA access to LEA Data Vertical Reports instead of “raw” student data – Demographic summaries instead of Student ID info – State-requested reports – no dynamic data changes – State sees only “processed” LEA student data – No State Agent registered in District Zones Vertical Reporting Choreography (example) – State Agent posts “Report Manifest” Event in State Zone – District RG Agents in State Zone see it, start report gathering process in their Districts – After completion District Agent posts “Report Ready” – Data obtained via Request / Response to District Agent

34. #2: District Agents Register in State Zone State VR Collector Agent District VR Gen Agent District Zone B District VR Gen Agent State Zone District Zone A Student Info Gatherer Application State Student Database SQL & DB Updates SOAP / WSDL HTTP/S

35. Architecting Zone Solutions

36. Architecting Multi-Zone Solutions: Sequence of Operations 1.Identify the Applications 2.Identify the Zones 3.Attempt to Scope the Applications to the Zones (Repeat steps 2 and 3 as needed) 4.Determine the Agent Instances 5.Configure All ZIS 6.Connect the Agent Instances to the Zones. 7.Verify Data Flow All needed Events can be Subscribed to All needed Data Objects can be Requested

37. Step 1. Determine the Applications Student Information System Library System Transportation System Data Warehouse Network System State Id System

38. Step 2. Identify The Zones

39. Step 3. Attempt to Scope the Applications to the Zones Student Information System – Agent (multiple zone support) District data scope School data scope Library System – Agent 1 (single zone support) School data scope – Agent 2 (single zone support) School data scope Transportation System – Agent (single zone support) District data scope Data Warehouse – Agent (multiple zone support) School data scope Network System – Agent 1 (single zone support) School data scope – Agent 2 (single zone support) School data scope State Id System – Agent (single zone support) District data scope Back to Step 2 if needed 

40. Zone Allocation Questions Number of application and agent instances. – Can one ZIS support all needed Zones? Data scope requirements for Zones. – Which Object Types spans Multi-zone and which not? SIF Versions supported. – Will deployed Applications span SIF Releases? Major releases are not backward compatible SIF Provision information – Which application provides each object type? Where?

41. Step 4. Configure the Agent Instances (assign Agent IDs / provide Zone IDs)

42. Step 5: Configure All ZIS Create one or more Zones in each ZIS Configure each Zone with the Agent IDs Configure the ACL for each Agent / Application – What Object Types they can publish Events for – What Object Types Events they can Subscribe to – What Object Types they can Provide Data for – What Object Types they can Request Data for

43. Step 6: Connect Agent Instances to Zones

44. Step 7: Verify Cross Zone Data Flow

45. Advanced Techniques: Ensuring Data Privacy XML Filtering (Data Privacy) – Remove object elements for selected recipients – Ex: Protect Student Discipline & Health information Directed Requests – Request Object Data from non-provider – Ex: Alternative “Student Object Provider” limits Work Study app access to only 5 of 1000 students Directed Events – Only specified application sees event (not subscribers) – Ex: Events on 5 students “reposted” direct to Work Study App which can’t receive general Student Events

46. Questions? EduGuy

47. Zone Functionality: Summary Quality of Service – Guaranteed once only Message Delivery – Content Based Routing – Publish / Subscribe support Security – Message Encryption – Application level Authorization – Element Level Security (proposed) Application Registry Auto-Discovery of Object Providers Zone Management (monitoring & control) Certification Program for Compliance