What’s new in the SIF3 World? SIF 3.2.1 that is…

Overview of SIF 3.2.1 Infrastructure Why “3.2.1” Changes Since: An alternative to SIF Events Query By Example (QBE): An intuitive Query Mechanism When would I use it? Functional Services: What, Why, When … Components of a Functional Service Real World Examples Others HTTP HEAD Extended Mime Type Support (i.e. CSV) Other Query related changes © Access 4 Learning Community

Why SIF 3.2.1? SIF 3.2 was released in June 2016 UK started implementing first Functional Service. Discovered a couple of errors in specification (Events for Functional Services). Hit some undesired limitations (no multiple-object operations). => Need a fix to and expand on Functional Services SIF 3.2.1 is a Fix Release Addresses errors and limitations in relation to Functional Services. Provides more clarity around the operations available for Functional Services and related objects. => SIF 3.2.1 will “replace” SIF 3.2 © Access 4 Learning Community

Changes Since Alternative Mechanism to SIF Events. SIF Events are an asynchronous message pattern. SIF Events require a number of SIF Interfaces to be implemented. SIF Events almost certainly only available in a BROKERED Environment (i.e. middleware component required). Enables systems to stay “in-sync” when SIF Events are not available. DIRECT Environments where asynchronous message patterns are generally not available. Either Provider or Consumer doesn’t support SIF Events. © Access 4 Learning Community

Changes Since (cont.) Consumer can request data that has changed since it last asked for changed data. Can be considered as a “partial” sync! Must indicate to the provider when it has requested the last “changes since” using the “Opaque Marker”. Provider can tell a consumer if it supports “Changes Since”: Providing consumers with an opaque marker which the consumers must pass to provider for each “changes since” request. Provider must manage all opaque markers for all potential consumers. © Access 4 Learning Community

What is the “Opaque Marker”? Changes Since (cont.) What is the “Opaque Marker”? It is some sort of a “token” that is issued and managed by the provider to track which changes have been provided to a particular consumer. Format and meaning is known to and managed by the Provider. Opaque to Consumer! Consumers must pass this Opaque Marker to the provider for each “Changes Since” request. Can be… Timestamp at object level or in change log. Sequence within a change log. Version number (i.e. Daily Refresh Number). Offset in a queue (i.e. Kafka Queue). etc. © Access 4 Learning Community

Changes Since (cont.) Response Payload of “Changes Since” Request: Changes Since call must be done for each SIF Object separately. Payload in response to “Changes Since” call is a collection of a particular SIF Object (i.e. xStudents collection). Consumer must inspect response payload to determine Event Type: RefID of Object exists in Consumer Data Store => UPDATE EVENT RefID of Object does not exist in Consumer Data Store => CREATE EVENT RefID is ONLY element/attribute provided => DELETE EVENT © Access 4 Learning Community

Query By Example (QBE) It is a Query Mechanism! The concept is based on providing the “Query Executor”, the provider, an example or template of the data to be returned. Query Condition is given by a standard SIF Object (can be XML or JSON): <xStudent> <name> <type>LegalName</type> <familyName>Jones</familyName> </name> </xStudent> => Give me all students with legal family name of Jones. © Access 4 Learning Community

Query By Example (QBE) (cont.) Key Characteristics for QBE: Conditions are related to elements contained within a specific Object Type (i.e. xStudent) Example: Get me all female students with a name of XYZ. Each Element/Attribute provided in the payload equate to one query condition. Does not work across objects (i.e. associations) Example: Get me the attendances for student ABC at School 123. => “Service Path” cover this type of queries! Supports all standard features of a GET method: Paging Zone, Context © Access 4 Learning Community

Query By Example (QBE) (cont.) Payload and Query Condition Interpretation: Each element in the payload forms a specific query condition. The conjunction between the conditions is 'AND‘. Comparator for all types except strings is EQUAL. Strings have a LIKE with the only supported wildcard of '%': 'ABC': No wildcards in the value is equivalent to an 'EQUAL'. '%ABC': Wildcard at the start of the value means 'ENDS IN'. In this case anything that ENDS IN 'ABC'. 'ABC%': Wildcard at the end of the value means 'STARTS WITH'. In this case anything that STARTS WITH 'ABC'. '%ABC%': Wildcard at the start and end of the value means 'CONTAINS'. In this case anything that CONTAINS 'ABC'. 'AB%C': Wildcard anywhere in the value means 'STARTS WITH' and 'ENDS IN'. In this case anything that STARTS WITH 'AB' followed by any number of characters and ENDS IN 'C' © Access 4 Learning Community

Functional Services What is a Functional Service? “A Functional (or Job Object) Service encapsulates stateful process behaviour as well as the data exchanged between applications implementing that process.” Other terms associated with Functional Service => Why or When to use it? Workflow Transaction Long-Running Process Orchestrated Multi-Step Process © Access 4 Learning Community

Components of Functional Services “The Job” Encapsulates the high level information about the Service: When created Overall Status of Job Phases of the Job A Job is created when the Functional Service is kicked-off. => Performed by Consumer. © Access 4 Learning Community

Components of Functional Services “The Job” (cont.) A Job is removed once the functional service concludes or is cancelled. Can be performed by: Provider that informs consumer(s) about the deletion with standard SIF Delete Event. Consumer with Delete Request on Job. Status updates on the Job are done by: Provider with standard SIF Update Event. Consumer with Update Requests on “Phases”. © Access 4 Learning Community

Components of Functional Services “Phases” A Job is made up of several Phases (Steps). Phases have an order and can be optional in some cases: Invoke phase A first. Optionally followed by phase B etc… Phases can carry data (i.e. Student, Class). ACL associated that defines what can be done with the data that is associated with a Phase. Phases have states (i.e. in-progress, on-hold). Updated by Consumer or Provider © Access 4 Learning Community

Functional Service Examples Real World Examples of Functional Services: Australia NSW: Timetable Publishing – In progress (design phase) End-of-Year Rollover - Potential Student Transfer: Use Case investigated a few years ago. Student Enrolment - Potential North America Student Record Exchange (SRE) – In progress (partly implemented in WA) Student Locator – On hold: May require Request Forwarding. United Kingdom DfE Data Exchange – In progress (i.e. being implemented as we speak…) © Access 4 Learning Community

Other: NEW - HTTP HEAD Get Information/Metadata about a Service. Implemented as “HTTP HEAD”: The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. … This method can be used for obtaining metainformation about the request … It returns all the HTTP Headers the equivalent HTTP GET would return. Specific use in SIF3: Changes Since! Get Initial “Opaque Marker”. Other usages: Get the number of records a query may return. Check of enabled features. © Access 4 Learning Community

Other: Extended Mime Type Support Up to SIF 3.1 any Service allowed: XML & Limited JSON as payloads (mime type) Had need to transport CSV files: Infrastructure independent of Data Model Why not enable any mime type in the Data Model Object Services? SIF 3.2 allows any mime type to be transported for all Data Model Object Services but not for Infrastructure Services! Only XML & JSON allowed for Infrastructure Services. I.e. There is no “Get Environment” data in CSV. © Access 4 Learning Community

Other: Query related Changes Static xQuery with parameters as URL Query Parameters: =>…/ StudentsBySectionId?sectionId=acb66d4b-0140-1000-0006-14109fdcaf83 Provider runs xQuery called StudentsBySectionId with parameter of sectionId. All Query Mechanisms allow for “Order By” Clause as an URL Query Parameter: =>…?order=[primarysortfield=direction;secondarysortfield=direction...] =>…/students?order=[name/nameOfRecord/familyName=ascending] Note: “sort-fields” are given as xPath! (Sorry JSON). © Access 4 Learning Community

SIF 3.3 Infrastructure Functionality Functionality that may make up all or parts of SIF 3.3: JSON Support for Events (long awaited) – very likely Request Forwarding – Broker to Broker request forwarding – not sure yet. Might be a need in the US for some Functional Services such as SRE (Student Record Exchange). “Pull” Providers - waiting Might be a need in the UK – wait for them to provide use case and documentation. Only for Brokered Environments: Broker puts requests on queue to be picked up by providers. Others? © Access 4 Learning Community

Resources & Supporting Tools A4L Website http://www.a4l.org Github SIF3 Spec, XSDs http://specification.sifassociation.org/Implementation/Infrastructure/3.2.1/contents.xhtml SIF3 Framework Java: https://github.com/Access4Learning/sif3-framework-java .Net: https://github.com/Access4Learning/sif3-framework-dotnet SIF Infrastructure Group (see A4L website) Your Locale Tech Board (AU, NA, UK) © Access 4 Learning Community

Questions? ? © Access 4 Learning Community