1. Drawn from TAPI: oimt.2019.ND.017.00-TapiStreaming.mht
Streaming Summary
Nigel Davis
Drawn from TAPI: oimt.2019.ND TapiStreaming.mht

2. Application
RESTCONF client gets current state followed separately by fine-grain delta notifications
Streaming proposal alignment and change notification through connection to a single stream
Ensuring eventual consistency with reduced fidelity when under extreme stress
Suits an M2M application:
A small number of direct clients, ~2 (orchestrator etc.)
Clients scoped to remain connected long term
Clients maintain history and live view of state
Hence, the client systems do not query state over TAPI.
RESTCONF notification application appears to assume that the client will get current state followed by delta notifications that give only details that have changed in the context of some identity.
The streaming approach proposed here provides alignment and notification of change through connection to a single stream. This is proposed initially as an alternative. The scheme has the potential to be a replacement in the longer run.
This stream ensures eventual consistency and, in the full realization, offers reduced fidelity when under extreme stress. Under normal conditions the stream offers full fidelity. The measure of stress depends upon system scaling (to achieve a particular point on the scale of cost effective fidelity).
This form of streaming suits an application where the streaming system is feeding a small number of direct clients, ~2 (orchestrator etc.), that are scoped to remain connected long term and to maintain history along with a live view of state of the things in the network so as to do some necessary analysis so as to perform some solution relevant functions. Hence the client systems do not query state over TAPI.

3. Characteristics
Whole entity (i.e., instances of classes with UUIDs) snapshots on change
Connection topics on per group of entity classes basis
All instances of entities of that class within the context.
Client system works on Context (which can be adjusted) hence it is not necessary to have
Individual queries
Notification filters
The provider streams snapshots of whole entities (i.e., instances of classes with UUIDs) within the context. The streamed information is divided into Topics on a per entity class basis where a topic covers all instances of entities of that class within the context. It is assumed that the context is what the client system wants to see and hence individual queries on the server and notification filters that are not simply a realization of the view should not be necessary.

4. Characteristics Implications of full entity streaming
Entities has properties that change at a consistent rate in their lifecycle
Larger entities change less
Things in the context may change and the context may change (add/remove things)
Each event record includes time of change from ultimate source, i.e., time of occurrence of event
Stream pipeline
“Guarantees” delivery “at least once”
Provides backpressure to a reactive producer
As full entities are streamed it is important that entities have properties that change at a consistent rate in their lifecycle and that the larger the entity the lesser the rate of change.
The stream mechanism including the delivery pipeline guarantees delivery “at least once” and provides backpressure to a reactive producer. The client can choose to consume from any point to ensure guaranteed delivery.

5. Characteristics Suitable realization is a compacted log
Fundamental to its operation is that it keeps the latest snapshot of events
An optimum realization should be
Highly scalable and available
Highly reliable (distributed, partitioned, replicated, and fault tolerant)
Low latency and high throughput on big data scale
Candidate stream settings to suit normal system behaviour:
Compaction delay = 10 minutes (the client can afford to be a little behind)
Tombstone retention = 4 hours (comms issues should be fixed within 4 hours)
Non-tombstone retention = “forever”
Optimised stream settings achieve cost effective fidelity

6. Alarm behaviour
Future Design: Detector stream stabilized to indicate detector is intermittently active then clear
Stream detector identifier derived from detector native identifier
Alarm normally reported against valid TAPI entity
Model allows alarms without full/any TAPI resource id (rare)
Alarm reported against containing TAPI entity where Detector related to thing not fully modelled in TAPI, e.g., a power supply function
Alarm clear followed immediately by tombstone record.
Detected condition:
Minimally: In native form, i.e., as provided by the NE, 
Additionally: In normalized form, i.e., complying with standard alarm type
Traditional alarm embellishments are inappropriate in future systems as they do not add relevantly to detected condition and location
However, legacy system, assume these as mandatory hence the structure supports them
The following should be populated if available:
Device level severity
Service affecting
Repair action etc.
Many of the traditional alarm embellishments are seen as inappropriate in a future system. They originated when there were no management/control systems, i.e., when the only viable alert system involved filament bulbs and when telecoms systems were relatively simple compared to those of today. At that stage the device was the only place for criticality analysis. In the current environment and certainly in the target environment the device cannot make any relevant contribution to the determination of the criticality of repair or the impact on revenue. The traditional device level severity, service affecting, repair action etc. do not add relevantly to the information conveyed by the detected condition and its location. Metadata and system structure should provide relevant information to allow a higher system to develop necessary priority indications related to primary tasks of repair etc.

7. Use cases Connect to Stream and align - new client
Steady state reception - well engineered client
Event storm – bad day (or slow client) - several micro-bends with active percussive interference, overloaded client with reduce compute power available
Event storm – very bad day (or very slow client) - many micro-bends and reduce compute power etc.
Short loss of communications
Long loss of communications requiring realignment

8. In oimt.2019.ND.017.00-TapiStreaming.mht
Two solutions illustrated
Simple message sequence
Data model discussion
Messaging