1. Event Stream Processing with Out-of-Order Data Arrival Mo Liu Database System Research Group Worcester Polytechnic Institute

17. Outline Introduction Problem with Out-of-Order Event Arrival Solutions Conclusion Related Work

18. Introduction: Event Stream Processing Raising interest on the database community Wild-range and growing applications Retail Management

19. Introduction: Algebraic Query Plan Sel (s.id = c.id AND c.id = e.id) Input Event Stream WinSeq (S, E, 10) Q: EVENT WSeq (S, !C, E) WHERE s.id = c.id AND c.id = e.id WITHIN 10 mins WinNeq (!C,10) (S.ts<C.ts<E.ts) S—Shelf reading C—Checkout counter reading E—Exit reading

20. Total Order Assumption in event arrivals  Order in which the events are received by the query system is the same as their timestamp order  By this assumption, “later arrival” means “larger timestamp” In the Case of Out-of-Order Event Arrival Unbounded operator Blocking operator Problems with Out-of-order arrivals

21. 012 SE * EVENTSEQ(S, !C, E) WITHIN10 mins s 3 e 6 sesce 3671015 Timestamp e 11 () s 3 (s 3 ) e 4 (s 7 ) e 11 S1 S2 Purge in WinSeq You see e15 then purge s3 and so on After that, OOO e4 comes  Unbounded operator! Output in WinNeg OOO c5 will cancel “ s3 e6”  Blocking operator! Problem with OOO: c 5 () s 7 (s 7 ) e 15 s 3 e 11 s 3 e 12 … … e 4 (s 3 ) e 6 s 3 e 4

22. Two Solution Frameworks Conservative Solution Exploits partial order guarantees (POGs) to produce permanent correct results. Aggressive Solution Outputs sequence results immediately and we design a compensation solution.

23. Conservative method … Metadata  Query Processor Operator state Network Purge with  Unblock with  data Metadata  Source1 SourceN

24. POG: Partial Order Guarantee POG Event type Pi time_stamp ts Example No More out-dated D events with timestamp smaller than 6 will come. d a d a Time Stamp

25. POG Functionalities in WinSeq Insert Sort semantics Compute Backward & Forward computation Purge Algorithm Singleton Purge using POGs e e P P e e e P

26. POG Functionalities in WinNeg Insert: Holding Set: store the spurious results Compute: POG P e1 e2...,ei, ej,...em (ej.ts < P.ts) WinNeg EVENT SEQ(E1,E2,..., Ei, !NE,Ej...Em) WITHIN10 mins

27. Aggressive solution Assumption: Most data comes in time and in order. Our goal is to send out results with small latency and send out compensation tuples when out-of-order data arrival occurs.

28. Compensation tuple Insertion message (+,Seq): (induced by an out-of-order positive event), where “Seq” is a new sequence. Deletion messages (-,Seq): (induced by an out-of-order negative event), such that “Seq” consists of the previously processed sequence.

29. e4e4 c5c5 WinSeq Functionalities WinNeg Compute EVENTSEQ(S, !C, E) WITHIN10 mins sesce 3671015 Timestamp c 11 c 5 … e 4 s3 e6 Results

30. Comparison ConservativeAggressive LatencyHighLow CorrectnessPermanentEventually Memory Consumption LowHigh

31. CONCLUSION Problems analysis Blocking operator Unbounded operator Two solution frameworks Pessimistic solution Optimistic solution Experiments are on-going…

32. Related Work Borealis system http://www.cs.brown.edu/research/borealis/public CEDR system http://research.microsoft.com/db/CEDR TuckerMSF03 UJ04 …

33. Happy Summer!