Presentation is loading. Please wait.

Presentation is loading. Please wait.

Thial: A Client Notication Service for Internet-Scale Applications Atul Adya, Gregory Cooper, Daniel Myers, Michael Piatek Google Seattle 1.

Similar presentations


Presentation on theme: "Thial: A Client Notication Service for Internet-Scale Applications Atul Adya, Gregory Cooper, Daniel Myers, Michael Piatek Google Seattle 1."— Presentation transcript:

1 Thial: A Client Notication Service for Internet-Scale Applications Atul Adya, Gregory Cooper, Daniel Myers, Michael Piatek Google Seattle 1

2 A Case for Notifications Problem: Ensuring cached data is fresh across users and devices 2

3 Common Application Patterns Clients poll to detect changes – Simple and reliable, but slow and inefficient Push updates to the client – Fast but complex – Add backup polling to get reliability – Tail latencies can be high: masks bugs – Application-specific protocol sacrifice reliability 3

4 Our Solution: Thialfi Scalable: tracks millions of clients and objects Fast: notifies clients in less than a second Reliable: even when entire data centers fail Easy to use: deployed in Chrome Sync, Contacts, Google Plus 4

5 Talk Outline Thialfis abstraction: reliable signaling Delivering notifications in the common case Detecting and recovering from failures Evaluation and experience 5

6 Thialfi Overview Thialfi client library Register XNotify X Client Data center X: C1, C2 Client C1 Client C2 Thialfi Service Update X Register Update X Application backend Notify X 6

7 Thialfi Abstraction Objects have unique IDs and version numbers, monotonically increasing on every update Delivery guarantee – Registered clients learn latest version number – Reliable signal only: cached object ID X at version Y 7

8 Why Signal, Not Data? Developers want reliable, in-order data delivery Adds complexity to Thialfi and application, e.g., – Hard state, arbitrary buffering – Offline applications flooded with data on wakeup For most applications, reliable signal is enough – Invoke polling path on signal: simplifies integration 8

9 API Without Failure Recovery Thialfi Service Publish(objectId, version) Client Library Register(objectId) Unregister(objectId) Notify(objectId, version) 9

10 Talk Outline Thialfis abstraction: reliable signaling Delivering notifications in the common case Detecting and recovering from failures Evaluation and experience 10

11 Architecture Client Bigtable Client Bigtable Matcher: Object ID registered clients, version Registrar: Client ID registered objects, notifications Client Registrar Matcher Object Bigtable Object Bigtable Data center Notifications Application Backend Registrations, notifications, acknowledgments Client library 11

12 C1: x, v7 C2: x, v7 C1: x, v5 C2: x, x: v5; C1, C2 x: v7; C1, C2 x Life of a Notification Client Bigtable Client Bigtable C1: x, v7 C2: x, v7 Notify: x, v7 Client C2 Matcher Object Bigtable Object Bigtable Data center Publish(x, v7) x, v7 Ack: x, v7 12 Registrar

13 Talk Outline Thialfis abstraction: reliable signaling Delivering notifications in the common case Detecting and recovering from failures Evaluation and experience 13

14 Data center loss Server state loss/ schema migration Partial storage unavailability Possible Failures Client Library Client Bigtable Client Bigtable Registrar Matcher Object Bigtable Object Bigtable Client Bigtable Client Bigtable Registrar Matcher Object Bigtable Object Bigtable... Data center 1 Data center n Thialfi Service Client Store Client Store Client restart Client state loss Publish Feed Network failures 14

15 Failures Addressed by Thialfi Client restart Client state loss Network failures Partial storage unavailability Server state loss / schema migration Publish feed loss Data center outage 15

16 Main Principle: No Hard State Thialfi remains correct even if all state is lost – All registrations – All object versions Detect and reconstruct after failures using: – ReissueRegistrations() client event – Registration Sync Protocol – NotifyUnknown() client event 16

17 Recovering Client Registrations 17 Registrar Matcher Object Bigtable Object Bigtable x y x y ReissueRegistrations() Register(x); Register(y) ReissueRegistrations : Not a burden for applications – Application stores objects in its cache, or – Object list is implicit, e.g., bookmarks for user X

18 Registrar Matcher Object Bigtable Object Bigtable Register: x, y Syncing Client Registrations x y Hash(x, y) x y Goal: Keep client-registrar registration state in sync Every message contains hash of registered objects Registrar initiates protocol when detects out-of-sync Allows simpler reasoning of registration state Reg sync 18 Hash(x, y)

19 Recovering From Lost Versions Versions may be lost, e.g. schema migration Refreshing from backend requires tight coupling Inform client with NotifyUnknown(objectId) – Client must refresh, regardless of its current state 19

20 Talk Outline Thialfis abstraction: reliable signaling Delivering notifications in the common case Detecting and recovering from failures Evaluation and experience 20

21 Notification Latency Breakdown Batching accounts for significant fraction of latency 21

22 Thialfi Usage by Applications 22 ApplicationLanguageNetwork Channel Client Lines of Code (Semi-colons) Chrome SyncC++XMPP535 ContactsJavaScriptHanging GET40 Google+JavaScriptHanging GET80 Android ApplicationJavaC2DM + Standard GET 300 Google BlackBerryJavaRPC340

23 Some Lessons Learned Add complexity at the server, not the client – Deploy at server: minutes. Upgrade clients: years+ Asynchronous events, not callbacks – Spontaneous events occur: need to handle them Initial applications have few objects per client – Earlier use of polling forces such a model 23

24 Thialfi Summary Fast, scalable notification service Reliable even when data centers fail Two key ideas simplify failure handling – Deliver a reliable signal, not data – No hard state: reconstruct after failure Deployed in Chrome Sync, Contacts, Google+ 24


Download ppt "Thial: A Client Notication Service for Internet-Scale Applications Atul Adya, Gregory Cooper, Daniel Myers, Michael Piatek Google Seattle 1."

Similar presentations


Ads by Google