Andrea D’Orazio 28 April 2010 ©

 Synchronization models what, when, how versioning sync process sync architectures type-specific data algorithms & tools  Sync in middleware  Case studies SyncML Funambol

 data has to be modified? NO: just a copy YES: propagation of updated data to all devices what if data is modified on two devices?  always ensure that no data is lost!

 WHEN to sync? manually automatically  keep in mind network / connection availability!  synchronization ~ replication store data in a number of locations fault tolerance, but... most copies unavailable most of the time

 HOW to sync? permit only one modifiable copy of data  lock on data  hub-and-spoke model multiple copies can be independently modified  more flexible  more complex to implement the sync process

 each state of data is assigned a version number that increases as modifications are made  2 basic rules:( α, β locations) if A α 3 originates from a modification(s) of A α 1, A α 2 has a version number greater than A α 1 if A β 4 and A α 6 originate from A α 1, version is not comparable (both A β 4 and A α 6 have been modified in a different location)

location α location β A1 m=0 A1 m=0 A5 m=1 A1 m=0 SYNC modify A5 m=0 A5 m=0 propagation m modified flag

location α location β A1 m=0 A1 m=0 A5 m=1 A3 m=1 SYNC modify propagation m modified flag version not comparable how to reconcile? modify

 update detection recognition that a data has been modified  update propagation transmission of changes among all the data replica  reconciliation combination of all the updated data to build a synchronized version

 update detection recognition that a data has been modified triggers the start of sync process clean / dirty status (modification flag) modification timestamp hash of the content

 update detection modification timestamp  store only the timestamp of last modification? (e.g. in file system)  comparison of all the timestamps  time / resource consuming!  monitoring file system for changes  use dir timestamp if equal to last modified contained file semantics of modification time  when a file timestamp is actually modified?  content modifying, file renaming, file moving?

 update detection modification / creation / access timestamps  (e.g. with NTFS) file size improved detection timestamp(s) + hash  effective and safe way to compare data  only the hash is transmitted for comparison  bandwidth saving  can be slow if cpu / storage performance is poor

 reconciliation combination of all the updated data to build a synchronized version  opaque data (e.g. binary, pictures)  ask the user which version to use  structured data (e.g. XML)  edit logs  state comparison  both (can) use the latest common ancestor as comparison aid

 reconciliation type of modifications (inside a single file):  insertion  deletion  moving  changing  moving often as deletion + insertion  use of unique ID per piece of structured content

 concepts single user owns multi-located data device where data is located  component connections between (two) devices  synchronization connections

 centralized model each device synchronizes with the master device bi-directional communication! not all links might be established at the same time! A master B C D E

 tree model each piece of data is branched from another one each device synchronizes only with its parents child embeds parent’s version A master B C D E D

 general model it’s a mess! composite version number, each part for one device how to locate common ancestor? how many older versions stored? A B C F E D

reconciliation capabilities development effort opaque structure-known unique identifier semantics-known application specific

 opaque Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. change

 structured Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. change

 structured + id Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. move

 rsync  Remote Differential Compression (RDC) Microsoft Windows Server 2003 R2 / 2008 allows data to be synchronized between two or more computers, using compression techniques to minimize the amount of data sent across the network.

 Remote Differential Compression (RDC) no need of storing previous versions no assumptions regarding similarity or common origin of data independent from transfer protocol (HTTP,...)

 Remote Differential Compression (RDC) files (to be synced) divided into chunks of data  chunk bounded using an incremental fingerprint function MD4 hash calculated for each chunk comparison of MD4 lists (signature, one per file) transfer only of missing / different chunks can be applied recursively!  original file size 9GB  signature 81MB  signature of signature 6MB

 most of the sync (propagation) procedure can be generic put it in a mw platform! modules / hooks applications dependent  update propagation typically relies on the network control channel  messaging system data channel  customized (over HTTP?)

 publish / subscribe for update propagation - data channel  each edit gets immediately published as a event  continuous reconciliation  multiple users for update detection  an update is advertised to all subscribers

 Synchronization Markup Language  now integrated in: Open Mobile Alliance (OMA) Data Synchronization and Device Management  interoperable protocol to sync data

update propagation  transfer of updates among devices client-server architecture based on “edit log” model  addition, delete, replace of objects  unique ID data type & data store independent notification (only!) of conflicts

“Many people want quick and easy access to their , contacts, calendars, tasks and notes, regardless of where that information is stored.” started in 2001 open source Java implementation of the SyncML (OMA DS) standard sync data with  billions of phones  thousands of applications and online services (Gmail, Yahoo!, AOL, Hotmail, Outlook,...)

“Funambol is the leading provider of mobile cloud sync. Its mobile open source platform can be used for many types of mobile applications, including push , PIM data synchronization and device management. It provides C++ and Java client APIs and server side Java APIs. It facilitates the development, deployment and management of a wide range of mobile projects.”

 Synchronization models what  data modified? when  auto / manually how  one / multiple copies modified versioning sync process  update detection / propagation, reconciliation

 Synchronization models sync architectures  centralized, tree, general model type-specific data  opaque, structured, unique ID, semantic, app. specific algorithms & tools  rsync, Remote Differential Compression

 Sync in middleware  update propagation  Case studies SyncML  protocol for transfer of updates among devices Funambol  “cloud implementation” of SyncML

Sasu Tarkoma, Mobile Middleware, chap. 8. Wiley. Remote Differential Compression SyncML Funambol