1. IBM Research / Linux Technology Center OpenLDAP Developer’s Conference – Vienna – 18 July 2003 © 2003 IBM Corporation Presentation subtitle: 20pt Arial Regular, teal R045 | G182 | B179 Recommended maximum length: 2 lines Confidentiality/date line: 13pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Disclaimer information may also be appear in this area. Place flush left, aligned at bottom, 8-10pt Arial Regular, white IBM logo must not be moved, added to, or altered in any way. Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements Presentation title: 28pt Arial Regular, black Recommended maximum length: 2 lines Group name: 17pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations For client presentations, client’s logo may go in this area Sync-based Replication : Protocol and OpenLDAP Implementation Jong Hyuk ChoiKurt Zeilenga IBM ResearchOpenLDAP Project Title slide

2. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 2 Agenda  Directory Replication / Synchronization -Replication -Synchronization -OpenLDAP Slurpd  LDAP Content Synchronization Protocol -Why not LCUP ? -Basic Protocol Description -Optimized Protocol for Traffic Reduction  SyncRepl : A New Replication Engine -Sync-based Replication Engine Design -Client-based Replication Engine Design  Target Applications  Summary Agenda slide

3. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 3 Directory Replication  Replication for High Availability, Performance, Security, Locality …  When the directory is updated in a replicated setup, replicas need to be synchronized to each other to provide a single directory image  Partition : unit of replication Replica : copy of a partition  Master-slave vs. Multi-master -Distributed directories via referral : referral chasing or chaining -Separate masters for different roles  Partial vs. Whole replication  Replication topology Basic text slide enterprise.company.com ou=users cn=foo ou=clients ou=assets nos.company.com ou=users ou=computers ou=servers ou=computers

4. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 4 Directory Synchronization  Keeping multiple directories up-to-date with each other  Stateful vs. Stateless -Stateful : synchronization action is based on replica status -Stateless : provider assumes the replica status and synchronize accordingly  State-based vs. History-based -State-based : synchronization action is determined based on the current replica status -History-based : history lookup is required for synchronization  Incremental vs. Full Reload -Incremental : only changes made after last sync be transmitted -Full Reload : requires full reloads per every (or most) sync  Push vs. Pull : provider-initiated or consumer-initiated  Polling vs. Listening : periodic sync or event-driven sync  Unit of synchronization : Entry-level vs. Attribute-level Basic text slide

5. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 5 Slurpd  Slurpd : standalone LDAP Update Replication Daemon -Master-slave -Multi-master for one level replication only without predefined URP (Update Reconciliation Protocol) -Stateless, History (replog) based, Push, Incremental synchronization  Example 1.Initial replication (db copy or ldif load) with master read-only 2.Promote master to read-write 3.Incremental synchronization Basic text slide slapd replog slurpd slapd partition replica

6. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 6 LDAP Content Synchronization Protocol  Stateful : cookie represents current replica status  State-based : does not mandate history store  Incremental : only changes after last sync are to be transmitted  Pull : clients initiate synchronization sessions  Polling & Listening : refreshOnly & refreshAndPersist  Partial replication : supports arbitrary search specification  Eventual consistency  Unit of synchronization : entry  Does not require predefined synchronization arrangement per-consumer information history Basic text slide

7. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 7 LDAP Sync : Example provider.company.com ou=users cn=foo com2 2003071809:00:00Z0000:00:0000 ou=assets com1 2003071712:00:00Z0000:00:0000 ou=computers consumer.company.com ou=users cn=foo com2 ou=assets com1 ou=computers Sync search base=assets (objectClass=computers) attrs=* refreshOnly sync control cookie= 2003071807:30:00Z0000:00:0000 Sync search entry cn=com2,ou=computers, dc=company,dc=com attributes, add state Sync Info Cookie= 2003071809:00:00Z0000:00:0000 Sync search result Sync done control Sync search entry cn=com1 present state com1 < cookie com2 > cookie

8. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 8 Why not LCUP ?  LCUP (LDAP Client Update Protocol) -Sends {updates + deletes} -Requires history information for reasonably efficient implementation -OpenLDAP doesn’t maintain history information (tombstone, changelog …) searchResultDone +syncDone(cookie) add modify/moddn/delete searchRequest +syncRequest(syncOnly) (cookie) searchResultEntry +syncUpdate(leftSet=0) searchResultEntry +syncUpdate(leftSet=1) modify/moddn/delete searchResultDone +syncDone ( cookie |reloadRequired ) searchRequest +syncRequest(syncAndPersist) ( cookie | reload ) leftSet=1 case of syncOnly requires history information such as log or tombstone to determine whether the entry was within the search result before the operation if consitency is too costly or impossible to achieve, producer issues reloadRequired searchResultEntry +syncUpdate (persistPhase, leftSet=0|1) add/modify/moddn/delete

9. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 9 Basic Protocol : Refresh Basic text slide Send Entries Initial Content Request Done with Cookie Send “Changed” Entries Content Refresh Request Done with “Cookie” Repeat Send “Present” Messages { update + present / delete } or “Delete” Messages

10. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 10 Basic Protocol : Refresh & Persist Send Entries Content Request Refresh Done Send “Changed” Entries and/or “Deleted” messages Cancel Done with Cookie / Acknowledge Cancel New “Cookie” Repeat

11. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 11 Protocol Optimization (Present Phase + Delete Phase) Repeat Send Entries Initial Content Request Done with Cookie Send “Changed” Entries Content Refresh Request Done with “Replica Cookie” Send “Present” Messages { update + present + delete } Send “Delete” Messages  Delete mode : requires full reload if replica state is out of history  Present mode : requires present entry transmission even if replica is within history  Present + Delete : sends deletes for the scope-outs covered by the history store sends presents for those not covered by the history Send “History Cookie”

12. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 12 Replication Engine Design  Periodic execution of refresh tasks : scheduling facility in slapd_daemon_task() simple runqueue implementation  Storage of sync cookie in consumer : Subentry : syncConsumerSubentry  Operational attributes : Regenerated at replica as needed  structuralObjectClass : generate on-the-fly in slap_mods_opattrs()  UUID, CSN based implementation UUID : stable entry identifier CSN as cookies  Configuration example test017 : refreshOnly, test018 : refreshAndPersist, test020 : cascading r1 p1 master p3 r2 p2

13. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 13 Replication Engine Design : Glueing  Glues : Naming entries for holes in DIT -Delivery can be out of hierarchy order after several rounds of updates -Partial replication  Glue construction -syncrepl_add_glue() -Find first non-glue superior object -Create glues from down to the entry  Schema checking bypass for glues (rdn attribute requirement)  Glueing for LDAP Proxy Cache consumer.company.com ou=users cn=foo com2 ou=assets com1 ou=computers

14. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 14 Client-based Replication Engine  Heterogeneous replication - SyncRepl engine needs to talk to generic LDAP servers  Synchronization without LDAP Content Synchronization 1.Search for (&(original filter)) asking only UUID and CSN attributes - Present phase 2.Delete replica entries not returned by (1) 3.Search for (&(original filter)(entryCSN>cookie)(entryCSN=<maxCSN(1)) asking replicated attributes + UUID + CSN - Update phase -Replica is synchronized to the point maxCSN(1)  Comparison with SyncRepl with LDAP Content Synchronization -Only supports polling -Extra requests / replies -Extra traffic (only present mode)

15. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 15 Target Applications  Slurpd replacement -OpenLDAP to OpenLDAP replication based on LDAP content sync protocol -Heterogeneous replication by using client-based replication engine  LDAP Proxy Cache synchronization -Replace current TTL based scheme -Replication and Caching  IBM Directory Integrator Connector –Heterogeneous Directory Synchronization : Meta-directory

16. Sync-based Replication OpenLDAP Developer’s Conference – Vienna – 18 July 2003 IBM logo must not be moved, added to, or altered in any way. Background should not be modified. Title/subtitle/confidentiality line: 10pt Arial Regular, white Maximum length: 1 line Information separated by vertical strokes, with two spaces on either side Slide heading: 28pt Arial Regular, blue R120 | G137 | B251 Maximum length: 2 lines Slide body: 18pt Arial Regular, black Square bullet color: teal R045 | G182 | B179 Recommended maximum text length: 5 principal points Group name: 14pt Arial Regular, white Maximum length: 1 line Copyright: 10pt Arial Regular, white Optional slide number: 10pt Arial Bold, white Template release: Oct 02 For the latest, go to http://w3.ibm.com/ibm/presentations Indications in green = Live content Indications in white = Edit in master Indications in blue = Locked elements Indications in black = Optional elements 16 Summary  LDAP Content Sync Protocol -draft-zeilenga-ldup-sync-xx.txt  OpenLDAP SyncRepl Engine -servers/slapd/syncrepl.c -tests/data/slapd-syncrepl-master.conf - slapd-syncrepl-slave-xxxxx.conf  Any Questions ?