1. 1 PATTERNS FOR AUTOMATIC INTEGRATION OF THE DOMAIN-DATA LAYERS IN ENTERPRISE-SYSTEMS Mira Balaban Joint work with Lior Limonad Ben Gurion University Beer-Sheva, Israel

2. IBM Haifa, Oct 062 Example: an e-Mail client system Persistency requirements:

3. IBM Haifa, Oct 063 Demo Persistent class Persistent  In-memory Persistent  Persistent Persistent collection Domain Layer Storage Data Access Layer

4. IBM Haifa, Oct 064 Modern Systems Built into layers Typical layers are: Presentation Domain Data Grasp changing requirements as a norm Such as: persistency requirements Adhering to the MDA spirit  View DB as a natural extension of the application

5. IBM Haifa, Oct 065 Persistency Motivation A strict demand to keep an object state, when its data is important for the continuity of the system. Objects are too big. Large amount of objects. Too much for loading all together to memory (hardware limit). Need to deal with objects one at the time. Need to save historical states of the system. Sharing object states between multiple applications.

6. IBM Haifa, Oct 066 Agenda 1. Demo Motivation 2. The problem of partial persistency The research goal 3. Partial solutions 4. The DAL Patterns solution 5. Application Example 6. Evaluation Correctness proof Merits and shortages 7. Conclusions and future work

7. IBM Haifa, Oct 067 Partial Consistency Input: Mapping between application (Domain layer) and storage (Data layer). Based on a semantic linkage between layers. Output: integrated layers. Task: Introduce Partial persistency. Requires system restructuring (refactoring, transformation). Requirements: Persistency of (only) marked objects Behavior preservation Transparency – minimal changes to the domain layer

8. IBM Haifa, Oct 068 Persistency of (only) marked objects complications (1) Co-existence of storage and in-memory objects Mixture – ingoing and outgoing references: Ingoing: In-memory  Persistent Availability Outgoing: Persistent  In-memory Reference preservation

9. IBM Haifa, Oct 069 Persistency of (only) marked objects complications (2) Duplication handling : Consistency– Inter-layer – Duplicated of objects Duplicated references Intra-layer – preventing duplication of objects

10. IBM Haifa, Oct 0610 Insertion of persistency requirements Goal: Achieve automatic persistency insertion in a running application Integration must fulfill Partial persistency Persistency of (only) marked objects Behavior preservation Transparent integration Domain-layer stays intact

11. IBM Haifa, Oct 0611 Agenda 1. Demo Motivation 2. The problem of partial persistency The research goal 3. Partial solutions 4. The DAL Patterns solution 5. Application Example 6. Evaluation Correctness proof Merits and shortages 7. Conclusions and future work

12. IBM Haifa, Oct 0612 Current Technologies (1) e.g. J2EE / Hibernate Support: Mapping: may be generated automatically Intra-layer Consistency Availability – handled by the environment e.g. EJB container

13. IBM Haifa, Oct 0613 Current Technologies (2) e.g. J2EE / Hibernate Necessary manual work: Modification of persistent classes Adhering to standards (e.g. EntityBean interface) Mixture handling Handle ingoing and outgoing references e.g. serialization Interlayer consistency Object consistency Manual invocation of, e.g., session.save(obj), session.load(class,id), ejbLoad() / ejbStore() / ejbCreate() / db.bind(…) / db.deletePersistent(…) Reference consistency e.g. between two stored objects  Still require writing tedious (bug infected…) code!  Domain-Layer is modified – No Transparency!

14. IBM Haifa, Oct 0614 Current Technologies (3) e.g. J2EE / Hibernate  Missing: An algorithmic, correct, automated solution for integration of the Domain and the Persistent layers

15. IBM Haifa, Oct 0615 Partial solutions: Fowler’s Active Record Pattern Transparent Partial persistency Preserving references between objects Preservation of objects availability Behavior preserving Object structure preservation

16. IBM Haifa, Oct 0616 Partial solutions: Fowler’s Data Mapper pattern Transparent Partial persistency Preserving references between objects Preservation of objects availability Behavior preserving

17. IBM Haifa, Oct 0617 Agenda 1. Demo Motivation 2. The problem of partial persistency The research goal 3. Partial solutions 4. The DAL Patterns solution 5. Application Example 6. Evaluation Correctness proof Merits and shortages 7. Conclusions and future work

18. IBM Haifa, Oct 0618 The DAL Patterns Solution 1. A small set of Data-Access Layer (DAL) patterns (refactorings, transformations): I. Core pattern: Proxy-Data-Mapper: Context independent insertion of persistency II. Context Patterns: Context tailored refactorings of: Persistent-to-In-Memory interactions Persistent-to-Persistent interactions 2. DAL-Insertion algorithm A combined procedure for pattern application

19. IBM Haifa, Oct 0619 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application:

20. IBM Haifa, Oct 0620 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application: Extract subclass “MessageImp”

21. IBM Haifa, Oct 0621 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application: Extract subclass “MessageImp” Move implementation: Message  MessageImp

22. IBM Haifa, Oct 0622 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application: Extract subclass “MessageImp” Move implementation: Message  MessageImp Add a MessageDataMapper

23. IBM Haifa, Oct 0623 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application: Extract subclass “MessageImp” Move implementation: Message  MessageImp Add a MessageDataMapper Add a MessageProxy

24. IBM Haifa, Oct 0624 Proxy Data Mapper Pattern Application precondition: message has a factory method Automatic application: Extract subclass “MessageImp” Move implementation: Message  MessageImp Add a MessageDataMapper Add a MessageProxy Turn Message to be abstract Redirect the factory method

25. IBM Haifa, Oct 0625 Example of Proxy Data Mapper Usage

26. IBM Haifa, Oct 0626 The Proxy Data Mapper Automatable Refactoring (1) Input: persistentClass, dbSchemaElement 1. Extract the implementation from the persistent class: persistentClassImp = extractSubClass(persistentClass); 2. Construct the data mapper class: dataMapper = addClass(concat(persistentClass.name+''DataMapper'')); addMethod(dataMapper, “load”, constructLoadImpMethod(persistentClassImp, dbSchemaElement)); addMethod(dataMapper, “store”, constructStoreImpMethod(persistentClassImp, dbSchemaElement)); addMethod(dataMapper, “update”, constructUpdateImpMethod(persistentClassImp, dbSchemaElement)); addMethod(dataMapper, “delete”, constructDeleteImpMethod(persistentClassImp, dbSchemaElement));

27. IBM Haifa, Oct 0627 The Proxy Data Mapper Automatable Refactoring (2) 3. Construct the proxy class: proxy = addClass(concat(persistentClass.name+''Proxy'')); addField(proxy, “_idField”, Integer); addField(proxy, “_mapperField”, dataMapper); Add finalize, load, update methods. add all constructors: for each constructor c in constructors(persistentClassImp) addMethod(proxy, proxy.name, constructConstructore(dataMapper, persistentClassImp)); add all delegated methods to the proxy class: for each method m in instance_methods(persistentClassImp) addMethod(proxy, m.name, constructDelegatedMethod(m, persistentClassImp);

28. IBM Haifa, Oct 0628 The Proxy Data Mapper Automatable Refactoring (3) 4. Redirect factory methods: for each factoryMethod f in factories(persistentClass) redirectFactoryMethod(proxy); 5. Tag modifications: stereotype(persistentClass, ``abstract'');

29. IBM Haifa, Oct 0629 Proxy Data Mapper – Example Applying PDM to the Message class:

30. IBM Haifa, Oct 0630 Problems solved by the Proxy Data Mapper Pattern Persistency of only marked classes Mixture Availability Reference preservation Duplication Consistency Inter layer Object – handled by the proxy Reference Intra layer – factory method uses Fowler’s “Identity Map” Behavior preservation Full Transparency

31. IBM Haifa, Oct 0631 The DAL Patterns Solution 1. A small set of Data-Access layer patterns: I. Core pattern: Proxy-Data-Mapper: Context independent insertion of persistency II. Context Patterns: Context tailored refactoring of: Persistent-to-In-Memory interactions Persistent-to-Persistent interactions 2. DAL-Insertion algorithm A combined procedure for pattern application

32. IBM Haifa, Oct 0632 Context Patterns Problem parameters: In/Out associations into/from the persistent class Cardinality constraints Persistent/In-memory marking of the Context class 5 context patterns > Context Class > fooClass 1 * * 1

33. IBM Haifa, Oct 0633 DAL Patterns

34. IBM Haifa, Oct 0634 I. Persistent Collection Data Mapper context

35. IBM Haifa, Oct 0635 I. Persistent Collection Data Mapper – problems(1) The in-memory collection realization problem: The collection of all proxy objects may be too large to fit in-memory m1:message Imp p1:Message Proxy :Message DataMapper Messages m 1..m n vs:Virus Scan col:Messages Collection m1:message Imp p1:Message Proxy :Message DataMapper m 3 :message Imp p 3 :Message Proxy :Message DataMapper m n :message Imp p n :Message Proxy :Message DataMapper

36. IBM Haifa, Oct 0636 I. Persistent Collection Data Mapper – problems(1) Solution to the collection realization problem: Introduce a collection that loads corresponding proxy objects by demand Messages m 1..m n vs:Virus Scan col:Messages CollectionMapper m n :message Imp p n :Message Proxy :Message DataMapper getMsg(n) id 1..id n col:Messages Collection

37. IBM Haifa, Oct 0637 I. Persistent Collection Data Mapper – problems(2) The deletion problem: Collection of IDs  A new implicit reference to persistent objects

38. IBM Haifa, Oct 0638 I. Persistent Collection Data Mapper – problems(2) The deletion problem: When a proxy is deleted, how to prevent message deletion from storage? Messages m 1..m n vs:Virus Scan col:Messages CollectionMapper mn:message Imp pn:Message Proxy :Message DataMapper getMsg(n) id 1..id n col:Messages Collection

39. IBM Haifa, Oct 0639 I. Persistent Collection Data Mapper– problems (2) Solution to the deletion problem: Extend the garbage collector to handle persistent objects. References to persistent objects = clients holding their ids. So far: In-memory proxy objects. Collections of persistent objects.

40. IBM Haifa, Oct 0640 I. Persistent Collection Data Mapper– problems (2) Solution to the deletion problem: Insert TableManager (object) for the message – guards storage deletion by tracking reference count to it All clients: proxys/collections, notify it about reference changes :Message TableManager Increase(id) Decrease(id) mn:message Imp pn:Message Proxy :Message DataMapper

41. IBM Haifa, Oct 0641 I. Persistent Collection Data Mapper Solution structure:

42. IBM Haifa, Oct 0642 I. Persistent Collection Data Mapper– problems (3) The collection itself is too large to reside in- memory: col:Messages CollectionMapper id 1 …………………………………………………………………..id n >

43. IBM Haifa, Oct 0643 I. Persistent Collection Data Mapper– problems (3) Extended solution: Make the collection persistent Apply the Proxy Data Mapper pattern col:Messages CollectionMapperImp id 1 ……………………………………………………………… …..id n Messages CollectionMapper proxy:Messages CollectionMapperProxy mapper:Messages CollectionMapperMapper

44. IBM Haifa, Oct 0644 I. Extended Persistent Collection Data Mapper

45. IBM Haifa, Oct 0645 II. Memoization Data Mapper patterns context Memoization data mapper Multi memoization data mapper

46. IBM Haifa, Oct 0646 II. Memoization Data Mapper patterns problems The memoization/multi-memoization problems When a message is stored, how to handle outgoing references to other in-memory objects or in-memory collections? m:messageI mp :Message Proxy :Message DataMapper Messages m e:Encryption Key

47. IBM Haifa, Oct 0647 II. Memoization Data Mapper patterns problems Solution: Extend the proxy with a reference to the outgoing referenced object or collection :Message Proxy :Message DataMapper Messages m e:Encryption Key m:messageI mp

48. IBM Haifa, Oct 0648 II. Memoization data mapper pattern Solution structure:

49. IBM Haifa, Oct 0649 III. Multi memoization data mapper pattern Solution structure:

50. IBM Haifa, Oct 0650 IV. Persistent to Persistent Patterns – context Persistent to Persistent Persistent to multi persistent

51. IBM Haifa, Oct 0651 Main problem – Interlayer reference consistency Example: element1  element3

52. IBM Haifa, Oct 0652 IV. Persistent to Persistent Pattern – problems (1) The load problem: Realization of an outgoing reference to another persistent object m:messageImp:Message Proxy :Message DataMapper Messages m Attachments a

53. IBM Haifa, Oct 0653 IV. Persistent to Persistent Pattern – problems (1) Solution to the load problem: A lightweight representative (proxy) is loaded by the persistent data mapper m:messageImp:Message Proxy :Message DataMapper Messages m Attachments a :Attachment Proxy

54. IBM Haifa, Oct 0654 IV. Persistent to Persistent Pattern – problems (2) Update/create problems of the referenced object: If attachment is replaced, how to know that a new one need to be stored. If attachment is updated, how to know that it should be updated in storage. m:message Imp :Message Proxy :Message DataMapper Messages m Attachments a m:Attach Imp. :Attach. Proxy :Attach, DataMapper

55. IBM Haifa, Oct 0655 IV. Persistent to Persistent Pattern – problems (2) Solution to the referenced object update/create problem: Message data mapper handles reference replacement Attachment update handled by attachment data-mapper (PDM) m:message Imp :Message Proxy :Message DataMapper Messages m Attachments a m:Attach Imp. :Attach. Proxy :Attach, DataMapper

56. IBM Haifa, Oct 0656 IV. Persistent to Persistent Pattern – problems (3) The deletion problem: When message is deleted from memory, should attachment be deleted from storage? m:messageImp:Message Proxy :Message DataMapper Messages m Attachments a m:Attach Imp, :Attach. Proxy :Attach, DataMapper

57. IBM Haifa, Oct 0657 IV. Persistent to Persistent Pattern – problems (3) Solution to the deletion problem: Use the extended garbage collection mechanism that handles persistent objects. Insert TableManager (object) for the attachment – As in the “persistent collection data mapper” pattern. m:Attach Imp, :Attach. Proxy :Attach, DataMapper :Attach, TableManager Increase(id) Decrease(id)

58. IBM Haifa, Oct 0658 V. Persistent to Multi Persistent Pattern – problem (4) Persistent collection realization problem How to represent an in-memory 1 to Many relation in storage? Domain Layer Data Layer

59. IBM Haifa, Oct 0659 V. Persistent to Multi Persistent Pattern – problem (4) Solution to the persistent collection realization problem: Insert a collection of proxies (loaded by demand) m:message Imp :Message Proxy :Message DataMapper m:MailServer Imp :MailServer Proxy :MailServer DataMapper :MessagesC ollection MapperImp load instantiates DB resolve The collection is created and identified by the client data-mapper

60. IBM Haifa, Oct 0660 IV. Persistent to Persistent Pattern Solution structure:

61. IBM Haifa, Oct 0661 V. Persistent to Multi Persistent Pattern Solution structure:

62. IBM Haifa, Oct 0662 DAL Patterns characterizations Realized as refactorings Can be automated Transparent insertion: No modifications/deletions. Introduce a new Data-Access Layer (DAL) New classes introduced.

63. IBM Haifa, Oct 0663 The DAL Patterns Solution 1. A small set of Data-Access layer patterns: I. Core pattern: Proxy-Data-Mapper: Context independent insertion of persistency II. Context Patterns: Context tailored refactoring of: Persistent-to-In-Memory interactions Persistent-to-Persistent interactions 2. DAL-Insertion algorithm A combined procedure for pattern application

64. IBM Haifa, Oct 0664 Dal-Insertion Algorithm Combined procedure for patterns application:

65. IBM Haifa, Oct 0665 Application Example Input System: Analysis Model

66. IBM Haifa, Oct 0666 Application Example Input System

67. IBM Haifa, Oct 0667 Application Example Input System

68. IBM Haifa, Oct 0668 Application Example Input System: persistency marking

69. IBM Haifa, Oct 0669 Application Example Input System: factory methods

70. IBM Haifa, Oct 0670 Application Example Apply Proxy-Data-Mapper pattern to all persistent classes:

71. IBM Haifa, Oct 0671 Application Example Apply memoization-proxy-data-mapper pattern

72. IBM Haifa, Oct 0672 Application Example Apply multi-memoization-proxy-data-mapper pattern

73. IBM Haifa, Oct 0673 Application Example Apply persistent-to-persistent pattern

74. IBM Haifa, Oct 0674 Application Example Apply persistent-to-multi-persistent pattern

75. IBM Haifa, Oct 0675 Application Example Apply persistent-collection-data-mapper pattern

76. IBM Haifa, Oct 0676 Application Example Result

77. IBM Haifa, Oct 0677 Evaluation – Integration Correctness Persistency of all and only marked objects Behavior preservation: Hard to quantify!  Measurable observables: Object structure preservation Preservation of references between objects Preservation of objects availability Method preservation Transparent integration No domain-layer changes

78. IBM Haifa, Oct 0678 DAL-Insertion algorithm – characterizations : Requires only local information – no data flow analysis: The system Marked persistent classes Marked persistent collections Non deterministic Minimal – based on a minimal set of constraints for DAL pattern combination Correct – satisfies all observables - solves the problem of partial persistency

79. IBM Haifa, Oct 0679 Overall Persistency Insertion Steps 1. Domain layer marking of: Persistent classes Persistent class collections 2. Data Layer: Mapping specification between domain-layer classes and storage elements (i.e. Tables) Access specifications (i.e. DB driver) 3. Automatic Data Access Layer insertion Application of DAL-Insertion algorithm

80. IBM Haifa, Oct 0680 Conclusion The DAL patterns approach provides the missing link towards full automation of Domain-Data layers integration Patterns can be implemented as independent software refactorings (in a compositional refactoring manner). Experimental tool for static aspects was implemented Restricted (toy) implementation of the patterns exists.

81. IBM Haifa, Oct 0681 Future Hierarchy related patterns should be finalized. Full implementation into an existing IDE (e.g. Eclipse) – started. Extend the solution to a shared storage Extension to other paradigms: Networking, distributed computing, web services Problems: recovery handling, transactions (current patterns assume auto commit), performance – query optimization Collection persistency

82. 82 Thank You

83. IBM Haifa, Oct 0683 Proxy Data Mapper Pattern Avoiding in memory duplications Handled by the implementation of factory method e.g. using fowler’s “Identity Map”