1. Designing a Flexible Framework for a Table Abstraction H. Conrad Cunningham 1 Yi Liu 2 Jingyi Wang 3 1 University of Mississippi 2 South Dakota State University 3 Acxiom Corporation

2. Hot Spots, Patterns, and Contracts: An Exploration of Software Framework Design H. Conrad Cunningham 1 Yi Liu 2 1 University of Mississippi 2 South Dakota State University

3. 26-Jan-2010 3 Hot Spots References  Coplien, J., Hoffman, D., Weiss, D. Commonality and variability in software engineering, IEEE Software, vol. 15, no. 6, pp. 37-45, 1998.  Schmid, H. A., Framework design by systematic generalization, In: Fayad, M. E., Schmidt, D. C., Johnson, R. E., editors, Building Application Frameworks: Object- Oriented Foundations of Framework Design, Wiley, pp. 353-378, 1999.

4. 26-Jan-2010 4 Patterns References  Gamma, R., Helm, R., Johnson, R., Vlissides, J., Design Patterns: Elements of Reusable Object-Oriented Software, Addison Wesley, 1995.  Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M., Pattern-Oriented Software Architecture: A System of Patterns, Wiley, 1996.

5. 26-Jan-2010 5 Contracts References  Meyer, B., Object-Oriented Software Construction, 2nd Edition, Prentice Hall PTR, 1997.

6. 26-Jan-2010 6 Source Papers  Cunningham, H. C., Liu, Y., Wang. J., Designing a flexible framework for a table abstraction, Chapter 13 in: Chan, Y., Talburt, J., Talley, T., editors, Data Engineering: Mining, Information, and Intelligence, Springer, pp. 279-314, 2010.  Cunningham, H. C., Wang, J., Building a Layered framework for the table abstraction, In Proceedings of the 16th ACM Symposium on Applied Computing, pp. 668-674, 2001.

7. 26-Jan-2010 7 Software Interfaces Project (1996-2000) Context: development of an instructional data and file structures library (for Java in CS 3)  artifacts for study of good design techniques  system for use, extension, and modification Motivation: study techniques for  presenting important methods to students (frameworks, software design patterns, design by contract, etc.)  unifying related file and data structures in framework

8. 26-Jan-2010 8 Table Abstract Data Type (ADT)  Collection of records o each a finite sequence of data fields o key field value uniquely identifies record within collection  Operations on collection and on record using key  Several different implementations possible key1data1 key2data2 key3data3 key4data4

9. 26-Jan-2010 9 Table ADT Operations  Insert new record  Delete existing record given key  Update existing record  Retrieve existing record given key  Get number of records  Query whether contains given key  Query whether empty  Query whether full

10. 26-Jan-2010 10 Software Framework  Generic application allowing creation of members of family of related programs  Reusable design expressed as set of abstract classes and way they collaborate  Common and variable aspects known as frozen spots and hot spots Framework Framework library User-supplied code (application specific) Hot spots Frozen spots

11. 26-Jan-2010 11 Software Design Patterns  Describe recurring design problems arising in specific contexts  Present well-proven generic solution schemes  Describe solution’s components and their responsibilities and relationships  To use: o select pattern that fits problem o structure solution to follow pattern

12. 26-Jan-2010 12 Software Design Contracts  Preconditions for correct use of operation  Postconditions for correct result of operation  Invariant conditions for correct implementation of class Insert record operation pre: record is valid and not already in table post: record now in table Invariant for table all records are valid, no duplicate keys

13. 26-Jan-2010 13 Initial Table Framework Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

14. 26-Jan-2010 14 Hot Spot Analysis of Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

15. 26-Jan-2010 15 Table Hot Spot Analysis Frozen spot 1. Table ADT functionality Hot spots 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients) 3. Variability in external representation of record state 4. Variability in access (i.e, indexing) mechanism 5. Variability in storage mechanism Hot spots #1 and #2 not completely independent Hot spot #3 quite subtle Desire to separate concerns (i.e., the hot spots)

16. 26-Jan-2010 16 Layered Architecture Pattern  Distinct groups of services  Hierarchical arrangement of groups into layers  Layer implemented with services of layer below  Enables independent implementation of hot spots in layers Client Layer Access Layer Storage Layer

17. 26-Jan-2010 17 Applying Layered Architecture Pattern Client Layer – Hot spots #1 and #2 o includes client application programs o implements record and key abstractions o uses layer below to store and retrieve records Access Layer – Frozen spot and Hot spot #4 o includes table implementations o provides key-based access to records for layer above o calls back to record and key implementations in level above o uses physical storage in layer below Storage Layer – Hot spot #5 o storage managers o provides physical storage for records o calls back to record implementations in layers above

18. 26-Jan-2010 18 Client Layer of Architecture Encapsulate hot spots #1 and #2 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients)

19. 26-Jan-2010 19 Client Layer Design Challenges o enabling Access Layer to access client- defined records and keys o avoiding unnecessary programming to use common data types Pattern o Interface – applied independently for hot spots #1 (keys) and #2 (records)

20. 26-Jan-2010 20 Client Layer Interfaces (1 of 2) Hot spot #1 – Variability in keys  Must be able to (at a minimum) compare with total ordering  Choose to use Comparable interface from Java API for convenience Comparable interface for keys o int compareTo(Object key) compares object with argument

21. 26-Jan-2010 21 Client Layer Interfaces (2 of 2) Hot spot #2 – Variability in records  Must be able to extract keys for comparison  Choose to define an appropriate new interface Keyed interface for records o Comparable getKey() extracts key from record

22. 26-Jan-2010 22 Client Layer Model Abstract predicates – for use in specification of this and other layers (not necessarily implemented) o boolean isValidKey(Object key) yields true for valid key values o boolean isValidRec(Object rec) yields true for valid record values

23. 26-Jan-2010 23 Client Layer Design Contracts Hot spot #1 Comparable int compareTo(Object key) int compareTo(Object key) Pre: isValidKey(this) && isValidKey(key) Pre: isValidKey(this) && isValidKey(key) Post: result == (if this < key then -1 Post: result == (if this < key then -1 else if this == key then 0 else if this == key then 0 else 1) else 1) Hot spot #2 Keyed Comparable getKey() Pre: isValidRec(this) Pre: isValidRec(this) Post: ( result == this.key ) Post: ( result == this.key ) && isValidKey(result) && isValidKey(result)

24. 26-Jan-2010 24 Client Layer Challenges Met?  Enabling Access Layer to access client- defined records and keys o records and keys implementations encapsulated within objects that implement Keyed and Comparable  Avoiding unnecessary programming to use common data types o reuse Comparable from Java API

25. 26-Jan-2010 25 Access Layer of Architecture Provide frozen spot Table ADT functionality Encapsulate hot spot #4: Variability in access (i.e, indexing) mechanism

26. 26-Jan-2010 26 Access Layer Design Challenges o enabling use of client-defined keys and records o enabling diverse implementations of indexing structure o enabling use of diverse storage structures Pattern o Interface – applied for frozen spot (Table ADT) and hot spot #4 (access)

27. 26-Jan-2010 27 Access Layer Interface Table void insert(Keyed r) inserts r into table void delete(Comparable key) removes record with key void update(Keyed r) changes record with same key Keyed retrieve(Comparable key) returns record with key int getSize() returns size of table boolean containsKey(Comparable key) searches for key boolean isEmpty() checks whether table is empty boolean isFull() checks whether table is full o for unbounded, always returns false

28. 26-Jan-2010 28 Access Layer Model Partial function table :: Comparable  Keyed o represents abstract table state o consider this: number of entries finite or infinite? If finite, bounded or unbounded? If infinite, countable or not? o #table in postcondition denotes table before operation o use as either function or set of key-record pairs Abstract predicates (to capture environmental assumptions) o isValidKey(Object) and isValidRec(Object) defined in Client Layer to identify valid keys and records o isStorable(Object) defined in Storage Layer to identify records that can be stored

29. 26-Jan-2010 29 Table Design Contract (1 of 5) Invariant (  k, r : r == table(k) : isValidRec(r) && isStorable(r) && k == r.getKey() ) && k == r.getKey() ) All stored keys and records in the table are valid and capable of being stored on the chosen external device, and the records can be assessed by their keys.

30. 26-Jan-2010 30 Table Design Contract (2 of 5) void insert(Keyed r) inserts r into table Pre: isValidRec(r) && isStorable(r) && !containsKey(r.getKey())&& !isFull() Post: table == #table  {(r.getKey(),r)} void delete(Comparable key) removes record with key from table Pre: isValidKey(key) && containsKey(key) Post: table == #table - {(key,#table(key))} Question: Does this allow table be initialized nonempty?

31. 26-Jan-2010 31 Table Design Contract (3 of 5) void update(Keyed r) changes record with same key Pre: isValidRec(r) && isStorable(r) && containsKey(r.getKey()) Post: table == (#table - {(r.getKey(),#table(r.getKey()))} )  {(r.getKey(),r)} Keyed retrieve(Comparable key) returns record with key Pre: isValidKey(key) && containsKey(key) Post: result == #table(r.getKey())

32. 26-Jan-2010 32 Table Design Contract (4 of 5) int getSize() returns size of table Pre: true Post: result == cardinality(#table) Note: If size not finite, need way to represent infinity boolean containsKey(Comparable key) searches table for key Pre: isValidKey(key) Post: result == defined(#table(key))

33. 26-Jan-2010 33 Table Design Contract (5 of 5) boolean isEmpty() checks whether table is empty Pre: true Post: result == (#table =  ) boolean isFull() checks whether table is full – for unbounded, always returns false Pre : true Post: result == (#table implementation has no free space to store record) no free space to store record)

34. 26-Jan-2010 34 Client/Access Layer Interactions  Client calls Access Layer class implementing Table interface  Access calls back to Client implementations of Keyed and Comparable interfaces

35. 26-Jan-2010 35 Access Layer Challenges Met?  Enabling use of client-defined keys and records o callbacks to Comparable and Keyed abstractions that hide the implementation details  Enabling diverse implementations of the table o careful design of table interface semantics using design by contract  Enabling use of diverse storage mechanisms o calls to Storage Layer interfaces

36. 26-Jan-2010 36 Storage Layer of Architecture Encapsulate hot spot #5: Variability in storage mechanism

37. 26-Jan-2010 37 Storage Layer Design Challenges o supporting client-defined records o supporting diverse table implementations in Access Layer (simple indexes, hashing, balanced trees, etc.) o allowing diverse physical media (in-memory, on- disk, etc.) o enabling persistence of table, including access layer o decoupling implementations as much as possible Patterns o Bridge o Proxy

38. 26-Jan-2010 38 Bridge Pattern  Decouple “interface” from “implementation” o table from storage in this case  Allow them to vary independently o plug any storage mechanism into table Table RecordStore Simple Indexed File Hashed File Slotted File Store Vector File uses

39. 26-Jan-2010 39 Proxy Pattern  Transparently manage services of target object o isolate Table implementation from nature/location of record slots in RecordStore implementation  Introduce proxy object as surrogate for target handle Table RecordSlot RecordStore

40. 26-Jan-2010 40 Storage Layer Interfaces RecordStore o operations to allocate and deallocate storage slots RecordSlot o operations to get and set records in slots o operations to get handle and containing RecordStore

41. 26-Jan-2010 41 Storage Layer Model (1 of 2) Partial function store :: int  Object o represents abstract RecordStore state Set Handles  int, NULLHANDLE  Handles Set alloc  Handles o represents set of allocated slot handles Set unalloc == Handles - alloc o represents set of unallocated slot handles Abstract predicate isStorable(Object) o depends on storage mechanism

42. 26-Jan-2010 42 Storage Layer Model (2 of 2) Invariant: (  h,r : r == store(h) : isStorable(r)) && (  h :: h  alloc == defined(store(h))) All records in the store are capable of being stored on the selected medium and the stored records can be access by their handles.

43. 26-Jan-2010 43 RecordStore Interface RecordSlot newSlot() allocates a new record slot RecordSlot getSlot(int handle) rebuilds record slot using given handle void releaseSlot(RecordSlot slot) deallocates record slot

44. 26-Jan-2010 44 RecordStore Design Contract (1 of 2) RecordSlot newSlot() allocates a new record slot, but it might be done in a lazy fashion Pre: true Post: result.getContainer() == this && result.getRecord() == NULLRECORD && result.getHandle()  #alloc && result.getHandle()  alloc  {NULLHANDLE} Note: NULLRECORD built according to Null Object pattern RecordSlot getSlot(int handle) rebuilds record slot using given handle Pre: handle  alloc Post: result.getContainer() == this && result.getRecord() == #store(handle) && result.getHandle() == handle

45. 26-Jan-2010 45 RecordStore Design Contract (2 of 2) void releaseSlot(RecordSlot slot) deallocates record slot Pre: slot.getHandle()  alloc  {NULLHANDLE} && slot.getContainer() == this Post: alloc == #alloc - {slot.getHandle()} && store == #store – store == #store – {(slot.getHandle(),slot.getRecord())} {(slot.getHandle(),slot.getRecord())}

46. 26-Jan-2010 46 RecordSlot Interface void setRecord(Object rec) stores rec in this slot allocation of handle done here or already done by getSlot allocation of handle done here or already done by getSlot Object getRecord() returns record stored in this slot int getHandle() returns handle of this slot RecordStore getContainer() returns reference to RecordStore holding this slot boolean isEmpty() determines whether this slot empty

47. 26-Jan-2010 47 RecordSlot Model Each slot has two attributes:  reference to RecordStore to which this RecordSlot belongs, which is immutable  handle for the associated physical storage slot in the RecordStore (as constrained in RecordStore ) Invariant: getHandle()  alloc  {NULLHANDLE} The handle of a RecordSlot object denotes a slot of the store that has been allocated unless it has the value NULLHANDLE.

48. 26-Jan-2010 48 RecordSlot Design Contract (1 of 3) void setRecord(Object rec) stores rec in this slot o allocation of handle done here or already done by getSlot() Pre: isStorable(rec) Post: Let h == getHandle(): (h  #alloc  store == (#store - (h  #alloc  store == (#store - {(h,#store(h))})  {(h,rec)} ) && (h = NULLHANDLE  (h = NULLHANDLE  (  g : g  #unalloc : alloc == #alloc  {g} && alloc == #alloc  {g} && store = #store  {(g,rec)} )) store = #store  {(g,rec)} ))

49. 26-Jan-2010 49 RecordSlot Design Contract (2 of 3) Object getRecord() returns record stored in this slot Pre: true Post: Let h == getHandle() : ( h  #alloc  result == #store(h)) && ( h == NULLHANDLE  result == NULLRECORD ) int getHandle() returns handle of this slot Pre: true Post: result == this.handle

50. 26-Jan-2010 50 RecordSlot Design Contract (3 of 3) RecordStore getContainer() returns reference to RecordStore holding this slot Pre: true Post: result == this.container boolean isEmpty() determines whether this slot empty Pre: true Post: result == (getHandle() == NULLHANDLE || getRecord() == NULLRECORD)

51. 26-Jan-2010 51 Storage Layer Challenges Met? (1 of 2)  Supporting client-defined records o deferred to Externalization Module  Supporting diverse table implementations in Access Layer o careful design of RecordStore and RecordSlot abstractions to have sufficient functionality  Allowing diverse physical media o careful design of RecordStore abstraction to hide media, implementable in many ways, o partly deferred to Externalization Module

52. 26-Jan-2010 52 Storage Layer Challenges Met? (2 of 2)  Decoupling implementations as much as possible o use of RecordSlot, handle, and Externalization Module  Enabling persistence of Table, including Access Layer o store RecordStore identifier and handles

53. 26-Jan-2010 53 Access/Storage Layer Interactions  Access Layer Table calls RecordStore in Storage Layer to get RecordSlot object  Access Layer calls RecordSlot to store and retrieve its records  If needed, RecordSlot calls back to Record implementation in Access Layer o interface Record defined in Externalization Module

54. 26-Jan-2010 54 Externalization Module of Architecture Encapsulate Hot Spot #3: Variability in external representation of record state

55. 26-Jan-2010 55 Externalization Module Design Challenges (e.g., to support Storage Layer) o supporting client-defined records o allowing diverse physical media o decoupling implementations Pattern: Interface

56. 26-Jan-2010 56 Externalization Module Interface Record o void writeRecord(DataOutput out) writes the client’s record to stream out o void readRecord(DataInput in) reads the client’s record from stream in o int getLength() returns number of bytes in external representation of client’s record (e.g., written by writeRecord)

57. 26-Jan-2010 57 Externalization Module Contract (1 of 2) void writeRecord(DataOutput out) writes this record to stream out Pre: true Post: suffix of stream out == this record’s state encoded as byte sequence this record’s state encoded as byte sequence void readRecord(DataInput in) reads this record from stream in Pre: true Post: this record’s state == prefix of stream in decoded from byte sequence prefix of stream in decoded from byte sequence

58. 26-Jan-2010 58 Externalization Module Contract (2 of 2) int getLength() returns number of bytes in external representation of this record (e.g., written by writeRecord) Pre: true Post: result == number of bytes in external representation of this record representation of this record State Restoration Property: If, for some record, writeRecord() is followed by readRecord() of the same byte sequence, the observable state of the record will be unchanged.

59. 26-Jan-2010 59 Externalization Challenges Met?  Supporting client-defined records o records needing to be externalized implement Record interface  Allowing diverse physical media o Record interface implementation provides low-level physical representation of record storable on most media  Decoupling implementations o Record decouples use from implementation

60. 26-Jan-2010 60 Externalization Module Interactions  Record used by Storage Layer may be defined by either layer above o hold one Client Layer Keyed record o hold multiple, or fractional, Client Layer records managed by Access Layer (e.g, multiway tree nodes)  Storage Layer calls back to Record implementation in layer above o implementation in Access Layer might call back to implementations in Client Layer

61. 26-Jan-2010 61 Abstraction Usage Relationships Table KeyedComparable RecordStoreRecordSlot Record Access Layer Storage Layer Client LayerExternalization Module

62. 26-Jan-2010 62 Other Design Patterns Used  Null Object (for NULLRECORD in Storage Layer)  Iterator o extended Table operations o query mechanism o utility classes  Template Method  Decorator  Strategy

63. 26-Jan-2010 63 Evolving Frameworks Patterns  Generalizing from three examples  Whitebox framework  Component library o Wang prototype: two Table s and three RecordStore s  Hot spots  Pluggable objects  Fine-grained objects  Blackbox framework

64. 26-Jan-2010 64 Conclusions  Hot spot analysis reveals generalization opportunities  Design patterns offer systematic way to discover reliable designs  Design contracts help make specifications precise  Novel design separates access and storage mechanisms  Case study potentially useful for educational purposes

65. 26-Jan-2010 65 Future Work  Study more formal use of Schmid’s generalization methodology  Study use of function generalization methodology  Modify prototypes to match revised design  Redevelop prototypes to use Scala with generics instead of Java without generics  Rethink Externalization, perhaps using Strategy pattern  Study hot spots and build finer-grained component library  Adapt earlier work of students on AVL and B-Tree class libraries  Integrate into SoftwareInterfaces library  Develop instructional materials

66. 26-Jan-2010 66 Acknowledgements  Jingyi Wang for her work on prototype framework  Yi Liu and Jingyi Wang for their work on the papers  Wei Feng, Jian Hu, and Deep Sharma for their work on earlier table-related libraries  Sudharshan Vazhkudai, Jennifer Jie Xu, Vandana Thomas, Cuihua Zhang, Xiaobin Pang, and Ming Wei for work on other frameworks  Pallavi Tadepalli for collaboration on function generalization  Various reviewers including Bob Cook, Jennifer Jie Xu, Chuck Jenkins, and Pallavi Tadepalli  Acxiom Corporation for its support