1. Designing a Flexible Framework for a Table Abstraction H. Conrad Cunningham 1 Yi Liu 2 Jingyi Wang 3 1 University of Mississippi 2 South Dakota State University 3 Acxiom Corporation

2. 08-Nov-2007 2 Software Interfaces Project (1996-2000) Context: development of an instructional data and file structures library (for Java in CSci 211)  artifacts for study of good design techniques  system for use, extension, and modification Motivation: study techniques for  presenting important methods to students (frameworks, software design patterns, design by contract, etc.)  unifying related file and data structures in framework

3. 08-Nov-2007 3 Table Abstract Data Type (ADT)  Collection of records o each a finite sequence of data fields o key field value uniquely identifies record within collection  Operations on collection and on record using key  Several different implementations possible key1data1 key2data2 key3data3 key4data4

4. 08-Nov-2007 4 Table ADT Operations  Insert new record  Delete existing record given key  Update existing record  Retrieve existing record given key  Get number of records  Query whether contains given key  Query whether empty  Query whether full

5. 08-Nov-2007 5 Software Framework  Generic application allowing creation of members of family of related programs  Reusable design expressed as set of abstract classes and way they collaborate  Common and variable aspects known as frozen spots and hot spots Framework Framework library User-supplied code (application specific) Hot spots Frozen spots

6. 08-Nov-2007 6 Software Design Contracts  Preconditions for correct use of operation  Postconditions for correct result of operation  Invariant conditions for correct implementation of class Insert record operation pre: record is valid and not already in table post: record now in table Invariant for table all records are valid, no duplicate keys

7. 08-Nov-2007 7 Software Design Patterns  Describe recurring design problems arising in specific contexts  Present well-proven generic solution schemes  Describe solution’s components and their responsibilities and relationships  To use: o select pattern that fits problem o structure solution to follow pattern

8. 08-Nov-2007 8 Table Framework Requirements (from 2001 paper) 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

9. 08-Nov-2007 9 Hot Spot Analysis of Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

10. 08-Nov-2007 10 Table Hot Spot Analysis Frozen spot 1. Provide Table ADT functionality Hot spots 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients) 3. Variability in external representation of record state 4. Variability in access (i.e, indexing) mechanism 5. Variability in storage mechanism Hot spots #1 and #2 not completely independent Hot spot #3 quite subtle Desire to separate concerns (i.e., the hot spots)

11. 08-Nov-2007 11 Layered Architecture Pattern  Distinct groups of services  Hierarchical arrangement of groups into layers  Layer implemented with services of layer below  Enables independent implementation of hot spots in layers Client Layer Access Layer Storage Layer

12. 08-Nov-2007 12 Applying Layered Architecture Pattern Client Layer – Hot spots #1 and #2 o includes client application programs o implements record and key abstractions o uses layer below to store and retrieve records Access Layer – Frozen spot and Hot spot #4 o includes table implementations o provides key-based access to records for layer above o calls back to record and key implementations in level above o uses physical storage in layer below Storage Layer – Hot spot #5 o storage managers o provides physical storage for records o calls back to record implementations in layers above

13. 08-Nov-2007 13 Client Layer of Architecture Encapsulate hot spots #1 and #2 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients)

14. 08-Nov-2007 14 Client Layer Design Challenges o enabling Access Layer to access client- defined records and keys o avoiding unnecessary programming to use common data types Pattern o Interface – applied independently for hot spots #1 (keys) and #2 (records)

15. 08-Nov-2007 15 Client Layer Interfaces (1 of 2) Hot spot #1 – Variability in keys  Must be able to (at a minimum) compare with total ordering  Choose to use Comparable interface from Java API for convenience Comparable interface for keys o int compareTo(Object key) compares object with argument

16. 08-Nov-2007 16 Client Layer Interfaces (2 of 2) Hot spot #2 – Variability in records  Must be able to extract keys for comparison  Choose to define an appropriate new interface Keyed interface for records o Comparable getKey() extracts key from record

17. 08-Nov-2007 17 Client Layer Model Abstract predicates – for use in specification of this and other layers (not necessarily implemented) o boolean isValidKey(Object key) yields true for valid key values o boolean isValidRec(Object rec) yields true for valid record values

18. 08-Nov-2007 18 Client Layer Design Contracts Hot spot #1 Comparable int compareTo(Object key) int compareTo(Object key) Pre: isValidKey(this) && isValidKey(key) Pre: isValidKey(this) && isValidKey(key) Post: result == (if this < key then -1 Post: result == (if this < key then -1 else if this == key then 0 else if this == key then 0 else 1) else 1) Hot spot #2 Keyed Comparable getKey() Pre: isValidRec(this) Pre: isValidRec(this) Post: ( result == attribute key of this record) Post: ( result == attribute key of this record) && isValidKey(result) && isValidKey(result)

19. 08-Nov-2007 19 Client Layer Challenges Met?  Enabling Access Layer to access client- defined records and keys o records and keys implementations encapsulated within objects that implement Keyed and Comparable  Avoiding unnecessary programming to use common data types o reuse Comparable from Java API

20. 08-Nov-2007 20 Access Layer of Architecture Encapsulate hot spot #4: Variability in access (i.e, indexing) mechanism

21. 08-Nov-2007 21 Access Layer Design Challenges o enabling use of client-defined keys and records o enabling diverse implementations of indexing structure o enabling use of diverse storage structures Pattern o Interface – applied for hot spot #4 (access) and frozen spot (Table ADT)

22. 08-Nov-2007 22 Access Layer Interface Table void insert(Keyed r) inserts r into table void delete(Comparable key) removes record with key void update(Keyed r) changes record with same key Keyed retrieve(Comparable key) returns record with key int getSize() returns size of table boolean containsKey(Comparable key) searches for key boolean isEmpty() checks whether table is empty boolean isFull() checks whether table is full o for unbounded, always returns false

23. 08-Nov-2007 23 Access Layer Model Partial function table :: Comparable  Keyed o represents abstract table state o consider this: number of entries finite or infinite? If finite, bounded or unbounded? If infinite, countable or not? o #table in postcondition denotes table before operation o use as either function or set of key-record pairs Abstract predicates (to capture environmental assumptions) o isValidKey(Object) and isValidRec(Object) defined in Client Layer to identify valid keys and records o isStorable(Object) defined in Storage Layer to identify records that can be stored

24. 08-Nov-2007 24 Table Design Contract (1 of 5) Invariant (  k, r : r == table(k) : isValidKey(k) && isValidRec(r) && isStorable(r) && k == r.getKey() ) isStorable(r) && k == r.getKey() )

25. 08-Nov-2007 25 Table Design Contract (2 of 5) void insert(Keyed r) inserts r into table Pre: isValidRec(r) && isStorable(r) && !containsKey(r.getKey())&& !isFull() Post: table == #table  {(r.getKey(),r)} void delete(Comparable key) removes record with key from table Pre: isValidKey(key) && containsKey(key) Post: table == #table - {(key,#table(key))} Question: Does this allow table be initialized nonempty?

26. 08-Nov-2007 26 Table Design Contract (3 of 5) void update(Keyed r) changes record with same key Pre: isValidRec(r) && isStorable(r) && containsKey(r.getKey()) Post: table == (#table - {(r.getKey(),#table(r.getKey()))} )  {(r.getKey(),r)} Keyed retrieve(Comparable key) returns record with key Pre: isValidKey(key) && containsKey(key) Post: result == #table(r.getKey())

27. 08-Nov-2007 27 Table Design Contract (4 of 5) int getSize() returns size of table Pre: true Post: result == cardinality(#table) Note: If size not finite, need way to represent infinity boolean containsKey(Comparable key) searches table for key Pre: isValidKey(key) Post: result == defined(#table(key))

28. 08-Nov-2007 28 Table Design Contract (5 of 5) boolean isEmpty() checks whether table is empty Pre: true Post: result == (#table =  ) boolean isFull() checks whether table is full – for unbounded, always returns false Pre : true Post: result == (#table implementation has no free space to store record) no free space to store record)

29. 08-Nov-2007 29 Client/Access Layer Interactions  Client calls Access Layer class implementing Table interface  Access calls back to Client implementations of Keyed and Comparable interfaces

30. 08-Nov-2007 30 Access Layer Challenges Met?  Enabling use of client-defined keys and records o callbacks to Comparable and Keyed abstractions which hide the implementation details  Enabling diverse implementations of the table o careful design of table interface semantics using design by contract  Enabling use of diverse storage mechanisms o calls to Storage Layer interfaces

31. 08-Nov-2007 31 Storage Layer of Architecture Encapsulate hot spot #5: Variability in storage mechanism

32. 08-Nov-2007 32 Storage Layer Design Challenges o supporting client-defined records o supporting diverse table implementations in Access Layer (simple indexes, hashing, balanced trees, etc.) o allowing diverse physical media (in-memory, on- disk, etc.) o enabling persistence of table, including access layer o decoupling implementations as much as possible Patterns o Bridge o Proxy

33. 08-Nov-2007 33 Bridge Pattern  Decouple “interface” from “implementation” o table from storage in this case  Allow them to vary independently o plug any storage mechanism into table Table RecordStore Simple Indexed File Hashed File Slotted File Store Vector File uses

34. 08-Nov-2007 34 Proxy Pattern  Transparently manage services of target object o isolate Table implementation from nature/location of record slots in RecordStore implementation  Introduce proxy object as surrogate for target handle Table RecordSlot RecordStore

35. 08-Nov-2007 35 Storage Layer Interfaces RecordStore o operations to allocate and deallocate storage slots RecordSlot o operations to get and set records in slots o operations to get handle and containing RecordStore

36. 08-Nov-2007 36 Storage Layer Model (1 of 2) Partial function store :: int  Object o represents abstract RecordStore state Set Handles  int, NULLHANDLE  Handles Set alloc  Handles o represents set of allocated slot handles Set unalloc == Handles - alloc o represents set of unallocated slot handles Abstract predicate isStorable(Object) o depends on storage mechanism

37. 08-Nov-2007 37 Storage Layer Model (2 of 2) Invariant: (  h,r : r == store(h) : isStorable(r)) && (  h :: h  alloc == defined(store(h)))

38. 08-Nov-2007 38 RecordStore Interface RecordSlot newSlot() allocates a new record slot RecordSlot getSlot(int handle) rebuilds record slot using given handle void releaseSlot(RecordSlot slot) deallocates record slot

39. 08-Nov-2007 39 RecordStore Design Contract (1 of 2) RecordSlot newSlot() allocates a new record slot, but it might be done in a lazy fashion Pre: true Post: result.getContainer() == this_RecordStore && result.getRecord() == NULLRECORD && result.getHandle()  #alloc && result.getHandle()  alloc  {NULLHANDLE} Note: NULLRECORD built according to Null Object pattern RecordSlot getSlot(int handle) rebuilds record slot using given handle Pre: handle  alloc Post: result.getContainer() == this_RecordStore && result.getRecord() == #store(handle) && result.getHandle() == handle

40. 08-Nov-2007 40 RecordStore Design Contract (2 of 2) void releaseSlot(RecordSlot slot) deallocates record slot Pre: slot.getHandle()  alloc  {NULLHANDLE} && slot.getContainer() == this_RecordStore Post: alloc == #alloc - {slot.getHandle()} && store == #store – store == #store – {(slot.getHandle(),slot.getRecord())} {(slot.getHandle(),slot.getRecord())}

41. 08-Nov-2007 41 RecordSlot Interface void setRecord(Object rec) stores rec in this slot allocation of handle done here or already done by getSlot allocation of handle done here or already done by getSlot Object getRecord() returns record stored in this slot int getHandle() returns handle of this slot RecordStore getContainer() returns reference to RecordStore holding this slot boolean isEmpty() determines whether this slot empty

42. 08-Nov-2007 42 RecordSlot Model Each slot has two attributes:  reference to RecordStore to which this RecordSlot belongs, which is immutable  handle for the associated physical storage slot in the RecordStore (as constrained in RecordStore ) Invariant: getHandle()  alloc  {NULLHANDLE}

43. 08-Nov-2007 43 RecordSlot Design Contract (1 of 3) void setRecord(Object rec) stores rec in this slot o allocation of handle done here or already done by getSlot() Pre: isStorable(rec) Post: Let h == getHandle(): (h  #alloc  store == (#store - (h  #alloc  store == (#store - {(h,#store(h))})  {(h,rec)} ) && (h = NULLHANDLE  (h = NULLHANDLE  (  g : g  #unalloc : alloc == #alloc  {g} && alloc == #alloc  {g} && store = #store  {(g,rec)} )) store = #store  {(g,rec)} ))

44. 08-Nov-2007 44 RecordSlot Design Contract (2 of 3) Object getRecord() returns record stored in this slot Pre: true Post: Let h == getHandle() : ( h  #alloc  result == #store(h)) && ( h == NULLHANDLE  result == NULLRECORD ) int getHandle() returns handle of this slot Pre: true Post: result == handle associated with this slot

45. 08-Nov-2007 45 RecordSlot Design Contract (3 of 3) RecordStore getContainer() returns reference to RecordStore holding this slot Pre: true Post: result == RecordStore associated with this slot boolean isEmpty() determines whether this slot empty Pre: true Post: result == (getHandle() == NULLHANDLE || record associated with slot is NULLRECORD)

46. 08-Nov-2007 46 Storage Layer Challenges Met? (1 of 2)  Supporting client-defined records o deferred to Externalization Module  Supporting diverse table implementations in Access Layer o careful design of RecordStore and RecordSlot abstractions to have sufficient functionality  Allowing diverse physical media o careful design of RecordStore abstraction to hide media, implementable in many ways, o partly deferred to Externalization Module

47. 08-Nov-2007 47 Storage Layer Challenges Met? (2 of 2)  Decoupling implementations as much as possible o use of RecordSlot, handle, and Externalization Module  Enabling persistence of Table, including Access Layer o store RecordStore identifier and handles

48. 08-Nov-2007 48 Access/Storage Layer Interactions  Access Layer Table calls RecordStore in Storage Layer to get RecordSlot object  Access Layer calls RecordSlot to store and retrieve its records  If needed, RecordSlot calls back to Record implementation in Access Layer o interface Record defined in Externalization Module

49. 08-Nov-2007 49 Externalization Module of Architecture Encapsulate Hot Spot #3: Variability in external representation of record state

50. 08-Nov-2007 50 Externalization Module Design Challenges (e.g., to support Storage Layer) o supporting client-defined records o allowing diverse physical media o decoupling implementations Pattern: Interface

51. 08-Nov-2007 51 Externalization Module Interface Record o void writeRecord(DataOutput out) writes the client’s record to stream out o void readRecord(DataInput in) reads the client’s record from stream in o int getLength() returns number of bytes in external representation of client’s record (e.g., written by writeRecord)

52. 08-Nov-2007 52 Externalization Module Contract (1 of 2) void writeRecord(DataOutput out) writes this record to stream out Pre: true Post: suffix of stream out == this record’s state encoded as byte sequence this record’s state encoded as byte sequence void readRecord(DataInput in) reads this record from stream in Pre: true Post: this record’s state == prefix of stream in decoded from byte sequence prefix of stream in decoded from byte sequence

53. 08-Nov-2007 53 Externalization Module Contract (2 of 2) int getLength() returns number of bytes in external representation of this record (e.g., written by writeRecord) Pre: true Post: result == number of bytes in external representation of this record representation of this record State Restoration Property: If, for some record, writeRecord() is followed by readRecord() of the same byte sequence, the observable state of the record will be unchanged.

54. 08-Nov-2007 54 Externalization Challenges Met?  Supporting client-defined records o records needing to be externalized implement Record interface  Allowing diverse physical media o Record interface implementation provides low-level physical representation of record storable on most media  Decoupling implementations o Record decouples use from implementation

55. 08-Nov-2007 55 Externalization Module Interactions  Record used by Storage Layer may be defined by either layer above o hold one Client Layer Keyed record o hold multiple, or fractional, Client Layer records managed by Access Layer (e.g, multiway tree nodes)  Storage Layer calls back to Record implementation in layer above o implementation in Access Layer might call back to implementations in Client Layer

56. 08-Nov-2007 56 Abstraction Usage Relationships Table KeyedComparable RecordStoreRecordSlot Record Access Layer Storage Layer Client LayerExternalization Module

57. 08-Nov-2007 57 Other Design Patterns Used  Null Object (for NULLRECORD in Storage Layer)  Iterator o extended Table operations o query mechanism o utility classes  Template Method  Decorator  Strategy

58. 08-Nov-2007 58 Evolving Frameworks Patterns  Generalizing from three examples  Whitebox framework  Component library o Wang prototype: two Table s and three RecordStore s  Hot spots  Pluggable objects  Fine-grained objects  Blackbox framework

59. 08-Nov-2007 59 Conclusions  Hot spot analysis reveals generalization opportunities  Design patterns offer systematic way to discover reliable designs  Design contracts help make specifications precise  Novel design by separating access and storage mechanisms  Case study potentially useful for educational purposes

60. 08-Nov-2007 60 Future Work  Study more formal use of Schmid’s generalization methodology  Study use of functional generalization methodology  Modify prototypes to match revised design  Rethink Externalization, perhaps using Strategy pattern  Study hot spots and build finer-grained component library  Adapt earlier work of students on AVL and B-Tree class libraries  Integrate into SoftwareInterfaces library  Develop instructional materials

61. 08-Nov-2007 61 Acknowledgements  Jingyi Wang for her work on the prototype framework  Wei Feng, Jian Hu, and Deep Sharma for their work on earlier table-related libraries  Sudharshan Vazhkudai, Jennifer Jie Xu, Vandana Thomas, Cuihua Zhang, Xiaobin Pang, and Ming Wei for work on other frameworks  Pallavi Tadepalli for collaboration on function generalization  Various reviewers including Bob Cook and Jennifer Jie Xu  Acxiom Corporation for its support