Designing a Flexible Framework for a Table Abstraction H. Conrad Cunningham 1 Yi Liu 2 Jingyi Wang 3 1 University of Mississippi 2 South Dakota State University 3 Acxiom Corporation

Hot Spots, Patterns, and Contracts: An Exploration of Software Framework Design H. Conrad Cunningham 1 Yi Liu 2 1 University of Mississippi 2 South Dakota State University

26-Jan Hot Spots References  Coplien, J., Hoffman, D., Weiss, D. Commonality and variability in software engineering, IEEE Software, vol. 15, no. 6, pp ,  Schmid, H. A., Framework design by systematic generalization, In: Fayad, M. E., Schmidt, D. C., Johnson, R. E., editors, Building Application Frameworks: Object- Oriented Foundations of Framework Design, Wiley, pp , 1999.

26-Jan Patterns References  Gamma, R., Helm, R., Johnson, R., Vlissides, J., Design Patterns: Elements of Reusable Object-Oriented Software, Addison Wesley,  Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M., Pattern-Oriented Software Architecture: A System of Patterns, Wiley, 1996.

26-Jan Contracts References  Meyer, B., Object-Oriented Software Construction, 2nd Edition, Prentice Hall PTR, 1997.

26-Jan Source Papers  Cunningham, H. C., Liu, Y., Wang. J., Designing a flexible framework for a table abstraction, Chapter 13 in: Chan, Y., Talburt, J., Talley, T., editors, Data Engineering: Mining, Information, and Intelligence, Springer, pp ,  Cunningham, H. C., Wang, J., Building a Layered framework for the table abstraction, In Proceedings of the 16th ACM Symposium on Applied Computing, pp , 2001.

26-Jan Software Interfaces Project ( ) Context: development of an instructional data and file structures library (for Java in CS 3)  artifacts for study of good design techniques  system for use, extension, and modification Motivation: study techniques for  presenting important methods to students (frameworks, software design patterns, design by contract, etc.)  unifying related file and data structures in framework

26-Jan Table Abstract Data Type (ADT)  Collection of records o each a finite sequence of data fields o key field value uniquely identifies record within collection  Operations on collection and on record using key  Several different implementations possible key1data1 key2data2 key3data3 key4data4

26-Jan Table ADT Operations  Insert new record  Delete existing record given key  Update existing record  Retrieve existing record given key  Get number of records  Query whether contains given key  Query whether empty  Query whether full

26-Jan Software Framework  Generic application allowing creation of members of family of related programs  Reusable design expressed as set of abstract classes and way they collaborate  Common and variable aspects known as frozen spots and hot spots Framework Framework library User-supplied code (application specific) Hot spots Frozen spots

26-Jan Software Design Patterns  Describe recurring design problems arising in specific contexts  Present well-proven generic solution schemes  Describe solution’s components and their responsibilities and relationships  To use: o select pattern that fits problem o structure solution to follow pattern

26-Jan Software Design Contracts  Preconditions for correct use of operation  Postconditions for correct result of operation  Invariant conditions for correct implementation of class Insert record operation pre: record is valid and not already in table post: record now in table Invariant for table all records are valid, no duplicate keys

26-Jan Initial Table Framework Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

26-Jan Hot Spot Analysis of Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

26-Jan Table Hot Spot Analysis Frozen spot 1. Table ADT functionality Hot spots 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients) 3. Variability in external representation of record state 4. Variability in access (i.e, indexing) mechanism 5. Variability in storage mechanism Hot spots #1 and #2 not completely independent Hot spot #3 quite subtle Desire to separate concerns (i.e., the hot spots)

26-Jan Layered Architecture Pattern  Distinct groups of services  Hierarchical arrangement of groups into layers  Layer implemented with services of layer below  Enables independent implementation of hot spots in layers Client Layer Access Layer Storage Layer

26-Jan Applying Layered Architecture Pattern Client Layer – Hot spots #1 and #2 o includes client application programs o implements record and key abstractions o uses layer below to store and retrieve records Access Layer – Frozen spot and Hot spot #4 o includes table implementations o provides key-based access to records for layer above o calls back to record and key implementations in level above o uses physical storage in layer below Storage Layer – Hot spot #5 o storage managers o provides physical storage for records o calls back to record implementations in layers above

26-Jan Client Layer of Architecture Encapsulate hot spots #1 and #2 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients)

26-Jan Client Layer Design Challenges o enabling Access Layer to access client- defined records and keys o avoiding unnecessary programming to use common data types Pattern o Interface – applied independently for hot spots #1 (keys) and #2 (records)

26-Jan Client Layer Interfaces (1 of 2) Hot spot #1 – Variability in keys  Must be able to (at a minimum) compare with total ordering  Choose to use Comparable interface from Java API for convenience Comparable interface for keys o int compareTo(Object key) compares object with argument

26-Jan Client Layer Interfaces (2 of 2) Hot spot #2 – Variability in records  Must be able to extract keys for comparison  Choose to define an appropriate new interface Keyed interface for records o Comparable getKey() extracts key from record

26-Jan Client Layer Model Abstract predicates – for use in specification of this and other layers (not necessarily implemented) o boolean isValidKey(Object key) yields true for valid key values o boolean isValidRec(Object rec) yields true for valid record values

26-Jan Client Layer Design Contracts Hot spot #1 Comparable int compareTo(Object key) int compareTo(Object key) Pre: isValidKey(this) && isValidKey(key) Pre: isValidKey(this) && isValidKey(key) Post: result == (if this < key then -1 Post: result == (if this < key then -1 else if this == key then 0 else if this == key then 0 else 1) else 1) Hot spot #2 Keyed Comparable getKey() Pre: isValidRec(this) Pre: isValidRec(this) Post: ( result == this.key ) Post: ( result == this.key ) && isValidKey(result) && isValidKey(result)

26-Jan Client Layer Challenges Met?  Enabling Access Layer to access client- defined records and keys o records and keys implementations encapsulated within objects that implement Keyed and Comparable  Avoiding unnecessary programming to use common data types o reuse Comparable from Java API

26-Jan Access Layer of Architecture Provide frozen spot Table ADT functionality Encapsulate hot spot #4: Variability in access (i.e, indexing) mechanism

26-Jan Access Layer Design Challenges o enabling use of client-defined keys and records o enabling diverse implementations of indexing structure o enabling use of diverse storage structures Pattern o Interface – applied for frozen spot (Table ADT) and hot spot #4 (access)

26-Jan Access Layer Interface Table void insert(Keyed r) inserts r into table void delete(Comparable key) removes record with key void update(Keyed r) changes record with same key Keyed retrieve(Comparable key) returns record with key int getSize() returns size of table boolean containsKey(Comparable key) searches for key boolean isEmpty() checks whether table is empty boolean isFull() checks whether table is full o for unbounded, always returns false

26-Jan Access Layer Model Partial function table :: Comparable  Keyed o represents abstract table state o consider this: number of entries finite or infinite? If finite, bounded or unbounded? If infinite, countable or not? o #table in postcondition denotes table before operation o use as either function or set of key-record pairs Abstract predicates (to capture environmental assumptions) o isValidKey(Object) and isValidRec(Object) defined in Client Layer to identify valid keys and records o isStorable(Object) defined in Storage Layer to identify records that can be stored

26-Jan Table Design Contract (1 of 5) Invariant (  k, r : r == table(k) : isValidRec(r) && isStorable(r) && k == r.getKey() ) && k == r.getKey() ) All stored keys and records in the table are valid and capable of being stored on the chosen external device, and the records can be assessed by their keys.

26-Jan Table Design Contract (2 of 5) void insert(Keyed r) inserts r into table Pre: isValidRec(r) && isStorable(r) && !containsKey(r.getKey())&& !isFull() Post: table == #table  {(r.getKey(),r)} void delete(Comparable key) removes record with key from table Pre: isValidKey(key) && containsKey(key) Post: table == #table - {(key,#table(key))} Question: Does this allow table be initialized nonempty?

26-Jan Table Design Contract (3 of 5) void update(Keyed r) changes record with same key Pre: isValidRec(r) && isStorable(r) && containsKey(r.getKey()) Post: table == (#table - {(r.getKey(),#table(r.getKey()))} )  {(r.getKey(),r)} Keyed retrieve(Comparable key) returns record with key Pre: isValidKey(key) && containsKey(key) Post: result == #table(r.getKey())

26-Jan Table Design Contract (4 of 5) int getSize() returns size of table Pre: true Post: result == cardinality(#table) Note: If size not finite, need way to represent infinity boolean containsKey(Comparable key) searches table for key Pre: isValidKey(key) Post: result == defined(#table(key))

26-Jan Table Design Contract (5 of 5) boolean isEmpty() checks whether table is empty Pre: true Post: result == (#table =  ) boolean isFull() checks whether table is full – for unbounded, always returns false Pre : true Post: result == (#table implementation has no free space to store record) no free space to store record)

26-Jan Client/Access Layer Interactions  Client calls Access Layer class implementing Table interface  Access calls back to Client implementations of Keyed and Comparable interfaces

26-Jan Access Layer Challenges Met?  Enabling use of client-defined keys and records o callbacks to Comparable and Keyed abstractions that hide the implementation details  Enabling diverse implementations of the table o careful design of table interface semantics using design by contract  Enabling use of diverse storage mechanisms o calls to Storage Layer interfaces

26-Jan Storage Layer of Architecture Encapsulate hot spot #5: Variability in storage mechanism

26-Jan Storage Layer Design Challenges o supporting client-defined records o supporting diverse table implementations in Access Layer (simple indexes, hashing, balanced trees, etc.) o allowing diverse physical media (in-memory, on- disk, etc.) o enabling persistence of table, including access layer o decoupling implementations as much as possible Patterns o Bridge o Proxy

26-Jan Bridge Pattern  Decouple “interface” from “implementation” o table from storage in this case  Allow them to vary independently o plug any storage mechanism into table Table RecordStore Simple Indexed File Hashed File Slotted File Store Vector File uses

26-Jan Proxy Pattern  Transparently manage services of target object o isolate Table implementation from nature/location of record slots in RecordStore implementation  Introduce proxy object as surrogate for target handle Table RecordSlot RecordStore

26-Jan Storage Layer Interfaces RecordStore o operations to allocate and deallocate storage slots RecordSlot o operations to get and set records in slots o operations to get handle and containing RecordStore

26-Jan Storage Layer Model (1 of 2) Partial function store :: int  Object o represents abstract RecordStore state Set Handles  int, NULLHANDLE  Handles Set alloc  Handles o represents set of allocated slot handles Set unalloc == Handles - alloc o represents set of unallocated slot handles Abstract predicate isStorable(Object) o depends on storage mechanism

26-Jan Storage Layer Model (2 of 2) Invariant: (  h,r : r == store(h) : isStorable(r)) && (  h :: h  alloc == defined(store(h))) All records in the store are capable of being stored on the selected medium and the stored records can be access by their handles.

26-Jan RecordStore Interface RecordSlot newSlot() allocates a new record slot RecordSlot getSlot(int handle) rebuilds record slot using given handle void releaseSlot(RecordSlot slot) deallocates record slot

26-Jan RecordStore Design Contract (1 of 2) RecordSlot newSlot() allocates a new record slot, but it might be done in a lazy fashion Pre: true Post: result.getContainer() == this && result.getRecord() == NULLRECORD && result.getHandle()  #alloc && result.getHandle()  alloc  {NULLHANDLE} Note: NULLRECORD built according to Null Object pattern RecordSlot getSlot(int handle) rebuilds record slot using given handle Pre: handle  alloc Post: result.getContainer() == this && result.getRecord() == #store(handle) && result.getHandle() == handle

26-Jan RecordStore Design Contract (2 of 2) void releaseSlot(RecordSlot slot) deallocates record slot Pre: slot.getHandle()  alloc  {NULLHANDLE} && slot.getContainer() == this Post: alloc == #alloc - {slot.getHandle()} && store == #store – store == #store – {(slot.getHandle(),slot.getRecord())} {(slot.getHandle(),slot.getRecord())}

26-Jan RecordSlot Interface void setRecord(Object rec) stores rec in this slot allocation of handle done here or already done by getSlot allocation of handle done here or already done by getSlot Object getRecord() returns record stored in this slot int getHandle() returns handle of this slot RecordStore getContainer() returns reference to RecordStore holding this slot boolean isEmpty() determines whether this slot empty

26-Jan RecordSlot Model Each slot has two attributes:  reference to RecordStore to which this RecordSlot belongs, which is immutable  handle for the associated physical storage slot in the RecordStore (as constrained in RecordStore ) Invariant: getHandle()  alloc  {NULLHANDLE} The handle of a RecordSlot object denotes a slot of the store that has been allocated unless it has the value NULLHANDLE.

26-Jan RecordSlot Design Contract (1 of 3) void setRecord(Object rec) stores rec in this slot o allocation of handle done here or already done by getSlot() Pre: isStorable(rec) Post: Let h == getHandle(): (h  #alloc  store == (#store - (h  #alloc  store == (#store - {(h,#store(h))})  {(h,rec)} ) && (h = NULLHANDLE  (h = NULLHANDLE  (  g : g  #unalloc : alloc == #alloc  {g} && alloc == #alloc  {g} && store = #store  {(g,rec)} )) store = #store  {(g,rec)} ))

26-Jan RecordSlot Design Contract (2 of 3) Object getRecord() returns record stored in this slot Pre: true Post: Let h == getHandle() : ( h  #alloc  result == #store(h)) && ( h == NULLHANDLE  result == NULLRECORD ) int getHandle() returns handle of this slot Pre: true Post: result == this.handle

26-Jan RecordSlot Design Contract (3 of 3) RecordStore getContainer() returns reference to RecordStore holding this slot Pre: true Post: result == this.container boolean isEmpty() determines whether this slot empty Pre: true Post: result == (getHandle() == NULLHANDLE || getRecord() == NULLRECORD)

26-Jan Storage Layer Challenges Met? (1 of 2)  Supporting client-defined records o deferred to Externalization Module  Supporting diverse table implementations in Access Layer o careful design of RecordStore and RecordSlot abstractions to have sufficient functionality  Allowing diverse physical media o careful design of RecordStore abstraction to hide media, implementable in many ways, o partly deferred to Externalization Module

26-Jan Storage Layer Challenges Met? (2 of 2)  Decoupling implementations as much as possible o use of RecordSlot, handle, and Externalization Module  Enabling persistence of Table, including Access Layer o store RecordStore identifier and handles

26-Jan Access/Storage Layer Interactions  Access Layer Table calls RecordStore in Storage Layer to get RecordSlot object  Access Layer calls RecordSlot to store and retrieve its records  If needed, RecordSlot calls back to Record implementation in Access Layer o interface Record defined in Externalization Module

26-Jan Externalization Module of Architecture Encapsulate Hot Spot #3: Variability in external representation of record state

26-Jan Externalization Module Design Challenges (e.g., to support Storage Layer) o supporting client-defined records o allowing diverse physical media o decoupling implementations Pattern: Interface

26-Jan Externalization Module Interface Record o void writeRecord(DataOutput out) writes the client’s record to stream out o void readRecord(DataInput in) reads the client’s record from stream in o int getLength() returns number of bytes in external representation of client’s record (e.g., written by writeRecord)

26-Jan Externalization Module Contract (1 of 2) void writeRecord(DataOutput out) writes this record to stream out Pre: true Post: suffix of stream out == this record’s state encoded as byte sequence this record’s state encoded as byte sequence void readRecord(DataInput in) reads this record from stream in Pre: true Post: this record’s state == prefix of stream in decoded from byte sequence prefix of stream in decoded from byte sequence

26-Jan Externalization Module Contract (2 of 2) int getLength() returns number of bytes in external representation of this record (e.g., written by writeRecord) Pre: true Post: result == number of bytes in external representation of this record representation of this record State Restoration Property: If, for some record, writeRecord() is followed by readRecord() of the same byte sequence, the observable state of the record will be unchanged.

26-Jan Externalization Challenges Met?  Supporting client-defined records o records needing to be externalized implement Record interface  Allowing diverse physical media o Record interface implementation provides low-level physical representation of record storable on most media  Decoupling implementations o Record decouples use from implementation

26-Jan Externalization Module Interactions  Record used by Storage Layer may be defined by either layer above o hold one Client Layer Keyed record o hold multiple, or fractional, Client Layer records managed by Access Layer (e.g, multiway tree nodes)  Storage Layer calls back to Record implementation in layer above o implementation in Access Layer might call back to implementations in Client Layer

26-Jan Abstraction Usage Relationships Table KeyedComparable RecordStoreRecordSlot Record Access Layer Storage Layer Client LayerExternalization Module

26-Jan Other Design Patterns Used  Null Object (for NULLRECORD in Storage Layer)  Iterator o extended Table operations o query mechanism o utility classes  Template Method  Decorator  Strategy

26-Jan Evolving Frameworks Patterns  Generalizing from three examples  Whitebox framework  Component library o Wang prototype: two Table s and three RecordStore s  Hot spots  Pluggable objects  Fine-grained objects  Blackbox framework

26-Jan Conclusions  Hot spot analysis reveals generalization opportunities  Design patterns offer systematic way to discover reliable designs  Design contracts help make specifications precise  Novel design separates access and storage mechanisms  Case study potentially useful for educational purposes

26-Jan Future Work  Study more formal use of Schmid’s generalization methodology  Study use of function generalization methodology  Modify prototypes to match revised design  Redevelop prototypes to use Scala with generics instead of Java without generics  Rethink Externalization, perhaps using Strategy pattern  Study hot spots and build finer-grained component library  Adapt earlier work of students on AVL and B-Tree class libraries  Integrate into SoftwareInterfaces library  Develop instructional materials

26-Jan Acknowledgements  Jingyi Wang for her work on prototype framework  Yi Liu and Jingyi Wang for their work on the papers  Wei Feng, Jian Hu, and Deep Sharma for their work on earlier table-related libraries  Sudharshan Vazhkudai, Jennifer Jie Xu, Vandana Thomas, Cuihua Zhang, Xiaobin Pang, and Ming Wei for work on other frameworks  Pallavi Tadepalli for collaboration on function generalization  Various reviewers including Bob Cook, Jennifer Jie Xu, Chuck Jenkins, and Pallavi Tadepalli  Acxiom Corporation for its support