Designing a Flexible Framework for a Table Abstraction H. Conrad Cunningham 1 Yi Liu 2 Jingyi Wang 3 1 University of Mississippi 2 South Dakota State University 3 Acxiom Corporation

08-Nov Software Interfaces Project ( ) Context: development of an instructional data and file structures library (for Java in CSci 211)  artifacts for study of good design techniques  system for use, extension, and modification Motivation: study techniques for  presenting important methods to students (frameworks, software design patterns, design by contract, etc.)  unifying related file and data structures in framework

08-Nov Table Abstract Data Type (ADT)  Collection of records o each a finite sequence of data fields o key field value uniquely identifies record within collection  Operations on collection and on record using key  Several different implementations possible key1data1 key2data2 key3data3 key4data4

08-Nov Table ADT Operations  Insert new record  Delete existing record given key  Update existing record  Retrieve existing record given key  Get number of records  Query whether contains given key  Query whether empty  Query whether full

08-Nov Software Framework  Generic application allowing creation of members of family of related programs  Reusable design expressed as set of abstract classes and way they collaborate  Common and variable aspects known as frozen spots and hot spots Framework Framework library User-supplied code (application specific) Hot spots Frozen spots

08-Nov Software Design Contracts  Preconditions for correct use of operation  Postconditions for correct result of operation  Invariant conditions for correct implementation of class Insert record operation pre: record is valid and not already in table post: record now in table Invariant for table all records are valid, no duplicate keys

08-Nov Software Design Patterns  Describe recurring design problems arising in specific contexts  Present well-proven generic solution schemes  Describe solution’s components and their responsibilities and relationships  To use: o select pattern that fits problem o structure solution to follow pattern

08-Nov Table Framework Requirements (from 2001 paper) 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

08-Nov Hot Spot Analysis of Requirements 1. Provide Table ADT functionality for client- defined records and keys 2. Support many implementations of access and storage mechanisms 3. Separate key-based access mechanism from storage mechanism 4. Present coherent abstractions with well-defined interfaces 5. Use software design patterns and design contracts

08-Nov Table Hot Spot Analysis Frozen spot 1. Provide Table ADT functionality Hot spots 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients) 3. Variability in external representation of record state 4. Variability in access (i.e, indexing) mechanism 5. Variability in storage mechanism Hot spots #1 and #2 not completely independent Hot spot #3 quite subtle Desire to separate concerns (i.e., the hot spots)

08-Nov Layered Architecture Pattern  Distinct groups of services  Hierarchical arrangement of groups into layers  Layer implemented with services of layer below  Enables independent implementation of hot spots in layers Client Layer Access Layer Storage Layer

08-Nov Applying Layered Architecture Pattern Client Layer – Hot spots #1 and #2 o includes client application programs o implements record and key abstractions o uses layer below to store and retrieve records Access Layer – Frozen spot and Hot spot #4 o includes table implementations o provides key-based access to records for layer above o calls back to record and key implementations in level above o uses physical storage in layer below Storage Layer – Hot spot #5 o storage managers o provides physical storage for records o calls back to record implementations in layers above

08-Nov Client Layer of Architecture Encapsulate hot spots #1 and #2 1. Variability in keys (defined by clients) 2. Variability in records (defined by clients)

08-Nov Client Layer Design Challenges o enabling Access Layer to access client- defined records and keys o avoiding unnecessary programming to use common data types Pattern o Interface – applied independently for hot spots #1 (keys) and #2 (records)

08-Nov Client Layer Interfaces (1 of 2) Hot spot #1 – Variability in keys  Must be able to (at a minimum) compare with total ordering  Choose to use Comparable interface from Java API for convenience Comparable interface for keys o int compareTo(Object key) compares object with argument

08-Nov Client Layer Interfaces (2 of 2) Hot spot #2 – Variability in records  Must be able to extract keys for comparison  Choose to define an appropriate new interface Keyed interface for records o Comparable getKey() extracts key from record

08-Nov Client Layer Model Abstract predicates – for use in specification of this and other layers (not necessarily implemented) o boolean isValidKey(Object key) yields true for valid key values o boolean isValidRec(Object rec) yields true for valid record values

08-Nov Client Layer Design Contracts Hot spot #1 Comparable int compareTo(Object key) int compareTo(Object key) Pre: isValidKey(this) && isValidKey(key) Pre: isValidKey(this) && isValidKey(key) Post: result == (if this < key then -1 Post: result == (if this < key then -1 else if this == key then 0 else if this == key then 0 else 1) else 1) Hot spot #2 Keyed Comparable getKey() Pre: isValidRec(this) Pre: isValidRec(this) Post: ( result == attribute key of this record) Post: ( result == attribute key of this record) && isValidKey(result) && isValidKey(result)

08-Nov Client Layer Challenges Met?  Enabling Access Layer to access client- defined records and keys o records and keys implementations encapsulated within objects that implement Keyed and Comparable  Avoiding unnecessary programming to use common data types o reuse Comparable from Java API

08-Nov Access Layer of Architecture Encapsulate hot spot #4: Variability in access (i.e, indexing) mechanism

08-Nov Access Layer Design Challenges o enabling use of client-defined keys and records o enabling diverse implementations of indexing structure o enabling use of diverse storage structures Pattern o Interface – applied for hot spot #4 (access) and frozen spot (Table ADT)

08-Nov Access Layer Interface Table void insert(Keyed r) inserts r into table void delete(Comparable key) removes record with key void update(Keyed r) changes record with same key Keyed retrieve(Comparable key) returns record with key int getSize() returns size of table boolean containsKey(Comparable key) searches for key boolean isEmpty() checks whether table is empty boolean isFull() checks whether table is full o for unbounded, always returns false

08-Nov Access Layer Model Partial function table :: Comparable  Keyed o represents abstract table state o consider this: number of entries finite or infinite? If finite, bounded or unbounded? If infinite, countable or not? o #table in postcondition denotes table before operation o use as either function or set of key-record pairs Abstract predicates (to capture environmental assumptions) o isValidKey(Object) and isValidRec(Object) defined in Client Layer to identify valid keys and records o isStorable(Object) defined in Storage Layer to identify records that can be stored

08-Nov Table Design Contract (1 of 5) Invariant (  k, r : r == table(k) : isValidKey(k) && isValidRec(r) && isStorable(r) && k == r.getKey() ) isStorable(r) && k == r.getKey() )

08-Nov Table Design Contract (2 of 5) void insert(Keyed r) inserts r into table Pre: isValidRec(r) && isStorable(r) && !containsKey(r.getKey())&& !isFull() Post: table == #table  {(r.getKey(),r)} void delete(Comparable key) removes record with key from table Pre: isValidKey(key) && containsKey(key) Post: table == #table - {(key,#table(key))} Question: Does this allow table be initialized nonempty?

08-Nov Table Design Contract (3 of 5) void update(Keyed r) changes record with same key Pre: isValidRec(r) && isStorable(r) && containsKey(r.getKey()) Post: table == (#table - {(r.getKey(),#table(r.getKey()))} )  {(r.getKey(),r)} Keyed retrieve(Comparable key) returns record with key Pre: isValidKey(key) && containsKey(key) Post: result == #table(r.getKey())

08-Nov Table Design Contract (4 of 5) int getSize() returns size of table Pre: true Post: result == cardinality(#table) Note: If size not finite, need way to represent infinity boolean containsKey(Comparable key) searches table for key Pre: isValidKey(key) Post: result == defined(#table(key))

08-Nov Table Design Contract (5 of 5) boolean isEmpty() checks whether table is empty Pre: true Post: result == (#table =  ) boolean isFull() checks whether table is full – for unbounded, always returns false Pre : true Post: result == (#table implementation has no free space to store record) no free space to store record)

08-Nov Client/Access Layer Interactions  Client calls Access Layer class implementing Table interface  Access calls back to Client implementations of Keyed and Comparable interfaces

08-Nov Access Layer Challenges Met?  Enabling use of client-defined keys and records o callbacks to Comparable and Keyed abstractions which hide the implementation details  Enabling diverse implementations of the table o careful design of table interface semantics using design by contract  Enabling use of diverse storage mechanisms o calls to Storage Layer interfaces

08-Nov Storage Layer of Architecture Encapsulate hot spot #5: Variability in storage mechanism

08-Nov Storage Layer Design Challenges o supporting client-defined records o supporting diverse table implementations in Access Layer (simple indexes, hashing, balanced trees, etc.) o allowing diverse physical media (in-memory, on- disk, etc.) o enabling persistence of table, including access layer o decoupling implementations as much as possible Patterns o Bridge o Proxy

08-Nov Bridge Pattern  Decouple “interface” from “implementation” o table from storage in this case  Allow them to vary independently o plug any storage mechanism into table Table RecordStore Simple Indexed File Hashed File Slotted File Store Vector File uses

08-Nov Proxy Pattern  Transparently manage services of target object o isolate Table implementation from nature/location of record slots in RecordStore implementation  Introduce proxy object as surrogate for target handle Table RecordSlot RecordStore

08-Nov Storage Layer Interfaces RecordStore o operations to allocate and deallocate storage slots RecordSlot o operations to get and set records in slots o operations to get handle and containing RecordStore

08-Nov Storage Layer Model (1 of 2) Partial function store :: int  Object o represents abstract RecordStore state Set Handles  int, NULLHANDLE  Handles Set alloc  Handles o represents set of allocated slot handles Set unalloc == Handles - alloc o represents set of unallocated slot handles Abstract predicate isStorable(Object) o depends on storage mechanism

08-Nov Storage Layer Model (2 of 2) Invariant: (  h,r : r == store(h) : isStorable(r)) && (  h :: h  alloc == defined(store(h)))

08-Nov RecordStore Interface RecordSlot newSlot() allocates a new record slot RecordSlot getSlot(int handle) rebuilds record slot using given handle void releaseSlot(RecordSlot slot) deallocates record slot

08-Nov RecordStore Design Contract (1 of 2) RecordSlot newSlot() allocates a new record slot, but it might be done in a lazy fashion Pre: true Post: result.getContainer() == this_RecordStore && result.getRecord() == NULLRECORD && result.getHandle()  #alloc && result.getHandle()  alloc  {NULLHANDLE} Note: NULLRECORD built according to Null Object pattern RecordSlot getSlot(int handle) rebuilds record slot using given handle Pre: handle  alloc Post: result.getContainer() == this_RecordStore && result.getRecord() == #store(handle) && result.getHandle() == handle

08-Nov RecordStore Design Contract (2 of 2) void releaseSlot(RecordSlot slot) deallocates record slot Pre: slot.getHandle()  alloc  {NULLHANDLE} && slot.getContainer() == this_RecordStore Post: alloc == #alloc - {slot.getHandle()} && store == #store – store == #store – {(slot.getHandle(),slot.getRecord())} {(slot.getHandle(),slot.getRecord())}

08-Nov RecordSlot Interface void setRecord(Object rec) stores rec in this slot allocation of handle done here or already done by getSlot allocation of handle done here or already done by getSlot Object getRecord() returns record stored in this slot int getHandle() returns handle of this slot RecordStore getContainer() returns reference to RecordStore holding this slot boolean isEmpty() determines whether this slot empty

08-Nov RecordSlot Model Each slot has two attributes:  reference to RecordStore to which this RecordSlot belongs, which is immutable  handle for the associated physical storage slot in the RecordStore (as constrained in RecordStore ) Invariant: getHandle()  alloc  {NULLHANDLE}

08-Nov RecordSlot Design Contract (1 of 3) void setRecord(Object rec) stores rec in this slot o allocation of handle done here or already done by getSlot() Pre: isStorable(rec) Post: Let h == getHandle(): (h  #alloc  store == (#store - (h  #alloc  store == (#store - {(h,#store(h))})  {(h,rec)} ) && (h = NULLHANDLE  (h = NULLHANDLE  (  g : g  #unalloc : alloc == #alloc  {g} && alloc == #alloc  {g} && store = #store  {(g,rec)} )) store = #store  {(g,rec)} ))

08-Nov RecordSlot Design Contract (2 of 3) Object getRecord() returns record stored in this slot Pre: true Post: Let h == getHandle() : ( h  #alloc  result == #store(h)) && ( h == NULLHANDLE  result == NULLRECORD ) int getHandle() returns handle of this slot Pre: true Post: result == handle associated with this slot

08-Nov RecordSlot Design Contract (3 of 3) RecordStore getContainer() returns reference to RecordStore holding this slot Pre: true Post: result == RecordStore associated with this slot boolean isEmpty() determines whether this slot empty Pre: true Post: result == (getHandle() == NULLHANDLE || record associated with slot is NULLRECORD)

08-Nov Storage Layer Challenges Met? (1 of 2)  Supporting client-defined records o deferred to Externalization Module  Supporting diverse table implementations in Access Layer o careful design of RecordStore and RecordSlot abstractions to have sufficient functionality  Allowing diverse physical media o careful design of RecordStore abstraction to hide media, implementable in many ways, o partly deferred to Externalization Module

08-Nov Storage Layer Challenges Met? (2 of 2)  Decoupling implementations as much as possible o use of RecordSlot, handle, and Externalization Module  Enabling persistence of Table, including Access Layer o store RecordStore identifier and handles

08-Nov Access/Storage Layer Interactions  Access Layer Table calls RecordStore in Storage Layer to get RecordSlot object  Access Layer calls RecordSlot to store and retrieve its records  If needed, RecordSlot calls back to Record implementation in Access Layer o interface Record defined in Externalization Module

08-Nov Externalization Module of Architecture Encapsulate Hot Spot #3: Variability in external representation of record state

08-Nov Externalization Module Design Challenges (e.g., to support Storage Layer) o supporting client-defined records o allowing diverse physical media o decoupling implementations Pattern: Interface

08-Nov Externalization Module Interface Record o void writeRecord(DataOutput out) writes the client’s record to stream out o void readRecord(DataInput in) reads the client’s record from stream in o int getLength() returns number of bytes in external representation of client’s record (e.g., written by writeRecord)

08-Nov Externalization Module Contract (1 of 2) void writeRecord(DataOutput out) writes this record to stream out Pre: true Post: suffix of stream out == this record’s state encoded as byte sequence this record’s state encoded as byte sequence void readRecord(DataInput in) reads this record from stream in Pre: true Post: this record’s state == prefix of stream in decoded from byte sequence prefix of stream in decoded from byte sequence

08-Nov Externalization Module Contract (2 of 2) int getLength() returns number of bytes in external representation of this record (e.g., written by writeRecord) Pre: true Post: result == number of bytes in external representation of this record representation of this record State Restoration Property: If, for some record, writeRecord() is followed by readRecord() of the same byte sequence, the observable state of the record will be unchanged.

08-Nov Externalization Challenges Met?  Supporting client-defined records o records needing to be externalized implement Record interface  Allowing diverse physical media o Record interface implementation provides low-level physical representation of record storable on most media  Decoupling implementations o Record decouples use from implementation

08-Nov Externalization Module Interactions  Record used by Storage Layer may be defined by either layer above o hold one Client Layer Keyed record o hold multiple, or fractional, Client Layer records managed by Access Layer (e.g, multiway tree nodes)  Storage Layer calls back to Record implementation in layer above o implementation in Access Layer might call back to implementations in Client Layer

08-Nov Abstraction Usage Relationships Table KeyedComparable RecordStoreRecordSlot Record Access Layer Storage Layer Client LayerExternalization Module

08-Nov Other Design Patterns Used  Null Object (for NULLRECORD in Storage Layer)  Iterator o extended Table operations o query mechanism o utility classes  Template Method  Decorator  Strategy

08-Nov Evolving Frameworks Patterns  Generalizing from three examples  Whitebox framework  Component library o Wang prototype: two Table s and three RecordStore s  Hot spots  Pluggable objects  Fine-grained objects  Blackbox framework

08-Nov Conclusions  Hot spot analysis reveals generalization opportunities  Design patterns offer systematic way to discover reliable designs  Design contracts help make specifications precise  Novel design by separating access and storage mechanisms  Case study potentially useful for educational purposes

08-Nov Future Work  Study more formal use of Schmid’s generalization methodology  Study use of functional generalization methodology  Modify prototypes to match revised design  Rethink Externalization, perhaps using Strategy pattern  Study hot spots and build finer-grained component library  Adapt earlier work of students on AVL and B-Tree class libraries  Integrate into SoftwareInterfaces library  Develop instructional materials

08-Nov Acknowledgements  Jingyi Wang for her work on the prototype framework  Wei Feng, Jian Hu, and Deep Sharma for their work on earlier table-related libraries  Sudharshan Vazhkudai, Jennifer Jie Xu, Vandana Thomas, Cuihua Zhang, Xiaobin Pang, and Ming Wei for work on other frameworks  Pallavi Tadepalli for collaboration on function generalization  Various reviewers including Bob Cook and Jennifer Jie Xu  Acxiom Corporation for its support