A State Perspective Mentoring Conference New Orleans, LA 2/28/2005 RCRAInfo Network Exchange

State Perspective - Overview  New process – everyone involved is learning – New technologies – New business processes – New relationships  Multiple parties – no true Project Lead or Authority – Competing priorities – Resource restrictions  Beta states should expect more changes/rework as schema is tested, modified, and finalized  Positive Note - this is progress. Future projects should benefit from our efforts.

RCRA Implementation Challenges – the Flow  Implementer of Record (IOR)  Primary Keys - Staying in Synch  Business Data and Composite Keys  Look-Up Code Maintenance  Data Conversion  Gaps in Data Validation (for historical data)  Developing additional RCRA exchange process flows – Data sharing and synchronization services – Get Handler ID; Get x by Handler

RCRA Implementation Challenges – the Project  Transfer of Knowledge – Business team and Technical team – Contractors and Internal Resources – EPA and State team  Project Phases with schema fluctuations require flexible relationships with resources (internal and external staff)  Grant constraints create tension with project goals – Satisfying time and financial constraints could lead to tension between broad goals of network exchange and immediate scope decisions  Disparate Parties – Identify appropriate expert resources / decision makers – Communication standards are evolving

RCRA Implementation Challenges – the Project  What are the affects of Schema changes? – Optional fields, element names, namespace – Structural modifications and mandatory fields  Identify phases that cause schema changes – determine navigation path – Drafting schema, Translator Design and Build, State Mapping, State/Pilot Testing, Production – Timing of activities and parties involved in the activities can affect schema stability  Other Factors – Source and Target system modifications need to be monitored

MDEQ TEMPO-RCRAInfo Data Extraction

RCRAInfo Mapping Challenges  Corresponding Events to Units  Permitting Actions – time to complete  Event Capture  Tracking Primary Keys  Implementing Transaction Codes  The Payload Operation Decision

Corresponding Events to Units

Permitting Actions – Time to Complete  Permits require long periods of time to complete.  Interim periods required data submissions  Traditional method of permit “complete” could not be used to indicate submissions.  What would be the trigger?  What would be the data?

Permitting Actions – Time to Complete Basics  Trigger: Completion of RCRA Event  Data: All data available linked to the event – …with conditions Conditions  Event Capture: Data added or changed events  Date criteria: – Last date file generation executed – Current date – Comfort Zone

Permitting Actions – Time to Complete Comfort Zone  Current Date: 10/22/2004  Last Run Date: 10/15/2004  Comfort Zone: 3 days

MDEQ RCRAInfo Event Capture  Track data events in source system – Insert new data – Update previously existing data – Delete previously existing data  Event, table, primary keys and “data over time”  Components – Database Triggers – Event Capture table structure – Event Capture interpreter

Event Capture Table Structure

MDEQ RCRAInfo Event Capture - Interpreter  Identify Event as occurring on table.  Group table and column events since last execution  Determine “net” value for data element(s)  Store “conclusion” in memory to generate file  Loop  Clear Event Capture tables using comfort zone

Waste Codes and Event Capture  Some Primary Keys of RCRAInfo tables require distinct event logic  Tracking the change of data and composing the correct net transaction can be difficult  Example: RCRAInfo Waste Codes PK – Waste Code, Waste Code Owner, Handler ID, Unit Sequence Number, Unit Detail Sequence Number – No waste code sequence number

Waste Codes and Event Capture C t1 B t1 A t1 A t2 Q t2

Tracking Primary Keys  Method of tracking/generating correct Primary Key (PK) for data in RCRAInfo  Capture PK structure of RCRAInfo and map to PK structure of source system  Challenge: – Learning the RCRAInfo model – Keeping in synch – sequence numbers – Similar challenges and structure in look-up code context

Implementing Transaction Codes

 Valid Transaction Codes – A (Add/Update) – D (Delete) – X (Submitted for Context)  Add and Update are handled in the same manner - the translator will interpret an Add as an Update when necessary – If PK already exists, add is interpreted as update – Record is replaced/updated, not the element (send all data for record) – Potential lack of feedback on incorrect data synchronization

Implementing Transaction Codes  Nesting a delete under update transaction codes  Cascade Delete ramifications – Sending delete on the parent table row only – “Child” deletes will trigger critical file submission errors  Cannot delete co-implemented records (transactional or full-replace) – Keep in mind with cascade deletions

Implementing Transaction Codes: De-Normalized Data

 How does one delete a link between an event and a unit?  How does one completely delete a unit?  Applying the Delete transaction at the child and root levels has different effects

Miscellaneous Challenges  Establish SOPs - new source system features  Data configuration and conversion resources  Facilities that exist in multiple counties  Address data – multiple lines to single elements  Converting geographic coordinate data to Decimal Degrees  Owner Operator Data – Polling entity types – Active/Inactive – Type Codes

Discussion