1. A Warehouse and Reporting Architecture for Healthcare using Oracle Technologies [CON3926]
Nicholas Collins
Clinical Analytics and Informatics
24 September 2013

2. Topics About MD Anderson (and me!)
The Future of Cancer Treatment and Research
Oracle at MD Anderson
Our Warehouse and Reporting Architecture
Implementation Conclusions 2

3. 1
About MD Anderson 3

4. About MD Anderson
Non-profit cancer hospital and research institution, founded in as part of The University of Texas System
Named after Monroe Dunaway Anderson (a banker and cotton trader, not an MD)
“Making Cancer History” – our mission is to eradicate cancer
Consistently ranked as the #1 hospital for cancer care 4

5. About MD Anderson 5

6. About MD Anderson Over 19,000 employees, majority in the Houston area
Occupying over 20 buildings in the Texas Medical Center
The Texas Medical Center has over 50 member institutions, together over 100,000 employees 6

7. About Me
Began working at MD Anderson while an undergraduate at Rice University (across the street)
Currently in the Clinical Analytics and Informatics (CAI) Department, but before that was in HR Information Management (HRIM), working with PeopleSoft and other HR apps
While in HRIM, built a custom HR data warehouse and reporting system using a combination of Microsoft and Oracle technologies
After hours, a professional stage actor/director in Houston 7

8. The Future of Cancer Treatment and Research2
The Future of Cancer Treatment and Research 8

9. MD Anderson Moon Shots Program
“The Time is Now. Together we will end cancer.”
Target six cancers: Breast/Ovarian, Leukemia (AML/MDS & CLL), Melanoma, Lung, Prostate
Clear focus on the concept that the answer to curing cancer lies in both clinical and genomic data 9

10. MD Anderson Moon Shots Program 10

11. It’s in the Data! 11

12. MD Anderson Moon Shots Platforms
Massive Data Analytics – An infrastructure for complex analytics and clinical decision support using integrated patient information, including clinical and research data
Big Data – An Information Technology infrastructure/environment that enables centralization, integration and secured access of patient and research data and analytical results 12

13. It’s in the Genes! 13

14. MD Anderson Moon Shots Platforms
Clinical Genomics – Clinical gene sequencing infrastructure, including centralized bio-specimen repository and processing
Omics – Bioinformatics – A high-throughput infrastructure for generation and standardization of large-scale “omic” data, including genomics, proteomics and immune profiling
Adaptive Learning in Genomic Medicine – A framework for bringing clinical medicine and genomic research together to enable rapid learning to improve patient management using Clinical Genomics, Omics-Bioinformatics and Massive Data Analytics platforms within the Big Data environment 14

15. Genomics in the News 15

16. 3
Oracle at MD Anderson 16

17. Oracle Health Sciences Products at MD Anderson
Oracle Healthcare Data Warehouse Foundation (HDWF)
Oracle Healthcare Analytics Data Integration (OHADI)
Oracle TRC (Translational Research Center) Cohort Explorer
Oracle TRC Omics Data Bank (ODB) 17

18. Oracle Technology at MD Anderson
Oracle Database 11gR2
Oracle Exadata (x3)
Oracle Business Intelligence (OBIEE)
Oracle GoldenGate*
*Oracle GoldenGate was used to demonstrate replication capabilities in a significant POC, but has not been purchased or put into production. Informatica is commonly used at MD Anderson for data integration; ODI is not currently in use at the institution. 18

19. Oracle Healthcare Data Warehouse Foundation (HDWF)
HDM
HDI 19

20. Oracle Healthcare Analytics Data Integration (OHADI)
HDM
HDI
Integration code that maps from the interface tables (HDI) to the warehouse tables (HDM)
Available as either Informatica or ODI mappings 20

21. Oracle Cohort Explorer
CDM 21

22. Oracle Cohort Explorer22

23. Oracle Omics Data Bank (ODB)23

24. 24

25. Review of Oracle Health Sciences Products
Cohort Explorer
HDI
HDM
CDM
OHADI
ODB 25

26. Our Warehouse and Reporting Architecture4
Our Warehouse and Reporting Architecture 26

27. The MD Anderson FIRE Program
FIRE - Federated Institutional Reporting Environment
A program level initiative, with many projects and products involved, to provide a unified BI/Reporting solution for all of MD Anderson
Managed by the Clinical Analytics and Informatics (CAI) Department, part of Oracle SDP (Strategic Development Parnter) Program 27

28. FIRE Program Team Structure (Early Proposal)28

29. FIRE Program Team Structure (Current)29

30. First FIRE Release – Pharmacy Dashboard
Custom-built OBIEE Dashboard for orders data, pulling data from HDM, with HDI populated from the GE Centricity source
Dimensional model with orders as the core fact
Included a smaller pre-release of HR data for staff details and testing the FIRE Architecture
Purchased HDWF and OHADI in May 2012, aiming for a fall go-live for our first FIRE release 30

31. Pharmacy Dashboard (OBIEE)31

32. Unifying the Solution
Oracle Health Sciences GBU provides products that are a part of an overall solution, the rest is organization specific
We needed a way to effectively get data into HDWF and deliver it to custom Data Marts
Having a pre-built warehouse model helps with speed of delivery, but not necessarily source system mapping and integration 32

33. Architectural Concept
Bring all the data processing together on a single Oracle instance for performance benefits of local movement and transformation
Abstract across all commonalities and patterns to the largest extent possible, avoiding needless one-off solutions, use code generation and automation
Ideal candidate for a later “forklift” to Exadata 33

34. The FIRE Architecture 34

35. Source Systems at MD Anderson
Currently no centralized EHR solution in place, a best-of- breed model with many disparate source systems
Data currently brought together for patient-care clinical use in a single UI by a SOA-based custom .NET app called ClinicStation
In July 2013, the institution announced its intention to migrate to Epic’s EHR solution 35

36. The FIRE Architecture
HDI/HDM SR SI UI UD 36

37. SR (Staging Replica) Layer
Stores replicated data from source systems in the consolidated warehouse environment, provides buffer from sources and their technology, also allows custom indexing and partitioning if needed
Replication can be accomplished in a variety of ways, using a “bag of dirty tricks” to get the data into relational form in Oracle
GoldenGate proposed as standard tool to replicate in from relational sources, transparent gateway and Informatica/ODI are other options
Repliaction done at table level for consistency and ease of change 37

38. SI (Staging Interface) Layer
Pull data directly from the corresponding SR schema (i.e. the SI_CENTRICITY schema pulls data from the SR_CENTRICITY schema)
Contains views that match the target HDI tables, one-to-one, same column names and data types
Accomplishes selection of the appropriate data, and any necessary pre-transformation
In complex cases, can have materialized pre-processing data in tables or materialized views, in practice this meets the 80/20 rule 38

39. HDI Layer Oracle-defined HDWF interface tables
A “landing zone” schema, can be used as a Persistent Staging Area (PSA)
Conceptually, where you place unrefined and unvalidated data to be processed by OHADI before it goes into the HDM (warehouse tables)
Tables are designed to be insert-only (source-change dated) 39

40. HDM Layer Oracle-defined HDWF warehouse tables
HDM stands for Healtcare Data Model
Keys and Referential Integrity (RI) in place for this layer, but RI is disabled by default
Conceptually, where all your data is persistently stored, though reloads from HDI are possible
Can be configured for effective-dating or only current state 40

41. UI (User Interface) Layer
Similar in concept to the SI Layer, but pulling data from HDM for use in the UD layer
Contains views that match the target UD tables, one-to-one, same column names and data types
Accomplishes selection of the appropriate data, and any necessary pre-transformation (like SI)
In complex cases, can have materialized pre-processing data in tables or materialized views, in practice this meets the 80/20 rule (like SI) 41

42. UD (User Data Mart) Layer
Pull data directly from the corresponding UI schema (i.e. the UD_RX schema pulls data from the views in the UI_RX schema)
Contains the user-layer target tables that are used in the dimensional (star schema) models
Can be used directly by OBIEE or schemas can be replicated to a separate user database for isolated/off-loaded dashboard processing
Data-wise, it’s the end result of the warehouse pipeline 42

43. Data Movement 43

44. Data Movement (Planned)
There was a desire from our integration team to use Informatica for ETL because of experience base on the team, not much PL/SQL or ODI knowledge
Architecture proposed use of abstracted code generation via Informatica APIs, jointly used with the push-down optimization option for all non-OHADI internal data movement (i.e. SI to HDI, UI to UD) 44

45. Data Movement (Actual)
Our integration team initially indicated that code generation with Informatica (or other tools) could not be done on account of complexity, and that the push-down optimization option was too expensive
To demostrate the feasibility, I programmed a PL/SQL-based version of the code generation as proposed in the FIRE Architecture documentation, we used this code in the first release 45

46. Data Movement Code Generation46

47. PL/SQL Procedures for Code Generation
Procedure iv_tv_ip_gen(name_of_sv_view) for SI layer, generates objects for change detection and movement from SR to HDI
Procedure iv_uv_dv_ip_gen(name_of_sv_view) for UI layer, generates objects for change detection and movement from HDM to UD
All that is needed for generation is the SV view, which conforms to the HDI- based structure, data in certain standard HDI columns determine action
A benefit of the generated views is the ability to see what will happen during the next run, without actually running anything 47

48. Results/Next Steps
Had approximately three months to implement, process was difficult, but in the end everything worked and we went to production with the first FIRE release in November 2012
OHADI was slower than expected but got the job done, Informatica version used, but might be faster with ODI?
Integration team wanted a second chance to get code generation going for Informatica, and wanted more Informatica and less SQL and PL/SQL
Committee voted to try Informatica alternatives for the next release 48

49. Second FIRE Release – Moon Shot Analytics and Pharmacy Dashboard
In this release, there were four total projects under the FIRE Program:
(1) Second Pharmacy Release, (2) Cohort Explorer and ODB (Moon Shot Analytics), (3) Exadata Implementation for Cohort Explorer and ODB, and (4) OBIEE Infrastructure Upgrade
Beginning to use Omics Data Bank, data volumes required Exadata
License restrictions and no budget yet to put HDWF on Exadata 49

50. Second Release
Cohort Explorer
CDM
ODB 50

51. Architectural Changes51

52. Architectural Changes
Informatica Code Generation using Java to generate Informatica objects, so not using the PL/SQL code generation with SI and UI views for this release
Integration team wanted an instantiated SI Layer and UI Layer for Informatica-based code generation instead of views in the SI and UI Layer 52

53. Informatica Code Generation with Java53

54. Results
Successfully implemented all projects within six months, Cohort Explorer and ODB now live, along with the new Pharmacy Dashboard
Informatica code generation worked successfully, but perfomance issues surfaced, particularly with the new SI and UI instantiated data
The old Pharmacy code (views) was easy to change/update and the integration team did not want to convert old code to the new Informatica- based methodology, hence we have a hybrid model in place currently
Some of the Informatica code had to be abandoned close to go-live, replaced by quickly created SQL in views (materialized views) 54

55. More Results
Had to tinker a little bit with the Exadata install of ODB and CDM, ended up to drop all indexes and implement some materialized views for improved performance, now being incorporated into the tool via our SDP partnership
We noticed that Oracle was now using a view-based methodology for the CDM ETL, Informatica mappings use views off of HDM, differs from OHADI, more similar to our original architecture
Indexing, partitioning, and SQL tuning were especially necessary in working with our conventional HDWF environment, Exadata can help in the future
Aggregation was important to getting OBIEE to perform well with ROLAP 55

56. Exadata Implmentation for Cohort Explorer and Omics Data Bank
(Moon Shot Analytics)
CDM
ODB 56

57. Exadata Architecture Database Clients Administrator Database Servers
Infiniband Switch
Storage Servers 57

58. Rack Configurations (x3-2)
Full Rack – 8 DB Servers, 14 Storage Servers
Half Rack – 4 DB Servers, 7 Storage Servers
Quarter Rack – 2 DB Servers, 3 Storage Servers
Eighth Rack – 2 DB Servers*, 3 Storage Servers**
* The eighth rack’s DB servers each have one processor disabled via software.
** The eighth rack’s storage servers have half the drives and half the flash cards. 58

59. Exadata x3-2 Capabilities59

60. Exadata Database Features
Smart Scans (Query Offloading)
Storage Indexes
Hybrid Columnar Compression
Exadata Smart Flash Cache
These are software-based “Exdata-only” features. 60

61. Standard Database I/O Exadata I/O
Conventional Storage
select …
I/O Request
Client
Database Server
Exadata I/O
Exadata Storage Servers
Smart Scan Request
select …
Client
Exadata DB Server 61

62. CDM/ODB Implementation - Exadata Equipment Purchased
Development/Test
Environment
Production
Environment 62

63. Exadata Pre-installation Activities
Oracle Exadata Readiness Checklist
Prepare Network and Power Connections
Shipping needs to align with Sun technician arrival
SFP Modules – be sure to order if needed
Training from Enkitec 63

64. MD Anderson CAI “War Room” Exadata Implementation Team
January 2013 64
Photo shown courtesy of Mr. Robert Jeffries, Project Manager

65. 65

66. Exadata Validation Scripts
Scripts to verify functionality
Eighth Rack scripts expected Quarter Rack
Most documentation still for x2 equipment, most likely updated by now 66

67. Exadata Storage Allocation
Crucial to plan storage allocation in advance
Recommended DATA/RECO disk groups: OLTP – 60%/40%, DW 80%/20%, we did a 70%/30% split
DB instances share these disk groups per rack
Implemented DBFS (Oracle Database File System) for use with the ODB Loaders, omics data files are large, particularly genomic reference data 67

68. Exadata Instance Creation
Performed after storage allocation
We installed two separate instances on production rack, four separate instances on dev/test rack
All instances are clustered via RAC, ODB and CDM are in the same database instance
Instance creation went fairly quickly for us, but would not have been the case for a consolidation project 68

69. Additional Components
Backup – IBM Tivoli (non-Oracle)
Anti-virus Software (non-Oracle)
Oracle Enterprise Manager 12c
Oracle Platinum Gateway 69

70. Oracle Enterprise Manager 12c70

71. 71

72. Exadata Implementation Lessons Learned
Single points of contact are ideal (Oracle, Enkitec, Internal Departments)
Early engagement project planning important
“War Room” concept very effective
Some documentation for new products hard to find (i.e. x3, OEM 12c)
Enkitec training was fantastic, but probably should have happened earlier than the delivery, we were on an accelerated implementation schedule 72

73. GoldenGate POC at MD Anderson – July 2012
Assess GoldenGate’s viability toward MD Anderson Use Cases requiring Heterogeneous Data Replication, Flexible Data Deployment and Continuous Availability
Determine ease-of-configuration, deployment, manageability and reliability of GoldenGate as implemented within the use cases
“Should GoldenGate handle these use cases convincingly, the POC will be considered as successful” 73

74. GoldenGate POC Use Cases
Use Case #1 - Heterogeneous Data Replication from multiple Source Databases, namely, PICIS (CareSuite), OR Manager (Security, Surgery, Interface DBs), and Sybase (RADDATA) to an Oracle 11g R2 Target Database (staging area for HCM Data Warehouse)
Use Case #2 - Heterogeneous Data Replication from multiple Source Databases, namely PICIS (CareSuite), OR Manager (Security, Surgery, Interface DBs), and Sybase (RADDATA) to a SQL Server 2008 R2 Target Database (Operations Reporting)
Use Case #3 - Flexible Data Deployment topology to merge activity data from multiple Source Tables (Sybase) to a Single SQL Server Target Table (Operations Reporting Database)
Use Case #4 - Architecture capable of handling schema differences across platforms (Audit Reporting Columns added at Oracle Target Database) 74

75. GoldenGate POC Architecture75

76. GoldenGate POC Results76

77. Implementation Conclusions5
Implementation Conclusions 77

78. Implementation Conclusions
The HDWF, OHADI, Cohort Explorer, and Omics Data Bank products helped us deliver a lot of functionality very quickly, in only one year – our president was very impressed with Cohort Explorer when coupled with ODB, calling it a “game changer”
Having a warehouse model in place helps you avoid a lot of the headaches and time that could be lost to developing your own intermediary models and related enforcement of standards
OHADI gives you a good starting point for the HDI to HDM ETL so you can effectively generate your RI relationships and manage exceptions
Taking some time to think about our overall architecture and try out some new techniques paid off in the long run, still important to deliver functionality of course 78

79. Implementation Conclusions
The HDM tables cover a large swath of clinical concepts, but sometimes things might not fit perfectly, working with Oracle as an SDP has its advantages, as does understanding how to customize the model
OHADI is supposed to be the step in the process where data is cleansed and validated, but a lot of customization would be required to do so thoroughly, Oracle will be working to improve this portion of the product, until then you can cleanse pre-HDI if OOTB functionality is not there yet
The Informatica version of OHADI had some performance issues on our environments, but I would predict the ODI (Oracle Data Integrator) version would run faster
Having a good vocabulary/terminology approach in place will help tremendously with your implementation of code systems in the model, Oracle is beginning to integrate OHADI with HLI 79

80. Implementation Conclusions
The Cohort Explorer application aims to deliver the exact functionality that will be needed in cancer research – merging the clinical and genomic data, but the application still needs some more maturity and better use of HDWF structures, we do not want the tail to wag the dog
Omics Data Bank helps to centralize a variety of genomics data using set standards from the academic and scientific community, this concept seems to work well in practice
The complexity of the queries on ODB and CDM were sometimes challenging from a performance perspective – it is clear that Exadata helps tremendously in this area and we would not have been nearly as successful without it
Be sure you have a knowledgeable SME resource for these Oracle technologies to assist you through implementation, they would be difficult to implement on your own 80

81. MD Anderson Future Steps
Get HDWF on Exadata, already ordered two new eighth racks
Upgrade Cohort Explorer and ODB to newer versions, when released
Focus on architectural refinements/changes for the FIRE Architecture
Need to get better code system infrastructure in place, will take work, but worth it in the end – it is unclear how use of Epic/HLI will affect us
Beginning NLP pipeline for unstructured data, currently prototyping using IBM ICA platform, but possibly investigate use of Big Data Appliance and/or Hadoop? Semantic technologies?
Prepare for Epic as source, hopefully acquire GoldenGate eventually too 81

82. Collins Axioms – Parting Maxims
Abstraction over commonalities is key in a world driven increasingly by “Big Data”
High-end performance is crucial for BI data delivery, every bit counts
Always mind the “Mythical Man-Month,” the more knowledge and capability a single individual has, the better - streamline your teams for effective agility and delivery, understand the business, beware of “design by committee”
You will never get it right the first time, expect changes and have flexibility to adapt quickly
Virtualization, in-memory database objects, and real-time data are gaining momentum in the realm of BI and warehousing, learn to embrace them 82

83. Questions? 83