Presentation on theme: "Wrangling With Oracle Ray Murray Sirius Users Group June 2001."— Presentation transcript:
Wrangling With Oracle Ray Murray Sirius Users Group June 2001
Introduction How can OS/390 Oracle Benefit Wrangler Supplement Model 204 Complement Model 204 Replace Model 204 Tangible Benefits? Organizational Benefits? Intangible Benefits?
Complementing Model 204 How can OS/390 Oracle Complement M204 Standard DBMS Interfacing with Power User tools MS Office MS Query Access, Excel VBA Applications Maintain a subset of Wrangler Data Interfacing with common Web environments Cold Fusion...
Supplementing Model 204 Scaleable to Other Environments UNIX (Most Flavors) Linux NT Enhance Wrangler Integration with other related applications Replicate related Oracle database on Mainframe True Replication Export from UNIX - Import to OS/390
Replacing Model 204 Can OS/390 Oracle Host the entire Wrangler database? Does living on the mainframe have advantages not found in the our other Oracle environments? Database Conversion approach and effort Ability to re-use parts of Application Software
Questions Can Wrangler data be ported to Oracle? What are the storage usage differences? What are the performance differences? What is life with OS 390 Oracle like? Tuning, Administration, Support In house Vendor What other hurdles might there be?
Try it out Approach - II Re-Attempt to Port Database to Oracle Second iteration using lessons learned Database Design Issues More De-normalization Database Loading Retrievals Coding Performance
Current Database Single Major Record type (avg.. 500 bytes) Unique System Generated Primary Key (PK) 35 OCC 1 mandatory fields 58 Mult Occ fields holding discrete values Color, Make, Model … 26 Mult Occ fields of range values Cost, Premium, Weight … 137 Optional single occ fields - descriptive info
Current Database Substantial indexing (68 fields) Two NR fields 41 ORD CHAR PK is ORD CHAR 11 of 58 Mult Occ discrete fields are ORD CHAR 25 ORD NUM 19 of the Mult occ range fields are ORD NUM 2/3 Billion records - across 75 files
Oracle Test Database Minimally Normalized Port 2 Tables Base table (I) contains 152 columns 63 columns are indexed Child table (O) contains 19 columns replaces many Mult Occ M204 fields 15 columns are indexed No Reference or Join tables defined Didnt define columns not used in M204 file sample Sirius FUNLOAD FSTATS report very useful here First try 7-table port too inefficient
Database Porting Issues Dates/Time pairs Combined in Oracle Numeric values Varchar or Number? Text Concatenated a multiply occ M204 field Invisible fields Oracle indexing is very flexible, but..
Database Preparation Model 204 test database Extracted 3 million production DB records Funload UAI job ran in 3 minutes Created 3234 Cyl M204 file in 10.1 minutes Sirius LAI job loaded M204 file in 46 minutes Loaded data and built all indexes
Oracle Database Preparation Extraction of data from M204 database Prepared Sequential files to load I and O M204 User Language extract job took 67 minutes FUEL would have been much faster Prepared SQL*Loader Positional scripts M204-Centric approach to preparing data for Oracle is most straightforward due to power of User Language and Funload/FUEL tools Added 3234 Cyl dataset to Oracle database VSAM job took 28.5 minutes to format it
Loading Oracle Test Database Loaded I table in 27 minutes (direct=true) 3 Million rows Loaded O table in 23 minutes(direct=true) 5,449,864 rows Created 63 I table indexes in 540 minutes Created 15 O table indexes in 66 minutes Total 656 minutes to build this heavily indexed Oracle test database Best numbers from several runs
Space Usage M204 file Asize=4, Bhighpg=230769, Csize=2300, Dpgsused=56102 Total: 1788.4 M space used Oracle I table 1228.8 M O table 358.4 M I indexes 1867.1 M O indexes 685.7 M Total: 4140 M space used (231% x M204)
Query Comparisons Oracle Queries Run via SQL*Plus running under TSO Statistics reported from SET TIMING ON Oracle Expert product guidelines used to tune Oracle M204 Queries Run in an M204 online test region - large buffer pool Statistics reported from TIME REQUEST Both M204 and Oracle running on same machine using same type of disk drives Identical query results required
Timing Comparisons Run time Oracle Elapsed time of 0.02.15.55 means 0 hours, 2 minutes, 15 seconds, 550 milliseconds M204 CNCT (connect time) is reported in seconds. Above Oracle time would be the same as CNCT=136 CPU time Oracle CPU time is reported in seconds. CPU of 20.25 is twenty and 1/4 second. M204 CPU is reported in milli-seconds. 20.25 Oracle seconds is the same as CPU=20250
Query Scenario 1 How many claims were filed having collision damage (CD) between 9000 and 9150 dollars? liability damage (LD) between 3000 and 4000 dollars? injury amount (INJ) between 200 and 300 dollars?
Query 1 - M204 User Language: BEGIN CLAIMS.FD: IN CLAIMS FPC RECTYPE = CLAIM CD IS IN RANGE FROM 9000 TO 9150 LD IS IN RANGE FROM 3000 TO 4000 INJ IS IN RANGE FROM 200 TO 300 END Answer: 9155 CNCT=4, CPU=922, DKRD=381, RQTM=1681
Query 1 - Oracle SQL*Plus (O table only, no join) Select Count(*) from O where CD between 9000 and 9150 and LD between 3000 and 4000 and INJ between 200 and 300; Answer: 11004 Elapsed=0:00:03:58, CPU=2.14 remember: Oracle reported CPU=2.14 is like M204 reported CPU of 2140
Query 1 - completed Join added to match M204 answer Select count(distinct I.PK) from I, O where CD between 9000 and 9150 and LD between 3000 and 4000 and INJ between 200 and 300 and I.PK = O.PK; Answer: 9155 (matches M204) Elapsed=0.00:42:02 and CPU=5.65
Query 1 - narrowed ranges LD between 9100 and 9110 CD between 3500 and 3600 INJ between 200 and 300 Answer: 38 from both systems Oracle Elapsed=0:00:00.53 CPU=.25 secs M204 CNCT=1, RQTM=416, CPU=340 ms
Query 1 - Widened Ranges LD between 5000 and 12000 CD between 2000 and 5000 INJ between 200 and 300 Answer: 215959 I rows Oracle Elapsed 0:07:17.06 Cpu 146.20 M204 Cnct=30, Cpu=16446, Dkrd=4754, Rqtm=25252
Query 2 - Simple Count Oracle - Answer: 2,999,995 Select count(*) from I; Elapsed 0.02:15.55 CPU 20.25 M204 Begin FPC End CNCT=2, DKRD=127, RQTM=411, DKPR=251
Query 3 - Range Query Oracle - Answer: 2,410,583 Select count(*) where mydate between 970705 and 970725; Elapsed: 0.00.26.02 CPU 8.82 M204 Begin FPC mydate is in range from 970705 to 970725 end M204 - cnct=1, cpu=25, dkpr=816, rqtm=543
Query 4 - Multiple criteria Oracle Select count(*) from I where mydate between 07-JUL-97 and 25-JUL-97 and make = AQ and model = BC and ratinga between -600 and 1500 and ratingb between -10000 and 4000; Answer: 861 Elapsed 0:01:10.87 CPU 22.28
Query 4 - M204 version M204 Begin FPC mydate is in range from 970705 to 970725 make = AQ and model = BC ratinga is in range from -600 to +1500 ratingb is in range from -10000 to 4000 End Answer: 861 cnct=3, cpu=303, dkrd=1171, rqtm=3391
Query 5 - Scanning rows Oracle - Answer: 685 Select count(*) from I where make = AQ and mydate between 07-JUL-97 and 25-JUL-97 and ratingc like 235% Elapsed: 0:02:33.48 Cpu 26.51 Scanned through 445904 rows M204 equivalent (ratingc like 235*) CNCT=10, CPU=2637, RQTM=3094, DIRRCD=445904, DKPR=127506 (many pages in buffer pool) CNCT=100, CPU=8072, DKRD=62201, RQTM=100528, DKPR=127506 (no pages in buffer pool)
Query 6 - Is Present Scan Select same 445904 rows/records to scan Look for presence of a column/field Answer: 235997 Oracle - mynamefield is not null Elapsed 0.02.39.31 CPU 25.47 M204 - mynamefield is present CPU=8785, CNCT=10 (yes - its true), RQTM=9997, DKPR=563314 (reused buffers) CPU=14306, CNCT=129, DKRD=62201, RQTM=129608, DKPR=563314 (fresh pages)
Query 7 - Extracting data Output 14 fields from 445904 rows in a columnar format Both extracts output 1100 tracks of data Identical DCB for output files Oracle (Using SQL*Plus) Elapsed: 0:06:20.81Cpu: 118.54 M204 User Language CNCT=222, CPU=45228, DKRD=62196, RQTM=201628 Sirius FUEL via $FUNLOAD Total Time: 57.557 CPU=9.875
Query 8 - Count of each value Oracle Select make,count(make) from I group by make; Elapsed 0:02:39.76 cpu=118.54 M204 Make.frv: frv make print value in make.frv tab... FPC make = value in make.frv cpu=112, dkrd=485, finds=11, rqtm=996, cnct=1
Changes since Wrangling I Simplified Oracle Database design Handles 98% of I type Wrangler data Need to handle other 2% with another design O table design less generic An intermediate table between I and O was eliminated
Changes to reduce Joins Three multiply occurring M204 fields rigidified in Oracle to reduce joins Eliminated PK repetition in 3 tables Field A became A1, A2, A3, A4, A5, A6 Field M became M1, M2, M3, M4 Field R became R1, R2, R3, R4, R5, R6, R7, R8 (Field R is used in every Wrangler Find)
Changes - continued Index tuning Reduced segments used for indices Several major O table indices created with the BITMAP option Matched M204 column indexing Refined column attributes Combined three date/time pairs
Observations: Space Mgmt Oracle - More pools of Space to work with Flexibility Can become overdone Additional Job Security for tuners M204 Need to tune indexes together Share same pools (C, D)
Observations: Programming Simple Ad-hocs (1 table) Much easier with SQL Our end users dont do such requests Would have to limit using Views Complex queries SQL appears delusionally simple Short code can be very complex Pgmr has to learn about paths through database M204 Procedural cross referencing approach is easier to follow.
Living With Oracle vs M204 Retrofitting Performance De-normalization Non-Standard SQL (Hints) Learning how optimizer works Moving Indices around / Eliminating Indices Enqueueing Less enqueueing issues in Oracle Read-consistent views
Backup/Recovery Oracle Recovery Rollback Segment oriented can set up App specific RBS Each databases recovery is independent Closer to TBO than Chkp/Rf Backup/Exports Nothing like Fast Dump or UAI/LAI Dumps DDL and data as Inserts... Losing Oracle Control table is disastrous
System Administration Oracle - Everything is in a bunch of tables Tools use those tables Experts can be helpful but... M204 Information in system stats, journal.. Special commands and tools Monitor Sirmon
DBMS Data Integrity Enforcement Oracle has Standard DBMS features Triggers Referential Integrity Field constraints Major ongoing effort to setup, tune, maintain Programs still have to handle exceptions Decoupled approach used by Wrangler Updates limited to controlled processes that validate data
Should Wrangler be Ported? Porting the Wrangler Database to Oracle is not economically feasible Substantially more design, prototyping, and analysis is necessary Special Techniques necessary The effort to develop and support an Oracle database grows exponentially as size and scope of database increases
Complementing M204 The OS 390 Version of Oracle works well and can effectively support hot backup of a Wrangler adjunct Oracle database that is currently under development. No need to setup and administer another Unix or NT hardware/software environment for this. System Administration, Storage management, Security processes already in place
Supplementing M204 Oracle makes it possible to port subsets of Wrangler data to Unix and NT machines that run Oracle for special purpose applications. Ported Wrangler subsets should be de-indexed Port can be rehearsed effectively on mainframe Oracle integrates effectively with the rest of the World OS Environments Java (COBOL of the 00s) Standards
Wrangler - Currently with Model 204 Current M204 Offerings (CCA/Sirius) Leverage M204s unique strengths Leverage knowledge of User Language Leverage Mainframe instruction set and other features Wrangler performance stays ahead Sirtune Evaluated MP/204 Clever product truly helps - easy to misunderstand
Wrangler - relies on M204 Wrangler survives in the M204 world because of Current features Reorganized entire database for structural changes in one week. Sirius UAI/LAI Dump operational M204 files while transactions are in-flight. Sirius Fast Dump Simple monitoring and problem chasing Sirscan Sirmon
Connecting Wrangler Wrangler survives in M204 because it is becoming more and more interoperable Janus Web Server Supports majority of retrieval user base now Software for selected updating users being tested Janus Sockets Proof of Concept Wrangler Retrieval API API for talking to Java, Perl, VB, C/C++... Tested with Perl on Unix Connect * Evaluated too
Tangible Benefits ? $$$ Performance Not obvious Database Organization Major effort to define Database and Views that would make Standard simple SQL programming work with Wrangler data. Programmability - need to use proprietary Oracle features
Organizational Benefits? Less database systems to support? Major effort to convert Extra effort necessary to port Wrangler applications to Oracle (or DB2) will more than offset cost of continuing to support M204 on mainframe Inherent RDBMS complexity Oracle Knowledge is much more common - bookstore & libraries.
Intangible Benefits? Career Fear Issue M204 - Should we call it M XCC instead? USER part of language name Mainframe? Remaining Current call PAI the self-describing data statement? Ongoing Supportability What is Current now anyway There are antique Oracle databases around too
OS/390 Oracle and Wrangler Serve as a replicated site for a smaller (150M) Wrangler related database. Use Janus Open Client to extract from Oracle database Extract subsets of Wrangler data, then load them into Oracle form to be exported to offsite Oracle databases.
Mainframe Oracle Observations about OS 390 Oracle 8i SQL*Plus works well with TSO and Batch SQL*Loader works well in Batch JCL env Works with NT based administration tools Truly indistinguishable from Unix and NT versions via Net 8 connections Very stable Leverages Mainframe behind the scenes VSAM, Media Manager...
Oracle for Ad-hoc support? Use of generalized Power User tools to query Wrangler data not feasible. Just too much data User Language oriented Selection best Specialized $FUNLOAD for high speed extracts - limited purpose Ongoing improvements to UL Extract speed
M204 - Alive and well Evolutionary approach benefited Wrangler Incremental system upgrades Incremental database upgrades Incremental app performance upgrades Our team is effective at combining knowledge of legacy system processes with ongoing incremental M204 refinements
M204 - Alive and Well Modernized User Interface Ever-expanding Janus Web Server Application suite Providing Wrangler services to external apps Specialized FUEL extracts Store and Forward Specialized API Janus Sockets based Wrangler data server
Porting to Oracle? Porting a big M204 database to Oracle will require iteratively trying many different design ideas. In the first port attempt, 1 million Wrangler M204 records became 14 million Oracle rows. In the second port attempt, 3 million Wrangler M204 records became 8.5 million Oracle rows. This design is less general.
Technical Observations Only one hiccup (OC4) was encountered with OS/390 Oracle during this exercise. Oracle Bitmap indexes used 30-50% less space than Normal indexes. 3 O table indexes were created as Bitmapped Once formatted, space assigned to Oracle is easily re-used.
Living with Oracle - 1 World of M204 is less subject to Press 1 for … support. Most Oracle support requests require entering request via Web and waiting for a formal reply. As is the case with M204 the DBA for a large Oracle database must be familiar with the application. Oracle documentation is readily available. Useful for OS 390 Oracle OS 390 based examples not readily available
Living with Oracle - 2 Ongoing effort to maintain performance of Oracle database is more involved than with M204. Oracle recommends rebuilds of indexes periodically Programmers have to learn how optimizer reacts to their database. Not feasible for end users
Living with Oracle - 3 World of Oracle encourages Technique-ing M204 puzzling to those inclined to Technique- ing Development of Wall Poster database designs Many parts to Oracle What do you choose? Pro C, SQLJ, JSQL, ???
Living with Oracle - 4 Upgrading an Oracle database can be cumbersome Ex: Changing length of a field that is used in referential integrity constraints. Knowledge Base Lots of basic knowledge duplicated Enough to run in circles forever
Unexpected findings CCA and Sirius are not burdened with keeping M204 features working on multiple platforms. Oracle helps give new respect to the M204 Coded field attribute.
Whats Next ? Can SQL access be provided to existing Wrangler M204 data? Is it really needed? Janus Open Server/ Janus Specialty data store? Connect * Brief initial testing with WebGate suggests better performance than with Oracle