1. Lucas Jellema Oracle OpenWorld 2014, San Francisco, CA, USA Everything That Is Really Useful in Oracle Database 12c for Application Developers

2. 2 Oracle Database 12c = Multitenant Architecture aka Pluggable Database

3. 3 Oracle Database 12c – 12.1.0.2 = In-Memory Database Option

4. 4 Oracle Database 12c for Database Developers Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+)

5. 5 License Flashback Data Archive in all database editions

6. Flashback Introduced in 9i Based on UNDO Initially only for recovery As of 11g – Total Recall option with Flashback Data Archive –Controlled history keeping per table select ename, job, sal from emp as of timestamp (systimestamp - INTERVAL '1' DAY) yester_emp

7. Flashback Look back into history –Query trends (version history) –Difference reporting –Restore Test Data Sets [to pre-test situation] –Audit trails (Replace journaling tables) Require trick for transaction history: WHO? Also: when is the start of history? What went on before? What to do with existing archives? By the way: Flashback Data Archive requires EE & Advanced Compression database option

8. Total Recall Flashback Data Archive Improvements: –Complete schema evolution support All table definition, partitioning, and space management DDLs are supported on FDA-enabled tables. –The metadata information for tracking transactions including the user context is now tracked. The addition of user-context tracking makes it easier to determine which user made which changes to a table. This could mean that journaling tables are now officially deprecated Also given that the current contents of journaling tables can even be migrated to Flashback Data Archive

9. Total Recall (2) Import and export of history –Support for import and export using Data Pump for FDA-enabled tables. Data Pump can now be used to export and import an FDA- enabled base table along with its schema-evolution metadata and historical row versions. User generated history –Support for importing user-generated history has been added. Customers who have been maintaining history using other mechanisms, such as triggers, can now import that history into Total Recall. Database Hardening –Register "Application" (a group of tables) and enable/disable flashback data archive for the application (also available: lock an application – make all tables read only)

10. 10 Generate History – Actions by SYS create table oow.emp as select * from scott.emp grant execute on dbms_flashback_archive to oow; grant execute on dbms_flashback to oow; CREATE FLASHBACK ARCHIVE DEFAULT one_year TABLESPACE users QUOTA 100M RETENTION 1 YEAR; grant flashback archive on one_year to oow; exec dbms_flashback_archive.create_temp_history_table('OOW', 'EMP'); -- This statement once in a database instance --This will extend mappings to the past so that import of old history can be done. Goes back to 01-JAN-88. EXEC DBMS_FLASHBACK_ARCHIVE.extend_mappings(); --- -- after some history has been created: EXEC DBMS_FLASHBACK_ARCHIVE.IMPORT_HISTORY('oow','EMP');

11. 11 Generate History – Actions by Application Insert records describing each stage of history that has existed –Including start and end time of historic state insert into temp_history (RID, STARTSCN, ENDSCN, XID, OPERATION,EMPNO, ename, job, hiredate, sal, deptno ) values (NULL, timestamp_to_scn(to_date('01-04-2001', 'DD-MM-YYYY')), timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')), NULL, 'I',1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 10); insert into temp_history (RID, STARTSCN, ENDSCN, XID, OPERATION,EMPNO, ename, job, hiredate, sal, deptno) values (NULL, timestamp_to_scn(to_date('01-07-2003', 'DD-MM-YYYY')), timestamp_to_scn(to_date('01-10-2006', 'DD-MM-YYYY')), NULL, 'U',1567,'SELLERS','CLERK',to_date('01-04-2001','DD-MM-YYYY'),2200, 20); …

12. 12 Query the generated history select ename, job from emp as of timestamp (sysdate - INTERVAL '10' YEAR) minus select ename, job from emp select ename, job from emp as of timestamp (systimestamp - INTERVAL '3' YEAR) minus select ename, job from emp

13. 13 Ensure transaction context is recorded (and set) exec dbms_flashback_archive.set_context_level(level=> 'ALL'); exec dbms_session.set_identifier('The Creepy User from Finance '); update oow.emp set sal = sal * 1.4 where ename = 'ALLEN' / commit; exec dbms_session.set_identifier('Scary Janitor from the Annex'); update oow.emp set sal = sal * 0.7 where ename = 'MILLER' / commit;

14. 14 Audit the generated history SELECT versions_xid, versions_starttime, empno, ename, sal new_sal, s.client_identifier FROM oow.emp VERSIONS BETWEEN TIMESTAMP minvalue AND maxvalue, sys.sys_fba_context_aud s where versions_xid = s.xid

15. 15 Alternative: retrieve context with dbms_flashback_archive.get_sys_context SELECT versions_xid, versions_starttime, empno, ename, sal new_sal, dbms_flashback_archive.get_sys_context (versions_xid,'USERENV','CLIENT_IDENTIFIER') who FROM emp VERSIONS BETWEEN TIMESTAMP minvalue AND maxvalue

16. J ava S cript O bject N otation Lightweight data-interchange format Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+) Support in Oracle Database 12c – 12.1.0.2

17. Name : Value

18. Name : Value, Name : Value

19. Name : Value, Name : Value Name : Value, Name : Value Name : Value, Name : Value,,

20. { "NAME" :"ACCOUNTING", "EMPLOYEES" : [ {"ENAME" : "KING", "JOB" : "PRESIDENT", "SAL" : 5000 }, {"ENAME" : "MILLER", "JOB" : "CLERK", "SAL" : 1300 }] }

21. KING PRESIDENT 5000 MILLER CLERK 1300

22. 22 JSON Light Weight, Structured, fairly tied coupled (low bandwidth) interactions between application[component]s Very popular in rich client web applications and mobile apps – usually as the format used in REST-services Oracle Database 12c is JSON aware –Store documents (in a column VARCHAR2, CLOB, NCLOB, BLOB, RAW, BFILE – and have them checked for JSON validity –Create Indexes on JSON contents –Access JSON content directly from SQL queries –Note: similar to but only a subset of XML support in Oracle Database (as JSON in terms of functionality is a somewhat pale subset of XML) alter table X add CONSTRAINT ensure_json_chk CHECK (column_y IS JSON))

23. 23 JSON in the Oracle Database Why JSON in relational database? –Consistency (integrity, transaction ACIDity) for storing JSON documents –Quickly interpret data received (and stored?) in JSON format –Leverage JSON as a convenient way to handle structured data as a string –Easy data transfer over some protocols, including HTTP – especially when database engages directly in HTTP interactions

24. 24 A sample JSON document {"matches": [ {"matchLineUp": "NED-ESP", "score": "5-2","matchDate":"13-06-2014" }, {"matchLineUp": "SWI-FRA", "score": "2-5","matchDate":"20-06-2014" }, {"matchLineUp": "GER-BRA", "score": "1-7","matchDate":"08-07-2014" } ] }

25. 25 Check for a valid JSON document and/or path result IS JSON can be used in the WHERE clause to filter on rows that contain a valid JSON document –It can be used in CHECK Constraints for that same reason JSON_EXISTS is typically used in the WHERE clause to filter on rows whose JSON content satisfies some JSON path condition select * from customers where json_exists(doc, '$.addresses.privateAddress') select * from customers where doc IS JSON

26. 26 JSON_VALUE to retrieve scalar value from JSON document JSON_VALUE is a SQL Function used to retrieve a scalar value from a JSON document –JSON_VALUE can be used like any other SQL Function in queries and DML – in various clauses like SELECT, WHERE, GROUP BY, etc. select json_value ( '{"matches": [ {"matchLineUp": "NED-ESP", "score": "5-2","matchDate":"13-06-2014"}, {"matchLineUp": "SWI-FRA", "score": "2-5","matchDate":"20-06-2014"}, {"matchLineUp": "GER-BRA", "score": "1-7","matchDate":"08-07-2014"} ] }', '$.matches[1].score') as "2nd_match_score" from dual

27. 27 JSON_QUERY to retrieve JSON- snippets from a JSON document select json_query ( '{"matches": [ {"matchLineUp": "NED-ESP", "score": "5-2","matchDate":"13-06-2014"}, {"matchLineUp": "SWI-FRA", "score": "2-5","matchDate":"20-06-2014"}, {"matchLineUp": "GER-BRA", "score": "1-7","matchDate":"08-07-2014"} ] }', $.matches[*].score' WITH WRAPPER) as scores from dual JSON_QUERY is a SQL Function used to retrieve a JSON snippet from a JSON document - the result is a string that contains valid JSON –Use WRAPPER to wrap an Array result in an object to return a valid document

28. 28 JSON_TABLE to expose records from a JSON document relationally with match_results as ( select '...' json_doc from dual) select lineUp, score, to_date(matchDate, 'DD-MM-YYYY') matchDate from match_results, json_table( json_doc, '$.matches[*]' COLUMNS ( lineUp varchar2(20) PATH '$.matchLineUp', score varchar2(20) PATH '$.score', matchDate varchar2(20) PATH '$.matchDate' ) JSON_TABLE is used in the FROM clause of SQL statements to project data from a JSON document as a virtual relation view –JSON_TABLE is very similar to XML_TABLE

29. 29 12.1.0.2 – JSON What it does not do? Support for variable strings as JSON_PATH in JSON-functions Deal with JSON in PL/SQL –JSON_VALUE and JSON_QUERY cannot be used directly from PL/SQL Construct JSON documents –Not supported: conversion from ADT (to XMLType) to JSON type and vice versa –Not supported: a SQL/JSON syntax similar to SQL/XML for querying together a JSON document –Not supported: facilities that inject JSON into RESTful Services implemented through the PL/SQL Embedded Gateway Types OBJECTS, NESTED TABLE Types OBJECTS, NESTED TABLE XMLType

30. The PL/JSON library This library has been around for several years – and is still pretty much relevant Especially useful –For composing JSON documents –And for converting back and forth from and to JSON to XMLType and ADT/UDT

31. 31 Bringing closer together PL/SQL and SQL Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+)

32. In-line PL/SQL Functions and procedures Procedures are also allowed in-line In-Line Functions and Procedures can invoke each other WITH procedure increment( operand in out number, incsize in number) is begin operand:= operand + incsize; end; FUNCTION inc(value number) RETURN number IS l_value number(10):= value; BEGIN increment(l_value, 100); RETURN l_value; end; SELECT inc(sal) from emp

33. In-line PL/SQL Functions and procedures In-Line Functions and Procedures can invoke each other –And themselves (recursively) WITH FUNCTION inc(value number) RETURN number IS BEGIN if value < 6000 then return inc(value+100); else return value + 100; END if; end; SELECT inc(sal) from emp

34. Faculty calculation Inline PL/SQL function Recursively invoked Using APEX as IDE

35. Dynamic (PL/)SQL is allowed inside inline functions EXECUTE IMMEDIATE can be used inside an inline PL/SQL function to dynamically construct and execute SQL and PL/SQL WITH FUNCTION EMP_ENRICHER(operand varchar2) RETURN varchar2 IS sql_stmt varchar2(500); job varchar2(500); BEGIN sql_stmt := 'SELECT job FROM emp WHERE ename = :param'; EXECUTE IMMEDIATE sql_stmt INTO job USING operand; RETURN ' has job '||job; END; SELECT ename || EMP_ENRICHER(ename) from emp Note: do not try this at home! It is a horrible query! (looks too much like POST_QUERY for comfort) In-Line PL/SQL is not an excuse for lousy SQL

36. Combining in-line Views and PL/SQL Functions & Procedures WITH procedure camel(p_string in out varchar2) is begin p_string:= initcap(p_string); end; function obfuscate(p_string in varchar2) return varchar2 is l_string varchar2(100); begin l_string:= translate(upper(p_string), 'AEUIO','OIEUA'); camel(l_string); return l_string; end; my_view as ( select obfuscate('Goedemorgen') from dual ) select * from my_view

37. PL/SQL Functions That Run Faster in SQL As of Oracle Database Release 12c, two kinds of PL/SQL functions might run faster in SQL: –PL/SQL functions that are defined in the WITH clauses of SQL SELECT statements, described in Oracle Database SQL Language Reference –PL/SQL functions that are defined with the "UDF Pragma" Pragma UDF means: compile in the ‘SQL way’ as to eliminate SQL  PL/SQL context switch FUNCTION inc(string VARCHAR2) RETURN VARCHAR2 IS PRAGMA UDF; value number(10):= to_number(string); BEGIN if value < 6000 then return inc(value+100); else return to_char(value + 100); end if; end;

38. 38 Maximum length for VARCHAR2 is now 32KB (up from 4KB) Invisible Columns One unified audit trail PL/SQL DBMS_UTILITY.EXPAND_SQL_TEXT can be used to uncover the real SQL executed for a given query –Expressed only in the underlying base tables, including VPD policies New PL/SQL Package UTL_CALL_STACK provides API for inspecting the PL/SQL Callstack –Programmatic interpretation, not pretty like DBMS_ UTILITY.FORMAT_CALL_STACK Export View as Table with Data Pump – fine grained projection of columns and rows that will be part of the Dump and subsequent Import Creation of multiple indexes on same set of columns is allowed –Although only one can be live at any one time Cross PDB queries

39. DEFAULT Default applied (also) when NULL was explicitly specified Default based on Sequence Identity Column that is automatically assigned generated sequence number value Meta Data Only Defaults –Data applies to potentially many records but hard takes up any space – only some meta-data are required to declaratively describe the data alter table emp modify (sal number(10,2) DEFAULT ON NULL 1000 ) alter table emp modify (empno number(5) NOT NULL DEFAULT ON NULL EMPNO_SEQ.NEXTVAL )

40. The Top-3 Earning Employees What can you say about the result of this query with respect to the question: "Who are our top three earning employees?" A.Correct Answer B.Sometimes correct C.Correct if there are never duplicate salaries D.Not Correct

41. In-Line Views

42. TOP-N Queries in 12c Last part of a query to be evaluated – to fetch only selected rows from the result set: –To select the next set of rows: select * from emp order by sal desc FETCH FIRST 3 ROWS ONLY; select * from emp order by sal desc OFFSET 3 FETCH NEXT 4 ROWS ONLY;

43. 43 Join (+) enriched to overcome some of the prior limitations: –the native syntax for a LEFT OUTER JOIN has been expanded to allow multiple tables on the left hand side –So you do not have to use LEFT OUTER JOIN/RIGHT OUTER JOIN –But I still think that you should! Source: Connor McDonald (http://www.slideshare.net/hamcdc/feb14-12c-for-developers-31412678)

44. Apply for joining APPLY is used to join with a Collection The function employees_in_department returns a collection (TABLE OF VARCHAR2 in this case) The function takes the DEPTNO value from the DEPT records as input Only when the returned collection is not empty will the DEPT record be produced by this join Use OUTER APPLY to get a result row even for an empty collection SELECT * FROM DEPT d CROSS APPLY employees_in_department(deptno) staff D D D D D D D D

45. Data Masking Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+)

46. Data Redaction At runtime, you can optionally have the query results modified to reset/scramble/randomize sensitive data –Through ‘data redaction’ policies associated with tables and view and applied at query time Because the data is masked in real-time, Data Redaction is well suited to environments in which data is constantly changing. You can create the Data Redaction policies in one central location and easily manage them from there. SQL engine SQL POLICY RESULTS

47. My first Data redaction policy Access to DBMS_REDACT package Create Data Redaction Policy for SAL column in EMP table – hide salaries from view Find that querying EMP has changed forever… –Note: the expression can be used to dynamically decide whether or not to apply the policy grant execute on dbms_redact to scott; BEGIN DBMS_REDACT.ADD_POLICY( object_schema => 'scott', object_name => 'emp', column_name => 'sal', policy_name => 'hide_salary', function_type => DBMS_REDACT.FULL, expression => '1=1' ); END;

48. Querying EMP with DATA REDACTION in place Note: drop Redaction Policy DBMS_REDACT.DROP_POLICY ( object_schema => 'scott', object_name => 'emp', policy_name => 'hide_salary' );

49. Partial Column masking – mask Day and month from hiredate BEGIN DBMS_REDACT.ADD_POLICY( object_schema => 'scott', object_name => 'emp', column_name => 'hiredate', policy_name => 'partially mask hiredate', expression => 'SYS_CONTEXT(''USERENV'',''SESSION_USER'')!= ''GOD''' function_type => DBMS_REDACT.PARTIAL, function_parameters => 'm1d31YHMS', expression => '1=1' ); END;

50. SQL Statement Preprocessor A mechanism to allow the text of a SQL statement, submitted from a client program (e.g. via ODBC or JDBC), to be translated by user-supplied code before it is submitted to the Oracle Database SQL compiler. –Additionally, this feature can satisfy any other use case where it is expedient to intervene between the SQL statement that the client submits and what is actually executed –Some associatations: VPD policies, Data Redaction, MV query rewrite, Advanced Query Rewrite, Temporal Validity (time slice) Application SQL Pre proce ssor SQL engine SQL

51. SQL Translation – set up AS SYS: grant create sql translation profile to oow; grant alter session to oow; AS Application Owner: -- Create a Translation Profile exec dbms_sql_translator.create_profile('HR_PROFILE'); BEGIN DBMS_SQL_TRANSLATOR.REGISTER_SQL_TRANSLATION( profile_name => 'HR_PROFILE', sql_text => 'select ename, job, hiredate from emp', translated_text => 'select initcap(ename) as ename, job, hiredate from emp where job<>''MANAGER'' ‘ ); END; select * FROM USER_SQL_TRANSLATION_PROFILES; select * from USER_SQL_TRANSLATIONS;

52. SQL Translation – in action -- to enable the profile (usually in logon trigger) alter session set sql_translation_profile = HR_PROFILE -- to pretend we are a foreign application, subject to -- SQL Translation alter session set events = '10601 trace name context forever, level 32'; -- execute query that is to be translated select ename, job, hiredate from emp -- results are produced as if the translated text had been submitted SQL Pre proce ssor SQL engine SQL

53. 53 SQL Translation Support for bind parameters Support for rewriting PL/SQL calls Translation of Error Messages (ORA-xxxx to SYB-yyyy or YOURAPP-zzz) Out of the box translators for Sybase, SQL Server and some DB2 Optionally: register a custom translator package –PROCEDURE TRANSLATE_SQL( sql_text IN CLOB, translated_text OUT CLOB);

54. SQL Statement Preprocessor The translation code is named and is installed in the database using a PL/SQL API. It can be implemented programmatically, or by look-up, or by a suitable mixture of these. The mechanism also allows Oracle error codes and American National Standards Institute (ANSI) SQLSTATES to be translated by user-supplied code. The motivating use case is to allow extant client-side application code, written for a different vendor's database (and therefore for a SQL dialect other than Oracle's), to run unchanged against an Oracle Database by emulating the syntax and semantics of the other SQL dialect thereby greatly reducing the cost of migration. Additionally, this feature can satisfy any other use case where it is expedient to intervene between the SQL statement that the client submits and what is actually executed. See: Oracle Database SQL Translation Installation, Configuration, and User's Guide for details

55. 55 Looking into the future… OUR_PRODUCTS NAME PRICE select name, price from our_products

56. 56 Looking further into the future… OUR_PRODUCTS NAME PRICE select name, price from our_products begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'ASOF', query_time => TO_TIMESTAMP('01-10-2018', 'DD-MM-YYYY') ); end;

57. 57 Current situation … OUR_PRODUCTS NAME PRICE select name, price from our_products begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'CURRENT' ); end;

58. 58 All data in the table (the default setting) OUR_PRODUCTS NAME PRICE select name, price from our_products begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'ALL' ); end;

59. 59 All data in the table (the default setting) OUR_PRODUCTS NAMEPRICE select name, price, start_date, end_date from our_products order by start_date START_DATEEND_DATE begin DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME ( level => 'ALL' ); end;

60. Make the database aware of the time based business validity of records Add timestamp columns indicating start and end of valid time for a record Specify a PERIOD for the table Note: –A table can have multiple sets of columns, describing multiple types of validness –Beyond 12.1.0.2 many Temporal Validity enhancements are expected: Unique Constraints, Foreign Key references, (auto) Join conditions, gap and overlap checks, aggregation create table our_products ( name varchar2(100), price number(7,2), start_date timestamp, end_date timestamp, PERIOD FOR offer_time (start_date, end_date) );

61. Valid time aware flashback queries Select all product prices on offer at a certain moment in time Perform all queries for records that are valid at a certain point in time – past or future Return all records currently (session time) valid Return all records (default) SELECT * FROM OUR_PRODUCTS AS OF PERIOD FOR offer_time TO_TIMESTAMP('01-10-2014','DD-MM-YYYY') EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time ( 'ASOF', TO_TIMESTAMP('01-05-2016','DD-MM-YYYY') ); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT'); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');

62. Creating a table with valid time dimension Table with explicit valid time columns: Table with valid time dimension and implicit columns: columns contract_time_start and contract_time_end (TIMESTAMP) are added implicitly CREATE TABLE EMP ( employee_number NUMBER, salary NUMBER, department_id NUMBER, name VARCHAR2(30), hiredate TIMESTAMP, firedate TIMESTAMP, PERIOD FOR user_time (hiredate, firedate) ); CREATE TABLE EMP ( employee_number NUMBER, salary NUMBER, department_id NUMBER, name VARCHAR2(30), PERIOD FOR contract_time );

63. Valid time aware flashback queries Select all employees who were employed at a certain moment in time Perform all queries for records that are valid at a certain point in time Return all records currently (session time) valid Return all records (default) SELECT * FROM EMP AS OF PERIOD FOR user_time TO_TIMESTAMP('01-JUN-2012 12.00.01 PM') EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time ( 'ASOF', TO_TIMESTAMP('29-JUL-15 12.00.01 PM') ); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('CURRENT'); EXECUTE DBMS_FLASHBACK_ARCHIVE.enable_at_valid_time('ALL');

64. 64 Oracle Database 12c for Database Developers Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+) Security

65. White List A white list of allowed invokers can be defined for a PL/SQL unit –supports the robust implementation of a module, consisting of a main unit and helper units, by allowing the helper units to be inaccessible from anywhere except the unit they are intended to help. schema

66. accessible by clause package Helper authid Definer accessible by (Good_Guy, Bad_Guy) is procedure p; end Helper; package body Good_Guy is procedure p is begin Helper.p();... end p; end Good_Guy; package body Bad_Guy is procedure p is begin Helper.p();... end p; end Bad_Guy; PLS-00904: insufficient privilege to access object HELPER _______

67. 67 (More) Security related features Attach database roles to the program units functions, procedures, packages, and types. –The role then becomes enabled during execution of the program unit (but not during compilation of the program unit). A new built-in namespace, SYS_SESSION_ROLES, allows you to determine if a specified role is enabled for the querying user View is either –BEQUEATH DEFINER (the default), which behaves like a Definer’s Rights unit (functions in the view are executed using the view owner’s rights) –or BEQUEATH CURRENT_USER, which behaves somewhat like an invoker’s rights unit (functions in the view are executed using the current user’s rights) An Invoker's Rights Function Can Be Result-Cached READ privilege to allow SELECT but no SELECT FOR UPDATE Invoker’s rights procedure calls only can run with the privileges of the invoker if the procedure’s owner has the INHERIT PRIVILEGES privilege on the invoker (do not stealthily use invoker’s privileges as owner) –or if the procedure’s owner has the INHERIT ANY PRIVILEGES privilege

68. Inherit or not in Invoker rights program units When a user runs an invoker's rights procedure (or program unit), it runs with the privileges of the invoking user. As the procedure runs, the procedure’s owner temporarily has access to the invoking user's privileges. [If the procedure owner has fewer privileges than an invoking user,] the procedure owner could use the invoking user’s privileges to perform operations procedure owner Invoker’s rights invoker procedure Special_ Table select privilege Tap_ Table

69. Pattern Matching Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+)

70. 70 Raw Data Refinement based on pattern matching 14,0 16,1 14,1 16,1 16,0 13,1 14,0 16,0 13,1 13,0 14,1 16,0 14,1 13,0 14,1 16,0 13,1 14,0 16,1 14,1 16,1 16,0 13,1 14,0 16,0 13,1 13,0 14,1 16,0 14,1 13,0 14,1 16,0 13,1 14,0 Processing Information Conclusion Alert Recommendation Action

71. 71 Raw, fine grained Tennis Results: rally points Challenge: Derive the match winner and final score (per set and for the entire match) from this raw data Match Id, Player [who scored] 14,0 16,1 14,1 16,1 16,0 13,1 14,0 Match Id, Player [who scored] 14,0 16,1 14,1 16,1 16,0 13,1 14,0

72. 72 Using MATCH_RECOGNIZE to process data looking for patterns MATCH_RECOGNIZE will analyze, filter, aggregate and reformat the data – based on matches patterns between subsequent rows in the source rows with rallypoints as ( select column_value player, rownum seq from table(number_tbl(1,0,0,1,0,0,1,1,1,0,1,1,1,0,0,0,0)) ) SELECT winner, gameNo FROM rallypoints MATCH_RECOGNIZE (... ) MR source row set MATCH_RECOGNIZE Result Set fed to SELECT

73. 73 Using MATCH_RECOGNIZE to process data looking for patterns with rallypoints as ( select column_value player, rownum seq from table(number_tbl(1,0,0,1,0,0,1,1,1,0,1,1,1,0,0,0,0)) ) SELECT winner, gameNo FROM rallypoints MATCH_RECOGNIZE ( ORDER BY seq MEASURES C.seq AS seq, C.player as winner, MATCH_NUMBER() AS gameNo ONE ROW PER MATCH PATTERN (A+? C) DEFINE C as (case C.player when 1 then sum(A.player) else (sum(abs((A.player-1)))) end >= 3 and case C.player when 1 then sum(A.player*2-1) else (sum(1-A.player*2)) end >= 1 ) ) MR -The first player to have won more than 4 points -and have won two or more points more than his opponent -The first player to have won more than 4 points -and have won two or more points more than his opponent

74. 74 Multi-Level processing RallypointsWon GamesWon SetsWon Matches

75. 75 Using MATCH_RECOGNIZE to process data looking for patterns with rallypoints as ( select column_value player, rownum seq from table(number_tbl(1,0,0,1,0,0,1,1,1,0,1,1,1,0,0,0,0, …)) ), gamepoints as (SELECT winner, gameNo FROM rallypoints MATCH_RECOGNIZE () MR ), setpoints as (SELECT winner, gameNo FROM gamepoints MATCH_RECOGNIZE () MR ), matchpoints as (SELECT winner, gameNo FROM setpoints MATCH_RECOGNIZE () MR ) select * from matchpoints

76. Who is afraid of Red, Yellow and blue Table Events –Column Seq number(5) –Column Payload varchar2(200)

77. Solution using Lead With LEAD it is easy to compare a row with its successor(s) –As long as the pattern is fixed, LEAD will suffice with look_ahead_events as ( SELECT e.*, lead(payload) over (order by seq) next_color, lead(payload,2) over (order by seq) second_next_color FROM events e ) select seq from look_ahead_events where payload ='red' and next_color ='yellow' and second_next_color='blue'

78. Find the pattern red, yellow and blue Using the new 12c Match Recognize operator for finding patterns in relational data SELECT * FROM events MATCH_RECOGNIZE ( ORDER BY seq MEASURES RED.seq AS redseq, MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (RED YELLOW BLUE) DEFINE RED AS RED.payload ='red', YELLOW AS YELLOW.payload ='yellow', BLUE AS BLUE.payload ='blue' ) MR ORDER BY MR.redseq, MR.seq;

79. Match_recognize for finding patterns in relational data The expression MATCH_RECOGNIZE provides native SQL support to find patterns in sequences of rows Match_recognize returns Measures for selected (pattern matched) rows –Similar to MODEL clause Match Conditions are expressed in columns from the Table Source, aggregate functions and pattern functions FIRST, PREV, NEXT, LAST Patterns are regular expressions using match conditions to express a special sequence of rows satisfying the conditions Table Source & Where Table Source & Where Match_ Recognize Process and Filter Match_ Recognize Process and Filter Select & Order By Select & Order By

80. Did we ever consecutively hire three employees in the same job? Find a string of three subsequent hires where each hire has the same job Order by hiredate, pattern is two records that each have the same job as their predecessor SELECT * FROM EMP MATCH_RECOGNIZE ( ORDER BY hiredate MEASURES SAME_JOB.hiredate AS hireday, MATCH_NUMBER() AS match_num ALL ROWS PER MATCH PATTERN (SAME_JOB{3}) DEFINE SAME_JOB AS SAME_JOB.job = FIRST(SAME_JOB.job) ) MR

81. Pattern clause is a regular expression Supported operators for the pattern clause include: –* for 0 or more iterations –+ for 1 or more iterations –? for 0 or 1 iterations –{ n } for exactly n iterations (n > 0) –{ n, } for n or more iterations (n >= 0) –{ n, m } for between n and m (inclusive) iterations (0 <= n <= m, 0 < m) –{, m } for between 0 and m (inclusive) iterations (m > 0) –reluctant qualifiers - *?, +?, ??, {n}?, {n,}?, { n, m }?, {,m}? –| for alternation (OR) –grouping using () parentheses –exclusion using {- and -} –empty pattern using () –^ and $ for start and end of a partition

82. Find the longest sequence of related observations Records are ordered Each record is qualified: assigned to a certain category Examples: –Voting records –Ball possession in football –Days with or without rain –Passing vehicles (make and model or category) –DNA records The challenge: find the longest string of consecutive observations in the same category

83. Find the longest sequence of related observations SELECT section_category, section_start FROM observations MATCH_RECOGNIZE ( ORDER BY seq MEASURES SAME_CATEGORY.category as section_category, FIRST(SAME_CATEGORY.seq) as section_start ONE ROW PER MATCH PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY) -- as many times as possible DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category), DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != NEXT(DIFFERENT_CATEGORY.category) ) MR order by rows_in_section desc )

84. Suppose we allow a single interruption of a sequence One record with a different category will not end the sequence – it might after all be a fluke or an incident Rewrite the pattern match to also accept one entry with a different category ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW -- a next row in the current match may be start of a next string PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY{0,1} SAME_CATEGORY* ) DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category), DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != SAME_CATEGORY.category

85. Find sequence (with one accepted interruption) from all records SELECT substr(section_category,1,1) cat, section_start, seq FROM observations MATCH_RECOGNIZE ( ORDER BY seq MEASURES SAME_CATEGORY.category as section_category, FIRST(SAME_CATEGORY.seq) as section_start, seq as seq ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW -- a next row in the current match may be -- start of a next string PATTERN (SAME_CATEGORY* DIFFERENT_CATEGORY{0,1} SAME_CATEGORY* ) DEFINE SAME_CATEGORY AS SAME_CATEGORY.category = FIRST(SAME_CATEGORY.category), DIFFERENT_CATEGORY AS DIFFERENT_CATEGORY.category != SAME_CATEGORY.category ) MR order by rows_in_section desc

86. Suspicious transactions Find occurrences of three or more money transfers (> 10k) within 24 hours – not necessarily consecutive Account Transfer Timestamp Amount To_Account Account_Number Holder_Name

87. Suspicious transactions select * from transfers t MATCH_RECOGNIZE ( PARTITION BY from_account ORDER BY transfer_time MEASURES MATCH_NUMBER() AS match_num, sum(amount) as total_amount, classifier() as match_role ALL ROWS PER MATCH PATTERN (SUSPICIOUS_TRANSFER NORMAL_TRANSFER* SUSPICIOUS_TRANSFER NORMAL_TRANSFER* SUSPICIOUS_TRANSFER) DEFINE SUSPICIOUS_TRANSFER as SUSPICIOUS_TRANSFER.amount > 10000 and SUSPICIOUS_TRANSFER.transfer_time < FIRST(SUSPICIOUS_TRANSFER.transfer_time + INTERVAL '24' HOUR), NORMAL_TRANSFER as NORMAL_TRANSFER.amount <= 10000 ) MR

88. Two steps forward… and one step back Common expression … common pattern as well? Investigate salary evolution –using Flashback Query and Match_Recognize match_recognize ( partition by empno order by the_time MEASURES MATCH_NUMBER() AS match_num, classifier() as match_role ALL ROWS PER MATCH PATTERN (STRT UP UP DOWN ) DEFINE UP as UP.sal > PREV(UP.SAL), DOWN as DOWN.SAL < PREV(DOWN.SAL) ) MR

89. Two steps forward… and one step back with sal_history as ( select empno, ename, sal, nvl(versions_starttime, versions_endtime) the_time from emp versions between timestamp minvalue and maxvalue ) select * from sal_history match_recognize ( partition by empno order by the_time MEASURES MATCH_NUMBER() AS match_num, classifier() as match_role ALL ROWS PER MATCH PATTERN (STRT UP UP DOWN ) DEFINE UP as UP.sal > PREV(UP.SAL), DOWN as DOWN.SAL < PREV(DOWN.SAL) ) MR

90. No Corner cutting Registration Registration Point CheckPoint_Label Distance_from_Start Runner_Id Timestamp

91. No Corner cutting Patterns to look for: –Checkpoints passed in the wrong order or checkpoints missed altogether –Suspicious accelerations – stretch with > 20% higher average speed than prior or later stretches Additional analysis –Fastest stretch by anyone –Section that is the fastest section for most runners Because of downhill or favorable wind –Top 3 runners over any selected section Registration Registration Point CheckPoint_Label Distance_from_Start Runner_Id Timestamp

92. Find the lacking checkpoint registrations When a runner is registered at a checkpoint that is not the next one (but a farther one) then a previous checkpoint was missed select previous_cpid, checkpoint_id, checkpoint_label, runner_id from ( select r.*, rp.*, lag( checkpoint_id ) over ( partition by runner_id order by rp.id ) previous_cpid from registrations r join registration_points rp on (rp.id = checkpoint_id) ) where checkpoint_id - previous_cpid > 1

93. Find suspicious speeds… When a runner increases speed by more than 20% - something irregular may be going on… match_recognize ( partition by runner_id order by id ALL ROWS PER MATCH PATTERN (SUSPICIOUS_SPEED+ ) DEFINE SUSPICIOUS_SPEED as (distance_from_start- PREV(distance_from_start)/ ( extract ( hour from registration_time) + (extract(minute from registration_time)/60) + (extract(second from registration_time)/3600) - extract ( hour from PREV(registration_time)) + (extract(minute from PREV(registration_time))/60) + (extract(second from PREV(registration_time))/3600) )) > 1.2 * (PREV(distance_from_start)- PREV(distance_from_start,2)/ ( extract ( hour from PREV(registration_time)) + (extract(minute from PREV(registration_time))/60) … - extract ( hour from PREV(registration_time,2)) + (extract(minute from PREV(registration_time,2))/60) + … )) ) MR

94. Find suspicious speeds… (the easier LAG based solution) with runner_data as ( select r.*, leg_distance/( extract ( hour from leg_time) + (extract(minute from leg_time)/60) + (extract(second from leg_time)/3600) ) leg_speed from ( select r.*, rp.*, rp.distance_from_start - lag(rp.distance_from_start,1,0) over (partition by runner_id order by rp.id) leg_distance, r.registration_time – lag(r.registration_time,1,INTERVAL '0' MINUTE) over (partition by runner_id order by rp.id) leg_time from registrations r join registration_points rp on (rp.id = checkpoint_id) ) r ), runner_data_now_and_previous as ( select runner_data.*, lag(leg_speed) over ( partition by runner_id order by checkpoint_id) previous_speed from runner_data ) select * from runner_data_now_and_previous where leg_speed > 1.2 * previous_speed

95. 95 Summary Flash back JSON Top N Pattern Match Inline PL/SQL SQL Translation Data Masking (+) Security Scripts can be downloaded from https://github.com/lucasjellema/OracleDatabase12c-development-demonstrationhttps://github.com/lucasjellema/OracleDatabase12c-development-demonstration