Download presentation
Presentation is loading. Please wait.
Published byYulia Lesmana Modified over 5 years ago
1
JSON in Database 18c/19c JSON as it was meant to be
Stew Ashton UKOUG Techfest 2019 Can you read the following line? If not, please move closer. It's much better when you can read the code ;)
2
Who am I? 1981-2015: 2005-present: focus on Oracle DB development
Developer / application architect From Mainframe to client / server to Web 2005-present: focus on Oracle DB development Advocate of data-centric application architecture Contribute to asktom & ODC/OTN SQL forum Presented at OOW, conferences, meetups… Author of in-house CSV and XML handler
3
JSON is Everywhere Storage of JSON in database tables
JSON Data Guide SODA (Simple Oracle Document Access): NoSQL type APIs to the database PL/SQL Object Types for JSON SQL Developer / sqlcl JSON output from SQL data APEX_JSON ORDS : REST-enabled SQL services SQL/JSON : query (JSONSQL) or generate (JSONSQL)
4
Why "JSON as it was meant to be"?
json.org : "JSON is a lightweight data-interchange format." Not meant to store data Deals in data structures, not "documents" Interchange between programming languages Many advantages over CSV or XML So: use JSON to exchange data to or from SQL 👍
5
Data Exchange Headaches
CSV Encoding ✗ Who knows? Decimal ✗ NLS settings Datetime formats Field terminator ✗ Open bar Field enclosure Escaping ✗ field enclosure * 2 Record delimiter
6
Data Exchange Headaches
CSV XML Encoding ✗ Who knows? ✗ Open bar Decimal ✗ NLS settings Datetime formats ISO 8601 😖 except decimal! Field terminator ✓ </tag> Field enclosure ✓ <tag></tag> Escaping ✗ field enclosure * 2 ✓ Standard Record delimiter Another tag?
7
Data Exchange Headaches
CSV XML JSON Encoding ✗ Who knows? ✗ Open bar ✓ UTF-8 (AL32UTF8) Decimal ✗ NLS settings ✓ Period, period! Datetime formats ISO 8601 😖 except decimal! ✓ ISO 8601 (Oracle) Field terminator ✓ </tag> ✓ comma Field enclosure ✓ <tag></tag> ✓ Double quotes Escaping ✗ field enclosure * 2 ✓ Standard Record delimiter Another tag? ✓ {} or [] JSON alone is independent of database parameters
8
JSON is a data structure
Object Array
9
JSON is a data structure
Object: unordered collection of name/value pairs {"Empno":7369,"Ename":"SMITH"} {"Ename":"SMITH","Empno":7369} Like XML: <Empno>7369</Empno><Ename>SMITH</Ename> Path expression: '$.Ename' , '$.Empno' Array: ordered list of values [7369,"SMITH","CLERK"] Like CSV: 7369,"SMITH","CLERK" Path expression: '$[0]' , '$[1]' , '$[2]' JSON_EQUAL
10
JSON is a data structure
Object: unordered collection of name/value pairs {"Empno":7369,"Ename":"SMITH"} {"Ename":"SMITH","Empno":7369} Like XML: <Empno>7369</Empno><Ename>SMITH</Ename> Path expression: '$.Ename' , '$.Empno' Array: ordered list of values [7369,"SMITH","CLERK"] Like CSV: 7369,"SMITH","CLERK" Path expression: '$[0]' , '$[1]' , '$[2]' "value" can be another object or array
11
JSON Data in SQL Statements
It’s not a special data type (yet!) VARCHAR2 CLOB BLOB in AL32UTF8 "Oracle recommends that you use AL32UTF8 as the database character set..." Use JSON_QUERY (or JSON_SERIALIZE in 19c) to get the BLOB in readable format When does Oracle consider data to be JSON? '{ }' could be an empty JSON object or a JSON string value "{ }" Stored data: column with IS JSON constraint Output: Result of JSON generation function Input: FORMAT JSON in JSON generation function TREAT(… AS JSON)
12
SQL <> JSON Transformation
Direction Source Function Returns Prefer for Generate 1 row JSON_OBJECT object OLTP JSON_ARRAY array ETL
13
SQL <> JSON Transformation
Direction Source Function Returns Prefer for Generate 1 row JSON_OBJECT object OLTP JSON_ARRAY array ETL N rows JSON_OBJECTAGG JSON_ARRAYAGG
14
SQL <> JSON Transformation
Direction Source Function Returns Prefer for Generate 1 row JSON_OBJECT object OLTP JSON_ARRAY array ETL N rows JSON_OBJECTAGG JSON_ARRAYAGG Query JSON JSON_QUERY 1 JSON part PRETTY, BLOB JSON_VALUE 1 Scalar JSON_EXISTS true/false
15
SQL <> JSON Transformation
Direction Source Function Returns Prefer for Generate 1 row JSON_OBJECT object OLTP JSON_ARRAY array ETL N rows JSON_OBJECTAGG JSON_ARRAYAGG Query JSON JSON_QUERY PRETTY JSON_VALUE SQL data type JSON_EXISTS true/false JSON_TABLE Multiple ETL / OLTP
16
SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string "varchar2" NVARCHAR2 "nvarchar2" CLOB "clob"
17
SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string "varchar2" NVARCHAR2 "nvarchar2" CLOB "clob" RAW hex string "1F2F3F4F" BLOB "1F2F3F4F5F6F7F8F"
18
SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string "varchar2" NVARCHAR2 "nvarchar2" CLOB "clob" RAW hex string "1F2F3F4F" BLOB "1F2F3F4F5F6F7F8F" NUMBER number 6 BINARY_DOUBLE 7 BINARY_FLOAT 8
19
SQL Data Type to JSON Converted to UTC Input data types JSON result
Example VARCHAR2 string "varchar2" NVARCHAR2 "nvarchar2" CLOB "clob" RAW hex string "1F2F3F4F" BLOB "1F2F3F4F5F6F7F8F" NUMBER number 6 BINARY_DOUBLE 7 BINARY_FLOAT 8 DATE ISO 8601 " T20:00:07" TIMESTAMP " T20:00: " TIMESTAMP WITH LOCAL TZ " T18:00: Z" TIMESTAMP WITH TZ INTERVAL DAY TO SECOND "P13D" INTERVAL YEAR TO MONTH "P1Y2M" Converted to UTC
20
create table t ( C_VARCHAR2 VARCHAR2(16), C_NVARCHAR2 NVARCHAR2(16), C_CLOB CLOB, C_RAW RAW(16), C_BLOB BLOB, C_NUMBER NUMBER(3,1), C_BINARY_DOUBLE BINARY_DOUBLE, C_BINARY_FLOAT BINARY_FLOAT, C_DATE DATE, C_TIMESTAMP TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO TIMESTAMP WITH LOCAL TIME ZONE, C_TIMESTAMP_WITH_TIME_ZONE TIMESTAMP WITH TIME ZONE, C_INTERVAL_DAY_TO_SECOND INTERVAL DAY TO SECOND, C_INTERVAL_YEAR_TO_MONTH INTERVAL YEAR TO MONTH ); Table
21
CSV without field enclosures
select C_VARCHAR2||','|| C_NVARCHAR2||',’|| C_CLOB||','|| rawtohex(C_RAW)||',’|| <blob is complicated>||','|| to_char(C_NUMBER, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_BINARY_DOUBLE, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_BINARY_FLOAT, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_DATE, 'yyyy-mm-dd"T"hh24:mi:ss')||','|| to_char(C_TIMESTAMP, 'yyyy-mm-dd"T"hh24:mi:ssxff')||','|| to_char(C_TIMESTAMP_WITH_LOCAL_TIME_ZO, 'yyyy-mm-dd"T"hh24:mi:ssxff')||','|| to_char(C_TIMESTAMP_WITH_TIME_ZONE, 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM')||','|| C_INTERVAL_DAY_TO_SECOND||','|| C_INTERVAL_YEAR_TO_MONTH csv_text from T; CSV without field enclosures
22
select xmlforest( C_VARCHAR2, C_NVARCHAR2, C_CLOB, rawtohex(C_RAW), <blob is complicated>, to_char(C_NUMBER, 'TM', 'nls_numeric_characters=''. ''') as C_NUMBER, to_char(C_BINARY_DOUBLE, 'TM', 'nls_numeric_characters=''. ''') as C_BINARY_DOUBLE, to_char(C_BINARY_FLOAT, 'TM', 'nls_numeric_characters=''. ''') as C_BINARY_FLOAT, to_char(C_DATE, 'yyyy-mm-dd"T"hh24:mi:ss') as C_DATE, to_char(C_TIMESTAMP, 'yyyy-mm-dd"T"hh24:mi:ssxff', 'nls_numeric_characters=''. ''') as C_TIMESTAMP, to_char(to_timestamp_tz(C_TIMESTAMP_WITH_LOCAL_TIME_ZO), 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM', 'nls_numeric_characters=''. ''') as C_TIMESTAMP_WITH_LOCAL_TIME_ZO, to_char(C_TIMESTAMP_WITH_TIME_ZONE, 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM', 'nls_numeric_characters=''. ''') as C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) T_XML from T; XML
23
What if C_BINARY_FLOAT is null?
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) sql_to_json from T; [ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, 8, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ] What if C_BINARY_FLOAT is null?
24
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, -- disappeared 😧 C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) sql_to_json from T; [ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ] Why?
25
ABSENT ON NULL is the default
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ABSENT ON NULL) sql_to_json from T; [ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ] ABSENT ON NULL is the default
26
Always add this for arrays.
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, -- it's back 🤗 C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH NULL ON NULL) sql_to_json from T; [ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, null, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ] Always add this for arrays.
27
select json_object( { key 'empno' value empno, "empno" : 7788, 'ename' value ename, "ename" : "SCOTT", 'job' : job, c "job" : "ANALYST", DEPTNO, c "DEPTNO" : 20, sal, c "sal" : 3000, comm c "comm" : null ← NULL ON NULL ) jo } from scott.emp where ename = 'SCOTT'; Select json_object(*) c
28
JSON to SQL Data Type Original SQL data type JSON Target SQL data type VARCHAR2 string NVARCHAR2 CLOB RAW (hex) string BLOB NUMBER number BINARY_DOUBLE BINARY_FLOAT DATE ISO 8601 TIMESTAMP TIMESTAMP WITH LOCAL TZ TIMESTAMP WITH TZ INTERVAL DAY TO SECOND INTERVAL YEAR TO MONTH These are the data types that can be returned by JSON_VALUE and JSON_TABLE.
29
Round trip Varchar2 Clob N Date Timestamp Timestamp with time zone
select jt.* from json_data, json_table(sql_to_json, '$' columns ( "Varchar2" VARCHAR2(16 BYTE) path '$[0]', "Clob" CLOB path '$[2]', "N" NUMBER path '$[5]', "Date" DATE path '$[8]', "Timestamp" TIMESTAMP path '$[9]', "Timestamp with time zone" TIMESTAMP WITH TIME ZONE path '$[11]' )) jt; Varchar2 Clob N Date Timestamp Timestamp with time zone varchar2 clob 6 :00 :26:22,67459 :26:22,67459 GMT Time portion lost JSON > SQL Time zone lost SQL > JSON
30
JSON to SQL Data Type Input data types JSON result Return type
For explicit conversion VARCHAR2 string NVARCHAR2 to_nchar() CLOB RAW hex string VARCHAR2(nn CHAR) hextoraw() BLOB must write function NUMBER number BINARY_DOUBLE to_binary_double() BINARY_FLOAT to_binary_float() DATE ISO 8601 TIMESTAMP cast(<value> as date) TIMESTAMP WITH LOCAL TZ TIMESTAMP WITH TZ cast(<value> as timestamp with local TZ) INTERVAL DAY TO SECOND VARCHAR2(16 BYTE) to_dsinterval() INTERVAL YEAR TO MONTH to_yminterval()
31
Hex_to_blob function create or replace FUNCTION hex_to_blob (hex CLOB) RETURN BLOB AUTHID CURRENT_USER IS b BLOB := NULL; s VARCHAR2(32766 BYTE) := NULL; l NUMBER := 32766; BEGIN if hex is not null then dbms_lob.createtemporary(b, FALSE); FOR i IN 0 .. floor(LENGTH(hex) / l) LOOP dbms_lob.read(hex, l, i * l + 1, s); dbms_lob.append(b, hextoraw(s)); END LOOP; end if; RETURN b; END hex_to_blob;
32
So Far so Good Oracle can generate JSON from all scalar data types (within reason). Oracle (with our help) can extract all scalar data types from JSON. 19c: support for SQL objects and collections Data can go from SQL to JSON to SQL without loss But for time zones (and careful with time part of date) Note: except for "proof of concept", do not use JSON to exchange data between Oracle DBs.
33
ETL: external table, 1 array per row
Extract Load Select json_array(<columns> null on null) from <table>; (write to file) select <converted columns> from <external table>, json_table(json_data, '$' columns ( a varchar2(99) path '$[0]', b number path '$[1]', ... )); Not "standard" but scalable Must agree on what separates each array
34
ETL: CLOB & array of arrays
Extract Load select json_arrayagg( json_array(<columns> null on null) returning clob ) from <table>; (write clob to file) select <converted columns> from json_table( bfilename('<dir>', '<filename>'), '$[*]' columns ( a varchar2(99) path '$[0]', b varchar2(99) path '$[1]', ... )); Standard, but memory hog?
35
OLTP: Managers
36
Just Join? with m as ( SELECT employee_id manager_id, last_name FROM employees WHERE employee_id = 103 ) select MANAGER_ID, m.LAST_NAME, d.DEPARTMENT_ID, d.DEPARTMENT_NAME, e.EMPLOYEE_ID, e.FIRST_NAME from m join departments d using(manager_id) join employees e using(manager_id);
37
MANAGER_ID LAST_NAME DEPARTMENT_ID DEPARTMENT_NAME EMPLOYEE_ID FIRST_NAME 103 Hunold 10 Administration 104 Bruce 20 Marketing 30 Purchasing 60 IT 105 David 106 Valli 107 Diana
38
with m as ( SELECT employee_id manager_id, last_name FROM employees WHERE employee_id = 103 ) SELECT json_object( 'manager_id' value manager_id, 'last_name' value last_name, 'departments' value ( ? ), 'employees' value ( ? ) ) ) json_data FROM m; SELECT JSON_ARRAYAGG( JSON_OBJECT( 'department_id' value department_id, 'department_name' value department_name ) FROM departments WHERE manager_id = m.manager_id SELECT JSON_ARRAYAGG( JSON_OBJECT( 'employee_id' value employee_id, 'first_name' value first_name ) FROM employees WHERE manager_id = m.manager_id
39
Now let's go from JSON to SQL
{ "manager_id" : 103, "last_name" : "Hunold", "departments" : [ "department_id" : 10, "department_name" : "Administration" }, ... "department_id" : 60, "department_name" : "IT" } ], "employees" : [ { "employee_id" : 104, "first_name" : "Bruce" }, ... "employee_id" : 107, "first_name" : "Diana" } ] Now let's go from JSON to SQL
40
Just the Scalar values MANAGER_ID LAST_NAME 103 Hunold
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name ) j; MANAGER_ID LAST_NAME 103 Hunold
41
Nested path: parent / child
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name, nested path '$.departments[*]' columns ( department_id, department_name ) ) j; MANAGER_ID LAST_NAME DEPARTMENT_ID DEPARTMENT_NAME 103 Hunold 10 Administration 20 Marketing 30 Purchasing 60 IT
42
Nested path: Ordinality
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name, nested path '$.departments[*]' columns ( d_ord for ordinality, department_id, department_name ) ) j; MANAGER_ID LAST_NAME D_ORD DEPARTMENT_ID DEPARTMENT_NAME 103 Hunold 1 10 Administration 2 20 Marketing 3 30 Purchasing 4 60 IT
43
Nested paths: Siblings
select j.* from mgr_json, json_table(json_data, '$' columns manager_id number, last_name, nested path '$.departments[*]' columns ( d_ord for ordinality, department_id number, department_name ), nested path '$.employees[*]' columns ( e_ord for ordinality, employee_id number, first_name ) ABSENT ON NULL) j;
44
Nested paths: Siblings
MANAGER_ID LAST_NAME D_ORD DEPT_ID DEPARTMENT_NAME E_ORD EMP_ID FIRST_NAME 103 Hunold 1 10 Administration 2 20 Marketing 3 30 Purchasing 4 60 IT 104 Bruce 105 David 106 Valli 107 Diana
45
Advantages for OLTP Out with ORM, in with JSONRM
Provide application-specific APIs in JSON format Use SQL*Net or REST Data access layer and DB are less tightly coupled Generate / interpret complex hierarchical data Read consistency for all data One SELECT per API
46
JSON in Database 18c/19c JSON as it was meant to be
Stew Ashton UKOUG Techfest 2019 Blog: stewashton.wordpress.com
47
If we had time… ETL: multi-table insert with nested JSON
OLTP modifications JSON_MERGEPATCH (19c) String manipulation Splitting strings stewashton.wordpress.com/splitting-strings-a-new-champion/ Listagg() for CLOBs
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.