1. JSON in Database 18c/19c JSON as it was meant to be
Stew Ashton
UKOUG Techfest 2019
Can you read the following line? If not, please move closer.
It's much better when you can read the code ;)

2. Who am I? 1981-2015: 2005-present: focus on Oracle DB development
Developer / application architect
From Mainframe to client / server to Web
2005-present: focus on Oracle DB development
Advocate of data-centric application architecture
Contribute to asktom & ODC/OTN SQL forum
Presented at OOW, conferences, meetups…
Author of in-house CSV and XML handler

3. JSON is Everywhere Storage of JSON in database tables
JSON Data Guide
SODA (Simple Oracle Document Access): NoSQL type APIs to the database
PL/SQL Object Types for JSON
SQL Developer / sqlcl JSON output from SQL data
APEX_JSON
ORDS : REST-enabled SQL services
SQL/JSON : query (JSONSQL) or generate (JSONSQL)

4. Why "JSON as it was meant to be"?
json.org : "JSON is a lightweight data-interchange format."
Not meant to store data
Deals in data structures, not "documents"
Interchange between programming languages
Many advantages over CSV or XML
So: use JSON to exchange data to or from SQL 👍

5. Data Exchange Headaches
CSV
Encoding
✗ Who knows?
Decimal
✗ NLS settings
Datetime formats
Field terminator
✗ Open bar
Field enclosure
Escaping
✗ field enclosure * 2
Record delimiter

6. Data Exchange Headaches
CSV
XML
Encoding
✗ Who knows?
✗ Open bar
Decimal
✗ NLS settings
Datetime formats
ISO 8601 😖 except decimal!
Field terminator
✓ </tag>
Field enclosure
✓ <tag></tag>
Escaping
✗ field enclosure * 2
✓ Standard
Record delimiter
Another tag?

7. Data Exchange Headaches
CSV
XML
JSON
Encoding
✗ Who knows?
✗ Open bar
✓ UTF-8 (AL32UTF8)
Decimal
✗ NLS settings
✓ Period, period!
Datetime formats
ISO 8601 😖 except decimal!
✓ ISO 8601 (Oracle)
Field terminator
✓ </tag>
✓ comma
Field enclosure
✓ <tag></tag>
✓ Double quotes
Escaping
✗ field enclosure * 2
✓ Standard
Record delimiter
Another tag?
✓ {} or []
JSON alone is independent of database parameters

8. JSON is a data structure
Object
Array

9. JSON is a data structure
Object: unordered collection of name/value pairs
{"Empno":7369,"Ename":"SMITH"} {"Ename":"SMITH","Empno":7369}
Like XML: <Empno>7369</Empno><Ename>SMITH</Ename>
Path expression: '$.Ename' , '$.Empno'
Array: ordered list of values
[7369,"SMITH","CLERK"]
Like CSV: 7369,"SMITH","CLERK"
Path expression: '$[0]' , '$[1]' , '$[2]'
JSON_EQUAL

10. JSON is a data structure
Object: unordered collection of name/value pairs
{"Empno":7369,"Ename":"SMITH"} {"Ename":"SMITH","Empno":7369}
Like XML: <Empno>7369</Empno><Ename>SMITH</Ename>
Path expression: '$.Ename' , '$.Empno'
Array: ordered list of values
[7369,"SMITH","CLERK"]
Like CSV: 7369,"SMITH","CLERK"
Path expression: '$[0]' , '$[1]' , '$[2]'
"value" can be another object or array

11. JSON Data in SQL Statements
It’s not a special data type (yet!)
VARCHAR2
CLOB
BLOB in AL32UTF8
"Oracle recommends that you use AL32UTF8 as the database character set..."
Use JSON_QUERY (or JSON_SERIALIZE in 19c) to get the BLOB in readable format
When does Oracle consider data to be JSON?
'{ }' could be an empty JSON object or a JSON string value "{ }"
Stored data: column with IS JSON constraint
Output: Result of JSON generation function
Input: FORMAT JSON in JSON generation function
TREAT(… AS JSON)

12. SQL <> JSON Transformation
Direction
Source
Function
Returns
Prefer for
Generate
1 row
JSON_OBJECT
object
OLTP
JSON_ARRAY
array
ETL

13. SQL <> JSON Transformation
Direction
Source
Function
Returns
Prefer for
Generate
1 row
JSON_OBJECT
object
OLTP
JSON_ARRAY
array
ETL
N rows
JSON_OBJECTAGG
JSON_ARRAYAGG

14. SQL <> JSON Transformation
Direction
Source
Function
Returns
Prefer for
Generate
1 row
JSON_OBJECT
object
OLTP
JSON_ARRAY
array
ETL
N rows
JSON_OBJECTAGG
JSON_ARRAYAGG
Query
JSON
JSON_QUERY
1 JSON part
PRETTY,
BLOB
JSON_VALUE
1 Scalar
JSON_EXISTS
true/false

15. SQL <> JSON Transformation
Direction
Source
Function
Returns
Prefer for
Generate
1 row
JSON_OBJECT
object
OLTP
JSON_ARRAY
array
ETL
N rows
JSON_OBJECTAGG
JSON_ARRAYAGG
Query
JSON
JSON_QUERY
PRETTY
JSON_VALUE
SQL data type
JSON_EXISTS
true/false
JSON_TABLE
Multiple
ETL / OLTP

16. SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string
"varchar2"
NVARCHAR2
"nvarchar2"
CLOB
"clob"

17. SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string
"varchar2"
NVARCHAR2
"nvarchar2"
CLOB
"clob"
RAW
hex string
"1F2F3F4F"
BLOB
"1F2F3F4F5F6F7F8F"

18. SQL Data Type to JSON Input data types JSON result Example VARCHAR2
string
"varchar2"
NVARCHAR2
"nvarchar2"
CLOB
"clob"
RAW
hex string
"1F2F3F4F"
BLOB
"1F2F3F4F5F6F7F8F"
NUMBER
number 6
BINARY_DOUBLE 7
BINARY_FLOAT 8

19. SQL Data Type to JSON Converted to UTC Input data types JSON result
Example
VARCHAR2
string
"varchar2"
NVARCHAR2
"nvarchar2"
CLOB
"clob"
RAW
hex string
"1F2F3F4F"
BLOB
"1F2F3F4F5F6F7F8F"
NUMBER
number 6
BINARY_DOUBLE 7
BINARY_FLOAT 8
DATE
ISO 8601
" T20:00:07"
TIMESTAMP
" T20:00: "
TIMESTAMP WITH LOCAL TZ
" T18:00: Z"
TIMESTAMP WITH TZ
INTERVAL DAY TO SECOND
"P13D"
INTERVAL YEAR TO MONTH
"P1Y2M"
Converted to UTC

20. create table t ( C_VARCHAR2 VARCHAR2(16), C_NVARCHAR2 NVARCHAR2(16), C_CLOB CLOB, C_RAW RAW(16), C_BLOB BLOB, C_NUMBER NUMBER(3,1), C_BINARY_DOUBLE BINARY_DOUBLE, C_BINARY_FLOAT BINARY_FLOAT, C_DATE DATE, C_TIMESTAMP TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO TIMESTAMP WITH LOCAL TIME ZONE, C_TIMESTAMP_WITH_TIME_ZONE TIMESTAMP WITH TIME ZONE, C_INTERVAL_DAY_TO_SECOND INTERVAL DAY TO SECOND, C_INTERVAL_YEAR_TO_MONTH INTERVAL YEAR TO MONTH );
Table

21. CSV without field enclosures
select C_VARCHAR2||','|| C_NVARCHAR2||',’|| C_CLOB||','|| rawtohex(C_RAW)||',’|| <blob is complicated>||','|| to_char(C_NUMBER, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_BINARY_DOUBLE, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_BINARY_FLOAT, 'TM', 'nls_numeric_characters=''. ''')||','|| to_char(C_DATE, 'yyyy-mm-dd"T"hh24:mi:ss')||','|| to_char(C_TIMESTAMP, 'yyyy-mm-dd"T"hh24:mi:ssxff')||','|| to_char(C_TIMESTAMP_WITH_LOCAL_TIME_ZO, 'yyyy-mm-dd"T"hh24:mi:ssxff')||','|| to_char(C_TIMESTAMP_WITH_TIME_ZONE, 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM')||','|| C_INTERVAL_DAY_TO_SECOND||','|| C_INTERVAL_YEAR_TO_MONTH csv_text from T;
CSV
without field enclosures

22. select xmlforest( C_VARCHAR2, C_NVARCHAR2, C_CLOB, rawtohex(C_RAW), <blob is complicated>, to_char(C_NUMBER, 'TM', 'nls_numeric_characters=''. ''') as C_NUMBER, to_char(C_BINARY_DOUBLE, 'TM', 'nls_numeric_characters=''. ''') as C_BINARY_DOUBLE, to_char(C_BINARY_FLOAT, 'TM', 'nls_numeric_characters=''. ''') as C_BINARY_FLOAT, to_char(C_DATE, 'yyyy-mm-dd"T"hh24:mi:ss') as C_DATE, to_char(C_TIMESTAMP, 'yyyy-mm-dd"T"hh24:mi:ssxff', 'nls_numeric_characters=''. ''') as C_TIMESTAMP, to_char(to_timestamp_tz(C_TIMESTAMP_WITH_LOCAL_TIME_ZO), 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM', 'nls_numeric_characters=''. ''') as C_TIMESTAMP_WITH_LOCAL_TIME_ZO, to_char(C_TIMESTAMP_WITH_TIME_ZONE, 'yyyy-mm-dd"T"hh24:mi:ssxffTZH:TZM', 'nls_numeric_characters=''. ''') as C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) T_XML from T;
XML

23. What if C_BINARY_FLOAT is null?
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) sql_to_json from T;
[ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, 8, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ]
What if C_BINARY_FLOAT is null?

24. select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, -- disappeared 😧 C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ) sql_to_json from T;
[ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ]
Why?

25. ABSENT ON NULL is the default
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH ABSENT ON NULL) sql_to_json from T;
[ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ]
ABSENT ON NULL is the default

26. Always add this for arrays.
select json_array( C_VARCHAR2, C_NVARCHAR2, C_CLOB, C_RAW, C_BLOB, C_NUMBER, C_BINARY_DOUBLE, C_BINARY_FLOAT, -- it's back 🤗 C_DATE, C_TIMESTAMP, C_TIMESTAMP_WITH_LOCAL_TIME_ZO, C_TIMESTAMP_WITH_TIME_ZONE, C_INTERVAL_DAY_TO_SECOND, C_INTERVAL_YEAR_TO_MONTH NULL ON NULL) sql_to_json from T;
[ "varchar2", "nvarchar2", "clob", "1F2F3F4F", "1F2F3F4F5F6F7F8F", 6, 7, null, " T20:00:07", " T20:00: ", " T18:00: Z", "P13D", "P1Y2M" ]
Always add this for arrays.

27. select json_object( {
key 'empno' value empno, "empno" : 7788,
'ename' value ename, "ename" : "SCOTT",
'job' : job, c "job" : "ANALYST",
DEPTNO, c "DEPTNO" : 20,
sal, c "sal" : 3000,
comm c "comm" : null ← NULL ON NULL
) jo }
from scott.emp
where ename = 'SCOTT';
Select json_object(*) c

28. JSON to SQL Data Type
Original SQL data type
JSON
Target SQL data type
VARCHAR2
string
NVARCHAR2
CLOB
RAW
(hex) string
BLOB
NUMBER
number
BINARY_DOUBLE
BINARY_FLOAT
DATE
ISO 8601
TIMESTAMP
TIMESTAMP WITH LOCAL TZ
TIMESTAMP WITH TZ
INTERVAL DAY TO SECOND
INTERVAL YEAR TO MONTH
These are the data types that can be returned by JSON_VALUE and JSON_TABLE.

29. Round trip Varchar2 Clob N Date Timestamp Timestamp with time zone
select jt.* from json_data, json_table(sql_to_json, '$' columns ( "Varchar2" VARCHAR2(16 BYTE) path '$[0]', "Clob" CLOB path '$[2]', "N" NUMBER path '$[5]', "Date" DATE path '$[8]', "Timestamp" TIMESTAMP path '$[9]', "Timestamp with time zone" TIMESTAMP WITH TIME ZONE path '$[11]' )) jt;
Varchar2
Clob N
Date
Timestamp
Timestamp with time zone
varchar2
clob 6
:00
:26:22,67459
:26:22,67459 GMT
Time portion lost JSON > SQL
Time zone lost
SQL > JSON

30. JSON to SQL Data Type Input data types JSON result Return type
For explicit conversion
VARCHAR2
string
NVARCHAR2
to_nchar()
CLOB
RAW
hex string
VARCHAR2(nn CHAR)
hextoraw()
BLOB
must write function
NUMBER
number
BINARY_DOUBLE
to_binary_double()
BINARY_FLOAT
to_binary_float()
DATE
ISO 8601
TIMESTAMP
cast(<value> as date)
TIMESTAMP WITH LOCAL TZ
TIMESTAMP WITH TZ
cast(<value> as timestamp with local TZ)
INTERVAL DAY TO SECOND
VARCHAR2(16 BYTE)
to_dsinterval()
INTERVAL YEAR TO MONTH
to_yminterval()

31. Hex_to_blob function
create or replace FUNCTION hex_to_blob (hex CLOB) RETURN BLOB AUTHID CURRENT_USER IS b BLOB := NULL; s VARCHAR2(32766 BYTE) := NULL; l NUMBER := 32766; BEGIN if hex is not null then dbms_lob.createtemporary(b, FALSE); FOR i IN 0 .. floor(LENGTH(hex) / l) LOOP dbms_lob.read(hex, l, i * l + 1, s); dbms_lob.append(b, hextoraw(s)); END LOOP; end if; RETURN b; END hex_to_blob;

32. So Far so Good
Oracle can generate JSON from all scalar data types (within reason).
Oracle (with our help) can extract all scalar data types from JSON.
19c: support for SQL objects and collections
Data can go from SQL to JSON to SQL without loss
But for time zones (and careful with time part of date)
Note: except for "proof of concept", do not use JSON to exchange data between Oracle DBs.

33. ETL: external table, 1 array per row
Extract
Load
Select json_array(<columns> null on null) from <table>; (write to file)
select <converted columns> from <external table>, json_table(json_data, '$' columns ( a varchar2(99) path '$[0]', b number path '$[1]', ... ));
Not "standard" but scalable
Must agree on what separates each array

34. ETL: CLOB & array of arrays
Extract
Load
select json_arrayagg( json_array(<columns> null on null) returning clob ) from <table>; (write clob to file)
select <converted columns> from json_table( bfilename('<dir>', '<filename>'), '$[*]' columns ( a varchar2(99) path '$[0]', b varchar2(99) path '$[1]', ... ));
Standard, but memory hog?

35. OLTP: Managers

36. Just Join?
with m as ( SELECT employee_id manager_id, last_name FROM employees WHERE employee_id = 103 ) select MANAGER_ID, m.LAST_NAME, d.DEPARTMENT_ID, d.DEPARTMENT_NAME, e.EMPLOYEE_ID, e.FIRST_NAME from m join departments d using(manager_id) join employees e using(manager_id);

37. MANAGER_ID
LAST_NAME
DEPARTMENT_ID
DEPARTMENT_NAME
EMPLOYEE_ID
FIRST_NAME
103
Hunold 10
Administration
104
Bruce 20
Marketing 30
Purchasing 60 IT
105
David
106
Valli
107
Diana

38. with m as ( SELECT employee_id manager_id, last_name FROM employees WHERE employee_id = 103 ) SELECT json_object( 'manager_id' value manager_id, 'last_name' value last_name, 'departments' value ( ? ), 'employees' value ( ? ) ) ) json_data FROM m;
SELECT JSON_ARRAYAGG(
JSON_OBJECT(
'department_id' value department_id,
'department_name'
value department_name )
FROM departments
WHERE manager_id = m.manager_id
SELECT JSON_ARRAYAGG(
JSON_OBJECT(
'employee_id' value employee_id,
'first_name' value first_name )
FROM employees
WHERE manager_id = m.manager_id

39. Now let's go from JSON to SQL
{ "manager_id" : 103, "last_name" : "Hunold", "departments" : [ "department_id" : 10, "department_name" : "Administration" }, ... "department_id" : 60, "department_name" : "IT" } ],
"employees" : [ { "employee_id" : 104, "first_name" : "Bruce" }, ... "employee_id" : 107, "first_name" : "Diana" } ]
Now let's go from JSON to SQL

40. Just the Scalar values MANAGER_ID LAST_NAME 103 Hunold
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name ) j;
MANAGER_ID
LAST_NAME
103
Hunold

41. Nested path: parent / child
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name, nested path '$.departments[*]' columns ( department_id, department_name ) ) j;
MANAGER_ID
LAST_NAME
DEPARTMENT_ID
DEPARTMENT_NAME
103
Hunold 10
Administration 20
Marketing 30
Purchasing 60 IT

42. Nested path: Ordinality
select j.* from mgr_json, json_table(json_data, '$' columns manager_id, last_name, nested path '$.departments[*]' columns ( d_ord for ordinality, department_id, department_name ) ) j;
MANAGER_ID
LAST_NAME
D_ORD
DEPARTMENT_ID
DEPARTMENT_NAME
103
Hunold 1 10
Administration 2 20
Marketing 3 30
Purchasing 4 60 IT

43. Nested paths: Siblings
select j.* from mgr_json, json_table(json_data, '$' columns manager_id number, last_name, nested path '$.departments[*]' columns ( d_ord for ordinality, department_id number, department_name ), nested path '$.employees[*]' columns ( e_ord for ordinality, employee_id number, first_name ) ABSENT ON NULL) j;

44. Nested paths: Siblings
MANAGER_ID
LAST_NAME
D_ORD
DEPT_ID
DEPARTMENT_NAME
E_ORD
EMP_ID
FIRST_NAME
103
Hunold 1 10
Administration 2 20
Marketing 3 30
Purchasing 4 60 IT
104
Bruce
105
David
106
Valli
107
Diana

45. Advantages for OLTP Out with ORM, in with JSONRM
Provide application-specific APIs in JSON format
Use SQL*Net or REST
Data access layer and DB are less tightly coupled
Generate / interpret complex hierarchical data
Read consistency for all data
One SELECT per API

46. JSON in Database 18c/19c JSON as it was meant to be
Stew Ashton
UKOUG Techfest 2019
Blog: stewashton.wordpress.com

47. If we had time… ETL: multi-table insert with nested JSON
OLTP modifications
JSON_MERGEPATCH (19c)
String manipulation
Splitting strings
stewashton.wordpress.com/splitting-strings-a-new-champion/
Listagg() for CLOBs