The Power of Hybrid – Informix & JSON Scott Pickett WW Informix Technical Sales For questions about this presentation, to:

… and the World of Mobile …… ARM  IBM has developed and sold a commercialized port of Informix Dynamic Server for Android running on the ARM processor.  ARM Processors have reduced costs, heat and power usage compared to more complex chip designs, traits which makes it desirable for light, portable, battery-powered devices like smartphones, laptops, tablet, and notepad computers and other embedded systems.  The use of a simpler design allows more efficient multi-core CPUs and higher core counts at lower cost, allowing higher levels of processing power and improved energy efficiency.  Mobile apps today need a reduced footprint and suspended database commits in the case of “no connection” availability.  Who did we sell this to? © 2014 IBM Corporation2

ARM and the Internet of Things, Sensor Data  Shaspa Home Electronics has built a product running Informix on an ARM processor: –The “In Home” Bridge acts as a central hub for data from a large selection of sensors and actuators such as air conditioners, thermostats and lighting systems. –Combined with the performance and reliability of the Informix database server and its native TimeSeries capability, Shapsa is able to store more data and respond to consumer requests, faster and more accurately. –Shaspa Bridge has everything necessary for smart instrumentation, can control any building or environment and implement features required for configuring devices and for the monitoring and control of those systems and appliances. © 2014 IBM Corporation3

The Consumerization of IT  Increasingly, users are driving technological choices. –Open source software represents the world of free –Free is always good, for a consumer, when it is less than a finished product Big companies cannot compete for long in the world of “free” due to Wall St.  Costs are coming down –Hardware speed is at record levels –Cheap memory is plentiful and fastest; we are on the verge of new memory- centric hardware architectures which will force lots of change. Think IPAD, all memory, no disk ……… just bigger systems.  Device adoption is at critical mass. Most everyone has one and users want one adaptable, simple, intelligent, cheap and quick device. –Think of the PALM pilot being integrated with a cell phone. –Think of the Blackberry Corp. not being comprehensive enough to have enough simple to use software in time to compete with the IPhone. –BSOD and 404 are not allowed here. –Familiarity with devices is leading to yet more devices to intelligently manage in a simple push button or autonomic way things not previously managed autonomically. The control of cost is a primary driving factor here. © 2014 IBM Corporation4

Informix in the World of Small and Big Data  The high-speed Internet is the common glue between the consumer world of IT and the business world of IT. –Commercial IT shops cannot keep up with the explosion of consumer IT devices being used for business purposes by their employees. Nor can the business, usually too small to risk losing key people in the already minimally staffed corporation. Not everyone has high-speed Internet. Its gets a little bigger each year. It enables “Big data.”  Informix has unique capabilities for this age: –It can store and process relational/non-relational data natively. –It can store and process high-volume machine generated “Big Data” and scale hugely on small, medium and large platforms. With free included analytics running at the speed of thought. –Or it can shrink to a very small footprint to be embed in devices of all kinds: Disappear ……… 75-150MB footprints in the world of 64 or 128 GB IPADS. –And it’s autonomic nature does all of this simply with a minimum of configuration. And simpler and more autonomic with each release. It always does more. –Works in a world of SSD and large memory quite well. © 2014 IBM Corporation5

Why MongoDB  Rapid Application Prototyping –Schema rides as objects inside the JSON Script, not as a separate item. Allows for constant unit test and retest as logic expands Schemas change quite quickly in modern apps as business needs change. Apps have shorter production lives and even shorter life-cycles as a result Knows and understands JSON natively. No need to normalize per se. 1 to 1 and many to 1 RDBMS relationships do not apply here. –Less bureaucratic Stripped down IS organizations, lean and mean Programmer can be the DBA, system administrator as well.  Scalability –Scalability is not by database tuning per se and adding resources, but rather by sharding (fragmenting the data by a key) the data when necessary across cheap, inexpensive commodity network hardware Automatic data rebalancing across shards by key occurs during the shard operation.  Costs of initial “up and running” are less  Quicker project completions times © 2014 IBM Corporation7

Why MongoDB - Business  600+ Customers worldwide  5,000,000+ Free Software Downloads  Enterprise Big Data customers  Training is Free  Charges for Services as an add on, complimentary business –Backup/Restores –Consulting © 2014 IBM Corporation8

Partnership with IBM and MongoDB  MongoDB and IBM announced a partnership in June 2013  There are many common use cases of interest addressed by the partnership –Accessing JSON Data in DB2, Informix MongoDB using JSON query –Schema-less JSON Data for variety of applications –Making JSON data available for variety of applications –Securing JSON Data  IBM and MongoDB are collaborating in 3 areas: –Open Governance: Standards and Open Source –Technology areas of mutual interest –Products and solutions © 2014 IBM Corporation9

NoSQL Market Data  2011: Market Research Media noted worldwide NoSQL market was expected to reach ~$3.4B by 2018, generating $14B from 2013-2018 with a CAGR of 21%  Comparison: data implies NoSQL market ~$895M –MySQL in 2011 ~$100M "NoSQL is in its infancy, many immature with only community support and limited tool capability; however, expect to see a few used more widely during the next 5 years." "Current adoption of NoSQL in enterprises is 4% in 2010. expected to double by 2014, and grow to 20% by 2015. 10% of existing apps could move to NoSQL. [Sept 2011] "MongoDB is the new MySQL." % NoSQL Enterprise Adoption 2010 2015 2014 © 2014 IBM Corporation10

Informix and MongoDB Have Free Editions EditionsInformixMongoDB FreeDeveloper Innovator-C Basic For PurchaseExpress, Workgroup, Advanced Workgroup, Enterprise, Advanced Enterprise Standard, Enterprise © 2014 IBM Corporation11

MongoDB Subscriptions BasicStandardEnterprise EditionMongoDB MongoDB Enterprise Support9am-9pm local, M-F24x7x365 LicenseAGPLCommercial Emergency PatchesNot IncludedIncluded Price$2,500 / Server / Year$5,000 / Server / Year $7,500 / Server / Year Subscription information obtained from 10Gen site, June 26, 2013. Additional monthly charges for backup services. © 2014 IBM Corporation12

Price Point Comparison Estimate, 3-year cost Dual Core Intel Nehalem Innovator-CExpress (4 core, 8 GB, 2 nodes) Workgroup (16 core, 16 GB, unlimited nodes) Product Cost $0$8,540$19,740 Support Subscription Year 1 24 x 7 x 365 Production System Down Development Call Emergency Patches Free Upgrades $1,680Included Support Renewal Year 2 $1,680$1,708$3,948 Support Renewal Year 3 $1,680$1,708$3,948 Total$5,040$11,956$27,636 MongoDB Enterprise, 3-year cost: $22,500 Subscription information obtained from 10Gen site, June 26, 2013. Retail U.S. prices subject to change, valid as of June 26, 2013. © 2014 IBM Corporation13

Informix as a Hybrid Data Storage Database  Many of MongoDB’s 600+ customers are enterprise that have both relational and non-relational data sitting in the same facility and need to make sense of the data they have for business purposes.  MongoDB has no transaction capabilities in the SQL sense –Informix will enforce transactions on all application statements, with consistency assured at the end of the transaction, MongoDB does not assure here and cannot It is possible that in MongoDB that 4 out of 7 deletes will delete and at the end the other 3 remain, to be dealt with manually …… someday.  Joining noSQL and SQL based queries in the same statement is not possible within MongoDB –Many customers using MongoDB in production circumstances with large document store databases need to join this data to their existing SQL based database repositories for business purposes, often stored, housed and queried separately: Informix can join the two kinds of data  Document store portion stored natively in a BSON as a single row data type within Informix, accessed as JSON, with a set full set of functions to access the document store data. © 2014 IBM Corporation14

Informix NoSQL High Level Functionality Comparison FunctionalityInformixDB2MongoDB SQL Data Native Types TimeSeries Native Type JSON & BSON Native Types Maximum Document Size2048 MB 16 MB Extended JSON Date Support Hybrid Storage Hybrid Applications Sharding Type-less JSON Indexes Typed JSON Indexes Hybrid Storage as native types © 2014 IBM Corporation16

Informix NoSQL Enterprises Functionality Comparison FunctionalityInformixDB2MongoDB Single Statement Transactions Multi-Statement Transactions Lock GranularityRow Database Ability to Join Collection to SQL Tables Ability to Join Collection to Collections Ability to Join Collection to TimeSeries Execute SPL with JSON & SQL Types Text Search Table & Row Level Auditing Data Compression Intelligent Buffer pool and read ahead Web Based Graphical Administration © 2014 IBM Corporation17

Informix NoSQL Analytics & SQL Functionality Comparison FunctionalityInformixDB2MongoDB Cost Based Optimizer Run SQL Analytics Tools (i.e. Cognos) on JSON Document and SQL Tables BLU Acceleration on JSON data OLAP analytics on JSON Collections Distributed access to Tables or Collections Triggers on Collections or Tables Truncate Collections or Tables Enterprise Level Security Online Index Maintenance © 2014 IBM Corporation18

Informix NoSQL Cloud + HA Functionality Comparison FunctionalityInformixDB2MongoDB Single Point in Time Online Backups Read-Only Secondary/Slave Servers Read-Write Secondary/Slave Servers Diskless Secondary/Slave Servers Multiple Secondary/Slave Servers Gridded Tables or Collections Policy Based Failover Automatic Connection Re-Direction Shard Tables or Collections © 2014 IBM Corporation19

JSON Features in Informix Flexible Schema  Use BSON and JSON data type.  Complete row is stored in a single column  BSON, JSON are multi-rep types and can store up to 2GB. Access Key Value Pairs within JSON  Translate the Mongo queries into SQL expressions to access key- value pairs.  Informix has added expressions to extract specific KV pairs.  Select bson_new(data, “{id:1, name:1, addr:1}”) from tab; Indexing  Support standard B-tree index via functional indexing.  Type Less Index  Create index ix_1 on t(bson_extract(data, “id”);  Typed Index  Create index ix_2 on t( bson_value_lvarchar(data, “name”), bson_value_date(data, “expire”);  Informix also supports indexing bson keys with different data types. Sharding  Supports range & hash based sharding.  Informix has built-in technology for replication.  Create identical tables in multiple nodes.  Add meta data for partitioning the data across based on range or hash expressions. © 2014 IBM Corporation21

JSON Features in Informix Select  Limited support required for sharding, Mongo API disallowing joins.  The query on a single sharded table is transformed into a federated UNION ALL query; includes shard elimination processing. Updates (single node)  INSERT: Simple direct insert.  DELETE: DELETE statement with WHERE bson_extract() > ?; or bson_value..() > ?  UPDATE:  bson_update(bson, “update expr”) will return a new bson after applying the bson expression. Simple updates to non_sharded tables will be direct UPDATE statement in a single transaction.  UPDATE tab bsoncol = bson_update(bsoncol, “expr”) where … Updates (sharded env)  INSERT – All rows are inserted to LOCAL shard, replication threads will read logs and replicate to the right shard and delete from local node (log only inserts underway).  DELETE – Do the local delete and replicate the “delete statements” to target node in the background  UPDATE – Slow update via select-delete-insert. Transaction  Each statement is a distinct transaction in a single node environment.  The data and the operation is replicated via enterprise replication. © 2014 IBM Corporation22

JSON Features in Informix Isolation levels  NoSQL session can use any of Informix isolation levels. Locking  Application controls only on the node they’re connected to.  Standard Informix locking semantics will apply when data is replicated and applied on the target shard. Hybrid Access (From MongoAPI to relational tables)  INSERT – All rows are inserted to LOCAL shard, replication threads will read logs and replicate to the right shard and delete from local node (log only inserts underway).  DELETE – Do the local delete and replicate the “delete statements” to target node in the background  UPDATE – Slow update via select-delete-insert. Hybrid Access (From SQL to JSON data)  Directly get binary BSON or cast to JSON to get in textual form.  Use expressions to extract to extract specific key-value pairs. NoSQL collections will only have one BSON object in the table. We can “imply” the expressions when the SQL refers to a column.  SELECT t.c1, t.c2 from t;  SELECT bson_extract(t.data, “{c1:1}”), bson_extract(t.data, “{c2:1}”) from t; © 2014 IBM Corporation23

 Clients exchange BSON document with the server both for queries and data.  Thus, BSON becomes a fundamental data type.  The explicit key-value(KV) pairs within the JSON/BSON document will be roughly equivalent to columns in relational tables.  However, there are differences! –The type of the KV data encoded within BSON is determined by the client –Server is unaware of data type of each KV pair at table definition time. –No guarantees that data type for each key will remain consistent in the collection. –The keys in the BSON document can be arbitrary; –While customers exploit flexible schema, they’re unlikely to create a single collection and dump everything under the sun into that collection. –Due to the limitations of Mongo/NoSQL API, customers typically de-normalize the tables (customer will have customer+customer addr + customer demographics/etc) to avoid joins. Flexible Schema © 2014 IBM Corporation 24

Flexible Schema – Informix Implementation  Informix has a new data type, BSON, to store the data.  Informix also has a JSON data type to convert between binary and text form.  BSON and JSON are abstract data types (like spatial, etc).  BSON and JSON multi-representational types. –Objects up to 4K is stored in data pages. –Larger objects (up to 2GB) are stored out of row, in BLOBs. MongoDB limits objects to 16MB  This is all seamless and transparent to applications. © 2014 IBM Corporation25

Flexible Schema – Informix Implementation  CREATE TABLE customer (data BSON)  BSON is the binary representation of JSON. –It has length and types of the key-value pairs in JSON.  MongoDB drivers send and receive in BSON form. © 2014 IBM Corporation26

Accessing KV pairs within JSON  Extract Expressions/functions returning base type bson_value_bigint(BSON, “key”); bson_value_lvarchar(bsoncol, “key.key2”); bson_value_date(bsoncol, “key.key2.key3”); bson_value_timestamp(bsoncol, “key”) bson_value_double(bsoncol, “key”); bson_value_boolean(bsoncol, “key”); bson_value_array(bsoncol, “key”); bson_keys_exist(bsoncol, “key”); bson_value_document(bsoncol, “key”) bson_value_binary(bsoncol, “key”) bson_value_objectid(bsoncol, “key”)  Expression returning BSON subset. Used for bson indices. bson_extract(bsoncol, “projection specification”)  Expressions to project out of SELECT statement. bson_new(bsoncol, “{key1:1, key2:1, key3:1}”); bson_new(bsoncol, “{key5:0}”); © 2014 IBM Corporation27

Accessing Key-Value (KV) pairs within JSON. Mongo QuerySQL Query db.customers.find();SELECT SKIP ? Data::bson FROM customers db.customers.find({},{num:1,na me:1}); SELECT SKIP ? bson_new( data, '{ "num" : 1.0, "name" : 1.0}')::bson FROM customers db.customers.find({}, {_id:0,num:1,name:1}); SELECT SKIP ? bson_new( data, '{_id:0.0, "num" : 1.0, "name" : 1.0}')::bson FROM customers db.customers.find({status:”A”})SELECT SKIP ? data FROM customers WHERE bson_extract(data, ‘status') = “A” db.customers.find({status:”A”}, {_id:0,num:1,name:1}); SELECT SKIP ? bson_new( data, '{ "_id" : 0.0, "num" : 1.0, "name" : 1.0}')::bson FROM customers WHERE bson_extract(data, 'name') = “A” © 2014 IBM Corporation28

Indexing (1)  Supports B-Tree indexes on any key-value pairs.  Indices could be on simple basic type (int, decimal) or BSON  Indices could be created on BSON and use BSON type comparison  Listener translates ensureIndex() to CREATE INDEX  Listener translates dropIndex() to DROP INDEX © 2014 IBM Corporation29

Indexing (2) Mongo Index Creation Statement SQL Query Statement db.customers.ensureIndex({ord erDate:1}) CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w", "key" : {"x" : [ 1.0, "$extract"]}, "name" : "x_1", "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({ord erDate:1, zip:-1}) CREATE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v", "key" : { "c1" : [ 1.0, "$extract"], "c2" : [ -1.0, "$extract"]}, "name" : "c1_1_c2__1", "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({ord erDate:1}, {unique:true) CREATE UNIQUE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v", "key" : { "c1" : [ 1.0, "$extract"], "c2" : [ -1.0, "$extract"]}, "name" :"c1_1_c2__1", "unique" : true, "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64 © 2014 IBM Corporation30

Indexing db.w.find({x:1,z:44},{x:1,y:1,z:1})  Translates to:  SELECT bson_new( data, '{ "x" : 1.0, "y" : 1.0, "z" : 1}  FROM w  WHERE ( bson_extract(data, 'x') = '{ "x" : 1.0 }'::json::bson ) AND  ( bson_extract(data, 'z') = '{ "z" : 44.0 }'::json::bson)  Estimated Cost: 2  Estimated # of Rows Returned: 1  1) keshav.w: SEQUENTIAL SCAN  Filters: (informix.equal(informix.bson_extract(keshav.w.data,'z' ),UDT )  AND informix.equal(informix.bson_extract(keshav.w.data,'x' ),UDT)) © 2014 IBM Corporation31

Indexing  Functional Index is built on bson expressions CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w", "key" : {"x" : [ 1.0, "$extract"]}, "name" : "x_1", "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64  Listener is aware of the available index and therefore generates right predicates:  db.w.find({x:1});  gets translated to  SELECT SKIP ? data FROM w WHERE bson_extract(data, 'x') = ? © 2014 IBM Corporation32

Indexing - sqexplain  db.w.find({x:5,z:5}, {x:1,y:1,z:1})  Translates to:  SELECT bson_new( data, '{ "x" : 1.0, "y" : 1.0, "z" : 1.0}')::bson  FROM w  WHERE ( bson_extract(data, 'x') = '{ "x" : 5.0 }'::json::bson )  AND ( bson_extract(data, 'z') = '{ "z" : 5.0 }'::json::bson )  Estimated Cost: 11  Estimated # of Rows Returned: 1  1) keshav.w: INDEX PATH  Filters: informix.equal(informix.bson_extract(keshav.w.data,'z' ), UDT )  (1) Index Name: keshav.w_x_1  Index Keys: informix.bson_extract(data,'x') (Serial, fragments: ALL)  Lower Index Filter: informix.equal(informix.bson_extract(keshav.w.data,'x' ),UDT ) © 2014 IBM Corporation33

MongoDB SHARDING (roughly)  Shard a single table by range or hashing.  Mongos will direct the INSERT to target shard.  Mongos tries to eliminate shards for update, delete, selects as well.  FIND (SELECT) can happen ONLY a SINGLE table.  Mongos works as coordinator for multi-node ops.  Once a row is inserted to a shard, it remains there despite any key update.  No transactional support on multi-node updates. –Each document update is unto its own. © 2014 IBM Corporation36

Informix Sharding App Server Mongo Driver Listener JDBC Enterprise replication + Flexible Grid App Server Mongo Driver Listener JDBC App Server Mongo Driver Listener JDBC App Server Mongo Driver Listener JDBC Location/1 Sales/1 Customer/1 Location/2 Sales/2 Customer/2 Location/3 Sales/3 Customer/3 Location/4 Sales/4 Customer/4 Location/5 Sales/5 Customer/5 Location/6 Sales/6 Customer/6 Informix/1 Informix/3 Informix/4 Informix/5 Informix/6 Informix/2 © 2014 IBM Corporation37

Informix Sharding + High Availability App Server Mongo Driver Listener JDBC Enterprise replication + Flexible Grid App Server Mongo Driver Listener JDBC App Server Mongo Driver Listener JDBC App Server Mongo Driver Listener JDBC Informix/1 Primary Informix/1 SDS/HDR Informix/1 RSS Informix/2 Primary Informix/2 SDS/HDR Informix/2 RSS Informix/3 Primary Informix/3 SDS/HDR Informix/3 RSS Informix/4 Primary Informix/4 SDS/HDR Informix/4 RSS Informix/5 Primary Informix/5 SDS/HDR Informix/5 RSS Informix/6 Primary Informix/6 SDS/HDR Informix/6 RSS © 2014 IBM Corporation38

Sharding – Informix Implementation  Shard a single table by range or hashing. cdr define shard myshard mydb:usr1.mytab \ –type=delete –key=”bson_get(bsoncol, ‘STATE’)” –strategy=expression \ versionCol=version \ servA “in (‘TX’, ‘OK’)” \ servB “in (‘NY’,’NJ’) “ \ servC “in (‘AL’,’KS’) “ \ servD remainder cdr define shard myshard mydb:usr1.mytab \ –type=delete –key=state –strategy=hash --versionCol=version \ servA servB servC servD  MongoDB shard a single table by hashing. –sh.shardCollection( "records.active", { a: "hashed" } ) © 2014 IBM Corporation39

Sharding – Informix Implementation  Shard a single table by range or hashing.  Sharding is transparent to application.  Each CRUD statement will only touch a single table. –Limitation of MongoDB…Makes it easier for Informix. –Lack of joins is a big limitation for SQL applications.  Lacks transactional support for distributed update. © 2014 IBM Corporation40

Informix Sharding  Identical table is created on each node and meta data is replicated on each node.  Schema based replication is the foundation for our sharding.  CRUD operations can go to any node. –We’ll use replication and other techniques to reflect the data in the target node, eventually. –Replication is asynchronous in Informix presently. –Informix also has synchronous replication … not used for sharding now. © 2014 IBM Corporation41

SELECT on sharded tables  The query can be submitted to any of the nodes via the listener.  That node acts as the “coordinator” for the distributed query.  It also does the node elimination based on the query predicate.  After that, the query is transformed to UNION ALL query SELECT SKIP ? bson_new( data, '{"_id":0.0,"num":1.0,"name" : 1.0}')::bson FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X” is transformed into: SELECT SKIP ? bson_new( data, '{"_id":0.0,"num":1.0,"name" : 1.0}')::bson FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X” UNION ALL SELECT SKIP ? bson_new( data, '{"_id":0.0,"num":1.0,"name" : 1.0}')::bson FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X” © 2014 IBM Corporation42

INSERT: Single node  If necessary, automatically create database & table (collection) on the application INSERT  Collections: CREATE TABLE t(a GUID, d BSON);  GUID column is needed to ensure unique row across the SHARDs. –Also used as a Primary Key (PK) in Enterprise Replication.  Client application inserts JSON, client API converts this into BSON, generates ‘_id’ (object id) if necessary & sends to server over JDBC/ODBC.  Server saves the data into this table as BSON, with an automatically generated GUID © 2014 IBM Corporation43

UPDATE: Single node  Simple set, increment, decrement updates are handled directly by the server. Mongo: db.w.update({x: {$gt:55}}, {$set:{z:9595}}); SQL: UPDATE w SET data = bson_update(data, “ {$set:{z:9595}}” ) WHERE bson_extract(data, 'x') > “{x:55}”::bson ?  bson_update is a built-in expression updating a BSON document with a given set of operations.  Complex updates are handled via select batch, delete, insert.  Informix always updates all the rows or no rows, under a transaction. © 2014 IBM Corporation45

INSERT: shard implementation  If the table is sharded: –Insert the data into local table as usual. –Replication threads in the background will evaluate the log record to see which rows will have moved. –Replication thread will move the necessary row to target and delete from the local table. –For each inserted row ending up in non-local shard, simply generate the logical log and avoid insert into local table & indices. © 2014 IBM Corporation46

DELETE : shard implementation  Application delete could delete rows from any of the shards in the table.  DELETE will come in as shard_delete() procedure. –execute procedure shard_delete(tabname, delete_stmt); –This procedure will issue delete locally. –It will then INSERT the delete statement into a “special” shard delete table (single for all tables). –Enterprise replication will propagate the delete to applicable target systems. © 2014 IBM Corporation47

UPDATE: shard implementation  When the update have to be done on multiple nodes: –Client application does UPDATE and CLIENT API converts that into three operations: SELECT, DELETE & INSERT Inserts JSON, client API converts this into BSON & sends to server.  GUID column is needed to ensure unique row across the SHARD © 2014 IBM Corporation48

Transactions (single node)  Mongo does not have the notion of transactions. –Each document update is atomic, but not the app statement  For the first release of Informix-NoSQL –By default, JDBC listener simply uses AUTO COMMIT option –Each server operation INSERT, UPDATE, DELETE, SELECT will be automatically be committed after each operation. No locks are held across multiple application operations. © 2014 IBM Corporation49

Transactions (sharded environment)  In sharded environment, mongo runs databases via two different instances: mongos and mongod. –Mongos simply redirects operations to the relevant mongod. –No statement level transactional support.  Informix –Does not have the 2-layer architecture –The server the application connected to becomes the transaction coordinator –Does have the 2-phase commit transaction support Unused for NoSQL presently. –SELECT statement goes thru distributed query infrastructure(ISTAR) as always –INSERT, UPDATE, DELETE direct thru Enterprise Replication as always. © 2014 IBM Corporation50

Isolation levels  Default isolation level is DIRTY READ.  Change this directly or sysdbopen()  You can also use USELASTCOMMITTED variable in the database server configuration file ($ONCONFIG).  If you’re using procedures for executing multi-statement transactions, you can set it within your procedure. © 2014 IBM Corporation51

Locking  Page level locking is the default.  You can change it to ROW level locks easily. –ALTER TABLE jc MODIFY lock mode (row); –DEF_TABLE_LOCKMODE server configuration file variable.  SET LOCK MODE can be set via sysdbopen()  Each statement is executed with auto-commit and locking semantics will apply there. © 2014 IBM Corporation52

MongoDB Drivers – The Power of Hybrid  One of the big secrets here is what the MongoDB company, formerly known as 10Gen, does not have to do.  Lots of user community support for the product and its drivers.  With the new MongoDB client interface to Informix as a hybrid data store, there are a series of MongoDB officially supported drivers, They are formally listed as the MongoDB “ecosystem” and linked to on the website for MongoDB.  These have support from MongoDB directly as a corporation and also broad support from the MongoDB user community  And now, those users can be part of the Informix “ecosystem” as well.  A whole new universe: –to go, where Informix has not gone before. © 2014 IBM Corporation54

Client Applications  New Wire Protocol Listener supports existing MongoDB drivers  Connect to MongoDB or Informix with same application! MongoDB native Client MongoDB web browser Mobile Applications MongoDB Wire Protocol Informix 12.1 MongoDB driver MongoDB © 2014 IBM Corporation55

MongoDB Drivers – Officially Supported (1)  With the new MongoDB client interface, with Informix as a hybrid data store, there comes a whole new world of drivers necessary for programmers to access and interface Informix, thru mongo:  C –C Driver LibraryC Driver Library  C++ –C++ Driver LibraryC++ Driver Library –Download and Compile C++ DriverDownload and Compile C++ Driver –Getting Started with the C++ DriverGetting Started with the C++ Driver –SQL to mongo Shell to C++SQL to mongo Shell to C++ –C++ BSON Helper FunctionsC++ BSON Helper Functions –C++ BSON Array ExamplesC++ BSON Array Examples  C# –CSharp Language CenterCSharp Language Center –CSharp Community ProjectsCSharp Community Projects –Getting Started with the CSharp DriverGetting Started with the CSharp Driver –CSharp Driver LINQ TutorialCSharp Driver LINQ Tutorial –Serialize Documents with the CSharp DriverSerialize Documents with the CSharp Driver –CSharp Driver TutorialCSharp Driver Tutorial  Erlang –Erlang Language CenterErlang Language Center © 2014 IBM Corporation56

MongoDB Drivers – Officially Supported (2)  Java –Java Language CenterJava Language Center –Java TypesJava Types –Java Driver ConcurrencyJava Driver Concurrency –Java Driver Replica Set SemanticsJava Driver Replica Set Semantics –Getting Started with Java DriverGetting Started with Java Driver –Java Driver and Aggregation FrameworkJava Driver and Aggregation Framework –Java DBObject to Perform SavesJava DBObject to Perform Saves  JavaScript –JavaScript Language CenterJavaScript Language Center  Node.js –Node.js Language CenterNode.js Language Center  Perl –Perl Language CenterPerl Language Center –Contribute to the Perl DriverContribute to the Perl Driver  PHP –PHP Language CenterPHP Language Center –PHP Libraries, Frameworks and ToolsPHP Libraries, Frameworks and Tools  Python –Python Language CenterPython Language Center –Write a Tumblelog Application with Flask and MongoEngineWrite a Tumblelog Application with Flask and MongoEngine  Ruby –Ruby Language CenterRuby Language Center –Ruby External ResourcesRuby External Resources –MongoDB Data Modeling and RailsMongoDB Data Modeling and Rails –Getting Started with RailsGetting Started with Rails –Getting Started with Rails 3Getting Started with Rails 3  Scala –Scala Language CenterScala Language Center © 2014 IBM Corporation57

MongoDB Drivers – Community Supported (1)  There is a large and vast MongoDB user community for the support and maintenance of various drivers used in interfacing MongoDB not officially supported by MongoDB.  They are programmers or dba’s controlling a database in many cases and are interested in getting solutions to complex programming and database technical issues.  They experiment and try out new technologies, as such they represent a new frontier for Informix. They support the drivers they use thru the standard modern day Internet forums, chat rooms, code repositories, etc. –These ecosystems outside of the MongoDB official environment are testament to the large user community following it has developed. –In part, its why they presently have 25% market share in the document store aspect of the database marketplace. –The drivers are numerous ……… worth looking at briefly. –Perhaps one of your customers is using one of these ……. © 2014 IBM Corporation58

MongoDB Drivers – Community Supported (2)  ActionScript3 ActionScript3 –http://www.mongoas3.comhttp://www.mongoas3.com  C –libmongo-clientlibmongo-client  C# and.NET Clojure C# and.NET –See the MongoDB Java Language CenterJava Language Center  ColdFusion ColdFusion –Blog post: Part 1 | Part 2 | Part 3Part 1Part 2Part 3 –http://github.com/virtix/cfmongodb/tr ee/0.9http://github.com/virtix/cfmongodb/tr ee/0.9 –http://mongocfc.riaforge.org/http://mongocfc.riaforge.org/  D –Port of the MongoDB C Driver for DPort of the MongoDB C Driver for D –DartDart  Delphi Delphi –Full featured Delphi interface to MongoDB built on top of the mongodb.org supported C driverC driver –PebongoPebongo Early stage Delphi driver for MongoDB –TMongoWireTMongoWire Maps all the VarTypes of OleVariant to the BSON types, implements IPersistStream for (de)serialization, and uses TTcpClient for networking  Entity –entity driver for mongodb on Google Code, included within the standard Entity Libraryentity driver for mongodb © 2014 IBM Corporation59

MongoDB Drivers – Community Supported (3)  Erlang –emongo - An Erlang MongoDB driver that emphasizes speed and stability. “The most emo of drivers.”emongo –Erlmongo - an almost complete MongoDB driver implementation in ErlangErlmongo  Factor Factor  Fantom Fantom  F# F#  Go –gomongogomongo –go-mongogo-mongo –MgoMgo  Groovy –gmongogmongo Also see the Java Language CenterJava Language Center –Blog Post: Groovy on Grails in the land of MongoDBGroovy on Grails in the land of MongoDB –Grails Bates:Grails Bates: Grails business audit trails plugin  Javascript Javascript  Lisp –https://github.com/fons/cl-mongohttps://github.com/fons/cl-mongo  Lua –LuaMongo on Google CodeLuaMongo on Google Code –LuaMongo fork on GithubLuaMongo  MatLab –mongo-matlab-drivermongo-matlab-driver  node.js node.js © 2014 IBM Corporation60

MongoDB Drivers – Community Supported (4)  Objective C –NuMongoDBNuMongoDB –ObjCMongoDBObjCMongoDB  OCaml –Mongo.mlMongo.ml  PHP –Asynchronous PHP driver using libevent|Asynchronous PHP driver using libevent|  PowerShell –mosh Powershell provider for MongoDBmosh Powershell provider for MongoDB –mdbc module cmdlets using the officially supported.NET drivermdbc module cmdlets using the officially supported.NET driver –Doug Finke’s blog post on using the original community C# driver with PowerShellDoug Finke’s blog post on using the original community C# driver with PowerShell  Prolog Prolog  Python –MongoEngineMongoEngine –MongoKitMongoKit –Django-nonrelDjango-nonrel –Django-mongodbDjango-mongodb –Django-mongonautDjango-mongonaut  R –rmongodbrmongodb Full featured R interface to MongoDB built on top of the mongodb.org supported C driverC driver –RMongo for R clientRMongo for R client R client to interface with MongoDB  Racket (PLT Scheme) Racket (PLT Scheme) –DocsDocs  REST REST © 2014 IBM Corporation61

MongoDB Drivers – Community Supported (5)  Ruby –MongoMapperMongoMapper –rmongo Ruby driverrmongo Ruby driver An event-machine-based Ruby driver for MongoDB –jmongojmongo A thin ruby wrapper around the mongo-java-driver for vastly better jruby performance. –em-mongoem-mongo EventMachine MongoDB Driver (based off of RMongo).  Scala –Java Language CenterJava Language Center  Smalltalk –Squeaksource MongotalkSqueaksource Mongotalk –Dolphin SmalltalkDolphin Smalltalk © 2014 IBM Corporation62

Nothing Like the Present …….  In addition to connectivity thru the wire listener, we are offering meaningful enhancements to Informix in 12.10.xC2 and beyond for JSON capabilities: –Native JSON and BSON data storage types With functions necessary to manipulate the data as well, in and out. –Hybrid Applications Execute SQL within JSON apps Execute JSON within SQL based apps. –Hybrid database Store Relational and non-relational data within the same database Join both data organization types within SQL –Standard MongoDB application performance scaling techniques via sharding We do it too, and have for years. We call it Enterprise Replication, and now we store on a node (shard) by key value and load balance the storage across the shards as does MongoDB for scalability. –Use standard Informix utilities and techniques on both. Backup/restore Enterprise Replication Compression Time Series, Etc. © 2014 IBM Corporation63

Summary: What is JSON’s Role in the Enterprise?  Flexible schema is agile, liberating for application developers  But will we abandon years of expertise in data modeling/normalization theory? –How to maintain control in an enterprise, mission critical DBMS?  Identification of appropriate applications is critical  Application deployment procedures need to adapt –New controls to prevent schema chaos –Application development groups need to implement controls  When combining with application that uses relational schema –Identify portions that need to remain dynamic –Allocate/accommodate space for that as JSON –Future – combination of SQL and JSON will make this easier © 2014 IBM Corporation64

What Data Store Format to Use?  Consider NoSQL JSON when –Application and schema subject to frequent changes –Prototyping, early stages of application development –De-normalized data has advantages Entity/document is in the form you want to save  Read efficiency – return in one fetch without sorting, grouping, or ORM mapping –"Systems of Engagement" Less stringent "CAP" requirements in favor of speed  Eventual consistency is good enough Social media  Relational still best suited when these are critical –Data normalization to Eliminate redundancy Ensure master data consistency –Database enforced constraints –Database-server JOINs on secondary indexes © 2014 IBM Corporation65

Data Normalization - Choose the Right Solution NoSQL JSON - Two approaches Embedded (de-normalized) Using references {dept: "A10", deptname:"Shipping", manager:"Samuel", emp:[ {empno:"000999", lastname:"Harrison", edlevel:"16"}, {empno:"370001", lastname:"Davis", edlevel:"12"} ] proj:[ {projno:"397", projname:"Site Renovation", respemp:"370001" }, {projno:"397", projname:"Site Renovation", respemp:"370001"} … ] } If you need normalization and database-enforced constraints, JSON may not be best choice {_id dept … ) {_id dept … ) {_id emp dept ref … } {_id emp dept ref … } {_id dept emp ref … ) {_id dept emp ref … ) Requires application- side join Chance for data redundancy create table department ( dept char(3), deptname varchar(40), manager varchar(20), empno integer, projno integer); create table employee( empno integer lastname varchar(20) edlevel smallint ); create table project( projno integer, projname varchar(20), respemp integer); Unique indexes on department.dept, employee.empno, project.projno Composite index on department (dept, empno, projno) Constraints (minimally) : Between department.empno and employee.empno, Between department.projno and project.projno  Possibility exists of mistyped data with the NoSQL schema.  Application joins a must  No mismatched data types on Informix.  No Application Joins. © 2014 IBM Corporation66

Using Schema-less Model In a Traditional Environment  During prototyping, early development, validation –As system stabilizes, stable schema parts could be moved to relational columns  For cases where web message or document will be retrieved as-is –Yet retrieval on internal fields is needed  When parts of the schema will always be dynamic © 2014 IBM Corporation67

JSON and Informix – Complementary Technologies  Does NoSQL mean NoDBA? NoInformix? –Definitely not - the relational database isn’t going away anytime soon –IBM sees JSON as becoming a complementary technology to relational  Transactional atomicity is essential for mission critical business transactions –Informix JSON solution brings commits, transaction scope  Informix was the first to store relational and non-relational datatypes side by side 16+ years ago and allow hybrid apps.  Choice for developers. © 2014 IBM Corporation68

JSON and Informix – Complementary Technologies  What do we offer a developer – Think Hybrid: –Hybrid database, –Hybrid applications, –Multi and single statement transactions, –Partial transactions, –Compression, –Free backup/restore utilities (with or without a free provided storage manager to disk, tape or remote storage), –Sharding, –Scale in/out, –Replication, –Shared disk, –Connection manager, –User Definable Data Types –Spatial, Time Series, Basic Text Search –Full JSON/BSON support natively –Configurable Autonomics –Free Informix Developer Edition  We are the hybrid applications solution … again. Right now. Informix © 2014 IBM Corporation69

JSON and Informix – User Experience Quotes  “This is just the beginning of the possibilities for this hybrid development structure. I believe this structure will be more common in the future of app creation than simply choosing only SQL or only NoSQL. The more tools are given to developers, the more creative they can be and more choices they have for tackling their problems. That is what developers want, reliable and varied tools to solve their problems. Informix is working hard to provide that with the JSON Listener. This technology will help drive the next generation of web based applications in an industry where time is precious, rapid prototyping is preferred and scalability is key. With NoSQL capabilities infused to IDS, Informix is off to a great start.” –  Hector Avala – Software Developer and College Student © 2014 IBM Corporation70

High Level Feature Description  Add Informix support for Mongo’s Aggregation Pipeline –Allow users to run aggregation queries on BSON documents –Enables users to do more than just retrieve back the same documents they inserted. Now they can run queries to aggregate and summarize groups of documents in a collection.  Motivation –Mongo compatibility –Customer requirements (IBM API Manager team)  Target Market –New customers - Mongo DB/NoSQL © 2014 IBM Corporation72

Mongo’s Aggregation Pipeline http://docs.mongodb.org/manual/core/aggregation-pipeline/ “The aggregation pipeline is a framework for data aggregation modeled on the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into an aggregated results.” © 2014 IBM Corporation73

Informix NoSQL Aggregation Queries MongoDB Client db.collection.aggregate([ {$match:{…}},{$group:{…}}, {$sort:{…}]) SELECT … FROM collection WHERE … GROUP BY … ORDER BY … Informix JSON listener Informix Database Server Informix JDBC driver BSON The JSON listener will handle pipelining by nesting various derived table queries inside of each other. © 2014 IBM Corporation74

Aggregation Pipeline Operations $matchWHERE clause. Uses standard Mongo query operators that are already supported in xC2. $groupGROUP BY clause. See next slide for group (aggregation) operators. $unwindNew Informix function bson_unwind was created. Takes an array of documents and returns them as a stream of documents $projectProjection clause For xC3, we just support specifying which fields of the documents to include in the projection. Mongo also has a set of projection operators to add new computed fields, as well as rename or rearrange fields in your documents. This support will be added in xC4. $limitLIMIT keyword in projection clause. $skipSKIP keyword in projection clause. $sortORDER BY clause. Mongo also has a $geoNear pipeline operator, which is not supported at this time. © 2014 IBM Corporation75

Group aggregation operators $sumUse bson_value_double to extract value and SUM $avgUse bson_value_double to extract value and compute average $maxUse bson_extract and then apply MAX function directly on BSON $minUse bson_extract and then apply MIN function directly on BSON $addToSetNew Informix addToSet function that returns an array of all the unique values for the selected field among for each document in that group. $pushNew Informix push function that returns an array of all values for the selected field among for each document in that group. Mongo also has a $first and $last group operators, which are not supported at this time. © 2014 IBM Corporation76

bson_unwind() - 12.10.xC3  One bson_unwind() per query block  Unwind one array per query block  Support Skip M / First N on unwind iterator  bson_unwind() is allowed only on column references ( e.g bson_unwind(bson_concat(...), “$tag”) not allowed )  Materialize Derived table with bson_unwind()  PDQ is disabled © 2014 IBM Corporation78

Future Work  Support for aggregation queries on sharded collections  Support for Mongo projection operators  Support for $first and $last group operators  Support Mongo’s map reduce aggregation paradigm © 2014 IBM Corporation82

Links, Helpful Information and Resources  Informix CTP wiki Informix CTP wiki  Informix Developer Edition free download: Informix Developer Edition –http://www-01.ibm.com/software/data/informix/downloads.htmlhttp://www-01.ibm.com/software/data/informix/downloads.html  Informix Virtual Appliance  MongoDB site MongoDB  MongoDB download: http://www.mongodb.org/downloadshttp://www.mongodb.org/downloads  MongoDB drivers MongoDB drivers  John Miller’s Blog John Miller’s Blog  John Miller’s NoSQL Chat With The Labs session John Miller’s NoSQL Chat With The Labs session  Keshava Murthy’s Blog Keshava Murthy’s Blog  Informix NoSQL Whitepaper Informix NoSQL Whitepaper © 2014 IBM Corporation83

Appendix A – MongoDB Shell Supported db.collection commands (1) MongoDB CommandSupportedDetails aggregateno countyes createIndexyesYou can use the MongoDB createIndex syntax to create an index that works for all data types. For example: db.collection.createIndex( { zipcode: 1 } ) db.collection.createIndex( { state: 1, zipcode: -1} ) You can use the Informix createIndex syntax to create an index for a specific data type. For example: db.collection.createIndex( { zipcode : [1, “$int”] } ) db.collection.createIndex( { state: [1, “$string”], zipcode: [-1, “$int”] } ) Tip: If you are creating an index on a field that has a fixed data type, you can get better query performance by using the Informix createIndex syntax. dataSizeyes distinctno dropyes © 2014 IBM Corporation85

Appendix A – MongoDB Shell Supported db.collection commands (2) MongoDB CommandSupportedDetails dropIndexyes dropIndexesyes ensureIndexyesYou can use the MongoDB ensureIndex syntax to create an index that works for all data types. For example: db.collection.ensureIndex( { zipcode: 1 } ) db.collection.ensureIndex( { state: 1, zipcode: - 1} ) You can use the Informix ensureIndex syntax to create an index for a specific data type. For example: db.collection.ensureIndex( { zipcode : [1, “$int”] } ) db.collection.ensureIndex( { state: [1, “$string”],zipcode: [-1, “$int”] } ) Tip: If you are creating an index on a field that has a fixed data type, better query performance can be had by using the Informix ensureIndex syntax. findyes findAndModifyyes © 2014 IBM Corporation86

Appendix A – MongoDB Shell Supported db.collection commands (3) MongoDB CommandSupportedDetails findOneyes getIndexesyes getShardDistributionno getShardVersionno getIndexStatsno groupno indexStatsno insertyes isCappedyes This command returns false because capped collections are not directly supported in Informix. mapReduceno © 2014 IBM Corporation87

Appendix A – MongoDB Shell Supported db.collection commands (4) MongoDB CommandSupportedDetails reIndexno removenoThe justOne option is not supported. This command deletes all documents that match the query criteria. renameCollectionyes saveyes statsyes storageSizeyes totalSizeyes updateyesThe multi option is supported only if update.one.enable=true in the jsonListener.properties file. If update.one.enable=false, all documents that match the query criteria are updated. validateno © 2014 IBM Corporation88

Appendix B - MongoDB Command Support (1)  User Commands - Aggregation Commands  User Commands - Query and Write Operation Commands MongoDB CommandSupportedDetails aggregateno countyesCount items in a collection distinctno groupno mapReduceno MongoDB CommandSupportedDetails evalno findAndModifyno getLastErroryesError code returned if any by an operation getPrevErrorno resetErrorno textno © 2014 IBM Corporation90

Appendix B - MongoDB Command Support (2)  Database Operations – Authentication Commands  Database Operations – Diagnostic Commands MongoDB CommandSupportedDetails logoutno authenticateno MongoDB CommandSupportedDetails Build infoyesWhenever possible, the Informix output fields are identical to MongoDB. There are additional fields that are unique to Informix. collStatsyesThe value of any field that is based on the collection size is an estimate, not an exact value. For example, the value of the field 'size' is an estimate. connPoolStatsno © 2014 IBM Corporation91

Appendix B - MongoDB Command Support (3)  Database Operations – Diagnostic Commands (cont’d) MongoDB CommandSupportedDetails cursorInfono dbStatsyesThe value of any field that is based on the collection size is an estimate, not an exact value. For example, the value of the field 'dataSize' is an estimate. featuresno getCmdLineOptsyes getLogno hostInfoyesThe memSizeMB, totalMemory, and freeMemory fields indicate the amount of memory available to the Java virtual machine (JVM) that is running, not the operating system values. indexStatsno © 2014 IBM Corporation92

Appendix B - MongoDB Command Support (4)  Database Operations – Diagnostic Commands (cont’d) MongoDB CommandSupportedDetails listCommandsno listDatabasesyesThe value of any field that is based on the collection size is an estimate, not an exact value. For example, the value of the field 'sizeOnDisk' is an estimate. pingyes serverStatusyes topno © 2014 IBM Corporation93

Appendix B - MongoDB Command Support (5)  Database Operations – Instance Administration Commands (cont’d) MongoDB CommandSupportedDetails cloneno cloneCollectionno cloneCollectionAsCappedno collModno compactno convertToCappedno. copydbNo createyesInformix does not support the following flags: capped autoIndexID size max dropyesInformix does not lock the database to block concurrent activity. dropDatabaseyes © 2014 IBM Corporation94

Appendix B - MongoDB Command Support (6)  Database Operations – Instance Administration Commands (cont’d) MongoDB CommandSupportedDetails dropIndexesyes The MongoDB deleteIndexes command is equivalent. filemd5yes fsyncno getParameterno logRotateno reIndexno renameCollectionno. repairDatabaseNo setParameterno shutdownyesThe timeoutSecs flag is supported. In the Informix, the timeoutSecs flag determines the number of seconds that the wire listener waits for a busy client to stop working before forcibly terminating the session. The force flag is not supported. touchno © 2014 IBM Corporation95

Appendix B - MongoDB Command Support (7)  Database Operations – Replication Commands MongoDB CommandSupportedDetails addShardyesThe MongoDB maxSize and name options are not supported. In addition to the MongoDB command syntax for adding a single shard server, you can use the Informix specific syntax to add multiple shard servers in one command by sending the list of shard servers as an array. enableShardingyesThis action is not required for Informix and therefore this command has no affect for Informix. flushRouterConfigno isdbgridyes listShardsyesThe equivalent Informix command is cdr list server. movePrimaryno. removeShardno shardCollectionyesThe equivalent Informix command is cdr define shardCollection. The MongoDB unique and numInitialChunks options are not supported. shardingstateno splitno © 2014 IBM Corporation96

Appendix B - MongoDB Command Support (8)  Array query operations  Comparison query operators MongoDB CommandSupportedDetails $elemMatchno $sizeyesSupported for simple queries only. The operator is only supported when it is the only condition in the query document. MongoDB CommandSupportedDetails $allyesSupported for primitive values and simple queries only. The operator is only supported when it is the only condition in the query document. $gtyesGreater than $gteyesGreater than or equal to $inyes $ltyesLess than $lteyesLess than or equal to $neyesNot equal © 2014 IBM Corporation97

Appendix B - MongoDB Command Support (9)  Comparison query operators (cont’d)  Element query operators  Geospatial query operators –The geospatial query operators are not supported.  JavaScript query operators –The JavaScript query operators are not supported. MongoDB CommandSupportedDetails $ninyesSupported for primitive values and simple queries only. The operator is only supported when it is the only condition in the query document. Not in operator or not exists $queryyesReturns documents matching its argument MongoDB CommandSupportedDetails $existsyesBoolean, yes or no (1 or 0) $modyesModulus, selects documents matching $typeyes © 2014 IBM Corporation98

Appendix B - MongoDB Command Support (10)  Logical Query Operators  Subdocument query elements –Queries that compare non-primitive data types, such as subdocuments, are not supported. MongoDB CommandSupportedDetails $andyesLogical and $oryesLogical or $notyesLogical not $noryesLogical nor © 2014 IBM Corporation99

Appendix B - MongoDB Command Support (11)  Bitwise Operators  Field Update Operators  Isolation update operators –The isolation update operators are not supported. MongoDB CommandSupportedDetails $bityes MongoDB CommandSupportedDetails $incyes $renameyes $setOnInsertyes $setyes $unsetyes © 2014 IBM Corporation100

Appendix B - MongoDB Command Support (12)  Projection Operators - Comparison query operators MongoDB CommandSupportedDetails $no $elemMatchno $sliceno $commentno $explainyes $hintyes $maxScanno $maxno $minno $orderbyyesNot supported for sharded data. $returnkeyno $showdiskLocno $snapshotno © 2014 IBM Corporation101

Appendix C – MongoDB API’s Supported  MongoDB community drivers can be used to store, update, and query JSON Documents with Informix as a JSON data store: –Java, –C/C++, –Ruby, –PHP, –PyMongo © 2014 IBM Corporation103

Appendix D – MongoDB Supported Command Utilities & Tools  You can use the MongoDB shell and any of the standard MongoDB command utilities and tools.  Supported MongoDB shell is version 2.4.3.  You can run the MongoDB mongodump and mongoexport utilities against MongoDB to export data from MongoDB to Informix.  You can run the MongoDB mongorestore and mongoimport utilities against Informix to import data to MongoDB from Informix. © 2014 IBM Corporation105

Similar Technical Naming – Informix & Mongo DB (1) MongoDB ConceptInformix ConceptDescription collectiontableThis is the same concept. In Informix this type of collection is sometimes referred to as a JSON collection. A JSON collection is similar to a relational database table, except it does not enforce a schema. documentrecordThis is the same concept. In Informix, this type of document is sometimes referred to as a JSON document. fieldcolumnThis is the same concept. © 2014 IBM Corporation106

Similar Technical Naming – Informix & Mongo DB (2) MongoDB ConceptInformix ConceptDescription master / slaveprimary server /secondary server This is the same concept. However, an Informix secondary server has additional capabilities. For example, data on a secondary server can be updated and propagated to primary servers. replica sethigh-availability cluster This is the same concept. However, when the replica set is updated, it then sent to all servers, not only the primary server. sharded cluster This is the same concept. In Informix, a sharded cluster consists of servers that are sometimes referred to as shard servers. sharded keyshard keyThis is the same concept. © 2014 IBM Corporation107

The Board of Directors of the International Informix Users Group (IIUG) announce the: 2014 IIUG Informix Conference 2014 IIUG Informix Conference April 27 – May 1, 2014 J.W. Marriott Hotel (Brickell) Miami, Florida, USA J.W. Marriott Hotel (Brickell) For more details visit the official conference web site www.iiug2014.org Registration will open November 1, 2013 – See you in Miami! Attention Speakers We are now accepting Presentation Proposals thru Nov. 20, 2013. Details at the conference speaker page: www.iiug2014.org/speakerswww.iiug2014.org/speakers All Non IBM speakers selected receive a Complimentary Conference pass!! ! © 2014 IBM Corporation108

Overview  Problem Statement: –We want the Informix database server to be as invisible as possible. –After the initial install of IDS and uptime, we need the server to be able to adapt to any workload that comes its way. –Prior to 12.10xC2, several key server resources were not dynamic This could and did lead to artificial performance ceilings, and an increase in downtime to fix, or degradation in performance until the issue was detected by human interface.  Not everyone knows how to detect ……. and if they do, getting the downtime from management is an issue …. always.  The improvements are in several different categories….. © 2014 IBM Corporation110

Requirement Categories  Physical Log  Logical Log  Bufferpool  CPUs  These will follow similar rules guiding automatic tuning: –How to detect when insufficient resources for each are causing a performance impact –How to increase the resource with minimal impact to the application –How to control the increase of the resource so it adapts in a timely manner… not too much, not too little, just right © 2014 IBM Corporation111

Physical Log Expansion  Purpose: –To help avoid performance loss associated with an undersized physical log caused by frequent server checkpoints which can block user transactions, located within an already constrained storage space with other objects therein, such as database and server objects.  Expansion –We support the changing of the physical log location already. –We want to make sure there sole occupancy in the space for the physical log hindering its successful expansion and requested size: A new single Physical Log Storage Space (PLOGspace) per instance, Has a single physical chunk containing only the server physical log with the physical page size of the host O/S.  Physical log expansion is triggered by any of the following –A checkpoint blocked user transactions because the physical log is too small. –5 checkpoints triggered by physical log activity: Since the last time the physical log was extended. In the last 20 checkpoints. © 2014 IBM Corporation112

Physical Log Space Creation – onspaces  General Syntax: –onspaces -c –P -p -o -s [-m ]  Examples: –onspaces -c –P plogdbs -p /dev/chk1 -o 0 -s 40000 –onspaces -c –P pdbs1 -p /dev/pchk1 -o 500 -s 60000 -m /dev/mchk1 0  Note –If a PLOGspace already exists: Creating a new one will create the new PLOGspace, Move the physical log to that space, Drop the old PLOGspace, All of the above done automatically. © 2014 IBM Corporation113

Physical Log Space Creation – SQL Admin API  Creating a PLOGspace with SQL Admin API  General Syntax: –create plogspace,,,,,,  Examples: –execute function task(“create plogspace”,”plogdbs”,”/dev/chk1”,30000,0); –execute function task(“create plogspace”,”plogdbs”,”/dev/chk1”,30000,0,”/dev/mchk1”,0); © 2014 IBM Corporation114

Physical Log Improvements - onspaces  Dropping a PLOGspace with onspaces –The same space-dropping command (onspaces –d) may be used to drop this new type of space, –Only if the space is empty.  General Syntax: –onspaces -d [-y] –  Example: –onspaces -d plogdbs -y © 2014 IBM Corporation115

Physical Log Improvements – SQL Admin API  Dropping a PLOGspace with SQL Admin –One must use a new “drop plogspace” command with SQL Admin API. “drop dbspace” will return an error. –“drop plogspace” takes no arguments, since there can be at most one PLOGspace in the instance. The PLOGspace must be empty to be dropped.  General Syntax: –drop plogspace  Example: –execute function task(“drop plogspace”); © 2014 IBM Corporation116

Warnings on Insufficient PLOGspace  In the message log file: –Thru the onstat –m command line utility: You will see a message saying “Physical Log too small ……”  Thru onstat –g ckp utility: –Look for a warning at the top about the physical log being too small or a “Trigger” of “Plog” with “Block Time” > 0.0 –The presence of long transaction rollbacks. A “Trigger” of “LongTX” with “Block Time” > 0.0 can be a too small physical log or too little Logical Log space. © 2014 IBM Corporation117

Logical Log Improvements  Purpose –To improve performance, having sufficient logical log space helps prevent frequent checkpoints from blocking transactions and also from having long or too frequent checkpoints. –Knowing an appropriate amount of logical log space for a dynamic workload is difficult to supply in advance without first knowing and estimating the transaction load. –Configuring a server with too much logical log space in advance can limit the out-of-box experience by requiring substantial disk space. –We will therefore allocate logical log space dynamically as needed and therefore dynamically tunable.  Detection –The best way to detect if the logical log is a bottleneck is to look at the recent checkpoint activity: The server keeps track of the last 20 checkpoints. If a substantial portion of the recent checkpoints were triggered because of the logical log, then we should increase the logical log space by adding another logical log. If a long transaction or checkpoint blocking occurs because of the logical log, we should also add another logical log to the log space. © 2014 IBM Corporation118

Logical Log Improvements  Expansion –We already have the adding logical logs dynamically feature: DYNAMIC_LOGS 2 in the configuration file defined by $ONCONFIG environment variable.  Typically executes only when the server runs out of logical log space. The server already contains the logic to extend a dbspace while adding a logical log. –There is a new, additional Logical log expansion parameter in the configuration file called AUTO_LLOG Enables the automatic addition of logical logs when the database server detects that adding logical logs improves performance. If enabled, log expansion occurs:  When a 5 of the last 20 checkpoints were caused by logical logs filling up  When a logical log causes a blocking checkpoint  When a logical log causes a long checkpoint AUTO_LLOG & DYNAMIC_LOGS do not interact © 2014 IBM Corporation119

Logical Log Improvements  Configuration –The new AUTO_LLOG configuration parameter specifies Dbspace for new logical log files Maximum size of all logical log files.  Estimation of maximum logical log space requirements: –Depending on the number of concurrent users accessing your database server……. 1 - 100: 200 MB 101 - 500: 5 MB 501 - 1000: 1 GB More than 1000: 2 GB © 2014 IBM Corporation120

AUTO_LLOG – Syntax  onconfig.std value 0  By default this is 0, disabled.  AUTO_LLOG 1, dbspace_name, max_size –1 - Add Logical logs when needed to improve performance. –dbspace_name - dbspace name in which to add logical log files. This dbspace must have the default page size for the operating system. –max_size = Optional. Default is 2048000 KB (2 GB). The maximum size in KB of all logical log files, including any logical log files that are not stored in the dbspace dbspace_name. If max_size is not specified, the AUTO_TUNE_SERVER_SIZE configuration parameter setting affects the maximum size. –Takes effect: After you edit the onconfig file and restart the database server. Reset the value dynamically in your onconfig file by running the onmode -wf command Reset the value in memory only by running the onmode -wm command. © 2014 IBM Corporation121

AUTO_LLOG – Syntax Notes  When the maximum size of the logical log files is reached, logical log files are no longer added to improve performance. However, if the DYNAMIC_LOGS configuration parameter is enabled, logical logs are added to prevent transaction blocking.  The settings of the DYNAMIC_LOGS and the AUTO_LLOG configuration parameters do not interact.  If the value of the max_size field is larger than the size of the specified dbspace, enable automatic expansion of storage spaces. © 2014 IBM Corporation122

Warnings on Insufficient Logical Logs  In the message log file: –Thru the onstat –m command line utility: You may see unusual numbers of logical logs have been added.  Thru onstat –g ckp utility: –Look for a “Trigger” of “Llog” with “Block Time” > 0.0 © 2014 IBM Corporation123

Bufferpool Added Capabilities  Dynamic Buffer pools have been requested for a while by our users.  We will therefore add the ability to dynamically extend a buffer pool and do so automatically, which will allow us to start with relatively small pools and increase the size of the pools only as needed.  This will improve performance in many cases by getting the memory resources necessary to the database server it needs, if available, to increase application transaction performance much sooner, in near real-time, rather than after the fact, which has been the norm. © 2014 IBM Corporation124

BUFFERPOOL Configuration Parameter Expansion  Three goals: –Use a new memory format method of allocating pool memory: Specifying buffer pool size in bytes (Kb, Mb, Gb) units. –Marking the pool as extendable. –Maintain the legacy format method of BUFFERPOOL for current user compatibility: Specifying buffer pool size and its limits in pagesize units.  Some BUFFERPOOL arguments are now incompatible with others.  Must follow rules to properly set this parameter: –Order of precedence for all BUFFERPOOL definitions: All definitions shall have the same format (all legacy or all memory) or the server will not boot. If an argument’s value is not defined, we use the value from the default BUFFERPOOL setting. If the default BUFFERPOOL setting is not defined, use the internal definition for the format being used. If unable to determine format (ex. BUFFERPOOL size=2k), use internal format based on any other BUFFERPOOL definitions present. If not, use internal legacy. © 2014 IBM Corporation125

Bufferpool Added Capabilities  Detection –We want to add more memory/buffers to a buffer pool only when the working set doesn’t fit. –To determine when to extend the buffer pool, we keep track of the cache hit ratio of each buffer pool by sampling the cache hit ratio once a minute for 5 minutes. –If the average cache hit ratio is less than a configured threshold, we extend the buffer pool.  Extension –In order to support buffer pool extension, we need to move buffer pools into their own shared memory segments.  Shared Memory in Informix can now have 5 possible segments –Resident –Virtual –Message –Bufferpool *** NEW **** –Extended © 2014 IBM Corporation126

Bufferpools  Bufferpool Segments –Buffer pools will no longer be allocated within the resident segment. –We will add a new IDS memory segment class that will contain only buffer pool memory structures and adhere to the RESIDENT flag. –The buffer pool class segment will have virtually no overhead (bitmap or TTree) for maintaining the segment. This will allow for maximum memory usage.  Bufferpool Extending for Legacy Format –The BUFFERPOOL configuration parameter’s legacy “buffers” format uses these attributes: buffers – starting number of buffers next_buffers – is the number of buffers to add when the buffer pool is extended. next_buffers - will be doubled every 4th extension *. max_extends – maximum number of extensions the buffer pool is allowed * (See the onstat –g buf slide example further below.) © 2014 IBM Corporation127

Bufferpools (cont’d)  Bufferpool Extending for Memory Format –The BUFFERPOOL configuration parameter’s new “Memory” format uses these attributes: memory – target amount of memory that a buffer pool can grow to. Values will be rounded up to 32mb alignments. start_memory – starting amount of memory –Both memory and start_memory values can be expressed in a variety of units. –For example: BUFFERPOOL size=4k,start_memory=32mb,memory=54gb BUFFERPOOL size=8k,start_memory=48000,memory=204mb © 2014 IBM Corporation128

BUFFERPOOL Configuration Parameter Expansion  A BUFFERPOOL definition can NOT include aspects of both formats; either it’s expressed in legacy format or memory format.  A BUFFERPOOL definition with mixed formats is rejected and the database server will not start.  The charts on the next several slides describe each argument to the BUFFERPOOL parameter: –both = works with legacy & memory format. –legacy & memory are mutually exclusive. © 2014 IBM Corporation129

BUFFERPOOL Argument Compatibility Chart ArgumentFormatDynamicValues sizebothnoRequired. Default,2k,4k,…. lrusbothnoOptional Proposed change internal default from numcpuvps to 4 * physical procs on the box. lru_min_dirtybothYes, autoOptional Internal default = 50 lru_max_dirtybothYes, autoOptional Internal default = 60 extendablebothnoOptional 0=FALSE 1=TRUE Determines if bufferpool is extendable or not. Internal default: legacy = 0, memory = 1 cache_hit_ratiobothYesOptional Min=0, Max=100, default = 90 Only applies if bufferpool is extendable  This chart describes each argument to the BUFFERPOOL parameter: both = works with legacy & memory format. legacy & memory are mutually exclusive. © 2014 IBM Corporation130

BUFFERPOOL Argument Compatibility Chart (cont’d) ArgumentFormatDynamicValues bufferslegacyNo# of buffers Initial # of buffers in 1st segment. Min=MIN_BUFFERS(1000) Max=MAX_BUFFERS(based on virtual memory) Internal default=1000 next_bufferslegacyYes, won’t impact existing segments or growth projections, only next segment Requires extendable=1, otherwise, ignored. Min=MIN_BUFFERS(1000) Max=MAX_BUFFERS(based on virtual memory) Internal default=1000 Specifies amount of growth the next bufferpool segment will be. max_extendslegacyNoOptional Requires extendable=1, otherwise, ignored. Maximum # of segments the bufferpool can expand to Min=0 Max=MAX_BUFFSEGS (platform specific… 64bit=24, 64bit NT=8, 32bit=16) Internal Default=8 © 2014 IBM Corporation131

BUFFERPOOL Argument Compatibility Chart (cont’d) ArgumentFormatDynamicValues start_memorymemoryNoOptional for new format determines size of initial bufferpool segment. Min=32Mb Max=Amount of free memory auto=IDS determines starting memory Internal default=Auto memory YesRequired for new format Determine max size bufferpool can grow to Min=32Mb Max=? auto=IDS determines maximum amount of memory to use Internal default=Auto © 2014 IBM Corporation132

BUFFERPOOL examples  BUFFERPOOL size=2k,extendable=1,buffers=1000,next_buffers=2000,max_extends=8 –Create a 2k pagesize buffer pool. –The buffer pool is extendable up to 8 times. The 1st 3 extensions will be of 2000 buffers, Next 4 will be of 4000 buffers and The 8th extension will be of 8000 buffers.  BUFFERPOOL size=10k,start_memory=auto,memory=4gb –Create a 10k pagesize buffer pool. –The buffer pool will be extendable and have a max size of 4gb. –IDS will determine the initial # of buffers and the extension sizes. –Note: when using legacy format, it is possible to ask for N buffers and get slightly more or fewer depending on memory alignment. We allocate extra memory to allow for alignment and then consume all of the memory allocated. © 2014 IBM Corporation133

onstat –p – Monitor your cache hit ratio  So if the %cached for reads is less that 95% then this is usually a good bet that you are doing too much disk i/o and not enough reading from memory, which does affect performance and this is usually because there are not enough memory buffers to cache all of the data. © 2014 IBM Corporation134

onstat –g seg – Memory Segments in Use  As before, the ‘class’ column indicates the type of memory segment, the ‘B’ within the column indicating the new ‘Buffer pool’ segment, and as before: ‘R’ indicating resident and ‘V’ indicating virtual segments. © 2014 IBM Corporation135

VPCLASS  This configuration parameter is now expanded to use with cpu class to make the cpu class extendable.  VPCLASS class,aff=range,noage,num=N,autotune,max=N –autotune is optional. When set, IDS will determine when to add another VP of class. –max is the maximum number of CPU VPs we can extend to.  When using autotune, affinity will not be used for any cpu vps added as part of automatic tuning.  This is again for performance reasons, adding an additional cpu vp dynamically thru autotune allows more resource availability to make those transactions complete quicker due to less likelihood of there being not enough cpu to complete a given operation quickly. –Manual intervention is likely to be not as quick in many such cases. © 2014 IBM Corporation138

CPU VP’s  Detection –CPU VPs are added whenever the average ready Q count across all the CPU VPs for the last minute exceeds 4. –If the average depth of the ready Q is >= 5, we add another CPU VP up to a maximum of the number of physical processors on the box. –Monitoring the ready queue (onstat –g rea, large number of threads sitting waiting), and the global usage of the CPU virtual processors (onstat –g glo, the eff column always at 100).  Configuration –A new argument to the VPCLASS configuration parameter format has been added: autotune. –This argument will be processed only for the CPU and AIO vp classes. For more info see VPCLASS below.  Expansion –Internal procedure to add a cpu vp. –See Appendix A for Event Alarm relevant to this. © 2014 IBM Corporation139

onstat –g rea – Threads Ready to Run onstat –g rea IBM Informix Dynamic Server Version 12.10.FC3 -- On-Line -- Up 2 days 23:53:12 -- 240192 Kbytes Ready threads: tid tcb rstcb prty status vp-class name 6 536a38 406464 4 ready 3cpu main_loop() 28 60cfe8 40a124 4 ready 1cpu onmode_mon 33 672a20 409dc4 2 ready 3cpu sqlexec © 2014 IBM Corporation140

AIO VP with AUTOTUNE  Use a VPCLASS configuration parameter entry for the AIO virtual processor class to specify an exact number of AIO virtual processors or to enable the database server to add AIO virtual processors as needed.  Again this being done for performance, it is sometimes hard for a DBA to know when he might need an AIO VP and where to look to get this information. © 2014 IBM Corporation143

AUTO_TUNE_SERVER_SIZE  Configuration parameter used to buffer pool size when automatic tuning is not possible.  When BUFFERPOOL:memory is set to auto, we set memory using AUTO_TUNE_SERVER_SIZE. –Sets internal structures to assume that’s how much memory will get used. –Therefore, it is possible to exceed these amounts.  AUTO_TUNE_SERVER_SIZE [OFF | SMALL | MEDIUM | LARGE | XLARGE] –OFF use 10% available memory –SMALL use 10% available memory –MEDIUMuse 20% available memory –LARGE use 33% available memory –XLARGE use 50% available memory © 2014 IBM Corporation144

Appendix A – New Event Alarms Class Id Internal NameEvent ID Description 24ALRMU_85_AUTOTUNE_D AEMON_FAIL 24013Auto Tune daemon has insufficient resources to perform tuning operations. Online log: Performance Advisory User action: Restart the server with more resources Class Id Internal NameEvent ID Description 24ALRMU_85_AUTOTUNE_A DD_CPUVP_FAIL 24014Auto tuning failed to start a CPU VP Online log: Performance Advisory User action: Restart the server with more resources © 2014 IBM Corporation145

Appendix A – New Event Alarms (cont’d) Class Id Internal NameEvent ID Description 24ALRMU_24_ADD_BP_FAIL24015Auto tuning was unable to extend the buffer pool due to insufficient resources. Online log: Performance Advisory User action: Restart the server with more resources Class Id Internal NameEvent ID Description 85ALRMU_85_OVER_LLOG85001Auto tuning failed to add another logical log, because adding another log would exceed the maximum log space as defined by configuration parameter AUTO_LLOG. Online log: Performance Advisory User action: Increase the maximum amount of log space by changing AUTO_LLOG configuration parameter. © 2014 IBM Corporation146

Appendix A – New Event Alarms (cont’d) Class Id Internal NameEvent ID Description 85ALRMU_85_OVER_BPOOL85002Auto tuning failed to extend a bufferpool, because the buffer pool would exceed the maximum amount of memory or extensions as defined by configuration parameter BUFFERPOOL. Online log: Performance Advisory User action: Increase the maximum amount defined by the configuration parameter BUFFERPOOL. © 2014 IBM Corporation147

Appendix A – New Event Alarms (cont’d) Class Id Internal NameEvent ID Description 85ALRMU_85_OVER_CPU85003Auto tuning failed to add a CPU VP because another CPU VP would exceed the maximum number specified in the configuration parameter VPCLASS or exceed the number of processors on the computer. User action: If there are more processors available on the computer, increase the maximum number of CPU VPs allowed by changing the VPCLASS configuration parameter. © 2014 IBM Corporation148

Slide Pen – Useful Slides, Not Enough Time – They Tell a Story Scott Pickett WW Informix Technical Sales For questions about this presentation contact: spickett@us.ibm.comspickett@us.ibm.com © 2014 IBM Corporation

Hybrid access: From MongoAPI to relational tables.  You want to develop an application with MongoAPI, but… –You already have relational tables with data. –You have views on relational data –You need to join tables –You need queries with complex expressions. E.g. OLAP window functions. –You need multi-statement transactions –You need to exploit stored procedure –You need federated access to other data –You have timeseries data. © 2014 IBM Corporation151

Mongo Application IBM Wire Listener IDXs Logs Enterprise replication + Flexible Grid + Sharding Distributed Queries Informix Tables IDXs Relational Tables JSON Collections SELECT bson_new(bson, ‘{}’) FROM customer WHERE bson_value_lvarchar(bson,‘state’)=“MO” db.customer.find({state:”MO”}) db.partners.find({state:”CA”}) SELECT * FROM partners WHERE state=“CA” Customer partners JSON Access Relational Access JSON MongoAPI Accessing Both NoSQL and Relational Tables © 2014 IBM Corporation152

How to Convert Relational Data as JSON Documents  Relational data can be treated as structured JSON documents; column name-value becomes key-value pair.  SELECT partner, pnum, country from partners; partnerpnumCountry Pronto 1748Australia Kazer 1746USA Diester 1472Spain Consultix 1742France {partner: “Pronto”, pnum:”1748”, Country: “Australia”} {partner: “Kazar”, pnum:”1746”, Country: “USA”} {partner: “Diester”, pnum:”1472”, Country: “Spain”} {partner: “Consultix”, pnum:”1742”, Country: “France”}  Informix automatically translates the results of a relational query to JSON/BSON form. © 2014 IBM Corporation153

MongoAPI Accessing Both NoSQL and Relational Tables  Typically NoSQL does not involve transactions –In many cases, a document update is atomic, but not the application statement –Example 7 targeted for deletion, but only 4 are removed  Informix-NoSQL provides transactions on all application statements –Each server operation INSERT, UPDATE, DELETE, SELECT will automatically be committed after each operation. –In Informix there is away to create multi-statement transactions is to utilize a stored procedure  Default isolation level is DIRTY READ  All standard isolation level support © 2014 IBM Corporation154

Accessing Data in Relational Tables db.partners.find({name:”Acme”}, {pnum:1, country:1}); SELECT pnum, country FROM partners WHERE name = “Acme”; db.partners.find({name:”Acme”}, {pnum:1, country:1}); SELECT pnum, country FROM partners WHERE name = “Acme”; db.partners.find({name:”Acme”}, {pnum:1, country:1}).sort({b:1}) SELECT pnum,country FROM partners WHERE name=“Acme” ORDER BY b ASC db.partners.find({name:”Acme”}, {pnum:1, country:1}).sort({b:1}) SELECT pnum,country FROM partners WHERE name=“Acme” ORDER BY b ASC CREATE TABLE partners(pnum int, name varchar(32), country varchar(32) ); CREATE TABLE partners(pnum int, name varchar(32), country varchar(32) ); © 2014 IBM Corporation155

Accessing data in relational tables. db.partners.save({pnum:1632,name:”EuroTop”,Country:“Belgium”}); INSERT into partners(pnum, name, country) values (1632, ”EuroTop”, “Belgium”); db.partners.save({pnum:1632,name:”EuroTop”,Country:“Belgium”}); INSERT into partners(pnum, name, country) values (1632, ”EuroTop”, “Belgium”); db.partners.delete({name:”Artics”}); DELETE FROM PARTNERS WHERE name = “Artics”; db.partners.delete({name:”Artics”}); DELETE FROM PARTNERS WHERE name = “Artics”; db.partners.update({country:”Holland”}, {$set:{country:”Netherland”}}, {multi: true}); UPDATE partners SET country = “Netherland” WHERE country = “Holland”; db.partners.update({country:”Holland”}, {$set:{country:”Netherland”}}, {multi: true}); UPDATE partners SET country = “Netherland” WHERE country = “Holland”; © 2014 IBM Corporation156

Views and Joins  Create a view between the existing partner table and a new pcontact table  Run the query across the view create table pcontact(pnum int, name varchar(32), phone varchar(32)); insert into pcontact values(1748,"Joe Smith","61-123-4821"); create table pcontact(pnum int, name varchar(32), phone varchar(32)); insert into pcontact values(1748,"Joe Smith","61-123-4821"); create view partnerphone(pname, pcontact, pphone) as select a.name, b.name, b.phone FROM pcontact b left outer join partners a on (a.pnum = b.pnum); db.partnerphone.find({pname:"Pronto"}) { "pname":"Pronto", "pcontact":"Joe Smith", "pphone":"61-123-4821"} © 2014 IBM Corporation157

Seamless federated access  create database newdb2;  create synonym oldcontactreport for newdb:contactreport; > use newdb2 > db.oldcontactreport.find({pname:"Pronto"}) { "pname" : "Pronto", "pcontact" : "Joel Garner", "totalcontacts" : 2 } { "pname" : "Pronto", "pcontact" : "Joe Smith", "totalcontacts" : 2 } SELECT data FROM oldcontactreport WHERE bson_extract(data, 'pname') = “Pronto”;  create synonym oldcontactreport for custdb@nydb:contactreport; © 2014 IBM Corporation158

Get results from a stored procedure. create function "keshav".p6() returns int, varchar(32); define x int; define y varchar(32); foreach cursor for select tabid, tabname into x,y from systables return x,y with resume; end foreach; end procedure; create view "keshav".v6 (c1,c2) as select x0.c1,x0.c2 from table(function p6())x0(c1,c2);  db.v6.find().limit(5) { "c1" : 1, "c2" : "systables" } { "c1" : 2, "c2" : "syscolumns" } { "c1" : 3, "c2" : "sysindices" } { "c1" : 4, "c2" : "systabauth" } { "c1" : 5, "c2" : "syscolauth" } © 2014 IBM Corporation159

Access Timeseries data create table daily_stocks ( stock_id integer, stock_name lvarchar, stock_data timeseries(stock_bar) ); -- Create virtual relational table on top (view) EXECUTE PROCEDURE TSCreateVirtualTab('daily_stocks_virt', 'daily_stocks', 'calendar(daycal),origin(2011-01-03 00:00:00.00000)' ); create table daily_stocks_virt ( stock_id integer, stock_name lvarchar, timestamp datetime year to fraction(5), high smallfloat, low smallfloat, final smallfloat, vol smallfloat ); © 2014 IBM Corporation160

Access Timeseries data db.daily_stocks_virt.find({stock_name:"IBM"}) { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-03T06:0 0:00Z"), "high" : 356, "low" : 310, "final" : 340, "vol" : 999 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-04T06:0 0:00Z"), "high" : 156, "low" : 110, "final" : 140, "vol" : 111 } { "stock_id" : 901, "stock_name" : "IBM", "timestamp" : ISODate("2011-01-06T06:0 0:00Z"), "high" : 99, "low" : 54, "final" : 66, "vol" : 888 } © 2014 IBM Corporation161

You want to perform complex analytics on JSON data  BI Tools like Cognos, Tableau generate SQL on data sources.  Option 1: Do ETL  Need to expose JSON data as views so it’s seen as a database object. –We use implicit casting to convert to compatible types –The references to non-existent key-value pair returns NULL  Create any combination of views –A view per JSON collection –Multiple views per JSON collection –Views joining JSON collections, relational tables and views.  Use these database objects to create reports, graphs, etc. © 2014 IBM Corporation162

Benefits of Hybrid Power  Access consistent data from its source  Avoid ETL, continuous data sync and conflicts.  Exploit the power of SQL, MongoAPI seamlessly  Exploit the power of RDBMS technologies in MongoAPI: – Informix Warehouse accelerator, – Cost based Optimizer & power of SQL – R-tree indices for spatial, Lucene text indexes, and more.  Access all your data thru any interface: MongoAPI & SQL  Store data in one place and efficiently transform and use them on demand.  Existing SQL based tools and APIs can access new data in JSON © 2014 IBM Corporation164

The Power of Hybrid – Informix & JSON Scott Pickett WW Informix Technical Sales For questions about this presentation, to:

Similar presentations

Presentation on theme: "The Power of Hybrid – Informix & JSON Scott Pickett WW Informix Technical Sales For questions about this presentation, to:"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

The Power of Hybrid – Informix & JSON Scott Pickett WW Informix Technical Sales For questions about this presentation, to:

Similar presentations

Presentation on theme: "The Power of Hybrid – Informix & JSON Scott Pickett WW Informix Technical Sales For questions about this presentation, to:"— Presentation transcript:

Similar presentations

About project

Feedback