1. Carlton Doe and his session – aka the warm-up band for Mark Scranton

2. Moving from IBM IDS 7.x to IBM IDS 9.x Carlton Doe Technical Sales Manager, Dallas, TX IBM

3. 3 ISBN 0-13-605296-7ISBN 0-13-080533-5 ? Who I Am  12 + years of experience as DBA and engine admin mainly in retail environments  Co-founded, presided over, and former member of the Board of Directors of the International Informix Users Group (IIUG)  Written two Informix Press books:

4. 4 What I’m Going to Cover The Basics  IDS Futures  What is IBM IDS and Foundation?  Things to Know Before Moving  Reserved Words  ONCONFIG Parameters  How Do I Move?  Dynamic Logical Log Files  Configurable Lock Modes  SQL Statement Cache  Raw / Standard Tables  Changes to “Explain Mode”  Fuzzy Checkpoints  IBM Informix MaxConnect

5. 5  Smart Large Objects and Dbspaces  Simple/Complex Datatypes and Casting  User Defined Routines and Functions  Collections  DataBlades and the Blade Manager  HDR / Enterprise Replication – A Snapshot  Java in the Engine What I’m Going to Cover The Good Stuff

6. 6 IDS Futures

7. 7 Projected IDS Roadmap IBM will deliver new releases of IDS about every 18 months 9.6 200320042005 2006 9.x In Planning Accepting Feature Requests Projected 4Q 2004 Delivery In Development Features Frozen Planned 2Q 2003 Delivery 9.49.5 In Concept Projected 2Q 2006 Delivery

8. 8 Development is focused on providing a release of IDS 9 of interest to all IDS customers. OLTP Features are the #1 Priority  Scalability  Security  SQL Compatibility  Performance  Ease of Upgrade  And More! IDS 9.40 Features

9. 9 What can you expect down the road?  Further Integration with IBM & DM Products  Focus on Security  Focus on SQL Compatibility  Focus on Performance  Leverage the SMART Initiative  Support for the Latest Hardware  Platforms  Operating Systems  Structured Storage – NAS & SAN IDS 9.50 & 9.60

10. 10 Platform Directions  Industry Consolidation, majority of IDS sales are on 5 platforms:  Tier 1 Operating Systems: AIX, Solaris, HPUX, Windows, Linux  Tier 1 Platforms: Sparc, Power, PA-RISC, Intel, IA64 Planned  Continuing support (Planned for IDS 9.40) for Tru 64, Irix, Caldera OpenUnix + Linux Kernel Personality (LKP)  Future is 64 bit.  Many vendors are slowly moving towards IA64  Affected operating systems include: Windows, Linux, HPUX  Whenever possible move to 64 bit servers, rather than 32 bit  Most applications will run unchanged against a 64 bit server  Some will not be 64 bit clean -> thorough testing prior to deployment is recommended  Full exploitation of 64 bit server may require recompiling and/or modifying application

11. 11 Downlevel Operating System Versions You need to move to newer OS levels and/or 64 bit servers (recommended if available)  Solaris 2.5.1  Solaris 2.7 (64 bit)  AIX 4.3.2 and below  HPUX 10.x  Tru 64 4.x  Dynix 4.4.x and below  Windows NT  Linux 2.2 Kernel  HPUX 11 32 bit  Caldera SCO OpenServer & UnixWare

12. 12 What’s the Difference Between IDS and Foundation? What’s the Difference Between IDS and Foundation?

13. 13 IBM Informix Dynamic Server 7.x IDS - UDO 9.1x What Is IBM IDS and Foundation?  Parallelism built-in  Parallel Data Query  SQL92 Entry Level  Enterprise Replication  Many, many more...  SQL 3 Support  DataBlade Support  DataBlade Developer Kit  User Defined Routines  User Defined Datatypes  User Defined Indexing  R-Tree Indexing  Extended B-Tree Support  Row Types  Collections (sets, multiset, lists)  Inheritance  Polymorphism  Partitioning with new data types

14. 14 IBM Informix Dynamic Server 7.x IDS - UDO 9.1x += IBM IDS 9.2 + What Is IDS and Foundation?

15. 15 Then What Is Foundation? Web DataBlade Excalibur Text DataBlade Object Translator Office Connect J/Foundation IBM Informix Dynamic Server

16. 16 The New Face of the Foundation Family:  IBM IDS with J/Foundation  New product -- IDS with the Hot-Spot JVM  Internet Foundation – has been discontinued  Financial Foundation for Capital Markets  TimeSeries, NAG DataBlade Modules, TimeSeries Real Time Loader, Office Connect, Object Translator  Law Enforcement Foundation  Visionics, fingerprint, and other biometric-oriented DataBlade Modules IBM Informix Spatial DataBlade Module is freely available to IDS 9.3 customers!!

17. 17 Which Version Is Loaded?? (IBM IDS) Odra: onstat - Informix Dynamic Server Version 9.30.UC1G1 (Foundation) Ebro: onstat - Informix Dynamic Server Version 9.30.UC1G1 (MSGPATH) Wed Oct 4 07:49:52 2000 07:49:52 Booting Language from module <> 07:49:52 Loading Module 07:49:57 Informix Dynamic Server Version 9.30.UC1G1 Software Serial Number AAB#J500705 07:50:12 Informix Dynamic Server Initialized -- Shared Memory Initialized 07:50:15 Physical Recovery Started 07:50:25 Physical Recovery Completed: 0 pages restored

18. 18 Which Version Is Loaded?? Ebro: cd /usr/informix/9_3/extend Ebro: ls -l drwxr-xr-x 4 informix informix 1024 Jul 19 08:08 ETX.1.30.UC5 drwxr-xr-x 4 informix informix 1024 Jul 19 08:07 TXT.1.10.UC5 drwxr-xr-x 4 informix informix 1024 Jul 19 09:02 VTS.1.20.UC1 drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxbuiltins.1.1 drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxmngr drwxr-xr-x 2 informix informix 1024 Jul 19 07:42 ifxrltree.2.00 drwxr-xr-x 4 informix informix 1024 Jul 19 07:49 krakatoa drwxr-xr-x 4 informix informix 1024 Jul 19 07:42 LLD.1.20.UC2 drwxr-xr-x 4 informix informix 1024 Aug 22 13:36 WEB.4.10.UC1

19. 19 Things to know before you move Things to know before you move

20. 20 Things to Know  You can not directly move from < 7.x versions  IBM STRONGLY suggests that you first move to the latest available 7.x version then move to 9.x  While you may want to export/import your data, in-place upgrades have been quite successful -- just remember to wear your belt and suspenders  HP-UX 11.0 - RUN_AS_ROOT.server script is wrong Links created for one library goes into /usr/lib instead of /usr/lib/pa20_64; if left alone, engine will not come up (“library not found” error). A “defect” has been entered ** Check 9.3 version, may be corrected.**

21. 21  With IBM IDS, CPU and user-defined VPs are run as user “informix” while other VPs are still run as root. This prevents UDRs from having root access  If you have auto-start scripts, you should modify them so they “su” to “informix” before starting the engine  Install engine after client products for “finderr” to work. The order is:  tools  network  engine Things to Know

22. 22  Precision change in float / smallfloat to decimal conversion smallfloat 8 -> 9 float 16 -> 17  Default behavior of “create index” is now to create detached indexes. 7.x attached indexes are supported by the upgrade process  Use the “DEFAULT_ATTACH” variable to temporarily mimic 7.x behavior, it will be discontinued however in a future release  9.4 Feature: LVARCHAR increases 2K to 31K Optional parameter added to lvarchar data-type declarations: LVARCHAR (n) where 0 < n < 32k-2. Current syntax still supported, but old 2k limit holds for that syntax. Things to Know

23. 23  If using IBM Informix Enterprise Gateway with DRDA version 7.31.UC1 with IBM IDS 7.x, you must get the DRDA patch to avoid defect #132619  HADR has changed. DRAUTO is no longer supported so after a HADR failure condition is declared, the secondary server goes to read-only mode  You must manually convert the secondary server to “standard” mode “onmode -d standard”  No changes to DRINTERVAL, DRTIMEOUT, DRLOSTFOUND Things to Know

24. 24  9.2 / 9.3 ER and HPL – does not support named and other complex datatypes, only distinct and opaque types  9.4 feature: new functionality! see ER/HDR portion of the presentation. Test dbexport / dbimport to see if more complex named types work  Shut down HADR / ER before upgrading Things to Know

25. 25  System catalog changes - columns added, moved, data types changed. New tables added and some dropped. Sysindexes and systables now a view rather than tables  Long identifiers!! The 8/18 rule is dead (the crowd cheers!!) user ids - 32 characters identifiers - 128 characters table name, server name, database name, column name, index name, synonym name, constraint name, procedure name, dbspace name, blobspace name, optical cluster name, trigger name, type name, routine language name, access method name, operator class name, trace message class name, trace message name, procedure variable names Things to Know

26. 26  9.4 Feature: File size limits for utilities have been increased from 2GB to 17 billion GB Utilities Affected: Oncheck, Onload, Onlog, Onmode, Onmonitor, Onparams, Onspaces, Onstat, Onunload, DB-Access, Dbexport, Dbimport, Dbload, Dbschema, Onpload, Ontape  Also updating Streams API for > 2GB support  9.4 Feature: Chunk size limit changes from 2 GB to 4TB  9.4 Feature: Maximum instance chunks changes to 32,767 Things to Know Individual instance size now 128 petabytes

27. 27 128 petabytes! Just how BIG is 128 petabytes?  At a load rate of 10 gigabytes an hour it would take well over a 1,000 years to fill a 128 petabyte database.  At the current price for high-end structured storage it would cost $3-4 billion to buy 128 petabytes of storage.  The US Library of Congress contains an estimated 10 terabytes of printed material. 128 petabytes is equal to 12,800 Libraries of Congress.

28. 28  9.4 Feature: # of columns in a Functional Index  Old Limit 16 columns  New Limit 341 Columns (Java, SPL)  New Limit 102 (C)  9.4 Feature: DBServerAliases  Old Limit 10  New Limit 32  9.4 Feature: Size of a Shared Memory Dump  Old Limit 2 GB limit  New Limit 17 Billion GB Things to Know

29. 29  9.4 Feature: Removed IDS libraries from /usr/lib  Each user connection uses about 5% more shared memory. Watch your shared memory allocations - both instance and operating system  This can be offset by using IBM Informix MaxConnect  Make sure you drop and rebuild your distributions after upgrading! Things to Know

30. 30 Things to Know: New Reserved Words  CACHE  COSTFUNC  ITEM  SELCONST  INNER  JOIN  LEFT   LOCKS   RETAIN   RAW   STANDARD   AVOID_EXECUTE   USE_SUBQF   AVOID_SUBQF

31. 31 Things to Know: One Example of New SQL Functionality 9.4 Feature: ORDER BY values not in select list  Most databases implement the ability to order a list based on values not returned by the select.  Solution: Implement support for this SQL syntax: SELECT d.dept_num FROM dept d, employees e WHERE d.dept_num = e.dept_num GROUP BY d.dept_num ORDER BY avg(e.salary);

32. 32 Things to Know: New ONCONFIG Parameters  ALLOW_NEWLINE  BLOCKTIMEOUT  DD_HASHMAX  DD_HASHSIZE  DS_HASHSIZE  DS_POOLSIZE  PC_HASHSIZE  PC_POOLSIZE  SBSPACENAME  SBSPACETEMP  SYSSBSPACENAME  STMT_CACHE  STMT_CACHE_SIZE  STMT_CACHE_HITS  STMT_CACHE_NOLIMIT  STMT_CACHE_NUMPOOL  VPCLASS  JDKVERSION  JVPHOME  JVPLOGFILE  JVPPROPFILE  JVPJAVAVM  JVPJAVAHOME  JVPJAVALIB  JVPCLASSPATH  JVMTHREAD

33. 33 Things to Know: New ONCONFIG Parameters VPCLASS  Enables you to designate and create uniquely named, user-defined, or system classes of VPs. User-defined VPs (UDVPs) have the same functional power as CPU VPs  User-defined VPs should be created to execute user- defined routines and/or DataBlades  Some DataBlades (Verity, Excalibur Text) require their own VP as does the JVM  User-defined VPs can be added/dropped on the fly with the “onmode -p class_name ” command

34. 34 Things to Know: New ONCONFIG Parameters VPCLASS Syntax tree: VPCLASS name,num= X,[max= X,aff=( X-Y ),noyield,noage]  VP name is not case sensitive, options are order independent, no whitespace, 128 char max/entry  Must declare multiple JVPs to execute Java UDRs in parallel (Unix)  If using “noyield”, only declare “1” vp since the UDR will execute serially, causing others to queue for their turn

35. 35 Things to Know: New ONCONFIG Parameters Use the VPCLASS parameter to replace the CPU and AIO ONCONFIG parameters VPCLASS cpu,num=3,max=10,noage  Comment out AFF_SPROC, AFF_NPROCS, NOAGE, NUMCPUVPS, SINGLECPUVP parameters VPCLASS aio,num=5,max=10  Comment out NUMAIOVPS parameter

36. 36  ON-Archive. Use either the ontape or ON-Bar backup utility instead.  DB/Cockpit. Use ISA instead.  Informix-DBA. Use Server Studio Java Edition by AGS instead.  ISM graphical user interface. The command line provides the same functionality. Things to Know: Discontinued Support

37. 37 Things to Know: How do I Move? 1. Stop database processing and force a change to a new logical log onmode -sy; onmode -l; onmode -c; onmode -ky; 2. Back up the instance(s) and critical configuration files in $INFORMIXDIR etc/ /aaodir/adtcfg $ONCONFIG (s)**/dbssodir/adtmasks ONCONFIG.std sm_versions sqlhosts** tctermcap termcap

38. 38 3. Restart the instance(s) and put them into single-user mode to ensure all open transactions are properly rolled back oninit -sv 4. Verify data and index integrity with the oncheck utility (-cr, -ce -cc, -c [ I / D ] db_name ) 5. Shut the instance(s) down again and create level 0 backup(s) and/or dbexports – make sure logical logs are backed up and clear!! 6. Install the new IBM IDS / Foundation software and change relevant scripts, global parameters, config files Things to Know: How do I Move?

39. 39 /usr/informix 7_3 9_3 scripts disks (also tapes) isa logs config_files max_conn koetari -- symbolic links odra -- symbolic links (current logs) /old_logs Environment Variables: INFORMIXDIR ONCONFIG INFORMIXSQLHOSTS My Recommended Directory Structure

40. 40 7. Change ALARMPROGRAM to [ null | no_log.sh ] if using On-Bar or LTAPEDEV to /dev/null 8. Install any DataBlades 9. Restart each instance into quiescent mode and monitor the MSG_PATH file. Sysmaster and reserved pages conversion is automatic (see $INFORMIXDIR/etc/dummyupds7x.sql). When completed, a notice is written to MSG_PATH. If upgrading from 9.2 to 9.3, watch for sysutils and sysmaster db build successful messages in MSG_PATH before doing anything else! 10. Verify data integrity with oncheck commands Things to Know: How do I Move?

41. 41 11. Execute an “update statistics low drop distributions” command on each database in the instance 12. Execute an “update statistics high” command on the sysmaster database 13. Execute an “update statistics medium distributions only” command on each database in the instance. 14. Change LTAPEDEV / ALARMPROGRAM back to its original value Things to Know: How do I Move?

42. 42 15. Create a level 0 backup. 16. Perform system tests to check performance. Execute queries that will stress indexes to see if the optimizer properly uses them as well as their performance. 17. Let the users back in If there are any problems -- RESTORE from tape Things to Know: How do I Move?

43. 43 Dynamic Logical Logs

44. 44 Dynamic Logical Logs: Introduction New feature in IBM IDS 9.3 – automatically create, insert, and activate logical logs when needed!!! Does not eliminate long transactions, just the server hangs that can occur especially on startup. Required:  Ability to add log on the fly  Insert log immediately after current logical log  Bring the log into active mode without a backup of rootdbs / critical Dbspace(s)

45. 45 Dynamic Logical Logs: Introduction In the pre-9.3 engine:  Could only be added when in quiescent mode  Always took the first available slot in the list  Required a backup of the rootdbs (level 0) to become active

46. 46 Dynamic Logical Logs: Introduction In IBM IDS 9.3:  LOGSMAX disappears from $ONCONFIG  Logs can be added while instance is processing transactions if so configured  Logs can be added after current log to avoid running into log with "begin work" statement  Newly added logs are immediately available  Need to use DYNAMIC_LOGS $ONCONFIG parameter

47. 47 Dynamic Logical Logs: How do you add a log? onparams –a [ -d dbspace ] [ -s size ] [ -i ]

48. 48 Dynamic Logical Logs: How do you add a log?

49. 49 Dynamic Logical Logs: Server actions The server itself will attempt to add a log file if two conditions are true:  The next active log file contains an open transaction  DYNAMIC_LOGS is set to 2 (the default) The server checks for those conditions at two important points in the code:  Immediately after a log switch  Beginning of the last phase of logical recovery (Transaction Cleanup)

50. 50 Dynamic Logical Logs: Server Actions What Dbspaces are used for automatic log allocation? 1 The dbspace that contains the newest log files. (If this dbspace is full, the engine searches other dbspaces.) 2 Mirrored dbspace that contains log files (but excluding the root dbspace) 3 All dbspaces that already contain log files (excluding the root dbspace) 4 The dbspace that contains the physical log 5 The root dbspace 6 Any mirrored dbspace 7 Any dbspace

51. 51 Dynamic Logical Logs: DYNAMIC_LOGS parameter Values for the DYNAMIC_LOGS parameter: 2 (default) the server is authorized to add log files automatically to avoid long transaction hangs 1 if a log file is needed, the server waits for the admin to add one manually 0 feature is turned off, mimics < 9.3 behavior

52. 52 Dynamic Logical Logs: DYNAMIC_LOGS parameter Why set DYNAMIC_LOGS=1? You want to make use of the feature in order to avoid long transaction hangs, but you don’t want to give up any control over log file location or size. Remember that once a log file is added to a non-critical Dbspace, that space becomes critical.

53. 53 Dynamic Logical Logs: DYNAMIC_LOGS parameter Why set DYNAMIC_LOGS=0? Hmm, tough question. We don’t know. Adding a log file to a Dbspace forces your next archive of that Dbspace (and the ROOT Dbspace) to be a level 0. If you’d rather risk a server hang than risk having to occasionally tweak your archive schedule, turn the feature off. If the server does hang due to a long transaction rollback, simply set DYNAMIC_LOGS to 1 or 2 and bounce the server. Now you may have to improvise with your archive schedule, but it’s better than prolonging a down system.

54. 54 Dynamic Logical Logs: DYNAMIC_LOGS parameter Dropping logical logs in 9.3 is similar to <9.3  Never-used log files (A) drop immediately  Used log files are marked “Deleted”. Once all spaces have been archived they will automatically drop  If system has never been archived, the new drop restriction isn’t necessary, and is not applied  NEW -- Can be done while instance is on-line!!!

55. 55 Configurable Lock Mode

56. 56 Configurable Lock Mode: Introduction An extension of the "lock mode [ row | page ]" syntax in "create table" command Allows users / admins to set the lock mode on newly created tables without having to explicitly set it in the table creation statement. Uses either an environment variable or an $ONCONFIG parameter setting

57. 57 Configurable Lock Mode: How's it set? IFX_DEF_TABLE_LOCKMODE environment variable setenv IFX_DEF_TABLE_LOCKMODE row This variable can also be set in $HOME/.informix or $INFORMIXDIR/etc/informix.rc files DEF_TABLE_LOCKMODE $ONCONFIG parameter DEF_TABLE_LOCKMODE row This sets LOCK MODE to ROW for all tables for all sessions in the server

58. 58 Configurable Lock Mode: Examples No additional syntax: {TABLE "carlton".test_table row size = 155 number of columns = 3 index size = 0 } create table "carlton".test_table ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page; TBLspace Report for my_nt_test:carlton.test_table Physical Address 200033 Creation date 09/09/2001 15:07:35 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155

59. 59 Configurable Lock Mode: Examples Explicit "lock mode page" syntax with environmental variable set to "row": {TABLE "carlton".test_table_3 row size = 155 number of columns = 3 index size = 0 } create table "carlton".test_table_3 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page; TBLspace Report for my_nt_test:carlton.test_table_3 Physical Address 200245 Creation date 09/09/2001 15:18:09 TBLspace Flags 901 Page Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155

60. 60 Configurable Lock Mode: Examples With environment variable set to "row" but no additional syntax in " create table " command: {TABLE "carlton".test_table_2 row size = 155 number of columns = 3 index size = 0 } create table "carlton".test_table_2 ( col1 integer, col2 char(50), col3 varchar(100) ) extent size 32 next size 32 lock mode page, row; TBLspace Report for my_nt_test:carlton.test_table_2 Physical Address 200034 Creation date 09/09/2001 15:18:09 TBLspace Flags 903 Page Locking Row Locking TBLspace contains VARCHARS TBLspace use 4 bit bit-maps Maximum row size 155 Bug in the code !!

61. 61 SQL Statement Cache

62. 62 SQL Statement Cache: Introduction As implied, prepared and optimized SQL statements are stored in a section of the virtual portion of shared memory. Users executing identical DML SQL statements can pull the pre-parsed query plan and even query data structures from the cache More complex queries (multiple columns, many filters in the WHERE clause) benefit most from caching Turned off by default, the STMT_CACHE $ONCONFIG parameter and the “SET STATEMENT CACHE” SQL command control how the caching process works. Can be overridden with the “onmode -e” command

63. 63 SQL Statement Cache: How It Works 1. Incoming SQL statements are hashed to a value which is compared to other hash values in the SSC section of the virtual portion of shared memory Caveats:  statements only optimized at the PREPARE phase  host variable names/placeholders are not included in the hash algorithm  white spaces and case of the statement is significant  session-specific parameters (OPTCOMPIND, OPT_GOAL) are significant and will affect hash value

64. 64 SQL Statement Cache: How It Works 2. If hash value matches an existing value, the cached copy is used. The existing copy is NOT re-optimized 3. If a match does not exist, the statement is evaluated for inclusion in the SSC Qualifications for inclusion in the SSC:  contains DML statements using built-in data types and operators  no UDRs/UDFs  local database connections only  no explicit temporary tables

65. 65 SQL Statement Cache: How It Works Qualifications for inclusion in the SSC (cont):  “select *” statements can be fully expanded to include all columns in the listed tables  is not a statement generated by a stored procedure  no embedded subselects in the SELECT statement

66. 66 SQL Statement Cache: Enabling and Sizing onmode -e {ENABLE|ON|OFF|FLUSH}  ENABLE  ENABLE - Allows statement caching to take place. This does not turn caching on, but allows it to be turned on  ON  ON - Turns on statement caching. Caching must first be enabled through the STMT_CACHE parameter or “ onmode ” command  OFF  OFF - Turns off the SSC and immediately disables sharing. Cached statements that are in use remain in the SSC until their use count reaches zero  FLUSH  FLUSH - Active statements are marked and subsequently flushed when they are released. When the SSC is turned off, sharing is immediately disabled

67. 67 SQL Statement Cache: Enabling and Sizing STMT_CACHE $ONCONFIG parameter: 0 0 - SQL Statement Cache disabled (default) 1 1 - SSC enabled though sessions must explicitly call for its use 2 2 - SSC enabled for all sessions unless explicitly denied on a session-by-session basis Set the STMT_CACHE environment variable to “0” (off) or “1” (on) Execute a “set statement cache [ on | off ]”command

68. 68 SQL Statement Cache: Enabling and Sizing STMT_CACHE_SIZE $ONCONFIG parameter: determines the size of the SSC in KBs, default = 524 kb If SSC is full and all statements are active, if additional statements qualify for inclusion the SSC will expand to hold them. As a statement’s user counts drop to zero, the SSC will flush them and reduce back to STMT_CACHE_SIZE

69. 69 SQL Statement Cache: Monitoring and Tuning  “onstat -g cac stmt” displays size of the SSC the cached statements, who’s currently executing a statement, and the number of times it’s been used  The “hash” column is not the statement’s hash value, rather the hash bucket in which the statement resides  To tune, monitor the size of the SSC over time, then resize as appropriate

70. 70 SQL Statement Cache: Monitoring and Tuning Use the STMT_CACHE_DEBUG environment variable to trace what happens with any given SQL statement export STMT_CACHE_DEBUG= debug_value:path_to_file debug_value can be “1” (basic) or “2” (extended)

71. 71 SSC identity (OK:status=1): select * from yoyo SSC link (semantic error): select * from yoyo SSC identity (stmt not found): create table yoyo (col1 int) in mustang_2 SSC qualify (invalid stmt type): create table yoyo (col1 int) in mustang_2 SSC identity (stmt not found): insert into yoyo values (1) SSC qualify (OK): insert into yoyo values (1) SSC key insert (OK): insert into yoyo values (1) SSC full insert (OK): insert into yoyo values (1) SSC identity (stmt not found): insert into yoyo values (2) SSC qualify (OK): insert into yoyo values (2) SSC key insert (OK): insert into yoyo values (2) SSC full insert (OK): insert into yoyo values (2) SSC identity (OK:status=1): insert into yoyo values (2) SSC link (OK): insert into yoyo values (2) SSC identity (stmt not found): drop table yoyo SSC qualify (invalid stmt type): drop table yoyo SQL Statement Cache: Monitoring and Tuning

72. 72 New $ONCONFIG parameters:  STMT_CACHE_HITS Number of statement hits before entered into the cache Values:  0 – statement immediately inserted  N – first time statement used, key-only entry made in cache, from 2 to N counter incremented. After N uses, full statement entered into cache Want to set to "1" (one) to prevent ad-hoc statements from clogging cache SQL Statement Cache: Monitoring and Tuning

73. 73  STMT_CACHE_NOLIMIT Controls insertion of qualified statements into cache after its size is greater than the STMT_CACHE_SIZE value. While statement cache can grow, this acts as a governor to stop unrestrained growth Values:  0 – no new statements are inserted after STMT_CACHE_SIZE reached  1 – unrestricted growth of cache is allowed SQL Statement Cache: Monitoring and Tuning

74. 74  STMT_CACHE_NUMPOOL Specifies number of pools to be allocated to the SQL Statement Cache Values:  1 (default) to 256 As statement cache increases, number of pools supporting the cache may become a bottleneck. Use the "onstat –g ssc pool" and "onstat –g spi" commands to monitor usage and spin waits. See pages 4-49 – 4-52 in the Performance Guide for more information. SQL Statement Cache: Monitoring and Tuning

75. 75 Raw / Standard Tables

76. 76 Raw / Standard Tables Raw tables have been in XPS for quite some time and are used in conjunction with its SQL-based HPL or massive static data manipulation Intended for initial loading and data validation Similar behavioral attributes regardless of database logging mode, i.e., don't try to use them in transactions.

77. 77 Raw / Standard Tables Raw table attributes  nonlogged permanent table, behaves like table in a nonlogging database  uses light append s, adds rows quickly to end of each table fragments  updates, inserts, and deletes are supported but not logged  Can not support indexes, referential constraints, or rollback  Can only restore from last physical backup if not updated since that backup, no MiT restore  Fast recovery rolls back incomplete transactions on STANDARD tables but not on RAW tables  Wicked fast to use if done properly!!!

78. 78 Raw / Standard Tables Syntax create raw table xyz ( yada, yada ) in etc., etc. fragment by whatever; alter table xyz type [ raw | standard ];  When altered for massive manipulation, make sure you backup the table after converting it back from raw mode!!

79. 79 Changes to "explain mode"

80. 80 Explain Mode Can now get sqexplain.out information without having to execute the DML statement!!!! Two ways to activate this feature:  per statement with a directive select --+explain avoid_execute * from xyz AVOID_EXECUTE directive works in conjunction with EXPLAIN directive. If EXPLAIN directives is omitted, i.e. if only AVOID_EXECUTE directive is used,the query will not be executed and no explain output will be generated.

81. 81 Explain Mode  Set as part of a statement block set explain on avoid_execute; After executing this statement, the server writes to the sqexplain.out file without executing any statements until SET EXPLAIN ON/OFF is executed.

82. 82 What Are Fuzzy Checkpoints??? What Are Fuzzy Checkpoints???

83. 83 Checkpoints IDS 9.2x introduced “fuzzy” operations and checkpoints to increase transactional throughput Two types of checkpoints:  Full or “sync”  Fuzzy

84. 84 Steps of a Sync Checkpoint 1. Engine blocks threads from entering “critical sections” of code 2. The page cleaner thread flushes the physical log buffer to log on disk 3. The page cleaner threads flush to disk all modified pages in the buffer pool (chunk write) 4. The checkpoint thread writes a checkpoint record to the logical log buffer 5. The logical log buffer is flushed to the current logical log file on disk 6. The physical log on disk is logically emptied (current entries can be overwritten) 7. The checkpoint thread updates the reserved pages with the checkpoint record information

85. 85 Good and bad: logically and physically data, interrupts user activity, expensive (I/O hit), must tune to reduce duration What Causes Sync Checkpoints to Occur?  Physical log becomes 75% full  “onmode -c” or “-ky”  Administrative actions (adding dbspaces, altering tables)  A backup or restore operation using ontape or ON-Bar  End of fast recovery or full recovery  Reuse of a logical log containing the oldest fuzzy operation not yet synced to disk  LTXHWM reached with fuzzy transactions  Through the onmonitor menu

86. 86 Fuzzy Checkpoints Record “results” of fuzzy operations in the logical log Goal: reduce or eliminate checkpoint interruptions, eliminate the physical log (over 3 - 4 phases) Cost: increased logical recovery time Note: Fuzzy checkpoints are not used in HADR environments

87. 87 Fuzzy Checkpoints Things to know:  Inserts, updates, delete DML statements  Currently supports “built-in” data types only (character, numeric values)  Logged databases  Not “old” pages. Determined by page timestamp (4 byte, cycles 2 gb -> -2 gb -> 0)  No BEFORE images are generated in the physical log  During “fuzzy” checkpoints, buffered information on fuzzy operations is **not** flushed to disk  After a fuzzy checkpoint, disks are not physically consistent

88. 88 Steps of a Fuzzy Checkpoint 1. Engine blocks threads from entering “critical sections” of code 2. The page cleaner threads flush to disk all **non-fuzzy** modified pages in the buffer pool (chunk write) Note: MLRU queues still full 3. A Dirty Page Table (DPT) is constructed containing entries (buffer addresses) for fuzzy operations 4. The DPT is flushed to the current logical log on disk 4A. The page cleaner thread flushes the physical and logical log buffers to disk 5. The checkpoint thread updates the reserved pages with the checkpoint information including the Log Sequence Number (LSN). 6. The physical log on disk is logically emptied (current entries can be overwritten)

89. 89 Log Sequence Number (LSN)  A value representing a position in the logical log (log # and log position)  Stored in each dirty buffer heading containing fuzzy operation results along with a newly-stored timestamp  The oldest LSN is tracked in shared memory because it contains the oldest fuzzy operation. Oldest LSN can not be further back than 1 logical log and 2 checkpoints  Logs after the LSN can not be freed for re-use until a checkpoint has occurred Net Net -- large logical logs are GOOD!!!

90. 90 What Causes Fuzzy Checkpoints to Occur?  Checkpoint interval has elapsed  Physical log becomes 75% full  onmode -c fuzzy  Reuse of a logical log containing the last checkpoint (either fuzzy or sync)

91. 91 Recovery - Sync Checkpoint 1. Physical log data used to return all disk pages to original “synced” state (physical restore) 2. The most recent checkpoint (sync) record is located in the logical log files 3. All subsequent logical log records are rolled forward 4. Uncommitted transactions are rolled back

92. 92 Recovery - Fuzzy Checkpoint 1. Physical restore proceeds as normal however disk pages are not physically consistent since there are modified pages that were not logged in the physical log 2. Logical recovery begins in two phases:  Phase A -- reapply DPT records  Phase B -- post checkpoint records

93. 93 Recovery - Fuzzy Checkpoint Phase A - Logical Recovery: 1. Locate the oldest LSN representing unwritten data 2. Selectively apply fuzzy operations from the DPT/logical log from the oldest remaining logical log up to the most recent checkpoint Conditional update based on page timestamp - if DPT record is “newer” than page timestamp, then the change is applied Result: similar to end of physical restore when using sync checkpoints

94. 94 Recovery - Fuzzy Checkpoint Phase B - Logical Recovery: 1. Beginning with last checkpoint, roll forward subsequent logical log records. 2. Uncommitted transactions are rolled back Result: similar to the logical restore when using sync checkpoints. Disks are physically and logically consistent to last recorded transaction

95. 95 Fuzzy Checkpoints  Buffer structures, logical log records, and reserved pages are automatically upgraded when migrating to 9.x  Fuzzy checkpoints enabled by default. Set the NOFUZZYCKPT environment variable to true/on/1 to disable them prior to initializing shared memory  Set the TRACEFUZZYCKPT environment variable to trace fuzzy checkpoints. It will affect instance performance but will log in the MSGLOG the number of buffers that remain dirty after a checkpoint is completed. It also lists the location of the old LSN

96. 96 IBM Informix MaxConnect

97. 97 IBM Informix MaxConnect IBM Informix MaxConnect is a middle-ware communications component that removes the overhead of managing user connections from the instance Helps you get scale your user connections past the effective 2,000 user connection limit without any change or resource impact to the database server It supports 7.3 and 9.2x+ Unix ports of the engine but accepts user connections from any platform NO CHANGE to existing applications!!

98. 98 IBM Informix MaxConnect Performance BaaN IVc World record:  11,445 BRUs on HP N4000 (8CPU, 16GB) and IDS  21% better than Oracle on identical HP system  Beat Oracle’s 64-CPU Sun E10000 result  18,650 database connections using 98 transports !!

99. 99 MaxConnect IDS/Fnd SQL Clients For large, hardware- partitionable SMP systems MaxConnect SQL Clients Database server system IDS/Fnd IBM Informix MaxConnect Architecture

100. 100 IBM Informix MaxConnect Configuration  An IBM MaxConnect instance connects to only one DB server instance  Connections are managed through the SQLHOSTS file with a new connection protocol -- imc  Clients change hostname and port address (maybe) to point to the IBM MaxConnect instance, instance name remains the same  IBM MaxConnect instance has an ADMIN entry, a client- facing entry, and a DB server-facing entry  DB instance has a DBSERVERALIAS and a DBSERVERNAME entry

101. 101  Client SQLHOSTS file: ebro_tcp ontlitcpmaxcon_11526  MaxConnect SQLHOSTS file: ebro_tcp ontlitcpmaxcon_11526 ebro_db ontliimccronus1575 ebro_adm onsoctcpmaxconn_11600  Server SQLHOSTS file: ebro onipcshmcronus ebro_ph (DBSERVERNAME) ebro_tcp ontliimccronus 1575 (DBSERVERALIAS) IBM Informix MaxConnect Configuration

102. 102 NETTYPE entry changes: ipcshm,1,20,CPU # tlitcp,5,200,CPU tliimc,1,2000,CPU or NET MaxConnect instance environment variables: IMCONFIG - location of MaxConnect config file - default is $INFORMIXDIR/etc/IMCconfig IMCSERVER - Client-side MaxConnect instance name IMCADMIN - Admin-side MaxConnect instance name IBM Informix MaxConnect Configuration

103. 103 IBM Informix MaxConnect config file contains: IMCLOG - location of MaxConnect message log IMCWORKERTHREADS - # of worker threads, range = 1 -> 64 IMCWORKERDELAY - amt of time MaxConnect will wait and pool messages to transmit to DB server default = 0, range = 0 -> 100,000 ms IMCTRANSPORTS - # of transport connections to DB server. Default = 2, range = 1 -> 64, ratio = 1/100 users IBM Informix MaxConnect Configuration

104. 104 IBM Informix MaxConnect Tuning  Use “onstat -g imc”, the imcadmin utility, or the ISA to monitor and tune IBM Informix MaxConnect  Tune the NETTYPE sessions parameter by monitoring the q-exceed and alloc/max values. May also need to tweak the IFX_NETBUF_PVPOOL_SIZE variable  Tune IMCTRANSPORTS by monitoring blocked or partial writes  Tune IMCWORKERTHREADS by monitoring AvgQlen if > 5, increase number of threads  Tune IMCWORKERDELAY by summing and comparing histograms against AvgQlen

105. 105 Large Objects, SBSpaces and Dynamic Lock Allocation Large Objects, SBSpaces and Dynamic Lock Allocation

106. 106 Simple Smart Large Objects and SBSpaces IBM IDS has 4 types of LOs:  Simple - BYTE & TEXT  Smart - CLOB & BLOB

107. 107 Large Objects and SBSpaces  Access to C/BLOBs is faster due to “portion-level” addressing  Default characteristics are easy to override  With sufficient resources, C/BLOBs are transactionally recoverable Simple Smart Logging Method logging depends on the table where the large-object is stored logging can be turned on or off at the large-object level Size2 GB maximum4 TB maximum Locking Granularity all or nothingcan lock portions or all of the object Reads and writes only the whole large-object can be read all or a portion of the large-object can be read Fragmentationall large-objects for a table are stored in the same blobspace ability to store the large-objects for one table in multiple sbspaces

108. 108 header (reserved) pages metadata area User Data Size and location of the metadata area is configurable at sbspace and/or chunk creation Metadata is always logged regardless of database or sbspace logging mode Large Objects and SBSpaces Smart BLOBs are stored in “smart” BLOBSpaces (sbspace) which contain meta-data as well as user data

109. 109 Large Objects and SBSpaces onspaces -S name -g pageunit -p pathname -o offset -s size [ -m pathname offset ] -Ms mdsize -Mo mdoffset -Df default list -t -Mo mdoffset offset, in kbs, into the disk partition where metadata will be stored -Ms mdsize the size, in kbs, of the metadata area for the initial chunk, remainder is user data space -- not recommended though possible. Each SLO needs ~500 bytes of metadata space -Df default list default specifications for objects stored in the sbspace. Comma separated, in double quotes (“ “)  -t new option in 9.3 for temporary sbspaces to store temporary smart LOBs

110. 110 Large Objects and SBSpaces

111. 111 Create a level 0 backup after creating a sbspace Large Objects and SBSpaces

112. 112 Large Objects and SBSpaces: New stuff Temporary SBSpaces:  Like temporary DBSpaces -- no logging whatsoever occurs  Set with the SBSPACETEMP $ONCONFIG parameter Temporary smart LOBs:  To create a temp smartblob, set the LO_CREATE_TEMP flag in the ifx_lo_specset_flags or mi_lo_specset_flags function. Use mi_lo_copy or ifx_lo_copy to create a permanent smart large object from a temporary smart large object.  If you put a temporary smartblob in a permanent sbspace it gets deleted at the end of session

113. 113 Metadata stealing:  40% of user data area reserved for either meta or user data, is allocated as needed to either portion (10% increments)  If metadata area is filling, messages in MSG_PATH  If metadata fills, and reserve is empty, no more S-BLOBs can be inserted even if user data area has free space  Must add more metadata space Large Objects and SBSpaces: New stuff header (reserved) pages metadata area User Data

114. 114 Large Objects and SBSpaces To add a chunk, you can specify whether the chunk will contain only user data, metadata, or both: onspaces -a sbspace -p pathname -o offset -s size [-Ms mdsize -Mo mdoffset | -U ] [ -m pathname offset ] To drop the sbspace: onspaces -d sbspacename -f (force)

115. 115 create table products (prod_sku int, prod_desc clob, prod_photo blob) put prod_desc in (sbspace_1, sbspace_2), prod_photo in (sbspace_9, sbspace_10), fragment by expression (prod_sku < 1000 in db_1, prod_sku >= 1000 in db_2); Large Objects and SBSpaces SLOs can be distributed in round-robin form among sbspaces like “regular” data Creates a 72 byte descriptor “handle” as opposed to 56 bytes with “simple” LOs

116. 116 Large Objects and SBSpaces IBM recommends that you convert “simple” LOs to their “smart” counterparts as soon as feasible You can, in many cases, use the "alter table" command to make the change(s): alter table t_name modify col_name [ clob | blob ] put col_name in ( sbspace(s) );

117. 117 Large Objects and SBSpaces Monitoring / tuning oncheck -cs -cS / -ps -pS to print/view metadata - ce / -pe has additional sbspace information onstat -g smb s | c | h | e | t smb s sbspace summary information smb c sbspace chunk information smb fdd LO file descriptor table smb lod LO header table smb t timing statistics for enabled server, not well documented smb e undocumented onstat -k has additional columns to document SLO byte range locks onstat -d has additional chunk-related columns for sbspaces

118. 118 Chunk adjunct partition Level 1 backup partition Level 2 backup partition LO header partition Sbspace descriptor partition user-data free-list partition header (reserved) pages metadata area Large Objects and SBSpaces Backing up and restoring SLOs and sbspaces now operates like regular dbspaces -- at the page level. As a result, you can now have granular and truly incremental SLO/sbspace backups.

119. 119 OtherspacesSbspaceheaderMetadatapagesArchive descriptor pages sbspace pages Large Objects and SBSpaces Backup:  Metadata LO headers are read and a list made of pages to be backed up  All individual page timestamps are compared too since updates can occur without affecting the LO header timestamp  Qualifying data is sent to the backup program through large buffer accesses Restore:  Sbspace header and metadata information is restored  Archive desriptor pages contain page addresses and extent sizes for the sbspace pages that follow. This is held in shared memory during the operation  Sbspace pages are restored to their correct places

120. 120 Dynamic Lock Allocation  In IBM IDS, the locking mechanism changes. LOCKS is now a starting allocation (resident portion)  If the lock table fills, additional locks are allocated (x 16) minimum allocation: smaller of current lock table or 100,00 (virtual portion)  To prevent lock table flooding, range locks will be upgraded to the whole SLO if a single transaction uses > 10% of lock table  “Byte range” locks are what lock portions of a SLO. They are sorted lists of rowids, partition #s, and byte address ranges

121. 121 Dynamic Lock Allocation  Range locks can be joined or split as needed if ranges to be locked adjoin, overlap, or become separate  Ranges locks are not stored in the lock hash table, rather an “intent” lock is placed for the LO header. Byte range locks are linked to associated hash intent lock then sorted by byte number  Byte ranges locks allocated if:  < 50% of lock table is being used  50% <= lock table <= 70%, whole SLO lock will be attempted, if unsuccessful, byte lock allocated  > 70% whole SLO locks only, existing locks will be upgraded automatically

122. 122 Complex and User-Defined Data Types Complex and User-Defined Data Types

123. 123 OpaqueDistinct Row Data Type Named Unnamed Collection Multiset List Set User-Defined Complex Extended Data Types BooleanInt8Serial8Lvarchar New Built-in Types Existing Built-in Types Data Types Complex and User-Defined Data Types

124. 124 Built-in Data Types New Built-In:  int8 (8 bytes)  serial8 (8 bytes)  Range is -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807  must add UC to serial8 to ensure uniqueness  one serial8 and serial per table  boolean  valid values: “t”, “f”, null. case sensitive  lvarchar (variable length character data type; 32k maximum – check your system for on-disk maximum)

125. 125 OpaqueDistinct Row Data Type Named Unnamed Collection Multiset List Set User-Defined Complex Extended Data Types BooleanInt8Serial8Lvarchar New Built-in Types Existing Built-in Types Data Types Complex and User-Defined Data Types

126. 126 No “alter type” statement, must drop and recreate Sys Adm is more complexIntuitive Not simple SQL Refers to a group of elements by a single name More complexLess coding DISADVANTAGESADVANTAGES Complex Data Types: Row analogous to C structure, come in two “kinds” NAMED - strongly typed, ID’ed by name, has inheritance, used to build columns and tables UNNAMED - weakly typed, ID’ed by structure, no inheritance, used to build columns Can contain built-in, collection, opaque, distinct, another row type data types  Caveat: no serial or serial8

127. 127 Named: create row type name_t (fname char(20), lname char(20)); create row type address_t (street_1 char(20), street_2 char(20), city char(20), state char(2), zip char(9)); create table student (student_id serial, name name_t, address address_t, company char(30)); Unnamed: ROW (a int, b char (10)) Note: is also equal to ROW(x int, y char(10)) create table part (part_id serial, cost decimal, part_dimensions row (length decimal,width decimal, height decimal, weight decimal)); Complex Data Types: Row

128. 128 Using Named Row Types in SQL statements: Insert statement: insert into student values (1234, row (“John”,”Doe”)::name_t, row ("1234 Main Street","", "Anytown","TX","75022")::address_t, "Informix Software") Select statement: select * from student where name.lname matches "Doe”; Result set: student_id 1234 name ROW('John ','Doe ') address ROW('1234 Main Street ',','Anytown ','TX','75022 ') company Informix Software Use of datatype keyword Cast the row type! Access data with ‘dot’ notation Complex Data Types: Row

129. 129 Complex Data Types: Row To drop a named row type: drop row type address_t restrict;

130. 130 OpaqueDistinct Row Data Type Named Unnamed Collection Multiset List Set User-Defined Complex Extended Data Types BooleanInt8Serial8Lvarchar New Built-in Types Existing Built-in Types Data Types Complex and User-Defined Data Types

131. 131 Complex Data Types: Collections Grouping of elements of the same datatype (char, int), max size = 32 KB Used when  The data is meaningless without the context of the other members in the collection (e.g., golf scores, to- do list, set of names)  Individual data elements are not likely to be directly queried by position  The maximum number of data elements is less than 32 Can be null

132. 132 Complex Data Types: Collections Three kinds of collections: Set - unordered, no duplicates allowed set {“apple”, ”orange”, ”grapefruit”, “plum”} Multiset - unordered, duplicates allowed multiset {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”} List - ordered, duplicates allowed list {“apple”, “orange”, “grapefruit”, “apple”, “plum”, “grapefruit”}

133. 133 Complex Data Types: Collections create table class (class_id serial, class_name varchar(60), description lvarchar, prereqs set(char(20) not null)); Insert syntax is similar to named row types: insert into class values (300, “Performance and Tuning”, “Covers advanced information on tuning the Informix Dynamic Server”, (SET{“RDD”,”BSQL”})); Use the “ in ” keyword to query values in a collection select * from class where (“ASQL”) in prereqs; define xyz char(20) define set_var set(char(20)) select prereqs into set_var from class where class_id = 300 foreach del_set_cursor for select * into xyz from table(set_var) if xyz matches “RDD” then delete from table(set_var) where current of del_set_cursor end if end foreach

134. 134 Complex Data Types: Collections You can not update one element in a collection, you must replace the whole collection: update class set prereqs = (set{“RDD”,”ASQL”,”BSQL”}) where class_id = 300; update class set prereqs = set_char where class_id = 300;

135. 135 OpaqueDistinct Row Data Type Named Unnamed Collection Multiset List Set User-Defined Complex Extended Data Types BooleanInt8Serial8Lvarchar New Built-in Types Existing Built-in Types Data Types Complex and User-Defined Data Types

136. 136 User-Defined Data Types: Distinct Two user-defined data types (UDTs): Distinct  data type modeled on an existing data type  has a unique name to distinguish it from other similar “types”  inherits the internal structure from the source type  might have operations and casts defined over its source type Opaque  data type that is unknown to the database server  you must define the internal structure, functions, and operations

137. 137 create distinct type dollar as decimal; create distinct type aus_dollar as decimal; create table sales ( sku int, sales_date date, us_sales dollar, aus_sales aus_dollar); insert into sales values (1234, today, 15.0::dollar,0::aus_dollar); insert into sales values (5678, today, 0::dollar, 75.0::aus_dollar); select sku, (sum(us_sales) + sum(aus_sales)) from sales where sales_date = today group by 1; error: 674 - routine (plus) can not be resolved User-Defined Data Types: Distinct

138. 138 Need to create some UDFs that handle the type and value conversion for you: create function usdlr_to_ausdlr(parm1 dollar) returning aus_dollar specific usd_to_ausd; return (parm1::decimal * 1.8)::aus_dollar; end function; create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar; end function; select sku, (sum(us_sales) + sum(ausdlr_to_usdlr(aus_sales))::dollar) from sales where sales_date = today group by 1; User-Defined Data Types: Distinct

139. 139 User-Defined Data Types: Opaque An opaque data type stores a single “value” that cannot be divided into components by the engine. Implemented as C or Java structures and manipulated by a set of routines written in C or Java An opaque “value” is stored in its entirety by the engine without any interpretation of the contents or its structure All access to an opaque type is through functions written by the user. You define the storage size of the data type and input and output routines

140. 140 User-Defined Data Types: Opaque 1. Create the C / Java data structure to represent the internal data structure 2. Write the support functions in C / Java 3. Register the opaque data type with the “create opaque type” statement create opaque type type_name ( internallength = length, alignment = num_bytes ); length is in bytes, alignment = 1,2,4,8 bytes (default = 4) create opaque type type_name ( internallength = variable, maxlen = length ); length - default = 2 KB, max value = 32 kb 4. Register the support functions with the “create function” statement. If in Java, will be stored in sbspace(s)

141. 141 create opaque type my_type(internallength=8, alignment=4); create function support_in(lvarchar) returning my_type with (not variant); external name “ /funcs/my_type.so ” language C end function; create implict cast (lvarchar as my_type with support_in); 5. Grant access to the opaque data type and support functions 6. Write any user-defined functions needed to support the opaque data type - input, output, destroy, compare, aggregates, send, receive, etc. 7. Provide any customized secondary-access methods for creating indexes User-Defined Data Types: Opaque

142. 142 Casts and Casting

143. 143  Casts allow you to make comparisons between values of different data types or substitute a value of one data type for a value of another data type. create function ausdlr_to_usdlr(parm1 aus_dollar) returning dollar specific ausd_to_usd; return (parm1::decimal / 1.8)::dollar; end function;  Engine provides a number of “built-in” casts (int to decimal, numeric to char, etc.) for most built-in datatypes  Must create user-defined casts for user-defined types. Must be unique with respect to source and target data types  Can not create casts for collections, LOBs, or unnamed row types Casts and Casting

144. 144 Casts and Casting Two kinds of user-defined casts: explicit... where price >= (cast aus_dollar as dollar)....... return (parm1::decimal / 1.8)::dollar; create explicit cast (aus_dollar as dollar with us_dlr_to_us_dlr); implicit create implicit cast (aus_dollar as dollar); create implicit cast (aus_dollar as dollar with aus_dlr_to_us_dlr); Automatically invoked when: one data type is passed to a user-defined routine whose parameters are of another data type expressions are evaluated that need to operate on two similar data types

145. 145 Casts and Casting – Implicit Casts select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved. create implicit cast (aus_dollar as dollar); select sum(us_sales) + sum(aus_sales) from sales; (expression) 120.00 drop cast (aus_dollar as dollar); create implicit cast (aus_dollar as dollar with ausdlr_to_usdlr); select sum(us_sales) + sum(aus_sales) from sales; (expression) 80.00

146. 146 select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved create explicit cast (aus_dollar as dollar); select sum(us_sales) + sum(aus_sales) from sales; # 674: Routine (plus) can not be resolved select sum(us_sales) + sum(aus_sales)::dollar from sales; (expression) 120.00 drop cast (aus_dollar as dollar); create explicit cast (aus_dollar as dollar with ausdlr_to_usdlr); select sum(us_sales) + sum(aus_sales)::dollar from sales; (expression) 80.00 Casts and Casting – Explicit Casts

147. 147 Previous examples were “straight” casts. You can also create cast “functions” to cast types with dissimilar data structures 1.write the cast function in C / Java / SPL create function opaque_to_opaque (input_arg my_type_1) returns my_type_2 return cast(cast(input_arg as lvarchar) as my_type_2); end function; 2. register the cast function with the “ create function ” command 3. register the cast with the “ create cast ” command create explicit cast (my_type_1 as my_type_2 with opaque_to_opaque); Casts and Casting

148. 148 User-defined Routines (UDRs) User-defined Routines (UDRs)

149. 149 UDRs There are User Defined Functions and Procedures (UDFs and UDPs) Can be internal or external in nature. Internal are written in SPL, external can be written in C or Java. If written in Java, compile into a class then.jar file. When registered in the engine,.jar file will be brought into a sbspace then executed when needed by the JVM in the engine. If written in C, create a shared object library in a directory owned by “informix” with 755 permissions.

150. 150 UDRs UDRs should be concise and precise. Don’t want engine wasting resources plowing though verbose code. The values returned should be relevant from a business perspective (e.g. proper domain) Make sure you include error handling in the code **ALWAYS** use a “specific” name, or alias, for every UDR to properly identify each one - used to drop, grant, or revoke permissions as well as to permit function overloading

151. 151 UDRs Function overloading occurs when two or more functions have the same name but different signatures signature = UDR name and parameter list create function plus (in_1 dollar, in_2 aus_dollar).. create function plus (in_1 aus_dollar, in_2 lira)... create function plus (in_1 lira, in_2 aus_dollar)... {Note: these probably should be formalized into user-defined aggregates} When properly registered, instance will use the data types passed to determine which UDR should be used

152. 152 UDRs create function routine_name (param_list) { returns | returning } typename [ specific specific_name ] [ with ( internal | handlesnulls | [ [not] variant] | class=“ vp_class ”) ] external name ‘ full_path_name_of_file‘ language language [ [not] variant ] end function; in the $ONCONFIG file: VPCLASS fince_vp,num=2 # vps for finance UDRs or onmode -p +2 fince_vp USE UDVPs FOR UDRs!!!!

153. 153 UDRs To drop UDRs, must either use full signature drop function plus(lire, aus_dollar); or the “specific” name drop specific function lir_to_ausdlr;

154. 154 UDRs Default behavior of UDRs is single-threaded, can be “parallelized” if the following conditions are met:  only in DSS environments (PDQPRIORITY > 1)  "parallelizable" keyword is added to UDR creation statement  external UDRs only (C / Java) that use PDQ thread- safe SAPI calls (see DataBlade manual for list of SAPI calls)  no complex types in input parameters or return values  multiple fragments must be scanned

155. 155 UDRs  Not directly called (singleton execution)  not an iterator - called in loops to return multiple values similar to with resume in SPL On some UDRs, you can add “selectivity” and “cost” modifiers that will affect optimizer activities and query plans used. Not really familiar with this

156. 156 UDRs create function bigger(int,int) returns int specific big_parallel with(parallelizable,class=“test_vps”) external name “/path/plludr.bld(bigger)” language C; alter function bigger (int,int) with (add parallelizable, add class=“test_vps”); Can monitor the parallel execution of UDRs through  “set explain out” command  the ISA  onstat -g mgm  onstat -g ses sess_id  onstat -g ath

157. 157 DataBlade Modules and the Blade Manager DataBlade Modules and the Blade Manager

158. 158 DataBlade Modules  DataBlade Modules extend the functionality of the engine by adding new UDTs, casts, interfaces, UDRs, error messages, client code (where necessary) to manipulate the UDTs  Functionality available through SQL, SPL, API calls to external functions  Can be mixed and matched to support any application  While the code is loaded at an instance level, blade registration occurs at a database level. With registration information contained in the database system tables, if you drop the database, blades become unregistered

159. 159 DataBlade Modules Install the blade in $INFORMIXDIR/extend (tar, cpio, ?) May have its own “install” script to brand binaries, use serial # / license key from engine install unless third-party requires unique key setup Use the Blade Manager to manage blade registration graphical version available with NT and ISA command line only on Unix / Linux Is case sensitive

160. 160 DataBlade Modules BladeManager supports the following commands: list [database] - print list of registered DataBlade modules set user [user] set server [server] show modules - print list of DataBlade modules on server show databases - print list of databases on server show servers - print list of servers show client - print DataBlade modules with files on client register [module] [database] unregister [module] [database] add client [module] - install DataBlade module's client files del client [module] - remove DataBlade module's client files info [module] - print information about a DataBlade module del logs - delete all the log files show log - prompt to display a log show last log - display log from last action help, ? bye, quit, exit set confirm on|off

161. 161 DataBlade Modules mustang>show database Databases on server: agg1 dcs_demo dps_demo ireach kiosk media360 my_test_db visclass20 mustang>list media360 DataBlade modules registered in database media360: lld.1.20.tc2 TXT.1.10.TC5 ifxrltree.2.00 ifxbuiltins.1.1 web.4.10.TC1 VTS.1.20.TC1 video.2.00.TC2

162. 162 HDR / Enterprise Replication – A Snapshot of New Features HDR / Enterprise Replication – A Snapshot of New Features

163. 163 HDR and 9.4  HDR now supports:  Enterprise Replication (co-existence)  Logged Smart Blob Spaces  Some DataBlades (when associated smart blob spaces are logged)  Supported DataBlades: Spatial Geodetic Web Time-Series Excalibur Text

164. 164 ER – A Snapshot New Stuff  DataSync performance improvements  Spooling improvements  Serial column primary key support  Changed columns sent with update, partial row update  Smart LOBs support  No raw table support  Distinct / opaque type support but not complex OLD!!  9.4 Feature: Transaction size limit increases from amount of shared memory to 4 TB  9.4 Feature: support for additional extensible types:  Row Types  Lists  Sets  Multi-Sets

165. 165 DataSync performance improvements include:  Support of intra-replicate parallelism  Parallel apply with ordered commits  Operations for a given row are serialized  Necessary to prevent ‘delete before insert’  Some tables are always serialized  Page level locking or small tables  Out-of-order commits  Allows commit order on target to be different than it was on source  No prior cross replicate relationships  No prior referential constraint relationships ER – A Snapshot New Stuff

166. 166  Decreased logical log consumption  DataSync now uses buffered logging  Acknowledgements are ‘buddy-bunched’  Shipping of ACKs coordinated with log flush, every 5 seconds or 50 Txn  Is automatic and self-regulated  CDRM_Monitor thread  Maintains weighted history of lock failures  Controls the degree of parallelism allowed  Monitors distance before DDRBLOCK  Coordinates ACKs with log flush  Recalculates Table Statistics, user may need to run update statistics on a target only system  Does not provide for intra-transaction parallelism (is not PDQ) ER – A Snapshot New Stuff

167. 167  9.30 ER interoperates with ER on 7.31 or 9.2x (but NOT 7.30)  Limitations  9.30 does not support replicate groups, all groups should be deleted from 9.2x/7.31 servers before inter-operation begins  9.2x/7.31 do not support replicate sets; no replicate sets should be defined until all 9.2x/7.31 servers have been converted to 9.30  PTS 128485: must be applied to 9.2x/7.31 server(s) before beginning interoperation ER – A Snapshot New Stuff

168. 168 ER – A Snapshot Result!! 9.30 9.21 Apply on Source Apply on Target

169. 169 Introduction to Java in the Server Introduction to Java in the Server

170. 170 EJB Java JVM Applet JavaBean RMI | Corba | IIOP Web Server Applet JavaBean Java DBDK JavaBean EJB Java in the Server J/Foundation is the marriage of the Java and SQL languages Can write internal or external Java routines that are accessed natively by the engine

171. 171 Java in the Server: How It Works Java UDRs execute on Java Virtual Processors  Java VPs have the same capability as CPU VP  It can process any SQL query  Avoids the inter-process communication overhead for executing Java UDRs  The Java Virtual Machine (JVM) is embedded directly into the VP code Java runs threads in parallel Dynamically load Java VPs  Increase JVPs to balance application load

172. 172  IBM IDS 9.30 was enhanced to take advantage of JDK 1.3  There are 3 different Java VM’s  Classic VM: jdk1.2.x VM  Client HotSpot VM: tuned for user-interaction apps needing short start- up & smaller memory footprint (JIT less aggressive) HotSpot is from Sun, IBM has other VMs that are not based on HotSpot technology.  Server HotSpot VM: tuned for servers, used in our J/Foundation product  Classic VM was used for IDS 9.21, and is not supported for the 9.30 release. This is in line with the vendor roadmaps for JDK. Java in the Server: New Stuff

173. 173 Performance improvements in 9.3 J/Foundation include:  thread pooling,  code optimization of Java UDR invocation,  support for HotSpot Server VM Performance improvements will vary by the application, but internally we’ve seen an improvement of approximately 135% for our test suites. Java in the Server: New Stuff

174. 174 Java in the Server: Configuration New ONCONFIG Parameters: VPCLASS jvp,num=1 JVPJAVAHOME /usr/java JVPHOME /opt/informix/extend/krakatoa JVPPROPFILE /opt/informix/extend/krakatoa/.jvpprops JVPLOGFILE /opt/informix/jvp.log

175. 175  9.21 $ONCONFIG had the settings  JDKVERSION 1.2  JVPJAVAVM hpi:jvm:java:net:math:zip:jpeg:dcpr  9.30 $ONCONFG will need the settings  JDKVERSION 1.3  JVPJAVAVM hpi:server:verify:java:net:zip:jpeg Look at examples in onconfig.std Java in the Server: Configuration

176. 176 Java in the Server: JDBC Driver Information We provide a “Type 4” JDBC driver Type 1 JDBC-ODBC bridge provides JDBC access via most ODBC drivers Type 2 A native API and partly Java driver that converts JDBC calls into calls on the client API for Oracle, Sybase, Informix, DB2 etc. Type 3 A net protocol, all Java driver that translates JDBC calls into a DBMS independent net protocol which is then translated to a DBMS protocol by a server Type 4 A native protocol all-Java driver converts JDBC calls into the network protocol used by DBMSs directly. This allows a direct call from the client machine to the DBMS server

177. 177 Java in the Server Development  Install JDK 1.3 on development machine(s)  Install any O/S patches required by vendor  We recommend that you recompile your Java UDR’s using JDK 1.3  For more information on development, please see the user guides and the Informix Developers Network website, “Java” and “DataBlade Developers” corners

178. 178 Java in the Server Debugging Tip #1 If you see this: 9431: Can't find system class or method or library (/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjv m.so). It may mean that you have an invalid entry for JVPJAVAM in your ONCONFIG file

179. 179 Java in the Server Debugging Tip # 2 If you see this in MSG_PATH file: 10:33:12 Cannot load lib /vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libjv m.so: error : ld.so.1: oninit: fatal: libCrun.so.1: open failed: No such file or directory This could also be a sign of an invalid entry for JVPJAVAM in your $ONCONFIG file

180. 180 Java in the Server Debugging Tip # 3 If you see this: 9431: Can't find system class or method or library (/vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/server/libmath.so). Or this in MSG_PATH: 10:33:12 Cannot load lib /vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so: error : ld.so.1: oninit: fatal: /vobs/tristarm/sqldist/extend/krakatoa/jre/lib/sparc/libmath.so : open failed: No such file or directory You have the math library specified in your JVPJAVAM, remove it.

181. 181 Questions???