1. Copyright 2007, Information Builders. Slide 1 Relational Efficiencies: Part II Overview and JOIN / Sort Scenarios Renee Teatro Information Builders

2.  Optimization Overview  Definition  Command syntax and settings  Optimization Hierarchy  Projection and Selection  Tracing  JOIN Scenarios  Sort Scenarios Relational Efficiencies Agenda

3. Interface optimization is the degree to which a TABLE request is translated to SQL In other words, the process in which the interface translates projection, selection, JOIN, sort, and aggregation operations of a report request into its SQL equivalent and passes it to the RDBMS for processing  TABLE, MODIFY, MAINTAIN  SQL  Direct SQL Passthru  Passing JOINs Relational Efficiencies: Overview Definition of Optimization

4. Copyright 2007, Information Builders. Slide 4 TSO CMS SQL DB2 SQLDS SQLDBC SQLORA SQLMSS SQLIDMS SET OPTIMIZATION SQLJOIN ON OFF SQL FOCUS * Components Description Target Database EngineDB2for DB2 SQLDSfor SQL/DS SQLDBCfor Teradata SQLORAfor Oracle SQLMSS for MS SQL Server SQLIDMS for IDMS SQL OUTER Relational Efficiencies : Overview Optimization Command SQL DB2 SET OPTIMIZATION ON SQL SQLORA SET OPTIMIZATION ON

5. Relational Efficiencies : Overview Optimization Hierarchy JOIN SORT AGGREGATION If the JOIN optimization fails, so will the SORT and AGGREGATION. If the SORT optimization fails, so will AGGREGATION. AGGREGATION will occur only if JOIN and SORT operations were optimized (and AGGREGATION requested).

6. Copyright 2007, Information Builders. Slide 6 Relational Efficiencies : Overview Selection and Projection What is selection and projection? Regardless of the optimization setting, the data adapter will always try and translate selection and projection. Why are selection and projection so important? They are both processes that reduce the volume of data being returned from the RDBMS, which helps to improve performance, efficiency and report response time. SelectionThe process of retrieving the table rows that meet the request criteria. ProjectionThe process of retrieving the table columns that meet the request criteria.

7.  SET TRACEON=component//destination  Component  SQLDI - FSTRACE - All Interface-RDBMS activity  SQLAGGR - FSTRACE3 - Optimization messages  STMTRACE - FSTRACE4 - SQL only  SQLCALL - commands and data exchange between the physical and the logical layers of the data adapter  Destination  FSTRACE - allocation for the ddname of FSTRACE  CLIENT - displays client session to the screen NOTE: Make sure to SET TRACEUSER=OFF afterwards Relational Efficiencies : Overview Data Adapter TRACE Facility

8. Relational Efficiencies : Overview Data Adapter TRACE Facility SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=SQLAGGR//CLIENT SET TRACEON=STMTRACE//CLIENT SET XRETRIEVAL=OFF TABLE FILE … SET XRETRIEVAL=[ON | OFF]  ON – the data adapter sends the request to the RDBMS and it processes the request  OFF – the data adapter attempts to optimize the request and trace output is generated but no RDBMS processing is done

9. Copyright 2007, Information Builders. Slide 9 JOIN Scenarios

10. Relational Efficiencies JOIN: Performance related to KEYS attribute JOINing 3 ORACLE tables results in very poor performance. These are newly created MFDs and AFDs via CREATE SYNONYM and the first time any requests / JOINs are being issued. Scenario Enable data adapter tracing to see if the JOIN is being passed over and if not, why. Suggestion SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=STMTRACE//CLIENT SET TRACEON=SQLAGGR//CLIENT SET XRETRIEVAL=OFF JOIN CLEAR * JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 JOIN SECTION_ID IN COURSE TO ALL SECTION_ID IN ENROLLMENT AS J2 TABLE FILE COURSE …

11. Relational Efficiencies JOIN: Performance related to KEYS attribute (FOC2510) FOCUS-MANAGED JOIN SELECTED FOR FOLLOWING REASON(S): (FOC2505) A SEGMENT IN THE STRUCTURE IS NON-KEYED: : SECTION (FOC2517) FST. OR LST. WHERE SORT FIELDS DO NOT COVER KEY (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2592) RDBMS-MANAGED JOIN HAS BEEN DISABLED SELECT T1."COURSE_NO",T1."DESCRIPTION" FROM STUDENT.COURSE T1; SELECT T2."SECTION_ID",T2."SECTION_NO",T2."START_DATE_TIME" FROM STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = :0001); SELECT T3."STUDENT_ID" FROM STUDENT.ENROLLMENT T3 WHERE (T3."SECTION_ID" = :0001); Output The.acx (ACCESS FILE DESCRIPTION) should have KEYS= with a NON-ZERO value to indicate the PRIMARY KEY for the table in question. The PRIMARY KEY is made up of the FIRST n fields in the.mas (MASTER FILE DESCRIPTION). Correction

12. Relational Efficiencies JOIN: Performance related to KEYS attribute SEGNAME=COURSE, TABLENAME=STUDENT.COURSE, CONNECTION=STUDENTA.IBI.COM, KEYS=1,$ SEGNAME=SECTION, TABLENAME=STUDENT.SECTION, CONNECTION=STUDENTA.IBI.COM, KEYS=0,$ *** Change to KEYS=1 SEGNAME=ENROLLMENT, TABLENAME=STUDENT.ENROLLMENT, CONNECTION=STUDENTA.IBI.COM, KEYS=2,$ Correction (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."COURSE_NO",T1."DESCRIPTION",T2."SECTION_ID", T2."SECTION_NO",T2."START_DATE_TIME",T3."STUDENT_ID", T3."SECTION_ID" FROM STUDENT.COURSE T1,STUDENT.SECTION T2, STUDENT.ENROLLMENT T3 WHERE (T2."COURSE_NO" = T1."COURSE_NO") AND (T3."SECTION_ID" = T2."SECTION_ID") ORDER BY T1."COURSE_NO", T2."SECTION_ID",T3."STUDENT_ID",T3."SECTION_ID";

13. Relational Efficiencies JOIN: Performance related to MULTI-PATH JOINing 3 ORACLE tables results in very poor performance. Certain JOIN combinations using these same tables do not have this issue. Scenario Suggestion SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=STMTRACE//CLIENT SET TRACEON=SQLAGGR//CLIENT SET XRETRIEVAL=OFF JOIN CLEAR * JOIN STUDENT_ID IN STUDENT TO ALL STUDENT_ID IN ENROLLMENT AS J1 JOIN STUDENT_ID IN STUDENT TO ALL STUDENT_ID IN GRADE AS J2 TABLE FILE STUDENT PRINT SECTION_ID ENROLL_DATE GRADE_TYPE_CODE BY STUDENT_ID BY LAST_NAME BY FIRST_NAME END

14. Relational Efficiencies JOIN: Performance related to MULTI-PATH (FOC2510) FOCUS-MANAGED JOIN SELECTED FOR FOLLOWING REASON(S): (FOC2512) MULTI-PATH REQUEST (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2592) RDBMS-MANAGED JOIN HAS BEEN DISABLED SELECT T1."STUDENT_ID",T1."FIRST_NAME",T1."LAST_NAME" FROM STUDENT.STUDENT T1; SELECT T2."GRADE_TYPE_CODE" FROM STUDENT.GRADE T2 WHERE (T2."STUDENT_ID" = :0001); SELECT T3."SECTION_ID",T3."ENROLL_DATE" FROM STUDENT.ENROLLMENT T3 WHERE (T3."STUDENT_ID" = :0001); Output  Adjust JOIN field(s) to create a SINGLE PATH structure  SQL Passthru  Create relational VIEW on RDBMS – single segment MFD  Create HOLD files & JOIN HOLD files vs. FOCUS-managed JOIN Correction

15. Relational Efficiencies JOIN: Performance related to MULTI-PATH CHECK FILE STUDENT PICTURE

16. Relational Efficiencies JOIN: Performance related to MULTI-PATH JOIN CLEAR * JOIN STUDENT_ID IN STUDENT TO ALL STUDENT_ID IN ENROLLMENT AS J1 JOIN ENROLLMENT.STUDENT_ID AND SECTION_ID IN STUDENT TO ALL STUDENT_ID AND SECTION_ID IN GRADE AS J2 -*CHECK FILE STUDENT PICTURE Correction (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."STUDENT_ID",T1."FIRST_NAME",T1."LAST_NAME", T2."STUDENT_ID",T2."SECTION_ID",T2."ENROLL_DATE", T3."GRADE_TYPE_CODE" FROM STUDENT.STUDENT T1,STUDENT.ENROLLMENT T2,STUDENT.GRADE T3 WHERE (T2."STUDENT_ID" = T1."STUDENT_ID") AND (T3."STUDENT_ID" = T2."STUDENT_ID") AND (T3."SECTION_ID" = T2."SECTION_ID") ORDER BY T1."STUDENT_ID",T1."LAST_NAME", T1."FIRST_NAME";

17. Relational Efficiencies JOIN: Performance related to MULTI-PATH

18. Relational Efficiencies JOIN: Separate Node / Subsystem JOINing 3 ORACLE tables results in very poor performance. Certain JOIN combinations using these same tables do not have this issue. Scenario Suggestion SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=STMTRACE//CLIENT SET TRACEON=SQLAGGR//CLIENT SET XRETRIEVAL=OFF JOIN CLEAR * JOIN STUDENT_ID IN STUDENT TO ALL STUDENT_ID IN ENROLLMENT AS J1 JOIN ENROLLMENT.STUDENT_ID AND SECTION_ID IN STUDENT TO ALL STUDENT_ID AND SECTION_ID IN GRADE AS J2 TABLE FILE STUDENT…

19. Relational Efficiencies JOIN: Separate Node / Subsystem (FOC2510) FOCUS-MANAGED JOIN SELECTED FOR FOLLOWING REASON(S): (FOC2519) THE JOINED SEGMENTS RESIDE IN MORE THAN ONE NODE OR SUBS (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2592) RDBMS-MANAGED JOIN HAS BEEN DISABLED SELECT T1."STUDENT_ID",T1."FIRST_NAME",T1."LAST_NAME" FROM STUDENT.STUDENT T1; SELECT T2."STUDENT_ID",T2."SECTION_ID",T2."ENROLL_DATE" FROM STUDENT.ENROLLMENT T2 WHERE (T2."STUDENT_ID" = :0001); SELECT T3."GRADE_TYPE_CODE" FROM STUDENT.GRADE T3 WHERE (T3."STUDENT_ID" = :0001) AND (T3."SECTION_ID" = :0002); Output  CONNECTION / HOST values in ACX file is not the same for all tables involved in JOIN structure  Another MFD/AFD by the same name higher in the application path list with the SAME name as one of the JOINed tables that did indeed point to a different node. Correction

20. Relational Efficiencies JOIN: Separate Node / Subsystem SEGNAME=STUDENT, TABLENAME=STUDENT.STUDENT, CONNECTION=STUDENTA.IBI.COM, KEYS=2,$ SEGNAME=GRADE, TABLENAME=STUDENT.GRADE, CONNECTION=STUDENTB.IBI.COM, KEYS=4,$ *** Change to STUDENTA.IBI.COM SEGNAME=ENROLLMENT, TABLENAME=STUDENT.ENROLLMENT, CONNECTION=STUDENTA.IBI.COM, KEYS=2,$ Correction (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."STUDENT_ID",T1."FIRST_NAME",T1."LAST_NAME", T2."STUDENT_ID",T2."SECTION_ID",T2."ENROLL_DATE", T3."GRADE_TYPE_CODE" FROM STUDENT.STUDENT T1,STUDENT.ENROLLMENT T2,STUDENT.GRADE T3 WHERE (T2."STUDENT_ID" = T1."STUDENT_ID") AND (T3."STUDENT_ID" = T2."STUDENT_ID") AND (T3."SECTION_ID" = T2."SECTION_ID") ORDER BY T1."STUDENT_ID",T1."LAST_NAME", T1."FIRST_NAME";

21. Copyright 2007, Information Builders. Slide 21 Sort Scenarios

22. Relational Efficiencies Sort Management by the RDBMS

23.  BY and ACROSS equate to SQL ORDER BY clause  BY / FOR ROWS OVER syntax will translate JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 TABLE FILE COURSE PRINT COST BY SECTION_ID FOR COURSE_NO 103 OVER 100 OVER 150 OVER 136 OVER 121 END (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."COURSE_NO",T1."COST",T2."SECTION_ID" FROM STUDENT.COURSE T1,STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") AND (T1."COURSE_NO" IN(100, 103, 121, 136, 150)) ORDER BY T2."SECTION_ID",T1."COURSE_NO";

24. Relational Efficiencies Sort Management by FOCUS / WebFOCUS

25. DEFINE FILE COURSE SHORT_DESC/A5=EDIT(DESCRIPTION,'99999'); COST_INC/P8=COST *.10; END TABLE FILE COURSE PRINT COST BY COURSE_NO BY COST_INC -*BY SHORT_DESC END (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."COURSE_NO",T1."COST" FROM STUDENT.COURSE T1; (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."COURSE_NO",T1."DESCRIPTION",T1."COST" FROM STUDENT.COURSE T1;

26. Relational Efficiencies Sort Management by the RDBMS DEFINE FILE COURSE SHORT_DESC/A5=EDIT(DESCRIPTION,'99999'); COST_INC/P8=COST *.10; END TABLE FILE COURSE SUM COST BY COURSE_NO BY COST_INC -*BY SHORT_DESC END AGGREGATION DONE... SELECT T1."COURSE_NO",(T1."COST" *.1), SUM(T1."COST") FROM STUDENT.COURSE T1 GROUP BY T1."COURSE_NO",(T1."COST" *.1) ORDER BY T1."COURSE_NO",(T1."COST" *.1); AGGREGATION DONE... SELECT T1."COURSE_NO",(SUBSTR(T1."DESCRIPTION", 1, 5)), SUM(T1."COST") FROM STUDENT.COURSE T1 GROUP BY T1."COURSE_NO", (SUBSTR(T1."DESCRIPTION", 1, 5)) ORDER BY T1."COURSE_NO", (SUBSTR(T1."DESCRIPTION", 1, 5));

27. Relational Efficiencies Sort Management by the RDBMS DEFINE FILE SSSALES MONTH_COMPONENT/A3=HNAME(ORD_DATE,'MONTH','A3'); END TABLE FILE SSSALES SUM QTY BY MONTH_COMPONENT -*BY ORD_DATE END Tracing via DEFINE field MONTH_COMPONENT sort: (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2609) CANNOT AGGREGATE BY NON-SQLIZABLE SORT KEY : MONTH_COMP SELECT T1."ord_date",T1."qty" FROM pubs.dbo.sales T1; Tracing via REAL field ORD_DATE sort: AGGREGATION DONE... SELECT T1."ord_date", SUM(T1."qty") FROM pubs.dbo.sales T1 GROUP BY T1."ord_date" ORDER BY T1."ord_date";

28. Relational Efficiencies Sort Management by the RDBMS DEFINE FILE SSSALES MONTH_COMPONENT/A3=HNAME(ORD_DATE,'MONTH','A3'); END TABLE FILE SSSALES SUM QTY BY ORD_DATEM -*BY MONTH_COMPONENT -*BY ORD_DATE END AGGREGATION DONE... SELECT T1."ord_date", SUM(T1."qty") FROM pubs.dbo.sales T1 GROUP BY T1."ord_date" ORDER BY T1."ord_date"; FILENAME=SSSALES, SUFFIX=SQLMSS, $ SEGMENT=SSSALES, SEGTYPE=S0, $ FIELDNAME=STOR_ID, ALIAS=stor_id, USAGE=A4, ACTUAL=A4, $ FIELDNAME=ORD_NUM, ALIAS=ord_num, USAGE=A20, ACTUAL=A20, $ FIELDNAME=TITLE_ID, ALIAS=title_id, USAGE=A6, ACTUAL=A6, $ FIELDNAME=ORD_DATE, ALIAS=ord_date, USAGE=HYYMDs, ACTUAL=HYYMDs, $ FIELDNAME=ORD_DATEM,ALIAS=ord_date, USAGE=MT, ACTUAL=DATE,$ FIELDNAME=QTY, ALIAS=qty, USAGE=I11, ACTUAL=I4, $

29.  From a performance standpoint, consider using TABLEF instead of TABLE for requests that you know are being returned in the requested sort order. This will free FOCUS / WebFOCUS from having to verify the sort order by loading its internal matrix. FOCUS / WebFOCUS will act directly on the data returned by the RDBMS.  ACROSS phrases are not supported.  It must be verified that an ORDER BY statement has been generated in the SQL for the correct columns prior to using TABLEF.  Multi-verb requests are not supported.  RETYPE is not available.  TABLEF can hold locks on data pages until the data adapter issues a COMMIT (usually when the report displays). Locks may prevent access to the data by other applications. (online core FOCUS)  Certain prefix operators are not available (TOT., PCT.)  Computes that use prefix operators are not supported. Relational Efficiencies Sort: Using TABLEF

30. Copyright 2007, Information Builders. Slide 30