1. Copyright 2007, Information Builders. Slide 1 Relational Efficiencies: Part III Overview and Aggregation / Selection Scenarios Renee Teatro Information Builders

2.  Optimization Overview  Definition  Command syntax and settings  Optimization Hierarchy  Tracing  Aggregation Scenarios  Selection Scenarios Relational Efficiencies Agenda

3. Interface optimization is the degree to which a TABLE request is translated to SQL In other words, the process in which the interface translates projection, selection, JOIN, sort, and aggregation operations of a report request into its SQL equivalent and passes it to the RDBMS for processing  TABLE, MODIFY, MAINTAIN  SQL  Direct SQL Passthru  Passing JOINs Relational Efficiencies: Overview Definition of Optimization

4. Copyright 2007, Information Builders. Slide 4 TSO CMS SQL DB2 SQLDS SQLDBC SQLORA SQLMSS SQLIDMS SET OPTIMIZATION SQLJOIN ON OFF SQL FOCUS * Components Description Target Database EngineDB2for DB2 SQLDSfor SQL/DS SQLDBCfor Teradata SQLORAfor Oracle SQLMSS for MS SQL Server SQLIDMS for IDMS SQL OUTER Relational Efficiencies : Overview Optimization Command SQL DB2 SET OPTIMIZATION ON SQL SQLORA SET OPTIMIZATION ON

5. Relational Efficiencies : Overview Optimization Hierarchy JOIN SORT AGGREGATION If the JOIN optimization fails, so will the SORT and AGGREGATION. If the SORT optimization fails, so will AGGREGATION. AGGREGATION will occur only if JOIN and SORT operations were optimized (and AGGREGATION requested).

6.  SET TRACEON=component//destination  Component  SQLDI - FSTRACE - All Interface-RDBMS activity  SQLAGGR - FSTRACE3 - Optimization messages  STMTRACE - FSTRACE4 - SQL only  SQLCALL - commands and data exchange between the physical and the logical layers of the data adapter  Destination  FSTRACE - allocation for the ddname of FSTRACE  CLIENT - displays client session to the screen NOTE: Make sure to SET TRACEUSER=OFF afterwards Relational Efficiencies : Overview Data Adapter TRACE Facility

7. Relational Efficiencies : Overview Data Adapter TRACE Facility SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=SQLAGGR//CLIENT SET TRACEON=STMTRACE//CLIENT SET XRETRIEVAL=OFF TABLE FILE … SET XRETRIEVAL=[ON | OFF]  ON – the data adapter sends the request to the RDBMS and it processes the request  OFF – the data adapter attempts to optimize the request and trace output is generated but no RDBMS processing is done

8. Copyright 2007, Information Builders. Slide 8 Aggregation Scenarios

9. Aggregation translation is important  RDBMS aggregation is more efficient: indices  An aggregated answer set reduces FOCUS-to-RDBMS communication  A smaller answer set reduces FOCUS local processing FOCUSSQL SUM..., WRITE... BY field SUM., CNT., MIN., MAX., AVE. SELECT SUM(...)GROUP BY column ORDER BY column SUM(...), COUNT(*), MIN(...), MAX(...), AVG(...) Relational Efficiencies Efficient Aggregation

10.  Verbs: SUM, COUNT, WRITE  Direct operators: MIN., MAX., AVE.  Aggregating DEFINEd fields:  Constant DEFINEd fields translated with CNT. The following defined expressions can be translated Type of expressions Expression components Examples Arithmetic Valued (Expressions that return a single number) Character String Valued (Expressions that return a character string) Real fields of datatype I, P, D, or F Numeric constants Arithmetic operators (+, -, *, / ) Real fields of datatype A String constants Concatenation operator (I) EDIT of alphanumeric fields DEFINE FILE ORAEMP NEW_SAL/D12.2=(CSAL * 0.10) + CSAL ; END DEFINE FILE ORAEMP NAME/A18=EDIT(FN,‘9.$’)|LN; END Relational Efficiencies Translatable Aggregation

11. Aggregation is not translated to SQL and optimization is automatically disabled when:  Optimization was set off by user  JOINs were not passed to RDBMS (and consequently optimization was disabled by the interface)  FOCUS sort phrase is not translated  Some screening conditions not passed to RDBMS  Some non-SQL operators are used  Multi-verb requests  COUNT with MISSING=ON If the verbs PRINT or LIST are used, no aggregation is requested and FSTRACE3 returns the following message: (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED Relational Efficiencies Non-Translatable Aggregation NOTE

12.  All IF or WHERE statements in the request are translated to SQL WHERE clause.  FOCUS / WebFOCUS must generate a single SQL statement  Only these aggregation commands or operators are used: SUM, COUNT, WRITE, MIN., MAX., AVE., SUM., DST., CNT. or CNT.DST.  Aggregation can occur on DEFINE fields that are valued expressions ONLY, not logical expressions. Relational Efficiencies Aggregation: Key Points for Aggregation Translation

13. Relational Efficiencies Aggregation Management by the RDBMS

14. Relational Efficiencies Aggregation: SUM on ALPHANUMERIC Field SET TRACEOFF=ALL SET TRACEUSER=CLIENT SET TRACEON=STMTRACE//CLIENT SET TRACEON=SQLAGGR//CLIENT SET XRETRIEVAL=OFF ?FF COURSE ?FF SECTION -RUN JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 TABLE FILE COURSE SUM COST DESCRIPTION LOCATION BY SECTION_ID BY COURSE_NO END

15. Relational Efficiencies Aggregation: SUM on ALPHANUMERIC Field FILENAME= COURSE COURSE_NO COURSE_NO P9 DESCRIPTION DESCRIPTION A50 COST COST P11.2 PREREQUISITE PREREQUISITE P9 … FILENAME= SECTION SECTION_ID SECTION_ID P9 COURSE_NO COURSE_NO P9 SECTION_NO SECTION_NO P4 START_DATE_TIME START_DATE_TIME HYYMDS LOCATION LOCATION A50 INSTRUCTOR_ID INSTRUCTOR_ID P9 CAPACITY CAPACITY P4 … (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2591) AVE ON ALPHA, DATE OR DATETIME CANNOT BE AGGREGATED SELECT T1."COURSE_NO",T1."DESCRIPTION",T1."COST", T2."SECTION_ID",T2."LOCATION" FROM STUDENT.COURSE T1, STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") ORDER BY T2."SECTION_ID",T1."COURSE_NO";

16. Relational Efficiencies Aggregation: SUM on ALPHANUMERIC Field JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 TABLE FILE COURSE SUM COST DESCRIPTION MAX.LOCATION BY SECTION_ID BY COURSE_NO -*BY LOCATION END AGGREGATION DONE... SELECT T2."SECTION_ID",T1."COURSE_NO", SUM(T1."COST"), MAX(T1."DESCRIPTION"), MAX(T2."LOCATION") FROM STUDENT.COURSE T1,STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") GROUP BY T2."SECTION_ID",T1."COURSE_NO" ORDER BY T2."SECTION_ID",T1."COURSE_NO"; SELECT T2."SECTION_ID",T1."COURSE_NO",T2."LOCATION", SUM(T1."COST"), MAX(T1."DESCRIPTION") FROM STUDENT.COURSE T1, STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") GROUP BY T2."SECTION_ID",T1."COURSE_NO",T2."LOCATION" ORDER BY T2."SECTION_ID",T1."COURSE_NO",T2."LOCATION";

17. Relational Efficiencies Aggregation: IF/WHEREs Translated for Aggregation JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 TABLE FILE COURSE SUM COST BY SECTION_ID BY COURSE_NO BY DESCRIPTION WHERE COURSE_NO GT 150 WHERE SECTION_NO EQ EDIT('1') -*WHERE SECTION_NO EQ 1 WHERE DESCRIPTION LIKE 'Intro%' END (FOC2598) FOCUS IF/WHERE TEST CANNOT BE PASSED TO SQL : WHERE exp (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2596) ONE OR MORE EXPRESSION(S) CAN NOT BE TRANSLATED TO SQL SELECT T1."COURSE_NO",T1."DESCRIPTION",T1."COST",T2."SECTION_ID", T2."SECTION_NO" FROM STUDENT.COURSE T1,STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") AND (T1."DESCRIPTION" LIKE 'Intro%') AND (T1."COURSE_NO" > 150) ORDER BY T2."SECTION_ID",T1."COURSE_NO",T1."DESCRIPTION"; AGGREGATION DONE... SELECT T2."SECTION_ID",T1."COURSE_NO",T1."DESCRIPTION",SUM(T1."COST") FROM STUDENT.COURSE T1,STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") AND (T1."DESCRIPTION" LIKE 'Intro%') AND (T1."COURSE_NO" > 150) AND (T2."SECTION_NO" = 1) GROUP BY T2."SECTION_ID",T1."COURSE_NO",T1."DESCRIPTION" ORDER BY T2."SECTION_ID",T1."COURSE_NO",T1."DESCRIPTION";

18. Relational Efficiencies Aggregation: DEFINE Field JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 DEFINE FILE COURSE LOGICAL_NUM/I1=IF CAPACITY GT 20 THEN 1 ELSE 0; LOGICAL_CHR/A1=IF CAPACITY GT 20 THEN 'Y' ELSE 'N'; END TABLE FILE COURSE SUM CAPACITY LOGICAL_NUM -*LOGICAL_CHR BY COURSE_NO END (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2597) USE OF DEFINE FIELD THAT CANNOT BE AGGREGATED : LOGICAL SELECT T1."COURSE_NO",T2."CAPACITY" FROM STUDENT.COURSE T1, STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") ORDER BY T1."COURSE_NO";

19. Relational Efficiencies Aggregation: DEFINE Field JOIN COURSE_NO IN COURSE TO ALL COURSE_NO IN SECTION AS J1 -**DEFINE FILE COURSE -**LOGICAL_NUM/I1=IF CAPACITY GT 20 THEN 1 ELSE 0; -**LOGICAL_CHR/A1=IF CAPACITY GT 20 THEN 'Y' ELSE 'N'; -**END TABLE FILE COURSE SUM CAPACITY COMPUTE LOGICAL_NUM/I1=IF CAPACITY GT 20 THEN 1 ELSE 0; -*COMPUTE LOGICAL_CHR/A1=IF CAPACITY GT 20 THEN 'Y' ELSE 'N'; BY COURSE_NO END AGGREGATION DONE... SELECT T1."COURSE_NO", SUM(T2."CAPACITY") FROM STUDENT.COURSE T1,STUDENT.SECTION T2 WHERE (T2."COURSE_NO" = T1."COURSE_NO") GROUP BY T1."COURSE_NO" ORDER BY T1."COURSE_NO";

20. Copyright 2007, Information Builders. Slide 20 Selection Scenarios

21. Copyright 2007, Information Builders. Slide 21 Type of Expressions Expression Components Examples Arithmetic-Valued Expressions (Expressions that return a single number) Character String- Valued Expressions (Expressions that return a character string) Logical Expressions (Expressions that return a single value, True or False) Real fields of datatype I, P, D, or F Numeric constants Arithmetic operators (+, -, *, / ) Aggregation operators (SUM., CNT., AVE., MIN., MAX.) Real fields of datatype A String constants Concatenation operator (I) EDIT of alphanumeric fields Real fields with any FOCUS datatype Constants of consistent datatype Relational operator (EQ, NE, GT, LE..) Logical operators (AND, OR, NOT) Valued expression operands WHERE TOTAL (AVE.CSAL * 0.10) +AVE.CSAL GT 55000; WHERE EDIT(FN, ‘9.$’) |LN EQ ‘J.WANG’ ; WHERE (CDIV EQ ‘CORP’ OR ‘NE’) AND ((CSAL* 0.10) +CSAL GT 55000); Screening conditions on DEFINEd fields, which calculate the above type of expressions, are passed to the RDBMS. NOTE Relational Efficiencies Selection: Translatable Screening

22. Copyright 2007, Information Builders. Slide 22 User-written subroutines IF–THEN–ELSE expressions *** optimized Self-referential expressions EDIT for field format conversions Strong concatenation (II) DECODE function Non-SQL relational operators (INCLUDES, EXCLUDES) FOCUS subroutines (ABS, INT, MAX, MIN, LOG, SQRT) Expressions using fields with ACTUAL=DATE Expressions using Example DEFINE... FNL/I3 = ARGLEN(15,LN,FNL); TABLE... IF FNL LE 6 DEFINE... DIVISION/A11=IF CDIV EQ ‘CORP’ THEN ‘CORPORATE’ ELSE IF CDIV EQ ‘NE’ THEN ‘NORTH-EAST’ ELSE ‘NA’; TABLE... IF DIVISION EQ ‘CORPORATE’ OR ‘NORTH-EAST” DEFINE... CPT/I2=CPT+1; TABLE... IF CPT NE 0 WHERE EDIT(ID) GT 20 DEFINE... NAME/A27=FN||(‘ ‘ | LN); TABLE... IF NAME EQ ‘DANIEL VALINO’ DEFINE... DEVISION/A11=DECODE CDIV (‘CORP’ ‘CORPORATE’ ‘NE’ ‘NORTH-EAST’ ELSE ‘NA’); TABLE... IF DIVISION EQ ‘CORPORATE’ OR ‘NORTH-EAST’ IF LN INCLUDES ‘VALINO’ DEFINE... HDAT2/YYMD=HDAT+365; TABLE... IF HDAT2 GT ‘1990/03/01’ WHERE SQRT(CSAL) GT 260 Relational Efficiencies Non-Translatable Screening Conditions

23. Relational Efficiencies Selection: Variable with Subroutine Reference -DEFAULT &LOCVAL='l509'; DEFINE FILE SECTION LOCTEST/A4=UPCASE(4,'&LOCVAL','A4'); END TABLE FILE SECTION PRINT LOCATION BY SECTION_ID BY SECTION_NO WHERE LOCATION EQ UPCASE(4,'&LOCVAL','A4') -*WHERE LOCATION EQ LOCTEST END (FOC2598) FOCUS IF/WHERE TEST CANNOT BE PASSED TO SQL : WHERE exp (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2596) ONE OR MORE EXPRESSION(S) CAN NOT BE TRANSLATED TO SQL SELECT T1."SECTION_ID",T1."SECTION_NO",T1."LOCATION" FROM STUDENT.SECTION T1 ORDER BY T1."SECTION_ID",T1."SECTION_NO";

24. Relational Efficiencies Selection: Variable with Subroutine Reference -DEFAULT &LOCVAL='l509'; -SET &UPLOCVAL=UPCASE(4,'&LOCVAL.EVAL','A4'); DEFINE FILE SECTION LOCTEST/A4 WITH SECTION_ID='&UPLOCVAL'; END TABLE FILE SECTION PRINT LOCATION BY SECTION_ID BY SECTION_NO WHERE LOCATION EQ '&UPLOCVAL' -*WHERE LOCATION EQ LOCTEST END (FOC2590) AGGREGATION NOT DONE FOR THE FOLLOWING REASON: (FOC2594) AGGREGATION IS NOT APPLICABLE TO THE VERB USED SELECT T1."SECTION_ID",T1."SECTION_NO",T1."LOCATION" FROM STUDENT.SECTION T1 WHERE (T1."LOCATION" = 'L509') ORDER BY T1."SECTION_ID",T1."SECTION_NO";

25. Copyright 2007, Information Builders. Slide 25 Relational Efficiencies Selection: DATEs DEFINE FILE ORDERS CONTROL/A1 WITH STORENAME='X'; Order_Year/YY=HPART(ORDER_DATE, 'YEAR', 'YY'); … END TABLE FILE ORDERS … WHERE Order_Year GE 2003 AND Order_Year LT 2004 FILENAME=ORDERS1, SUFFIX=SQLMAC, $ SEGMENT=ORDERS1, SEGTYPE=S0, $ … FIELDNAME=ORDER_DATE, ALIAS=ORDER_DATE, USAGE=HYYMDS, ACTUAL=HYYMDS, MISSING=ON, $, $ (FOC2598) FOCUS IF/WHERE TEST CANNOT BE PASSED TO SQL : Order_Year

26. Copyright 2007, Information Builders. Slide 26 Relational Efficiencies Selection: DATEs TABLE FILE ORDERS1 … WHERE Order_Year GE 2003 AND Order_Year LT 2004 FILENAME=ORDERS1, SUFFIX=SQLMAC, $ SEGMENT=ORDERS1, SEGTYPE=S0, $ … FIELDNAME=ORDER_DATE, ALIAS=ORDER_DATE, USAGE=HYYMDS, ACTUAL=HYYMDS, MISSING=ON, $ FIELDNAME=Order_Year, ALIAS=ORDER_DATE, USAGE=YY, ACTUAL=DATE, MISSING=ON, $ WHERE (T1.`ORDER_DATE` = {d '2003-01-01'})

27. Copyright 2007, Information Builders. Slide 27