1. The Power of the BY Statement SVSUG 2009.06.25 Paul Choate, California Developmental Services (& Toby Dunn, U.S. Army Medical Department Center & School)

2. BY Statement Syntax and Usage The BY statement is used in SAS to instruct the DATA step or procedures to process dataset observations in groups, rather than singly. It can be used whenever SAS data is ordered, or can be accessed in order through a SAS dataset index. In the DATA step this allows observations to be summarized or reorganized according to a group structure. In PROC steps it allows SAS to process and present data in groups. The basic syntax of the BY statement is the same throughout SAS, with the exception that the GROUPFORMAT option is only available in the DATA step. BY var1 varn> ;

3. BY Statement Syntax and Usage BY Sex Age Name; NAME SEX AGE HEIGHT WEIGHT Alice F 13 56.5 84 Barbara F 13 65.3 98 Carol F 14 62.8 102.5 Judy F 14 64.3 90 Jeffrey M 13 62.5 84 Alfred M 14 69 112.5 Ronald M 15 67 133 Philip M 16 72 150

4. BY Statement Syntax and Usage Sort order is platform dependent and is based on the internal ordering of the platform character set, called the collating sequence. ASCII (PC) character set order:..., 1, 2, 3,... A, B, C,... a, b, c......, 1, 2, 3,... A, B, C,... a, b, c... EBCDIC (MVS) character set order:..., a, b, c,... A, B, C,... 1, 2, 3,......, a, b, c,... A, B, C,... 1, 2, 3,...

5. BY Statement Syntax and Usage BY Sex DESCENDING Age Name; NAME SEX AGE HEIGHT WEIGHT Janet F 15 62.5 112.5 Carol F 14 62.8 102.5 Alice F 13 56.5 84 Barbara F 13 65.3 98 Philip M 16 72 150 Alfred M 14 69 112.5 Henry M 14 63.5 102.5

6. BY Statement Syntax and Usage BY Age NOTSORTED; NAME AGE HEIGHT WEIGHT Carol 14 62.8 102.5 Judy 14 64.3 90 Janet 15 62.5 112.5 Ronald 15 67 133 Mary 15 66.5 112 Alice 13 56.5 84 Jeffrey 13 62.5 84

7. BY Statement Syntax and Usage PROC FORMAT; VALUE $Initials 'A'-<'B'='A' VALUE $Initials 'A'-<'B'='A' 'B'-<'C'='B' 'B'-<'C'='B'......RUN; DATA Class; DATA Class; SET Class; SET Class; FORMAT Name $Initials.; FORMAT Name $Initials.; BY Name GROUPFORMAT; BY Name GROUPFORMAT;RUN;

8. BY Statement Syntax and Usage GROUPFORMAT (cont.) NAME AGE HEIGHT WEIGHT Alice 13 56.5 84 Alfred 14 69 112.5 Judy 14 64.3 90 Janet 15 62.5 112.5 Jeffrey 13 62.5 84 William 15 66.5 112

9. Introduction to Data Structure Variables may be divided into two classes: primary key variables, whose values may be combined uniquely to identify one observation or event, and primary key variables, whose values may be combined uniquely to identify one observation or event, and non-primary keys, whose values cannot be combined to uniquely identify an observation. non-primary keys, whose values cannot be combined to uniquely identify an observation. The primary and non-primary keys are all related to each other in some fashion known as functional dependencies. Primary keys must be unique or form unique combinations called composite keys.

10. Introduction to Data Structure The most fundamental rule is that no two rows shall have the same unique values for all primary key variables. VEHICLETYPE MODEL MAKE YEAR COLOR Truck 1500 Chevy 2008 Blue should be reduced to: VEHICLETYPE MODEL MAKE YEAR COLOR COUNT Truck 1500 Chevy 2008 Blue 2

11. Introduction to Data Structure Each variable in the dataset should have atomic values. VEHICLE MODEL YEAR COLOR NUMSOLD PACKAGE Truck 1500 2008 Blue 2 Sports, Standard Truck 1500 2008 Gold 3 Sports, Sports, Standard should be restructured as: VEHICLE MODEL YEAR COLOR NUMSOLD PACKAGE Truck 1500 2008 Blue 1 Sports Truck 1500 2008 Blue 1 Standard Truck 1500 2008 Gold 2 Sports Truck 1500 2008 Gold 1 Standard This is called First Normal Form with Redundancies.

12. BY Statement in the Data Step The BY statement provides two automatic temporary variables for each BY variable: FIRST.variable and LAST.variable. They indicate whether an observation is: the first in a BY group the first in a BY group the last in a BY group the last in a BY group neither the first nor the last in a BY group neither the first nor the last in a BY group both first and last, as is the case when there is only one observation in a BY group. both first and last, as is the case when there is only one observation in a BY group.

13. BY Statement in the Data Step SEXAGEFIRST.SEXLAST.SEXFIRST.AGELAST.AGE F131010 F130001 F150010 F150101 M131011 M140010 M140001 M160111 BY Sex Age;

14. BY Statement in the Data Step SEXAGEFIRST.SEXLAST.SEXFIRST.AGELAST.AGE F131011 F150111 M131011 M140011 M160111 Sorted variables with unique values have all FIRST.variable and LAST.variables set to 1. Here Age is unique within Sex: BY Sex Age;

15. BY Statement in the Data Step Examples: Unduplication example Unduplication example Counting records example Counting records example

16. Combining Datasets In the DATA step the BY statement is used for combining data with: Interleaving with the SET statement Interleaving with the SET statement Match-merging with the MERGE statement Match-merging with the MERGE statement Updating with the UPDATE statement Updating with the UPDATE statementand Modifying (beyond scope of presentation) Modifying (beyond scope of presentation)

17. Interleaving Datasets When a BY statement is used with a SET statement that specifies two or more datasets, the DATA step reads the files simultaneously, alternating between the files based on the BY variable order. This maintains the sort order of the data from the datasets as they are processed. For example, suppose there are two datasets, one for males and one for females, and both are sorted on Age. They can be interleaved into a single dataset sorted on Age and Gender.

18. Interleaving Datasets Example: SET statement interleaving example SET statement interleaving example

19. Interleaving Datasets With interleaving, the sum of a variable for each by-group may be attached back to the original non-aggregated dataset. This requires at least two passes of the data, but the efficiency and complexity may vary considerably based on the approach. This requires at least two passes of the data, but the efficiency and complexity may vary considerably based on the approach.

20. Interleaving Datasets Example: Howard Shreier look-ahead processing Howard Shreier look-ahead processing

21. Wookie One-Liners You?! It was your idea for Jar Jar?! And Lando never suggested a flea bath again. "I just need one head to finish my C3PO" Allright, allright. I promise: No more Colt-45 commercials! What do you mean,"We're OUT of shampoo??!!!!"

22. Match-Merging Datasets When a BY statement is used with a MERGE statement, the SAS datasets are read simultaneously, merging observations based on matching BY variables. When merging multiple datasets, usually at least all but one of the datasets should be unique on the BY variables. The combined unique observations are merged with each matching observation in the non-unique dataset. The unique observations are duplicated across the non-unique observations.

23. Match-Merging Datasets Example: MERGE statement MERGE statement

24. Updating Datasets The UPDATE statement only allows two datasets, a master dataset and a transaction dataset. The master dataset is specified first and the transaction dataset second, followed by a BY statement. As with MERGE, the two datasets are read simultaneously, updating observations from the master dataset with observations from the transaction dataset based on the lowest level groupings of the BY variables. When a transaction variable has a missing value, by default UPDATE does not overwrite the value in the master dataset, whereas the MERGE statement does.

25. Updating Datasets Examples: Updating prices in an inventory Updating prices in an inventory Flattening a dataset Flattening a dataset

26. Do-Loop of Whitlock (DoW) The SET statement may be wrapped inside a DO UNTIL loop with the BY statement controlling the loop. DATA...; ; ; DO UNTIL ; DO UNTIL ; SET...; SET...; By...; By...; ; ; END; END; ; ;RUN;

27. Do-Loop of Whitlock (DoW) The DoW works with the natural execution of the DATA step by isolating what happens between two consecutive break events. Statements and functions are placed within the loop, and the implicit action of the DATA step resets calculated values to missing after each BY group. In our example the break events are BY groups, but in other cases could be anything that triggers the DO loop to stop.

28. Do-Loop of Whitlock (DoW) Examples: Standard DATA step Standard DATA step Whitlock/Dorfman DoW Whitlock/Dorfman DoW Sequential DoWs Sequential DoWs

29. The BY Statement in SAS Procedures Nearly all SAS PROCs that process datasets allow for the BY statement. The syntax is the same as in the DATA step, except for the GROUPFORMAT option which is only available to the DATA step. Procedures that produce printed output, such as PROC PRINT, format printed output into BY groups. Procedures that produce printed output, such as PROC PRINT, format printed output into BY groups. Procedures that summarize datasets, like PROC FREQ or PROC SUMMARY process the data in groups, sometimes as an alternative to other statements such as TABLES or CLASS. Procedures that summarize datasets, like PROC FREQ or PROC SUMMARY process the data in groups, sometimes as an alternative to other statements such as TABLES or CLASS.

30. The PRINT Procedure PROC PRINT writes dataset values in columnar table form with the variable names or labels at the top of each column. The BY statement, and the related PAGEBY and SUMBY statements can be used with PROC PRINT.

31. The PRINT Procedure Examples: BY statement BY statement BY statement with ID statement BY statement with ID statement PAGEBY statement PAGEBY statement SUMBY statement SUMBY statement

32. The FREQ Procedure The FREQ procedure calculates frequencies and statistics on discrete variables. These can be printed or output. Levels of a tabulation are requested with a TABLES statement, or for sorted variables with a BY statement. PROC FREQ does not show rows or columns for missing categories of a variable in a BY group, but in the TABLE statement the row or column is zero filled. The BY and TABLE statements produce different statistics for tabulation levels with missing categories.

33. The FREQ Procedure Examples: PROC FREQ with the TABLES statement PROC FREQ with the TABLES statement PROC FREQ with the BY statement PROC FREQ with the BY statement

34. The SUMMARY or MEANS Procedure In PROC SUMMARY and PROC MEANS the BY statement is an alternate to the CLASS statement. All permutations of levels of CLASS variables are summarized. For three class variables A, B, and C, statistics are calculated for the overall data and all levels of A, B, C, A*B, A*C, B*C, and A*B*C. Sorted variables may be alternatively specified in a BY statement, but only permutations including that variable will be summarized. For example, if A is specified in the BY statement rather than the CLASS statement, then only statistics for A, A*B, A*C, and A*B*C are produced.

35. The SUMMARY or MEANS Procedure Examples: PROC SUMMARY with the CLASS statement PROC SUMMARY with the CLASS statement PROC SUMMARY with the BY statement PROC SUMMARY with the BY statement

36. The SQL Procedure PROC SQL performs actions both similar to the DATA step and summarizing procedures such as SUMMARY, TABULATE, and UNIVARIATE. PROC SQL has unique syntax conforming to the SQL programming language. The BY statement in PROC SQL is replaced by the GROUP BY statement. In PROC SQL if data are not sorted then the procedure will sort the data internally as needed by the GROUP BY statement.

37. The SQL Procedure Example: Aggregating grouped data with PROC SQL GROUP BY statement Aggregating grouped data with PROC SQL GROUP BY statement

38. Thanks to SVSUG Chair Andrew Karp

39. Contact Information Your comments and questions are valued and encouraged. Paul Choate, California Developmental Services Phone: (916) 654-2160 E-mail: Paul.Choate@dds.ca.gov