1. Timothy Forsyth Ashok Viswanathan Debbie McCullough Donn Garvert

2.  Some operations are more convenient when there is one observation per subject  Example: Regression analysis  Some operations are more convenient when there are several observations per subject  Example: Plot individual observations as a time series Participant #206 0.0 0.5 1.0 1.5 2.0 2.5 3.0 12345678910111213141516 Day Since Injury Ave. Positive Mood Ave. Negative Mood

3. First method: Using arrays – Can use arrays to transpose the data set – Gives user more control Original data:

4. Transposing the entire data set

5. Transposing the entire data set back to original form

6. Key commands: RETAIN: since these variables are not in the dataset rainfall1 we must retain these values as SAS will set to missing if we do not CALL MISSING: this will set any number of numeric values or character values to missing all at once In our example, if there was a missing data point somewhere in between month 1 and month 5 for each subject the CALL MISSING command will set these values to missing all at once. We use the OF command so that SAS looks at all 5 values.

7. Without adding options to PROC TRANSPOSE we will get a confusing output, but the data will be correct:

8. When we add options to PROC TRANSPOSE we can get a cleaner looking data set Renamed _name_ to “month” Renamed col1 to “rainfall” Dropped subjects with missing data

9. Transposing the rainfall data back to original form

10.  PREFIX= option: this was not used in the example, but can be used as an option in this conversion  Useful when your id variable is a number  For example if month was listed as 1, 2, 3, 4, 5 we could use PREFIX=Month ▪This will result in Month 1, Month 2, …, Month 5

11. Data Sets Often have two or more observations per subject (or other groupings)  Patients who have repeated visits to the Doctor’s office or clinic  Sales at a store on a given day.  Inches of rainfall over many months in a given city (Current Example)  This is known as Longitudinal Data SAS processes data one observation at a time so special techniques are needed to perform calculations across observations I will cover : 1)Identifying the first and last observation in a group. 2)A few different ways to count the number of occurrences of a subject (or other grouping.). In our example we will count, for a given city, the number of months that had their average rainfall recorded.

12. Listing of SAS Data Set Rainfall Ave Average City Rainfall Temp Sunlight Alameda 5.8 53.6 8.9 Fremont 5.2 52.9 8.7 Fremont 4.5 57.6 9.5 Fremont 3.3 59.7 10.1 Fremont 2.2 60.6 11.0 Fremont 1.8 63.9 11.5 Hayward 1.4 60.4 9.3 Hayward 3.6 57.8 10.1 Hayward 2.0 62.5 10.2 Hayward 4.1 55.2 10.8 Hayward 5.7 53.6 11.2 Oakland 3.1 59.2 9.5 Oakland 3.4 60.1 10.1 Sunnyvale 5.7 54.2 9.4 Sunnyvale 3.4 56.8 9.8 Sunnyvale 4.9 55.2 10.3 Sunnyvale 5.0 52.9 10.7 Three Step process: Step 1) Sort the data first by the grouping variable (City) and then by the counting variable(Rainfall). Step 2) Create First and Last Variables. Step 3) Counting the months of recorded rainfall using either a data step or proc sql.

13. city Rainfall Alameda 5.8 Fremont 5.2 Fremont 4.5 Fremont 3.3 Fremont 1.8 Hayward 1.4 Hayward 3.6 Hayward 2.0 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 5.7 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 proc sort data=rainfall5; by city rainfall; run; proc print data=rainfall5; var city rainfall; run; city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 Dataset Rainfall has been sorted first by city and second by rainfall

14. city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 data rainfall_last rainfall_first; set rainfall5; by city; if last.city then output rainfall_last; else if first.city then output rainfall_first; run; proc print data=rainfall_first; run; Listing of First_City city Rainfall Fremont 1.8 Hayward 1.4 Oakland 3.1 Sunnyvale 3.4 Using the set dataset; by var(city); creates two temporary SAS variables, first.var and last.var. These two are logical variables; They equal 1 if true and 0 if false. In our case we have generated variables first.city and last.city.

15. city=Alameda Rainfall=5.8 FIRST.city=1 LAST.city=1 city=Fremont Rainfall=1.8 FIRST.city=1 LAST.city=0 city=Fremont Rainfall=2.2 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=3.3 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=4.5 FIRST.city=0 LAST.city=0 city=Fremont Rainfall=5.2 FIRST.city=0 LAST.city=1 city=Hayward Rainfall=1.4 FIRST.city=1 LAST.city=0 city=Hayward Rainfall=2 FIRST.city=0 LAST.city=0 city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 Observation 1 for Alameda is both the first and the last variable so first.city =1(true )and Last.city=1(true). Obs 1 for Fremont first.city = 1 last.city =0, etc., etc. data rainfall_last rainfall_first; set rainfall5; by city; put city= rainfall= first.city= last.city=; if last.city then output rainfall_last; else if first.city then output rainfall_first; run; Contents from the log

16. Listing of Counts # of Months City Rainfall Recorded Alameda 5.8 1 Fremont 5.2 5 Hayward 5.7 5 Oakland 3.4 2 Sunnyvale 5.7 4 city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 data months_of_rec_rainfall; set rainfall5; by city; if first.city then N_months_rec = 0; N_months_rec +1; if last.city then output; run; title 'Listing of Counts'; proc print data=months_of_rec_rainfall label noobs; label N_months_rec = '# of Months Recorded'; var City Rainfall N_months_rec;; run; Prediction(actual) ; Alameda 1(1), Fremont 5(5), Hayward 5(5) Oakland 2(2), Sunnyvale 4(4).

17. data months_of_rec_rainfall; set rainfall5; by city; if first.city then N_months_rec= 0; 1) N_months_rec +1; 2) if last.city then output; 3) run; title 'Listing of Counts'; proc print data=months_of_rec_rainfall label noobs; label N_months_rec = '# of Months Recorded'; var City Rainfall N_months_rec;; run; 1) Initialize the counter at zero 2) Sum Statement 3) Conditional Statement

18. city Rainfall Alameda 5.8 Fremont 1.8 Fremont 2.2 Fremont 3.3 Fremont 4.5 Fremont 5.2 Hayward 1.4 Hayward 2.0 Hayward 3.6 Hayward 4.1 Hayward 5.7 Oakland 3.1 Oakland 3.4 Sunnyvale 3.4 Sunnyvale 4.9 Sunnyvale 5.0 Sunnyvale 5.7 months_rec_ city rain Alameda 1 Fremont 5 Hayward 5 Oakland 2 Sunnyvale 4 proc sql; create table Months_Rec_Rain as select city, count(city) as months_rec_rain from rainfall5 group by city; quit; The proc sql gives the same result as the data step.

19. It was shown, via a three step process, how to create a variable to count the number of occurrences of a grouping variable. The example shown dealt with the number of months of recorded rainfall in a given city. These techniques have utility in the medical or clinical setting.

20. SAS CODE /*eliminate missing values*/ /*create longitudinal data*/ data rainfall1; set rainfall; array rainfall_array{5} month_1-month_5; array temp_array{5} temp_1-temp_5; array hours_array{5} hours_1-hours_5; do month = 1 to 5; if missing(rainfall_array{month}) then leave; Rain = rainfall_array{month}; AveTemp = temp_array{month}; AverageSunlight = hours_array{month}; Output; end; keep city rain month AveTem AverageSunlight; run; proc print data=rainfall1; run; SAS OUTPUT Output: Average Obs city month Rain AveTemp Sunlight 1 Hayward 1 1.4 60.4 9.3 2 Hayward 2 3.6 57.8 10.1 3 Hayward 3 2.0 62.5 10.2 4 Hayward 4 4.1 55.2 10.8 5 Hayward 5 5.7 53.6 11.2 6 Oakland 1 3.1 59.2 9.5 7 Oakland 2 3.4 60.1 10.1 8 Alameda 1 5.8 53.6 8.9 9 Sunnyval 1 5.7 54.2 9.4 10 Sunnyval 2 3.4 56.8 9.8 11 Sunnyval 3 4.9 55.2 10.3 12 Sunnyval 4 5.0 52.9 10.7 13 Fremont 1 5.2 52.9 8.7 14 Fremont 2 4.5 57.6 9.5 15 Fremont 3 3.3 59.7 10.1 16 Fremont 4 2.2 60.6 11.0 17 Fremont 5 1.8 63.9 11.5

21. A FEW POINTS TO NOTE:  A proc freq approach can be used in addition to a proc means approach  We present here the proc means approach  Both ways can give us the same result/output CODE proc means data=rainfall1 nway noprint; class city; output out=counts (rename=(_freq_ = N_Recorded) drop = _type_); run; proc print data=counts; run; Things to notice:  nway  nprint  rename

22. The SAS System 14:46 Sunday, May 30, 2010 13 Average Obs city N_Recorded _STAT_ month Rain AveTemp Sunlight 1 Alameda 1 N 1.00000 1.00000 1.0000 1.0000 2 Alameda 1 MIN 1.00000 5.80000 53.6000 8.9000 3 Alameda 1 MAX 1.00000 5.80000 53.6000 8.9000 4 Alameda 1 MEAN 1.00000 5.80000 53.6000 8.9000 5 Alameda 1 STD.... 6 Fremont 5 N 5.00000 5.00000 5.0000 5.0000 7 Fremont 5 MIN 1.00000 1.80000 52.9000 8.7000 8 Fremont 5 MAX 5.00000 5.20000 63.9000 11.5000 9 Fremont 5 MEAN 3.00000 3.40000 58.9400 10.1600 10 Fremont 5 STD 1.58114 1.45430 4.0685 1.1261 11 Hayward 5 N 5.00000 5.00000 5.0000 5.0000 12 Hayward 5 MIN 1.00000 1.40000 53.6000 9.3000 13 Hayward 5 MAX 5.00000 5.70000 62.5000 11.2000 14 Hayward 5 MEAN 3.00000 3.36000 57.9000 10.3200 15 Hayward 5 STD 1.58114 1.71552 3.6469 0.7259 16 Oakland 2 N 2.00000 2.00000 2.0000 2.0000 17 Oakland 2 MIN 1.00000 3.10000 59.2000 9.5000 18 Oakland 2 MAX 2.00000 3.40000 60.1000 10.1000 19 Oakland 2 MEAN 1.50000 3.25000 59.6500 9.8000 20 Oakland 2 STD 0.70711 0.21213 0.6364 0.4243 21 Sunnyval 4 N 4.00000 4.00000 4.0000 4.0000 22 Sunnyval 4 MIN 1.00000 3.40000 52.9000 9.4000 23 Sunnyval 4 MAX 4.00000 5.70000 56.8000 10.7000 24 Sunnyval 4 MEAN 2.50000 4.75000 54.7750 10.0500 25 Sunnyval 4 STD 1.29099 0.96782 1.6460 0.5686

23.  We would like to see the differences in values by city and by month for the following variables:  The average rainfall recorded (given by ‘rain’)  The average temperature recorded (given by ‘avetemp’)  The average sunlight recorded (given by ‘avesunlight’)

24. proc sort data=rainfall1 out=rainfall1; by city month; run; data last; set rainfall1; by city; put city=month=first.city = last.city=; if last.city; run;

25. log file: NOTE: There were 17 observations read from the data set WORK.RAINFALL1. NOTE: The data set WORK.RAINFALL1 has 17 observations and 5 variables. NOTE: PROCEDURE SORT used (Total process time): real time 0.01 seconds cpu time 0.00 seconds 61 62 data last; 63 set rainfall1; 64 by city; 65 put city=month=first.city = last.city=; 66 if last.city; 67 run; city=Alameda month=1 FIRST.city=1 LAST.city=1 city=Fremont month=1 FIRST.city=1 LAST.city=0 city=Fremont month=2 FIRST.city=0 LAST.city=0 city=Fremont month=3 FIRST.city=0 LAST.city=0 city=Fremont month=4 FIRST.city=0 LAST.city=0 city=Fremont month=5 FIRST.city=0 LAST.city=1 city=Hayward month=1 FIRST.city=1 LAST.city=0 city=Hayward month=2 FIRST.city=0 LAST.city=0 city=Hayward month=3 FIRST.city=0 LAST.city=0 city=Hayward month=4 FIRST.city=0 LAST.city=0 city=Hayward month=5 FIRST.city=0 LAST.city=1 city=Oakland month=1 FIRST.city=1 LAST.city=0 city=Oakland month=2 FIRST.city=0 LAST.city=1 city=Sunnyval month=1 FIRST.city=1 LAST.city=0 city=Sunnyval month=2 FIRST.city=0 LAST.city=0 city=Sunnyval month=3 FIRST.city=0 LAST.city=0 city=Sunnyval month=4 FIRST.city=0 LAST.city=1 NOTE: There were 17 observations read from the data set WORK.RAINFALL1. NOTE: The data set WORK.LAST has 5 observations and 5 variables. NOTE: DATA statement used (Total process time): real time 0.01 seconds cpu time 0.01 seconds

26. data difference; set rainfall1; by city; if first.city and last.city then delete; Diff_Rain = rain - lag(rain); Diff_AveTemp = AveTemp - lag(AveTemp); Diff_AverageSunlight = AverageSunlight - lag(AverageSunlight); if not first.city then output; run; proc print data=difference; run;

27. The SAS System 14:46 Sunday, May 30, 2010 5 Ave Average Diff_ Diff_ Diff_ Average  Obs city month Rain Temp Sunlight Rain AveTemp Sunlight  1 Fremont 2 4.5 57.6 9.5 -0.7 4.7 0.8  2 Fremont 3 3.3 59.7 10.1 -1.2 2.1 0.6  3 Fremont 4 2.2 60.6 11.0 -1.1 0.9 0.9  4 Fremont 5 1.8 63.9 11.5 -0.4 3.3 0.5  5 Hayward 2 3.6 57.8 10.1 2.2 -2.6 0.8  6 Hayward 3 2.0 62.5 10.2 -1.6 4.7 0.1  7 Hayward 4 4.1 55.2 10.8 2.1 -7.3 0.6  8 Hayward 5 5.7 53.6 11.2 1.6 -1.6 0.4  9 Oakland 2 3.4 60.1 10.1 0.3 0.9 0.6  10 Sunnyval 2 3.4 56.8 9.8 -2.3 2.6 0.4  11 Sunnyval 3 4.9 55.2 10.3 1.5 -1.6 0.5  12 Sunnyval 4 5.0 52.9 10.7 0.1 -2.3 0.4

28. data first_last; set rainfall1; by city; if first.city and last.city then delete; if first.city or last.city then do; diff_rain = rain - lag(rain); diff_temp = avetemp - lag(avetemp); diff_sunlight = averagesunlight - lag(averagesunlight); end; if last.city then output; run; proc print data=first_last; run;

29. Ave Average Average diff_ diff_ diff_ Obs city month Rain Temp Sunlight rain temp Sunlight 1 Fremont 5 1.8 63.9 11.5 -3.4 11.0 2.8 2 Hayward 5 5.7 53.6 11.2 4.3 -6.8 1.9 3 Oakland 2 3.4 60.1 10.1 0.3 0.9 0.6 4 Sunnyval 4 5.0 52.9 10.7 -0.7 -1.3 1.3

30. Using RETAIN statement is one of the best ways to “remember” values from previous observations Variables that do not come from SAS data sets are set to a missing value during each iteration of the DATA step A RETAIN statement allows you to tell SAS not to do this

31. data new_data_set; set old_data_set; by ID; if first.ID and last.ID then delete; Retain First_Var_1 First_Var_2 … First_Var_3; If first.ID then do; First_Var_1 = Var_1 First_Var_2 = Var_2 … First_Var_n = Var_n If last.ID then do; Diff_Var_1 = Var_1 - First_Var_1; Diff_Var_2 = Var_2 - First_Var_2; … Diff_Var_n = Var_n - First_Var_n; End; Drop Frist_: ; Run; Need to sort by the ID, or which ever variable you are grouping by! The RETAIN statement ensures these variables are not set back to missing values during iteration. The Variables named within the RETAIN Statement are not replaced with missing values during the iterations.

32.  The RETAIN statement ensures your variables are not set back to missing values.  When processing first observation, the retained variables are set to respective variable values.  The last iteration subtracts the retained first values from the respective last variable values.

33. proc sort data=rainfall; by City Month; run; data last; set rainfall; by City; put City= Month= First.City= Last.City=; if last.City; run; data first_last; set rainfall; by City; if first.city and last.city then delete; retain First_Rainfall First_Temp First_Hours; if first.City then do; First_Rainfall = Rainfall; First_Temp = Temp; First_Hours = Hours; end; if last.City then do; Diff_Rainfall = Rainfall - First_Rainfall; Diff_Temp = Temp - First_Temp; Diff_Hours = Hours - First_Hours; output; end; drop First_: ; run;

34. Displaying the RETAIN Statement City Month Rainfall Temp Hours Diff_ Rainfall Diff_ Temp Diff_Hours Fremont 5 1.8 63.9 11.5 -3.4 11.0 2.8 Hayward 5 5.7 53.6 11.2 4.3 -6.8 1.9 Oakland 5 3.6 60.3 10.8 0.5 1.1 1.3 Sunnyvale 4 5.0 52.9 10.7 -0.7 -1.3 1.3

35. Output from LAG statement: city month Rain Temp Sunlight diff_rain diff_temp diff_Sunlight Fremont 5 1.8 63.9 11.5 -3.4 11.0 2.8 Hayward 5 5.7 53.6 11.2 4.3 -6.8 1.9 Oakland 2 3.4 60.1 10.1 0.3 0.9 0.6 Sunnyvale 4 5.0 52.9 10.7 -0.7 -1.3 1.3 Output from RETAIN statement City Month Rainfall Temp Hours Diff_ Rainfall Diff_ Temp Diff_Hours Fremont 5 1.8 63.9 11.5 -3.4 11.0 2.8 Hayward 5 5.7 53.6 11.2 4.3 -6.8 1.9 Oakland 5 3.6 60.3 10.8 0.5 1.1 1.3 Sunnyvale 4 5.0 52.9 10.7 -0.7 -1.3 1.3

36.  Suppose you want to know if a certain variable value is the maximum of all of you observations  The RETAIN Statement allows us to easily find this while preserving a variable’s value from previous iteration

37. data Maximums; Set rainfall; Retain Max_Rainfall Max_Temp Max_Hours; Max_Rainfall = Max(Max_Rainfall, Rainfall); Max_Temp = Max(Max_Temp, Temp); Max_Hours = Max(Max_Hours, Hours); run; title "Displaying the RETAIN Statement- Finding Maximums“; proc print data=Maximums noobs; run;

38. Displaying the RETAIN Statement- Finding Maximums city Month Rainfall Temp Hours Max_Rainfall Max_Temp Max_Hours Alameda 1 5.8 53.6 8.9 5.8 53.6 8.9 Fremont 1 5.2 52.9 8.7 5.8 53.6 8.9 Fremont 2 4.5 57.6 9.5 5.8 57.6 9.5 Fremont 3 3.3 59.7 10.1 5.8 59.7 10.1 Fremont 4 2.2 60.6 11.0 5.8 60.6 11.0 Fremont 5 1.8 63.9 11.5 5.8 63.9 11.5 Hayward 1 1.4 60.4 9.3 5.8 63.9 11.5 Hayward 2 3.6 57.8 10.1 5.8 63.9 11.5 Hayward 3 2.0 62.5 10.2 5.8 63.9 11.5 Hayward 4 4.1 55.2 10.8 5.8 63.9 11.5 Hayward 5 5.7 53.6 11.2 5.8 63.9 11.5 Oakland 1 3.1 59.2 9.5 5.8 63.9 11.5 Oakland 2 3.4 60.1 10.1 5.8 63.9 11.5 Oakland 5 3.6 60.3 10.8 5.8 63.9 11.5 Sunnyvale 1 5.7 54.2 9.4 5.8 63.9 11.5 Sunnyvale 2 3.4 56.8 9.8 5.8 63.9 11.5 Sunnyvale 3 4.9 55.2 10.3 5.8 63.9 11.5 Sunnyvale 4 5.0 52.9 10.7 5.8 63.9 11.5