Presentation on theme: "Welcome to CSC323 Data Analysis and Statistical Software I Winter 2004 Instructor: Raffaella Settimi Office: Loop campus: CST 716 - Tel. (312) 362 – 5556."— Presentation transcript:
Welcome to CSC323 Data Analysis and Statistical Software I Winter 2004 Instructor: Raffaella Settimi Office: Loop campus: CST 716 - Tel. (312) 362 – 5556 O'Hare campus: Email: firstname.lastname@example.org@cs.depaul.edu Contact hours: Monday 4:00 - 5:30pm O’Hare campus Tuesday 3:30 – 5:00 pm in the Loop campus
Course web page: http://facweb.cs.depaul.edu/rsettimi/323 http://facweb.cs.depaul.edu/rsettimi/323 Check the web page regularly for news and announcements. Course documents and homework assignments will be posted on the course homepage. Lectures can be seen online on the DL website: http://dlweb.cs.depaul.edu The lectures will be available in the morning following the class.
Course topics The course will discuss simple statistical methods and basic concepts of probability theory. The topics of the course are 1.descriptive statistics and representing data using graphs. 2.Linear regression models. 3.Sampling and experimental design. 4.An introduction to statistical inference 1.confidence intervals and 2.hypothesis testing. We will use the statistical package SAS The statistical software SAS runs on UNIX (accounts on Hawk are available to students) and on PC's (available in the computer labs) Check the course web page at http://facweb.cs.depaul.edu/rsettimi/323/sasinstructions.htm http://facweb.cs.depaul.edu/rsettimi/323/sasinstructions.htm for more information on SAS availability.
Required Texts: Introduction to the Practice of Statistics, Fourth Edition, by D.S. Moore and G.P. McCabe (2003). ISBN:0-7167-9657-0 Recommended SAS manual SAS Manual for Moore and McCabe's Introduction to the Practice of Statistics. Michael Evans, Freeman. Third edition, 1999. ISBN: 0-7167-3657-8 The course syllabus provides more detailed course information. The syllabus is posted on http://facweb.cs.depaul.edu/rsettimi/323/csc_323.htmhttp://facweb.cs.depaul.edu/rsettimi/323/csc_323.htm
Grading Homework and Programming assignments (35%). No homework this week!! Due on Monday in class or it can be submitted online at http://dlweb.cs.depaul.edu.http://dlweb.cs.depaul.edu Late assignments will be accepted not later than three days after the due date (typically by the following Thursday). Notice that a 10% point penalty will be applied for each day after the deadline. Quizzes (15%). There will be two short tests, scheduled tentatively on week 3 and week 8. Students are allowed to bring one single page of notes and a calculator. There will be no make up quizzes. Midterm (30%) on Feb 9 th, 2004 at 6:15-7:45pm. It is a closed book exam, students are allowed to bring one single page of notes and a calculator. Final (35%) on March 15 th at 6:15 - 8:30 pm. It is a closed book exam, students are allowed to bring two pages of notes and a calculator.
Homework assignments will be posted on Tuesdays. Homework solutions will be due in class on Monday. (Only legible, organized homework will be graded. Include your name, section number, date, and homework number on the first page of your assignment. Staple pages together.) Alternatively, homework can be submitted online at the dlweb site http://dlweb.cs.depaul.edu. The online submission application will let you submit only one document, so use a word processor to collate your solutions in a file. http://dlweb.cs.depaul.edu Duplication of homework solutions and computer output prepared in whole or in part by someone else is not permitted. Homework submission
Lecture 1 Outline Exploratory data analysis (Sec. 1.1, 1.2) Discovering information from the data through graphs and numbers. Introduction to the statistical package SAS
The goal of statistics is to gain information from the data. Data come from several sources: 1.Available data: Census data, Federal agencies, Governmental Statistical Offices (www.fedstats.gov), General Social Survey at the University of Chicago’s NORC (http://www.icpsr.umich.edu/GSS/).www.fedstats.govhttp://www.icpsr.umich.edu/GSS/ Several databases are available on the Internet or at DePaul library!! 2.New Data: Sampling from population of interest: Observational studies Conducting statistical experiments: medical trials, controlled experiments. When well designed, provide most reliable source of information!! Exploratory Data Analysis
What’s the next step after the data collection? Long listings of data are of little value. Statistical methods come to help us. Exploratory data analysis: set of methods to display and summarize the data. Data on just one variable: the distribution of the observations is analyzed by I.Displaying the data in a graph that shows overall patterns and unusual observations (histogram, box plot, density curve) II.Computing descriptive statistics that summarize specific aspects of the data (center and spread).
Data contain information about group of individuals / subjects A variable is a characteristic of an observed individual which takes different values for different individuals: Quantitative variable (continuous) takes numerical values. Ex.: Height, Weight, Age, Income, Measurements Qualitative/Categorical variable classifies an individual into categories or groups. Ex. : Sex, Religion, Occupation, Age (in classes e.g. 10-20, 20-30, 30- 40) The distribution of a variable tells us what values it takes and how often it takes those values Different statistical methods are used to analyze quantitative or categorical variables. Random variables
Graphs for categorical variables The values of a categorical variable are labels. The distribution of a categorical variable lists the count or percentage of individuals in each category. A sample of 400 wireless internet users. Counts: 212 168 20
Wireless internet users Male288 (72%) Female112 (28%) Total400 (100%)
SurvivedDead MaleFemaleMaleFemale First class621411184 Second class 25 9315413 Third class 88 90422106 Crew members 192 206703 Example: On the morning of April 10, 1912 the Titanic sailed from the port of Southampton (UK) directed to NY. Altogether there were 2,201 passengers and crew members on board. This is the table of the survivors of the famous tragic accident. Define the categorical variables
Bar chart representing the data in the table above (in percentages) Male Female Male Female Survivors Survivors Victims Victims
Graphs for qualitative variables: Stemplots Stemplot ~ stem-and-leaf plot To make a stemplot: 1.Separate each observation into a stem consisting of all but the final (rightmost) digit and a leaf, the final digit. 2. Write the stems in a vertical column with the smallest at the top, and draw a vertical line at the right of this column. 3.Write each leaf in the row to the right of its stem, in increasing order out from the stem.
Stemplots (cont.) Back-to-back stemplot How stemplots deal with large data sets? Splitting stems: One stem with leaves between 0 and 4 One stem with leaves between 5 and 9 How stemplots deal with observations with having many digits? Rounding
Stemplots (cont.) Advantages of stemplots: Describe the shape of a distribution for small numbers Disadvantages: Don’t work well with large data sets since they display the values of the variables Divide the observations into groups (stems) determined by the number system rather than by judgment
Graphs for qualitative variables: the histogram Example: CEO salaries Forbes magazine published data on the best small firms in 1993. These were firms with annual sales of more than five and less than $350 million. Firms were ranked by five- year average return on investment. The data extracted are the age and annual salary of the chief executive officer for the first 60 ranked firms. (Data at http://lib.stat.cmu.edu/DASL/DataArchive.html ) Salary of chief executive officer (including bonuses), in $thousands 145 621 262 208 362 424 339 736 291 58 498 643 390 332 750 368 659 234 396 300 343 536 543 217 298 1103 406 254 862 204 206 250 21 298 350 800 726 370 536 291 808 543 149 350 242 198 213 296 317 482 155 802 200 282 573 388 250 396 572
Drawing a histogram 1.Construct a distribution table: i.Define class intervals or bins (Choose intervals of equal width!) ii.Count the percentage of observations in each interval iii.End-point convention: left endpoint of the interval is included, and the right endpoint is excluded, i.e. [a,b[ 2.Draw the horizontal axis. 3.Construct the blocks: Height of block = percentages! The total area under an histogram must be 100%
Class intervals Frequency Use left end-point Percentage= (frequency/tot al)x100 Class intervals Frequency Use left end-point Percentage= (frequency/total) x100 0-10022/59x100=3.39600-70035.08 100-20044/59x100=6.78700-80035.08 200-3001830.50800-90046.78 300-4001423.73900-100000 400-50046.781000-110011.70 500-600610.18Total59100%
The area of each block represents the percentages of cases in the corresponding class interval (or bin). 3.39% 30..50% 23.73% 1.70%
Remarks A histogram represents percent by area. The area of each block represents the percentages of cases in the corresponding class interval. The total area under a histogram is 100% There is no fixed choice for the number of classes in a histogram: If class intervals are too small, the histogram will have spikes; If class intervals are too large, some information will be missed. Use your judgment! Typically statistical software will choose the class intervals for you, but you can modify them.
SMOKING In a Public Health Service study, a histogram was plotted showing the number of cigarettes smoked per day by each subject (male current smokers), as shown below. The density is marked in parentheses. The class intervals include the left endpoint, but not the right. 1.The percentage who smoked less than two packs a day but at least a pack, is around (There are 20 cigarettes in a pack.) 1.5% 15%30%50% 2.The percent who smoked at least a pack a day is around 1.5% 15%30%50% 3.The percent who smoked at least 3 packs a day is around 0.25 of 1% 0.5 of 1%10% 4.The percent who smoked 20 cigarettes a day is around 0.35 of 1% 0.5 of 1% 1.5% 3.5% 10%
Answers: 1.The percentage who smoked less than two packs a day but at least a pack, is around (There are 20 cigarettes in a pack.) It is given by the area of the third block: 1.5x(40-20)=1.5x20=30% 2.The percent who smoked at least a pack a day is around It is given by the area of the third and fourth blocks: 30+0.5x40=50% 3.The percent who smoked at least 3 packs a day is around It is the area of the block for number of cigarettes greater or equal to 60. This is half of the fourth block: 10% 4.The percent who smoked 20 cigarettes a day is around We use the left endpoint convention, so 20 belongs to the third block. The answer is 1.5%.
Fuel economy for model year 2001 compact and two- seater cars (Table 1.8 pg 38) City Consumption Highway consumption Using histograms for comparisons
Describing distributions with numbers A distribution can be described through the measures of its center and of its spread. Measuring the center The most common measures are the mean or average and the median. 1.The Mean or Average To calculate the average of a set of observations, add their value and divide by the number of observations: Data: Number of home runs hit by Babe Ruth as a Yankee 54, 59, 35, 41, 46, 25, 47, 60, 54, 46, 49, 46, 41, 34, 22 The mean number of home runs hit in a year is:
2.The median The median M is the midpoint of a distribution, the number such that half the observations are smaller and the other half are larger. To find the median: 1.Sort all the observations in order of size from smallest to largest 2.If the number of observations n is odd, the median M is the center observation in the ordered list; I.e. M=(n+1)/2-th obs. 3.If the number of observations n is even, the median M is the mean of the two center observations in the ordered list. Example 1: Ordered list of home run hits by Babe Ruth: 22 25 34 35 41 41 46 46 46 47 49 54 54 59 60 N=15 Median = 46 8 th Example 2: Ordered list of home run hits by Roger Maris in 1961: 8 13 14 16 23 26 28 33 39 61 N=10 Median = (23+26)/2=24.5
Mean versus Median 1.The mean and median of a symmetric distribution are close together Mean Median 50% 2.In skewed distributions, the mean is farther out in the long tail than is the median. The mean is more sensitive to extreme values. MedianMean MedianMean Right-skewed distribution Left-skewed distribution Symmetric distribution
Mean or median? v The mean is a good measure for the center of a symmetric distribution v The median is a resistant measure and should be used for skewed distributions. Its value is only slightly affected by the presence of extreme observations, no matter how large these observations are.
Example: Shopping in a supermarket A marketing consultant observed 50 consecutive shoppers at a supermarket. The histogram below shows how much each shopper spent in the store. Summary statistics: Mean = $ 34.70 Median = $ 27.855 The mean does not say much… The median says that about 50% of the shoppers spent less than 28 dollars What else would you like to know?
Q 1 M Q 3 Spread of a Distribution Two measures of spread: 1. The Quartiles: First quartile Q 1 = the value such that 25% of the observations fall at or below it, (Q 1 is often called 25th percentile). The third quartile Q 3 = the value such that 75% of the observations fall at or below it, (Q 3 is often called 75th percentile). Typically used if the distribution of the observations is skewed. IQR The Inter-Quartile Range IQR is defined as the distance between the two quartiles: IQR= Q 3 – Q 1
Example: Shopping in a supermarket A marketing consultant observed 50 consecutive shoppers at a supermarket. The histogram below shows how much each shopper spent in the store. Summary statistics: Mean = $ 34.70 Median = $ 27.855 Q 1 = $19.27 Q 3 = $ 45.40 IQR= 45.40-19.27= 26.13 About 50% of the shoppers spent less than 28 dollars, 25% spent less than 20 dollars and 25% of the customers of the store spent more that 45 dollars. Moreover, 50% of the customers spent between 20 and 45 dollars! Extreme values for purchases > Q 3 + 1.5xIQR=84.59