Research Methods for Counselors COUN 597 University of Saint Joseph Class # 6 Copyright © 2015 by R. Halstead. All rights reserved.

Class Objectives  Salkind Chapter 8 – Probability  Salkind Chapter 9 – Statistical Significance

Probability  The analysis of data that is generated from an experiment usually involves calculating the resulting means of the experimental and control groups and then test those means to establish whether or not they are “significantly different.”  The term “significantly different” is used to denote, with a specified degree of confidence, that any difference between the means did not occur by chance.

Probability and the Normal Curve  Any time we make reference to chance, we are automatically implying the application of probability.  The basic tool that helps us work effectively with probability is the normal curve.

The Normal Curve  Characteristics  Mean, Median, and Mode are equal  Perfectly symmetrical in shape  The tails of the curve approach the horizontal axis but never touch it  How do we establish a sample that creates the normal curve?  Randomly select a large enough sample (the larger the better but 30 is thought a minimum)

Standard Scores  Standard Scores are simply scores that have been standardized so as to make them consistent with the Standard Deviation of a distribution.  The value of setting up a standard score is that it allows us to compare scores from different test or measure on the same scale.  One type of Standard Score is the Z Score.  The Z Score has a mean of zero and and a standard deviation of one.

Converting a Test Score into a Standard Scores  The formula for this conversion is as follows: (X - X) s Z is the z score X is the individual score X is the mean of the distribution s is the distribution standard deviation Z =

The Z Score  Beside the z score allowing us to compare scores from different tests against one another on the same scale, we can also determine what percent of scores we would expect to find above and below that score.  Logic would tell us, given that the normal curve is really about probability, that we can also determine the probability of a certain score occurring at or above (or below) any z score.

Z Scores and Where this is Heading  To date, we have established three points.  First, we know that conducting research is about establishing a reasonable empirical conclusion.  Second, empirical knowing is established by a process of examining the nature of differences.  Third, any differences (or lack there of) could be the result of random chance.  Now lets move one more step forward.

Z Scores and Where this is Heading  If we want to be sure that the difference between two observations, say the average levels of depression between a cognitive therapy treatment group and a no treatment control group after 10 weeks of treatment, means something, we want to be relatively certain that any difference is due to the treatment and not due to chance.  The z score is the first step toward actually being able to empirically determine that type of reasonably acceptable conclusion.

The Concept of Significance  As we have discussed in previous classes, when we engage in a process of conducting quantitative research we use a specific method for establishing a certain form of knowing.  This method (the scientific method) utilizes a process of comparison.  Often what the researcher is comparing is one group against another group. When a difference is found the research draws some conclusion.

The Concept of Significance  Given that the researcher’s conclusions are going to be based on differences regarding some variable of interest, there are two major elements that must be taken into account.  First, we want to make sure that the difference found is due to the manipulation of the independent variable and not some other factor or factors.  The researcher addresses this by trying to control for confounding variables and minimizing sampling error and sampling bias.

The Concept of Significance  Second, once we have controlled, as much as possible, for confounding factors we want to make sure that the difference found is not due to those elements that we can not control for and are introduced by random chance.  Let’s say that you find a difference between your experimental and control group. How can you be certain that the resulting difference is due to the manipulation of the independent variable and not due to random chance?

The Concept of Significance  The answer to that question leads us to a special type of difference that a research seeks to find. That is the “Significant Difference” between groups.  Another way to express this is to say that the difference between the groups was found to be... “Statistically Significant”

Statistical Significance: Researchers as Chickens  Okay - that title is a bit of an over statement but in essence dealing with levels of statistical significance is about the risk of being wrong.  Remember you can never be 100% certain of any research outcome so what the researcher says is, “I am willing to state that my hypothesis holds and differences are due to my manipulation of the independent variable 95% of the time - but, to be on the safe side, 5% of the time this result could be due to other factors.”

Statistical Significance and Research Hypotheses  So you set a certain level of risk with which you are willing to live - p <.05 (less than 5%). This says that the probability (p) that any difference between the experimental and control group is due to some factor other than the manipulation of the independent variable is less than 5%.  In testing the null hypothesis, if your test results are shown to fall within that p <.05 region of the normal curve you conclude that the difference is due to the I.V. and not due to other factors.

Statistical Significance and Being Wrong  Step 1 - Establish the study you wish to conduct.  Step 2 - Write your null hypothesis  Step 3 - Conduct your study and collect data  Step 4 - Analysis your data by testing for differences accepting the a certain level of risk with which you are willing to live - say p <.05  Step 5 - Publish your results and become famous in the field of counseling  Piece of cake - right? No so fast researcher!!

Statistical Significance and Being Wrong  Remember that risk you were willing to live with?  Well sometimes it is not so easy - sometimes your results will really be due to something other than your I.V. even though you have stated that the your I.V. was responsible for the difference between the experimental and control groups. In the world of research this is called... Being Wrong!!

Statistical Significance and Being Wrong  When we deal with hypotheses we have to make a decision. Either the hypothesis is true or it is false.  There are two ways we can get into trouble.  The first kind of trouble is when we say the hypothesis is false (reject the null hypothesis) when it is true - Type I Error.  The second kind of trouble is when we say the hypothesis is true (accept the null hypothesis) when it is false - Type II Error.

Decisions about the Null Hypothesis and Type I & II Error Accepted the Null Hypothesis and it is true Reject the Null Hypothesis and it is true - Type I Error Accepted the Null Hypothesis and it is false - Type II Error Rejected the Null Hypothesis and it is false AcceptReject Null Hypothesis is True Null Hypothesis is False

Reducing the Likelihood of Type I and Type II Error  You have some control over Type I Error in that it is based on the p level that you set for your test.  Type II Error is a bit more difficult to deal with in that it results most often from the samples not being an accurate representation of the population from which they were drawn.  Reducing Type II Error is best accomplished by establishing samples that are large enough so as to increase confidence that the sample represents the population it is supposed to represent.

Significant but Meaningful?  A primary purpose of conducting research is to learn something about the world. The value in conducting research is the increase knowledge and sharing it with others.  When the analysis of your data results in statistically significant differences you must explain the meaning of those results.  Correlation Example - you have 45 cases how large of a correlation would you need for a one tailed test at a p level of <.05? Significant but meaningful?

Significant but Meaningful?  Significant relationship? Yes. Meaningfully different? Probably not.  Remember the idea behind conducting research is not to show that you were right about the hypothesis but rather to learn something useful for practice.

Useful Phrases for Reporting Research - NOT  “It has long been known...”  “A definite trend is evident.”  “Correct within an order of magnitude.”  “Three of the samples were chosen for a detailed study.” I didn’t look up any sources The results of the others didn’t make any sense Wrong Data are practically meaningless

Inferential Statistics  Up to this point in the semester we have used descriptive statistics to understand the characteristics of a sample with which we were working.  We are now about to go boldly into the fascinating world of inferential statistics.  Inferential statistics are used to allows the researcher to say something about the population given what can be known about the sample.

The Logic of Inference  The researcher randomly selects a sample from a population and assigned participants to two groups that are found not to differ prior to the start of the research study.  One group gets some form of treatment and one group gets another form of treatment.  After ten weeks of treatment the researcher administers a test to each group and computes and compares the means for each group.

The Logic of Inference  A conclusion is reached about the effectiveness of the two treatments relative to each other.  The researcher infers that the same conclusion would also hold for the entire population that the sample represents.  Thus the researcher can report these findings to the community of professional counselors and client’s lives are made better due to a discovery of a more effective treatment. Pretty neet!!

Statistical Tests for Finding Differences Between Groups  As the chart on page 152/183/173 of your Salkind text suggests there are a variety of tests available to the researcher.  That chart is a good one to keep handy in that it offers a quick method for helping to determine which statistical test is appropriate for a given testing situation.

General Steps in Applying Tests of Statistical Significance  1. A statement of the Null Hypothesis.  The assumption that, given no other information, all will balance out equally.  2. Set some level of risk (significance) by which you will be willing to accept or reject the hypothesis.  Remember that the risk you are taking here is that you will be wrong (Type I Error).

General Steps in Applying Tests of Statistical Significance  3. Select the appropriate test statistic.  Each research situation requires a specific test for the job. We will learn more about that in the weeks to come.  4. Compute the test statistic to arrive at the obtained value for the test.  5. Determine the value needed for rejecting the null hypothesis using the appropriate table of critical values for the statistic you are using.

General Steps in Applying Tests of Statistical Significance  6. Compare the obtained value to the critical value.  If the obtained value is more extreme than the critical value - reject the null hypothesis.  If the obtained value is less extreme than the critical value - accept the null hypothesis.