1 Validity Conclusions are appropriate Conclusion are true

2 Validity = Truth Truth about a thing –Person, event, group Truth about a relationship or difference –Servant Leadership affects Organizational Commitment –Men and Women are not different in Leadership Ability

3 SUMMARIZING and PRESENTING a large body of facts offers to IGNORANCE or MALICE ample opportunity for DISTORTION (lower validity). This is NO LESS, but also NO MORE, the case when the facts to be summarized are NUMBERS (Quantitative) rather than WORDS (Qualitative). We will therefore take due note of the TRAPS that the IGNORANT fall into and the TOOLS OF DUPLICITY that the MALICIOUS use.

4 12 Scenarios A research study will be described. Is the conclusion true (based on the research information presented)? If not, why not?

5 Scenario #1 A researcher studied the relationship between Parental Violence and School Learning for Hispanic 9th grade students in Los Angeles. Surveys were distributed to 10,000 Hispanic 9th grade students in Los Angeles. All students returned the survey. Students were asked two questions on the survey: 1. How often does your father beat your mother? The responses were on a scale of 1 to 10, with 1 = Never, and 10 = Very frequently. 2. What is your gpa? The responses were gpa on a scale from 0.00 (F) to 4.00 (A). The results showed that there was a correlation of -.80 between the two variables. The researcher concluded that “Ninth-grade Hispanic youth do better in school if they live in homes where there is less violence.”

6 Scenario #2 A researcher studied the relationship between Class Size and Achievement for Hispanic 9th grade students. A stratified random sample was selected. A random sample of 20 school districts was selected from each state that had at least 10% Hispanic residents. A random sample of 3 schools was selected from each school district. Schools with less than 10% Hispanic students were excluded. All 9th grade students from each of these schools were surveyed. A total of 34,264 students were sampled. All students returned the survey. At the end of the school year, students were asked two questions on the survey: 1. How many students are in your English class? 2. What is your grade in English this year? The responses were gpa on a scale from 0.00 (F) to 4.00 (A). The results showed that there was a correlation of -.60 between the two variables. The researcher concluded that “Ninth-grade Hispanic youth do better in school if they have smaller classrooms.”

7 Scenario #3 A researcher studied the relationship between Class Size and English Achievement for Hispanic 9th grade students. A stratified random sample was selected. A random sample of 20 school districts was selected from each state that had at least 10% Hispanic residents. A random sample of 3 schools was selected from each school district. Schools with less than 10% Hispanic students were excluded. Data for all 9th grade students from each of these schools were obtained. There were 34,264 students in the sample. At the end of the school year the following information was obtained for each student from computer records: 1. How many students were in the second semester English class in which the student was enrolled? and 2. What was the student’s grade in English for the spring semester? Grades were reported on a scale of A (4.0) to F (0.0). The results showed that there was a correlation of.60 between the two variables. The researcher concluded that “Ninth-grade Hispanic youth do better in 9th-grade English if they are in classes with fewer students.”

8 Scenario #4 A researcher studied the relationship between Class Attendance and Science Learning for 4th grade students in Los Angeles. All 4th grade students in Los Angeles were tested in October using the latest version of the Iowa Tests of Basic Skills. Students for whom the test was determined to be not valid were excluded from the testing. Attendance records were gathered from the administrative office of each school. The attendance variable was recorded as the number of days that the student missed school during the year. The results showed that there was a correlation of -.90 between the two variables. The researcher concluded that “Fourth-grade Hispanic students in Los Angeles do better in Science if they have better attendance in school.”

9 Scenario #5 A researcher studied the relationship between Class Attendance and Science Learning for Hispanic 4th grade students in Los Angeles. All Hispanic 4th grade students in Los Angeles were tested in May using the latest version of the Iowa Tests of Basic Skills. Attendance records were gathered from the administrative office of each school. The attendance variable was recorded as the number of days that the student missed school that year prior to the testing date. The results showed that there was a correlation of -.90 between the two variables. The researcher concluded that “Fourth-grade Hispanic students in Los Angeles do better in Science if they have better attendance in school.”

10 Scenario #6 A researcher studied the relationship between Class Size and Science Learning for 6th grade students. A stratified random sample was selected. A random sample of 20 school districts was selected from each state. A random sample of 3 schools was selected from each school district. Surveys were given to all 6th grade students from each of these schools. A total of 347,264 students were distributed. 73,539 surveys were returned. At the end of the school year, students were asked two questions on the survey: 1. How many students are in your Science class? 2. What is your grade in Science this year? The responses were gpa on a scale from 0.00 (F) to 4.00 (A). The results showed that there was a correlation of -.60 between the two variables. The researcher concluded that “Fourth-grade students do better in Science if they are in classrooms with fewer students.”

11 Scenario #7 A researcher studied the relationship between Drug Use and School Learning for 9th grade students in Los Angeles. Surveys were distributed to all 9th grade students in Los Angeles. All students returned the survey. Students were asked two questions on the survey. 1. How often do you use heroin? The responses were on a scale of 1 to 10, with 1 = Never, and 10 = Daily. 2. What is your gpa? The responses were gpa on a scale from 0.00 (F) to 4.00 (A). The results showed that there was a correlation of -.80 between the two variables. The researcher concluded that “Ninth-grade youth do better in school if they do not use heroin.”

12 Scenario #8 A researcher studied the impact of the doctoral program in Leadership program on the values of its participants. All graduates of the Leadership program during the latest year were interviewed. A total of 5 persons were interviewed. As part of the portfolio presentation, the director of the program asked each graduate to “Describe how the Leadership program has affected your values?” Members of the participant’s examining committee asked follow-up questions to clarify the participant’s responses. The responses were taped and transcribed. The director of the program listened to the tapes and analyzed the responses. The results showed that all participants stated that their values had changed positively as a result of the leadership program. The researcher concluded that “The doctoral program in Leadership has a positive impact on the values of the doctoral participants for the latest year.”

Scenario #9 A researcher studied the impact of the roundtable experience in Leadership program on its participants. A random sample of 40 graduates of the Leadership program over the past 10 years were selected. Two male and two female participants each year were selected. All persons selected agreed to be interviewed. Two doctoral students in Education who were not part of the Leadership program was trained to conduct the interviews. The interviewers attended each roundtable as a participant to become familiar with what occurred there. Prior to the first roundtable, the interviewers also met with the faculty and were briefed on the philosophy of the program and purposes of the roundtable experience. During one two-month period, each Leadership participant was interviewed in person for one hour by one of the doctoral students. The entire one-hour was spent discussing the impact of the roundtable experience and what aspects of the roundtable were most beneficial, how they were beneficial, and how the roundtable could be improved?” The responses were taped and transcribed. Both interviewers listened to all tapes and independently analyzed the responses. Where they disagreed on their interpretation, they discussed the responses until they agreed. The results showed that all participants stated that the roundtables were judged by the subjects to have been the most valuable part of their doctoral program. The researcher concluded that “The roundtable experience has a positive impact on Leadership participants.”

14 Scenario #10 A researcher studied the impact of the roundtable experience in Leadership program on its participants. Twenty Leadership students were interviewed, four participants each year for the last five years. Two doctoral students in Education who were not part of the Leadership program were trained to conduct the interviews. The interviewers attended each roundtable as a participant to become familiar with what occurred there. Prior to the first roundtable, the interviewers also met with the faculty and were briefed on the philosophy of the program and purposes of the roundtable experience. Each Leadership participant was interviewed in person for one hour by one of the doctoral students one month after the roundtable. The entire one-hour was spent discussing the impact of the roundtable experience and what aspects of the roundtable were most beneficial, how they were beneficial, and how the roundtable could be improved?” The responses were taped and transcribed. Both interviewers listened to all tapes and independently analyzed the responses. Where they disagreed on their interpretation, they discussed the responses until they agreed. The results showed that all participants stated that the roundtables were judged by the subjects to have been the most valuable part of their doctoral program. The researcher concluded that “The roundtable experience has a beneficial impact on Leadership participants.”

15 Scenario #11 A researcher studied the impact of the roundtable experience in Leadership program on its participants. Over a 10-year period, two participants of the Leadership program were randomly selected each year, one male participant and one female participant. All persons selected agreed to be interviewed. The faculty member in charge of the roundtable had a one-hour in-person interview with each subject at the close of roundtable each year. The entire one-hour was spent discussing the benefits of the roundtable experience and what aspects of the roundtable were most beneficial and how they were beneficial?” The responses were taped and transcribed. The faculty member listened to the tapes and analyzed the responses. The results showed that all participants stated that the roundtables were judged by the subjects to have been the most valuable part of their doctoral program. The researcher concluded that “The roundtable experience has a beneficial impact on Leadership participants.”

16 Scenario #12 A researcher studied the impact of the roundtable experience in Leadership program on its participants. During the latest Leadership roundtable, all participants were selected to be studied. All persons agreed to be interviewed. Ten doctoral students in Education who were not part of the Leadership program were trained to conduct the interviews. The interviewers attended roundtable as a participant to become familiar with what occurred there. Prior to the roundtable, the interviewers also met with the faculty and were briefed on the philosophy of the program and purposes of the roundtable experience. Each Leadership participant was interviewed in person for one hour by one of the doctoral students immediately after the roundtable. The entire one-hour was spent discussing the impact of the roundtable experience and what aspects of the roundtable were most beneficial, how they were beneficial, and how the roundtable could be improved?” The responses were taped and transcribed. Each tape was independently analyzed by four of the interviewers. Where they disagreed on their interpretation, they discussed the responses until at least three of the interviewers agreed. The results showed that all participants stated that the roundtable was judged by the subjects to have been the most valuable part of their doctoral program. The researcher concluded that “The roundtable experience has a beneficial impact on Leadership participants. The roundtable should be continued as part of the Leadership program.”

17

18 Validity Validity is a characteristic – an attribute Valid is an adjective In research, a valid ___ is something that is true or correct or appropriate. Validity is unique to the situation – what is appropriate in one situation may not be in another.

19 Validity Valid sample Valid data Valid analysis Valid conclusion

20 Threats to Validity - 1 Bad Data –Bad sample –Bad instrument Bad Analysis –Bad manipulation –Bad conclusion

21 Threats to Validity - 2 Internal Validity –Bad measurement (e.g., wrong information) –Bad data (e.g., clerical errors) –Bad analysis (e.g., assumptions violated) External Validity –Bad sample (e.g., not representative, too small) –Bad conclusions (e.g., not related to the data)

22 Decisions Points Where Validity is Affected Decision PointsQualitativeQuantitative 1. Choice of data-gathering methodology a. choice of procedure (e.g., interview)xx b. choice of test (e..g, GRE)x c. choice of data-producing element (e.g., item)xx 2. Choice of sample a. choice of method (e.g., stratified)xx b. choice of sample sizexx 3. Choice of analysis method a. statistical (e.g., ANOVA)x b. verifiable (e.g., reliable, triangulation)xx 4. Choice of conclusion a. data-relatedxx b. speculativexx

23 Conclusions The aim of most research agendas is to pursue truth. We usually only approach truth. The aim of most research studies is to arrive at conclusions that can be viewed as moving toward truth. We should aim for intellectual pluralism rather than relativism. –Relativism = All points of view are equally valid or equally correct (there is no truth or no way to evaluate truthfulness). –Intellectual pluralism = accept alternative points of view as being possibly correct but some are better (more likely to be true) than others. We should use methods that are as valid as possible. We should make conclusions taking limitations of our methods into consideration.