Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College,

Similar presentations


Presentation on theme: "1 Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College,"— Presentation transcript:

1 1 Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College, May 2009

2 2 Using Questions to Gather SoTL Evidence Types of Questions Types of Questions Question Design Question Design Question Issues and Problems Question Issues and Problems Types of Response Scales Types of Response Scales Design and Preparation of Survey Design and Preparation of Survey

3 3 Asking Questions To Gather Evidence

4 4 Methods for Gathering Evidence Evidence can include: Evidence can include: Course-Intrinsic Evidence Course-Intrinsic Evidence Course-Extrinsic Evidence Course-Extrinsic Evidence Using your research question, what is one type of course-intrinsic and one type of course- extrinsic evidence you might collect? Using your research question, what is one type of course-intrinsic and one type of course- extrinsic evidence you might collect?

5 5 Asking Questions All SoTL evidence gathering is focused on asking questions of students. All SoTL evidence gathering is focused on asking questions of students. Research on writing survey questions provides comprehensive information on how to do this best Research on writing survey questions provides comprehensive information on how to do this best

6 6 Survey Evaluation Look at survey in packet. Look at survey in packet. What works? What works? What doesn’t work? What doesn’t work? What is missing? What is missing?

7 7 What Survey Research Tells Us About Asking Good Questions

8 8 The Impact of Questions on the Research Process Poor questions lead to lousy results. Poor questions lead to lousy results. How might answers to these questions differ? How might answers to these questions differ? Do you attend class? Do you attend class? Do you regularly attend class? Do you regularly attend class? How many times per week do you attend this class? How many times per week do you attend this class? Asking survey questions is part science and part art. Asking survey questions is part science and part art.

9 9 Types of Survey Questions Open Ended and Closed Ended Question Formats Open Ended and Closed Ended Question Formats Attitude, Knowledge, and Behavior Question Formats Attitude, Knowledge, and Behavior Question Formats

10 10 Open-Ended Questions Open-ended questions ask respondents to state in their own words their response to the question. Open-ended questions ask respondents to state in their own words their response to the question. Closed-ended (or fixed alternative) questions ask respondents to respond by picking one or more alternatives from a list of available options. Closed-ended (or fixed alternative) questions ask respondents to respond by picking one or more alternatives from a list of available options. Can you find any examples of open-ended questions on the survey? Are these good open-ended questions? Can you find any examples of open-ended questions on the survey? Are these good open-ended questions?

11 11 Open Ended Questions What are advantages and disadvantages to open- and close-ended questions? What are advantages and disadvantages to open- and close-ended questions?

12 12 Advantages and Disadvantages Are often less time consuming Are often less time consuming May provide more complete answers May provide more complete answers May provide information researchers did not anticipate May provide information researchers did not anticipate Responses are often easier to analyze Responses are often easier to analyze May produce better response rate May produce better response rate

13 13 Surveys: Asking Questions Surveys are used most often in SoTL research when we are interested in asking questions about: Surveys are used most often in SoTL research when we are interested in asking questions about: how students FEEL or THINK about certain issues, topics, activities, related to their learning, how students FEEL or THINK about certain issues, topics, activities, related to their learning, what students KNOW about various topics, etc. what students KNOW about various topics, etc. how often students engage in certain learning BEHAVIORS how often students engage in certain learning BEHAVIORS

14 14 Attitude/Feelings Questions Attitude/feelings questions ask respondents for their views on certain topics or issues Attitude/feelings questions ask respondents for their views on certain topics or issues There are no right or wrong answers to attitude/feeling questions. There are no right or wrong answers to attitude/feeling questions. Attitude/feeling questions should be “neutrally” written so that the respondent is not led to believe that a certain response is desired. Attitude/feeling questions should be “neutrally” written so that the respondent is not led to believe that a certain response is desired.

15 15 Knowledge Questions Knowledge questions ask about what a respondent knows about a particular topic Knowledge questions ask about what a respondent knows about a particular topic Knowledge questions can be asked as a screening question to determine if respondents are qualified to answer attitude questions. Knowledge questions can be asked as a screening question to determine if respondents are qualified to answer attitude questions. “What do you know about evolution? “What do you know about evolution? “How do you feel about evolution as a theory of creation?” “How do you feel about evolution as a theory of creation?”

16 16 Behavior Questions Behavior questions ask respondents to report on their past or future behavior Behavior questions ask respondents to report on their past or future behavior Responses to behavior questions may be biased because of faulty or biased recall by respondents Responses to behavior questions may be biased because of faulty or biased recall by respondents

17 17 Attitude, Knowledge, and Behavior Questions Can you find an attitude/feelings, knowledge, and behavior question on the survey? Can you find an attitude/feelings, knowledge, and behavior question on the survey? Write one attitude/feelings, knowledge, or behavior question for your own SoTL project. Write one attitude/feelings, knowledge, or behavior question for your own SoTL project. Pre-assessment measure Pre-assessment measure Question for an assignment you will collect Question for an assignment you will collect Post-assessment Post-assessment

18 18 Question Issues and Problems

19 19 Recall Questions Is there a question on survey that asks students to “recall” some behavior? Is there a question on survey that asks students to “recall” some behavior? How would you evaluate these questions? How would you evaluate these questions? Would you know how to answer them? Would you know how to answer them? Is there a better way to ask these questions? Is there a better way to ask these questions?

20 20 Using Diaries for Recall Questions To minimize recall errors, researchers can ask respondents to keep personal diaries to record their actual teaching or learning activities over a period of time To minimize recall errors, researchers can ask respondents to keep personal diaries to record their actual teaching or learning activities over a period of time Caution: Respondents may forget to fill out their diaries on a timely basis and instead “catch up” their diaries just before they are due to be collected Caution: Respondents may forget to fill out their diaries on a timely basis and instead “catch up” their diaries just before they are due to be collected

21 21 Unaided vs. Aided Recall Questions Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use aided recall questions to “jog” their memories. Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use aided recall questions to “jog” their memories. Unaided recall: “Please list all of the Communication classes in which you participated in learning groups last semester.” (open-ended question) Unaided recall: “Please list all of the Communication classes in which you participated in learning groups last semester.” (open-ended question) Aided recall: “Please place a check by any of the Communication courses in which you participated in a learning group last semester.” (closed-ended question) Aided recall: “Please place a check by any of the Communication courses in which you participated in a learning group last semester.” (closed-ended question) Public Speaking Public Speaking Introduction to Interpersonal Communication Introduction to Interpersonal Communication Small Group Communication Small Group Communication

22 22 Recall Questions and Time Frame We often want respondents to focus on a specific time frame when answering a question. We often want respondents to focus on a specific time frame when answering a question. Avoid ambiguous terms like “past year” or “next year” because respondents may not know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period. Avoid ambiguous terms like “past year” or “next year” because respondents may not know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period. Avoid time frames that are too large (for the recall question) or too far in the past. Avoid time frames that are too large (for the recall question) or too far in the past. Write one recall question for gathering evidence for your SoTL research question. Write one recall question for gathering evidence for your SoTL research question.

23 23 Intimidating or Inappropriate Questions Respondents will consider a question to be intimidating or inappropriate if: Respondents will consider a question to be intimidating or inappropriate if: It addresses highly personal and private behavior (drinking behavior, sexual activity, illegal drug use, etc.) It addresses highly personal and private behavior (drinking behavior, sexual activity, illegal drug use, etc.) It addresses subjects for which the respondent believes there are socially desirable norms (communication with instructors, behavior in class, spiritual behavior, etc.) It addresses subjects for which the respondent believes there are socially desirable norms (communication with instructors, behavior in class, spiritual behavior, etc.) If respondents perceive a question to be intimidating, they are more likely to refuse to answer or to give an inaccurate answer. If respondents perceive a question to be intimidating, they are more likely to refuse to answer or to give an inaccurate answer.

24 24 Intimidating Questions Would you use an open- or closed-ended question to ask a potentially intimidating question to gather evidence from your students? Would you use an open- or closed-ended question to ask a potentially intimidating question to gather evidence from your students? Examples? Examples?

25 25 Forms for Asking Potentially Intimidating Questions About Behavior If you ask a closed-ended question with a set of response options, people typically do not choose the extreme options. If you ask a closed-ended question with a set of response options, people typically do not choose the extreme options. Therefore, it is often better to use an open- ended question and allow the respondents to indicate how often they engage in certain behaviors, or define the terms as they perceive them. Therefore, it is often better to use an open- ended question and allow the respondents to indicate how often they engage in certain behaviors, or define the terms as they perceive them.

26 26 Forms for Asking Potentially Intimidating Questions About Behavior Third person questions ask respondents to report on the behavior and activities of other people. Third person questions ask respondents to report on the behavior and activities of other people. Example: Instead of asking college students directly about their alcohol consumption behavior and its impact on their learning, you could ask students to report how much alcohol their roommates consume and how that affects their learning behavior. Example: Instead of asking college students directly about their alcohol consumption behavior and its impact on their learning, you could ask students to report how much alcohol their roommates consume and how that affects their learning behavior.

27 27 Double-Barreled Questions Double-barreled questions ask respondents for their opinions about two distinct issues in the same question. Double-barreled questions ask respondents for their opinions about two distinct issues in the same question. “Do you prefer working in groups and learning from your fellow students, or do you prefer lectures and learning from the instructor? “Do you prefer working in groups and learning from your fellow students, or do you prefer lectures and learning from the instructor? Any other examples? Any other examples?

28 28 Leading and Loaded Questions Some “surveys” seek to generate results that support a set of predetermined goals and objectives. Some “surveys” seek to generate results that support a set of predetermined goals and objectives. How might we re-write a question on the survey to make it a leading question? How might we re-write a question on the survey to make it a leading question?

29 29 Survey Question Design

30 30 Ordering of Questions Start with easy, salient, and non-threatening questions first. Start with easy, salient, and non-threatening questions first. Complete questions on a single topic before moving on to a new topic. Complete questions on a single topic before moving on to a new topic. When switching topics, use transitional phrases to make it easier for respondents to switch their train of thought. When switching topics, use transitional phrases to make it easier for respondents to switch their train of thought. Put demographic questions near the end. Put demographic questions near the end. Do a quick evaluation of survey to see how it fits criteria. Do a quick evaluation of survey to see how it fits criteria.

31 31 Funnel Questions Questions regarding a specific topic should usually start with the most general questions and then follow with increasingly more restrictive questions. Questions regarding a specific topic should usually start with the most general questions and then follow with increasingly more restrictive questions. If you were required to work in groups in your class, what types of students would you want in your group? If you were required to work in groups in your class, what types of students would you want in your group? If you had to select just three key characteristics that you would use to choose among these different students, what would these 3 factors be? If you had to select just three key characteristics that you would use to choose among these different students, what would these 3 factors be? Would “student’s GPA” be an important factor in your choice of students for your work group? Would “student’s GPA” be an important factor in your choice of students for your work group? If your instructor indicated that she was going to put students in groups according to similar GPAs, would you be interested in working in such a group? If your instructor indicated that she was going to put students in groups according to similar GPAs, would you be interested in working in such a group?

32 32 Skipping and Branching Surveys can instruct respondents to skip over certain questions if these questions do not pertain to them. Surveys can instruct respondents to skip over certain questions if these questions do not pertain to them. Question 7: “Have you worked in a classroom learning group in the past semester?” Question 7: “Have you worked in a classroom learning group in the past semester?” ____ Yes (please proceed to question 8) ____ Yes (please proceed to question 8) ____ No (please skip to question 12) ____ No (please skip to question 12) Any examples on survey? Any examples on survey?

33 33 Demographic Questions In most surveys, questions regarding demographic characteristics of the respondents are included at end. In most surveys, questions regarding demographic characteristics of the respondents are included at end. Typical demographic questions: Typical demographic questions: Sex Sex Ethnicity/Racial background Ethnicity/Racial background Age Age Year in School Year in School Major Major Are there better ways to write these demographic questions on the survey? Are there better ways to write these demographic questions on the survey?

34 34 Wording of Survey Questions

35 35 Avoid Ambiguity in Wording Some common words, such as “regularly” or “often”, may be interpreted very differently by respondents. Some common words, such as “regularly” or “often”, may be interpreted very differently by respondents. Do you go to talk to your instructor regularly (outside of class)? Do you go to talk to your instructor regularly (outside of class)? How often do you go talk to your instructor (outside of class): How often do you go talk to your instructor (outside of class): 1-2 times a week 1-2 times a week 3-4 times a week 3-4 times a week 5-7 times a week 5-7 times a week More than 7 times a week More than 7 times a week What might be the issue with these questions? What might be the issue with these questions?

36 36 Response Scales Used To Collect Survey Data

37 37 Features of Response Categories Any response category must be mutually exclusive. Any response category must be mutually exclusive. There should not be any overlap in the categories. There should not be any overlap in the categories. Any response category system should be collectively exhaustive. Any response category system should be collectively exhaustive. All possible responses should be included in the category system. All possible responses should be included in the category system. Usually it is advisable to include an “other” category to allow respondents to add a response that is not included in the category system. Usually it is advisable to include an “other” category to allow respondents to add a response that is not included in the category system. On survey, evaluate questions for exclusivity and collective exhaustion. Any issues? On survey, evaluate questions for exclusivity and collective exhaustion. Any issues?

38 38 Semantic Differential Scales A scale, typically used to measure attitudes or perceptions, that includes end points that are anchored by bipolar adjectives. A scale, typically used to measure attitudes or perceptions, that includes end points that are anchored by bipolar adjectives. Outstanding _ _ _ _ _ _ _ Unacceptable Strong _ _ _ _ _ _ _Weak Moral _ _ _ _ _ _ _Immoral Any examples of a semantic differential scale on survey? Any examples of a semantic differential scale on survey?

39 39 Likert Scales A scale, typically used to measure attitudes, that usually includes end points of “strongly agree” and “strongly disagree.” A scale, typically used to measure attitudes, that usually includes end points of “strongly agree” and “strongly disagree.” Abiding by page limits is important for writing a good paper: Strongly Agree AgreeNeutralDisagree Strongly Disagree Any examples of a Likert scale on the survey? Any examples of a Likert scale on the survey?

40 40 Numerical Scales A scale that uses multiple numbers, usually 5 to 10, with anchored end points to represent the strength or position of respondents’ attitudes. A scale that uses multiple numbers, usually 5 to 10, with anchored end points to represent the strength or position of respondents’ attitudes. 1 strongly disagree 1 strongly disagree 2 disagree 2 disagree 3 neutral 3 neutral 4 agree 4 agree 5 strongly agree 5 strongly agree Any examples of a numerical scale on survey? Any examples of a numerical scale on survey?

41 41 The Use of “Don’t Know” and “Not Sure” and “Not Applicable” Rather than have respondents feel forced to give an invalid response, researchers often add a category to scales that is labeled “don’t know” or “not sure” or “not applicable”. Rather than have respondents feel forced to give an invalid response, researchers often add a category to scales that is labeled “don’t know” or “not sure” or “not applicable”. Write a question for your research study that includes a response scale. Then explain why you wrote the question using that response scale. Write a question for your research study that includes a response scale. Then explain why you wrote the question using that response scale.

42 42

43 43 Survey Design and Preparation

44 44 Length of Surveys How long respondents will spend answering a survey is directly related to the saliency of the issues for them. How long respondents will spend answering a survey is directly related to the saliency of the issues for them. For salient topics, questionnaires of pages are possible. For salient topics, questionnaires of pages are possible. For non-salient topics, questionnaires are usually limited to 2 to 4 pages. For non-salient topics, questionnaires are usually limited to 2 to 4 pages. How does this survey rate on these criteria? How does this survey rate on these criteria?

45 45 Introducing the Survey In a cover letter or introductory comments, the following points should be addressed: In a cover letter or introductory comments, the following points should be addressed: What the study is about and its relevance to the participant; What the study is about and its relevance to the participant; Why the respondent’s cooperation is important; Why the respondent’s cooperation is important; Promise of confidentiality and security of information (IRB rules and regulations must be followed); Promise of confidentiality and security of information (IRB rules and regulations must be followed); Reward for participation (if relevant); Reward for participation (if relevant); Deadline for completion (if relevant); Deadline for completion (if relevant); Who to contact regarding any questions; Who to contact regarding any questions; Thank you Thank you Evaluate survey in terms of these criteria Evaluate survey in terms of these criteria

46 46 Pretest, pretest, pretest You should always pretest draft versions of your questions with representatives from the target population. You should always pretest draft versions of your questions with representatives from the target population. But never pretest your questionnaire with people who may be in your final sample of respondents. But never pretest your questionnaire with people who may be in your final sample of respondents. Who might I ask to pretest this survey? Who might I ask to pretest this survey?

47 47 Online survey help sites Advanced Survey Advanced Survey Advanced Survey Advanced Survey Zoomerang Zoomerang Zoomerang Web Online Surveys Web Online Surveys Web Online Surveys Web Online Surveys SurveyMonkey SurveyMonkey SurveyMonkey

48 48 Final Comments on Survey Design Never underestimate the time to takes to create a high quality survey instrument. Never underestimate the time to takes to create a high quality survey instrument. No survey is ever perfect. No survey is ever perfect. There is no substitute for pre-testing and soliciting feedback from other research experts. There is no substitute for pre-testing and soliciting feedback from other research experts. Shorter is always better. Shorter is always better. Only ask questions on surveys that are directly related to specific and important research issues. Only ask questions on surveys that are directly related to specific and important research issues.

49 49 Reflection How might I use questions to collect evidence for my research project? How might I use questions to collect evidence for my research project? What will be most important for me to remember about the questions I want to ask? What will be most important for me to remember about the questions I want to ask?

50 50


Download ppt "1 Gathering SoTL Evidence: Methods for Systematic Inquiry into Student Learning Renee A. Meyers Coordinator, UWS SoTL Leadership Site Faculty College,"

Similar presentations


Ads by Google