4Methods for Gathering Evidence Evidence can include:Course-Intrinsic EvidenceCourse-Extrinsic EvidenceUsing your research question, what is one type of course-intrinsic and one type of course-extrinsic evidence you might collect?
5Asking QuestionsAll SoTL evidence gathering is focused on asking questions of students.Research on writing survey questions provides comprehensive information on how to do this best
6Survey Evaluation Look at survey in packet. What works? What doesn’t work?What is missing?
7What Survey Research Tells Us About Asking Good Questions
8The Impact of Questions on the Research Process Poor questions lead to lousy results.How might answers to these questions differ?Do you attend class?Do you regularly attend class?How many times per week do you attend this class?Asking survey questions is part science and part art.
9Types of Survey Questions Open Ended and Closed Ended Question FormatsAttitude, Knowledge, and Behavior Question Formats
10Open-Ended QuestionsOpen-ended questions ask respondents to state in their own words their response to the question.Closed-ended (or fixed alternative) questions ask respondents to respond by picking one or more alternatives from a list of available options.Can you find any examples of open-ended questions on the survey? Are these good open-ended questions?
11Open Ended QuestionsWhat are advantages and disadvantages to open- and close-ended questions?
12Advantages and Disadvantages Are often less time consumingMay provide more complete answersMay provide information researchers did not anticipateResponses are often easier to analyzeMay produce better response rate
13Surveys: Asking Questions Surveys are used most often in SoTL research when we are interested in asking questions about:how students FEEL or THINK about certain issues, topics, activities, related to their learning,what students KNOW about various topics, etc.how often students engage in certain learning BEHAVIORS
14Attitude/Feelings Questions Attitude/feelings questions ask respondents for their views on certain topics or issuesThere are no right or wrong answers to attitude/feeling questions.Attitude/feeling questions should be “neutrally” written so that the respondent is not led to believe that a certain response is desired.
15Knowledge QuestionsKnowledge questions ask about what a respondent knows about a particular topicKnowledge questions can be asked as a screening question to determine if respondents are qualified to answer attitude questions.“What do you know about evolution?“How do you feel about evolution as a theory of creation?”
16Behavior QuestionsBehavior questions ask respondents to report on their past or future behaviorResponses to behavior questions may be biased because of faulty or biased recall by respondents
17Attitude, Knowledge, and Behavior Questions Can you find an attitude/feelings, knowledge, and behavior question on the survey?Write one attitude/feelings, knowledge, or behavior question for your own SoTL project.Pre-assessment measureQuestion for an assignment you will collectPost-assessment
19Recall QuestionsIs there a question on survey that asks students to “recall” some behavior?How would you evaluate these questions?Would you know how to answer them?Is there a better way to ask these questions?
20Using Diaries for Recall Questions To minimize recall errors, researchers can ask respondents to keep personal diaries to record their actual teaching or learning activities over a period of timeCaution: Respondents may forget to fill out their diaries on a timely basis and instead “catch up” their diaries just before they are due to be collected
21Unaided vs. Aided Recall Questions Because respondents often forget, or fail to recall accurately, their activities and behavior, you can use aided recall questions to “jog” their memories.Unaided recall: “Please list all of the Communication classes in which you participated in learning groups last semester.” (open-ended question)Aided recall: “Please place a check by any of the Communication courses in which you participated in a learning group last semester.” (closed-ended question)Public SpeakingIntroduction to Interpersonal CommunicationSmall Group Communication
22Recall Questions and Time Frame We often want respondents to focus on a specific time frame when answering a question.Avoid ambiguous terms like “past year” or “next year” because respondents may not know if this is a calendar year (i.e., 2006, 2007, 2008) or a 12-month period.Avoid time frames that are too large (for the recall question) or too far in the past.Write one recall question for gathering evidence for your SoTL research question.
23Intimidating or Inappropriate Questions Respondents will consider a question to be intimidating or inappropriate if:It addresses highly personal and private behavior (drinking behavior, sexual activity, illegal drug use, etc.)It addresses subjects for which the respondent believes there are socially desirable norms (communication with instructors, behavior in class, spiritual behavior, etc.)If respondents perceive a question to be intimidating, they are more likely to refuse to answer or to give an inaccurate answer.
24Intimidating Questions Would you use an open- or closed-ended question to ask a potentially intimidating question to gather evidence from your students?Examples?
25Forms for Asking Potentially Intimidating Questions About Behavior If you ask a closed-ended question with a set of response options, people typically do not choose the extreme options.Therefore, it is often better to use an open-ended question and allow the respondents to indicate how often they engage in certain behaviors, or define the terms as they perceive them.
26Forms for Asking Potentially Intimidating Questions About Behavior Third person questions ask respondents to report on the behavior and activities of other people.Example: Instead of asking college students directly about their alcohol consumption behavior and its impact on their learning, you could ask students to report how much alcohol their roommates consume and how that affects their learning behavior.
27Double-Barreled Questions Double-barreled questions ask respondents for their opinions about two distinct issues in the same question.“Do you prefer working in groups and learning from your fellow students, or do you prefer lectures and learning from the instructor?Any other examples?
28Leading and Loaded Questions Some “surveys” seek to generate results that support a set of predetermined goals and objectives.How might we re-write a question on the survey to make it a leading question?
30Ordering of QuestionsStart with easy, salient, and non-threatening questions first.Complete questions on a single topic before moving on to a new topic.When switching topics, use transitional phrases to make it easier for respondents to switch their train of thought.Put demographic questions near the end.Do a quick evaluation of survey to see how it fits criteria.
31Funnel QuestionsQuestions regarding a specific topic should usually start with the most general questions and then follow with increasingly more restrictive questions.If you were required to work in groups in your class, what types of students would you want in your group?If you had to select just three key characteristics that you would use to choose among these different students, what would these 3 factors be?Would “student’s GPA” be an important factor in your choice of students for your work group?If your instructor indicated that she was going to put students in groups according to similar GPAs, would you be interested in working in such a group?
32Skipping and Branching Surveys can instruct respondents to skip over certain questions if these questions do not pertain to them.Question 7: “Have you worked in a classroom learning group in the past semester?”____ Yes (please proceed to question 8)____ No (please skip to question 12)Any examples on survey?
33Demographic Questions In most surveys, questions regarding demographic characteristics of the respondents are included at end.Typical demographic questions:SexEthnicity/Racial backgroundAgeYear in SchoolMajorAre there better ways to write these demographic questions on the survey?
35Avoid Ambiguity in Wording Some common words, such as “regularly” or “often”, may be interpreted very differently by respondents.Do you go to talk to your instructor regularly (outside of class)?How often do you go talk to your instructor (outside of class):1-2 times a week3-4 times a week5-7 times a weekMore than 7 times a weekWhat might be the issue with these questions?
37Features of Response Categories Any response category must be mutually exclusive.There should not be any overlap in the categories.Any response category system should be collectively exhaustive.All possible responses should be included in the category system.Usually it is advisable to include an “other” category to allow respondents to add a response that is not included in the category system.On survey, evaluate questions for exclusivity and collective exhaustion. Any issues?Could respond more than once a day; need an ‘other’ category
38Semantic Differential Scales A scale, typically used to measure attitudes or perceptions, that includes end points that are anchored by bipolar adjectives.Outstanding _ _ _ _ _ _ _ UnacceptableStrong _ _ _ _ _ _ _ WeakMoral _ _ _ _ _ _ _ ImmoralAny examples of a semantic differential scale on survey?Question 4, 7
39Likert ScalesA scale, typically used to measure attitudes, that usually includes end points of “strongly agree” and “strongly disagree.”Abiding by page limits is important for writing a good paper:Strongly AgreeAgreeNeutralDisagreeStrongly DisagreeAny examples of a Likert scale on the survey?Aren’t any All are numbered
40Numerical ScalesA scale that uses multiple numbers, usually 5 to 10, with anchored end points to represent the strength or position of respondents’ attitudes.1 strongly disagree2 disagree3 neutral4 agree5 strongly agreeAny examples of a numerical scale on survey?
41The Use of “Don’t Know” and “Not Sure” and “Not Applicable” Rather than have respondents feel forced to give an invalid response, researchers often add a category to scales that is labeled “don’t know” or “not sure” or “not applicable”.Write a question for your research study that includes a response scale. Then explain why you wrote the question using that response scale.
44Length of SurveysHow long respondents will spend answering a survey is directly related to the saliency of the issues for them.For salient topics, questionnaires of pages are possible.For non-salient topics, questionnaires are usually limited to 2 to 4 pages.How does this survey rate on these criteria?
45Introducing the Survey In a cover letter or introductory comments, the following points should be addressed:What the study is about and its relevance to the participant;Why the respondent’s cooperation is important;Promise of confidentiality and security of information (IRB rules and regulations must be followed);Reward for participation (if relevant);Deadline for completion (if relevant);Who to contact regarding any questions;Thank youEvaluate survey in terms of these criteria
46Pretest, pretest, pretest You should always pretest draft versions of your questions with representatives from the target population.But never pretest your questionnaire with people who may be in your final sample of respondents.Who might I ask to pretest this survey?
47Online survey help sites Advanced SurveyZoomerangWeb Online SurveysSurveyMonkey
48Final Comments on Survey Design Never underestimate the time to takes to create a high quality survey instrument.No survey is ever perfect.There is no substitute for pre-testing and soliciting feedback from other research experts.Shorter is always better.Only ask questions on surveys that are directly related to specific and important research issues.
49ReflectionHow might I use questions to collect evidence for my research project?What will be most important for me to remember about the questions I want to ask?