Building a Better Survey Katie Busby, PhD Director of Student Affairs Assessment & Planning University of Alabama Keston Fulcher, PhD Director of Assessment, Evaluation and Accreditation Christopher Newport University International Assessment & Retention Conference 2007

Survey Please take a few minutes to complete this survey. It has some errors, note them when you read.

Overview of Session Introduce the assessment cycle as a foundation upon which to build assessment efforts Complete an exercise that will identify strong and weak components of a survey Discuss, in detail, the steps that are necessary to develop a good survey instrument Share practical techniques that can be applied to overcome common missteps in survey development

Definition of Assessment The systematic process of determining educational objectives, gathering, using, and analyzing information about student learning outcomes to make decisions about programs, individual student progress, or accountability - T. Dary Erwin (1991)

Assessment Model Conceptualize and Establish Intended Learning Outcomes Select Assessment Methods Collect Data Report and Use Results Analyze and Interpret Select Assessment Methods

Rules for Developing Survey Avoiding the don’ts and doing the do’s

Rules for Developing Surveys Be sure a survey is the way to go –Is there existing data you can use? –Is the survey the best method of collecting data? –Is there an existing survey you can use? Design your assessment project first –Identify your research questions –Determine what you want to know –Clarify your goals

Rules for Developing Surveys Survey development is a team sport –Include other experts in the process –Involve relevant decision makers –Combine efforts with others who are developing a survey instrument

Rules for Developing Surveys You never get a second chance to make a first impression –Keep survey as short as possible –Have a logical flow to questions –Ensure a good layout design –Requests for participation, title, and instructions should be clear and concise

Rules for Developing Surveys It’s all about the respondent –Keep the respondent in mind during the development phase –Make it easy for the respondent to respond –Consider offering incentives –Pilot the survey

Specific Hint 1 Keep sensitive items toward the end of survey (i.e., warm up the responder). Note: Do you need IRB approval?

Specific Hint 2 If intended, make sure response options are mutually exclusive. Bad: –0-15k –15k-30k Better: –Under $15,000 per year –$15,001 - $30,000 per year

Specific Hint 3 Group items with similar content and/or similar item types together. Notice that Item 2 and Item 4 are needlessly separated on Bogus Survey.

Specific Hint 4 “Don’t Know” or “Not Applicable” options should be included at the end of or aside from other response options (READ: NOT IN THE MIDDLE)

Specific Hint 5 Avoid Double-Barreled Items Bad: –(4) I enjoy working in the field of assessment and analyzing data. Better –(4) I enjoy working in the field of assessment. –(5) I enjoy analyzing data.

Specific Hint 6 Response options should accommodate all reasonable responses. On Bogus Survey, response options to Item 5 do not cover all responses.

Specific Hint 7 Attempt to maximize variability with responses For Item 6, the vast majority of attendees would mark the middle option.

Specific Hint 8 Proofread As with any document, errors connote a lack of professionalism.

Questions about survey development What challenges have you faced? How can we improve our “survey”?

Contact us Katie Busby, PhD Director, Student Affairs Assessment and Planning University of Alabama Box Tuscaloosa, AL Keston H. Fulcher, PhD Director of Assessment, Evaluation, and Accreditation Christopher Newport University McMurran Building Annex, Room 101-A 1 University Place Newport News, Virginia (757)

Resources for Survey Design Alreck, P. L., & Settle, R. B. (1995). The survey research handbook, 3rd edition. New York: McGraw Hill. Bradburn, N. M., Sudman, S., & Wansink, B. (1996). Asking questions: The definitive guide to questionnaire design – for market research, political polls, and social and health questionnaires. San Francisco: Jossey Bass.. Dillman, D. A. (2006). Mail and Internet surveys: The tailored design method 2007 update with new Internet, visual, and mixed-mode guide, 2nd edition. New York: John Wiley & Sons. Rea, L. M., & Parker, R. A. (1997). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass. Sudman, S., Bradburn, N. M., & Schwartz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey Bass.