Presentation is loading. Please wait.

Presentation is loading. Please wait.

CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN HAYWARD University of Saskatchewan Using Surveys to Maximize.

Similar presentations


Presentation on theme: "CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN HAYWARD University of Saskatchewan Using Surveys to Maximize."— Presentation transcript:

1 CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN HAYWARD University of Saskatchewan Using Surveys to Maximize Strategic Planning

2 Overview Surveys and the Planning Process Understanding Surveys Delivery Methods Closing Tips 23 February 2010 2 Dial and Discuss: Using Surveys

3 Surveys and the Planning Process 23 February 2010 Dial and Discuss: Using Surveys 3

4 Surveys and Planning 23 February 2010 Dial and Discuss: Using Surveys 4 Are surveys needed/useful for planning or do they just get in the way (slow things down)?  Depends on the timing of the planning  Useful for evidence-based decisions  Surveys can be an efficient means of collecting data  Properly done, they can be a valuable tool that increases the chances of success

5 Surveys are useful when… 23 February 2010 Dial and Discuss: Using Surveys 5 information is needed to examine population characteristics a larger response pool is needed examining subgroup data is important (e.g., gender or age differentials) breadth of response is more important than depth

6 Surveys & Evaluations 23 February 2010 Dial and Discuss: Using Surveys 6 Surveys are effective tools with most types of evaluations including:  Needs assessments  Collecting information about the target audience  Formative evaluations (midterm)  Collecting information that helps make adjustments & improvements  Summative evaluations  Collecting information that helps determine if goals and objectives are being met (merit, value, & worth)  Maintenance evaluations  Collecting information a period of time after what is being evaluated is complete, to see if the objectives are still being met

7 Surveys and Strategic Planning 23 February 2010 Dial and Discuss: Using Surveys 7 Surveys can help us: Better understand potential audiences/users/students (consider different segments) Collect baseline data that can be compared with subsequent survey data Test assumptions or preliminary products (trial balloons) Examine existing and potential markets

8 Understanding Surveys… 23 February 2010 Dial and Discuss: Using Surveys 8

9 23 February 2010 Dial and Discuss: Using Surveys 9 What Surveys Can Offer Collect suggestions to improve Catch problems Track trends Get anonymous feedback Get buy-in Gain understanding Increase awareness  2 way process Easy and effective way to collect data With an appropriate sample and response rate quantitative data can help generalize to the population

10 23 February 2010 Dial and Discuss: Using Surveys 10 Strengths of Surveys Easy to create Easy to analyze Relatively inexpensive Common/respondent familiarity Handle both quantitative and qualitative data Can be relatively time efficient to take Real time results Instant Feedback

11 Additional Reasons to Use Surveys 23 February 2010 Dial and Discuss: Using Surveys 11 Can be cost savings, to allow feedback before considerable time and resources spent  Storyboards/scripts Allow feedback while modification is still possible  Helpful to the developer and the user Determine if goals and objectives have been met

12 23 February 2010 Dial and Discuss: Using Surveys 12 Weaknesses of Surveys Lack of follow-up Not best suited for collecting qualitative data Limited flexibility Sampling problems not always obvious Can miss key results  Results are dependent on the questions/options that are presented Garbage In Garbage Out (GIGO)

13 Delivery Methods 23 February 2010 Dial and Discuss: Using Surveys 13

14 Survey Delivery Methods 23 February 2010 Dial and Discuss: Using Surveys 14 Surveys can be conducted using different delivery methods  These methods have different strengths and weaknesses that should be considered  The three different delivery methods considered here are:  Online  Paper-based  Telephone  Currently Online Surveys are probably the most popular method

15 23 February 2010 Dial and Discuss: Using Surveys 15 Online Advantages:  Minimal distribution costs  Automatic data entry  Automatic compilation possible  Branching possible  Potentially quickly modified or updated  Real time data collection potential  Can use multimedia Disadvantages:  SPAM Filters/Broken Links  Duplication a concern  User competency dependent (submit)  More complicated  Anonymity a concern  Potential user cost (Long distance/ISP)  Access

16 Other Online Considerations 23 February 2010 Dial and Discuss: Using Surveys 16 Response rate Length Item types  Forced choices Making changes Navigation  Going to previous pages Submitting survey Accommodating schedules Different browsers  Colors  Size Exactly who is responding Location of the survey server Online survey fatigue

17 23 February 2010 Dial and Discuss: Using Surveys 17 Paper and Pencil Advantages:  Simple (for respondents and evaluator)  No special training required  Easy to go back and change responses  Can have a higher response rate Disadvantages:  Distribution  Can be slow  Branching  Following instructions  Usually required manual data entry  Reading writing  Qualitative responses  Costs

18 23 February 2010 Dial and Discuss: Using Surveys 18 Phone Advantages  Can sometimes provides higher response rates  Ability to branch  Limited ability to follow up response  Allows assessment of response quality  Longer surveys possible Disadvantages  Cost  Interviewer  Long distance  Persistence  More obtrusive  Can be perceived anonymity problems  Selective responses  Caller ID & Voicemail  Telemarketers

19 General Survey Considerations 23 February 2010 Dial and Discuss: Using Surveys 19 Use (needs assessment, formative, or summative) Data entry method Number of response options  5, 7, 99, or unlimited  Even or odd Effective scale Anchors  Continuum versus descriptors Statement questions Normative referenced Validity Reliability

20 Other Considerations 23 February 2010 Dial and Discuss: Using Surveys 20 Completion Time  Written versus telephone  Class versus own time Mixed methods Triangulation Piloting Incentives Routing Funding Bias Terminology Acronyms Reading ability

21 23 February 2010 Dial and Discuss: Using Surveys 21 Other Considerations ( cont’d ) Anonymity  Aggregating data  Removing identifiers  Looking at trends across surveys Differential responses  Only the “keeners” complete or vice-versa  Some other important characteristic determines who responds Sampling Social desirability Time survey is available

22 Bias 23 February 2010 Dial and Discuss: Using Surveys 22 Can affect results, but is often not obvious Question wording  Leading questions Options available  Missing options Question order  Specific to general  Carry over Option order General survey expectations

23 Creating Surveys… Examples 23 February 2010 Dial and Discuss: Using Surveys 23

24 23 February 2010 Dial and Discuss: Using Surveys 24 Survey Creation - Writing Surveys Templates Think of how the information will be used Comparisons between surveys Respondent’s environment Inclusive  Include stakeholders Assumptions Terminology/Acronyms Reading ability

25 23 February 2010 Dial and Discuss: Using Surveys 25 Item Types Likert - Interval data  how many options  even or odd  additional options  direction  anchors I feel very comfortable taking online distance courses. 12345 StronglyDisagreeNeutralAgreeStrongly disagreeagree

26 23 February 2010 Dial and Discuss: Using Surveys 26 End-of-course Evaluations Uses Likert Items Five options Standardized Normative Effective scale Social desirability How would we know when assumptions have been violated? Intuitive meaning

27 23 February 2010 Dial and Discuss: Using Surveys 27 Item Types (continued) Modified Likert  Not agree/disagree items  Continuum (options) changes  Can use fewer anchors  2, 3, or 5 How comfortable do you feel online distance courses? 12345 Not at allSomewhatVery comfortablecomfortablecomfortable

28 23 February 2010 Dial and Discuss: Using Surveys 28 Item Types (continued) Multiple choice  Nominal data  Reporting results  Picking the correct choice  Instructions  Select best  Select all that apply  Missing options Where do you plan to complete most of your online coursework? a )Home b )Work c )Computer labs d )Not sure e )Other (please specify):

29 23 February 2010 Dial and Discuss: Using Surveys 29 Item Types (continued) Rankings  Following instructions  Can force distinctions  Can be harder to interpret results Fill-in-the-blanks  Can be time consuming to code  Generalization Open ended  Use open-ended questions carefully  Exploratory versus confirmatory  Time  Potentially rich

30 Closing Tips 23 February 2010 Dial and Discuss: Using Surveys 30

31 23 February 2010 Dial and Discuss: Using Surveys 31 Survey Writing Tips Demographic information  Focus on what information will be useful for analysis  Recognize many consider the information given to be sensitive  Caution about subject identification Trade-offs  Don’t make responses too easy

32 23 February 2010 Dial and Discuss: Using Surveys 32 Survey Tips ( cont’d ) Instructions  Clarify purpose/reasons  Anonymity  Voluntary  Withdrawal  Ethics  Implied Consent  Dissemination  When and where results will be found

33 23 February 2010 Dial and Discuss: Using Surveys 33 Survey Pitfalls Want to get the largest proportion possible to respond  Reduced the chance of self-selection  Increases likelihood of accurately generalizing results Increasingly difficult to get high response rates  Frequency being surveyed  Survey fatigue  Perceived need/relevance

34 Tips to Increase Response Rates 23 February 2010 Dial and Discuss: Using Surveys 34 Well written cover letter from meaningful individual  Explaining survey purpose and relevance Honorarium – given with survey  Draw (e.g., gift certificate) Keep survey as short as possible Provide reasonable deadline Reminders

35 23 February 2010 Dial and Discuss: Using Surveys 35 Reference Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009) Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method (Third Edition). New York: John Wiley & Son.

36 Questions? 23 February 2010 Dial and Discuss: Using Surveys 36

37 23 February 2010 Dial and Discuss: Using Surveys 37 Contacting Us Stanley Varnhagen  University of Alberta  phone: (780) 492-3641  email: stanley.varnhagen@ualberta.castanley.varnhagen@ualberta.ca Karen Hayward  University of Saskatchewan  phone: (306) 966-7384  email: Karen.Hayward@usask.caKaren.Hayward@usask.ca


Download ppt "CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN HAYWARD University of Saskatchewan Using Surveys to Maximize."

Similar presentations


Ads by Google