1 Assessing Undergraduate Sustainability Knowledge Campus Wide: Environmental and Social Sustainability LabAdam Zwickle - OSUTomas Koontz - OSUAndrew Bodine - OSUMark Stewart – UMDNicole Horvath - UMDAssessing Undergraduate Sustainability Knowledge Campus Wide:From design to implementation and analysis
2 OverviewHow we developed our Assessment of Sustainability Knowledge (ASK)Why an ASK is important & how it can helpAn aside on Knowledge and LiteracyConducting an assessmentThinking long term…
3 Developing an assessment Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s”Environmental (planet)Economic (prosperity)Social (people)Basic knowledge from each knowledge domain?~OR~Knowledge bridging the different spheres?“knowledge domains”
4 Developing an assessment Replicated questions used in the pastCoyle, “Environmental Literacy in America.”Solicited topics and questions from expertsHeld expert focus groupsPilot tested among professors, graduate, and undergraduate studentsNarrowed down to 30 questions
5 Developing an assessment Distributed those 30 to OSU studentsUsed IRT to throw out 14Added UMD’s 16Distributed those to OSU and UMD studentsUsed IRT to throw out 2Current ASK has 28 items:ess.osu.edu
6 Conceptualizing sustainability knowledge What is the most common cause of pollution of streams and rivers?Surface water running off yards, city streets, paved, lots, and farm fieldsConceptualizing sustainability knowledgeWhich of the following is the most commonly used definition of sustainable development?Meeting the needs of the present without compromising the ability of future generations to meet their own needsSocialEnvironmentalSustainabilityADD EXAMPLE QUESTIONSMany economists argue that electricity prices in the U.S. are too low because…They do not reflect the costs of pollution from generating the electricityEconomic
8 Need for measuring knowledge University goals-More along the lines of:“Become carbon neutral by 2050”Less common:“Create sustainably minded citizens of tomorrow”Can help track improvement over timeBy creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.
9 Need for measuring knowledge Serves as an evaluation for specific academic efforts:Interdisciplinary programsSustainability majors/minorsSTARS Credit:ER 6 (STARS 2.0) – 3 points availableBy creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.
10 “Knowledge” vs. “Literacy” Can be objectively measuredCan be used to evaluate academic programsLiteracy= KnowledgeKnowledge + values, attitudes, and behaviorsCan be used to evaluate outreach efforts, campaignsBy creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.
11 Measuring Behaviors, Values, and Attitudes Make sure you are asking the right questionsCollaborate with sustainability departments to target specific behaviors (e.g., leaving lights on)Include questions on behavioral barriersOther behaviors that may get at the same conceptBy creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.
12 Measuring Behaviors, Values, and Attitudes Collaborate with academic departments to develop a good survey designNo need to reinvent the wheelEach survey could be a master’s thesisBy creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.
14 Conducting an Assessment Find your partners!IRB approvalRequired for publicationExempt statusRegistrar approval*Student’s s, majors, and demographicsSurvey software*We use Qualtrics, but there are others*If a large scale assessment is planned
15 Maximizing Response Rates Maximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest.Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person.There is growing research on electronic survey response rates.
16 Research MethodsWe compare response rates from five different survey implementations:1: 2012 spring OSU (n=10,000)2: 2013 spring OSU sample A (n=10,000)3: 2013 spring OSU sample B (n=10,000)4: 2013 spring OSU School of ENR (n=538)5: 2013 spring UMD (n=10,000)We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)
17 Key Variables Affecting Response Rates among College Students TimingWhen to send the invitation and remindersIncentivesQuestionnaire formatLong list of questions vs. more page clickstextWho it is from
18 Timing Matters Time of Semester: Time of Day: Reminders are critical Last week of semester and into finals weekMiddle of semesterTime of Day:6:00 am6:00 pmReminders are criticalbig spike in completed surveys after each reminder, with a fast decay (see next slide)
19 Timing Matters Overall, you want students sitting at their computers… but wanting to be distracted
20 IncentivesSome disagreement in research on best way to provide incentivesahead of time (Dillman 2008)randomly select winner of 1 big prizegive more/all respondents smaller prizes.May impact the validity of the data
22 textHow the invitation s are phrased affects response rates.Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters.An appeal from a trusted authority increases response rates.
23 Email text: Appeal from Authority We compared an appeal from authority versus an appeal from a peer (student).Two surveys, A and B, had an appeal from a higher authority (School Director or University VP) for first 3 contacts.For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student
24 Email text: Appeal from Authority Results:= additional 538= additional 348over the next 65 hours
25 Questionnaire Format Trade-off: We analyzed respondent drop-outs spots Long list of questions to scroll downShorter lists with a page click to get to the next pageWe analyzed respondent drop-outs spotsHighest spots were just after clicking to the next pageWe recommend finding a balance…
26 Non-Respondent BiasEven after efforts to maximize response rates, are the non-respondents different from the respondents?We conducted a non-respondent short survey (5 questions + some demographics).Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.
27 Planning ahead… For longitudinal studies: Write a multiple year IRB Let registrar know this is a yearly surveyPartnering up with an academic department for helpSustainability officeAcademic departments with survey expertise (interdisciplinary social science, communications, sociology, political science, psychology)
28 Acknowledgements Funded by: & Natural Resources The Ohio State Office of SustainabilityOSU’s School of Environment& Natural Resources
29 Thank You! Questions? Environment and Social Sustainability Lab Email ess.osu.eduContains:This presentationThe 28 question ASKForthcoming article
31 Example Analysis Compare sub score by major Economic score Aerospace engineering: 4.00Animal science: 3.15Finance: 3.02English: 2.80Accounting: 2.63Best to worst
32 Phase 2Combined our shortened set with a separate assessment developed at the University of MarylandAdministered survey at both campuses
33 Phase 2 - Analysis IRT to compare information richness of questions across campusesusing local descriptives (“Fishermen on Lake Erie” vs. “Fishermen in the Chesapeake Bay”)different ways of asking about the same topic
34 Survey Implementation Best practices to maximize response rates:Invitation script (from authority vs. student)Page length vs. number of pagesUse of incentivesTime of academic calendarTime of day, day of weekReminders
36 Example Analysis ANOVA F(3,1330)=8.09, p<.001 Sig differences Display graphsBar graphsFlag“Out of 16”ANOVAF(3,1330)=8.09, p<.001
37 Moving Forward Adjust the instrument: Be clear about limitations: Some answers change over timeDevelop test bank of validated questions – try this on your campus and send us questionsBe clear about limitations:Multiple choice formatMeasures knowledge onlyPublishPhase 1 – International Journal of Sustainability in Higher Education, 2014Discussion:Weakness: some answers change over time, should aim for static knowledge concepts.Give us your questions
38 Moving Forward Goals: Hopes: Evaluate sustainability themed majors, programs, and coursesServe as an indicator for comparison within institutionsEncourage friendly competitionHopes:Help solidify the concept of “sustainability”Provide some credence for more abstract educational goals
40 Questions - Environmental What is the most common cause of pollution of streams and rivers?Dumping of garbage by citiesSurface water running off yards, city streets, paved lots, and farm fieldsLitter near streams and riversWaste dumped by factoriesDon't knowOzone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from?Acid rainClimate changeSudden changes in temperatureHarmful UV rays
41 Questions - Environmental What is the name of the primary federal agency that oversees environmental regulation?Environmental Protection Agency (the EPA)Department of Health, Environment, and Safety (the DHES)National Environmental Agency (the NEA)Federal Pollution Control Agency (the FPCA)Don't knowWhat is the primary benefit of wetlands?Promote floodingClean the water before it enters lakes, streams, rivers, or oceansKeep the number of undesirable plants and animals lowProvide good sites for landfills
42 Questions - Environmental Which of the following is an example of sustainable forest management?Setting aside forests to be off limits to the publicNever harvesting more than what the forest produces in new growthProducing lumber for nearby communities to build affordable housingPutting the local communities in charge of forest resourcesDon’t knowIn the U.S., what do we currently do with the nuclear waste generated by nuclear power plants?Use it as nuclear fuelSell it to other countriesDump it in landfillsStore and monitor the wasteDon't know
43 Questions - SocialWhich of the following is the most commonly used definition of sustainable development?Creating a government welfare system that ensures universal access to education, healthcare, and social servicesMeeting the needs of the present without compromising the ability of future generations to meet their own needsSetting aside resources for preservation, never to be usedBuilding a neighborhood that is both socio-demographically and economically diverseDon’t knowThe wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth?20%35%50%85%Don't know
44 Questions - SocialOver the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans?The difference has increasedThe difference has stayed about the sameThe difference has decreasedDon’t knowHigher levels of education generally lead to...Lower levels of voter turnoutGreater annual earningsLarger family sizeHigher self esteem
45 Questions - SocialWhich of the following populations has the highest rate of growth?North AmericaEuropeChinaAfricaDon’t know
46 Questions - EconomicMany economists argue that electricity prices in the U.S. are too low because…They do not reflect the costs of pollution from generating the electricityToo many suppliers go out of businessElectric companies have a monopoly in their service areaConsumers spend only a small part of their income on energyDon’t knowWhich of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide?ChinaSwedenBrazilJapan
47 Questions - EconomicWhich of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean?Fishermen seeking to maximize their catchReduced fish fertility due to genetic hybridizationOcean pollutionGlobal climate changeDon’t knowWhich of the following is the most commonly used definition of economic sustainability?Maximizing the share price of a company's stockLong term profitabilityWhen costs equal revenueContinually expanding market share
48 Questions - EconomicWhich of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.?Growing percentage of gas stations owned by large corporationsIncreasing oil discoveries overseasHigher rates of state and federal gasoline taxIncreasing global demand for oilDon’t know