Assessing Undergraduate Sustainability Knowledge Campus Wide: Environmental and Social Sustainability Lab Adam Zwickle - OSU Tomas Koontz - OSU Andrew Bodine - OSU Mark Stewart – UMD Nicole Horvath - UMD Assessing Undergraduate Sustainability Knowledge Campus Wide: From design to implementation and analysis

Overview How we developed our Assessment of Sustainability Knowledge (ASK) Why an ASK is important & how it can help An aside on Knowledge and Literacy Conducting an assessment Thinking long term…

Developing an assessment Built upon the “triple bottom line”, the “three legged stool”, the “3 p’s” Environmental (planet) Economic (prosperity) Social (people) Basic knowledge from each knowledge domain? ~OR~ Knowledge bridging the different spheres? “knowledge domains”

Developing an assessment Replicated questions used in the past Coyle, 2005. “Environmental Literacy in America.” Solicited topics and questions from experts Held expert focus groups Pilot tested among professors, graduate, and undergraduate students Narrowed down to 30 questions

Developing an assessment Distributed those 30 to OSU students Used IRT to throw out 14 Added UMD’s 16 Distributed those to OSU and UMD students Used IRT to throw out 2 Current ASK has 28 items: ess.osu.edu

Conceptualizing sustainability knowledge What is the most common cause of pollution of streams and rivers? Surface water running off yards, city streets, paved, lots, and farm fields Conceptualizing sustainability knowledge Which of the following is the most commonly used definition of sustainable development? Meeting the needs of the present without compromising the ability of future generations to meet their own needs Social Environmental Sustainability ADD EXAMPLE QUESTIONS Many economists argue that electricity prices in the U.S. are too low because… They do not reflect the costs of pollution from generating the electricity Economic

How is this helpful?

Need for measuring knowledge University goals- More along the lines of: “Become carbon neutral by 2050” Less common: “Create sustainably minded citizens of tomorrow” Can help track improvement over time By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

Need for measuring knowledge Serves as an evaluation for specific academic efforts: Interdisciplinary programs Sustainability majors/minors STARS Credit: ER 6 (STARS 2.0) – 3 points available By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

“Knowledge” vs. “Literacy” Can be objectively measured Can be used to evaluate academic programs Literacy = Knowledge Knowledge + values, attitudes, and behaviors Can be used to evaluate outreach efforts, campaigns By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

Measuring Behaviors, Values, and Attitudes Make sure you are asking the right questions Collaborate with sustainability departments to target specific behaviors (e.g., leaving lights on) Include questions on behavioral barriers Other behaviors that may get at the same concept By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

Measuring Behaviors, Values, and Attitudes Collaborate with academic departments to develop a good survey design No need to reinvent the wheel Each survey could be a master’s thesis By creating a way to quantify an abstract educational goal such as this, we hope to encourage more institutions to make this a priority.

Conducting an Assessment

Conducting an Assessment Find your partners! IRB approval Required for publication Exempt status Registrar approval* Student’s emails, majors, and demographics Survey software* We use Qualtrics, but there are others *If a large scale assessment is planned

Maximizing Response Rates Maximizing response rates is important to reduce uncertainty about how well your completed sample matches the population of interest. Dillman (2008) and others have long studied how to maximize response rates for surveys that were telephone, mailed, or in-person. There is growing research on electronic survey response rates.

Research Methods We compare response rates from five different survey implementations: 1: 2012 spring OSU (n=10,000) 2: 2013 spring OSU sample A (n=10,000) 3: 2013 spring OSU sample B (n=10,000) 4: 2013 spring OSU School of ENR (n=538) 5: 2013 spring UMD (n=10,000) We tried different treatments and tracked survey responses with survey software (Survey Monkey and Qualtrics)

Key Variables Affecting Response Rates among College Students Timing When to send the invitation and reminders Incentives Questionnaire format Long list of questions vs. more page clicks Email text Who it is from

Timing Matters Time of Semester: Time of Day: Reminders are critical Last week of semester and into finals week Middle of semester Time of Day: 6:00 am 6:00 pm Reminders are critical big spike in completed surveys after each reminder, with a fast decay (see next slide)

Timing Matters Overall, you want students sitting at their computers… but wanting to be distracted

Incentives Some disagreement in research on best way to provide incentives ahead of time (Dillman 2008) randomly select winner of 1 big prize give more/all respondents smaller prizes. May impact the validity of the data

Survey Responses UMD n=1,556 Day OSU n=2,621 Day

Email text How the invitation emails are phrased affects response rates. Besides making the invitation personal, clear, and as short as possible, prior research has found that who the invitation comes from matters. An appeal from a trusted authority increases response rates.

Email text: Appeal from Authority We compared an appeal from authority versus an appeal from a peer (student). Two surveys, A and B, had an appeal from a higher authority (School Director or University VP) for first 3 contacts. For the 4th contact kept the higher authority for A, but switched B to have to an appeal from a grad student

Email text: Appeal from Authority Results: = additional 538 = additional 348 over the next 65 hours

Questionnaire Format Trade-off: We analyzed respondent drop-outs spots Long list of questions to scroll down Shorter lists with a page click to get to the next page We analyzed respondent drop-outs spots Highest spots were just after clicking to the next page We recommend finding a balance…

Non-Respondent Bias Even after efforts to maximize response rates, are the non-respondents different from the respondents? We conducted a non-respondent short survey (5 questions + some demographics). Results indicate that non-respondents are slightly but significantly less knowledgeable about sustainability topics, but no difference in GPA or pro-environmental behaviors.

Planning ahead… For longitudinal studies: Write a multiple year IRB Let registrar know this is a yearly survey Partnering up with an academic department for help Sustainability office Academic departments with survey expertise (interdisciplinary social science, communications, sociology, political science, psychology)

Acknowledgements Funded by: & Natural Resources The Ohio State Office of Sustainability http://sustainability.osu.edu/ OSU’s School of Environment & Natural Resources http://senr.osu.edu/

Thank You! Questions? Environment and Social Sustainability Lab Email ess.osu.edu Contains: This presentation The 28 question ASK Forthcoming article Email Zwickle.1@osu.edu Koontz.31@osu.edu

Example Analysis Compare sub score by major Economic score Aerospace engineering: 4.00 Animal science: 3.15 Finance: 3.02 English: 2.80 Accounting: 2.63 Best to worst

Phase 2 Combined our shortened set with a separate assessment developed at the University of Maryland Administered survey at both campuses

Phase 2 - Analysis IRT to compare information richness of questions across campuses using local descriptives (“Fishermen on Lake Erie” vs. “Fishermen in the Chesapeake Bay”) different ways of asking about the same topic

Survey Implementation Best practices to maximize response rates: Invitation script (from authority vs. student) Page length vs. number of pages Use of incentives Time of academic calendar Time of day, day of week Reminders

Results Total Environmental Social Economic 73% Mean: 11.08/16 6 questions 5 questions Mean: 4.39/6 Mean: 3.55/5 Mean: 3.03/5 SD: 1.48 SD: 1.23 SD: 1.27 73% 71% 61% Average scores % Total 16 questions Mean: 11.08/16 SD: 3.21 69%

Example Analysis ANOVA F(3,1330)=8.09, p<.001 Sig differences Display graphs Bar graphs Flag “Out of 16” ANOVA F(3,1330)=8.09, p<.001

Moving Forward Adjust the instrument: Be clear about limitations: Some answers change over time Develop test bank of validated questions – try this on your campus and send us questions Be clear about limitations: Multiple choice format Measures knowledge only Publish Phase 1 – International Journal of Sustainability in Higher Education, 2014 Discussion: Weakness: some answers change over time, should aim for static knowledge concepts. Give us your questions

Moving Forward Goals: Hopes: Evaluate sustainability themed majors, programs, and courses Serve as an indicator for comparison within institutions Encourage friendly competition Hopes: Help solidify the concept of “sustainability” Provide some credence for more abstract educational goals

Appendix Phase 1 -16 Question Sustainability Literacy Assessment

Questions - Environmental What is the most common cause of pollution of streams and rivers? Dumping of garbage by cities Surface water running off yards, city streets, paved lots, and farm fields Litter near streams and rivers Waste dumped by factories Don't know Ozone forms a protective layer in the earth's upper atmosphere. What does ozone protect us from? Acid rain Climate change Sudden changes in temperature Harmful UV rays

Questions - Environmental What is the name of the primary federal agency that oversees environmental regulation? Environmental Protection Agency (the EPA) Department of Health, Environment, and Safety (the DHES) National Environmental Agency (the NEA) Federal Pollution Control Agency (the FPCA) Don't know What is the primary benefit of wetlands? Promote flooding Clean the water before it enters lakes, streams, rivers, or oceans Keep the number of undesirable plants and animals low Provide good sites for landfills

Questions - Environmental Which of the following is an example of sustainable forest management? Setting aside forests to be off limits to the public Never harvesting more than what the forest produces in new  growth Producing lumber for nearby communities to build affordable housing Putting the local communities in charge of forest resources Don’t know In the U.S., what do we currently do with the nuclear waste generated by nuclear power plants? Use it as nuclear fuel Sell it to other countries Dump it in landfills Store and monitor the waste Don't know

Questions - Social Which of the following is the most commonly used definition of sustainable development? Creating a government welfare system that ensures universal  access to education, health­care, and social services Meeting the needs of the present without compromising the ability of future generations to meet their own needs Setting aside resources for preservation, never to be used Building a neighborhood that is both socio-demographically  and economically diverse Don’t know The wealthiest 20% of people in the U.S. own approximately what percent of the nation’s privately held wealth? 20% 35% 50% 85% Don't know

Questions - Social Over the past 3 decades, what has happened to the difference between the wealth of the richest and poorest Americans? The difference has increased The difference has stayed about the same The difference has decreased Don’t know Higher levels of education generally lead to... Lower levels of voter turnout Greater annual earnings Larger family size Higher self esteem

Questions - Social Which of the following populations has the highest rate of growth? North America Europe China Africa Don’t know

Questions - Economic Many economists argue that electricity prices in the U.S. are too low because… They do not reflect the costs of pollution from generating the  electricity Too many suppliers go out of business Electric companies have a monopoly in their service area Consumers spend only a small part of their income on energy Don’t know Which of the following countries has now passed the U.S. as the biggest emitter of the greenhouse gas carbon dioxide? China Sweden Brazil Japan

Questions - Economic Which of the following is a leading cause of the depletion of fish stocks in the Atlantic Ocean? Fishermen seeking to maximize their catch Reduced fish fertility due to genetic hybridization Ocean pollution Global climate change Don’t know Which of the following is the most commonly used definition of economic sustainability? Maximizing the share price of a company's stock Long term profitability When costs equal revenue Continually expanding market share

Questions - Economic Which of the following is the primary reason that gasoline prices have risen over the last several decades in the U.S.? Growing percentage of gas stations owned by large  corporations Increasing oil discoveries overseas Higher rates of state and federal gasoline tax Increasing global demand for oil Don’t know