1

 Most of the information presented in this workshop represents the presenters’ opinions and not an official NSF position  Local facilitators will provide the link to the workshop slides at the completion of the webinar.  Participants may ask questions by “raising their virtual hand” during a question session. We will call on selected sites and enable their microphone so that the question can be asked.  Responses will be collected from a few sites at the end of each Exercise. At the start of the Exercise, we will identify these sites in the Chat Box and then call on them one at a time to provide their responses. 2

 Learning must build on prior knowledge ◦ Some knowledge correct ◦ Some knowledge incorrect – Misconceptions  Learning is ◦ Connecting new knowledge to prior knowledge ◦ Correcting misconceptions  Learning requires engagement ◦ Actively recalling prior knowledge ◦ Sharing new knowledge ◦ Forming a new understanding

 Effective learning activities ◦ Recall prior knowledge -- actively, explicitly ◦ Connect new concepts to existing ones ◦ Challenge and alter misconceptions  Active & collaborative processes ◦ Think individually ◦ Share with partner ◦ Report to local and virtual groups ◦ Learn from program directors’ responses 4

 Coordinate the local activities  Watch the time ◦ Allow for think, share, and report phases ◦ Reconvene on time -- 1 min warning slide  Ensure the individual think phase is devoted to thinking and not talking  Coordinate the asking of questions by local participants and reporting local responses to exercises 5

 Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Short Exercise min ◦ Think individually ~2 min ◦ Report in local group ---- ~2 min  Individual Exercise min  Questions min  Reports to Virtual Group min 6

The session will enable you to collaborate more effectively with evaluation experts in preparing credible and comprehensive project evaluation plans …. it will not make you an evaluation expert.

After the session, participants should be able to:  Discuss the importance of goals, outcomes, and questions in the evaluation process ◦ Cognitive and affective outcomes  Describe several types of evaluation tools ◦ Advantages, limitations, and appropriateness  Discuss data interpretation issues ◦ Variability, alternative explanations  Develop an evaluation plan in collaboration with an evaluator ◦ Outline a first draft of an evaluation plan

 The terms evaluation and assessment have many meanings ◦ One definition  Assessment is gathering evidence  Evaluation is interpreting data and making value judgments  Examples of evaluation and assessment ◦ Individual’s performance (grading) ◦ Program’s effectiveness (ABET and regional accreditation) ◦ Project’s progress and success (monitoring and validating)  Session addresses project evaluation ◦ May involve evaluating individual and group performance – but in the context of the project  Project evaluation ◦ Formative – monitoring progress to improve approach ◦ Summative – characterizing and documenting final accomplishments

Project Goals, Expected Outcomes, and Evaluation Questions

 Effective evaluation starts with carefully defined project goals and expected outcomes  Goals and expected outcomes related to: ◦ Project management  Initiating or completing an activity  Finishing a “product” ◦ Student behavior  Modifying a learning outcome  Modifying an attitude or a perception

 Goals provide overarching statements of project intention What is your overall ambition? What do you hope to achieve?  Expected outcomes identify specific observable or measureable results for each goal How will achieving your “intention” be reflected by changes in student behavior? How will it change their learning and their attitudes?

 Goals → Expected outcomes  Expected outcomes → Evaluation questions  Questions form the basis of the evaluation process  The evaluation process consists of the collection and interpretation of data to answer evaluation questions

 Read the abstract -- Goal statement removed  Suggest two plausible goals ◦ One on student learning  Cognitive behavior ◦ One on some other aspect of student behavior  Affective behavior  Focus on what will happen to the students ◦ Do not focus on what the instructor will do  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion, Selected local facilitators report to virtual group

The goal of the project is …… The project is developing computer-based instructional modules for statics and mechanics of materials. The project uses 3D rendering and animation software, in which the user manipulates virtual 3D objects in much the same manner as they would physical objects. Tools being developed enable instructors to realistically include external forces and internal reactions on 3D objects as topics are being explained during lectures. Exercises are being developed for students to be able to communicate with peers and instructors through real-time voice and text interactions. The project is being evaluated by … The project is being disseminated through … The broader impacts of the project are … Non engineers should substitute: “Organic chemistry” for “statics and mechanics of materials” “Interactions” for “external forces and internal reactions”

One Minute

GOAL: To improve conceptual understanding and processing skills  In the context of course ◦ Draw free-body diagrams for textbook problems ◦ Solve 3-D textbook problems ◦ Describe the effect(s) of external forces on a solid object orally  In a broader context ◦ Solve out-of-context problems ◦ Visualize 3-D problems ◦ Communicate technical problems orally ◦ Improve critical thinking skills ◦ Enhance intellectual development

GOAL: To improve ◦ Self- confidence ◦ Attitude about engineering as a career

 Write one expected measurable outcome for each of the following goals: ◦ Improve the students’ understanding of the fundamental concepts in statics (cognitive) ◦ Improve the students’ self confidence (affective)  Individual exercise ~ 2 minutes ◦ Individually write a response

One Minute

Understanding of the fundamentals ◦ Students will be better able to:  Describe all parameters, variable, and elemental relationships  Describe the governing laws  Describe the effects of changing some variable in a simple problem  Changes in the frictional force on a block when the angle of an inclined plane changes  Changes in the forces in the members of a simple three element truss when the connecting angles change Self-Confidence ◦ Students will:  Do more of the homework  Have less test anxiety  Express more confidence in their solutions  Be more willing to discuss their solutions

 Write an evaluation question for these expected measurable outcomes:  Understanding of the fundamentals ◦ Students will be better able to describe the effects of changing some variable in a simple problem Self-Confidence ◦ Students will express more confidence in their solutions  Individually identify a question for each  Report to the group

One Minute

Understanding of the fundamentals ◦ Are the students better able to describe the effects of changing some variable in a simple problem ◦ Are the students better able to describe the effects of changing some variable in a simple problem as a result of the intervention Self-Confidence ◦ Do the students express more confidence in their solutions ◦ Do the students express more confidence in their solutions as a result of the intervention

Tools for Evaluating Learning Outcomes

 Surveys ◦ Forced choice or open-ended responses  Concept Inventories ◦ Multiple-choice questions to measure conceptual understanding  Rubrics for analyzing student products ◦ Guides for scoring student reports, tests, etc.  Interviews ◦ Structured (fixed questions) or in-depth (free flowing)  Focus groups ◦ Like interviews but with group interaction  Observations ◦ Actually monitor and evaluate behavior Olds et al, JEE 94:13, 2005 NSF’s Evaluation Handbook

Surveys  Efficient  Accuracy depends on subject’s honesty  Difficult to develop reliable and valid survey  Low response rate threatens reliability, validity & interpretation Observations  Time & labor intensive  Inter-rater reliability must be established  Captures behavior that subjects are unlikely to report  Useful for observable behavior Olds et al, JEE 94:13, 2005

 Use interviews to answer these questions: ◦ What does program look and feel like? ◦ What do stakeholders know about the project? ◦ What are stakeholders’ and participants’ expectations? ◦ What features are most salient? ◦ What changes do participants perceive in themselves? The 2002 User Friendly Handbook for Project Evaluation, NSF publication REC

 Originated in physics -- Force Concept Inventory (FCI)  Several are being developed in engineering fields  Series of multiple choice questions ◦ Questions involve single concept  Formulas, calculations or problem solving skills not required ◦ Possible answers include detractors  Common errors -- misconceptions  Developing CI is involved ◦ Identify misconceptions and detractors ◦ Develop, test, and refine questions ◦ Establish validity and reliability of tool ◦ Language is a major issue

 Pittsburgh Freshman Engineering Survey ◦ Questions about perception  Confidence in their skills in chemistry, communications, engineering, etc.  Impressions about engineering as a precise science, as a lucrative profession, etc.  Validated using alternate approaches: ◦ Item analysis ◦ Verbal protocol elicitation ◦ Factor analysis  Compared results for students who stayed in engineering to those who left Besterfield-Sacre et al, JEE 86:37, 1997

 Levels of Intellectual Development ◦ Students see knowledge, beliefs, and authority in different ways  “ Knowledge is absolute” versus “Knowledge is contextual”  Tools ◦ Measure of Intellectual Development (MID) ◦ Measure of Epistemological Reflection (MER) ◦ Learning Environment Preferences (LEP) Felder et al, JEE 94:57, 2005

 Suppose you where considering an existing tool (e. g., a concept inventory) for use in your project’s evaluation of learning outcomes  What questions would you consider in deciding if the tool is appropriate?  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

One Minute

 Nature of the tool ◦ Is the tool relevant to what was taught? ◦ Is the tool competency based? ◦ Is the tool conceptual or procedural?  Prior validation of the tool ◦ Has the tool been tested? ◦ Is there information concerning its reliability and validity? ◦ Has it been compared to other tools? ◦ Is it sensitive? Does it discriminate between a novice and an expert?  Experience of others with the tool ◦ Has the tool been used by others besides the developer? At other sites? With other populations? ◦ Is there normative data?

Question s Hold up your “virtual hand” to ask a question.

 Data suggest that the understanding of Concept #2 increased  One interpretation is that the intervention caused the change  List some alternative explanations ◦ Confounding factors ◦ Other factors that could explain the change  Individual Exercise min ◦ Individually write a response

One Minute

 Students learned the concept out of class (e. g., in another course or in study groups with students not in the course)  Students answered with what they thought the instructor wanted rather than what they believed or “knew”  An external event distorted the pretest data  The instrument was unreliable  Other changes in the course and not the intervention was responsible for the improvement  The characteristics of groups were not similar

 Data suggest that the understanding of the concept tested by Q1 did not improve  One interpretation is that the intervention did cause a change that was masked by other factors  Think about alternative explanations  How would these alternative explanations (confounding factors) differ from the previous list?

Evaluation Plan

 List the topics that need to be addressed in the evaluation plan  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group

One Minute

 Name & qualifications of the evaluation expert ◦ Get the evaluator involved early in the proposal development phase  Goals, outcomes, and evaluation questions  Instruments for evaluating each outcome  Protocols defining when and how data will be collected  Analysis & interpretation procedures  Confounding factors & approaches for minimizing their impact  Formative evaluation techniques for monitoring and improving the project as it evolves  Summative evaluation techniques for characterizing the accomplishments of the completed project.

 Workshop on Evaluation of Educational Development Projects ◦  NSF’s User Friendly Handbook for Project Evaluation ◦  Online Evaluation Resource Library (OERL) ◦  Field-Tested Learning Assessment Guide (FLAG) ◦  Student Assessment of Their Learning Gains (SALG) ◦  Science education literature

Identify the most interesting, important, or surprising ideas you encountered in the workshop on dealing with project evaluation 47

Question s Hold up your “virtual hand” to ask a question.

BREAK 15 min 49

BREAK 1 min 50

 NSF proposals evaluated using two review criteria ◦ Intellectual merit ◦ Broader impacts  Most proposals ◦ Intellectual merit done fairly well ◦ Broader impacts done poorly 52

 To increase the community’s ability to design projects that respond effectively to NSF’s broader impacts criterion 53

 At the end of the workshop, participants should be able to: ◦ List categories for broader impacts ◦ List activities for each category ◦ Evaluate a proposed broader impacts plan ◦ Develop an effective broader impacts plan 54

Broader Impacts: Categories and Activities 55

TASK: ◦ What does NSF mean by broader impacts?  Individual Exercise min ◦ Individually write a response 56

One Minute

 Every NSF solicitation has a set of questions that provide context for the broader impacts criterion  Suggested questions are a guide for considering broader impacts  Suggested questions are NOT ◦ A complete list of “requirements” ◦ Applicable to every proposal ◦ An official checklist

Will the project…  Advance discovery - promote teaching & learning?  Broaden participation of underrepresented groups?  Enhance the infrastructure?  Include broad dissemination?  Benefit society? NOTE: Broader impacts includes more than broadening participation

Will the project…  Involve a significant effort to facilitate adaptation at other sites?  Contribute to the understanding of STEM education?  Help build and diversify the STEM education community?  Have a broad impact on STEM education in an area of recognized need or opportunity?  Have the potential to contribute to a paradigm shift in undergraduate STEM education?

TASK:  Identify activities that “broadly disseminate results to enhance scientific and technological understanding”  Pay special attention to activities that will help transport the approach to other sites  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group 61

One Minute

 Dissemination to general public ◦ Applies to research and education development proposals ◦ See handout  Dissemination to peers (other instructors) ◦ Education projects should include strategies for-  Making other instructors aware of material and methods  Enabling other instructors to use material and methods

 Partner with museums, nature centers, science centers, and similar institutions to develop exhibits in science, math, and engineering.  Involve the public or industry, where possible, in research and education activities.  Give science and engineering presentations to the broader community (e.g., at museums and libraries, on radio shows, and in other such venues).  Make data available in a timely manner by means of databases, digital libraries, or other venues such as CD-ROMs 64

 Publish in diverse media (e.g., non-technical literature, and websites, CD-ROMs, press kits) to reach broad audiences.  Present research and education results in formats useful to policy-makers, members of Congress, industry, and broad audiences.  Participate in multi- and interdisciplinary conferences, workshops, and research activities.  Integrate research with education activities in order to communicate in a broader context. 65

 Standard approaches ◦ Post material on website ◦ Present papers at conferences ◦ Publish journal articles  Consider other approaches ◦ NSDL ◦ Specialty websites and list servers (e.g., Connexions) ◦ Targeting and involving a specific sub-population ◦ Commercialization of products ◦ Beta test sites  Focus on active rather than passive approaches 66

Question s Hold up your “virtual hand” to ask a question.

Reviewing a Project’s Broader Impacts 68

 Review the Project Summary & the excerpts from the Project Description  Assume the proposal is a TUES Type 1 with a $200K budget and a 3-year duration and that the technical merit is considered to be meritorious Write the broader impacts section of a review ◦ Identify strengths and weaknesses ◦ Use a bullet format  (Extra) Long Exercise min ◦ Think individually ~4min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 8 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group 69

One Minute

 Scope of activities ◦ Overall-very inclusive and good ◦ Well done but “standard things" ◦ Did not address the issue of quality ◦ No clear-cut plan ◦ Activities not justified by research base  Dissemination ◦ Limited to standard channels ◦ Perfunctory 71

 Industrial advisory committee a strength  Collaboration with other higher education institutions ◦ Institutions appear to be quite diverse but use of diversity not explicit ◦ Interactions not clearly explained ◦ Sends mixed message – raises questions about effectiveness of partnership  High school outreach ◦ Real commitment not evident ◦ Passive -- not proactive ◦ High school counselors and teachers not involved 72

 Modules are versatile  Broader (societal) benefits ◦ Need for materials not well described ◦ Value of the product not explained ◦ Not clear who will benefit and how much  Assessment of broader impacts not addressed 73

TASK: ◦ Identify desirable features of a broader impacts plan or strategy  General aspects or characteristics  Long Exercise min ◦ Think individually ~2 min ◦ Share with a partner ~2 min ◦ Report in local group ---- ~2 min  Watch time and reconvene after 6 min  Use THINK time to think – no discussion  Selected local facilitators report to virtual group 74

One Minute

 Include strategy to achieve impact ◦ Have a well-defined set of expected outcomes ◦ Make results meaningful and valuable ◦ Make consistent with technical project tasks ◦ Have detailed plan for activities ◦ Provide rationale to justify activities ◦ Include evaluation of impacts ◦ Have a well-stated relationship to the audience or audiences 76

WRAP-UP 77

 Use and build on NSF suggestions ◦ List of categories in solicitations ◦ Representative activities on website  Not a comprehensive checklist  Expand on these -- be creative  Develop activities to show impact impact  Integrate and align with other project activities 78

 Help reviewers (and NSF program officers) ◦ Provide sufficient detail  Include goals, objectives, strategy, evaluation ◦ Make broader impacts obvious  Easy to find  Easy to relate to NSF criterion 79

 Make broader impacts credible ◦ Realistic and believable  Include appropriate funds in budget ◦ Make broader impacts consistent with  Project’s scope and objectives  Institution's or College’s mission and culture  PI’s interest and experience  Assure agreement between content of Project Summary and Project Description 80

Identify the most interesting, important or surprising ideas you encountered in the workshop on dealing with broader impacts 81

Grant Proposal Guide Proposal & Award Policies & Procedures Guide Broader Impacts Activities 82

Question s Hold up your “virtual hand” to ask a question.

 To download a copy of the presentation- go to:  Please complete the assessment survey-go to: