Presentation on theme: "National Science Foundation Project Evaluation and Assessment Dennis W. Sunal Susan E. Thomas Alabama Science Teaching and Learning Center"— Presentation transcript:
National Science Foundation Project Evaluation and Assessment Dennis W. Sunal Susan E. Thomas Alabama Science Teaching and Learning Center College of Education University of Alabama
Evaluation and Assessment of National Science Foundation Projects Outline Consider your present approach and difficulties toward evaluation Consider NSF’s perception of evaluation difficulties Explore the process and skills associated with proficiently planning a project evaluation Investigate difficulties with a case study example Creating your own evaluation plan
Group Activity: Form small groups of two or three Group members will list “project evaluation” difficulties they have recognized in the past Create a question from one of the listed items " What is the difficulty?”
Difficulties with NSF Project Evaluation Whole Audience Activity: Consider the question; What are the critical areas that define “evaluation difficulties” in your projects?
NSF Recognized Common Difficulties with Project Evaluations?
NSF Diagnosed Difficulties ? Lack of funding planned for evaluation Too short a time, (e.g. longer than life of the project, follow grad student for years) Use of only quantitative methods Lack of knowledge of evaluation techniques Did not use information available about what has worked in other projects Data collection not extended extended over time Ambiguous requirements and/or outcomes
Evaluation Evaluation has varied definitions…. Accepted definition for NSF Projects: “Systematic investigation of the worth or merit of an object…” -Joint committee on Standards for Education Evaluation
NSF Rationale and Purpose of Conducting Project Evaluation Develop a diverse internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens; Enabling discoveries across the frontiers of science and engineering connected to learning, innovations, and service to society; Providing broadly accessible, state-of-the-art information bases and shared research and education tools.
How NSF Thinks About Evaluation! A component that is an integral part of the research and development process It is not something that comes at the end of the project It is a continuous process that begins during planning Evaluation is regularly and iteratively performed during the project and is completed at the end of the project Different questions are appropriate at different phases of the project
Evaluation can be accomplished in different ways There is no single model that can be universally applied Data gathered may be quantitative, qualitative, or both
Important Criteria for Evaluation Focus on an important question about what is being accomplished and learned in the project Emphasis on gathering data that can be used to identify necessary mid- project changes Plan a strong evaluation design (e.g. with comparison groups and well-chosen samples) that clearly addresses the main questions and rules out threats to validity Use sound data collection instruments, appropriate to the questions addressed Establish procedures to assure the evaluation is carried out objectively and sources of bias are eliminated Data analysis appropriate to questions asked and data collection methodologies being used A reasonable budget given the size of the project, about 5 to 10 percent
NSF Expects Grantee will clearly lay out and evaluation plan in the proposal Refine the plan after the award Include in the final report a separate section on the evaluation, its purpose, and what was found In some cases a separate report or interim reports may be expected.
Example RFP - Faculty Early Career Development Proposal Content - A. Project Summary: Summarize the integrated education and research activities of the plan B. Project description: Provide results from prior NSF support. Provide a specific proposal for activities over 5 years that will build a firm foundation for a lifetime of integrated contributions to research and education.
Plan for development should include: The objectives and significance of the research and education activities. The objectives and significance of the research and education activities. The relationship of the research to current knowledge in the field. The relationship of the research to current knowledge in the field. An outline of the plan; including evaluation of the educational activities on a yearly basis. An outline of the plan; including evaluation of the educational activities on a yearly basis. The relationship of the plan to career goals and objectives. The relationship of the plan to career goals and objectives. A summary of prior research and educational accomplishments. A summary of prior research and educational accomplishments.
NSF Merit Review Broader Impacts Criterion: Representative Activities NSF criteria relate to 1) intellectual merit and 2) broader impact. Criteria: Broader Impacts of the Proposed Activity Does the activity promote discover, understanding, teaching, training and learning? Does the proposed activity include participants of underrepresented groups? Does it enhance the infrastructure for research and education Will the results be disseminated broadly to enhance scientific and technological understandings? What are the benefits of the proposed activity to society?
1. Advance discovery and understanding while promoting teaching, training and learning Integrate research activities into the teaching of science at all levels Involve students Participate in recruiting and professional development of teachers Develop research based educational materials and databases Partner researchers and educators Integrate graduate and undergraduate students Develop, adapt, or disseminate effective models and pedagogic approaches to teaching
2. Broaden participation of underrepresented groups Establish research and education collaborations with students and teachers Include students from underrepresented groups Make visits and presentations on school campuses Mentor early year scientists and engineers Participate in developing new approaches to engage underserved individuals Participate in conferences, workshops, and field activities
3. Enhance infrastructure for research and education Identify and establish collaborations between disciplines and institutions Stimulate and support next generation instrumentation and research and education platforms Maintain and modernize shared research and education infrastructure Upgrade the computation and computing infrastructure Develop activities that ensure multi-user facilities are sites of research and mentoring
4. Broaden dissemination to enhance scientific and technological understanding Partner with museums, science centers, and others to develop exhibits Involve public and industry in research and education activities Give presentations to broader community Make data available in a timely manner Publish in diverse media Present research and education results in formats useful to policy makers Participate in multi- and interdisciplinary conferences, workshops, and research activities. Integrate research with education activities
5. Benefits to society Demonstrate linkage between discovery and societal benefit through application of research and education results. Partner with others on projects to integrate research into broader programs Analyze, interpret, and synthesize research and education results in formats useful to non- scientists
Planning Evaluation Necessary to assess understanding of a project’s goals, objectives, strategies and timelines. The criteria can also indicate a baseline for measuring success.
Planning Evaluation Addresses the Following… Why was the project developed? Who are the stakeholders? What do the stakeholders want to know? What are the activities and strategies that will address the problem which was identified? Where will the program be located? How long will the program operate? How much does it cost in relation to outcomes? What are the measurable outcomes to be achieved? How will data be collected?
Two Kinds of Evaluation Program Evaluation determines the value of the collection of projects (e.g. the Alabama DOE-EPSCoR Program) Project Evaluation focuses on an individual project and its many components funded under an umbrella of the program (research plan components, educational plan components). It answers a limited number of questions.
Types of Evaluation Formative Evaluation –Implementation –Progress Summative Evaluation –Near end of a major milestone or –At the end of a project
Formative Evaluation Assesses ongoing project activities … Assesses ongoing project activities … Purpose: assess initial and ongoing project activities Done regularly at several points in throughout the developmental life of the project. Main goal is to check, monitor, and improve - to see if activities are being conducted and components are progressing toward project goals
Implementation Evaluation Early check to see if all elements in place Assess whether project is being conducted as planned Done early, several times during a life cycle Cannot evaluate outcomes or impact unless you are sure components are operating according to plan Sample question guides: Were appropriate students selected, was the make-up of group consistent with NSF’s goal for a more diverse workforce? Were appropriate recruitment strategies used? Do activities and strategies match those described in the plan?
Progress Evaluation: Assess progress in meeting goals Collect information to learn if benchmarks of progress were met, what impact activities have had, and to determine unexpected developments Useful throughout life of project Can contribute to summative evaluation Sample question guides: Are participants moving toward goals, improving understanding of the research process Are numbers of students reached increasing? Is progress sufficient to reach goals?
Summative Evaluation Assesses project’s success and answers… Assesses project’s success and answers… Purpose: Assess the quality and impact of a fully mature project Was the project successful? To what extent did the project meet the overall goals? What components were the most effective? Were the results worth the projects cost? Is the project replicable?
Characteristics Collects information about outcomes and impacts and the processes, strategies, and activities that led to them Needed for decision making – disseminate, continue probationary status, modify, or discontinue Important to have external evaluator who is seen as objective and unbiased or have an internal evaluation with an outside agent review of the design and findings Consider unexpected outcomes
Evaluation Compared to other Data Gathering Activities Evaluation differs from other types of activities that provide information on accountability Different information serves different purposes
Formative vs. Summative “When the cook tastes the soup, that is formative; when the guest taste the soup, that is summative.”
The Evaluation Process Steps in conducting an evaluation (six phases) 1. Develop a conceptual model of the program and identify key evaluation points 2. Develop evaluation questions and define measurable outcomes 3. Develop an evaluation design 4. Collect data 5. Analyze data 6. Reporting
Conditions to be met Information gathered is not perceived as valuable or useful (Wrong questions asked) Information gathered is not seen as credible or convincing (wrong techniques used) Report is late or not understandable (does not contribute to decision making process)
1. Develop a Conceptual Model InputsActivitiesShort-Term Outcomes Long-Term Outcomes Start with a conceptual model to which an evaluation design is applied. In the case below a “logic model” is applied. Identify program components and show expected connections among them.
Inputs (Examples) Resource streams NSF Funds Local and State funds Other partnerships In-kind contributions Activities (Examples) services, materials, and actions that characterize project goals Recruit traditionally underrepresented students Infrastructure development Provisions of extended standards-based professional development Public outreach Mentoring by senior scientist
Short-Term Outcomes (Examples) Effective use of new materials Numbers of people, products or institutions reached (17 students mentored) Changes resulting from experience (impact on choice of major of research RAs) Long-Term Outcomes (Examples) Broader more enduring impact Changes in instructional practice leading to enhanced student learning and performance Selecting a career in NSF-related research activity
Next steps: Determine, review, and/or clarify timeline Identify critical achievements and times that need to be met.
2. Question Development Identify key stakeholders and audiences early to help shape questions. Multiple audiences exist. (Scientists, NSF, students, administration, community..) Formulate potential evaluation questions of interest considering stakeholders and audiences. Define outcomes in measurable terms, including criteria for success. Determine feasibility and prioritize and eliminate questions.
Questions to consider when developing an evaluation approach… 1. Who is the information for and who will use the findings? 2. What kinds of information are needed? 3. How is the information to be used? 4. When is the information needed? 5. What resources are available to conduct the evaluation? 6. Given the answers to the preceding questions, what methods are appropriate?
2. …Defining Measurable Outcomes 1. Briefly describe the purpose of the project. 2. State in terms of a general goal. 3. State an objective to be evaluated as clearly as you can. 4. Can this objective be broken down further? 5. Is the objective measurable? If not – restate. 6. Once you have completed the above steps, go back to # 3 and write the next objective. Continue with steps 4, 5, and 6.
3. Develop and Evaluation Plan Select a methodological approach and data collection instruments –Quantitative or qualitative –Lead to different questions asked, timeframe, skills needed, type of data seen as credible Determine who will be studied and when –Sampling, use of comparison groups, timing, sequencing, frequency of data collection, and cost.
4. Data Collection Obtain necessary clearance and permission. Consider the needs and sensitivities of the respondents. Make sure your data collectors are adequately trained and will operate in an objective, unbiased manner. Obtain data from as many members of your sample as possible. Cause as little disruption as possible to the ongoing effort.
4. Data Collection Sources and Techniques Checklists or inventories Rating scales Semantic differentials QuestionnairesInterviews Written responses Samples of work TestsObservationsAudiotapesVideotapes Time-lapse photographs
5. Analysis of Data Check raw data and prepare data for analysis. Conduct initial analysis based on the evaluation plan. Conduct additional analysis based on the initial results. Integrate and synthesize findings Develop conclusions regarding what the data shows
6. Reporting Final Reports typically include six major sections: a) Background b) Evaluation study questions c) Evaluation procedures d) Data analysis e) Findings f) Conclusions (and recommendations)
a. Background The background section describes: The problem or needs addressed A literature review (if relevant) The stakeholders and their information needs The participants The project’s objectives The activities and components Location and planned longevity of the project The project’s expected measurable outcomes
b. Evaluation and Study Questions Describes and lists the questions the evaluation addressed. Based on: Need for specific information Stakeholders
c. Evaluation Procedures This section describes the groups and types of data collected and the instruments used for the data collection activities. For example: Data for identified critical indicators Ratings obtained in questionnaires and interviews Descriptions of activities from observations of key instrumental components of the project Examination of extant data records
d. Data Analysis Describes the techniques used to analyze the data collected Describes the various stages of analysis that were implemented Describes checks that were carried out to make sure that the data were free of as many confounding factors as possible Contains a discussion of the techniques used
e. Findings Presents the results of the analysis described previously Organized in terms of the questions presented in the section on evaluation study questions Provides a summary that presents the major conclusions
f. Conclusions Reports the findings with more broad-based and summative statements Statements must relate to the findings of the project’s evaluation questions ad to the goals of the overall program. Sometimes includes recommendations for NSF or the other undertaking projects similar in goals, focus, and scope. Recommendations must be based solely on robust findings that are data-based and not on anecdotal evidence.
Other Sections An Abstract: a summary of the study and its findings presented in approximately one half page of text. An executive summary: a summary which may be as long as 4 to 10 pages, that provides an overview of the evolution, its findings, and implications.
Formal Report Outline Summary Sections –Abstract –Executive summary Background –Problems or needs addressed –Literature review –Stakeholders and their information needs –Participants –Projects’ objectives –Activities and components
–Location and planed longevity of the project –Resources used to implement the project –Constraints Evaluation study questions –Questions addressed by the study –Questions that could not be addressed by the study Evaluation Procedures –Sample: Selection procedures Representativeness of the sample –Data collection MethodsInstruments
–Summary matrix Evaluation questions Variables Data gathering approaches Respondents Data collection schedule Findings –Results of the analysis organized by study questions Conclusions –Broad-based, summative statements –Recommendations, when applicable
Disseminating the Information Consider what various groups need to know Best manner for communicating information to them Audiences –Funding sources and potential funding sources –Others involved with similar projects or areas of research –Community members, especially those who are directly involved with the project or might be involved –Members of the business or political community, etc.
Finding an Evaluator University setting - contact department chairs for availability of staff skilled in project evaluation Independent contractors – department chairs, phone book, state departments, private foundations (Kellogg Foundation in Michigan), and other local colleges and universities will be cognizant of available services Contact other researchers or peruse research and evaluation reports
Overview of Quantitative and Qualitative Data Collection Methods
Data Collection Methods: Some Tips and Comparisons Theoretical Issues Practical Issues Using the Mixed-Method Approach
Theoretical Issues The value of the types of data The relative scientific rigor of the data Basic, underlying philosophies of evaluation
Practical Issues Credibility of findings Staff skills Costs Time constraints
Using the Mixed-Method Approach Methodology:Qualitative QuantitativeQualitative Data Collection Approach: ExploratorySurveyPersonal Interview focus group
Review and Comparison of Selected Techniques Surveys Interviews Focus Groups Observations Tests Other Methods Document Studies Key Information Case Studies
Surveys Advantages: Good for gathering descriptive data Can cover a wide range of topics Are relatively inexpensive to use Can be analyzed using a variety of existing software Disadvantages: Self-report may lead to biased reporting Data may provide a general picture but lack depth May not provide adequate information on context
Interviews Use interviews to answer the following questions: 1. What does the program look and feel like to the participants? To other stakeholders? 2. What do stakeholders know about the project? 3. What thoughts do stakeholders knowledgeable about the program have concerning program operations, processes, and outcomes? 4. What are the participants’ and stakeholders’ expectations? 5. What features of the project are most salient to the participants? 6. What changes do participants perceive in themselves as a result of their involvement in the project?
Interviews Advantages: Usually yield richest data, details, new insights Permit face-to-face contact with respondents Provide opportunity to explore topics in depth Allow interviewer to experience the affective as well as cognitive aspects of responses Allow interviewer to explain or help clarify questions, increasing the likelihood of useful responses Disadvantages: Expensive and time-consuming Need well-qualified, highly trained interviewers Interviewee may distort information through recall error, selective perceptions, dire to please interviewer Flexibility can result in inconsistencies across interviews Volume of information vary large; may be difficult to transcribe and reduce data
Focus Groups When to use focus groups: Identifying and defining problems in project implementation Pre-testing topics or ideas Identifying project strengths, weaknesses, and recommendations Assisting with interpretation of quantitative findings Obtaining perceptions of project outcomes and impacts Generating new ideas
Observations Advantages: Provide direct information about behavior of individuals and groups Permit evaluator to enter into and understand situation/context Provide good opportunities for identifying unanticipated outcomes Exist in natural, unstructured, and flexible setting Disadvantages: Expensive and time consuming Need well-qualified, highly trained observers; may need to be content experts May affect behavior of participants Selective perception of observer may distort data Behavior or set of behaviors observed may be atypical
Test Advantages: Provide objective information on what the test taker knows and can do Can be constructed to match a given curriculum or set of skills Can be scored in a straightforward manner Are accepted by the public as a credible indicator of learning Disadvantages: May be oversimplified and superficial May be very time consuming May be biased against some groups of test takers May be subject to corruption via coaching or cheating
Other Methods – Document Studies Advantages: Available locally Inexpensive Grounded in setting and language in which they occur Useful for determining value, interest, positions, political climate, public attitudes Provide information on historical trends or sequences Provide opportunity for study of trends over time Unobtrusive Disadvantages: May be incomplete May be inaccurate or of questionable authenticity Location suitable documents may pose challenges Analysis may be time consuming and access may be difficult
Other Methods – Key Informants Advantages: Information concerning causes, reasons, and/or best approaches is gathered form an “insider” point of view Advice/feedback increases credibility of study pipeline to pivotal groups May have side benefit to solidify relationships among evaluators, clients participants, and other stakeholders Disadvantages: Time required to select and get commitment may be substantial Relationship between evaluator and informants may influence type of data obtained Informants may interject own biases and impressions Disagreements among individuals may be hard to resolve
Other Methods – Case Studies Advantages: Provide a rich picture of what is happening, as seen through the eyes of many individuals Allow a through exploration of interactions between treatment and contextual factors Can help explain changes or facilitating factors that might otherwise not emerge form the data Disadvantages: Require a sophisticated and well-trained data collection and reporting team Can be costly in terms of the demands on time and resources Individual cases may be over-interpreted or over- generalized
Evaluation Application Activity Group Activity: Form small groups and assign roles. All group members should read the “Sample Proposal Outline” on the next set of slides. Consider the question, Develop an Evaluation Plan for the NSF Project. " What should be considered?”
Proposal Outline Sample Research Proposal CAREER: Fundamental Micromechanics and Materials Dynamics of Thermal Barrier Coating Systems Containing Multiple Layers A. Research Plan 1.Introduction General background General definitions Explanation of procedures Benefits Objectives Characterize the dynamics of the process Characterize the dynamics of the process Monitoring techniques Monitoring techniques Development of models Development of models
2. Proposed Research Introduction Experimental Approach Materials Micro-structural Characterization Mechanical Properties Micro-mechanical Characterization – Nano- indentation Bulk Mechanical Properties Residual Stresses and Techniques Modeling 3. Prior Research Accomplishments 4. Significance and Impact of Research 5. Industrial Interest
B. Education Plan 1. Objectives Enhance the undergraduate curriculum Encourage the best undergraduate students to pursue graduate studies To increase diversity by attracting underrepresented minority students
2. Education Activities – –The education of undergraduate and graduate students in materials and mechanical characterization and laboratory report preparation. – –Encourage undergraduates to pursue graduate work. – –Actively recruit undergraduate minority students 3. Teaching Activities 4. Teaching & Education Accomplishments
References NOVA Web Site (NASA Opportunities for Visionary Academics) NSF (2002). Division of Research, Evaluation and Communication National Science Foundation (2002).The 2002 User-Friendly Handbook for Project Evaluations. Henson, K. (2004). Grant writing in higher education, Boston: Pearson Publishers. The first time grant writer's guide to success, Knowles, C. (2002). The first time grant writer's guide to success, Thousand Oaks CA: Corwin Press Simplified grant writing, Burke, M. (2002). Simplified grant writing, Thousand Oaks CA: Corwin Press NSF (2004). A Guide for Proposal Writing nsf04016_Criteria for Evaluation: Intellectual merit and broader impacts. -A Guide for Proposal Writing nsf04016_Criteria for Evaluation
NSF NSF : The 2002 User-Friendly Handbook for Project Evaluation, a basic guide to quantitative and qualitative evaluation methods for educational projects NSF NSF : User-Friendly Handbook for Mixed Method Evaluations, a monograph "initiated to provide more information on qualitative [evaluation] techniques and... how they can be combined effectively with quantitative measures" Online Evaluation Resource Library (OERL)Online Evaluation Resource Library (OERL) for NSF's Directorate for Education and Human Resources, a collection of evaluation plans, instruments, reports, glossaries of evaluation terminology, and best practices, with guidance for adapting and implementing evaluation resources Field-Tested Learning Assessment Guide (FLAG)Field-Tested Learning Assessment Guide (FLAG) for Science, Math, Engineering, and Technology Instructors, a collection of "broadly applicable, self-contained modular classroom assessment techniques and discipline-specific tools for... instructors interested in new approaches to evaluating student learning, attitudes, and performance."
National Science Foundation Project Evaluation and Assessment Dennis W. Sunal Susan E. Thomas Alabama Science Teaching and Learning Center College of Education University of Alabama