1. Assessment Strategies for Inquiry Units Delwyn L. Harnisch University of Nebraska, Lincoln Teacher’s College National Center for Information Technology in Education (NCITE)NCITE harnisch@unl.edu

2. Initial partners: UNL Teachers College, NET, and CSE Planning participants: UNL, UNO, UNK, NDE, Aim Institute Where, why, and how does educational technology apply to learning Basic research, applied research, technology transfer National Center for Information Technology in Education... bringing together researchers and practitioners to appropriately and effectively apply technology to learning.

3. NCITE Major Themes R&D: The role of technology in teaching and learning –Supporting learners with technology –Supporting teaching with technology –Technology supported assessment/evaluation R&D: Interrelationship of technology, education, and society –Social and cultural issues and technologies Moving research into practice: people and tools –Creation/development of educational technology leaders –Develop, adapt, and evaluate educational technology tools

4. Initial Goals Create initial organization, governance, and staffing “Put NCITE on the map” –Take steps to ensure respected, sustainable role in educational technology R&D and transition to practice –Establish educational, commercial, and government collaborations and contacts

5. Build Upon Existing Expertise Centers and Institutes –Center for Innovative Instruction –BUROS Institute for Mental Measurement –Nebraska Educational Telecommunications –Office of Internet Studies (UNO) Current Research –Affinity Learning –Tools Promoting Deeper Online Learning –Cognitive Diagnostic Assessment –Measuring Technology Competencies

6. Build New Opportunities Seed Grants –$75K - $100K per year set aside –Grants will most likely be $15K to $30K –Selected based on probability of outside funding and relevance to NCITE Potential areas – Funded Proposal Writing/Review Teams –Research into the role of human teachers/coaches in computer- based learning At the current state of technology, having humans “in the loop” is vital –Programmatic, education, and technical look at Blackboard and WebCT –Assessment and effectiveness of visualization tools –Intelligent Agents for “leveling the peer playing field” –Literacy/Writing Assessment –Measurement of educator technical competency

7. Create Collaborations Extend partnerships –Faculty/student affiliates –Nebraska Department of Education, K-12 –National linkages (e.g., Higher Education, ADL Initiative, NCSA) Nebraska Symposium on Information Technology in Education (2002 and beyond)http://www.ncite.org/http://www.ncite.org/

8. NCSA and NCITE Current initiatives –Joint proposals and projects –Faculty affiliation –Seminars with discipline leaders –Research linkages and support –Research resources and infrastructure –Publication and dissemination

9. Evaluation of Inquiry Units Requires Reasoning Strategies Problem-Based Approach Real-Life Connections

10. Reasoning in areas of academic work Where Students pose questions or identify issues and through research, search for answers Where students interpret Texts and data Where students Search for ways to reach goals and Overcome obstacles Where students use imagination and discipline to design novel and expressive products

11. What is Evaluation? evaluation addresses a range of questions about quality and effectiveness --both implementation and outcome– uses a range of methods --qualitative and quantitative--to gather and analyze data. evaluation explicitly uses a range of criteria upon which to judge program success, e.g., student interest and motivation, parent satisfaction, teacher enthusiasm, standards of the profession,and student learning as measured in multiple ways. Handbook for Mixed Method Evaluations

12. Types of Evaluations: Planning Assesses understanding of project goals, objectives, timelines and strategies Key questions –Who are the people involved? –Who are the students? –What are the activities that will involve the students? –What is the cost? –What are the measurable outcomes? –How will the data be collected? –How long is the program?

13. Types of Evaluations: Formative Assesses ongoing project activities –Implementation evaluation: –Progress Evaluation

14. Formative: Implementation Evaluation Assesses whether project is being conducted as planned. Key questions: –Were appropriate participants selected? –Do activities and strategies match those in the plan? –Were appropriate staff members hired and trained and working according to the plan? –Were activities conducted according to plan? –Was management plan followed?

15. Formative: Progress Evaluation Assesses the progress made by the participants in meeting goals Key questions: –Are participants moving toward the anticipated project goals? –Which activities and strategies are aiding the participants toward the goal? Mid-year GK12 Evaluation Form

16. Summative Evaluation Assesses project successful in meeting goals Key questions –Was the project successful? –Did the project meet overall goals? –Did participants benefit from the project? –What components were most effective? –Were the results worth the project’s cost? –Is the project replicable and transportable?

17. Stages in Conducting an Evaluation Develop evaluation questions Match questions with appropriate data collection methods Collect data Analyze data Report information to interested audiences

18. Developing Evaluation Questions Clarify goals of the evaluation Identify and involve key stakeholders Describe activity or intervention to be evaluated Formulate potential relevant evaluation questions Determine available resources Prioritize and eliminate questions if necessary

19. Match Questions with Data Collection Methods Select appropriate methodological approach –Quantitative –Qualitative –Mixed Method Determine data sources Select appropriate data collection method to collect data and build data base.

20. Collecting Data Obtain necessary clearances Consider needs and sensitivities of respondents Train data collectors Avoid causing disruption to data collection site

21. Analyze Data Inspect raw data and clean if necessary Conduct initial evaluation based on evaluation plan Conduct additional analyses based on initial results Integrate and synthesize findings

22. Reporting Information Report information to targeted audiences Deliver reports in a timely manner Customize reports and presentations

23. Sample Assessment Strategies for Inquiry Units Adapted from Teaching, Learning, & Assessment Together by Arthur K. Ellis

24. Assessment Possibilities – Jigsaw Groups of students given different parts of an event or problem Groups research their part Groups come together and “teach” others about their part Entire group draws conclusions Promotes collaborative learning

25. Question Authoring Students write down questions on content or concepts they don’t know or areas of interest at the beginning of a lesson Students compare questions in dyads Teacher posts questions Teacher suggests reading or activities that help student answer question

26. Problem Solving: Talk about it Students engage in self-talk or talk with a friend while solving a problem Goals –Self-feedback mechanism –Testing ideas in public –Make thought process deliberate Students share strategies with the class Teacher comments on strategies

27. Creative Work: Learning illustrated Ask students to visualize the concept –Concept map –Picture –Flow chart –Diagram –Map Promotes visualization

28. Examples Science ReviewMath Concept Map

29. Analytic Work: Key Idea Ask what is the “Key” idea at the end of a lesson or unit Ask students to explain why this is a “key idea” Can be done in groups Goal –Students see learning as composed of ideas –Teacher sees what students have valued

30. Analytic Work: I Can Teach Students write, diagram, or map concepts that they could teach to other students Shows how well students know the idea Helps students who have not grasped concepts the first time Allows teacher to evaluate their teaching methods

31. Real Life Connections: Authentic Applications Have students see how an idea is applied in real life Students visualize this idea in posters, projects, or diagram Students reflect on the concept and how it is applied in their visual presentation Example evaluation form

32. Example: Roller coasters

33. Real Life Connections: Getting a Job Role play activity Students imagine they hold a particular occupation Students see how the work they are doing in school is related to the occupation Could be used as a problem-based exercise

34. Summary Activities: I Learned Statements Statements of personal learning at end of lesson or unit Can be done in writing or orally Teachers see what was really important to students Example evaluation form

35. Summary Activities: Clear and Unclear Windows Students sort out what is clear or unclear to them on paper Students who have clear understanding can help those who don’t Example evaluation form

36. Need for Careful Planning Need to have evaluation plan guided by a theory that takes into account –the type of information needed to make decisions –the suitability of different kinds of methods for learning about different kinds of phenomena –the technical limitations and biases of various methods –the importance of purpose of evaluation in mixed- method evaluation –that different purposes invoke different designed mixes of methods

37. Conclusion Mixed method framework for evaluation invites the "uncertainty and openness needed for evaluators to use their findings for action and change" The ability to examine multiple questions about a program offers "exciting and potentially meaningful opportunities for connectedness and solidarity" for evaluators, program managers, and program participants The "practical possibilities" of mixing inquiry methodologies contribute to, and reflect, the pluralistic nature of modern society

38. www.ncite.org NCITE