Are you Using Technology to Assess Learning or Assessing Learning to Improve Technology? Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary Education and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego State University 3590 Camino Del Rio North San Diego, California, U.S.A

Bresciani, M.J. Overview  Overview of Outcomes-Based Assessment  Examples of Using Technology to Assess Learning or Assessing Learning to Improve Technology  Questions to Consider.

Bresciani, M.J. Ask Yourself These Questions How would you describe what outcomes-based assessment is to your faculty, staff, and students? How would you explain its importance?

Bresciani, M.J. Ask Yourself These Questions How would your description change if you were describing assessment for  how well technology enhances student learning?  how well technology can be used to assess and report out on student learning ?

Bresciani, M.J. The Assessment Cycle (Bresciani, 2006) The key questions… What are we trying to do and why? or What is my program supposed to accomplish? or What do I want students to be able to do and/or know as a result of my course/workshop/orientation/program? How well are we doing it? How do we know? How do we use the information to improve or celebrate successes? Do the improvements we make contribute to our intended end results?

The Iterative Systematic Assessment Cycle Bresciani, 2006 Mission/Purposes Goals Outcomes Implement Methods to Deliver Outcomes and Methods to Gather Data Gather Data Interpret Evidence Make decisions to improve programs; enhance student learning and development; inform institutional decision- making, planning, budgeting, policy, public accountability

Bresciani, M.J. Drivers  Regional Accreditation Requirement since 1985  National Commission on the Future of Higher Education (2006)  Demand for Public Information about Performance Transparency of outcomes and results Comparable measures of quality Demonstration of value-added of the entire educational experience

Bresciani, M.J. Drivers, Cont.  Accountability requirements handed back to states Performance indicators Standardized tests  Discipline Standards could be designed by disciplines  Increased focus on private education  Still no focus on evaluating transferability of learning

Bresciani, M.J. Now turn to each other  Would you change any of your responses to the aforementioned questions? How would you describe what outcomes-based assessment is? How would you explain its importance?

Bresciani, M.J. This Assessment Process Can be Used To… 1. Assess learning, where technology is a data collection tool or teaching tool (to promote deep learning) 2. Improve technology as a teaching tool or a data collection tool in order to make assessment of learning more effective

Bresciani, M.J. This Assessment Process Can be Used To… 3. Improve professional development for use of technology 4. Improve technological skills of students, faculty, and staff 5. Improve process of planning for improved student learning by using technology as an instructional organizational tool

So, One Process, which is outcomes-based assessment, for many uses

Bresciani, M.J. The Key is to Determine  What you are assessing?  Therefore, what are your outcomes?  Informs Design  Informs Methodology  Informs Decisions

Bresciani, M.J. Student Learning Example  Students will be able to identify potential ethical research dilemmas and conflicts of interest and propose solutions to address them Case Study exercise in class Assignment - Student’s actual research project using Blackboard discussion

Bresciani, M.J. Student Learning Example, Cont.  The majority of students were able to demonstrate this learning on blackboard and challenged their peers’ comments as well.  However, three students did not perform as well as they had on the in- class case studies.

What was the problem? Lack of student learning Or concern with technology as a data collection tool?

Bresciani, M.J. Two things Going on at Once 1. Assess learning, where technology is a data collection tool or teaching tool 2. Improve technology as a data collection tool in order to make assessment of learning more effective

Bresciani, M.J. Another Example  Assessing effectiveness of professional development for faculty using TracDat to manage their process for evaluating student learning Evidence of assessment plans and reports entered

Bresciani, M.J. Another Example  All 10 who attended the workshop entered assessment plans But 6 of the 10 had poorly written outcomes, and disconnected evaluation methods even though all items were in their “correct” box  Only 2 of the 10 faculty had entered assessment reports

What was the problem? Poor professional development workshop Or concern with faculty and their technology skills?

Bresciani, M.J. Three Things Going on at Once 2. Improve technology as a data collection tool in order to make assessment of learning more effective 3. Improve professional development for use of technology 4. Improve technological skills of students, faculty, and staff

Bresciani, M.J. One Final Example  Assessed instructional technology for its accessibility from the Pacific Islands Delays in assignments posted Troubles posting to discussion board

Bresciani, M.J. Example, Cont.  Poor accessibility by students but could not discern if it was the software or the students  Interestingly, students did not complain about access and the designer said it was because the course expectations were so clear

What was the problem? Poor technology Or concern with students and their technological skills?

Bresciani, M.J. Three Things Going On 2. Improve technology as a teaching tool in order to make assessment of learning more effective 4. Improve technological skills of students, faculty, and staff 5. Improve process of planning for improved student learning by using technology as an instructional organizational tool

Bresciani, M.J. Re-Visiting: The Key is to Determine  What you are assessing?  Therefore, what are your outcomes?  Informs Design  Informs Methodology  Informs Decisions

Take a moment and share with each other a scenario where you determined that you needed to clarify whether you were assessing technology or learning

Bresciani, M.J. Questions to Consider (Bresciani, 2007)  When considering the use of Technology to promote outcomes-based assessment… Does the use of this technology reinforce the reasons we are engaged in outcomes-based assessment? How does the use of this technology promote formative assessment, summative assessment, or the assessment of transferable skills/knowledge?

Bresciani, M.J. Questions to Consider, Cont. (Bresciani, 2007)  Is our professional development to train users of our technology coupled with the professional development for outcomes- based assessment or instructional design?  How does the use of this technology enable us to better use the results to improve student learning and development?

Bresciani, M.J. Questions to Consider, Cont. (Bresciani, 2007)  How does the use of this technology improve efficiencies? And who benefits from those improved efficiencies?  How does this technology constrain or promote creativity/flexibility within each discipline?  How does this technology aid the coordination of the process? Or does it contribute to isolated thinking and doing?

Bresciani, M.J. Questions to Consider, Cont. (Bresciani, 2007)  How can data that this technological tool gathers help us evaluate the extent that we are transforming our organization through the use of evidence-based decision-making? “Keep your eye on the prize”

Bresciani, M.J. Challenge Questions to Consider (Bresciani, 2007)  How does this technology help illustrate the transferability of learning/skills?  How does this technology help us illustrate, evaluate, and improve the learning/skills as they relate to the requisite skills required by employers?

Bresciani, M.J. Challenge Questions to Consider, Cont. (Milne, 2007)  How does this technology help us determine whether the technology is promoting or prohibiting social interaction?  How do we ensure that the technology is assuring the deepest engagement with the subject matter?

Bresciani, M.J. Challenge Questions to Consider, Cont. (Bresciani, 2007)  How will you determine that technology is promoting deep learning?  What do you need to refine in order to make this happen?

Which questions do you need to focus on as you move forward with your learning at this conference?

Questions?

Bresciani, M.J. References  Bresciani, M.J. (2006). Outcomes-Based Academic and Co-Curricular Program Review: A Compilation of Institutional Good Practices. Sterling, VA: Stylus Publishing  Bresciani, M.J. (2007). Implementing Lessons Learned: Establishing A Culture of Evidence-Based Assessment. Closing Keynote at Course Technology Conference. Las Vegas, NV.