1. Keeping it Simple: Helping Faculty and Staff Create Meaningful, Measurable, and Manageable Assessment Plans and Reports
Amber Malinovsky
Assistant Director Institutional Assessment
Texas A&M University

2. About Us Texas A&M University, College Station, Texas
48,000 students
2,700 faculty
5,500 staff
5,200 acres
WEAVE users and programs
470+ programs in entity tree
700 WEAVE users

3. About Us Office of Institutional Assessment
Four full-time employees, two graduate assistants
Variety of duties that support assessment
Liaison structure
Ten colleges
Two branch campuses (Galveston and Qatar)
Multiple support units (student affairs, undergraduate programs, etc.)

4. Setting up a Workshop Choose a topic
Often suggested by users/Assessment Liaisons
Sometimes specific to audience (academic vs. support, by discipline, etc.)
Conduct research and collect examples
Draft and revise slides and materials
Our “simple” formula: definition + tips + examples + cautions + putting it all together = meaningful, measurable, manageable

5. Setting up a Workshop Choose dates and locations
Advertise and track registrants
Announcement sent on lists and posted on website
Online form for registration
Evaluation and follow-up
Past workshops:

6. The Assessment Plan & Report
Dr. Loraine Phillips
Amber Malinovsky

7. Agenda Part One Introduction to Assessment
Components of an Assessment Plan
Mission
Outcomes
Measures
Achievement Targets
Components of an Assessment Report
Findings
Analysis of Findings
Action Plans
Annual Reporting

8. Suggested Assessment Report Deadlines
May 31, 2009 Data collected
August 1, 2009 Findings entered
September 1, 2009 Analysis Questions answered, along with any appropriate action plans
September 15, 2009 Annual Report question answered
October 1, Cycle Year CLOSES no more editing

9. SACS Comprehensive Standard 3.3.1
SACS Expectations
SACS Comprehensive Standard 3.3.1
3.3 Institutional Effectiveness
The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)
educational programs, to include student learning outcomes
administrative support services
educational support services
research within its educational mission, if appropriate
community/public service within its educational mission, if appropriate

10. The Assessment Process
Develop Program Mission & Outcomes
Design an Assessment Plan
Implement the Plan & Gather Information
Interpret/ Evaluate Information
Modify & Improve
Adapted from: Trudy Banta, IUPUI

11. Mission Statement
The mission statement links the functions of your unit to the overall mission of TAMU.
A few questions to consider in formulating the mission of your unit:
What is the primary function of your unit?
What core activities are involved?
What should those you serve experience after interacting with your unit?

12. Characteristics of a Well-Defined Mission Statement
Brief, concise, distinctive
Clearly identifies the program’s purpose
Clearly aligns with the mission of the division and the University
Explicitly articulates the essential functions/activities of the program
Clearly identifies the primary stakeholders of the program: i.e., students, faculty, parents, etc.

13. Example of a Mission Statement
“University Career Services, an integral part of the educational process, assists students and alumni in assessing their career possibilities, setting their personal goals and achieving their objectives toward becoming productive citizens in the global community. While assisting its clients in identifying professional employment opportunities, University Career Services also provides the university community with insights into the ever- changing world of work to help develop realistic ways to better educate tomorrow’s leaders.” (Texas Christian University)

14. Learning Outcomes
When writing Learning Outcomes, the focus must be on the stakeholders (e.g., students, faculty, staff, and others) and what they will think, know, do, or value following the provision of the service.

15. Program Outcomes Process statements Satisfaction statements
Relate to what the unit intends to accomplish
Level or volume of activity
Efficiency with which you conduct the processes
Compliance with external standards of “good practice in the field” or regulations
Satisfaction statements
Describe how those you serve rate their satisfaction with your unit’s processes or services

16. Common Issues Addressed in Program Outcomes
Access
Accountability
Affordability
Climate
Educational Effectiveness/Quality
Facilities & Equipment
Faculty
Financial Support
Student Preparation
Productivity
Technology

17. Writing Program Outcomes
Consider such questions as:
What are the most important results or impacts that should occur as a result of your unit’s activities?
What are your critical work processes and how should they function?
What does the end user experience through interaction with your unit?

18. Examples of Outcomes Learning Outcomes Process statements
Students receiving advising services will be able to identify and utilize campus resources.
Staff and faculty will be able to use the Oracle system for purchasing.
Process statements
The number of faculty training workshops and participation rates for WebCT will relate to the needs.
The travel office will promptly process travel requests.
Satisfaction statements
Faculty will report satisfaction in the training received for WebCT.
Faculty and staff will report satisfaction with travel processing under the Oracle system.

19. Writing Outcomes Outcomes should be: linked to the unit’s mission
realistic and attainable
limited in number (manageable)
something that is under the control of the unit
measurable and/or observable
meaningful
target key services or change points
use action verbs

20. Assessment Measures After establishing your outcomes…
Define and identify the sources of evidence you will use to determine whether you are achieving your outcomes.
Detail what will be measured and how
Identify or create, if necessary, measures that help your unit in making critical decisions about its processes and services.

21. Choosing Assessment Measures
Some things to think about:
How would you describe the end result of the outcome?
How will you know if this outcome is being accomplished?
What will provide you with this information?
Where are you currently delivering the outcome?
Are there any naturally occurring assessment opportunities?
What measures are currently available?

22. Example of a Delivery Map
Learning Outcome
Method of Delivery
How, When, and Where Information will be Gathered
Outcome 1
Outcome 2
Outcome 3
Etc.

23. Characteristics of an Effective Assessment Measure
Measurable and/or observable
You can either observe it, count it, quantify, etc.
Meaningful
If captures enough of the essential components of the objective to represent it adequately
Manageable
It can be measured without excessive cost or effort

24. Direct Measures
Direct measures are those designed to directly measure:
what a stakeholder knows or is able to do (i.e., requires a stakeholder to actually demonstrate the skill or knowledge)
The benefit of programming or intervention

25. Common Direct Measures
Participation data
Observation of behavior
Culminating experiences (e.g., presentation, project, internships, etc.)
Collection of work samples (portfolios)
Pre- and post-measures
Volume of activity
Level of efficiency (average response time)
Measure of quality (average errors)

26. Indirect Measures Indirect measures focus on:
stakeholders’ perception of their level of learning
stakeholders’ perception of the benefit of programming or intervention
stakeholders’ satisfaction with some aspect of the program or service

27. Common Indirect Measures
Surveys
Exit interviews
Retention/graduation data
Demographics
Focus groups

28. Achievement Targets
An achievement target is the result, target, benchmark, or value that will represent success at achieving a given outcome.
Achievement targets can be specific numbers or trends.

29. Examples of Achievement Targets
Students will achieve a score of 3 or 4 on the rubric.
Average score on rubric is a 3 or better (on a scale of 0 to 4).
90% of the transcripts will be sent within three days.
Each employee will participate in a minimum of two training/development programs per year.
Acquisition statistics will indicate growth in the overall collections that support academic programs.

30. Putting It All Together
Outcome: Improve and strengthen media relations and publications
Measure 1 (Direct): Track media pitches/releases.
Achievement Target: Average of 250 media pitches/releases for the year.
Measure 2 (Indirect): Conduct annual readership survey of university magazine.
Achievement Target: Feedback via readership survey will indicate average ratings of 7 or better on a 1-10 scale.

31. Putting It All Together
Outcome: To provide a library web site that enables users to locate and use information on their own
Measure 1 (Direct): Usability testing with small groups
Achievement Target: Participants in usability testing demonstrate web site’s ease of use
Measure 2 (Indirect): LibQUAL+ assessment program
Achievement Target: LibQUAL+ results indicate faculty and student satisfaction

32. Things to be careful of…
Writing your Achievement Target as part of your Outcome
Example:
Outcome: 80% satisfaction rate from instructors
Measure: Biannual instructor satisfaction survey
Achievement Target: 80% satisfaction rate
Possible Steps to Revision:
Satisfied with what?
New Outcome: Instructors will report satisfaction with the resources available for course development.

33. Things to be careful of…
Having the same Measure and Achievement Target
Example 1:
Measure: Annual review of forms
Achievement Target: Annual review of forms
Possible Steps to Revision:
Go back and look at outcome. Outcome states that Forms and paperwork will be clear, concise & accurate.
If the Annual review shows that they are not clear, concise & accurate, what will need to happen?
New Achievement Target: Decrease in number of revisions needed to make forms more understandable and accurate.

34. Things to be careful of…
Having the same Measure and Achievement Target
Example 2:
Measure: 50 slots for scholarship recipients secured
Achievement Target: 50 slots secured for scholarship recipients
Possible Steps to Revision:
Remember: Measures are more general. Achievement Targets are specific.
Identify what it is that you’re doing to determine whether or not you’ve met the achievement target. THIS is your measure.
New Measure: Track number of scholarship recipients.

35. Things to be careful of…
Writing your Achievement Target as part of your Measure
Example:
Measure: Score a 3 or 4 on an identified technology assignment
Possible Steps to Revision:
Remember: Measures are more general. Achievement Targets are specific.
Outcome states: Demonstrate competency in technology
Revise Measure to read: Identified technology assignment.
Add Achievement Target: Students will achieve a 3 or 4 on the rubric.

36. Assessment Plans & Reports
Mission/Purpose
Outcomes, learning & program
Assessment Methods with achievement targets
Assessment Report
Mission/Purpose
Outcomes, learning & program
Assessment Methods with achievement targets
Findings
Action Plans
Analysis Questions
Annual Report Section

37. SACS Comprehensive Standard 3.3.1
SACS Expectations
and provides evidence of improvement based on analysis of the results…
SACS Comprehensive Standard 3.3.1
3.3 Institutional Effectiveness
The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness)
educational programs, to include student learning outcomes
administrative support services
educational support services
research within its educational mission, if appropriate
community/public service within its educational mission, if appropriate

38. Findings Findings = Assessment data
On an Assessment Report, “findings” refers to a concise summary of the results you gathered from a given assessment measure.
The language of this statement should parallel the corresponding achievement target
Describe your results in enough detail to prove whether you have met, partially met, or not met your achievement target.
It is not necessary to provide any interpretation of your data in your findings summary.

39. Findings
(Optional) Attach documents to support your data. These can include survey instruments or results, reports, committee members and minutes from meetings, etc.

40. Examples of Findings Statements
Achievement Target: Overall mean score of students from program meets or exceeds state average score.
Findings: The overall mean score of students from the Teaching, Learning, and Culture program exceeded that of the state average score of the state certification exam. Results: Program overall mean scaled score—91.50, State overall mean scaled score—79.13.

41. Examples of Findings Statements
Achievement Target: Decrease by 40% the number of undeclared students from the entering major (Fall 2008) to current major (Fall )
Findings: 34% of undeclared students who had registered for Fall classes by the beginning of June had declared a major.

42. Examples of Findings Statements
Achievement Target: 10 campus-wide workshops conducted in November and December of 2008.
Findings: 8 campus-wide workshops were conducted in November and December of 2008.

43. Analyze your Findings
Reflect on what has been learned during an assessment cycle
Identify areas that need to be monitored, remediated, or enhanced
Three key questions are at the heart of your analysis:
What did you find and learn?
So What does that mean for your academic program or support unit?
Now What will you do as a result of the first two answers?

44. Analyze your Findings You will want to reflect on the following areas:
Student Learning Outcomes, if applicable
Program Outcomes, if applicable
The Assessment Process

45. Action Plan
After reflecting on the findings, you and your colleagues should determine appropriate action to improve the program. This will lead to at least one action plan.
Actions outlined in the action plan should be specific and relate directly to the outcome and the results of assessment.

46. Action Plan Recommendations
We do not recommend having an assessment report without any action plans.
SACS 3.3.1
We do recommend that you keep the number of action plans manageable.

47. Annual Reporting
The final step in your assessment report is to consider how your program contributed to one or more of the following:
Departmental goals or imperatives
College goals or imperatives
University mission, goals or imperatives

48. Rubrics for Evaluation
Assessment Plans
Mission/Purpose Statements
Outcomes
Measures and Achievement Targets
Findings/Report
Achievement Targets
Findings
Analysis
Action Plans

49. Take-Home Messages You do not have to assess everything every year
Modify something already being done that is meaningful to the program
Summarize your Findings
Limit your Action Plans and keep them manageable
Consider how your program fits within the university’s mission
Be flexible—this is an iterative process

50. References
The Principles of Accreditation: Foundations for Quality Enhancement. SACS COC Edition.
Banta, Trudy W., & Palomba, C. (1999). Assessment Essentials. San Francisco: Jossey-Bass.
Banta, Trudy W. (2004). Hallmarks of Effective Outcomes Assessment. San Francisco: John Wiley and Sons.
Walvoord, Barbara E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey- Bass.
Assessment manuals from Western Carolina University and Texas Christian University were very helpful in developing this presentation.
Putting It All Together examples adapted from Wright State University Libraries Assessment Plan, UHCL President’s Office Assessment Plan

51. Resources
North Carolina State University _others.htm
University of Central Florida—Handbook for Admin Units
Texas Christian University
Each Other
OIA consulting

52. WEAVEonline Overview Amber Malinovsky, Assistant Director
Office of Institutional Assessment

53. Questions?

54. One Minute Evaluation What was the most valuable thing you learned?
What is one question that you still have?
What do you think is the next step that your program needs to take in order to implement systematic program assessment?

55. Contact Us Our Staff Our Website TAMU Assessment Conference Website
Dr. Loraine Phillips,
Amber Malinovsky,
Our Website
TAMU Assessment Conference Website

56. Plenary Speakers: Dr. Belle S. Wheelan Dr. Clifford Adelman
February 21-23, 2010
College Station, Texas
Plenary Speakers:
Dr. Belle S. Wheelan
Dr. Clifford Adelman
Registration now open!