1. Real-time Student Assessment: A Shared Commitment to Improve Currently Enrolled Students’ Equitable Performance
Peggy Maki
Education Consultant Specializing in
Assessing Student Learning
Workshop Presented at Syracuse University
September 11, 2017

2. Integrated and Enduring Learning
Cognitive
Affective
Forms of Representation
Within Different
Contexts
Psychomotor

3. How well do your students…
Integrate
Transfer
Analyze
(Re)Apply
Re-use
Synthesize
Restructure previous incorrect learning…

4. Within a course or module or learning experience?
Along the chronology of their studies and educational experiences?
From one subject or topic or focus or context to another one such as from an exercise, to a case study, to another course, to an internship?

5. What Real-time Student Assessment Asks….
What chronological barriers or difficulties do students encounter as they learn--from the moment they matriculate or transfer or re- enter the University?
How and when will you identify and address those barriers—sometimes persistent barriers-- so that students across your demographics equitably progress toward achieving high-quality outcomes?

6. Example: What’s The Problem in Physics?
Through the use of concept inventories, it has been well documented that entry-level physics majors hold onto incorrect understanding of physics concepts—even ones that have been historically proven to be incorrect.

7. Despite all attempts to correct these incorrect concepts, such as incorporating more labs into early courses, students typically continue to draw on their incorrect understanding years after their early coursework often accounting for their weak performance as they progress in their studies or resulting in their dropping out of the major.

8. How to restructure incorrect understanding of physics concepts became the work of physics faculty at the University of Colorado,Boulder (PhET project).

9. University of Colorado, Boulder: PhET Project

10. Foci The Case for Real-time Student Assessment (RTSA)
A. The National Picture…11
B. The Current State of Assessment…20
C. Identification of Learning Obstacles or Barriers along
Students’ Undergraduate and Graduate Pathways…26
5 Core Learner-centered Commitments That Support RTSA…38
The 6 Guiding Principles and Related Processes That Underlie Continuous Reporting and Acting on Assessment Results…56
Technology That Provides On-time Access to and Visual Representation of Students’ Learning…70
Possible Approaches to Taking RTSA to Scale…77
Works Cited...79

11. I. The Case for RTSA A. The National Picture
B. The Current State of Assessment
C. Obstacles/Barriers to Student Learning

12. I.A. The National Picture
Continuing diversification of our student demographics (representative of our democracy),representing broad ranges of academic preparation and readiness and personal needs

13. Of High School Graduates Amer. Indian/Alaska Native
Projections
Of High School Graduates
Amer. Indian/Alaska Native
Asian/Pacific Islander
Black Non-Hispanic
Hispanic
White
Non-Hispanic
184,201
205,657
699,757
1,140,483
2,441,828
2,426,542
2,517,313
4,248,975
8,872,046
7,389,783
Source: Western Interstate Commission on Higher Education Knocking at the College Door: Projections of High School Graduates.

14. Persistent gaps in achievement and degree completion rates between historically represented and historically underrepresented students’

15. Degree Attainment Rates for U. S
Degree Attainment Rates for U.S. Residents (ages 25-64) by Population Group as of 2013
U.S. Census Bureau American Community Survey PUMS Files

16. The nation’s dependence on an educated citizenry from across our student demographics to ensure our prosperity, long-term growth, and attentiveness to our democratic values

17. High demands for associate- and bachelor-degreed students with relevant 21st-Century skills to address evolving needs of the workplace and challenges of globalization

18. Employers’ Focus on High-quality
Achievement in Graduates ? ? ? ? ?

19. Projections of Workforce Needs Requiring a Degree
“By 2018, the postsecondary system will have produced 3 million fewer college graduates than demanded by the labor.” To meet projected employment needs, Carnevale, Smith, and Strohl project that “our colleges and universities will need to increase the number of degrees they confer by 10 percent annually” (2014. p. 16).

20. I.B. The Current State of Assessment
“Assessment is certainly being conducted, evidence is being generated, and reports are being written and filed.  But translating that work, and the significant energy and resources it requires, into real improvements for learners continues to be a challenge.”
Hutchings, et als ”AAHE Principles of Good Practice: Aging Nicely.” NILOA Resources.

21. “Although more assessment evidence is available on campuses than ever before, information is not as widely shared as it should be and using it to guide institutional actions toward improving student outcomes is not nearly as pervasive as it could be.”
Kuh, et als. 2015, 122.

22. 2016 Gallup Poll of 654 CAOs and 1,671Faculty Reported by IHE in 2017
Attitudes on Assessment
Statement
% Private Provosts Agree
% Public Provosts Agree
% Community College Provosts Agree
% Faculty
Agree
Assessment has improved the quality of teaching and learning.
56%
46%
50%
27%
Assessment is more about keeping accreditors and politicians happy than it is about teaching and learning.
20%
35%
37%
65%

23. Provosts’ Views of Where Institutions Are “Very Effective”
Area
Private
All Public
Community Colleges
Identifying and assessing student outcomes
32%
29%
31%
Using data to aid and inform decision making
20%
30%

24. Doubts on Data-Driven Assessment
Statement
% Faculty Members Who Agree/Strongly Agree
% Academic Tech Administrators Who Agree/Strongly Agree
Assessment efforts seem primarily focused on satisfying outside groups.
65%
46%
Faculty members play meaningful role in planning assessment.
37%
38%
My college has meaningful discussion on how to use assessment data.
35%
These assessments have improved quality of teaching and learning at my college.
27%
34%

25. Major Sources of Faculty Frustrations
Lack of time, resources, rewards
Lack of meaningful role in determining what to assess
Fear of standardization that ignores disciplines and institutional missions
Conviction that they already do assessment
Fear of impact on minority-serving institutions
Sense that elite institutions are exempt
Jaschik, March, 2017

26. I.C. Identification of Learning Obstacles or Barriers along Students’ Undergraduate and Graduate Pathways
Persistent patterns of underperformance in general education outcomes such as quantitative reasoning, higher order thinking, writing (Wabash Study, Arum and Roksa, Multi State Collaborative). (See also Handout 1: AAC&U survey results)

27. Percent Students Forget When They Log off or Leave Your Learning Experience
30%

28. Barriers and Obstacles to Students’ Learning
Learners create meaning: egocentricity, sociocentricity, narrow-mindedness, routinized habits

31. Deep learning occurs over time—transference
People learn differently and may hold onto folk or naive knowledge, incorrect concepts, misunderstandings, false information
Deep learning occurs over time—transference

33. (curricular-co-curricular)
Transfer of new knowledge into different contexts is important to deepen understanding
(curricular-co-curricular)
NRC, 2002

34. Meta-cognitive processes are a significant means of reinforcing learning (thinking about one’s thinking)
Learning involves creating relationships between short-term and long-term memory

35. Research on…..
Threshold Concepts: pathways central to the mastery of a subject or discipline that change the way students view a subject or discipline, prompting students to bring together various aspects of a subject that they heretofore did not view as related (Land, Meyer, Smith).

36. Practice in various contexts creates expertise or enduring learning
Surface Learning (inert)
Deep Learning(activated)

37. Learning Progressions: knowledge-based, web-like interrelated actions or behaviors or ways of thinking, transitioning, self- monitoring. May not be developed successfully in linear progression thus necessitate formative assessment along the trajectory of learning. Movements towards increased understanding (Hess).

38. II. Five Core Learner-centered Commitments that…
Altogether form a canvas that stretches over an outcomes-based framework
Support and facilitate real-time student assessment that
monitors/gauges continuously students’ progress toward achieving a high-quality degree at the institution- and program-levels; and
triggers nimble development of short- and longer-term practices or interventions to improve students’ learning from point of matriculation, transfer, or re-entry to point of graduation

39. Commitment 1
A Shared Commitment to Close Measurably Existing Achievement and Graduation Gaps
An inclusive commitment to real-time student assessment stretches across an institution and engages internal stakeholders at multiple levels and for various purpose— from institution or system leaders to administrators, academic leaders, full- and part-time faculty, the institution’s network of experts who contribute to or support student learning and currently enrolled students—beneficiaries of this assessment approach.

40. Quality: What Does Quality Achievement Look Like in Student Work?

41. Establish transparent expectations for all students.
Commitment 2
Agreement on the Language of Outcomes and Scoring Rubrics to Identify Continuously Patterns of Performance and Underperformance
Outcomes:
Generate collaborative discussions across a campus and across silos about shared expectations for student work.
Provide students with a compass for learning (Orientation to Institution/Program).
Establish transparent expectations for all students.

42. DQP/TUNING Standards and Criteria of Judgment AAC&U LEAP
21st-Century National Outcomes-Based Frameworks
Standards and Criteria of Judgment
AAC&U LEAP
DQP/TUNING
Competency-based Education
Specialized Accreditation
See Handout 2

43. Degree Qualifications Profile

44. Outcomes-based Frameworks
Equity
Transparency
Quality
Explicitness

45. Use Collaboratively Agreed upon Rubrics to…
Monitor, report, and improve patterns of under-performance as students progress towards equitable achievement of a high-quality degree.
Assess the multiple dimensions of student learning achievement in a range of contexts.
Develop a wide-angle lens to document students’ achievement along their educational pathways—beyond course grades—based on their authentic work.

46. Without an agreed upon outcomes-based framework and aligned criteria and standards of judgment there will be uneven attention to students’ equitable progress toward achieving the institution’s GE outcomes or those of its programs.
Both agreed upon outcomes and aligned rubrics:
Monitor students’ equitable progress;
Provide consistent feedback to students, holding them accountable for their learning and future improvements; and
Prompt students’ self-reflection on their progress, often in the presence of a trusted “other.”
(See Handouts 3 and 4)

47. Example: We held norming sessions using the
WC VALUE rubric that served as critical times to involve a wider circle of faculty and other contributors to student learning and to build a common language, a collective set of expectations, and shared tools for teaching writing across the curriculum. (Brockport College, SUNY)
Maki, 2015

48. Commitment 3
Coherence across The Curriculum, Co-Curriculum, and Other Educational Experiences
Students examine and compare information from various sources to evaluate reliability, validity, accuracy, timeliness, and point of view or bias. (I)
Students examine and compare information from various geological sources to evaluate reliability, validity, accuracy, timeliness, and point of view or bias. (P)
In searching a ProQuest assignment in Geology 102, students examine and compare information from various geological sources to evaluate reliability, validity, accuracy, timeliness, and point of view or bias. (C)

49. Collaboratively Agreed upon Learning Outcomes at Multiple Levels Foster Enduring Learning
Institution-level Outcomes
Program- or Dept.-Level Outcomes
Course/Experience Outcomes

50. Excerpt of Co-Curricular Map
Introduce (I)
Reinforce (R)
Emphasize (E)
Multicultural Week
Leadership Council
Community Service
Outcome 1:
Speak to different audiences for a range of purposes
(I) Group Activities involving Interpersonal Communication
(I) Community Presentations
(R) Collaborative Projects with Surrounding Community
Outcome 2:
Outcome 3:
Outcome 4:
(See Handout 5)

51. Commitment 4
Alignment of Courses, Educational Experiences, and Assignments with Outcomes and Standards and Criteria of Judgment
Institution’s General Education Outcomes
Program
Outcomes
That Include
General
Education
Course or Educational Experience Outcomes That Include Relevant General Education Outcomes
Agreement on Criteria and Standards of Judgment
Course/ Experience Design:
*Pedagogy
*Sequencing
*Context for Learning
*Input from Other Experts
Assignment Design:
*Aligned
Scaffolded
Assignments or
Exercises
*Chronological
Feedback and
Student Self-
reflection
*Major
Assignment
Demonstrating
Targeted
Outcome(s)
(See Handout 6)

52. Example:
GE Task Force faculty representatives mentored faculty in each baccalaureate program to develop signature assignments, embedded within core courses in each undergraduate program and designed to measure achievement of the five ILOs. (Brandman University)
Maki, 2015

53. Examples of Quantitative Literacy across the Curriculum:
English 101 assignment that includes analyzing statistics and graphs in research papers
• Anthropology assignment in a social action
project that includes justifying choices with
surveys and data, also required in writing a
grant to get funding for the project.
• Introduction to Nutrition assignment that
includes personal diet analysis with
recommendations for ways to improve that
diet.

54. Criminology assignment that includes analyzing crime data through the lens of individuals’ different roles such as a police chief or a journalist.
• ESL assignment that includes using self-
collected survey data to create and justify an
ideal community.
• Economics assignment that includes analyzing
the federal budget (Holyoke Community
College).

55. Commitment 5
Faculty Collaboration with The Institution’s Network of Experts, Such as Those in…
student affairs/ support services
academic support services
registrar’s office
library and information resources
institutional research
assessment
professional development
online education
student enrollment and management

56. 1. Internally Driven and Motivated
III: The 6 Guiding Principles of RTSA and Related Processes
1. Internally Driven and Motivated
A shared commitment to enrolled students’ equitable and measurable progress toward achieving a high-quality degree is driven by institutional mission and/or leadership

57. 2. Inclusive of Internal Stakeholders
Involvement of institutional leaders, faculty, administrators, education professionals (such as those in academic and student support, education or instructional technology and tutoring) and students to improve measurably students’ achievement of a high-quality degree.

58. 3: Bolstered by Collaboration That Harnesses Others’ Expertise and Practices
Sustained through ongoing collaboration and cooperation among institutional and administrative leaders, faculty, other campus experts, and students to advance all enrolled students.

59. 4: Anchored in Continuous Reporting and Interrogation of Assessment Results
End-of-semester real-time assessment results are displayed on two dashboards: program-level dashboards and an institution-level dashboard

60. Flow of End-of-Semester Assessment Results into Program- and Institution-level Dashboards
General Education Interventions or Practices to Improve Patterns of Underperformance Continuously in Collaboration with Major Programs (COPs)
Program-level Interventions or Practices to Improve Patterns of Underperformance Continuously (COPs)
Reported End-of-Semester Course- and Experience-Based Assessment Results

61. Performance Dashboard
Course or Education Experience Exit-Level Assessment Results Are Continuously Reported on Institution- and Program-Level Dashboards to Monitor Students’ Equitable Progress towards Achieving High-quality GE and major program outcomes (that also include GE outcomes).

62. This slide demonstrates how data can easily be aggregated in real time.62

63. This slide demonstrates how data can be visualized to make it easier to consume. Moreover, discussing data in a visual format can help facilitate and foster meaningful conversations regarding curriculum content, alignment, and delivery. 63

64. Used with Permission from Taskstream

65. Filter data by course
Used with permission from Taskstream

66. 5. Responsive to Students’ Needs in “The Present Tense”
Context-specific assessment results are nimbly used on-time to improve patterns of underperformance in the short-term and over the long-term as those patterns persist in enrolled students’ work.(See Handouts 7,8,9)

67. Rio Salado College uses a dashboard approach featuring green, yellow and red lights to indicate to students and their instructors whether they are on track towards success, failure or somewhere in between. The immediacy of this information allows problems to be addressed before it is too late in the semester to make a difference (Barrington, 2013)

68. 6. Institutionally Valued
Recognized and valued in an institution’s aligned rewards system.

69. Recognition may value, for example....
The importance of time--calendaring in days/times for collaborative interrogation leading to actions to improve learning;
The need for technological resources/support to facilitate the commitment; or
Contributions to teaching and learning in P&T criteria, such as identifying effective teaching strategies, taking them to scale to improve learning measurably, engaging in SOTL.

70. IV. Technology That Provides On-time Access to and Visual Representation of Students’ Learning
ePortfolio: A Real-time Record of Student Achievement

71. What the Future May Hold…..
Eportfolio or Learning Record
Criteria and Standards of Judgment
Signature Assignments
Milestone Work
Self-reflection
Evidence of Career and Life Experiences
Evidence of Co-curricular Experiences

72. Adaptive Technology Platforms
Integrated into Learning Management Systems (MyMathLab)
Developed as Independent Programs such as McGraw-Hill’s
ALEX and LEARNSMART

73. Data Analytics in Software Programs (Learning and Predictive)
Help Visualize Trends in Performance for Early Intervention
Provide Data about Each Student’s Progress
Build in Watch List Alerts that Let Educators Know Which Students Need Extra Attention

75. Dashboards for students

76. Analytics Tools Built into LMS Plugins
Ex.: Progress and Course Engagement (PACE) that tracks student progress in a course
alerting faculty to the need for intervention

77. V: Possible Approaches to Taking RTSA to Scale
Identify High-Risk Courses or Modules That Stretch Across Students’ Degree Pathways, From Point of Matriculation, Transfer, or Re-entry Into Higher Education to Point of Graduation
Identify High-Risk Learning Outcomes
Initially Reduce Intervals of Time Between Current Assessment Reporting Times to Identify and Improve Enrolled Students’ Chronological Barriers

78. Identify and Then Track the Most Vulnerable Cohorts of Students
Initially Assess Students’ Near-Graduation Work to Identify Patterns of Underperformance That Need to Be Addressed Longitudinally
Learn How Colleagues in Specialized or Nationally Accredited Programs Continuously Assess Their Students’ Progress and Use Results to Advance Enrolled Students to increasingly higher levels of performance

79. Works Cited
Barrington. R. ( 2013, August).“What Colleges Can Learn from Big Data.
Carnevale, A., Smith, N, and Strohl, J. (2014, June). Recovery: Job Growth and Education Requirements through Washington, D.C.: Georgetown University.
Hess, K. (2008). Developing and Using Learning Progressions as a Schema for Measuring Progress. National Center for Assessment,
Hutchings, Pat, Peter Ewell, and Trudy Banda.( 2012). ”AAHE Principles of Good Practice: Aging Nicely.” NILOA Resources. nt.htm
Jaschik, S. March 7, Provosts, Professors and the State of Assessment: Data from Inside Higher Ed Surveys. Taskstream Webinar
Kuh, G., Ikenberry,S., Jankowski,N., Cain,T., Ewell,P., Hutchings,P., Kinzie,J. (2015). Using Evidence of Student Learning to Improve Higher Education. CA: Jossey Bass.

80. Maki, P. 2017. Real-time Student Assessment
Maki, P Real-time Student Assessment. Sterling, VA: Stylus Publishing, LLC.
Maki, P Assessment That Works: A National Call, A Twenty- First Century Response. Washington, D.C.: AAC&U.
Meyer, J.H., Land, R., and Baillie, C Eds. Threshold Concepts and Transformational Learning. Rotterdam: Sense Publishers.
National Research Council Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C
Taskstream and TK20 have recently merged.
U.S. Census Bureau American Community Survey PUMS Files
University of Colorado, Boulder. PhET project.
Western Interstate Commission on Higher Education Knocking at the College Door: Projections of High School Graduates.