1. Assessing Learning-Centered Leadership Andrew C. Porter University of Pennsylvania Joseph Murphy, Ellen Goldring, & Stephen N. Elliott Vanderbilt University Spring 2008

2. Acknowledgments The VAL-ED instrument will be published and distributed by Discovery Education Assessment starting in July 2008. We are authors of the VAL-Ed, and while we have made every effort to be objective and data-based in my statements about this instrument in this presentation, readers should judge the facts and related information materials for themselves and make independent decisions regarding the use of the instrument. 2008

3. Outline of Presentation Background on VAL-ED The VAL-ED instrument Psychometric properties Score reports VAL-ED and professional development 2008

4. Background on VAL-ED

5. The Case for Leadership Assessment Most school leadership improvement focuses on professional development, mentoring, licensing policies, and standards. Minimal attention has been paid to assessment, feedback, and subsequent action. Leadership assessment and feedback is an important missing link to improving and strengthening school leadership. 2008

6. Learning-Centered Leadership Leaders should be assessed on leadership behaviors associated with student learning. Learning-centered leadership is leadership for student performance. Learning-centered leadership is the framework for our leadership assessment system. 2008

7. Our Conceptual Model 2008

8. Leadership Behavior Framework 2008

9. Definitions of Core Components High Standards for Student Learning —There are individual, team, and school goals for rigorous student academic and social learning. Rigorous Curriculum (content) —There is ambitious academic content provided to all students in core academic subjects. Quality Instruction (pedagogy) —There are effective instructional practices that maximize student academic and social learning. Culture of Learning & Professional Behavior —There are integrated communities of professional practice in the service of student academic and social learning. There is a healthy school environment in which student learning is the central focus. Connections to External Communities —There are linkages to family and/or other people and institutions in the community that advance academic and social learning. Performance Accountability — Leadership holds itself and others responsible for realizing high standards of performance for student academic and social learning. There is individual and collective responsibility among the professional staff and students. 2008

10. Definitions of Key Processes Planning—Articulate shared direction and coherent policies, practices, and procedures for realizing high standards of student performance. Implementing—Engage people, ideas, and resources to put into practice the activities necessary to realize high standards for student performance. Supporting—Create enabling conditions; secure and use the financial, political, technological, and human resources necessary to promote academic and social learning. Advocating—Promotes the diverse needs of students within and beyond the school. Communicating—Develop, utilize, and maintain systems of exchange among members of the school and with its external communities. Monitoring—Systematically collect and analyze data to make judgments that guide decisions and actions for continuous improvement. 2008

11. The Development of VAL-ED The development of VAL-ED has been supported by a 3-year grant from The Wallace Foundation. Three phases of our work:  Phase 1 – Leadership conceptualization and assessment system development  Phase 2 – Field testing the behavior rating scale and exploring its properties  Phase 3 – Dissemination of results and products 2008

12. The VAL-ED Instrument

13. The instrument consists of 72 items defining six core component subscales and six key process subscales. Principal, Teachers, & Supervisor provide a 360-degree, evidenced-based assessment of leadership behaviors. Respondents rate effectiveness of 72 behaviors on scale 1=Ineffective to 5=Outstandingly effective. Each respondent rates the principal’s effectiveness after indicating the sources of evidence on which the effectiveness is rated. Two parallel forms of the assessment facilitate measuring growth over time. The instrument will be available in both paper and online versions. 2008

14. Purpose & Uses The VAL-ED can be used as part of a comprehensive assessment of the effectiveness of a leader's behaviors. The VAL-ED reports principal performance through  Norm-referenced scores and  Criterion-reference scores. VAL-ED can be used annually or more frequently to:  Facilitate a data-based performance evaluation,  Measure performance growth, and  Guide professional development. 2008

15. Implementation Identify respondents and invite participation. Discuss use of results & confidentiality. Decide paper or online version. Time and Timing  Average respondent requires 20 to 25 minutes.  Schedule completion after respondents have had a reasonable time to observe/experience the leader’s work and its effects on the school. Designate person(s) to manage collection and submission of response forms, if paper version used. Ensure teacher confidentiality. 2008

16. Directions for Completing Rating Scale 2008

17. An Example Set of Responses 2008

18. Psychometric Properties

19. Assessing Learning-Centered Leadership: The VAL-ED vision… A leadership assessment system that has the following properties: Works well in a variety of settings and circumstances, Is construct valid, Is reliable, If feasible for widespread use, Provides accurate and useful reporting of results. Is unbiased, Yields a diagnostic profile for summative and formative purposes. Can be used to measure progress over time in the development of leadership, and Predicts important outcomes. 2008

20. Psychometric Evidence Item and response scale development  Based on review of learning-centered leadership literature and alignment to ISLLC standards.  Critiqued by education leaders and leadership researchers. Item sorting study  Established content validity by asking education leaders to sort the items into 36 cells. Cognitive interviews of paper/pencil version  Two rounds of cognitive interviews in three districts each.  Three respondents evaluated the format and items. Nine-school pilot test  Estimated reliability of each of 12 scales.  Established construct validity through factor analysis.  Established face validity through questions to respondents. 2008

21. Psychometric Evidence Cognitive interviews of online instrument including revisions based on 9-school pilot Bias review  Submitted to urban districts to evaluate language. 11-school pilot test  Confirmed changes made after 9-school pilot test. 300-school field test  Conducting differential item functioning to determine biases.  Establishing norms.  Setting performance standards. Proficiency standard setting  Using bookmark approach with 24 education leaders. 2008

22. Cognitive Interviews—Validity First round  Sources of evidence cumbersome  Item stem lost  Instructions “wordy” Modifications  Instructions bulleted  Stem included in each item  “Not done” added Conclusions  VAL-ED captured all the relevant leadership behaviors  Response scale was clear  Respondents able to complete task on their own Cognitive interviews of online prototype 2008

23. Methods Confirmatory factor analysis  High Parsimonious Goodness of Fit Index (.93-.96)  First, second, and third-order factor loadings salient Responses to final survey questions  Understood items, sources of evidence Teacher and principal effectiveness ratings were correlated r =.47 Cronbach’s Alpha above.92 for 108-item-form scales Scores high (generally >4 on the five-point effectiveness scale) 9-School Pilot Study—Validity & Reliability 2008

24. 11-School Pilot Study—Validity & Reliability Methods High agreement between schools  r =.79 for teacher-principal,.51 for principal- supervisor,.68 for supervisor-teacher Mean effectiveness ratings lower, more variable Alpha above.89 for all 72-item-form scales 2008

25. Respondent Feedback 2008

26. Score Reports

27. Interpretation of Rating Scale Results Descriptive Analysis  Total Score  Core Components Subscale Scores  Key Process Subscale Scores Norm-Referenced Profiles  Principal  Teacher  Supervisor  Total respondent composite Criterion-Referenced Profiles  Distinguished  Proficient  Basic  Below basic 2008

28. VAL-ED Results: Performance Descriptors Distinguished  A distinguished leader exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are virtually certain to influence teachers to bring the school to a point that results in strong value-added to student achievement and social learning for all students. Proficient  A proficient leader exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for all students. Basic  A leader at the basic level of proficiency exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for some sub-groups of students, but not all. Below basic  A leader at the below basic level of proficiency exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are unlikely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for students. 2008

29. Aggregated Effectiveness Ratings 2008

30. Comparisons Across Respondent Groups 2008

31. Comparisons Across Respondent Groups 2008

32. VAL-ED and Professional Development

33. Cell-by-Cell Feedback 2008

34. VAL-ED and Professional Growth Cell-by-cell feedback highlights up to 6 potential areas of growth. Behaviors from these 6 domains are listed. Areas of growth provide principals with information about key targets for professional development. 2008

35. Leadership Behaviors for Possible Improvement Example of a potential area of growth: 2008

36. Supporting Research & Publications Goldring, E., Porter, A.C., Murphy, J., Elliott, S.N., & Cravens, X. (2007, March). Assessing learning-centered leadership: Connections to research, professional standards, and current practice. New York, N.Y.: Wallace Foundation. Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (2007). Leadership for learning: A research-based model and taxonomy of behaviors. School Leadership & Management, 27 (2), 179-201. Murphy, J., Elliott, S.N., Goldring, E., & Porter, A.C. (in press). Leaders for productive schools. In M. Brundrett & M. Crawford (Eds.), Developing school leaders: An international perspective, London: Routledge. Murphy, J., Elliott, S.N., Goldring, E.B., & Porter, A.C. (2006). Learning-centered leadership: A conceptual foundation. New York, NY: Wallace Foundation. Porter, A.C., Goldring, E.B., Murphy, J., Elliott, S.N., & Cravens, X. (2006). A framework for the assessment of learning-centered leadership. New York, NY: Wallace Foundation. Murphy, J.F., Goldring, E.B., Cravens, X.C., Elliott, S.N., Porter, A.C. (2007, August). The Vanderbilt Assessment of Leadership in Education: Measuring Learning-Centered Leadership. Journal of East China Normal University. These and other publications are all available for download at http://www.vanderbilt.edu/lsi/valed/featured.html. 2008

37. To Learn More … Visit: http://www.val-ed.comhttp://www.val-ed.com Contact : Discovery Education/ThinkLink Assessment 2416 21st Avenue, South, Suite 300 Nashville, TN 37212 Jackie Shrago (o) 615-324-6091 (e) Jackie_Shrago@discovery.com or William Dycus (o) 615-324-6090 (e) William_Dycus@discovery.comJackie_Shrago@discovery.comWilliam_Dycus@discovery.com 2008