1. The World of Performance Measurement Purpose of the literature review Scope of review conducted Key Findings: Background information Performance measurement basics Academic, social and personal measures Different perspectives and experiences: o Provincial o National o International What findings mean for FNSSP work

2. Literature Review: Purpose Examine and summarize background material Inform the development of a Performance Measurement system for the Sioux Lookout District

3. Literature Review: Scope Web search of publications, presentations, web sites (associations, government, organizations) Training materials and sources Focused on performance measurement discipline Examined exemplars of academic, social and personal measures Different perspectives and experiences: Provincial National International

4. Overall Finding: NB Success Performance measurement is about measuring success. Before success can be measured, it needs to be defined. The development of Performance Measures must be based on a vision of success.

5. What is Success? There are many definitions of success: Academic Readiness Program vision NNEC vision Assembly of First Nations Secretariat Cynthia Wesley-Esquimaux’s presentation to Directors of Education February 2010

6. Visions of Student Success KERC Academic Readiness Project Definition of Success: What would constitute success for Sioux Lookout District Students The original vision set by the District Chiefs in establishing the Academic Readiness Project was to close the achievement gap of First Nation Students in the Sioux Lookout District on measurable assessments parallel to Ontario students. Since then DEPC has noted in its strategic planning process that success for students would be measured by achievement of academic and cultural success.

7. Visions of Student Success This definition of success would be evidenced by students : Reaching their individual potential Becoming self-sustaining members of the society Understanding and functioning in both cultures Conversing and being literate in both languages in the grade level as compared to the Ontario academic standards Attaining spiritual and cultural values held by the Sioux Lookout communities Developing skills necessary to become good providers Understanding global issues

8. Visions of Student Success NNEC Vision For our youth, a sound successful future with a viable and meaningful role in society, and their rightful place as First Peoples.

9. Visions of Student Success Assembly of First Nations Education Secretariat First Nations learners will achieve their full potential supported by a comprehensive system under First Nation jurisdiction that addresses their intellectual, spiritual, emotional and physical needs through quality lifelong learning, grounded in First Nations’ languages, culture, traditions, values and worldviews.

10. Visions of Student Success Assembly of First Nations Education Secretariat The point of measurement is not just to measure, it is to use the measurement system to show achievement of the vision of holistic lifelong learning, or to show what needs to be addressed to get closer to the vision.

11. Visions of Student Success: Assembly of First Nations Education Secretariat

12. Visions of Student Success Cynthia Wesley-Esquimaux Value of Education Gives our people the ability to develop new adaptable, holistic learning models that will help to map the relationships between learning purposes, life experiences and knowledge outcomes across an entire life span; affirm FN, Inuit and Métis values and beliefs; thereby providing a basis for developing positive new frameworks for measuring learning success. * Excerpts from Presentation to Sioux Lookout District: Directors of Education and District Planning Committee

13. Visions of Student Success Cynthia Wesley-Esquimaux Spiritual We have respect for education and school attendance and we relate our children’s education to the spiritual relationship we maintain to the land. Political The most positive and unique political reality we have as a people is good chiefs and councils in our territory. Social The most positive and unique social reality we have as a people is our strong sense of family and community. Economic The most positive and unique economic reality we have in our communities is that people want to better their lives and know education will help. * Excerpts from Presentation to Sioux Lookout District: Directors of Education and District Planning Committee

14. Visions of Student Success Cynthia Wesley-Esquimaux The demonstration of respect is an important trait in our young people. * Excerpts from Presentation to Sioux Lookout District: Directors of Education and District Planning Committee

15. Findings

16. What is a Performance Measurement System? (PM) A comprehensive and systematic process for collecting performance measures and using them to: Assess Monitor Improve results of programs and policies Should allow room for experimentation and innovation; test what works

17. Performance Measurement at a Glance Performance Measurement is used to continually assess efficiency, sustainability and progress to achieving goals and vision of success It is important to identify and to collect the right data The process of measuring performance requires the use of statistical and experiential evidence to determine progress toward specific defined objectives

18. What are the Benefits and uses of Performance Measurement? Used to set goals and standards Identify and correct problems Manage, describe and improve processes Document accomplishments Gain insight into and make judgments about effectiveness and efficiency Determine if fulfilling vision of success Provide measurable results to demonstrate progress towards goals and objectives

19. Definitions of Performance Measurement The process of developing measurable indicators that can be systematically tracked to assess progress made in achieving predetermined goals and using such indicators to assess progress in achieving these goals A system for assessing performance against stated goals relative to what was planned in terms of achievement of outputs and outcomes Involves ongoing data collection which measures inputs, outputs and outcomes over time

20. Definitions of Performance Measurement (cont’d) A system where pre and post comparisons are used to assess change The ongoing monitoring and reporting of accomplishments A process of assessing achievement of pre- determined goals and objectives through measurement of types of indicators: o inputs, processes, outputs and outcomes A way to produce objective, relevant information on performance

21. PM and Success If you don’t measure results, you can’t tell success from failure and you can’t claim or reward success If you can’t recognize success you can’t learn from it; if you can’t recognize failure, you can’t correct it If you can’t measure it, you can neither manage it or improve it

22. Good Performance Measures Provide a way to see if things are working Focus on what matters most to success Allow measurement of accomplishments Provide common language for communication Are explicit and defined (unit of measure, collection frequency, data quality, targets) Are valid to make sure measurement of the right things Are provable to ensure accuracy of data collection

23. PM Focus The focus should be on a holistic and balanced framework for PM what measures to choose and why? how to use these measures; what to do with the results? who will be using the results? who has to know the results?

24. PM Fundamental Questions What are you doing? How well are you doing it? How do you know how well you are doing it? How can you demonstrate to others how well you are doing?

25. Why Measure Performance? The fundamental purpose behind measures is improve performance, the number of measures that are not directly connected to improving performance are measures that are means to achieving that ultimate purpose.

26. Reasons to Measure Performance To evaluate To control To budget To motivate To celebrate To promote To learn

27. Basic Principles of PM If we don’t measure….. How do you know what to improve? How do you know if change is working? How do you know how to compare to others?

28. The most asked question in PM How do you find the right measure?

29. Steps to find the right measure Begin with the end in mind Which outcomes are most worth tracking? What differences are you trying to create? Be sensory and specific (if applicable) What would you see, hear, feel, do, taste or smell? Check the bigger picture What are positive and negative results of measuring a result?

30. Steps to find the right measure What is the evidence? What can you count to tell you the extent to which something is happening? Name the measure State exactly what you will be measuring

31. PM as a Cycle Measure: Track performance using selected indicators Report: Communicate performance internally and externally Learn: Extract knowledge from data; identify opportunities for improvement and make data driven decisions Improve: Implement changes to activities

32. What is the starting point once you have a vision of success? In order to measure anything you need: Baseline: A starting point for assessing changes in performance and for establishing goals or targets for the future Benchmark (standard): A level of achievement against which can measure own progress and can be used for comparisons

33. What are the key types of data? Comprehensive data collection will combine both: Quantitative data Qualitative data

34. What are the key types of data? Quantitative data: Information collected as numerical or statistical values Focus on quantities, percentages, frequencies, statistics, Provide information that is statistical and reliable and can be easily analyzed o Includes surveys, tests, reports, records

35. What are the key types of data? Qualitative data: Information gathered expressed in words and concerned with describing meaning Focus on descriptions of situations, events, interactions, behaviours and people’s experiences, attitudes and beliefs o Includes interviews, case studies, key informant interviews, focus groups, observations, literature reviews

36. Where will you find this data? Important to define key Data Sources on which performance measurement will be based Develop data collection methods Determine frequency and/or timing

37. What are the Key Terms used in PM? Goal Objective Indicator Measure Target

38. Goal A statement of direction, purpose or intent A broad statement of what is expected to be achieved sometime in the future A goal is a broadly defined objective Often goals and objectives are used interchangeably

39. Objective Something to be accomplished in a specific, well defined and measurable terms Should be achievable within a specified timeframe Are more specific than goals, but often are used interchangeably

40. Indicator Measure of progress toward specified outcomes or goals Lagging indicators: provide information about past conditions or outcomes Leading indicators: provide information on future conditions or outcomes Key Performance Indicator (KPI) A measurable factor linked to desired end outcome

41. Indicator A unit of measurement, or metric, for assessing short and long term results of programs and activities

42. Examples of Indicators Organizational Program Social impact Economic impact

43. Measure Specific quantitative or qualitative assessment of results Also referred to as metric

44. Target A desired numerical value related to a performance measure Sometimes is used to mean goal or objective

45. Inputs and Outputs Input measures show amount of resources, activities used Output measures report on result of given activity Show whether expected results have been achieved

46. Outcomes Result of an action or event Imply a change in behaviour or environment Intended to measure progress towards achieving an objective Types of outcomes: Initial Intermediate Long term

47. Outcomes (cont’d) Initial outcome: The result soon after an action, event or program has begun Intermediate outcome: A result during the action, event or program Is usually temporary or dynamic Intent is that intermediate outcomes will lead to long term outcomes Long term outcome: The result of an action, event or program after its conclusion This is usually the final result

48. Performance Measurement Tools SMART Goal SWOT Analysis Logic Model

49. SMART Goal Criteria for a focused and achievable goal: Specific Do you know exactly what you want to accomplish with all its details? (who, what, when how and why) Measurable Are you able to assess your progress? (How much/How many) Attainable Is it within your reach? (is it incremental? Is it within scope?) Realistic Is it doable? Can it be accomplished? Timely/time sensitive What is the timeframe? What is target date?

50. SWOT Analysis Assessment of external and internal environment to support formulation of goals. Strengths: focus on positive elements that help achieve objectives Weaknesses: focus on elements that are harmful to achieving objectives Opportunities: focus on external conditions helpful to achieve objectives Threats: focus on external conditions that could have major negative consequences on objectives

51. SWOT Analysis (cont’d) Internal Analysis: Examination of internal conditions to identify potential strengths and weaknesses Examples: Objectives, strategies, resources

52. SWOT Analysis (cont’d) External Analysis: Examination of outside conditions to identify potential opportunities and threats Examples: Demographic, economic, technological, social and cultural, political and natural forces

53. Template for SWOT Item InternalExternal StrengthsWeaknessesOpportunitiesThreats

54. Logic Model A visual representation displaying all inputs, outputs and outcomes and their linkages Used to link individual inputs to goals

55. International PM Methodologies and Standards Balanced Scorecard Management instrument that translates an organization’s mission and strategy into a comprehensive set of performance measures Measures organizational performance across different perspectives Baldridge Award Given by President of U.S. to different organizations who excel in 7 key areas Total Quality Management Set of management practices involving everyone in an organization in control and continuous improvement of work processes designed to meet customer quality expectations Six Sigma Methodology that uses performance measurement as a tool for systematically managing and eliminating process variations Objective is to deliver high performance, reliability and value to the end customer

56. Education Quality and Accountability Office (EQAO) Mandate: The agency is dedicated to enhancing the quality and accountability of the education system in Ontario and to work with the education community. This will be achieved through student assessments that produce objective, reliable information, through the public release of this information and through the profiling of the value and use of EQAO data across the province.

57. EQAO Activities Designs and implements a comprehensive program of student assessment within government-established parameters. Advises the Minister of Education on assessment policy. Develops and implements a provincial education quality indicators program. Leads Ontario’s participation in national and international assessments Promotes research in best practices in assessment and accountability. Reports to the Minister, the public and the education community on assessment and education issues and makes recommendations for improvement.

58. What testing does EQAO do? Develops and implements provincial assessment programs for primary, junior and secondary school students in Ontario. Assesses all students in Grade 3 and Grade 6 in reading, writing and mathematics. EQAO also administers two secondary school assessments. The Ontario Secondary School Literacy Test (OSSLT) is administered in March each year. The Grade 9 Assessment of Mathematics is administered in January to students enrolled in first-semester mathematics courses and in May/June to students enrolled in second- semester and full-year courses.

59. What testing does EQAO do? Coordinates Ontario’s participation in several national and international assessments: The Pan-Canadian Assessment Program (PCAP) is conducted by the Council of Ministers of Education, Canada and evaluates 13- year-olds’ skills in reading, mathematics and science. The Programme for International Student Assessment (PISA) is sponsored by the Organisation for Economic Co-operation and Development (OECD) and involves random samples of 15-year-olds in similar subjects. The Trends in Mathematics and Science Study (TIMSS) assesses student learning worldwide in Grades 4 and 8. The Second International Technology in Education Studies (SITES) focuses on computer use in primary and junior schools. *Both TIMSS and SITES are programs of the International Association for Evaluation of Educational Achievement (IEA).

60. National Tests Canadian Achievement Test (CAT) Measures the achievement in the basic skills commonly taught in schools across the country. The curricular areas measured are: reading, language, spelling, writing and math. This type of test indicates how well the child is doing in comparison to others at the same grade level across Canada and profiles the strengths and needs in the above areas. Canadian Cognitive Abilities Test (CCAT) Tests is a reliable general levels of students’ cognitive abilities in three areas: verbal, quantitative, and nonverbal. Canadian Tests of Basic Skills (CTBS) Tests language (vocabulary, reading, spelling, capitalization, punctuation and usage), mathematics (concepts, problems and computation) and work-study skills (visual materials and references)

61. National and International Tests Trends in International Mathematics and Science Study (TIMSS) Progress in International Reading Literacy Study (PIRLS) Programme for International Student Assessment (PISA) Pan Canadian Assessment Program (PCAP) Second Information Technology in Education Study (SITES)

62. Canadian Experience Examples British Columbia: Aboriginal Education Enhancement Agreement Purpose: to meet educational needs of Aboriginal students Performance Goals: measurable indicators to show annual improvement for Aboriginal students in these areas: o Improve academic achievement o Improve school attendance and extra curricular participation o Improve students self esteem

63. Canadian Experience Examples Sample school district Plan to Improve Student Achievement Purpose: to improve the achievement of Aboriginal students (elementary and high school) Key performance measures include data collection for attendance and transition rates

64. Canadian Experience Examples Closing the Aboriginal and Non-Aboriginal Education Gaps Report Study of student performance based on FSA tests in reading, writing and numeracy (grades 4 – 7) Student performance in elementary grades powerful indicator of successful graduation from high school Important to evaluate school performance in early grades Look at both on-reserve and off-reserve practices

65. Canadian Experience Examples Assembly of First Nations (AFN): School Based Performance Indicators Focus on development of FN indicators for performance measurement at the school level Includes description of: o Regional and National indicators o Culture and language indicators o School and community based indicators o Information and communication technology (ICT)

66. Canadian Experience Examples Alberta: First Nations, Métis and Inuit (FNMI) Education Policy Framework: Native Education Policy Review Includes goals for accessibility, student learning and achievement, education progression and preparation for entry into labour force Description of strategies and performance measures for each goal

67. Canadian Experience Examples Saskatchewan: Aboriginal Learning Knowledge Centre (AbLKC), Naturalizing Indigenous Knowledge Includes conceptual framework for indigenous knowledge (definition, cultural, aboriginal paradigms) Focus on goal of Aboriginal education Describes ‘the way it should be’ with regards to curriculum and indigenous knowledge

68. International Experience Examples Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA): A Measurement Framework for National Key Performance Measures Membership comprises State, Territory, Australian Government and New Zealand Ministers with responsibility for the portfolios of school education, early childhood development and youth affairs, with Papua New Guinea, Norfolk Island and East Timor having observer status

69. International Experience Examples A Measurement Framework for National Key Performance Measures Provides basis for reporting progress towards achieving National Goals by Australian students Draws upon agreed definitions in Key Performance Measures to report on: o Literacy, numeracy, science, information and communication technology, participation and attainment, civics and citizenship education Students characteristics and outcomes reported Describes definitions of KPMs, benchmarks and standards, targets and reporting

70. International Experience Examples New South Wales implementation plan for Schools National Partnerships : Goals: Improvements in literacy and numeracy achievement, including Aboriginal and disadvantaged students Increase in the proportion of students completing Year12 or AQF certificate, including Aboriginal and disadvantaged students Increase in the proportion of students participating in post school education or training Expansion of the pathways into teaching, including Aboriginal students and in subject shortage areas such as Maths and Science.

71. What do these findings mean for the development of Performance Measures for the Sioux Lookout District?

72. Types of Measures Usually Considered Academic Social Cultural Spiritual

73. Academic Readiness Project Academic, social, cultural, performance measures required for: Kwayaciiwin Bilingual and Bicultural Program Immersion program Ontario curriculum *Spiritual?

74. FNSSP The program goals are: Help First Nation educators plan and make improvements in 3 priority areas: o literacy, numeracy and student retention Enhance First Nation student learning experiences and improve student and school outcomes and results Assist schools in assessing and tracking how their students are performing

75. FNSSP Performance Measurement Implications The program aims to: Help First Nation educators plan and make improvements in 3 priority areas: o literacy, numeracy and student retention Enhance First Nation student learning experiences and improve student and school outcomes and results Assist schools in assessing and tracking how their students are performing 9

76. FNSSP Performance Measurement Implications Plan and make improvements requires goals, indicators, measures and targets Any improvement requires baseline information in order to track progress and change Tracking performance requires data sources that are monitored over time

77. Example PM Template GoalsIndicatorsMeasuresTargets Literacy Numeracy Student Retention Student Learning Experience Student school outcomes Student performance

78. We want to set the goals that… Our heart conceives Our mind believes and that Our bodies will carry out Jim Rohn

79. Conclusions Tasks to develop performance measurement system for the District : Definition of success: accepted by District Establishment of related goals, objectives, indicators, measures, and targets Identification of data requirements, data sources and collection and reporting processes and protocols

80. Next Steps Strategy and work plan for moving forward