Title Slide Making Evaluation Judgements A quick overview He Oranga Poutama Provider Hui, October Te Manukanuka o Hoturoa Airport Marae Nan Wehipeihana, Kataraina Pipi and Kate McKegg Grappling with Uncertainty in Innovative and Complex Settings: Weaving Quality in Developmental Evaluation Kataraina Pipi, Nan Wehipeihana and Kate McKegg

Introduction Kataraina, Nan and Kate are the developmental evaluators for the SPARC He Oranga Poutama Programme – A culturally based sport and recreation programme delivered in 12 locations in New Zealand In this presentation we explore quality in a cultural context – on a developmental evaluation journey

The programme aims to increase participation in sport and recreation in culturally specific ways – ‘As Māori’ A programme review resulted in a ‘re-visioning’ A new focus on culturally led and driven participation Key drivers: – Tino rangatiratanga (self determination) – Drawing on 20 years of Māori socio-cultural and political development The story of He Oranga Poutama

Why developmental evaluation? Uncertainty around the new programme concept and direction – Exploration of ‘as Māori’ participation in sport and recreation Traditional evaluation methods not suitable – Emergent understanding of culturally based indicators and outcomes No formal culturally based evaluation evidence, lack of data available to measure key cultural concepts and variables Complex environment – contracting economy, political environment, changing organization

What does quality look like in Developmental Evaluation in indigenous contexts? Kataraina Pipi

Quality evaluation in a cultural context Cultural practice at the centre of our process The right thing to do Context + relationships = quality Cultural norms and values guide our practice as evaluators

Cultural tools – the key to generating quality information Cultural icebreakers – Creating connections Cultural critical thinking tools – Tapping into Māori ways of thinking and being Cultural contemporary evaluative synthesis tool – Cultural analytic lens

Cultural ‘ice breakers’ create connections In pairs – one thing you love about the place you call home In fours – one thing we have in common The point is… – Relationships matters!

Values Exercise ‘As Māori’ critical thinking (adapted from Te Ngaru Learning Systems) What were the messages you received from your grandparents about being Māori? What were the messages you received from your parents about being Māori? What messages would you like your grandchildren to have about being Māori? What messages are you giving / have you given your children about being Māori? Being Māori

‘Surfaces’ implicit values and assumptions Draws out implicit values and assumptions Makes explicit – The things we privilege – The things we prioritize – The source of our ‘valuing’ It is our belief systems which determine the outcomes that we value – and bring to evaluation

‘As Māori’ evaluation synthesis activity ActivityEvaluative CriteriaCredible evidence of excellence Data / tools Provide line dancing classes for kaumatua/k uia (elders) at the marae (traditional meeting place) 1.The number of kaumatua and kuia who turn up to the first class 2.The extent of positive feedback from kaumatua and kuia following the class 1.The numbers of kaumatua and kuia who come back for more classes 1.The level of interest from other marae High turnout of kaumatua and kuia at classes There is lots of positive feedback from kaumatua and kuia about the classes ‘It’s just one of those events you don’t miss’ High numbers are sustained over the course of the classes. Kaumatua and kuia spread the word about the classes to others. Demand from other marae to run the classes Registrations Attendance records Kaumatua and Kuia feedback Other whānau (family) feedback Videos and photos Attendance records Follow up calls Log of enquiries (phone, and verbal) from others

Cultural traditions and values at the heart of quality… Cultural traditions and practices privileged Context + relationships = quality

‘Talking Past Each Other’: The language of the developmental evaluator in indigenous contexts and its link to quality Nan Wehipeihana

It is through language that we come to know, understand and give expression to all aspects of who we are (Takao et al, 2010) It’s through language that we give meaning to the world… It’s in our language that our values are expressed… It’s in our language that identity is embedded The centrality of language

“It’s the damn English…” (Kirimatao Paipa) The importance of language for understanding The importance of language for validity Inaccurate or incomplete understandings of culture introduce systematic error that threatens validity Evaluation quality is about accuracy and trustworthiness of our understanding and the evaluative judgements we make

Examples from the field that worked Use the word ‘Poutama’ – Depicts a stairway or journey – Signal developmental or ongoing progression Mauri Ora – An expression of wellbeing – likened to excellence Rehua – Literally a constellation of stars – draws on traditional knowledge – The rehua (the constellation) is a visual anchor and reference for the evaluation framework

Evaluation language - that didn’t work ‘Rich description’ – For some, funders historically not interested in the cultural detail – For others, the term was not understood ‘Credible evidence’ – Reframed as what would you see, hear, feel – that would convince critics of success? – What is valid in cultural contexts and settings – to elders, to communities, to tribes? ‘Criteria’ – ‘the values that we practice, that we can measure’ (Thomas Tawhiri 2010)

Language is the connector Language can marginalize, disempower and disenfranchise Sometimes meaning is lost in translation – Start with language that connects It’s not about dumbing down… – Use of Māori language, metaphor and cultural symbols as a starting point for understanding  quality in evaluation

Drawing on deep values to ensure evaluation quality in emergent and uncertain contexts Kate McKegg

Applying developmental evaluation in a cultural context Draws on cultural values, needs, strengths and aspirations Cultural concepts, language and values are foundational within evaluation thinking, processes, tools, frameworks, data collection, reporting etc Achieves genuine and valid evaluation experience in the eyes of our communities

Applying developmental evaluation questions in a cultural context DE QuestionApplied Example (HOP) What?What’s being developed?What does / might ‘as Māori’ look like in the sport and recreation sector? What’s emerging? What sense can we make of emerging issues, evidence, data about this development? What are other’s doing in the ‘as Māori space? What are the patterns of ideas and practice emerging from the provider communities about ‘as Māori’ So what?What’s the value / significance / importance of what we’re doing and achieving? Whose values are we applying to our judgments? And what importance do different values place on what’s emerging? Now what?What does this mean for how we should now act? Into the future? How will be take our new understandings of ‘as Māori’ and apply these to the next phase of development?

The cultural framework (Rehua) dimensions By MāoriRefers to the extent of rangatiratanga / control by Māori in the governance, management and delivery of an initiative For MāoriEmphasises the importance of whānau, hapu, iwi, (family and tribal relationships) and other Māori collectives bought together for Māori purposes With te reo me ona tikanga Points to the importance of language and customs as central to the survival and affirmation of what it means to live ‘as Māori’ In/on places of cultural significance to Māori From ancestral domains (e.g., rivers, mountains and gathering places such as meeting houses), to more contemporary places such as kōhanga reo, kura etc. founded on Māori principles ThroughThose activities that are distinguished in uniquely cultural ways e.g., traditional and contemporary sport and recreational activities

Applying the framework Used to inform program design – to ensure that program activities aligned to the ‘as Māori’ outcome goal / vision Informed the development of evaluative criteria Influenced and changed the funders contract process Informed baseline data collection Reporting

How significant is the location? Extremely significant Strong connections to whakapapa, whanaungatanga and history (identity as Māori) It is the ideal place and cultural fit for the activity For example, it’s the right environment because you don’t have to explain it, it belongs there, there is no questioning of the authenticity The location or place enriches and enhances (strongly adds to) the overall experience For example, tikanga tells you it’s the right place because of the mauri and the wairua of the place Very significant There are some connections to whakapapa, whanaungatanga and history The location or place adds to or enhances the overall quality but connections are more general rather than specific It’s a culturally safe environment For example, it’s a place where it still feels OK – no one can say what are we doing here? There is no question of the appropriateness of undertaking the activity in that place

Reporting Cultural framework reflected in data collection and reporting

Culture is foundational to quality evaluation Valuing, affirming and privileging cultural practices, relationships, language and concepts leads to credible, valid evaluation Developmental evaluation is responsive to culture and cultural context