Presentation on theme: "Evaluating learning about partnership with service users using Concept Mapping Roxana Anghel Joanna Fox"— Presentation transcript:
Evaluating learning about partnership with service users using Concept Mapping Roxana Anghel firstname.lastname@example.org Joanna Fox email@example.com
Level 2b outcome (Kirkpatrick, 1997; Barr, 2000) – Acquisition of knowledge – concepts, procedures and principles of working with service users and carers (in Carpenter, 2005:6) Focus: Evaluate the progress in learning about working in partnership with service users throughout the first two years of the BA course
Rationale The BA (Hons) SW degree - strong emphasis on training focused on partnership with service users (DoH, 2002) All social workers will learn and be assessed on partnership (p16) Core expectation that defines good practice (NOS, GSCC, QAA) Service users and carers consider very important for students to learn how to involve them in assessing their needs, and how to treat them with respect (Barnes et al, 2000; Anghel & Ramon, 2009) However a complex concept with varying definitions, in practice and education often implied rather than made explicit (Taylor & Le Riche, 2006)
Generally taught in an embedded or discrete way – no defined partnership curriculum, not explicitly defined as an outcome (Taylor & Le Riche, 2006). Anglia Ruskin - embedded and discrete approaches to learning (involvement of service users in curriculum delivery, and teaching about aspects of partnership on various modules). As learning about partnership is spread across various learning opportunities on the BA it is important to assess the students' gradual understanding of the concept and acquisition of a meaningful framework before becoming practitioners.
David Ausubel (1968, 2000) – Meaningful Learning (opposed to rote learning) is achieved when the learner assimilates new concepts into a pre-existing conceptual framework stored in the long-term memory the learner deliberately seeks the logical connections between new and old learning thus choosing a meaningful learning mindset the concepts progress in acquiring meaning as new concepts are added retention depends on the strength of the meaningful connections between the concepts stored meaningful learning is personal Joseph Novak (1984) – Concept Mapping
Concept Mapping (CM) A visual map of the summary of an individual's understanding of a knowledge domain (Novak & Canas, 2006) Concepts – perceived regularities in events or objects, or records of events or objects, designated by a label. Events, objects, even emotions or feelings (Freeman & Jessup, 2004) Differs from brainstorming and mind maps by making explicit the concept-links Initially designed as a research instrument used mainly in science education, but increasingly being used in a variety of teaching and research fields including social work (Anghel & Fox, 2008; Webber et al., unpublished)
T1 map before MA module Concept link Cross link
Research Design Before and After quasi-experimental longitudinal design - three measurements - two academic years CM and questionnaire at T1 (induction week BA); T2 (end year 1); T3 (end year 2) Questionnaire Demographic details Students previous experience of social work and Rating possible impacting factors Non-academic (media, workplace, personal experience) Classroom-based (various modules, lecturer teaching style, discussing with colleagues) Non-classroom-based academic factors (practice/ observational placement) no impact, little impact, some impact, great impact.
Research Process University colleagues consulted on the idea of outcomes evaluation, on the research design, and later on using CM. Some allies but initially difficult to propose an outcomes evaluation culture Advisory group – mixed – working mostly on the development of the CM analysis method 2 service users 2 year three SW students 3 researchers (one familiar with SPSS analysis) 3 senior lecturers Students and service users paid for their time and travel (additional £5000 secured from University fund)
Research Procedure During induction week BA students received one hour instruction on CM, a handout with the instructions, and the participant information sheet Two days later they were invited to participate by signing a consent form – participants given 30 mins (negotiation with lecturers) to complete handwritten CMs individually. Student with dyslexia supported. Instruction: complete the conceptual map of working in partnership with service users Questionnaire completed after each CM Before the subsequent measurements the students received 15 mins of instruction as reminder
Sample At T1 – 13 (17% of BA cohort) At T2 – 6 – attrition 53% T2 sample Females 4 White British 1 student with dyslexia 3 students with previous experience of social work (employment, personal experience) 1 Asian – previous experience of social work (volunteering) 1 Black Caribbean – no previous experience Age ranges: 36-45
Analysis Novak (1984) – structural method – scoring the maps by giving specific values to hierarchies, concept-links and cross-links. Arbitrary values Mechanical counting of correct links Hierarchies may not be relevant to all types of knowledge (Freeman, 2004) West el al. (2000) – relational method – based on degree of foundational or core relationship to the main concept.
Scoring instructions developed by the team Combination of structural and relational methods Acknowledges the importance of the theoretical basis of concept- links and cross links, but rejects the value of hierarchies and the mechanical counting of concepts Acknowledges the importance of the degree of core relevance of the ideas included in the map to the main concept
Description of linkScoreEx. False connections, whether indicated through concept links or only by ??? Tautologies 0 Relevant concepts linked by ???? (no concept link) * Where the ??? cannot be assumed with certainty by the scorer, as there could be more than one alternative explanation to ??? * Where the scorer can assume with certainty the meaning of ??? as the only possible alternative, please score depending on the relevance of the concept to the main concept/question 1 2 or 3 Debatable links - Propositions have sense, but are true only in some circumstances2 Contextual – propositions that are relevant to social work but are far from the core of the main concept 2 Lists – where a number of concepts are attached to one concept link but individually do not add theoretical value to the map – score once the entire bunch 2 Relevant – propositions that are relevant to the concept (the degree of relevance should be captured in the qualitative analysis) 3 Cross-links The links that connect two concepts belonging to two different branches that helps the reader see how concepts in two different domain of knowledge in the map are related and indicate meaningful learning - where the cross-link is not specifically relevant to the main subject – score as contextual link - where the cross-link appears as ??? but the meaning is obvious and its close to the main concept – give full cross-link value 626626
Developing the Scoring instructions Advisory group – service user involvement Blind scoring 2 raters – scoring independently, meeting to discuss were wide differences in scoring (only a few) Instructions altered three times and scoring repeated with three different sets of raters The final version of the scoring method was then applied to the entire set of maps in one go by two new raters: the lead researcher and a senior lecturer Pearson (0.96) and Spearman (0.87) – Strong inter-rater reliability Wilcoxon (r=0.854) – Temporal stability
Findings Increased knowledge demonstrated Significant limitation – only 6 pairs of maps, then 4 Map Scores Most impacting sources of learning at T1: Ethics and Values; observational practice; Poverty, Social Exclusion and Social Work, and academic reading. At T2: Social Work with Children and Families, Social Work with Adults, practice placement; Principles and Skills of Social Work; academic reading; and discussing with colleagues
Student 4 profile Female 36-40 White British No disability Previous contact with social work through employment in NHS – generally good appraisal of her experience Scores T1: 35.5 T2: 80.5 T3: 136.5
Student 3 profile Age range – 36-40 Asian ethnic origin Previous volunteering work – appraised as excellent in opening my eyes to a lot of issues concerning social services. Scores T1: 29.5 T2: 26.5 T3: 48
Conclusions Some evidence of significant increase in knowledge – however, inconclusive due to small number of maps – thus CM as method for measuring social work education outcomes needs to be explored further In particular the balance between quantitative and qualitative evaluation of maps has to be strengthen. CM – has potential for being used routinely as evaluation method – validity depends on familiarity with the tool and adequate time given to learn it and complete it. Being based on the theory of meaningful learning CM pertains to a wide range of subject matters in social work and other fields. Workshop: CM as formative evaluation method, learning and teaching aid, and empowering tool in social work practice
Selected Bibliography Anghel R. & Ramon S. (2009) Service users and carers involvement in social work education: lessons from an English case study. European Journal of Social Work, 12 (2), p. 1 – 15. Ausubel D. (1968) Educational psychology: a cognitive view. New York: Holt, Rinehart and Winston Carpenter J. (2005) Evaluating Outcomes in Social Work Education, SCIE, SWAP and SIESWE, London DoH (2002) Requirements for social work training [online]. Cmap software FREE download - http://cmap.ihmc.us/download/http://cmap.ihmc.us/download/ Freeman L.A. & Jessup L.M. (2004), The power and benefits of concept mapping: measuring use, usefulness, ease of use, and satisfaction, International Journal of Science Education, 26(2), 151-169 Novak J.D. & Gowin D.B. (1984), Learning how to learn, Cambridge University Press, Cambridge Novak J. & Canas, A.J. (2008) The Theory Underlying Concept Maps and How to Construct and Use Them, http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMap s.htm http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMap s.htm Taylor I. & Le Riche P. (2006) What do we know about partnership with service users and carers in social work education and how robust is the evidence base?, Heath and Social Care in the Community, 14(5), 418-425 West D.C., Pomery J.R., Park J.K., Gerstenberger E.A., & Sandoval J. (2000), Critical thinking in graduate medical education – a role for concept mapping assessment?, American Medical Association, September 6, 284(9), p 1105-1110 West D.C., Park J.K., Pomeroy J.R. & Sandoval J. (2002) Concept mapping assessment in medical education: a comparison of two scoring systems, Medical Education, 36, 820-826