1. WLE Centre for Excellence Institute of Education University of London 20 Bedford Way London WC1H 0AL Tel +44 (0)20 7911 5531 Fax +44 (0)7092 288 882 Email n.pachler@ioe.ac.uk Web www.wlecentre.ac.uk E-learner narratives of experience: innovating evaluation Caroline Daly, Norbert Pachler, Jon Pickering and Jeff Bezemer Institute of Education, University of London

2. The Master of Teaching (MTeach)  approximately 140 participants on various routes (post-induction; experienced; school-based, Institute for Ismaili Studies)  based on the notion of shared knowledge construction at a distance through computer-mediated communication  implies a corporate responsibility for developing teacher knowledge, reflected in the course infrastructure, module content, models of assessment and key players  focuses on the ‘art’ and ‘science’ of teaching, requiring participants to examine closely their own classrooms as places of learning  not preoccupied with the acquisition and critical analysis of established orthodoxies and epistemologies in the field, but concerned with understanding more deeply their practice in the context of a learning culture

3.  stresses the ‘agentive’ dimension of teacher development, i.e. the reciprocal, symbiotic relationship between practice and intellectual reflection (‘scholarship of teaching’)  the knowledge base of the course is not located in the Institute of Education, nor in the ‘set’ (digitised) readings, nor in the tutors but in the interactive environment of the electronic forums themselves  use of portfolio assessments (containing ‘evidence studies’), nurturing an enquiry approach to ‘understanding teaching and learning’ and raising levels of ‘research literacy’ and practitioner research capacity  develops ‘research readiness’ in the early stages of the course, taking participants beyond the ‘what works’ agenda, through an expanded understanding of critical classroom enquiry  based on the notion of a ‘professional learning academy’, which has at its heart a number of linked online learning communities (MTeach tutor groups) consisting of programme ‘participants’, university tutors and some ‘others’

4.  requires participants to provide public accounts of several aspects of teaching, e.g. the establishment of learning goals, the management of learning environments, the design of materials, the evaluation of outcomes, the assessment of learning etc.  much effort is expended on enquiries as a basis for productive professional conversations (‘talking teaching’)

5. Pedagogical considerations  the pedagogy of the course emerged over time through the shared spaces of interdisciplinary, cross-curricular and cross-phase planning meetings  an antidote to a prescriptive initial teacher education curriculum which progressively incapacitates teacher educators and conceives of them as little more than ‘delivery agents’ of government prescriptions (‘standards’)  leadership is conceived of outside of current managerial agendas, focussing instead on ‘pedagogical leadership’ and leadership of learning  the potential afforded by new technologies has significantly informed the design of the MTeach and its ‘delivery’: ‘mixed mode’ employing online networking as well as some traditional face-to-face modes  course tutors have, by-and-large, not had a lot of prior experience in online tutoring and had to ‘learn by doing’ how to create an environment conducive to the constructional and conversational discourses upon which the MTeach is predicated

6.  access to discourse transcripts documenting collaborative learning (i.e. the ‘closed’ web-archive of contributions to online discussions)  some technical and conceptual challenges: around certain aspects of the particular online environment used (originally YahooGroups, now Moodle) and at perceptual-pedagogical level, for example around the effective use of threading and the development of strategies for monitoring the quantity and quality of participants’ contributions  anxieties around ensuring that participants’ work meets with requisite (i.e. Masters level) standards: course participants invariably contribute differently to electronic forums compared with formal seminar rooms, i.e. much of the work is ‘conversational’ in nature and it involves participants continually in a discourse about their own learning and that of their peers with reference to relevant conceptual and theoretical frameworks  the MTeach is characterised by a strong belief in social constructivist approaches to learning, i.e. the co-operative nature of knowledge building and pursuit

7.  ‘template-based approach’: tasks as key cognitive tools (cf. social interactionism)  templates encourage the ‘on-the-fly’ recording of thoughts and impressions whilst examining learning materials and they provide a framework for information gathering, the stimulation of recall of prior knowledge and the guiding of knowledge construction  problem-solving in orientation, encouraging MTeach participants to ‘go meta’ about their teaching  importance of three educational processes from divergence to convergence:  idea generating (and gathering)  idea linking, organization, and intellectual progress; and  idea structuring (and convergence)

8.  ‘case methodology’: generative potential of discourse; potential of narrative or ‘storying’ in learning (about teaching)  MTeach participants to develop case studies which we see as a ‘re-collected, re-told, re-experienced and re-flected version’ of direct experiences

9. Work-based learning Continuing Professional Development Collaborative learning Criticality & research literacy critical incidents practitioner research learning journals response partners online discussions portfolios scholarly teaching accredited CPD mixed mode challenging orthodoxies socio-constructivist pedagogy

10. Rationale We need our course development to be informed by deeper understanding of our e-learners’ experiences and consider questions like:  what are (their perceptions of) the pedagogical advantages?  does the use of technology create social barriers? Otherwise we are driven by other factors which are important, but can dominate our course design while we strive to keep ahead in the world of e- learning e.g.  Keeping up with technology  Widening participation  Increasing student numbers  Keeping costs down … … so that we feel like this

11. Why focus on e-learners’ experiences?  The ‘primacy of improving the student learning experience’ (HEA Strategic Plan 2005 – 2015) for overseas and home students.  The inadequacies of evaluation based on ‘satisfaction’ and ‘exit’ models which consult students after study is completed.  The ‘gap’ between our intentions and the students’ experiences might be bigger than we thought (cf. the efficacy of ‘designed-in’ narrative)…they can ‘cover up’ at a distance.  We perceive coherence in our learning designs where the students may not. Our assumptions about what is going on can be wrong.  We need a sustainable way of understanding our learners’ experiences when they are at a distance and encountering new ways of learning with technologies.  We need ways which can help us know what the e-learning experiences on our courses are really like – are our efforts worth it?!

12.  to identify e-learners’ experiences in the mixed-mode context  to trial a new (embedded) approach to evaluation  to analyse the implications for e-learning course development  to see if, therefore, we should redesign our course in the light of what we found out about learners’ experiences Aims of the research project

13. Research ContextResearch Design Mixed-mode degree (MTeach)Longitudinal – 7 months (1 module) Participants: a tutor group of 12 teachers beginning the course; aged 25 – 55; 3 men, 9 women Quantitative data – pre-course questionnaire on prior learning experiences & course expectations Text-based online discussions - compulsory - asynchronous (not ‘live’) - 3 per module - lasting 5 weeks each Qualitative data – learner narratives - narrative interviews - focus group discussions - online focus groups - online commentaries  All concurrent with their e-learning Only 2 participants were experienced e-learners Analytical approach based on systematic reading and coding of the range of learner narratives

14. Why narrative evaluation methods?  a way of capturing the learner voice  a way of better understanding the realities of ‘being an e-learner’ on our courses as a prime evaluation tool (cf. verisimilitude: the intention was not to ‘prove’ anything, not about logico-scientific ways of understanding, rather than to understand and adopt a stance which makes a difference to teaching)  to provide ongoing information about the course while it is happening: meaning located in place and time; gain autobiographical insights  to impact on the learners’ abilities to adapt to new learning contexts

15. What narrative evaluation methods did we use?  individual, loosely-structured narrative interviews using ‘postmodern’ techniques (Gubrium, 2003):  naïve interviewer stance  emplotment  chronology  “without overspecifying the substance or the perspective of the talk” (McCracken,1988)  group narratives aimed at collective sense-making of experiences  online forums as ‘virtual focus groups’ (Bloor, 2001) to elicit reflective group discussion on key issues  online think aloud to elicit individual responses to mediating objects, e.g. images (Eraut, 2000)

16. Greenhalgh’s criteria for a ‘good story’ in narrative research aesthetic appealthe narrative is pleasing to hear and recount, it contains an internal harmony coherencethe narrative is clear and makes a logical whole, it contains a ‘moral order’ or sense authenticitythe narrative has credibility, based on the experiences of the listeners/readers reportability“ the ‘so what’ value” of what is narrated, its significance persuasivenessthe narrative convinces of the teller’s own perspective

17. Analysis of narrative data  inductive  systematic  inter-researcher coding  continuous dialogue within a research team  often involves stages of interpretation  identification of themes in the first instance – surface level, declaratory  identification of sub-textual or latent themes and patterns  development of categories revealing controlling principles, motivations, unarticulated experiences…  hypothesising about the meaning of the narratives  development of vehicle by which the narrative data can be presented and findings communicated – a diagram and sample ‘cases’ using quotations from the narratives themselves

18. I am aware that other professionals are reading my work before anyone has told me whether it is any good or not. Now, I know I'm a teacher and I'm nearly 26 yrs old and I should probably have grown up a bit by now but I hate getting things wrong (!) and it feels a bit like being asked to read your work aloud to the class before your teacher has checked it. Does anyone else know what I mean ? it's a bit daunting posting work, without it being checked first!! I'm sure I'll get used to it … What participants told us about being an e-learning ‘community’

19. it's a big learning curve for me: like this before Now that responses are being posted I am beginning to feel more confident in this on- line learning process and that a community is beginning to emerge. It certainly takes some getting used to, as I too have not done anything like this before. I am finding that when I ’ ve read someone ’ s correspondence I want to talk about it immediately and explore the issues involved

20. The style in which a person writes I feel gives an insight into their personality, as you read their writing you can often build a sense of who they are and the situation they are coming from. … or perhaps not knowing the person who posted the tasks could be more benefit to bringing a sense of community as it allows us to take a more non- judgemental view as we do not have the ‘ baggage ’ that people bring with them? Just a thought.

21. E-learners’ experiences – four types of ‘newness’ which affect participation for all These are the new areas of experience which the e-learners chose to talk about in their learner narratives:  the experience of managing work, life and learning with technology  the experience of writing with fellow students in an online discussion in order to learn  the experience of collaborative learning in an online tutor group  the experience of social relationships with other e-learners

22. ‘Learning about learning’ – a positive outcome for participants in a new approach to evaluation based on learner narratives  Most participants show considerable interest in how they learn with others and what impedes/supports this  They assume responsibility for the conduct of the online forum  They reflect on un/desirable features of online discussions  There is increased commitment to the collaborative processes (e.g. responding to online partners, sharing files and documents)  They take on ‘teacherly’ roles  They develop a respect for learner differences  They create their own learning culture, shaping the context of their own learning and how it works at a distance  Both novices and experienced e-learners gain from such activities and are extremely candid – they want to tell

23. Second level of interpretation – latent categories of experience (‘getting used to e-learning’) vulnerabilitysecurity expertise incompetence formalityinformality newnesscontinuum

24. Meta-learning  An unexpected outcome has been the impact on the learners of discussing their learning and reflecting on how they learn within the collaborative online environment  Considerable degree of interest in how they learn with others and what impedes/supports this  Considerable commitment to the collaborative processes (e.g. response partners, sharing files and documents, detailed consideration of issues raised in online course discussions)  They create a ‘culture of use’, shaping the context of their own learning and how it works at a distance  Articulations are shaped by the social and physical environment in which they were produced:  among each other in tutor-prompted online commentaries  with an interviewer who is not an online tutor  at school, in a classroom, at IoE, with or without filming

25. Implications for programme teams  There are benefits for learners of participating in narrative evaluation activities, by which they address explicitly their (new) experiences of learning with technologies.  There are benefits for practitioners in gaining knowledge of unanticipated learner experiences, though these may not always lead to simple modifications of course design or learner tasks.  There is an argument for course design to invest learner time in developing the abilities of participants to reflect on how they are learning. There is growing research evidence that this has a positive impact on what they learn.  There is a need for meaningful evaluation activities which also support learning. Engaging with evaluation becomes a different type of experience for both tutors and students

27. Bibliography Bloor, M., Frankland, J., Robson, K., Thomas, M. (2001) ‘Virtual Focus Groups’ Focus Groups in Social Research. Bruner, J. (1985) ‘Narrative and Paradigmatic Modes of Thought’ in Eisner (ed) Learning and Teaching the Ways of Knowing Chicago: University of Chicago Press Eraut, M. (2000) ‘Non-formal learning, implicit learning and tacit knowledge in professional work’ in (ed.) Coffield, F. The necessity of informal learning Bristol: Policy Press ESRC Learning Society Programme 12-31 Greenhalgh, T. (2006) What seems to be the trouble? Stories in illness and healthcare Oxford: Radcliff Publishing Greenhalgh, T., Russell, J. and Swinglehurst, D. (2005) Narrative methods in quality improvement research, Quality & Safety in Health Care 14: 443-449. Gubrium, J. (2003) Postmodern Interviewing London: Sage. McCracken, G. (1988) The Long Interview Newbury Park: Sage.