1. Helen Beetham Consultant in Pedagogy JISC e-learning programme
Pedagogic Evaluation
Helen Beetham Consultant in Pedagogy
JISC e-learning programme
Colloquia

2. Activities for this session
Discuss what is meant by ‘pedagogic evaluation’
Identify project aims and rephrase as evaluation questions
Identify stakeholders in the evaluation
Consider appropriate means of data collection and analysis
Establish contact with ‘peer’ projects and begin sharing ideas/expertise

3. Lab testing Does it work? Functionality test Compatibility test
Destruction test …

4. Usability testing Can other people make it work?
Are the menus clearly designed?
Is there a logical page structure?
...

5. Pedagogic evaluation Does anyone care if it works?
i.e. is it useful in learning and teaching contexts?
How is it useful?
Who needs software anyway? …

6. Relating the three phases of evaluation
Software may progress from lab testing through usability testing to contextual evaluation…
… or (e.g. in RAP) through many iterative cycles, with users involved at each stage
Increasing authenticity of context
Evaluation moves from simple, lab-based to complex, authentic contexts of use
Different questions are asked, different kinds of data are collected, and different issues arise
Complex, authentic contexts rarely provide yes/no answers to development questions
Causative factors may be difficult to untangle
Findings may be highly context-related (so several different contexts are better for evaluation than one)

7. Three approaches to learning and teaching
There are basically three ways of understanding how people learn
Associative
Constructive
individual/cognitivist
social constructivist
Situative
Lead to different pedagogic strategies and approaches
Any of these approaches may be appropriate
depending on the priority outcomes and the needs of learners

8. Associative approach In learning In teaching In assessment
Routines of organised activity
Progression through component concepts or skills
Clear goals and feedback
Individualised pathways matched to prior performance
In teaching
Analysis into component units
Progressive sequences of component-to-composite skills or concepts
Clear instructional approach for each unit
Highly focused objectives
In assessment
Accurate reproduction of knowledge or skill
Component performance
Clear criteria: rapid reliable feedback
Examples
Guided instruction
Drill and practice
Traditional ISD (e.g. Gagne)
Socratic dialogue

9. Constructive approach (cognitivist)
In learning
Active construction and integration of concepts
Ill-structured problems
Opportunities for reflection
Ownership of the task
In teaching
Interactive environments and appropriate challenges
Encourage experimentation and the discovery of principles
Coach and model skills
Include meta-cognitive outcomes
In assessment
Conceptual understanding (applied knowledge and skills)
Extended performance
Processes as well as outcomes
Crediting varieties of excellence
Developing self-evaluation and autonomy in learning
Examples
Cognitive scaffolding (e.g. Piaget)
Experiential learning (based on Kolb’s learning cycle)
Experimental learning
Constructivist learning environments
Problem-based learning
Research-based or exploratory learning

10. Constructive approach (social)
In learning
Conceptual development through collaborative activity
Ill-structured problems
Opportunities for discussion and reflection
Shared ownership of the task
In teaching
Collaborative environments and appropriate challenges
Encourage experimentation, discussion and collaboration
Coach and model skills, including social skills
Learning outcomes may be collectively negotiated
In assessment
Conceptual understanding (applied knowledge and skills)
Extended performance
Process and participation as well as outcomes
Crediting varieties of excellence
Developing peer-evaluation and shared responsibility
Examples
Reciprocal teaching
Conversational model (Laurillard/Pask)
(Computer-supported) collaborative learning
Dialogue/argumentation

11. Situative approach In learning In teaching In assessment
Participation in social practices of enquiry and learning
Acquiring skills in contexts of use
Developing identity as a learner
Developing learning and professional relationships
In teaching
Creating safe environments for participation
Supporting development of identities
Facilitating learning dialogues and relationships
Elaborating authentic opportunities for learning
In assessment
Crediting participation
Extended performance, including variety of contexts
Authenticity of practice (values, beliefs, competencies)
Involving peers
Examples
Apprenticeship
Cognitive apprenticeship
Situated learning
(Legitimate peripheral) participation
(Continuing) professional development
Work-based learning

12. 3 ways of learning about (educational) software
Does it work?
Routines of organised activity
Analysis into component units
Component performance
Highly focused objectives
Can other people make it work?
Ill-structured problems
Opportunities for reflection
Ownership of the task
Extended performance
Processes as well as outcomes
Is it useful in authentic (educational) contexts?
Creating supportive environments for use
Supporting development of users’ skills
Facilitating dialogues and relationships
Elaborating authentic opportunities
Extended performance, including variety of contexts
Authenticity of practice

13. Evaluation is learning!
Evaluation for development, not accountability
Sharing lessons (including failures)
Sharing concepts and approaches
Moving software into more authentic contexts of use
in order to find out how it is useful, and how it should be supported and embedded for effective use
Using the outcomes of evaluation to inform development and take-up
Learning across peer projects
Learning across different strands of the e-learning programme (and beyond)
Learning about your own software
how to have effective dialogues with users
range and/or specificity of application
usability implications?

14. Principles of evaluation
Ask the right questions
Involve the right people
Collect useful and reliable data
Analyse and draw conclusions appropriately

15. 1. Asking the right questions
How does the use of this e-tool support effective learning, teaching and assessment (LTA)?
What LTA activities?
Be specific and pragmatic
Understand how e-tool fits with existing LTA practice
But expect it to alter practice – sometimes unpredictably
Which users?
Range of user needs, roles and preferences
Consider stakeholders who are not direct users
What counts as ‘effective’?
Enhanced outcomes for learners? Enhanced experience of learning?
Enhanced experience for teachers/support staff? Greater organisational efficiency?
Consider what claims you made in your bid, your big vision
Effective in what LTA contexts?
Does e-tool support a particular pedagogic approach?
Does it require a particular organisational context?
Consider pragmatics of interoperability, sustainability and re-use
Are you aiming for specificity of breadth of application?

16. Example: Interactive Logbook project
identify how the IL supports learners wrt ‘access, communication, planning & recording’
How is access to learning resources improved?
How is communication for learning improved?
Does the IL provide a useful tool for planning and recording learning in the pilot programme? In what ways?
Does the IL support planning and recording outside of the pilot programme and does it provide a durable basis for future planning and recording? In what ways?
How does it compare with other systems offering similar benefits?
identify how the IL supports teachers or programme developers and organisations respectively, and understand how best ‘to implement and embed the Interactive Logbook… as distributed e-learning becomes more mainstream’.
What skills do learners and tutors need to make effective use of IL? What features of the programme support integration and use of IL (assessment strategies, support available, mode of access)?
What technical systems and support are needed for IL to be integrated effectively? What features of the organisation support effective use of IL by learners and teachers?

17. Over to you (1) Evaluation should be interesting, so:
What do you really want to find out about your software?
What is the most important lesson your project could pass on to others?
Don’t set out to prove what we already know!
Look back at your project aims – what claims are you making for impact on LTA?
Translate these aims/claims into questions. Is there evidence of this impact? How does it happen?
Good claims are achievable but also challenging
Good questions are tractable but also interesting
Do your original aims fit with what interests you now?
Prioritise the issues that seem important now, with the benefit of insights from the development process
But use this as an opportunity to revisit and review
What are other projects investigating?
Do you have any questions in common in your peer group? Or questions that complement each other?
What could you usefully share of the evaluation process?

18. 2. Involving the right people
Who are your users?
What activities will they carry out with the system?
What functions of the system are important to them?
What roles do they play in those activities?
What are the important differences between them?
Significant for sampling (e.g. dyslexics for V-MAP)
walk-throughs, use cases (user testing and evaluation design)
real groups of learners and teachers (pedagogic evaluation)
Who are your other stakeholders?
Non-users whose work or learning may be impacted by use of the system in context e.g. administrators
Other ‘interested’ parties e.g. institutional managers, project funders, researchers/developers, potential users…

19. Over to you (2) Who are your important stakeholder groups?
distinguish your user group in ways that are significant for your evaluation questions (e.g. learners with or without an existing e-portfolio)
consider non-users as stakeholders and as potential sources of information
Share your outcomes with your peer group.
What types of user do you need to include? (e.g. model users you have developed for walk-throughs)
How will you identify real user groups for evaluation?
How will you ensure all your significant types are included?
(NB including different types of learner is much easier than finding a ‘statistically representative’ sample: for this the proportions of different kinds of learner must be the same as in the target population)

20. 3. Collecting useful data
Data collected should be directly relevant to the questions!
Data should show triangulation:
using a variety of methods (e.g. focus group and questionnaire)
from a range of stakeholders (e.g. learners, teaching staff)
over a span of time (e.g. ‘before’ and ‘after’)
Quantitative data = How much? How often? How many?
Also providing yes/no answers to simple questions
Generalising from instances to rules
Converting opinions into data for analysis (Likert scales)
Qualitative data = explanatory, narrative
What happened? What was it like for you? Why?
Identifying themes and providing local evidence
Preserving the voices of participants
How authentic the context? If authentic, how will you support embedding and use of the software?
Feasibility and costs of data collection?
Skills and costs of analysis?

21. Over to you (3) Use the matrix to plan what data you will collect
Data should be designed to answer specific questions (left column) and should be collected from specific stakeholder groups (top row)
Add details if possible, e.g. when (time) and how (method) this data could be collected
You need not fill all the boxes but try to have something in each row and column
You can merge boxes!

22. Final discussion: analysing and drawing conclusions
Basic choices for data analysis:
Quantitative analysis – statistical data that may be presented as pie charts, graphs etc; likert scales
Qualitative analysis within a given analytical framework –comparison, correlation, explanation
Qualitative analysis without a given analytical framework – case histories, narratives, themes
Outcomes need to be useful to different audiences
Your project, and other development projects
Implementers and users of your software
How will we draw conclusions across the different projects?
Peer review groups
Links to other projects in the e-learning programme
Pedagogy strand – refer to previous workshop
DeL regional pilots
ELF reference models (also standards community)
Sharing scenarios, roles and walk-throughs (model users)?
Mapping activities to a common framework (model uses)?

23. Learner Differences (‘model users’)
Refer to hand-out
But note that in many situations these differences will not be educationally significant
And in your context of use, there may be other important differences to consider (see your ‘stakeholder’ activity)
Could we develop a databank of ‘scenarios’ and roles’?
See Peter Rees-Jones’ work on scenarios for e-portfolios

24. Outline of a learning activity
Identities: preferences, needs, motivations Competences: skills, knowledge, abilities
Roles: approaches and modes of participating
learner(s)
specific interaction of learner(s) with other(s), using specific tools and resources, oriented towards specific outcomes
learning environment
learning activity
learning outcome(s)
Tools, resources, artefacts Affordances of the physical and virtual environment for learning
New knowledge, skills and abilities on the part of the learner(s)
Artefacts of the activity process
other(s)
Other people involved and the specific role they play in the interaction e.g. support, mediate, challenge, guide

25. Developmental processes
Environment (tools, resources) can be adapted to meet the needs of learners, or provides a range of options for differentiation
learner(s) do
reflect
adapt
differentiate
Learning outcomes are captured for reflection, planning and review
learning environment
learning activity
learning outcome(s)
share
respond
Outcomes can also be shared for formal assessment, informal feedback and peer review
other(s)

26. Next steps Appoint evaluators (if not already in place)
Finalise evaluation plan
Based on phase 2 bid
Using a pro-forma (optional)
Identify opportunities to liaise with other projects, e.g. to share
Evaluation questions and approaches
Actual data (comparative analysis?)
Process of analysis and drawing conclusions
I will be in touch to discuss these (or contact me at any time)