1. Vignettes as Evaluation Tool: Insights and Issues
Tanja Westfall-Greiter & Helga Diendorfer
Center for Learning Schools
Austria

2. The Challenges: Sustaining the Network and Fostering Lerndesigners
How can Lerndesigners be kept up-to-date in order to sustain development at their schools?
How can they remain connected to the Lerndesigner-Network as a community of practice on the national level?
How can they be encouraged to participate actively and fully in nationwide development processes?

3. The Innovation: Virtual PLCs
Topics from survey in Oct 13:
Teacher Leadership
Differentiated Instruction
Assessment
Competence Orientation

4. Focus of Evaluation: Participants‘ Experience
Key question :
How do the learners experience the learning trajectory of this environment?

5. The Evaluation Tools Vignettes during the PLC work
Questionnaire focused on technology and perceived value of the PLC work at the end of the pilot

6. The Method
Vignette writers co-experience in the innovation  protocols of experience
The writers capture the experiences in writing  first drafts
The first drafts are sent to the subjects for validation  feedback
The vignettes are revised and anonymized  revised vignette
Revised vignettes are explored in a workshop to polish vignettes  final vignette

7. A Revised Vignette: PLC „Teacher Leadership“
Anne is furious. Everyone else in the PLC is reporting that although „teacher leadership“ isn‘t part of their school vocabulary the role of teacher leaders is nonetheless central. Neither is true for her school. Her principal delegates responsibilities when there is need. There is even constant fluctuation in their School Development Team! How did she become a Lerndesigner? She just happened to be in the principal‘s office when the form arrived to register a Lerndesigner and got the job. She wants to know if teacher leaders also have authority so they can, for example, call for team meetings. And she also wants to know what benefits others see in their teacher leader roles and who would put themselves through all of that under the given circumstances. Actually she‘s not really thinking about teacher leaders in general; she‘s really thinking about herself at her school. Anne is growing impatient. She wants answers to the question of what processes are necessary to introduce teacher leaders in a school. (TLS1 V3)

8. Discussion
Imagine you are the innovator and are evaluating virtual professional learning communities.
What do you learn from this vignette?
How does Anne experience the virtual PLC environment?
Is this a learning experiene? If yes, explain why.
Is it helpful for evaluating the innovation? How?
What else might it be helpful for?

9. Validation
Dear Helga,  Re: Vignette 1: well captured. You mirror the situation at my school and my position perfectly. Please use „furious“ rather than „frustrated“.  

10. Evaluation Results
Technology and virtual environments enable global PLC work.
Technology can hinder the individual in participating fully.
Virtual PLCs foster participants‘ digital skills if they have a minimum prerequisite of digital skills.
Session facilitators need both technical and moderation skills.
Preparing for a PLC session can be more valuable than the session itself for the participant, particularly when the technology hinders individuals.
Participants expressed a need for outside expertise when they got stuck but did not always seek it out.

11. 7 Insights into Vignettes as Evaluation Tool
Vignettes are an appropriate method for accessing learners‘ experience.
Vignettes are specific and qualitative and therefore an expensive evaluation tool.
Vignettes provide rich data on both the innovation in focus and other innovations in the broader context.
Vignettes as an embedded tool can help innovators adapt the innovation while it is happening.
Vignettes reveal participants‘ innovations in the innovations.
Vignettes provide multiple perspectives and constitute a narrative of events over time.
Vignettes alone are not enough; post-pilot feedback from participants is also necessary (but relatively cheap).

12. 7 Insights into the Vignette Method
It is possible to co-experience virtually.
Acute perception, openness and writing ability are important skills for vignette writers.
Vignette writers need (and want) training in the method.
Validation is useful but not absolutely necessary; it is itself an intervention for the participant.
Innovators need know-how to synthesize the data in vignettes.
One hour of co-experiencing results in two to three vignettes; each vignette requires approx. five work hours.
The evaluation questions (in this case „learning experiences“) direct attention and influence the co-experiencing of the vignette writers.

13. Questions for Discussion
When do vignettes seems to be a useful evaluation tool? What do they show the innovators? What do they not show?
Who should synthesize data? When? How? (Evaluators? Evaluators + Innovators? Innovators?)
What do innovators need to work with the vignettes as sources of data? Would guided conversations with the vignette writers be useful? Would readings from the vignette writers be a further source of useful data?
If the evaluation questions direct the attention of vignette writers, how does that affect their perception? What do they capture? What do they overlook?