1. PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY
VI KONFERENCJA EWALUACYJNA
WARSZAWA,
PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY
Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych
Some Visions for Fifth Generation Evaluation

2. Some background information
My interest in research methodology goes back to the beginning of 1990s.
Since 1994 I have been a lecturer in data analysis; since 2004 also lecturer in research methods.
My (5y+2y) Master’s thesis, defended in 1996, was about teaching statistics in the framework of research methods courses.
Got interested in mixed methods research in 1998 during my one-year study in Cambridge University
In the doctoral dissertation published in 2004 I focused on the combined use of quantitative and qualitative approaches in educational research
Belonging to the editorial board of Journal of Mixed Methods Research and International Journal of Multiple Research Approaches

3. Further reading related to the topic of speech
Niglas, K. (2010). The multidimensional model of research methodology. An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed Metods Research. 2nd Ed. Sage Publications Ltd
Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 1,
Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for mixed methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research: Theories and Applications. Sage Publications Ltd
Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research – Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29
Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational Research. Tallinn Pedagogical University. Dissertations on Social Sciences. Short version: PS! Published version of the dissertation (book) is available by request (
Several papers available at Education Line

4. Change in the aims and forms of evaluation
Guba and Lincoln (1989) describe 4 generations in evaluation of social and political programs:
Evaluation ≈ measurement, testing
Evaluation ≈ description, mapping
Evaluation ≈ substantive evaluation, judgement, advise
Evaluation ≈ collaboration, negotiation, initiation and guidance of change

5. From positivism to constructivism … Critique of the first three generations by G & L (1989)
A tendency towards managerialism managers who organise evaluation as well as evaluators are outsiders and their relationship with/to stakeholders is disempowering
A failure to accommodate value pluralism scientific mode comes with the claim of the value-freedom, but as evaluation is essentially about valuing, the value-free in practice means imposing the values of one group and not accommodating value differences
An overcommitment to the scientific paradigm of inquiry context stripping; overdependence on quantitative measurement; belief in scientific truth leaves no room for negotiation or alternative explanations and frees evaluators from moral responsibility

6. From positivism to constructivism … Main principles for 4th generation evaluation by G&L (1989)
is organised by the claims, concerns, and issues of stakeholding audiences
utilises the methodology of the constructionist paradigm
is a process whereby evaluators and stakeholders jointly and collaboratively create (or move towards) a consensual valuing construction of some evaluand This process is: local, sociopolitical, emergent, continuous, recursive, and divergent, involves teaching/learning of all parties and sets them into hermeneutic dialectic relationship

7. From positivism to constructivism … Where are we 20 years after manifestation of 4th Generation?
Are we there? (read: has constructivist paradigm won the paradigm wars and we all work within this framework?)
Do we want to be there? (read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?)
Do we have an alternative? (read: is our choice constrained by positivism and constructivism or …?)

8. From positivism to constructivism … My answers to the three questions set on the last slide:
Are we there? (read: has constructivist paradigm won the paradigm wars and we all work within this framework?)
No! – a quick glance at the evaluation practice tells us that wealth of evaluation studies are not conceptualised and carried out in this framework

9. From positivism to constructivism … My answers to the three questions set on the last slide:
Do we want to be there? (read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?)
No, not all of us! – Even though there is a lot to learn from constructivist tradition many of us resist the forced choice between positivism and constructivism and do not accept all the assumptions that constructivism brings with it (i.e relativist ontology, strong belief in and either or stance on paradigms, overemphasized belief in impossibility of representativeness and generalizations as well as that change cannot be engineered, etc.)

10. From positivism to constructivism … My answers to the three questions set on the last slide:
Do we have an alternative? (read: is our choice constrained by positivism and constructivism or …?)
Yes! We can: – replace the bi-polar paradigmatic view by the conceptualization of research methodology as a multidimensional integrated set of continua
– reason that the research/evaluation practice is inevitably lead by sophisticated and changing personal worldviews or mental models of the parties rather than well defined and incommensurable paradigms
– set facilitating change as the sovereign aim and the staple framework for the evaluation without having to declare either the stakeholders’ or the managers/experts’ position, needs, views, etc. being always paramount

11. The idea of incommensurable paradigms
The Conventional and Constructivist Belief Systems (Adapted from Guba and Lincoln 1989)

12. Common dichotomies in methodological literature

13. Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum
view which divides social research into limited number of paradigms, whether based on philosophical categories or methodological distinction, is misleading
research practice is much more variegated and complicated than that proposed by a paradigmatic view
misguided attribution of two distinct and often opposite sets of qualities to the two large families of methods

14. Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum
misrepresentation of complex philosophical ideas
major differences in philosophical as well as methodological preferences within the camp of qualitative researchers as well as within the ranks of quantitative researchers
it is rather a complex and multifaceted individual mental model, which is formed by many factors like education and training, personal values, disciplinary perspectives, philosophy of science, etc., than formalized paradigm that guides the work of social inquirers and evaluators

15. On the way towards the idea of continuum
Extension of the paradigm model to 5 paradigms
by Guba and Lincoln (2001/2005/2008)
The idea of mixed methods as an interactive continuum by Ridenour & Newman (2008)
QUAL-MM-QUAN continuum
Multidimensional continuum of research projects
by Teddlie & Tashakkori (2009)

16. On the way towards the idea of continuum
Integrated Multidimensional Continuum of Research Methodology (adapted from Niglas 2001)
not static or “all-inclusive”
functions as an illustration supporting an argument against validity of paradigmatic approach to methodology
helps to organize our thinking about methodology and thereby serves educative purposes

17. Original figure available at: www.tlu.ee/~katrin/mmrm/

19. Experiences from the field ...
“[...] experience in conducting research to bring about social change and influence social policy indicates that the most persuasive policy research includes both elements: numbers that define the scope and patterns of the problem, and a story that shows how the problem works in daily life and provides for emphatetic understanding. These two elements stem from quantitative and qualitative research. Integrated approaches are effective for [...] analysis of programs, policies, or actions that helped to overcome persistent inequalities.”
Spalter-Roth, 2000

20. The choice of methodology and methods for a study depends on the purposes and research questions

21. Best fit between aims/problems and designs/strategies (adapted from Creswell & Plano Clark 2007)
Need to ...
Usually best suited research design/strategy
see if treatment is effective
Experimental design
identify trends/attitudes/... in a population
Survey design
learn about and describe a culture shared by a group
Ethnography design
generate a theory of a process
Grounded theory design
imporve the subsistence and/or empower certain group
Action research design
develop or imporve some policies/artifacts
Design research approach

22. Cyclical nature of the research process:
Facilitating change as sovereign aim for evaluation - practice and change oriented research approaches
Action research
Design research
Cyclical nature of the research process:
Diagnose / Analyze
Plan action / design & develop
Act / test / implement
Evaluate / Learn
Need for variegated empirical data –> best results will be often achieved by utilizing mixed methods approach
Monitor

23. Traditional model of methodological aspects is here incorporated into more general framework!
Research problem (question, hypothesis, purpose, …)
Strategy (case study, survey, experiment, grounded theory, action research …)
Sampling (random sample, one case, purposefully chosen cases, … )
Data collection (structured questionnaire, unstructured/open interview, …)
Data analysis (statistical methods, open coding, discourse analysis,… )
Interpretation and conclusions (descriptions, empirical generalisations, …)

24. What is design research?
Design research investigates the process of designing with the overall aim to develop an accessible, robust body of knowledge that enhances our understanding of design processes, applications, methods and contexts. Often, this knowledge helps to define best practice and workable methods in dealing with design and design related problems.
Alternative approaches/terms used: design-based research, design-science research, design study, design experiment, developmental research, development study, formative research, ...

25. What is design research?
Design research is distinguished from design by the production of interesting (to a community) new knowledge
Research driven
Systematic documentation
Formative evaluation
Generalization
DR strives to combine the creativity of design communities with appropriate adherence to standards of traditional research

26. Characteristics of good design research
Development and research takes place through continuous cycles of design, enactment, analysis, and redesign
The central goals of designing artifacts or environments and developing theories about design process but also about the related phenomena are intertwined
DR must lead to sharable theories that help communicate relevant implications to practitioners and other designers
Research must account for how design functions in authentic settings
The development relies on methods that can document and connect processes of enactment to outcomes of interest

27. Design research – the methodological cycle
Indicating the need for development and related knowledge -> > initial plan of the study
Analysis of the problem (needs, goals, pre-existing knowledge, ... )
Designing / monitoring / documenting – planning of the design&development process (work allocation, schedule, applicable methods, ... ) – monitoring the d&d process (memos, team discussions, …) – describing the result of d&d process (sketches, alternatives, resulting design and/or artifact, …)
Evaluation (incl initial implementation) (testing the design result, evaluation according to standards, feedback from users and/or experts, ... )
Generalisations (suggestions, requirements, standards, concepts, models, ontologies, theories,...)

28. Our design research project: Induction year for novice teachers in Estonia
0. Need to design a support system
1. Analysing problem, collecting and syntezising theoretical information
2.Design process, constructing/ improving the model
3. Applying the model (being reflective)
4 Evaluating the model and theoretical standpoints
5. Shortcomings of the model and Generalizations

29. Induction year for novice teachers in Estonia Partners and implementation activities:
Schools (mentors, novie teachers, school leaders)
creating propitious environment for professional development
mentoring beginning teachers
Universities
(re-)developing theoretical model for mentoring
mentor training
support programme for beginning teachers
Ministry
developing policy
ensuring resources
Joint expert group
(re-)designing implementation model
facilitating networking
quality assurance

30. Induction year for novice teachers in Estonia Design research cycles
Design cycles of our project can be divided into two general phases:
Cycles I and II - emphasis on planning, designing and re-designing implementation model for induction programme
Analysis of theoretical and secondary materials
Focus group interviews with experts, novice, mentors and school leadres
Survey for beginning teachers
Semi-structured questionnaire for mentors
Open interviews for the school leaders
Monitoring and self-evaluation (diaries, peer visits, peer groups, ...)
Cycles III, IV, etc – emphasis on implementation of the induction programme
- Constant evaluation and improvement of the model => monitoring and research activities continued

31. Experiences from our design research project
Well established cyclical process of design research where QUAL and QUAN data is systematically collected and analysed in order to inform design & development process, empovered by strong partnership links provide good basis for evidence based educational restructuring ensuring
Continuity of the process
- ongoing evaluation and development
- ongoing partnership
Sustainability of the project
- acceptance by the practitioners
- wide-scale implementation

32. Towards Fifth Generation Evaluation … some conclusions (read: visions)
The 5th Generation Evaluation can be characterised by:
Conceptualising evaluation as a tool for facilitating development and therefore utilizing cyclical methodological designs which place evaluation into the process of continuous pursuit for improvement
(evaluation not as a purpose of the project per se but implemented within a project to serve wider purposes)
Valuing initiation from stakeholders as well as from experts to start and lead systematic process of evidence-based development
(utilizing designs from action research to design research with various hybrids in between)

33. Towards Fifth Generation Evaluation … some conclusions (read: visions)
Open and inclusive stance towards various worldview positions and methodological approaches where paradigmatic confrontation is commuted for continuum model accommodating overlaps between and being enriched by the differences there are between traditions
(dignifying personal mental model instead of paradigm, following the principle of informed creativity, choosing methods and techniques in concord with specific aims and questions considering QUAN, QUAL as well as MM approaches)
Acceptance that evaluation can generate results that apply to the unique local settings as well as results that serve more general shared needs for development
(accepting and utilizing different forms of generalizations)

34. Thank you !
Katrin Niglas PhD, Professor Institute of Informatics