1. Qualitative Evidence Synthesis (QES):
Karin Hannes Centre for Methodology of Educational Research

2. Qualitative Evidence Synthesis: Agenda
PART 1
Its context (and how I became triggered by QES)
How to formulate questions and search for qualitative insights
PART 2
Its popularity
Its role in systematic reviews (practical examples)
A list of developed approaches
A comparison of characteristics of two commonly used approaches
PART 3
New developments
Context specific versus multi-context reviews
Mixed methods reviews

3. CONTEXT AND STEPWISE APPROACH
PART 1:
CONTEXT AND STEPWISE APPROACH

4. CONTEXT
Emma,
Born the 6th of October 2010

5. CONTEXT

6. CONTEXT
Women who exercised did not lose significantly more weight than women in the usual care group.
Women who took part in a diet or diet plus exercise programme, lost significantly more weight than women in the usual care.
There was no difference in the magnitude of weight loss between diet and diet plus exercise group.
The interventions seemed not to affect breastfeeding performance adversely.

7. CONTEXT
A recent study in the Journal of the America College of Nutritian found that those who ate cereals where lower in weight compared to those who ate meat and eggs, bread or skipped breakfast.

8. Logical, rational reasoning:
CONTEXT
Logical, rational reasoning:
IF a diet helps to loose weight after pregancy.
IF cereals have proven to work well as a diet.
THEN the consumption of cereals will lead to weight loss after pregnancy!
After having consumed cereals for several months……………..

9. What is evidence? CONTEXT
Evidence of ‘effectiveness’: the extent to which an intervention, when used appropriately, achieves the intended effect.
Evidence of ‘feasibility’: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.
Evidence of ‘appropriateness’ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.
Evidence of ‘meaningfulness’: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.

10. Systematic Reviews CONTEXT Well formulated question (PICO)
Therapeutic, diagnostic, economic, preventive,…
Well formulated question (PICO)
Well defined search strategy (sensitive, all-inclusive)
Objective selection proces (inclusion/exclusion criteria)
Critical appraisal of methodological quality (risk of bias)
Objective data-extraction (standard sheets)
Synthesis of data
Conclusion and recommendations consistent with data
Quant: RCT, …
What distinguishes a systematic review from a regular review is its predefined methodology.
Is everyone comfortable with this methodology or do I just quickly run through it?
SR in which you clearly have to indicate what your PICO
Risk of bias or the risk that the results do not do justice to truth because due to an error in the conduct of the project.
Data from those that pass the test are then extracted to be pooled into a synthesis, which can be a meta-analysis, but does not have to be a meta-analysis.
Whether or not a meta-analysis is appropriate needs consideration, e.g. based on the level of heterogeneity between studies etc.
Meta-analysis: pooling results from individual studies

11. Systematic Reviews CONTEXT
IF I am not interested in evidence of effectiveness,
BUT in feasibility, appropriateness, meaningfulness,…
Qualitative Evidence Synthesis: A process of summarizing qualitative research findings, either aggregative or interpretive, by comparing and analysing texts derived from multiple accounts of an event or situation as reported in basic qualitative research studies.
Explore questions such as
how do people experience illness?
why does an intervention work (or not), for whom and in what circumstances…?
what are the barriers and facilitators to accessing health care?
what impact do specific barriers and facilitators have on people, their experiences and behavior?

12. Systematic Reviews CONTEXT
Could I still use the methodology outlined for SR to answer these questions?
Could I use it in the same way?
Question formulation  PICO becomes SPI(C)E
Searching  Sensitive (all-inclusive) versus Specific (Selective)
Critical appraisal  Proponents versus Opponents
Synthesis  Variety of Approaches
Recommendations  Not always the goal, could be building theory as well
But the story becomes completely different if you are not interested in the effectiveness of interventions, but in their feasibility, appropriateness, or more generally in the meaning people ascribe to certain phenomena or topics of interest.
In that case, could I…? YES and NO, I could try to be as transparant as possible on the different steps I am executing to produce my results.
However, I might not be able to execute my synthesis according to that one particular set of predefined steps.
I will not always have an intervention as topic of interest here, nor a comparison.
I am not evaluating an outcome, rather an experience, a meaning, an opinion, a set of values etc.
And most importantly, because qual research is by nature context-specific, I have to add the setting to my acronym, which gives me SPICE: setting, pop, topic of I and evaluation as ingredients of my question.
Searching in qual evidence synthesis might not always be all-inclusive. If the primary goal of a QES is to generate a particular theory, it might not be necessary to be all-inclusive, it could well be that a selection of those reports that provide building blocks for the theory is enough, that there is some kind of a saturation point in the search where additional reports really don’t add much more.
Critical appraisal has highly been supported by some qual researchers, but a majority is opposed to critical appraisal, because their main interest to include a particular study would not be its quality, but it relevance and that there is no such thing as an objective truth in qualitative research since all qualitative knowledge is socially constructed.
A lot of these discussions are epistemological discussions addressing questions such as what is knowledge, how is it acquired and how do we know what we know. Much of the debate in this field has focused on analyzing the nature of knowledge and how it relates to similar notions such as truth, belief, and justification.
It also deals with the means of production of knowledge, as well as skepticism about different knowledge claims, which links into ontological discussions such as what is existence and is there an world out there that we can objectively research.
This is again reflected in the discussions on which approach to synthesis is appropriate to certain questions. And on that point I wish I was a quant researcher because the variety of paradigms and theoretical frameworks outside of positivism is so much more complex, even on the level of what is considered truth and trusthworthiness.
It is not a matter of trying to convince the world of what is wrong or wright. Each of those perspectives has its value.
You just have to take a stance and realize that your method needs to be consistent with what you stand for and you need to be transparant about your particular stance.
With his in the back of your mind I would like to return to my scientific work on two components, crit app and synthesis.

13. QES: Stepwise approach PICO becomes SPICE
S Setting: Western
P Perspective: Mothers in a post-natal situation
I Intervention/(topic of) Interest: diet
(C) Comparison: (training, placebo)
For some qualitative questions there isn’t an intervention to be evaluated.
E Evaluation: Attitude, view, opinion on...
Elements that hinder the diet
Impact of the diet on the general welbeing of the mother
Opinions on how to best integrate the diet in daily family life
 Targeted suggestions on how to make things work.

14. QES: Stepwise approach Searching qualitative evidence: problems
Little result from searching the major databases
30% database & handsearch
50% ‘snowballing’
24% personal contacts
(Greenhalgh, 2005)
Major problems:
Bad indexing
Less developed and tested methodological filters
MeSH-term: qualitative research
General rules:
Use methodological filters
Examine references
Use the related article features in major databases
Search for citations (backword and forward)

15. QES: Stepwise approach Searching qualitative evidence: INTERTASC. www
QES: Stepwise approach Searching qualitative evidence: INTERTASC
Methodology
nursing methodology research OR (qualitative OR ethnolog* OR ethnog* OR ethnomethodolog* OR emic OR etic OR phenomenolog*[Title/Abstract]) OR (hermeneutic* OR heidegger* OR husserl* OR colaizzi* OR giorgi* OR glaser OR strauss [title/abstract]) OR (kaam* OR manen OR participant observ* OR constant compar* [title/abstract]) OR (focus group* OR grounded theory OR "narrative analysis" OR lived experience* OR life experience* [title/abstract]) OR (theoretical sampl* OR purposive sampl* OR ricoeur OR spiegelberg* OR merleau [title/abstract]) OR (metasynthes* OR meta-synthes* OR metasummar* OR meta-summar* OR metastud* OR meta-stud* [title/abstract]) OR (maximum variation OR snowball [title/abstract]) OR (field stud* OR field note* OR fieldnote* OR field record* OR action research [title/abstract]) OR (thematic analys* OR content analy* OR unstructured categor* or structured categor* [title/abstract]) OR (participant observation* OR nonparticipant observation* OR non participant observation* [title/abstract]) OR (tape recording OR "tape record*" OR "video record*" OR "audio record*" OR taperecord* OR audiorecord* OR videotap* OR videorecord*) 
Citations/authors
Techniques

16. QES: Stepwise approach Searching qualitative evidence: debate
We need a sensitive approach to searching that includes all potentially relevant studies.
We need a search strategy that is specific and purposeful, including studies that are relevant to our synthesis. This might include working with a saturation point for inclusion.
 Related to the goal and/or approach of a particular study.
Jamaar we hebben net geleerd dat we sensitief moeten zoeken.
Ja, in kwantitatief onderzoek, waarin we resultaten eigenlijk willen samenbrengen om een sterker effect te gaan bewijzen.
In een kwalitatieve synthese zijn we daar niet naar op zoek.
In veel typen van kwalitatieve syntheses streven we eigenlijk naar de formulering van een theorie of een model, en is het misschien niet altijd nodig om alle studies te overwegen om een theorie te genereren.
We komen op die verschillende soorten ook nog terug.
The debates around searching rests on whether being systematic and explicit, or comprehensive and exhaustive or all four are required
Advocates of purposive searching suggest thematic saturation is the goal, hence searching need not be comprehensive and exhaustive once reviewers have achieved a holistic interpretation of a phenomena
Advocates of comprehensive and exhaustive searching suggest the methods should be systematic, explicit, and predicated by the need to meet certain standards associated with “systematic reviews”
Searching: addressed in upcoming presentation

17. QES: Stepwise approach Critically appraising evidence: debate
The more you appraise the more you obstruct the creative, interpretive nature of qualitative research. The most important criterion is relevance!
The more you appraise the smaller the chance that you would end up with statements and conclusions that do not represent the truth. The most important criterion is (methodological) quality!
Verhaal van het aspirientje bij oma. Hoe meer water in het glas, hoe meer je moet opdrinken. Hoe minder water in het glas hoe straffer de smaak. Discussie waar je moeilijk uitgeraakt. Je moet daar een bepaald standpunt in innemen. Het zelfde geldt voor de discussie of je nu al dan niet een kritische beoordeling moet uitvoeren van kwalitatief onderzoek.
Gevecht tussen de olifant en de eland.
Kwalitatief onderzoek moet aan dezelfde criteria onderworpen worden
Een lijst van criteria specifiek ontworpen voor kwalitatief onderzoek is noodzakelijk
Kwaliteitscriteria moeten beschouwd worden als een leidraad voor ‘good practice’, eerder dan een rigied systeem
Criteriologie is geen goede zaak voor het beoordelen van kwalitatief onderzoek
‘Critical appraisal’ addressed in upcoming presentation

18. PART 2: Qualitative Evidence Synthesis (QES) approaches
QES in the scientific literature
The role of QES (in SR)
A list of developed approaches
Comparing characteristics of two commonly used approaches
This brings me to the last component of this session, QES itself.
And I want to introduce you to the role of QES, particularly in the context of SR.
I will present a list of developed approaches
Compare characteristics of a selection of these.

19. QES in the public health literature
Timetrend based on the reviews from:
Dixon-Woods & colleagues (2007)
Hannes & Macaitis (2011)
First of all, let me tell you that this graphic shows that the interest from researchers in QES is definately rising.
The first QES has been detected in 1994 and in 2007 we reached an acceptable number of around 30 QES published.
The 2004 and 2008 drop should not be taken too seriously, because I suspect it relates to indexing isssues from the major databases.
Dixon-Woods did her search early 2005 and I did mine early 2009.
Not all studies have been indexed by then.

20. The role of QES in SR: different aims
Nature
Aim
Studies
Result
Synthesis
Meta- analysis
Evidence-synthesis
Qualitative
Make sense of data
Basic comparability
Added value in content
Through interpretation
Quantitative
Accumulating
Strictly comparable
More power
Through data

21. The role of QES in SR
Inform: reviews by using evidence from qualitative research to help define and refine the question, and to ensure the review includes appropriate studies and addresses important outcomes (scoping review)
Enhance: reviews by synthesizing evidence from qualitative research identified whilst looking for evidence of effectiveness (process and implementation issues).
Extend: reviews by undertaking a search to specifically seek out evidence from qualitative studies to address questions directly related to the effectiveness review (mixed method or multilevel synthesis).
Supplement: reviews by synthesizing qualitative evidence within a stand-alone, but complementary review to address questions on other than effectiveness (stand-alone or parallel synthesis).
Note:These should be distinguished from a narrative report of a quantitative SR. When individual studies cannot be pooled quantitatively (reason: heterogeneity), they may still have useful qualitative information to be shared with the reader.
QES have a particular role to play, either as a stand-alone product or within the context of a SR.
They can inform….
This is often refered to as a scoping review, which is an extremely useful thing to do.
Let’s take the example of a review evaluating the effectiveness of inviting lower educated people to skills based courses.
Now, all but one trial included assumed that a written invitation would lead to a higher attendance.
Qualitative research, however, on the reasons why people do not attend course programs to increase their skills revealed that almost half of them had difficulties understanding the pamflets.
Which is why it is so damned important to get your question right, from the beginning.
They can also enhance a SR by looking at aspects of process and implementation related to included trials or quantitative studies in the review, either within the study or through an active search of reports that address these issues for the included trials.This would nuance a conclusion because researchers would be able to give indications on why a particular intervention might not have worked or why it might have worked very well. No seperate search or critical appraisal exercise. Selected original studies are screened for process and implementation issues or qualitative evidence as well.
QES can also be used to extent on a SR. In this case, researchers would specifically undertake a search to detect Qual research as well as quant research at the same time. This is generally refered to as mixed method reviews or multilevel synthesis. : conducted as seperate streams, but product of each synthesis is combined.
Or they can supplement reviews in addressing questions that cannot be answered quantitatively. These are generally labeled as stand-alone or parallel syntheses. These are also conducted as seperate streams, but the product of QES used in parallel and juxtaposed alongside to aid the interpretation of synthesized trials.

22. Example:Narrative report
Adding a bit won’t work. The need: full SR of QR !
Here, the table shown summarises the characteristics of each study.
There is a similar table (not shown) which details the outcomes for each study.
The text here is the interpretation of how the studies fit together.
Though this is not typical. There is really no such thing as ‘typical’ with NS, which brings me onto the next slide…
Narrative synthesis because pooling is not possible? Rather not labelled as QES…

23. Recommendation for interventions
Example: Extending Review
Which interventions match recommendations derived from children’s views and experiences?
‘Mixed Method Approach’ Children & Healthy Eating - EPPI-centre: eppi.ioe.ac.uk
Children’s Views
Trials
Recommendation for interventions
Good quality
Other
Do not promote fruit and vegetables in the same way
None
Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one 5
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 6
Here is an example of some of the matrix. We listed the recommendations for interventions down the left and then noted which interventions actually built on the recommendations and those which did not. We used the good quality outcome evaluations to assess whether or not there was evidence to support the children’s views, whether the evidence was contrary to what the children were suggesting, or whether there was a gap in available evidence. The interventions which had not been evaluated well were identified as building on a potentially fruitful facilitator and were recommended for rigorous evaluation.
So for the first recommendation we identified a research gap.
In the third recommendation – to reduce the emphasis on health messages – we identified 5 well evaluated interventions. Two of these provided results on the same outcome so we were able to conduct a statistical sub-group analysis dividing the studies between those which emphasised health messages and those which did not.

24. Example: Supplementing review
Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living with HIV Infection. Qual Health Res vol 10, nr 3, 2000.:
Understand experience of adults living with HIV infection
21 articles
Method: constant comparative analysis
Themes:
Finding meaning in HIV: death, loss, surviving
Shattered meaning: fear, irreparable
Human connectedness: meaningfull relationships alienation, isolation
Focusing on the self: actions to enhance fysical and emotional health
Negociating care: active role
Dealing with stigma: personal, family, society

25. The role of QES in SR
Provide evidence on the subjective experience of those involved in developing, delivering and receiving an intervention
OR/AND
Provide a research-based context for interpreting and explaining trial results
How to achieve change (more effectively)?
How to improve interventions?
How to ‘fit’ subjective needs?
What other type of interventions might be needed?
Reveal the extent to which effective interventions are actually adopted in policies and practice (what are barriers and bridges?)
Contribute to the understanding of heterogeneity in outcomes
To put it differently,

26. A list of developed approaches
QES-approach
Developed
Meta-ethnography
Noblit & Hare, 1988
Meta-summary
Sandelowski & Barosso (2008)
Meta-study
Paterson et al (2001)
Realist synthesis
Pawson et al (2004)
Meta-narrative mapping
Greenhalgh (2005)
Critical Interpretive Synthesis
Dixon-Woods et al (2006)
Narrative Synthesis
Popey et al (2006)
Textual narrative synthesis
Lucas (2007)
Ecological triangulation
Banning (unknown)
Framework synthesis
Brunton et al (2006), Oliver et al (2008)
Meta-interpretation
Weed (2005)
Meta-aggregation
Joanna Briggs Institute (2001), Hannes and Lockwood (2010)
Bayesian meta-analysis
Roberts et al (2002), Voils et al (2009)
Content analysis
Evans and Fitzgerald (2002), Suikkala & Leino-Kilpi (2000)
Case Survey
Yin & Heald (1975), Jensen & Rodgers (2001)
Qualitative Comparative analysis
Cress & Snow (2000)
Thematic synthesis
Thomas & Harden (2008)
Cross-case analysis
Miles & Huberman (1994)
Grounded theory
Finfgeld (1999) Kearney (2001), Eaves (2001) …
There is meta-analysis for summarizing results from quantitative studies and then there is a list of at least 19 different approaches to conduct a QES, depressing isn’t it.
For some the very idea of qualitative, cross-study ‘synthesis’ is epistemologically impossible
I can’t even say that I am all inclusive here.
But one could say that the first 12 have been developed particularly for synthesis purposes, while the latter half basically draws on basic qualitative research methodologies to outline an approach to synthesis.
Meta-ethnography, developed by Noblit and Hare (who were educational scientists by the way), is the most commonly used approach. It has been picked up by some distuinguished scholars in the field of health care and has been copied by other wannabees ever since. It is a highly interpretive approach with an aim to develop an understanding and a theory.
Meta-summary is one of the few meta-aggregative approaches on the market. Most are interpretive. Apart from grouping qualitative findings it also calculates frequency and intensity effects and it promotes itself as a useful technique to be considered in mixed-methods research.
Meta-study is one of the earlier approaches developed by B. Paterson. She has written one of the most easy accessible books on synthesis. It is not so much a concrete technique for synthesis, but rather a theoretical framework from which to start from.
Beautiful realist synthesis: the world would be so much better of with more researchers being engaged in realist synthesis but those who do, I declare them completely nuts. Ray Pawson developed the method from a policy perspective and being confronted with the complexity of decision making on this level it has been developed as a multi-method, multi-disciplinary evidence base with a focus on theories that underly interventions. It has particulary been designed to answer questions such as what works, for whom, why in which circumstances and how. Very appealing, but very difficult to handle indeed.
Meta-narrative mapping, has many parallels with realist synthesis in that it draws on a variety of insights, but its focus is on storied accounts, preferably a sequence of events in time, from different research traditions. It focusses on the epistemological understandings of different schools of thought. I must admit that I am not very familiar with this approach, neither the one from Popay on narrative synthesis. To me, narrative re
Critical interpretive synthesis: Highly inspired by meta-ethnography, but with a particular emphasis on a fundamental critique on what is summarized, questioning the ways in which the literature had constructed or handled a particular phenomenon of interest.
Textual narrative synthesis, ecological triangulation, meta-interpretation and framework synthesis have not yet known a substantive uptake in the field, neither am I able to define their characteristics. But it is good to know that they are there to be explored.
Another story is the meta-aggregative approach developed by JBI, highly promoted and used within their world wide network of centres. Like meta-summary it is one of the few that takes on aggregative and not an explicit interpretive approach to synthesis. It has mainly been inspired by pragmatism and its outcome is a series of recommendations to improve practice (and policy). It mirrors the procedures taken on by review org such as CC and C2. This is currently the only one I have made a substantial contribution to.
Bayesian approach to meta-synthesis should be situated in the context of mixed methods research. methods for using probability models it can be either quantizising qualitative data, qualitizing quantitative data by identifying themes in both or using data from the qualitative reports to create the prior distributions.
All others are just approaches that have been build on basic qualitative designs such as content analysis, case survey, thematic synthesis, cross-case analysis and grounded theory, with which you are probably familiar at least at an introduction level.

27. A list of developed approaches
MAKING SENSE OF THE MYRIAD OF QUALITATIVE EVIDENCE SYNTHESIS METHODS
No clear guidance about how to determine which of the existing synthesis methods best fits a particular purpose.
possible considerations…
For you all to make sense out of this myriad of QES approaches is hardly feasible.
It would give you the same feeling as standing in front of this traffic light having to decide which one applies to you.
So what I’ll do is give you a sneak peak in the introduction chapter of the book on QES approaches we are currently producing.

28. A list of developed approaches
Consider
Questions to be asked
The nature of the research
Will the synthesis method result in the expected and desired outcomes?
Is the method congruent with the goals of the synthesis project?
Does the primary research support the method?
The nature of the researcher
How tolerant is the researcher to the amount of structure and ambiguity that is inherent in the method?
Is his/her epistemological stance congruent with that of the synthesis method?
The nature of the research team
Is there the necessary mix of disciplinary, methodological, and other perspectives among the research team to enact this method?
 Is the expertise needed for this method (e.g., statistical analysis, theoretical) available?
Resource requirements
How much personnel, time and effort are required for this method?
Is there adequate funding to support expenses incurred in implementing this method?
Paterson, B. (2011). Introducing Qualitative Evidence Synthesis. In Hannes, K., & Lockwood, C. (eds.). Qualitative Ev. Synthesis: choosing the right approach. Wiley-Blackwell, UK.
There are some elements that could help you in deciding which approach best fits your purpose.
First of all we need to consider the nature of the research. What do we want to get out of it, a theory, recommendations, a comparative matrix etc? And if e.g. our goal is theory building, can we use the set of original articles to compile this?
Secondly, there is the nature of the researcher and his epistemological stance. Different ways of knowing of different researchers is one way of explaining why we have-and need- different methods for synthesis. There is a huge difference between those taking an ideological stance in the discussion and those having a realistic point of view. There is a huge difference between e.g. an idealist and a realist. While the former might be allergic to anything that structures or prescribes in advance or is aggregative in nature, the latter might be in favour of a concrete set of rules.
A research team should also be able to conduct a particular type of synthesis. You might need some specific skills for a bayesian synthesis or a critical interpretive synthesis that you do not require for others.
And of course, there is the issue of resources: how much personnel, time and effort does a particular method require?
If you are able to make that puzzel, it will definately guide your choice, although I hope that the latter two arguments are given far less weight than the first two considerations.

29. Qualitative Evidence Synthesis approaches
Decision to conduct a qualitative evidence synthesis
To aggregate / summarise / integrate qualitative data to address specific questions in relation to a Cochrane intervention review
To interpret synthesised qualitative evidence and develop explanatory theory or models
Primarily to integrate and interpret qualitative and quantitative evidence within a single approach
or integrated model
Can be used to develop
explanatory theory
Purpose of the additional
qualitative synthesis
Thematic analysis without theory generation
Meta-aggregation
Realist review
EPPI (mixed) approach (thematic)
Narrative synthesis
Meta-ethnography
Thematic analysis with theory generation
Grounded theory
Product
Explanatory theory, analytical or conceptual framework or interpretative framework/ mechanism
Aggregated findings from source papers
This is not an exclusive list. I think all the developed approaches deserve a spot here.
Bayesian synthesis
Critical interpretive synthesis

30. To aggregate findings of included studies and aid decision making
Comparing characteristics of different approaches (developed for synthesis)
Meta-ethnography
Meta-aggregation
Purpose
Seek and reveal similarities and differences, achieving a degree of innovation
To aggregate findings of included studies and aid decision making
Epistemology
Idealism
Realism
Quality Assessment
Not discussed, relevance argument
Required, using standardized critical appraisal instrument
Synthesis
Refutational and reciprocal translation, line of argument synthesis
Aggregation of findings into categories and of categories into synthesized findings
Outcome
Higher order interpretation of findings, theory
Standardised chart informing practice and policy
What follows is an attempt to outline some characteristics of a subset of approaches to QES.
There is no objective argument for why I have selected these approaches other than the fact that these apply to QES (could be used as a stand-alone approach) rather than mixed methods and the fact that I am somehow familiar with either the approach or the developers of the approach, which makes it a lot easier to reflect on comparative and distinguishing characteristics.
So let’s have a look at meta-ethnography and its sister approach critical interpretive synthesis. They are both underpinned by a rather idealistic paradigm subscribing to the theory that knowledge is a social construction and that therefore multiple realities exist as opposed to meta-aggregation that takes an explicit realistic stance. Its a method that is more grounded in reality that aims for what can be reached with a certain amount of assurance. This difference explains why meta ethno and critical interpretive synthesis are at the interpretive side of the spectrum while aggregation is more on the aggregative side of the spectrum trying to avoid too much of an interpretation.
The main purpose of meta-aggregation is to aid decision making in providing clear advice and implications for practice. Meta-ethnography and crit interpretive synthesis are more focussed on the generation of theory through revealing similarities and differences and building on these to create a new understanding. CIS applies a critic to the existing material and tries to deconstruct it as a means of contextualizing findings. It does include more of a judgment of what individual studies add to a particular discussion.
Quality assessment of included studies is not really an issue in meta-ethnography, although many examples in practice do use an appraisal in their projects. Like CIS inclusion is based on the relevance of a particular study to the synthesis, while in meta-aggregation it is a prerequisite for a study to be of high quality to be included.
The synthesis part in a meta-ethnography consists of a refutational or/and reciprocical translation. Reciprocal translation basically means that you would search for similarities between studies and than translate the metaphors used in one study in to another study. Refutational translation would focus on quite the opposite. It would seak out differences in meaning. They require a more elaborated set of arguments because you would not only translate the disagreements but also the relationship between different accounts made. Another possible way of presenting findings is a line of argument synthesis, which looks at the whole, based on selective studies of the parts. It would put similarities and disagreements in a new interpretive context (one step further).
Critical interpretive synthesis to a certain extent even more iterative and dynamic in that it critiques the way in which the literature in the area have chosen to represent a particular phenomenon of interest. And it is this critique that further informs sampling and in the end generation of theory.
Meta-aggregation if far more linear in its approach. It aggregates findings in categories and categories in synthesis that are displayed as concrete advice. Its outcome is a standardized chart informing practice, while the other two would come up with a theory or new interpretation of findings.

31. PART 3: Recent developments
Context-specific versus multi-context reviews
Mixed methods reviews
This brings me to the last component of this session, QES itself.
And I want to introduce you to the role of QES, particularly in the context of SR.
I will present a list of developed approaches
Compare characteristics of a selection of these.

32. Context-specific versus multi-context syntheses
Quantitative Review
Qualitative Review
A recent systematic review from Zief and colleagues (2006) evaluating the impact of after-school programs on student outcomes opted for a limitation to the North American region because school c
contexts worldwide would differ significantly. The search was limited to interventions implemented in North America. We did not specifically
search for any international studies because the intervention context would likely be much
different than that of U.S. and Canadian programs. Furthermore, Lauver’s (2002) preliminary
but extensive prior search found no relevant studies from countries other than the United States.
But more and more reviews appear that
I recently conducted a qualitative evidence synthesis on obstacles to implementing EBP in Belgium and I opted for a context specific synthesis for the vary same reason, that it makes little sense to do a multi-context review, since health care system vary substantially and so are the obstacles related to implementation.
This is not to say that there are no elements in this review that would not be applicable in other countries, but we found some issues that typically related to the Belgian health care system, such as the complained of the old fashioned referals from medicals. This would most likely not surface in countries where patient have direct access to physiotherapy, which is not the case in Belgium. And in searching for potential reasons for that we bumped into the strong lobby of medical doctors that would most likely defend their particular niche. To name only one example.

33. Context-specific versus multi-context reviews
Exhaustive search
Little access to or knowledge of local databases and experts
Targets a broad audience (but no-one in particular)
Findings may be too general
Risk of downplaying important local characteristics
Context may get lost
Potential low level of acceptance in end-users
Wide ranging in scope
Ability to cross compare different settings
Works for topics were little heterogeneity between settings is expected
Findings are more likely transferable to a broad range of settings
Context-specific reviews
Selective search
(related to context)
Access to and knowledge of local databases and experts
Only relevant to the ‘happy few’.
Findings are less likely transferable to other settings
Targeted audience
Highly relevant to practice and policy
Maintains integrity with the context reported in original studies
Findings may induce a higher level of acceptance in the end-users
Ofcourse there are risks in trying to be sensitive to a local context as well.
I recently did a fellowship in an australian institute where I was involved in developing a best practice sheet on the use of dummies and the potential negative and positive effects of it. It turned out that it had a negative effect on breast feeding, but at the same time it decreased the risk for sudden death syndrom. Now, australian health care workers are very much proponents of breast feeding. So they were very keen on presenting the negative effects, because it would fit the policy of australian government. We had discussions about the need to present a more balanced picture, but in the end, it was decided that the sheet would not be published. So there is certainly a risk in considering the global context.

34. Context-specific versus multi-context reviews
Context-specific syntheses do well in responding to the needs and policies of a targeted setting.
Multi-context syntheses assist in building a cumulative knowledge base and are an excellent choice when little heterogeneity is expected.
Integrating the best of both
Umbrella reviews, in which insights in a particular phenomenon generated from different settings could be summarized.
Transcontextual adaptation, which means modifying insights in such a way that they become relevant and reply to the needs and policies of a targeted setting.
We would maintain the links with a certain setting and we would allow readers to move back to the original context specific reviews. Although it is an interesting idea to explore, it is not very likely that this will happen in the short term.

35. Mixed methods reviews: definition
•‘Mixed research synthesis’ (Sandelowski et al., 2006)
•‘Mixed studies review’ (Pluye et al., 2009)
•‘Mixed methods synthesis’ (Harden & Thomas, 2005)
‘Mixed methods research synthesis’ = A synthesis in which researchers combine primary qualitative, quantitative, and mixed methods studies, and apply a mixed methods approach in order to integrate those studies, for the broad purposes of breadth and depth of understanding and corroboration (Heyvaert, Onghena & Maes, 2011).
•Promising utility for research and practice: Combining strengths of qualitative and quantitative techniques and studies

36. Paired: Concurrent but separate review example – 1 team
Cochrane
Intervention
Review
Qualitative
Evidence
Synthesis
Research Question(s)
Synthesis of Qualitative
and Quantitative Studies
Paired: A/S -QUAL + QUAN
–Equal status
-Concurrent
Lins S, Rücker G, Motschall E, Langer G, Antes G, Meyer G. Efficacy and experiences of telephone counselling for informal carers of people with dementia (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5. Art. No.: CD DOI: / CD009126
Qualitative thematic synthesis related to the efficacy of telephone counselling
Leiknes KA, Berg RC, Smedslund G, Jarosch-von Schweder L, Øverland S, Hammerstrøm KT, Høie B. Electroconvulsive therapy for depression (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5.
Meta-aggregative approach to qualitative synthesis
Lins, the same team conducts the whole review
Embudleniya, different teams are conducting the seperate streams of the review.

37. Paired: Concurrent but separate review example – 2 teams
Cochrane
Intervention
Review
Qualitative
Evidence
Synthesis
Research Question(s)
Synthesis of Qualitative
and Quantitative Studies
Paired: A/S -QUAL + QUAN
–Equal status
-Concurrent
Embuldeniya, G. Et al. Perceived impacts and experiences of peer support in chronic disease
Qualitative meta-ethnography exploring issues related to an ongoing Cochrane effectiveness review (Doull MJ, O'Connor AM, Robinson VA, Tugwell P, Wells G. Peer support strategies for people with chronic disease to enhance health and promote health equity. Cochrane Database of Systematic Reviews 2005, Issue 3.)

38. Retrospective, stand alone review example
-Equal status
-Sequential
Noyes, J., Popay, J (2007).Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced Nursing, 57,
Narrative Summary approach extending/enhancing a Cochrane intervention review Volmink J. & Garner P (1997) (updated 2002, 2006) Directly observed therapy for treating tuberculosis. Cochrane Database Systematic Reiews 19(2), CD
Jordan J, Rose L, Dainty KN, Noyes J, Clarke S, Blackwood B. Factors which impact on the use of weaning protocols for reducing the duration of mechanical ventilation in adults and children.
Meta-ethnography extending/enhancing a recent Cochrane intervention review (Blackwood B, Alderdice F, Burns KEA, Cardwell CR, Lavery G, O'Halloran P. (2010) Protocolized versus non-protocolized weaning for reducing the duration of mechanical ventilation in critically ill adult patients. Cochrane Database of Systematic Reviews (in press).

39. QES: conclusion
If you wish to embark in the field of qualitative evidence synthesis you will soon realise that
There are no fixed standards
There is no general consensus (and will there ever be one?)
You shall have to travel slowly,
By your own means (but the CQRM- and CP&I-groups can help)
On small and potentially difficult roads…
BE PREPARED FOR A POTENTIAL DELAY!
There are no magic bullets for qualitative evidence synthesis in the way they have been developed for conventional systematic reviews. There are, however, a wide range of methods available that, if used appropriately, can lead to trusthworthy qualitative evidence.
Those who want to engage in QES will soon realize that
There is only one certainty in the field of qualitative research and that is that you should be….
This is probably where I should end and leave you soom room for thoughts, questions and clarifications.