1. How do we know when something is implemented?
ImpleMentAll Midterm Workshop
10th October 2018
How do we know when something is implemented?
Tracy Finch
Northumbria University Newcastle

2. Why is understanding implementation (better) important?
Evidence to practice – still problematic
An intervention may be proven as safe, beneficial, or more effective – but it may not be implementable
May help us understand ‘non-significant’ study results – due to intervention lacking efficacy or not successfully implemented?
Implementation science - theories, frameworks and tools – but not always easy to understand or to use for own purposes.
Style: one bullet point appearing at a time
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018 2

3. Implementation: One definition
Implementation involves all activities that occur between making an adoption commitment and the time that an innovation either becomes part of the organizational routine, ceases to be new, or is abandoned (…) [and the] behavior of organizational members over time evolves from avoidance or non-use, through unenthusiastic or compliant use, to skilled or consistent use. (Linton, 2002: 65)
Linton, J.D. (2002) ‘Implementation Research: State of the Art and Future Directions’, Technovation 22(2): 65–79.
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018

4. Implementation outcomes: A multifaceted concept
Implementation, by which we mean the social organization of bringing a practice or practices into action. Embedding, by which we mean the processes through which a practice or practices become, (or do not become), routinely incorporated in everyday work of individuals and groups. Integration, by which we mean the processes by which a practice or practices are reproduced and sustained among the social matrices of an organization or institution.
May C, Mair, F, Finch, T et al. Building an interdisciplinary theory of implementation, embedding, and integration: the development of normalization process theory. Implementation Science 2009, 4.

5. Implementation outcomes: Proctor’s taxonomy
Enola Proctor, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Policy Ment Health Mar;38(2):65-76.

6. Implementation: Can we measure it?
Lewis et al (2015):
104 instruments measuring 8 implementation outcome ‘constructs’:
Acceptability (n = 50), adoption (n=19), appropriateness, cost, feasibility, fidelity, penetration, sustainability
Style: one bullet point appearing at a time
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci, 10. 6

7. Second, our review did not
distinguish between nomothetic instrumentation (i.e., instruments
in which interpretation of results are based on
comparisons to aggregated data for the instrument) and
idiographic instrumentation (i.e., individually selected
or tailored instruments of variables or functionality that
maximize applicability to individuals or context) [26].
Nomothetic measurement approaches provide the
benefit of cross-study comparison. However, the preponderance
of single use instruments may reflect a
valid argument for the need to employ idiographic
methods to optimally investigate implementation efforts.
The need for nomothetic versus idiographic instrumentation
approaches is an empirical question.
Subsequently, all instruments included
in this report will be re-assessed for their pragmatic
qualities to determine whether instruments can be
both pragmatic and psychometrically strong—a necessary
balance to advance both the science and practice of
implementation.
Lewis et al 2015
Lewis, C. C., Fischer, S., Weiner, B. J., Stanick, C., Kim, M., & Martinez, R. G. (2015). Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci, 10.

8. Will measures help to understand complexity?
“In order to better understand implementation, success measures must be standardized, the relative importance of various roles, the role of the network, and the interaction between technology and an organization must be better understood.” (Linton 2002, pg 76)
Style: one bullet point appearing at a time
Linton, J. (2002). Implementation research: state of the art and future directions. Technovation, 22:65-79.
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018 8

9. One approach to implementation outcomes: Normalization Process Theory (NPT)
“By normalization, we mean the work that actors do as they engage with some ensemble of activities (that may include new or changed ways of thinking, acting, and organizing) and by which means it becomes routinely embedded in the matrices of already existing, socially patterned, knowledge and practices.”
May & Finch. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology 2009, 43(3):

10. NoMAD: Normalization MeAsure Development

11. Available for download at www.normalizationprocess.org
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018

12. Finch TL, Girling, May et al
Finch TL, Girling, May et al. Improving the normalization of complex interventions: Part 2 - Validation of the NoMAD survey tool for assessing implementation work based on Normalization Process Theory (NPT) (in press)
Style: one bullet point appearing at a time 12

13. Structured measurement highlights complexity
Wording/ Ambiguity
The participant queries wording within the item, e.g. unsure of meaning
Timing relevance
The participant does not consider the item ‘relevant’ to the timing of the intervention
Role relevance
The participant does not consider the item ‘relevant’ to their role in the intervention
Who?
The participant has trouble with ‘who’ the item is relating to e.g. themself, or others (and who the ‘others’ may be)
Multiple Interpretations
The participant offers a response from their own perspective as well as that of others involved in the intervention, in a single response
Style: one bullet point appearing at a time
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018 13

14. Instrument development reveals complexity
Implementation work is complex and messy – revealed by ‘think aloud’
Need to think about:
Roles & participants – the ‘doers’
Intervention
Implementation trajectory/stage
To be meaningful/ fit for purpose, tailoring & adaptation is needed
– methodological tensions?
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018

15. Measuring complexity: standardizing vs adapting
“...implementation scientists might still find it challenging to develop measures of implementation climate that are sufficiently tailored to make them predictive in specific innovation-implementation contexts, yet not so tailored that they could not be used in other innovation-implementation contexts without substantial modification.” (p9)
Style: one bullet point appearing at a time
Weiner, BJ., Belden, CM., Bergmire, DM. & Johnson, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 9:118. 15

16. Pragmatic measures
Glasgow & Riley (2013). Pragmatic measures: what they are and why we need them. American Journal of Preventive Medicine. Vol 45 (2) pp 237–243.

17. How do we know when something is implemented?
Formal assessment is possible – we now have plenty of ‘tools’!
Implementation outcomes are multi-faceted:
Must be related to what you are trying to achieve
Must consider whose perspective is important
Determining if ‘something is implemented’ requires a balance of science, pragmatism & genuine stakeholder co-production
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018

18. Acknowledgements
NoMAD study – ESRC RES ImpleMentAll - European Union’s Horizon 2020 research and innovation programme under grant agreement No Colleagues – Tim Rapley, Melissa Girling, Carl May, ImpleMentAll team & many more
ImpleMentAll – G.A Midterm Workshop, Odense, 10 October 2018