1. Petter Øgland, Department of Informatics, University of Oslo
The effect of EFQM self-assessments on action research performance: Lessons on how to improve the efficiency and effectiveness of research on New Public Management
Petter Øgland, Department of Informatics, University of Oslo
NEON, 25th-27th November 2014

2. Plan for presentation Motivation (3 minutes)
Problem: Action research is important but risky design for studying NPM
Possible solution: Use TQM for controlling and improving action research
EFQM model and double-loop learning (7 minutes)
Hypothesis 1: EFQM improves EFFICIENCY of action research
Hypothesis 2: EFQM improves EFFECTIVENESS of action research
Hypothesis 3: EFQM improves LEARNING about improving action research
Testing the hypotheses in a NTAX action research study (7 minutes)
The NTAX action research is slow and costly
EFQM succeeds in diagnosing and suggesting treatments
Discussion: Controlling and improving AR within AR traditions based on interpretivism, post-positivism and critical theory
Conclusions (3 minutes)
Critical theory allows EFQM to align with action research
Action researchers should use NPM logic for improving AR performance

3. New Public Management (NPM): Managerial logic vs. practitioner logic
How can practitioner logic be understood in the context of an economic/managerial need to improve cost-efficiency?

4. Action research is a way of building knowledge by designing and evaluating social interventions (Clark, 1972)
But it is a research design with high risk for failure (Simonsen, 2007)

5. What if we applied the logic of NPM on action research?
1. Define the production process: Producing change and scientific knowledge
2. Improve the production process: Reducing costs and cycle time in AR process by use of TQM methods

6. Using the European Foundation for Quality Management (EFQM) model

7. Theoretical framework (Argyris & Schön, 1978)
1. EFFICIENCY of action research (strategy)
2. EFFECTIVENESS of
action research (goals)
3. LEARNING HOW TO DESIGN
Single-loop and double-loop learning

8. Use TQM-model (EFQM) to evaluate consequence of action strategy
Hypothesis 1:
EFQM improves
EFFICIENCY of action research (strategy)

9. Use TQM-model (EFQM) to reflect on governing variables
Hypothesis 2:
EFQM improves
EFFECTIVENESS of action research (goals)

10. Use TQM-model (EFQM) to improve AR performance
Hypothesis 3:
EFQM improves
LEARNING HOW TO DESIGN Single-loop and double-loop learning

11. Case: Trying to improve action research at NTAX by use of EFQM
Diagnosis:
NPM strategy of outsourcing IT services, managing through contracts, and controlling through ISO 9001 audits. No audit process.
Treatment plan:
Design audit process through use of action research approach. NTAX wants AR to be externally funded.
Execution and evaluation:
Eleven months of trying to get funded, but no success so far.
Learning:
NTAX research has to focus on AR method as no action is allowed.
Operational hypothesis:
Applying EFQM self-assessments will result in improved efficiency, effectiveness and learning on how to do Action Research.

12. Using the EFQM model to diagnose the action research project
Diagnosis AR:
AR strategy of proposing to design audit process through action research fails because NTAX will not allow AR without external funding.

13. Using the EFQM model to plan treatments for action research project
Treatment AR:
EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.

14. Using the EFQM model to plan treatments for action research project
Treatment AR:
EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.
Sub-criteria 4a, 4b, 4c, 4d, 4e

15. Using the EFQM model to plan treatments for action research project
Treatment AR:
EFQM model identifies improving customer satisfaction (NTAX) and improving partnership management (UiO) as the two most important issues.
Sub-criteria 1c, 2a, 2c, 3b, 3c, 4a, 5b, 5c, 5d, 5e

16. Using the EFQM model to evaluate the action research project
Evaluation AR:
Partnership with UiO improved by collaborating on grant application, but no measurable effect on customer (NTAX) because of poorly calibrated measurement methods.

17. Using the EFQM model to reflect on the action research project
Validity and reliability:
When comparing with EFQM results with earlier experiments, the observed change from 2013 to 2014 is not significant.
Learning AR:
Use feedback from rejection for writing new grant application.

18. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

19. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

20. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

21. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

22. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

23. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant

24. Analysis: Did the use of the EFQM model improve AR performance?
Analysis of AR project
Use of EFQM model
Diagnosis AR performance
Plan for improving AR performance
Execute plan and record what happens
Evaluate new AR performance level
Reflect on causes for change in AR performance
EFQM self-assessment
Use EFQM theory about causes and effects
New EFQM assessment of enablers
New EFQM assessment of results
Observed change was not statistically significant
H1, H2 & H3: Confirmed by qualitative data
H1 & H2: Not confirmed by quantitative data

25. Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)

26. Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING

27. Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if they do not result in better treatments

28. Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if they do not result in better treatments
EFQM suggests that TREATMENTS should be evaluated on objective terms

29. Discussion: Related research - interpretive AR approach
Improving AR through ethics and moral standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING, but not as a goal in itself
RICH DESCRIPTIONS are of limited use if they do not result in better treatments
EFQM suggests that TREATMENTS should be evaluated on objective terms

30. Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)

31. Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS

32. Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context

33. Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context
An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context

34. Discussion: Related research - positivist AR approach
Improving AR by making it more positivist (Kock et al, 1997)
EFQM is concerned with testing TREATMENTS, but not as a goal in itself
The truth about GENERALISED TREATMENTS are of limited use if they cannot be used in the local context
An interpretive approach is needed for DIAGNOSING and REFLECTING on the outcomes of the TREATMENTS within the local context

35. Discussion: Related research - critical AR approach
Herbert Simon was part of the progressive “research and reform” movement in Chicago in the 1930s

36. Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)

37. Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)

38. Discussion: Related research - critical AR approach
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)
Game theory as a foundation for social theory in general (Gintis, 2009)

39. Discussion: Related research - critical AR approach√
Improving AR by aligning it with Herbert Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical theory (Elster, 1982)
Game theory as a foundation for social theory in general (Gintis, 2009)

40. Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through SPC and related TQM methods

41. Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through SPC and related TQM methods
H2: EFQM improves AR effectiveness
The governing variables are updated due to causal relationships in EFQM model

42. Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through SPC and related TQM methods
H2: EFQM improves AR effectiveness
The governing variables are updated due to causal relationships in EFQM model
H3: EFQM improves AR learning
By conceptualising the social situation as a game it can be analysed in a rational manner (“empirical economics”)

43. Conclusion: Contributions to practice
Political consciousness
NPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory

44. Conclusion: Contributions to practice
Political consciousness
NPM logic is compatible with practitioner logic when viewing the conflict between the two groups through the use of game theory
Conflict resolution
Action researchers should use NPM logic for assessing and improving their own strategic performance within the game

45. Summary of presentation
Contributions to theory and implications for practice
Critical theory allows EFQM to align with action research
Action researchers should use NPM logic for improving AR performance
Why was critical theory important in this context?
The NTAX action research process needs to be improved
EFQM is successfully used for improving the NTAX action research process
Discussion: Comparison of EFQM approach for improving AR with literature on AR improvement based interpretivism, post-positivism and critical theory
Why did we believe the EFQM model would work?
Hypothesis 1: EFQM improves EFFICIENCY of action research
Hypothesis 2: EFQM improves EFFECTIVENESS of action research
Hypothesis 3: EFQM improves LEARNING about improving action research
Why was this interesting?
Problem: Action research is important but risky design for studying NPM
Possible solution: Use TQM for controlling and improving action research