Presentation is loading. Please wait.

Presentation is loading. Please wait.

Postgraduate Course 8. Evidence-based management Step 3: Critical appraisal of studies.

Similar presentations


Presentation on theme: "Postgraduate Course 8. Evidence-based management Step 3: Critical appraisal of studies."— Presentation transcript:

1 Postgraduate Course 8. Evidence-based management Step 3: Critical appraisal of studies

2 Postgraduate Course 5-step approach EBMgt is a 5-step approach 1.Formulate an answerable question (PICOC) 2.Search for the best available evidence 3.Critically appraise the quality of the found evidence 4.Integrate the evidence with managerial expertise and organizational concerns and apply 5.Monitor and evaluate the results

3 Postgraduate Course Intermezzo How to read a research article?

4 Postgraduate Course 1. Title 2. Abstract 3. Introduction 4. Background / review of literature 5. Organizational context 6. Methodology 7. Results 8. Discussion Structure of an article

5 Postgraduate Course 1.Title Not always a good indication of the content of the article Example: “ The Risks of Autonomy: Empirical Evidence for the Necessity of a Balance Management in Promoting Organizational Innovativeness” ?????? Structure of an article

6 Postgraduate Course 2. Abstract Sometimes unclear. What should be in it: a summary of the the research question, key methods, results and conclusions of the study Structure of an article

7 Postgraduate Course Research questions occur in the context of an already- formed body of knowledge. The background should address this context, help set the rationale for the study, and explain why the questions being asked are relevant. 4. Background / review of literature 3. Introduction Should contain the research question (PICOC!) or hypotheses tested Structure of an article

8 Postgraduate Course Should describe exactly how the research was carried out: 6. Methodology - Sample: characteristics, selection, number, non-response - Measures: description of tests / questionnaires (validated?), data, outcome measures - Procedure: study design (qualitative, quantitative, controlled?) 5. Research setting (organizational context) Structure of an article

9 Postgraduate Course - Interpretation of the results / relation to theory - Comparison with the results of other studies - Weaknesses / limitations of the study - Implications - Recommendations 8. Discussion 7. Results Should tell the reader what the findings were. All outcome measures must be reported and confidence intervals for effect sizes should be presented. Structure of an article

10 Postgraduate Course In general  Don’t let yourself be taken in by scientific jargon and complex use of language!! Good articles are written in plain English.  Even authorative journals with a high impact factor contain bad articles and vice versa.  Focus on research question, study design and outcome.  Don’t worry about statistics!  Be critical!! Always ask yourself: does this make sense?

11 Postgraduate Course Critical appraisal of studies

12 Postgraduate Course Critical appraisal: quick and dirty Is the study design appropriate to the stated aims? Are the measurements likely to be valid and reliable? Was there a relevant effect size? Is the outcome (population, type of organization) generalizable to your situation?

13 Postgraduate Course Levels of internal validity 1.Were there enough subjects in the study? 2.Was a control group used? 3.Were the subjects randomly assigned? 4.Was a pretest used? 5.Was the study started prior to the intervention or event? 6.Was the outcome measured in an objective and reliable way? 6x yes = very high (A) 5x yes = high (A) 4-3x yes = limited (B) 2x yes = low (C) 1-0x yes = very low (D)

14 Postgraduate Course Always ask yourself: How did they measure that? Is that a reliable way to measure?

15 Postgraduate Course Appraisal Critical appraisal questionnaires

16 Postgraduate Course Standard appraisal questions 1. Did the study address a clearly focused issue? 2. Is the sample size justified? 3. Is the design appropriate to the stated aims? 4. Are the measurements likely to be valid and reliable? 5. Are the statistical methods described? 6. Did untoward events occur during the study? 7. Were the basic data adequately described? 8. Do the numbers add up? 9. Was the statistical significance assessed? 10. What do the findings mean? 11. Are important effects overlooked? 12. What implications does the study have for your practice?

17 Postgraduate Course Appraisal of a controlled study 1.Did the study address a clearly focused issue? 2.Were subjects randomly allocated to the experimental and control group? If not, could this have introduced bias? 3.Are objective inclusion / exclusion criteria used? 4.Were groups comparable at the start of the study? 5.Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias) 6.Were outcomes assessed blind? If not, could this have introduced bias? 7.Is the size of effect practically relevant? 8.Are the conclusions applicable?

18 Postgraduate Course Appraisal of a cohort / panel study 1.Did the study address a clearly focused issue? 2.Was the cohort / panel recruited in an acceptable way? (selection bias) 3.Was the cohort/ panel representative of a defined population? 4.Was a control group used? Should one have been used? 5.Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias) 6.Was the follow up of cases/subjects long enough? 7.Could there be confounding? 8.Is the size of effect practically relevant? 9.Are the conclusions applicable?

19 Postgraduate Course 1.Did the study address a clearly focused issue? 2.Were the cases and controls defined precisely? 3.Was the selection of cases and controls based on external, objective and validated criteria? (selection bias) 4.Are objective and validated measurement methods used and were they similar in cases and controls? (misclassification bias) 5.Did the study incorporate blinding where feasible? (halo-effect) 6.Was there data-dredging? 7.Could there be confounding? 8.Is the size of effect practically relevant? 9.Are the conclusions applicable? Appraisal of a case-control study

20 Postgraduate Course Assessment of a survey 1.Did the study address a clearly focused issue? 2.Was the sample size justified? 3.Could the way the sample was obtained introduce (selection)bias? 4.Is the sample representative and reliable? 5.Are the measurements (questionnaires) likely to be valid and reliable? 6.Was the statistical significance assessed? 7.Are important effects overlooked? 8.Can the results be generalized?


Download ppt "Postgraduate Course 8. Evidence-based management Step 3: Critical appraisal of studies."

Similar presentations


Ads by Google