1. Domain Scoring Guide groupNon-Scoring Guide group ICC(95% C.I.)p-valueICC(95% C.I.)p-value Domain 1 0.821(-0.303~0.995)0.043-0.333(-0.806~0.966)0.512 Domain 2 0.769(-0.675~0.994)0.0680.769(-0.679~0.994)0.069 Domain 3 0.796(0.394~0.954)0.0020.424(-0.710~0.871)0.155 Domain 4 -1.333(-15.940~0.941)0.6700.000(-6.260~0.975)0.422 Domain 5 0.888(0.431~0.992)0.0050.272(-2.696~0.950)0.312 Domain 6 0.667(-4.814~1.000)0.1820.792(-2634~1.000)0.116 Overall 0.869(0.753~0.939)<0.0010.662(0.362~0.841)<0.001 Domain Scoring Guide groupNon-Scoring Guide group ICC(95% C.I.)p-valueICC(95% C.I.)p-value Domain 1 0.815(-0.344~0.995)0.0460.682(-1.307~0.992)0.116 Domain 2 0.430(-3.137~0.986)0.251-0.762(-11.791~0.955)0.595 Domain 3 0.722(0.175~0.938)0.0110.473(-0.565~0.882)0.121 Domain 4 0.718(-1.048~0.993)0.096-0.296(-8.411~0.967)0.503 Domain 5 0.424(-1.925~0.960)0.2290.273(-2.693~0.950)0.312 Domain 6 0.000(-16.443~0.999)0.3910.000(-16.443~0.999)0.391 Overall 0.826(0.671~0.918)<0.0010.680(0.395~0.850)<0.001 Statistics To identify the distribution Descriptive analysis (Domain specific) To evaluate the reliability Intra-class correlation (Domain specific) To evaluate the consistency Association among appraisers (Each items) Study design I. Background and Purpose IV. Discussion Scoring guide reduce the inter-rater disagreemet and improve the overall reliability of the K-AGREE II instrument.  Those effects remarkable in low level CPGs development  Inter-rater disagreement reflects the healthcare environment characteristics II. Methods Table 1. Inter-rater reliability of K-AGREE Ⅱ domain scores in CPG A Effects of Korean-AGREE Ⅱ Scoring Guide on Improving the Reliability of the Scores Reliability Higher ICC in SG  Observed in almost domain and stastically significant in overall assessment Moo-Kyung Oh 1, Heuisug Jo 1,2,3, Youkyoung Lee 3,4,5 1 Department of Preventive Medicine, Kangwon National University Hospital, Chuncheon, Korea 2 Department of Health Management and Policy, Kangwon National University School of Medicine, Chuncheon, Korea 3 Executive Committee for clinical practice guideline, The Korean Academy of Medical Sciences 4 Department of Laboratory Medicine and Genetics, Soonchunhyang University Bucheon Hospital, Bucheon, Korea 5 Department of Laboratory Medicine and Genetics, Soonchunhyang University College of Medicine, Cheonan, Korea Table 2. Inter-rater reliability of K-AGREE Ⅱ domain scores in CPG B Appraiser1234 11.622.348.481 2.6221.393.596 3.348.3931-.052 4.481.596-.0521 Fig. 1. Distribution of K-AGREE Ⅱ domain scores according Scoring Guide users in CPG A Scope and Purpose Stake involve- ment Rigour develop -ment Clarity and Presen- tation Applica- bility Editorial Indepen -dence Domain Scores Scope and Purpose Stake involve- ment Rigour develop -ment Clarity and Presen- tation Applica- bility Editorial Indepen -dence Fig. 2. Distribution of K-AGREE Ⅱ domain scores according Scoring Guide users in CPG B III. Results Distribution Higher Scores and higher variability in Non-SG  Distinctive in domain 2, 3, and 5 Consistency Higher association in SG  Association improve in SG, especially distinctive in CPG B Table 3. Association of SG in CPG A Table 4. Association of Non-SG in CPG A Table 5. Association of SG in CPG B Table 6. Association of Non-SG in CPG B Appraiser1234 11.853.453.491 2.8531.651.749 3.453.6511.641 4.491.749.6411 Appraiser1234 11-.225-.434-.459 2-.2251.373.502 3-.434.3731.833 4-.459.502.8331 Appraiser1234 11.556.441.127 2.5561.479.128 3.441.4791.372 4.127.128.3721 Stakeholder involvement Applicability DomainLimitationScoring Guide Providing clear standards regarding the stakeholder and the level of participation Providing clear standards regarding the stakeholder and the level of participation Providing clear definition of implementation Providing the methodologies and resources in detail Providing clear definition of implementation Providing the methodologies and resources in detail Lack of experience in stakeholder involvement Different understanding of stakeholder Low level of CPGs implementation Confuse implementation with dissemination Current status of Clinical Practice Guidelines in Korea This study aim to examine the effects of the K-AGREE II scoring guide to reduce inter-rater differences. This study aim to examine the effects of the K-AGREE II scoring guide to reduce inter-rater differences. Korean medical guideline information center (KoMGI) A couple of CPGs development handbook More than 100 of guidelines of the last decade Korean AGREE II instrument Web-based evaluation systems KoMGI Development Dissemination Implementation Approval Appraisal Inter-rater disagreement A lack of experiences in evaluation of CPGs Differences of healthcare environments Scoring Guide