1. IAEA International Atomic Energy Agency The IAEA Integrated Regulatory Review Service (IRRS) IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective Part 2: Follow-up Missions Moscow, 09 - 11 December 2014 Ahmad Al Khatibeh Section Head, Regulatory Infrastructure and Transport Safety Section, Division of Radiation, Transport and Waste Safety (NSRW) IAEA

2. IAEA Contents END 2 Assessment of IRRS Performance - Follow-up Missions: The Consultancy was asked to review seven follow-up (FU) missions to: identify evidence of the effectiveness of the IRRS in instigating improvements to national frameworks and infrastructures for safety; evaluate the effectiveness of the follow-up process; identify recurring issues; and gather evidence of progress or otherwise in implementing IRRS recommendations. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

3. IAEA Contents END 3 Assessment of IRRS Performance - Follow-up Missions: Analysis of seven follow-up (FU) missions indicates IRRS is effective in enhancing nuclear and radiation safety and regulatory effectiveness … … however, the IRRS FU team may choose only one of three conclusions regarding progress i.e.: the recommendation / suggestion (R/S) remains open or; the R/S is closed or; the R/S is closed on the basis of progress and confidence in effective completion. The analysis team concluded this may be inadequately objective since it takes no account of the significance of each recommendation or suggestion nor the priority associated with each of them. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

4. IAEA Contents END 4 Assessment of IRRS Performance - Follow-up Missions: Option 1 ‘Open’ is valuable because it is evidence-based and clearly demonstrates to the host that more effort or special attention is needed to achieve compliance. Option 2 ‘Closed’ is also useful since it is evidence-based and demonstrates real improvement to both the host and to the worldwide community Option 3 ‘Closed on the basis of progress and confidence’ however, offers little additional information to the host or worldwide community. In effect, it merely says the issue remains open. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

5. IAEA Contents END 5 Assessment of IRRS Performance - Follow-up Missions: Option 3 is often chosen on the understanding the host has proposals, plans or draft documents associated with improvement, but in some cases there is little evidence these good intentions will be realised in due course. Moreover, Option 3 is often added to the number of recommendations and suggestions stated to be ‘closed’ thus, potentially statistically distorting the real picture of progress. Analysis of IRRS FU reports indicates this tendency to be significant and thus, Option 3 may need to be reworded. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

6. IAEA Contents END 6 Assessment of IRRS Performance - Follow-up Missions: Without questioning the host country’s determination to ‘close’ as many recommendations and suggestions as possible, statistics alone do not indicate the extent of improvement to safety or overall progress. To fully assess the extent of improvement in safety requires in-depth examination of the extent and effectiveness of implemented measures validated during an IRRS FU mission. e.g.: Closing a recommendation on an inspection procedure is unlikely to be as fundamental to improvement of national infrastructure as promulgation of a new law. This suggests the need for changes in the IRRS FU process. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

7. IAEA Contents END 7 Assessment of IRRS Performance - Follow-up Missions: ‘Lesson-learned LL1’ IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

8. IAEA Contents END 8 IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective Lesson-Learned LL1 LL1 Lessons Learned: Closing recommendations and suggestions ‘on the basis of progress and confidence in effective completion’ offers little additional information to the host country or worldwide community and may distort the real picture of progress. LL1: Proposal for Consideration P1 Proposal: IAEA should consider rewording the conclusion option; ‘closed on the basis of progress and confidence in effective completion’ as follows: ‘The recommendation / suggestion remains open, but evidence of progress gives confidence of effective completion in due time’.

9. IAEA Contents END 9 Assessment of IRRS Performance - Follow-up Missions: The effectiveness of an IRRS mission is primarily evaluated through the conduct of a follow-up (FU) IRRS mission. The analysis team notes however, that the IRRS follow-up process does not work in accordance with IRRS Guidelines: At the end of 2010, 28 countries had received IRRS, but only 13 of these countries have since requested or received (7) an IRRS follow-up within the four year window proposed in the Guidelines. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

10. IAEA Contents END 10 Assessment of IRRS Performance - Follow-up Missions: There is insufficient evidence to determine why countries do not request a follow-up mission. A number of factors may have an impact, including the following: Follow-up missions use considerable resources and their value and benefit may not be fully recognised. The IRRS action plan may not be realistically synchronised with national plans and programmes for implementing improvement. Significant reorganization of the national framework since the original mission may have rendered previous recommendations and suggestions irrelevant. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

11. IAEA Contents END 11 Assessment of IRRS Performance - Follow-up Missions: There is insufficient evidence to determine why countries do not request a follow-up mission. A number of factors may have an impact, including the following: Absence of real progress may influence the decision to request a follow-up mission. The original recommendations may not have been realistically achievable by the Host Country or deliverable within timescales suggested in the IRRS Guidelines. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

12. IAEA Contents END 12 Assessment of IRRS Performance - Follow-up Missions: ‘Lesson-learned LL2’ IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

13. IAEA Contents END 13 IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective Lesson-Learned LL2 LL2 Lessons Learned: The follow-up process does not work in accordance with the IRRS Guidelines. Less than half IRRS Host Countries have requested or received an IRRS follow-up within the four year window proposed in the Guidelines. LL2: Proposal for Consideration P2 Proposal: To identify why IRRS Guidelines on FU missions are not being followed, the IAEA should consider an anonymous survey of those MS having hosted IRRS, but not yet requested a follow up within the four years suggested in the Guidelines.

14. IAEA Contents END 14 Assessment of IRRS Performance - Follow-up Missions: Analysis of FU mission reports reveals: most recommendations and suggestions directed to government remain open; and the majority of recommendations and suggestions directed to the RB are closed. This suggests that implementation of recommendations and suggestions directed to government are in some way more problematic or complex than those directed to other organisations IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

15. IAEA Contents END 15 Assessment of IRRS Performance - Follow-up Missions: Analysis of FU mission reports reveals: most recommendations and suggestions directed to government remain open; and the majority of recommendations and suggestions directed to the RB are closed. In view of the above, IAEA is advising that the host state involves high level officials of government and other relevant agencies, both during the preparatory mission and at the IRRS entrance and exit meetings. IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

16. IAEA Contents END 16 Assessment of IRRS Performance - Follow-up Missions: ‘Lesson-learned LL5’ IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective

17. IAEA Contents END 17 IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective Lesson-Learned LL5 LL5 Lessons-learned: IRRS follow up missions reveal that few recommendations and suggestions directed to governments are closed. LL5: 1 st Proposal for Consideration P5 Proposal: IRRS Team leaders should ensure recommendations and suggestions to Government are realistically achievable and supported by the Host Country. The IRRS Team Leader should seek governmental assurance that due priority and resources will be given to their implementation in accordance with Host’s action plan.

18. IAEA Contents END 18 IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective Lesson-Learned LL5 LL5 Lessons-learned: IRRS follow up missions reveal that few recommendations and suggestions directed to governments are closed. LL5: 2 nd Proposal for Consideration P6 IAEA may have to focus more closely on assistance at governmental level, including raising awareness of the benefits associated with an effective governmental, legal and regulatory infrastructure for safety.

19. IAEA Contents END 19 IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective