1. IRRS REFRESHER TRAINING Lecture 3
Evaluation and Documentation of Observations

2. Outline Learning Objectives Definitions Formulation of Observations
Experience from Past Missions
Refreshing IRRS Training - Evaluation of Observations

3. Learning objectives After finishing this module the trainee will:
be aware of the observation types to be offered in IRRS missions;
be able to select the observation-type adequate to the review findings;
know the process of formulating observations in a mission;
have some idea about the statistical properties of the observations.
Refreshing IRRS Training - Evaluation of Observations

4. definitions
Refreshing IRRS Training - Evaluation of Observations

5. Observations and their types
Observation: focused results of the review process, based on facts and related to possible ways of improvement or achievements to recognise, with reference to requirements in IAEA Safety Standards
Observations lead to:
Recommendation
Suggestion
Good Practice
(Remarkable Performance)
Findings
In brief: observations
Refreshing IRRS Training - Evaluation of Observations

6. Recommendation (1)
Recommendations are proposed where aspects relative to the IAEA Safety Requirements are missing, incomplete, or inadequately implemented.
Recommendations should be
Specific
Realistic
Designed to result in tangible improvements to regulatory effectiveness (within the individual context of the Host Country)
Refreshing IRRS Training - Evaluation of Observations

7. Recommendation (2) Recommendations should also
be based on IAEA Safety Requirements, with the basis documented (Safety Guide may not be a basis);
be formulated in such a way that they are succinct and self-explanatory;
clearly specify the responsible party;
use “should”-language.
Refreshing IRRS Training - Evaluation of Observations

8. Recommendation (3) When a recommendation is formulated it should
be based on checked and evidenced factual information
advise on what improvement in the regulatory regime should be made;
not advise on how it should be completed;
address the root causes, rather than the symptoms
Refreshing IRRS Training - Evaluation of Observations

9. Suggestion (1)
Suggestions are proposed when opportunities for improvement not directly related to inadequate conformance with IAEA Safety Requirements are identified.
Suggestions may contribute to improvements in national regulatory arrangements but are primarily intended to make the regulatory body’s performance more effective or efficient
Refreshing IRRS Training - Evaluation of Observations

10. Suggestion (2) Suggestions
should stimulate the RB’s management and staff to consider new approaches;
may be proposed in conjunction with a recommendation;
should be based either on IAEA Safety Requirements or Safety Guides, with the basis documented;
formulated in such a way that they are succinct and self-explanatory;
clearly specify the responsible party;
use “should consider”-language.
Refreshing IRRS Training - Evaluation of Observations

11. Good Practice (1)
A good practice is identified in recognition of an outstanding organization, arrangement, programme or performance superior to those generally observed elsewhere
A good practice
goes beyond the fulfilment of current requirements or expectations;
is worthy of the attention of other regulatory bodies as a model
might be applied elsewhere;
shall reference a basis in IAEA Safety Requirements or Safety Guides;
shall be clearly documented in the mission report.
Refreshing IRRS Training - Evaluation of Observations

12. Thus a good practice should have the following characteristics:
Has proven benefit;
Be replicable and applicable elsewhere;
Be effective and creative;
Produce good results;
Does not contradicts a finding;
Be novel or unique.
Refreshing IRRS Training - Evaluation of Observations

13. Formulation of observations
Refreshing IRRS Training - Evaluation of Observations

14. Format of an Observation
A description of what the reviewer observed, that has lead to the present finding or good practice
Quotation from the relevant IAEA Safety Standard that is to be / has been followed.
(There may be more than one Bases)
Recommendation / suggestion / good practice offered by the reviewer
Basic IRRS Training - Evaluation of Observations

15. Recording Observations
During interviews and direct observations, reviewers should make detailed notes of all relevant information gained.
On the techniques and content of taking notes see Training Module No. 5 (Methodology of the review)
Note here too that the number of observations in an IRRS mission does not represent a measure of the current performance of the reviewed RB relative to any other RB.
Refreshing IRRS Training - Evaluation of Observations

16. Formulation of Observations
The reviewer is expected to formulate in writing his/her observations daily
Observations of the day need to be presented on the Daily Team Meeting.
Be aware that the first set of findings (R+S) needs to be ready as early as on the third day of the mission!! (For more details see Training Module No. 8 on Scheduling IRRS Missions)
Good practices may be formulated later
Refreshing IRRS Training - Evaluation of Observations

17. Finalization of an Observation
Each observation is discussed and agreed upon by the team in Daily Team Meetings
Agreed observations are finalized and inserted into the mission report by the reviewer who formulated them
Observations agreed by the team should be discussed by the reviewers with the host counterparts for ensuring factual correctness of the observation
No agreement on the findings with the host counterpart is needed or sought!
Refreshing IRRS Training - Evaluation of Observations

18. Use of Action Plans
Hosts provide Action Plan(s) based on their Self-Assessment
Actions in the AP do not imply that they may not be IRRS findings (think of the follow-up m.)
Please consult the AP’s for input!
Refreshing IRRS Training - Evaluation of Observations

19. Experts say
Refreshing IRRS Training - Evaluation of Observations

20. Recommendation of Experienced Reviewers Give particular attention to:
R,S & GP should be adequately justified in the text The text should be based on fact and evidence (not assumptions) and the conclusions justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive explanation but should include some examples.
The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community
Some of the observations may fit in principle with the concept of Suggestions but are considered by the team to be beneath the threshold for inclusion as formal Suggestions – these should be considered for inclusion as observations in the main text only
In the formulation of the mission findings the team leaders should consider the application of the principle of graded approach
Refreshing IRRS Training - Evaluation of Observations

21. Experience form past missions
Refreshing IRRS Training - Evaluation of Observations

22. Collections of Observations
Refreshing IRRS Training - Evaluation of Observations

23. Statistics of Observations (1)
All missions
Without outliers
Follow-up missions
Refreshing IRRS Training - Evaluation of Observations

24. Statistics of Observations (2)
Follow-ups
R + S is negatively correlated with GP
In most missions the no. of findings is in the range of
Refreshing IRRS Training - Evaluation of Observations

25. Statistics of Observations (3)
The number of recommendations and that of suggestions are strongly positively correlated
The number of recommendations and that of good practices are strongly negatively correlated
Refreshing IRRS Training - Evaluation of Observations

26. Performance Indicators (1)
Balance of findings:
Many Rs implies few GPs and vice versa
 NR*NGP  const
The No of Rs and that of Ss are nearly proportional
 NR/NS  const
Refreshing IRRS Training - Evaluation of Observations

27. Performance Indicators (2)
Action Plan Coverage:
No. of findings also in the AP
No. of findings not in the AP
Refreshing IRRS Training - Evaluation of Observations

28. Warning Note once more:
The number of observations have certain definite statistical properties,
as such their relative values may have orientating merits within a given mission,
yet their absolute values should not be used for any comparison of separate missions.
Refreshing IRRS Training - Evaluation of Observations

29. Summary (1) Three types of observations have been defined:
Recommendation
Suggestion
Good Practice
Observations need to refer to IAEA Safety Standards (Requirements or Guides)
The various observations have specific features, yet all need to be based on factual information and not on assumptions
Inputs: ARM (including SA), Action Plans, interviews, direct obervations
Refreshing IRRS Training - Evaluation of Observations

30. Summary (2)
Reviewers have to take notes and formulate their observations daily
The first set of observations is expected to be ready at the end of the third day of the mission
Observations are inserted into the mission report in a standard table format
The number of recommendations, suggestions and good practices follow certain correlated tendencies. Yet comparison of such numbers between IRRS missions to various countries should not be attempted
Refreshing IRRS Training - Evaluation of Observations

31. THANK YOU FOR YOUR ATTENTION
This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission
Photo: V. Friedrich
Refreshing IRRS Training - Evaluation of Observations

32. Remarkable Performance*
Remarkable performance is identified in recognition of an organization, arrangement, programme or performance that – although is not unique and thus does not qualify for a good practice – is substantially superior to the general international practice
Remarkable performance
is meant to appreciate high level performance or practice of the RB;
in all other attributes is similar to good practices.
* Under consideration, not yet official
Refreshing IRRS Training - Evaluation of Observations