Presentation on theme: "Research Grade Evaluation (RGE) and Equipment Development Grade Evaluation (EDGE) Panel Member Training August 2014."— Presentation transcript:
1 Research Grade Evaluation (RGE) and Equipment Development Grade Evaluation (EDGE) Panel Member TrainingAugust 2014
2 ResourcesOffice of Science Quality and Integrity intranet site,RDSR portal (http://internal.usgs.gov/quality_integrity/rge_edge/RDSRportal.htm) - used to submit and access RDSR packages.Office of Personnel Management’s (OPM’s) Research Grade Evaluation (RGE) and Equipment Development Grade Evaluation (EDGE) guidesUSGS Research and Development Evaluation Process Handbook - how USGS implements the OPM guidesRGE, EDGE, and ST ChecklistsTraining materialsRGE/EDGE coordinators:John Thompson and Judy Back,
3 First-level RGE and EDGE panels Rate the research/development contributions and impact of an employee using the RGEG/EDGEG and USGS checklists.Evaluate the R&D and related work in terms of scientific quality and impactAddress career progress, balance of scientific activities, relevance of work to USGS goals, & content and quality of RDSR.Identify significant work that is not R&D in nature. ID approximate % of time and impact on the position. Positions where R&D is less than 25% should not be rated by the panel.Make final recommendations for all retentions in grade, conversions to RGE/EDGE, and promotions through GS-13.Provide promotion recommendations and justifications to GS-14/15 and ST for the second-level panelReview operational employees who request conversion to RGE/EDGE positionProvide constructive feedback to employeesAll evaluations, discussions, and documentation must be kept confidential
4 Research versus Development Research (RGE)Development (EDGE part 3)PurposeExtending knowledge andunderstandingNew or improvedproducts, processes, andtechniquesAssignmentsProblems to be solved:* entail relative freedom to explore promising areas in relation to organizational programs;* may stem from an intent to close gaps in knowledge in a given field, or develop new theories or explanations of phenomena; and* are difficult to define in terms of expected outcomes and measurable results.* are defined in advance or assigned;* may stem from an intent to exploit an understanding of phenomena and principles; or* have predictable outcomes or measurable results.ResultsProducts are:* papers describing new and modified theories and principles;* explanations of phenomena; and* information to improve the understanding of techniques and processes.* papers describing application of theories and principles;* design concepts, models, patents, and inventions; and* equipment, techniques, and processes.4
5 Four RGE/EDGE factors RGE EDGE Factor 1: Research Assignment Nature of the Assignment(scope, complexity, importance) (scope and complexity, objectives, means available, expected results)Factor 2: Supervisory Controls Supervision ReceivedFactor 3: Guidelines and Originality Guidelines and OriginalityFactor 4: Contributions, Impact, and Qualifications and ContributionsStature (demonstrated ability, products,advisory activities)When scoring a package, each of the four factors is assigned a level, defined by OPM, with corresponding points. The sum of the scores you assign for the four factors determines the grade in which an employee is working.
6 Core R&D Work Peer Reviewed Research and Development activities to considerCore R&D Work Peer ReviewedTechnical AssistanceProfessional Society ServicesEducation and OutreachStrategic PlanningResearch Leadership and DirectionSocietal RelevanceScience Impact
7 1st-level panel composition Chair – Scientist who has served on previous panels.Panel members – Panel members are research/development peers. The panel chair assigns members to be primary reviewers for specific scientists.RGE/EDGE coordinator – Guides panel process and takes notes during the deliberations.HR specialist – Attends panel as non-voting member to advise on procedural and interpretive issues.
8 Roles and responsibilities – panel members Be prepared before the panel meets:Thoroughly review the appropriate OPM RGE or EDGE (part 3) guide and the USGS RGE/EDGE/ST Factor Checklists. They define the key elements in each factor and aid in evaluation of each factor.Thoroughly review the RDSR package for each employee.Review and consider career experience, significant contributions, and bibliographyRead all publications listed under each scientist’s significant contributions (section 19)Carefully consider evidence of impact and stature of the scientistScore each package (for each of the four factors).Use the package and the appropriate guide and checklist (RGE or EDGE) to determine a degree level and corresponding score for each the four factors. If you think a GS-15 employee exceeds the maximum scores, consult the ST checklist.If an individual package does not fully meet a particular degree level, the next lower degree level must be assigned.Take notes that indicate the rationale for each factor score. Record initial scores, notes, and questions or discuss during the meeting on the Panel Member Evaluation Score Form (exhibit A-4 for RGE and B-4 for EDGE).
9 Roles and responsibilities – panel members (cont’d.) During the panel:Be prepared, objective, and actively participate!Confidentiality – nothing leaves the panelAll documentation, files, and notes are returned/destroyed after panel deliberationsDelete all digital files from your hard drives; destroy CDs/DVDs that contain panel informationAfter the panel:No outside discussions about panel deliberations or findingsDo not disclose identity of panel members
10 Roles and responsibilities – primary reviewer Prior to the meeting:As with all other employees being reviewed, thoroughly review the RDSR and PD, read the significant publications and score the individual.Supplement the package with additional information – As a minimum, consult the supervisor; then, consult as many references as necessary to understand the body of work. You may use, but are not limited to, the references provided at the end of the RDSR. You may not consult with the individual being reviewed. If you run into problems, contact the RGE/EDGE coordinator.Make certain you will be able to address each of the four factors and clarify scientist’s role on scientific teams/co-authored publications if it is not clearly indicated in RDSR.For each scientist you will present: Provide a digital primary reviewer write up for each of the four factors to the RGE/EDGE coordinator before the panel begins. The panel will use this to help craft the feedback, so do not quote from scientist’s PD, RDSR, or written responses from references.All communications with references must be confidential.
11 Roles and responsibilities – primary reviewer (cont’d.) Areas that may need clarification:Does the scientist generate ideas/make interpretations independently?What is the employee's role in cooperative efforts? Lead role or team member role? What were the scientist's contributions?To what extent are existing technologies/methodologies applicable to the work? Has the employee moved science/technology forward?What was the scientist's role in listed significant contributions, significance of results, and impact? Do the individual and (or) collective team contributions move the science/technology forward? Advance USGS strategic goals? Have societal impact?
12 Roles and responsibilities – primary reviewer (cont’d.) Areas that may need clarification (cont’d):How widely is the employee recognized and consulted in their field? Employee’s reputation among scientific and technical peers? How is leadership manifested in the employee’s record?Have recent products been assessed by scientific/technical peers? Has sufficient time lapsed to assess the impact and relevance of findings?Does the employee perform a full range of activities in the research/development cycle (from conceiving ideas through writing interpretive reports for technical and non-technical audiences)? Communicate with non-science groups or policymakers regarding nature, impact, and application of results to societal problems?
13 Roles and responsibilities – primary reviewer (cont’d.) During the meeting:Present information about the assigned employee’s career, current assignment, scope of responsibility, contributions, and impact. Clarify written materials for other panel members and provide information that is lacking in the written material but is needed to make a fair evaluation.Do not advocate. Be objective in presenting and clarifying information for the panel.
14 Roles and responsibilities – panel chair Before the panel meets:Assign a primary reviewer for each scientist being evaluated. May consult with Science Center Directors, National Capabilities Mangers, RDs, ADs, RGE/EDGE coordinator and other appropriate personnel on these assignments. Remind primary reviewers of their responsibilities. Strongly encourage panel members to attend training.Emphasize to all panel members the importance of conducting a thorough and objective review and rating of each scientist before the panel meets and the importance of maintaining confidentiality. Panel members should not rely on the primary reviewer as a primary source of information.Ask primary reviewer to provide a descriptive write up for each of the four factors to the RGE/EDGE coordinator before the panel begins.
15 Roles and responsibilities – panel chair (cont’d.) At the start of the panel – provide guidelines and emphasize:Importance of objective, unbiased ratings using RGEG or EDGEGThe importance of providing constructive feedback to the employeeImportance of evaluating the quality and impact of publication and non-publication contributions when scoring factor 4How to evaluate contributions that are part of a scientific teamImportance of identifying when RGEG or EDGEG is not appropriate for evaluation. Note: research/development must be at least 25% of efforts to be rated by the panel
16 Roles and responsibilities – panel chair (cont’d.) During the panel:Ensure that each panel member’s scores are posted for each of the factors before beginning discussion of employee.Ensure primary reviewer presents information to the panel for discussion. The panel is free to consult with anyone (except for the scientist being reviewed) who has additional information relevant to a case, in person or by phone during the panel.Lead and manage discussion. Ensure that all panel members participate and that discussion stays on track.
17 Roles and responsibilities – panel chair (cont’d.) During the panel (cont’d.):Discuss all four factors to provide documentation for each consensus score.Ensure that differences among scores are thoroughly discussed (especially >2 points for factors 1-3 and >4 points for factor 4).Work towards getting consensus scores. Ensure that the final consensus score for each factor is justified in writing and that the justification correlates with the criteria for that score under the RGEG/EDGEG.
18 Roles and responsibilities – panel chair (cont’d.) At the conclusion of the panel:Ensure that the final summary comments on each Individual Panel Results form is approved by all panel members.Complete and sign the Panel Recommendation Summary form.Send final documentation to the appropriate RGE/EDGE coordinator, John Thompson or Judy Back
19 Roles and responsibilities – RGE/EDGE coordinator Before the panel meets:Ensure that all panel members have access to RDSR packages at least two weeks before the panel meets.Provide training. Strongly encourage panel members to attend training.Work with panel chair to determine date and location of panel.Start of panel – provide guidelines and emphasize:Importance of confidentiality of materials and discussions – nothing is to be shared outside of the panel deliberationsThe process that will be used to discuss and evaluate each employee and the importance of reaching consensusIt is never appropriate to discuss personal/performance issues about an employee during peer-review panel discussions
20 Roles and responsibilities – RGE/EDGE coordinator (cont’d.) At the conclusion of the panel:Re-emphasize the importance of confidentiality. Panel members must not disclose any information/panel deliberations. It is line management’s responsibility to provide feedback to the employee.Send panel results (final Individual Panel Results forms for scientists not going forward to the second-level panel) to the Science Center Director/National Capability Manager.Reiterate that all digital and non-digital documentation (RDSRs, notes, etc.) from all panel members, other than the final copies of the Individual Panel Results and Panel Recommendation Summary forms must be deleted or shredded.
21 Special topics: Role of the position description The current position description (PD), including a copy of the signed coversheet, is used by the panel to clarify the official work assignment (factor 1).If a PD is inaccurate, the employee is responsible for drafting a statement that outlines the current work assigned by management and obtaining the supervisor’s signature to show concurrence. This should be part of the RDSR package.Panel members should note discrepancies between the official PD of record and the assignment (section 11) and accomplishment (section 12) portions of the RDSR. Where appropriate, panel results should recommend that the PD be updated and should note the impact of discrepancies on the scientist’s career.
22 Special topics: Mixed positions Panels only evaluate the research/development portion of the position, but need to understand the scope of the whole position to give proper weight to the recency of the work.Panels may have difficulty distinguishing between research/development and operational work. If necessary,Discuss questions regarding the nature of the work with the supervisor.Obtain position classification advice from the servicing personnel office.
23 Special topics: Scientific Teams When assessing Factor 1, consider the individual’s responsibility for research priorities, guidance, and productivity and credit the assignment in the following manner:Assessment of Team Leaders/Project Chiefs should reflect the scope and character of the entire project being conducted by the team, provided the scientist actively guides the research being done by the entire project. If the Team Leader/Project Chief does not actively guide the research of the project, his/her role should be assessed in the same manner as a team member.Assessment of a team member’s assignment should reflect the specific project, nature, scope, and difficulty of the assignment conducted by the individual and the contribution of that work to the impact of the larger project.
24 Special topics: Assessments and Monitoring Assessments and monitoring are mission critical for the USGS and can include a mix of operational as well as R&D components.An employee’s R&D role in an assessment or monitoring project depends on the performance (originality, creativity, etc) of the person in the job. If a scientist is spending at least 25% of his/her time and effort conducting R&D, it is appropriate for the scientist to be evaluated by a peer-review panel.Examples of assessment/monitoring-related research activities:Development of new assessment/monitoring protocols and toolsDevelopment of innovative decision support system toolsUse of assessment/monitoring data to develop/test hypothesesDevelopment of new predictive modelsInnovative integration of interdisciplinary assessment /monitoring resultsDiscovery of previously unknown processes or principles
25 Special topics: Technology Transfer Synthesizes and translates scientific findings and/or results into a more usable format for decision makers and adds value to the science through additional interpretation, synthesis, modeling, or translation.Should be directly related to and derived from official peer-reviewed R&D accomplishments and considered as evidence of the employee’s stature in the field.Panel members should consider:the impact of these activities on R&D in the field of expertise;contribution to the resource and the customer; andmeasurement of quality and quantity.Examples of science application and transfer products:Web-delivered analytical toolsModelsMaps and GIS productsDecision Support ToolsReports (USGS series and other) to actions agencies (e.g., BLM, FWS, EPA and their state and local counterparts, NGOs and the private sector)
26 Conversion scales for RGE and EDGE GradeRGEEDGEGS-118-148-11GS-1216-2413-16GS-1326-3418-21GS-1436-4423-26GS-1546-5028-31ST>56>3326
27 Where to find the information in the RDSR RGE 1. Research assignmenta) assignmentSections 11-12, 20b) complexityc) importance of resultsSections 11-12, 192. Supervisory controlsSections 11-13, 203. Guidelinesa) guidelines availableb) originality requiredSections 11-12, 18-204. Contributions, impact and staturea) demonstrated abilitySections 9-10, 13-14, 17-19b) products/publicationsSections 15-16, 18-19c) advisory/consult/scientific activitiesSections 13-15, 1727
28 Where to find the information in the RDSR EDGE 1. Nature of assignmenta) scopeSections 11-12, 20b) complexityc) importance of resultsSections 11-12, 192. Supervision receivedSections 11-13, 203. Guidelines and originalitya) guidelines availableb) originality requiredSections 11-12, 18-20c) demonstrated originalitySections 12, 16, 18-204. Qualifications and Scientific Contributiona) Contributions of new or improved products, processes, and techniquesSections 12, 15, 19b) products/publications(papers; design concepts, criteria and data; laboratory fabrication techniques and processes; and patents and inventions)Sections 12, 15-16, 18-19c) advisory/consult/scientific committee activitiesSections 13-15, 1728
29 Final panel documentation – only two panel forms are preserved: (1) Individual Panel Results Form – for each scientist reviewed
30 Individual Panel Results Form Documents panel’s assessment in relation to four factors. Include final consensus score and justification (1-2 paragraphs/factor) for each factor.Provide helpful and constructive feedback to employee.Be accurate. Cite the scientist’s record and refer to the checklist. Substantiate panel findings.Do not quote from the scientist’s RDSR, position description, or others you have consulted for the review.Forwarded to the second-level panel when reviewing recommendations for promotions to GS-14 and above and below-grade findings.
31 Individual Panel Results Form, cont’d. Factor 1: AssignmentWhat research/development is the scientist currently doing and what is the scope of the work?How complex are the problems being addressed?What is the importance of results/expected results?Factor 2: SupervisionHow is the work assigned?How much guidance is given?What level of responsibility does the scientist have?How are the results reviewed?
32 Individual Panel Results Form, cont’d. Factor 3: Guidelines and OriginalityWhat guidelines (literature, procedures, instructions, precedents) are available for the current work?What is the level of originality that the scientist brings to the work?Factor 4: Contributions, Impact, and Stature (RGE)Qualifications and Contributions (EDGE)What are the scientists publication and non-publication contributions?What impact have the scientist’s contributions had? Include evidence.What is the stature of the scientist in his/her field? In the broader scientific community? Include evidence.
33 Final panel documentation – only two panel documents are preserved: (2) Panel Recommendation Summary Form – for each panel
34 Resources:Office of Science Quality and Integrity intranet site,RDSR portal (http://internal.usgs.gov/quality_integrity/rge_edge/RDSRportal.htm) - used to submit RDSR packages for review.Office of Personnel Management’s (OPM’s) Research Grade Evaluation (RGE) and Equipment Development Grade Evaluation (EDGE) guidesUSGS Research and Development Evaluation Process Handbook -how USGS implements the OPM guidesRGE, EDGE, and ST ChecklistsTraining materialsRGE/EDGE coordinators:Judy Back and John Thompson,