1. Technical Reviews Module Space Systems Engineering, version 1.0
SOURCE INFORMATION:
The material contained in this lecture was developed by Lisa Guerra of NASA’s Exploration Systems Mission Directorate while on assignment in the Department of Aerospace Engineering at the University of Texas at Austin. As part of a course entitled, Space Systems Engineering, the lecture was piloted at UT-Austin in Spring 2008.
The content that follows was also reviewed and edited by Dr. Paul Graf, Adjunct Professor at the University of Colorado at Boulder.

2. Module Purpose: Technical Reviews
To understand the purpose and value of conducting technical reviews.
To discuss the timing of technical reviews over the course of a project’s life cycle.
To consider the entrance criteria and success criteria for the standard project technical reviews.
To understand when a technical review is complete.

3. Definition of Technical Reviews
Technical reviews are scheduled within a project or program to communicate an approach, demonstrate an ability to meet requirements, or establish status.
Technical reviews often serve as control gates for management to make a go/no-go decision on a project.
Control gates involve formal examination of a project’s status in order to obtain approval to proceed.
Technical reviews may be held for subsystems too; depending on their size and complexity. Subsystem reviews precede their corresponding system review.
When do technical reviews occur?
Throughout the entire life cycle of a project, as shown on the following life cycle chart.
Green book definition:
Reviews - Generically, a review is defined as an in-depth assessment, by an independent team of discipline experts and managers, that the design (or concept) is realistic and attainable from a programmatic and technical sense.
Extra thoughts from Marshall SE Handbook:
Many of the technical reviews, in particular the PDR and CDR, may be conducted on the overall system or incrementally on the subsystems.
Incremental reviews are typically conducted on large programs where it is necessary or desirable to allow design of the system or its sub-elements to proceed in the most efficient manner or to allow initiation of long lead-time procurement or manufacturing. In those cases where incremental reviews are utilized, it is mandatory that summaries of the results be included in the overall, comprehensive reviews to assure that the incremental activity is compatible and satisfies project requirements.

4. Reviews in a NASA Project’s Life Cycle
Source: NASA NPR D
Note: focus of this module is on a subset of the critical reviews, more closely aligned with the robotic missions.
This module does not discuss the section called Project Life Cycle Gates and Major Events.

5. The Value of Technical Reviews
Technical reviews are key development milestones used to measure progress, assess project maturity and to infuse lessons from the past. They…
Provide confirmation of completed effort and readiness to commit additional resources for the next phase.
Encourage and establish project discipline with an integrated project team perspective.
Identify risks and review mitigation options.
Describe plans and priorities for the next phase.
Technical reviews give everyone (design team, non-advocate discipline experts, customer and consumer) the opportunity to agree on the current project baseline (requirements, interfaces, allocations, margins, schedules, risks, budgets, etc.).
Functional, resource and performance allocations and assumptions are described and confirmed.
System development constraints and interfaces are described and agreed to by both sides.
Risks and development problems are identified and mitigation options discussed.

6. More Values of Technical Reviews
A significant, and perhaps unexpected, value to the development team is the preparation for a review. Usually everyone is busy in their own domain - preparing for a review forces the team to describe what they have done and understand what others are doing.
While there is great value in the preparation, there is also unique value in the execution. One Air Force officer once took Dwight D. Eisenhower’s famous quote:
“Plans are nothing. Planning is everything.”
too seriously and on the day before it was scheduled cancelled the PDR of a $100 million space mission. After a minor revolt, the PDR was reinstated and the team benefited from the preparation and the review.
Periodic project reviews are held to demonstrate that the appropriate products, accomplishments and plans have been completed before proceeding to the next phase.
Appropriate products, accomplishments and plans are based on the lessons of hundreds of past projects (best practices). NASA, for example, describes standard entrance and success criteria for all standard milestone reviews.
For entrance and success criteria see NASA NPR Appendix G or The NASA Systems Enginering Handbook section 6.7.2

7. NASA’s Minimum Set of Technical Reviews
Mission Concept Review (MCR),
System Requirements Review (SRR) and/or Mission Definition Review (MDR),
Preliminary Design Review (PDR),
Critical Design Review (CDR),
System Integration Review (SIR),
Test Readiness Review (TRR),
Operational Readiness Review (ORR),
Flight Readiness Review (FRR),
Post-Launch Assessment Review (PLAR),
Critical Event Readiness Review (CERR), and
Decommissioning Review (DR).
While this may seem like a large number of technical reviews, each has its own focus and proven value.
In addition, since mission lifetimes are several years, typically there is 6 months or more between major reviews.
Source: NASA NPR Chapter 5. Section
The technical team shall execute the required FS&GS project technical reviews… as above.
Note that the * reviews discussed in this module are most directly applicable to the senior design class the students will take. Often the senior design class does a mock PDR at the end of the semester.

8. The Primary Questions of Each Review
Mission Concept Review - Does the proposed concept meet the mission need and objectives?
System Requirements Review and/or Mission Definition Review - Do the functional and performance requirements and the selected concept satisfy the mission?
Preliminary Design Review - Does the preliminary design meet all the system requirements within acceptable cost, schedule, and risk?
Critical Design Review - Is the system design mature enough to proceed with full-scale fabrication, assembly, integration and test?
System Integration Review - Are the system, facilities, personnel, plans and procedures ready for system integration?
Test Readiness Review - Is the project ready to commence with verification testing?
Operational Readiness Review - Are all systems hardware, software, personnel, and procedures in place to support operations?
Flight Readiness Review - Is the system ready for launch? Are the ground facilities and personnel ready to support launch?
Post-Launch Assessment Review - After launch and deployment, are the spacecraft systems ready to proceed with routine operations?
Critical Event Readiness Review - Is the project ready to execute the mission’s critical activities during flight operation?
Decommissioning Review - Is the system ready to be terminated?
Source: NASA NPR Chapter 5.

9. Pre-Phase A Reviews Mission Concept Review (MCR)
A validation that the mission has clearly established needs, objectives, and top-level functional/performance requirements, and that at least one way of conducting the proposed mission is realistic and attainable within existing or projected technology and ROM cost.
An internal Center review(s) of all Pre-Phase A activities and products should be conducted prior to forwarding the Pre-Phase A report to Headquarters. Technical, management, resources, and scientific personnel should conduct the review.
Peer Reviews
Used informally during Pre-Phase A and Phase A.
The peer group is composed of individuals selected from outside the project according to their expertise in the applicable disciplines.
Throughout Pre-Phase A and Phase A, peer reviews should informally check the evolving mission concept against objectives, requirements, and constraints.
Peer reviews help you take advantage of other engineering experience from colleagues who have worked on different missions. They can point out issues they confronted that may be similar for your mission concept.

10. Phase A Reviews System Requirements Review (SRR)
The primary focus of the SRR is to verify the realism of the functional and performance requirements, ensure their congruence with the mission and system configuration, and ensure the mission objectives can be satisfied.
The SRR encompasses all major participants (NASA and contractors), and is chaired by the Project Manager.
A product from the SRR is the project system specification that is formally baselined and placed under configuration management control.
Mission Definition Review (MDR)
A validation that the mission objectives can be satisfied, the partitioning of the functionality to each of the systems is adequate, the top-level performance requirements for each system have been defined, and the technology required to develop the systems and implement the mission is attainable.
The MDR is keyed to the end of Phase A and evaluates the mission definition, system design, operational concepts, schedule, and cost estimates.
Additional Notes from Source: NASA NPR Appendix G.
The SRR and/or MDR examines the functional and performance requirements defined for the system and the preliminary program or project plan and ensures that the requirements and the selected concept will satisfy the mission.
SRR and/or MDR is typically conducted during the concept development phase (Phase B) following completion of the concept studies phase (Phase A), following baselining of the Systems Engineering Management Plan (SEMP) and before the preliminary design phase, the Agency Pre-Non-Advocate Review (PNAR), and System Definition Review (SDR).
Entrance Criteria. Prior to the execution of the SRR and/or MDR the activities and products identified in Table G-2 should be completed and documentation provided to all participants prior to the review. Also, precursor reviews should be completed.
Success Criteria. The review board was able to conclude that the success criteria in Table G-2 was accomplished to complete the objectives of the SRR and/or MDR.

11. Example Entrance Criteria — System Requirements Review
1. Successful completion of the Mission Concept Review (MCR) and responses made to all MCR Requests for Actions (RFAs).
2. A preliminary SRR and/or MDR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the SRR and/or MDR.
3. The following technical products for hardware and software system elements are available to the cognizant participants prior to the review:
a. System Architecture.
b. System requirements document.
c. System software functionality description.
d. Updated concept of operations.
e. Updated mission requirements, if applicable.
f. Baselined SEMP.
g. Preliminary system requirements allocation to the next lower level system.
h. Updated cost estimate.
i. Technology Development Maturity Assessment Plan.
j. Preferred system solution definition including major trades and options.
k. Updated risk assessment and mitigations.
l. Updated cost and schedule data.
m. Logistics documentation (preliminary maintenance plan, etc.).
n. Preliminary human rating plan, if applicable.
o. Software Development Plan (SDP).
p. System safety and mission assurance plan.
q. Configuration management plan.
r. Project management plan.
s. Initial document tree.
t. Verification and validation approach.
u. Preliminary hazard analysis (PHA).
Source: NASA NPR Appendix G.

12. Example Success Criteria — System Requirements Review
The resulting overall concept is reasonable, feasible, complete, responsive to the mission requirements, and is consistent with system requirements and available resources (cost, schedule, mass power, etc.).
The project utilizes a sound process for the allocation and control of requirements throughout all levels, and a plan has been defined to complete the definition activity within schedule constraints.
Requirements definition is complete with respect to top level mission and science requirements, and interfaces with external entities and between major internal elements have been defined.
Requirements allocation and flow down of key driving requirements have been defined down to subsystems.
System and subsystem design approaches and operational concepts exist and are consistent with the requirements set.
The requirements, design approaches, and conceptual design will fulfill the mission needs within the estimated costs.
Preliminary approaches have been determined for how requirements will be verified and validated down to the subsystem level
Major risks have been identified, and viable mitigation strategies have been defined.
Source: NASA NPR Appendix G.

13. Constellation Program Office Schedule of SRRs
You are Here!
2007
PBS
March
April
May
5/23
Issue ID, Form Entry
3/26
5/10
Provide Issue list w/ Resolution Plan & Impacts
Rev. Issue Table, POC, & Sched.
3/29
Provide Status on Issue Resolution
(every Cx SE&I 8:30 telecon)
4/6
4/9
4/16
4/23
4/30
Develop PBS Presentation
5/14
CLV SRR
(send updates to Cx SE&I COB every other day in May)
3/26
Issue ID, Form Entry
5/10
Provide Issue list w/ Resolution Plan & Impacts
4/6
Rev. Issue Table, POC, & Sched.
Provide Status on Issue Resolution
(every Cx SE&I 8:30 telecon)
4/9
4/16
4/23
4/30
5/14
Develop PBS Presentation
CEV SRR
(send updates to Cx SE&I COB every other day in May)
Issue ID, Form Entry
4/9
5/10
Provide Issue list w/ Resolution Plan & Impacts
4/13
Rev. Issue Table, POC, & Sched.
5/14
Develop PBS Presentation
Provide Status on Issue Resolution
(every Cx SE&I 8:30 telecon)
4/16
4/23
4/30
MO SRR
(send updates to Cx SE&I COB every other day in May)
Example review schedule for the Constellation Program. Each project/system within the program conducts their individual SRR. Once each has successfully completed their SRR, the Program conducts their review, designated above as PBS (green triangle). The PBS is the Program Baseline System requirements review.
Abbreviations:
CLV - crew launch vehicle
CEV - crew exploration vehicle
MO - Mission Operations
GO - Ground Operations
EVA - extra vehicular activity (space suits)
5/10
Provide Issue list w/ Resolution Plan & Impacts
Issue ID, Form Entry
5/4
5/5
Rev. Issue Table, POC, & Sched.
5/14
Develop PBS Presentation
GO SRR
Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)
5/10
Provide Issue list w/ Resolution Plan & Impacts
Issue ID, Form Entry
5/8
5/9
Rev. Issue Table, POC, & Sched.
5/14
Develop PBS Presentation
EVA SRR
Provide Status on Issue Resolution
(send updates to Cx SE&I COB every other day in May)

14. Constellation Program Office Status - Closure of SRR
RIDs
Total RID Count: 6,283
6225 Closed
58 Open
Majority closed prior to PBS
All RIDs should be closed prior to CxP SDR
Board AI
Total AI Count: 48 Total
9 Closed
39 Open
30 are “Past Due”
9 are in work, “Not Due Yet”
TBD/TBR
Total Count: 2,532
Received plans for 2,064 TBRs/TBDs in 72 of 95 documents (76% of docs.)
Need burn-down plans for 196 TBRs/TBDs in 16 documents (17% of docs.)
7 documents (274 TBRs/TBDs) will not be updated/baselined until post CxP PBS. These TBRs/TBDs will not be closed prior to CxP PBS
AI - Action Item
CxP - Constellation Program
PBS - Program Baseline Synchronization
RID = Review Item Discrepancies (RIDs)
The conduct of a major review is not complete until all resulting RIDs and action items are thoroughly and accurately closed out and their effect on the project properly
implemented. Although there may be a climate after such a review of relief or let down, the follow-up work must be pursued aggressively. This effort will help assure that the results of the review are expeditiously reflected in the project and will also serve as a solid basis for the next review.
TBD/TBR = To Be Determined / To Be Resolved

15. Preliminary Design Review (PDR)
The PDR is not a single review but a number of reviews starting with the specific component PDRs, followed by the system-level review.
The PDR establishes the “design-to” baseline and ensures that it meets the program, project , system, subsystem, or specific component baseline requirements.
The PDR process should:
Establish the ability of the selected design approach to meet the technical requirements (i.e., Verifiability/ Traceability);
Establish the compatibility of the interface relationships of the specific end item with other interfacing items;
Establish producibility of the selected design;
Establish the operability of the selected design;
Assess compliance with reliability and system safety requirements;
Establish the feasibility of the approach;
Address status, schedule and cost relationships.
Goddard Green book definition (SDR used instead PDR)
System(s) Design Review
The objective of the SDR is to demonstrate that an acceptable system configuration has been defined and the requirement allocations are complete for all portions of the mission. The primary focus of the SDR is to show that a system can be built which will satisfy the mission objectives. More than one review may be necessary if more than one system is required to conduct the mission. If necessary, the system(s) design reviews are summarized and then evaluated from a total mission standpoint.
From Marshall Handbook:
The PDR is conducted when the basic design approach has been selected and the necessary documentation is available (usually when design maturity is approximately 50% and drawing release is 10% complete). PDRs may be conducted at the program level and/or the project level. This is a technical review of the basic design to assure compliance with project requirements.

16. When is a Review Complete?
Reviews are considered complete when the following is accomplished:
Agreement exists for the disposition of all Review Item Discrepancies (RID) and Request for Actions (RFA).
The review board report and minutes are complete and distributed.
Agreement exists on a plan to address the issues and concerns in the review board’s report.
Agreement exists on a plan for addressing the actions identified out of the review.
Liens against the review results are closed, or an adequate and timely plan exists for their closure.
Differences of opinion between the project under review and the review board(s) have been resolved, or a timely plan exists to resolve the issues.
A report is given by the review board chairperson to the appropriate management and governing program management committees charged with oversight of the project.
Appropriate procedures and controls are instituted to ensure that all actions from reviews are followed and verified through implementation to closure.
Source: NASA NPR Chapter 5.

17. Example Agenda for a Project System Design Review by a Standing Review Board (SRB)
Purpose of Review & Charge to SRB SRB Chair
Project Overview & Status Project Manager
System Engineering & Status Project SE
Requirements & V&V plans
Trade studies
Technical margins
WBS-level 2 Design State & Status for each area WBS Managers
System Design
Key Requirements
Trade Studies
Technology Readiness
Acquisition Strategy & Long Lead
Logistics & Facilities
Challenges & Risks
Integrated System (e.g., power) State & Status for each area Discipline Leads
Integration & Test Integration Manager
Safety & Mission Assurance (S&MA) S&MA Manager
Human Rating Project HR Rep
Risk Risk Manager
Schedule Project Planner
Cost Cost Manager
Source: NASA’s Independent Program Assessment Office (IPAO); Mark Saunders
This sample agenda was generated for the projects under the Constellation Program Office, such as the Orion crew vehicle.
A report on human rating is unique to this program which is responsible for the development of spacecraft and equipment (such as space suits) for astronauts.

18. Pause and Learn Opportunity
Student role-play:
You are the chief systems engineer for a New Frontiers-class mission to Europa. Your PDR is scheduled in 3 months.
What do you do?
What do you ask the development team to do?
What benefits would you expect from having a PDR?
How might it waste time? Or
Read and discuss the Crosslink article: The Role of Independent Assessments for Mission Readiness
(Crosslink_Independent Review.pdf)

19. Module Summary: Technical Reviews
Technical reviews are key development milestones used to measure progress, assess project maturity and to infuse lessons from the past. They…
Provide confirmation of completed effort and readiness to commit additional resources for the next phase.
Encourage and establish project discipline with an integrated project team perspective.
Identify risks and review mitigation options.
Describe plans and priorities for the next phase.
There are 11 reviews in the minimum set of technical reviews for a NASA robotic mission. These reviews cover the entire mission life — assessing the concepts and designs early; readiness for test, flight and operations in mid-life and plans for disposal at the mission’s end.
These reviews are held to demonstrate that the appropriate products, accomplishments and plans have been completed before proceeding to the next phase.
Appropriate products, accomplishments and plans are based on the lessons of hundreds of past projects (best practices).

20. Backup Slides for Technical Review Module

21. Major Project Reviews Precede Each Key Decision Point
FORMULATION
IMPLEMENTATION
Pre-A A B C D E F
Project
Phases
Concept
Studies
Concept &
Technology
Development
Preliminary
Design &
Technology Completion
Final
Design &
Fabrication
System Assembly,
Test, & Launch
Operations &
Sustainment
Closeout
Key Decision Points A B C D E F
Mission Concept Review
Systems Requirements Review
Major Reviews
Mission/System Definition Review
Preliminary Design Review
Key Definitions
Formulation: The first part of the NASA management life cycle where system requirements are baselined, feasible concepts are determined, a system definition is baselined for the selected concept(s), and preparation is made for progressing to the Implementation Phase.
Implementation: The part of the NASA management life cycle the detailed design of system products is completed and the products to be deployed are fabricated, assembled, integrated and tested; and the products are deployed to their customers or users for their assigned use or mission.
Critical Design Review
Independent Cost
Estimates
Systems Integration Review
Operational Readiness Review
Flight Readiness Review
Post Launch Assessment Review
Decommissioning
Review

22. Purpose of Technical Reviews
A technical review is an evaluation of the project, or element thereof, by a knowledgeable group for the purposes of:
Assessing the status of and progress toward accomplishing the planned activities.
Validating the technical tradeoffs explored and design solutions proposed.
Identifying technical weaknesses or marginal design and potential problems (risks) and recommending improvements and corrective actions.
Making judgments on the activities’ readiness for the follow-on events, including additional future evaluation milestones to improve the likelihood of a successful outcome.
Making assessments and recommendations to the project team, Center, and Agency management.
Providing a historical record that can be referenced of decisions that were made during these formal reviews.
Assessing the technical risk status and current risk profile.
Source: NASA’s NPR A

23. Need for Management Reviews
The progress between life-cycle phases is marked by key decision points (KDPs). At each KDP, management examines the maturity of the technical aspects of the project. For example, management examines whether the resources (staffing and funding) are sufficient for the planned technical effort, whether the technical maturity has evolved, what the technical and non-technical internal issues and risks are, or whether the stakeholder expectations have changed. If the technical and management aspects of the project are satisfactory, including the implementation of corrective actions, then the project can be approved to proceed to the next phase.
Source: NASA’s NPR A

24. Blue text indicates changes from last update
This is a Constellation Program Office schedule regarding completion of serial SRRs required to be complete prior to setting the Program Baseline Synchronization (PBS). It is from this point that the mission/system baseline goes under configuration control.
Blue text indicates changes from last update