1. JTAMS POST-CDR IT/SIS ISSUES
PM JTAMS
Lesson Point of Contact:
Name: Mike Thorne
Length of Presentation:
Presentation: .5 hours
Exercise: 3 hours
Lesson 20 – Practicum#4

2. Scope of Presentation Practicum Sections
Section 1: Software Quality Analysis
Section 2: Cyber Controls Analysis
Section 3: Cloud Business Case Analysis Section

3. Software Quality Analysis
Task 1: Agile Measures Analysis
Task 2: JRATS/JTAMS Measures Analysis
Task 3: Software Management
Task 4: Agile/Lean Assessment
Task 1: Agile Measures Analysis
Overview: It is approximately 3 months after CDR (May FY3). AMRU has continued STIM development and has produced measures and reports highlighting their progress as well as working software.
Objective: Evaluate information and present an overview of the current status of STIM development (see Student Template)
Task 2: JRATS/JTAMS Measures Analysis
Overview: It is approximately 6 months after CDR. Your team in the JTAMS program office has received a set of slides from Juggernaut depicting the current status of the overall JTAMS development.
Objective: Evaluate information and provide recommended alternatives.
Assess the updated measures delivered by Juggernaut
Recommend alternatives for the PM
Task 3: Software Management
You will find data in the Job Aids relating to an emerging software management issue. The PM wants a summary of this issue and any insights your team has regarding the issue (see the Software Error Report included below and other artifacts in the Job Aids).
Are there procedures to ensure that management information is transmitted throughout the software organization, and does the plan call for integrated information across all subcontracts as well as prime contractor developed software?
Have plans been developed for resolution of actual and potential problems?
Task 4: Agile/Lean Assessment
The PM has indicated he would like an assessment of any aspect of the program that is using Agile software development.
The PM understands that there are some Lean concepts that should be implemented as part of any Agile development and he wants to ensure that increment 2 of JTAMS fully leverages Lean concepts as part of any Agile development approach. For the lean-Agile assessment you will find an article the PEO provided.
You will assess AMRU’s current Agile development approach and determine which aspects are in keeping with a lean approach. You will also recommend areas where AMRU might benefit from investigating additional Lean concepts.
Section 1

4. AMRU S/W Release Schedule
Task 1: Agile Measures Analysis
Present an overview of the schedule and identify any updates to the overall JRATS/JTAMS scenario
Joint Training and Maintenance System (JTAMS) Increment 1 Government Roadmap Schedule Showing STIM (in green)
TODAY is 30 May FY3
FY 00
FY 01
FY 02
FY 03
FY 04
FY 05
FY 07
TECH DEV
PROD & DEPLOYMENT
Milestones
& Phases
MS A
MS B
IOC
Final RFP
Release
Contract
Award
Technical
Reviews
CDR
Development
Testing
IOT&E
LFT&E
Deliveries
(Prototypes / EDMs)
Operational
FY 06
FRP
DECISION
EOA OA
Post CDR Assessment
EMD
Prototype
DT&E
EDMs
MDD
FY 08
MSA
MS C
Engineering and Manufactuing Demonstration
ITR
ASR
PDR
SSR
SRR TD
JTAMS-1 IOC
PRR
FCA
PCA
MRU Development
Block 1
Block 2
Block 3
Section 1
SEE: Practicum 4\Exercise 1 Post-CDR Agile Measures
These charts come from the Release 4 report in the Section folder. The intent here is for the students to identify that we are now past release 2. most of the rest of their briefing will be based on this report and the charts and s.
Instructor notes:
Most notable aspect of the schedule is that AMRU is producing software that can be used by the rest of the MRES team as they proceed with their implementations. This means that other teams can use working STIM elements as part of their development instead of relying on interpreting interface documents. This will improve the quality of MRES as a whole. It also helps to identify interface and possibly architecture problems as well, which could be one of the contributors (certainly not the only one) to MRES being behind schedule overall.
AMRU S/W Release Schedule 4

5. New interface spec released
Using the Cumulative Flow Diagram,
explain what happened in April and May FY03.
identify the work completed, work in progress and the work remaining on 03/01/03.
Explanation
The bulk of the team’s effort was spent re-planning and understanding and decomposing the new interface spec and very little was expended developing / testing existing tasks.
New interface spec released
Production falls
Work Remaining
In-Progress
Section 1
SEE: Practicum 4\Exercise 1 Post-CDR Agile Measures
Again, this chart comes from the release report. They should have briefed a version of this and explained the purpose of the chart in Practicum 3. Here we are pointing out that since we have now received the interface spec (a point missing from the risk slides), we see a large downturn in software developed and tested and a spike in the “ready” area as we learn how to develop the interface. We eventually catch up and continue an upward slope in development and testing.
Additional instructor notes:
Point out to students the difference between the overall rhythm of the Release 2 report (lots of variation in the bands) vs the early part of the Release 3 and 4 report. The Ready, Dev, and Test bands are much more consistent up until the time when the new interface spec comes into play. The CFD makes clear the effect of the extra definition work that has to be done on other activities, and at the very end you can see that the development pace is beginning to pick up agai. At this point, it is unclear if AMRU will be able to recover to achieve their release goals for the next block.
Instructor notes:
The answer correctly identifies that characterization of interface work as defects COULD be part of the defect story. One of the things that measurement visualizations provide is a new set of questions to ask when something anomalous shows up. The visualization (burn up chart) alone doesn’t provide the answer, typically.
Work Complete

6. Explain the defect closure measurement
Explain the defect closure measurement. Does it correlate with the other measurements? Why or why not?
The chart visually shows the rate at which defects are being opened (blue line), the rate at which defects are being closed (red line); Cumulative closed rate decreased starting in sprint 17
As was seen in the CFD, the development effort, which incorporates bug fixes, was put on hold while the new interface spec was added.
The Burn Up measurement shows an increase of 90 tasks added to the release effort.
Defects continued to be identified during this timeframe. It’s possible AMRU identified the interface work that needed to be redone as defects.
Section 1
SEE: Lesson 18 Practicum 4\Exercise 1 Post-CDR Agile Measures
In the release report, there are some “burn” charts for the students to review.
Instructor notes:
The answer correctly identifies that characterization of interface work as defects COULD be part of the defect story. One of the things that measurement visualizations provide is a new set of questions to ask when something anomalous shows up. The visualization (burn up chart) alone doesn’t provide the answer, typically.

7. Task 2: JRATS/JTAMS Measures Analysis
Summarize your analysis
Worksheet #1 – Part 1
Problems
Summarize any problems you identified, and identify root causes.
1. The program is seriously behind, and does not appear likely complete on time, within budget, and with acceptable quality. Multiple correlated measures validate this problem including earned value, code development, and the number of identified defects.
Root causes appear to be the late requirements changes, lagging definition of subsystem interfaces, the delay in staffing for the program, and quality issues.
Impacts
Describe the source and scope of the problem and assess the potential impacts on project success.
Current data indicates that the program will likely have a schedule that is late, will be over budget, and will not be at an acceptable level of quality.
- Earned value shows that the work completed to date has cost 30% more than planned. Only 63% of the work schedule to be completed by now is done.
- Only 80% of the planned staff hours to date have been expended.
- About 90% of the code planned to be complete by now is done. MRES has only completed about 75% of what it was expected to.
- The defect backlog is large, and growing. Large numbers of defects continue to be identified, in all components
- Requirements volatility is still occurring. New requirements continue to be identified, even though a significant part of the system has already been developed.
Outcome
Predict the outcome if no action is taken, and indicate how this prediction was derived.
If no action is taken, it is probable that the program performance parameters will need to be rebaselined.
Based on the current data, if no additional requirements are changed, the program will likely require at least 30% additional funds, and a significant addition to the schedule.
Section 2
SEE: Practicum 4\Exercise 2 Post-CDR JTAMS Measures Section
The intent is for the students to be able to summarize the key problem areas and summarize the problems/impacts/outcomes. Again, they may see things differently based on the areas they choose to focus on given the slides they are reviewing.

8. Justification Overview
Display the key indicator slides that justify your analysis on the previous slide:
Section 2
SEE: Practicum 4\Exercise 2 Post-CDR JTAMS Measures Section
This slide depicts some of the charts the students have been given to analyze. They should present the relevant charts and provide some analysis that supports what they have explained on the earlier charts. They can present this any way they see fit and do not necessarily have to use the slide templates.
Students may identify other slides to justify their conclusions

9. JTAMS MEASURES ASSESSMENT
Provide alternatives given your analysis
Worksheet #1 – Part 2 (Alternatives)
Alternatives
List the alternative courses of action you would consider for the program.
1. Do a program replan, and rationalize remaining work, schedule, and costs. Develop a realistic plan, based on progress to date and demonstrated capability. Freeze requirements and program changes until the initial delivery is complete.
2. Put together an expert technical team to prioritize remaining functionality. The team should have representatives from each system component. This team should assess the criticality of any undefined items, based on required functionality and cost/schedule impacts. Consider delaying implementation of any functionality that is less critical to a future release. Work with all stakeholders to produce a revised development and release schedule
3. Put in place a more rigorous control process for requirements additions and changes. Carefully evaluate any proposed requirements additions or changes, for impacts to cost, schedule, and quality. Defer non-critical additions and changes to a future release of the system.
(this was a recommendation from the last one, but it clearly did not happen)
Section 2
SEE: Lesson 18 Practicum 4\Exercise 2 Post-CDR JTAMS Measures Section
Finally, this is where the students are at liberty to present their recommended way ahead for the program office. You should be prepared to discuss their recommendations and note that some of these problems were identified in Practicum 3 and still have not been resolved – note that they may even have recommended similar actions in P3.

10. Task 3: Software Management
Are there procedures to ensure that management information is transmitted throughout the software organization, and does the plan call for integrated information across all subcontracts as well as prime contractor developed software?
Answer (Justify your answer): No. There are no procedures in effect for controlling parallel software development for the software development teams.
Development and testing takes place in 4 locations w/ little communication between them
Juggernaut has no plan documented
Suggested resolution:
Work with Juggernaut to improve communication among development teams.
Identify and track appropriate measures
Section 3
The students are provided with a “Software Error Report”. They also have error metrics “software error metrics” where they can identify variation. From these documents they can identify the answers to slides 40 and 41.
These teams have very little communication with each other and each has about a third of the effort. Each team is responsible for their own quality assurance with respect to meeting the requirements of their respective software requirements documents. Completed modules are sent to the system integration lab for subsystem and system level integration testing. Any problems with software modules are documented and returned to the originating activity for correction.
Students are provided with software error metrics for all software development and an individual breakdown on each team as well as earned value reports for each team. The overall software metric showing errors discovered and errors corrected shows a potential problem trend for the program in that new errors are being discovered at a rate that is greater than the errors are being corrected. This has the potential to cause a schedule overrun if there is a great deal of unplanned software corrections that must be made.
Note: software integrator is the one that has to get subordinate software developers communicating with each other

11. Answer (Justify your answer):
Have plans been developed for resolution of actual and potential problems?
Answer (Justify your answer):
Errors are building up faster than they are being corrected and there is no evidence of any process improvement for either teams 1 and 3.
Two of three software development teams (Teams 1 and 3) have special causes of variation in their software development processes. In addition, their EVs show negative cost and schedule variance.
Suggested resolution:
Work with Juggernaut to improve communication among development teams.
Migrate the practices of Team 2 that has repeatable processes to Teams 1 and 3.
Section 3
SEE: Section 3 CPI\JOB AIDS
The student is provided a normal distribution curve for six months of performance for each of the three teams. Team 2 has a low error rate with consistent errors over all six months. Teams 1 and 3 have higher overall error rates and a great deal of variation between months. The variation has no trend (i.e. teams 2 and 3 are not getting better or worse). The students are also provided Earned Value data for each team. Team 2 had negative variances over the first 3 months but has improved since then and the trend is that they will complete with positive cost and schedule variances. Teams 1 and 3 show the opposite trend starting with positive variances in the first three months and then declining variances. Teams 1 and 3 are on track to end with negative variances in both cost and schedule.
This task is to evaluate the performance variances. The intended response is that team 2 used the first three months to eliminate special causes of variation and establish a repeatable process. Teams 1 and 3 have not implemented repeatable processes and therefore have inconsistent performance with higher errors and variation. Both of these teams have failed to eliminate special causes of variation in their coding processes. As a result, team 2 EV variances continue to improve as they have very little rework of integration testing errors. Teams 1 and 3 both have negative EV variances as extra work piles up from the higher number of errors returned for corrections. The assignment asks the student to recommend corrective action in terms of both variation and best practices. For variation, the student should recommend that the contractor institute communications between the three teams. Team 2 has established repeatable processes that deliver quality software. Those practices should be migrated to teams 1 and 3.

12. Task 4: Agile/Lean Assessment
Agile/Lean Assessment: The team recommends the following areas where AMRU and/or Juggernaut might benefit from investigating additional Lean concepts.
Agile / Lean assessment recommendations:
Lean Concept
Description
Addressed in AMRU overview
Recommended for AMRU Sw Dev Plan
(Describe expected benefit)
Inventory in software development.
In knowledge work such as software development, we can think of inventory as being maintained in documentation and in non-deployed software. No
Establish a policy or procedures for documentation; minimize non-deployed SW
Motion in software development.
Motion in team members is manifested through activities like travel time, walking to meetings, task-switching between projects and work interruptions
Minimize task switching
Reduce meetings
Address reasons for work interruptions
Waiting in software development.
Examples include waiting for milestone and change request approvals and sign-offs, waiting for clarifications and elaborations from sponsors and analysts, waiting for builds and testing to complete, waiting for your turn in meetings and conference calls, waiting for deployment schedules and architectural and code reviews.
Yes
Adequately addressed in AMRU overview; recommend Juggernaut investigate options
Over-production in software development.
over-production in software development occurs when we build features that are either 1: never or rarely used or that are 2: deployed prematurely
Adequately addressed in AMRU overview
Over-processing in software development.
Over or redundant processing in software includes low and no value meetings and conference calls, duplicative approvals, redundant reporting, redundant specifications, over-engineering code and gold-plating specifications, design and code reviews that don’t result in technical improvements
Transport in software development.
waste of translating and handing-off customer requirements through subsequent phases such as functional specifications, UML diagrams, source code and tests
Yes/No
While not specifically addressed, the AMRU approach indicates they already adequately address this aspect of Lean
Defects in software development.
Testing and inspections that do not result in finding bugs are considered defect waste in software development. Other examples are features that were implemented but were never requested, inaccurate specifications, production bugs, and substandard user experience.
The testing process seems adequate – however, additional emphasis could be placed in this area
Section 3
SEE: CPI\JOB AIDS\Software Dev\when-you-are-agile-you-get-lean-solutionsiq-charlie-rudd-white-paper
The intent of this Section is to move the program (and the students) past CDR considerations and on to (or past) Milestone C. This Section brings in aspects of CPI (Lean, Six sigma, and TOC) and aspects of the software quality lesson. The intent is to have them integrate a variety of aspects, including software quality, into an overall assessment of the overall JRATS-JTAMS effort.
Note: compare/contrast the AMRU approach as described in their overview documents with the Lean/Agile article. The students will likely identify additional or other areas. The intent here is to provide an opportunity to discuss what AMRU is doing and the Lean concepts in the white paper.
Inventory in software development. In knowledge work such as software development, we can think of inventory as being maintained in documentation and in non-deployed software. Software work-in-process (WIP) is all activity (and therefore expense) subsequent to business case approval but preceding deployment to production. This includes requirements documents, project plans, functional and design specifications, source code, test plans, and tests plus the time to create these artifacts. If a project is a year or more in duration, WIP continues to build up over the entire period prior to production use. WIP is also tantamount to the accumulation of write-off risk for software investments because if the project ends prematurely or never is delivered, no business value is generated and the WIP investment is lost.
Motion in software development. Motion in team members is manifested through activities like travel time, walking to meetings, task-switching between projects and work interruptions. A 30 second work interruption can result in a five-minute think time reset for a developer.
Waiting in software development. Examples include waiting for milestone and change request approvals and sign-offs, waiting for clarifications and elaborations from sponsors and analysts, waiting for builds and testing to complete, waiting for your turn in meetings and conference calls, waiting for deployment schedules and architectural and code reviews. Any elapsed time in excess of what is needed to execute the added value step is a form of waiting. Looked at in that manner, even the most efficient operations still have a great deal of waiting in principle, leaving plenty of opportunity for future improvements.
Over-production in software development. Since the model for software use is that one unit (i.e., one release) is shared by many users (even more so in SAAS environments) and the physical material cost of a software unit is minimal, over-production is not expressed as producing units in excess of demand. Rather, over-production in software development occurs when we build features that are either 1: never or rarely used or that are 2: deployed prematurely.
Over-processing in software development. Over or redundant processing in software includes low and no value meetings and conference calls, duplicative approvals, redundant reporting, redundant specifications, over-engineering code and gold-plating specifications, design and code reviews that don’t result in technical improvements. Overly detailed work breakdown structures and overly precise estimations are also forms of processing waste.
Transport in software development. Since software comprises information electronically stored and accessed, the physically transport of materials or finished goods is of little concern. However, there is the analogous waste of translating and handing-off customer requirements through subsequent phases such as functional specifications, UML diagrams, source code and tests. In addition, there is also information loss (or the introduction of noise) that damages the information just as finished goods and materials are sometimes damaged through transportation.
Defects in software development. Testing and inspections that do not result in finding bugs are considered defect waste in software development. Other examples are features that were implemented but were never requested, inaccurate specifications, production bugs, and substandard user experience.

13. Section 2 Cybersecurity Controls
Critical Controls that Could Have Prevented Target Breach.
Case study file: “ISA201 SANS Target Case Study edited DAU Jun 2017.docx”
Complete the “Controls” templates by identifying the appropriate controls from NIST SP rev 4. Suggest 2-3 controls that would mitigate the threat if applied properly. Explain your choices.
Identify how the threat described in the case could be realized in JTAMS.
Explain whether or not applying the NIST controls would provide similar results for JTAMS?
Identify statements, language, or clauses concerning controls that should go into an RFP regarding the controls.
Section 4 Cybersecurity Controls

14. Controls – Slide 1
Attack Vector 1
Reconnaissance
Critical Controls 9 15 18
Security Skills Assessment and Appropriate Training to Fill Gaps
Controlled Access Based on the Need to Know
Uncover the malware and network traffic to mitigate losses
Notional Result w/ Critical Controls Applied
Result: Hackers would have not known vulnerabilities in advance of attack; losses could have been minimized/mitigated
* Suggested Controls-SP
 Controls
Audit and Accountability (AU-6),
Configuration Management (CM-6),
Contingency Planning (CP-2), CP-4,
(Incident Response (IR-2), IR-3, IR-8
Justification
Incident-related information can be obtained from a variety of sources including, for example, audit monitoring, network monitoring, physical access monitoring, user/administrator reports, and reported supply chain events.
Adequate, well- trained staff with time to appropriately analyze logs would have uncovered the malware and network traffic to mitigate losses
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:
An attack vector is a path or means by which a hacker can gain access to a computer or network server in order to deliver a payload or malicious outcome.

15. Control Slide 1 – Threat
While commercial threats and DoD threats are different, they often share similarities.
Identify how the threat described in the case could be realized in JTAMS.
Explain whether or not applying the NIST controls would provide similar results for JTAMS?
How could the threat described in the case be realized in a DoD program?
This threat occurs commonly in the DoD and is very serious. Major data breaches in the DoD are in the news regularly and we know that there are many passive threats by China and other countries to DoD assets that may be looking for reconnaissance data.
Would applying the NIST controls provide similar results?
It is difficult to quantify the results of security awareness programs, but we know from past experience that many breaches could have been thwarted by employees with greater security awareness. Access control procedures are extremely important in protecting confidentiality of government assets.
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:

16. Controls – Slide 2
Attack Vector 2
attack, Malware installed on vendor machine
Critical Controls 5 9
Malware Defenses
Security Skills Assessment and Appropriate Training to Fill Gaps
Notional Result w/ Critical Controls Applied
Result: Malware in phishing attack would have failed. Hackers would not have obtained access to vendor portal credentials.
Suggested Controls- SP
 Controls
AT-1, AT-2, AT-4 – Security Awareness and Training and Policy/Procedure, CA-2 – Security Assessment , CA-7 – Continuous Monitoring , CP-4 – Contingency Plan Testing, IR-3 – Incident Response Testing, SC-39 – Process Isolation, SI-3, SI-4 – Information System Monitoring , SA-11, SA-16 – Developer Security and Testing evaluation ,
Justification
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:
An attack vector is a path or means by which a hacker can gain access to a computer or network server in order to deliver a payload or malicious outcome.

17. Controls Slide 2 – Threat
Identify how the threat described in the case could be realized in JTAMS.
Explain whether or not applying the NIST controls would provide similar results for JTAMS?
While commercial threats and DoD threats are different, they often share similarities.
How could the threat described in the case be realized in a DoD program?
DoD gets containing malware on a daily basis. Good security programs and virus scanning can help prevent many, but not all, of these attacks. A good incident monitoring and reporting program can help make sure that if items “fall through the cracks” safeguards are immediately put in place. Penetration testing can also find where vulnerabilities lie before a threat is realized.
Would applying the NIST controls provide similar results?
Applying NIST control may be more effective than critical controls listed. Penetration testing allows you to proactively find and fix the vulnerability.
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:

18. Controls – Slide 3
Attack Vector 11
Data moved to drop locations
Critical Controls 17
Data Protection: Employ tools at perimeters to monitor for sensitive data leaving the company in clear text.
Notional Result w/ Critical Controls Applied
Result: Data moving out of the organization would have been spotted. Breach would have been stopped sooner.
Suggested Controls- SP
 Controls
AC-2 Account Management
AU-2 Audit Events
CM-7 Least Functionality
AC-4 Information Flow Enforcement
AC-17 Remote Access
Justification
Changing default passwords (ex. BMC’s Performance Assurance tool for Microsoft Servers)
Frequent password changes for all accounts should have been enforced
All unnecessary ports, protocols and services should be shutoff; NetBIOS has a lot of known vulnerabilities and probably wasn’t a necessary protocol.
File Transfer Protocol (FTP) should have been limited by firewall
The vendors’ remote access site should have been placed in a DMZ with a intrusion detection (or prevention) device placed between the DMZ and Target Intranet
Target should have established an intranet separating vital systems from the Internet.
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:
An attack vector is a path or means by which a hacker can gain access to a computer or network server in order to deliver a payload or malicious outcome.

19. Controls Slide 3 – Threat
While commercial threats and DoD threats are different, they often share similarities.
Identify how the threat described in the case could be realized in JTAMS.
Explain whether or not applying the NIST controls would provide similar results for JTAMS?
How could the threat described in the case be realized in a DoD program?
FTP runs on many DoD systems and can be a major issue. Continuous monitoring policy could keep this threat from occurring or at least provide greater visibility for where data is going. Please note that controls not listed here such as System and Information Integrity (SI) and Media Protection (MP) may also help in controlling this risk.
Would applying the NIST controls provide similar results?
Yes. SI-4(b) states Information System Monitoring “Identifies unauthorized use of the information system through [Assignment: organization-defined techniques and methods]:” In this case, organization-defined techniques should identify the improper use of FTP to transfer data.
SLIDE INFORMATION****************************************************************************************************************************
*Slide Type(Content or Exercise): Exercise
*ELO ID (Use a Comma to separate ELOs if more than one is addressed): ,
**************************************************************************************************************************************************
Key Points:
Overview Slide only
Key Questions to Ask and Anticipated Answers:
Q:? A:
Terms \ Definitions \ Acronyms:

20. Possible RFP Language Options
What statements, language, or clauses concerning controls should go into an RFP?
Contractor shall comply with DoD policies regarding the protection of data
Contractor shall provide layers of defense including preventative and detective countermeasures to mitigate risks
Contractor shall have a detailed understanding of networks, hardware, software and processes
Contractor shall provide a comprehensive approach for auditing to enable rapid updating of defense gathering and corrective actions
Contractor shall posses the KSAs needed to support defense of the enterprise
Contractor shall provide training and awareness programs

21. Possible RFP Language Options
What statements, language, or clauses concerning controls should go into an RFP?
Section L: Instructions, Conditions, and Notices to Offerors
[SecL001] The offeror, as part of its technical proposal, shall describe the use of its system security engineering (SSE) process in specifying and designing a system that is protected against external threats and against hardware and software vulnerabilities. As a part of describing this SSE process, the offeror shall describe the criticality analysis process used to determine Mission-Critical Functions and the protection techniques (countermeasures and sub-countermeasures) used to achieve system protection and mission effectiveness.
Section M: Proposal Evaluation Criteria
The offeror’s proposal will be evaluated based upon:
The extent to which the offeror employs a disciplined, structured system security engineering (SSE) process, including criticality analysis, in arriving at its system specification and design.
The extent to which the SSE process identifies and mitigates threat and vulnerability risks to the system mission effectiveness.
The extent to which the SSE process is integrated within the overall Systems Engineering, Integration and Test (SEI&T) process.
The extent to which supply chain risk protection, detection, and response procedures and activities are incorporated into the system acquisition.

22. Section 3: Cloud Business Case Analysis Section
The JTAMS PM is looking for a Cloud Service Provider (CSP) as an alternative hosting solution for its web-based applications because the Army has not designated the current Program Director Hosting Service (PD HS) at Fort Lincoln as an enduring data center.
Section Requirements:
Provide an overview (Executive Summary) of the overall scenario and the Section requirements to include the presentation overview.
Identify and justify the IIL required for the program; Describe the impacts to the program
Provide an overview of the program Cloud transition requirements
Compare and contrast the capabilities of the two cloud service providers
Identify 3 risks to the program (based on the requirements and capabilities provided)
Provide an Section summary/conclusion/recommendation
Section 5
SEE: Section 5 Cloud BCA

23. Scenario and Presentation Overview
The JTAMS PM is looking for a Cloud Service Provider (CSP) as an alternative hosting solution for its web-based applications because the Army has not designated the current Program Director Hosting Service (PD HS) at Fort Lincoln as an enduring data center.
JTAMS PM is looking for a CSP that can provide:
resource pooling
self-service on-demand capabilities for DEV and TEST environments only
elasticity capability
JTAMS PM needs a CSP that:
will manage all aspects up to the application level
Can provide monthly reporting on utilization and critical issues
A CSP that is accredited to impact level 4, due to the current ATO for JTAMS including PII
The CSP must be accessed via a DoD CAP or a facility hosted on DoD premises with in a facility operated under DoD security policies and control
JTAMS PM requires the CSP to:
Provide a COOP environment
An SLA or capability consistent with cloud services
JTAMS PM would like:
to leverage a pay-for use model
JTAMS will need to migrate its data with the help of a tested migration plan.
Section 5
SEE: Section 5 Cloud BCA
Use the Cloud BCA Scenario and Instructions

24. Scenario and Presentation Overview (CONT)
In this Section we will:
- provide an overview of the scenario
- define IIL, identify IIL for program, describe program impact of IIL
- provide an overview of cloud transition requirements
- compare and contrast the 2 CSP’s presented
- identify 3 risks to program/risk statements
-provide overall summary and recommendation
Section 5
SEE: Section 5 Cloud BCA

25. Information Impact Level
Identify and justify the IIL required for the program:
JTAMS requires IIL level 4 because the system contains PII; must be accessed via DoD CAP or hosted at DoD location/under DoD control
Describe the impacts to the program inherent in the required impact level:
JTAMS must use a CSP that is authorized at the FedRamp Moderate baseline; limits CSP options, location, level of required effort/who maintains responsibility for governance, risk management, and compliance
Section 5
SEE: Section 5 Cloud BCA
Controlled Unclassified Information (CUI):
Export Control
Privacy Information (PII)
Protected Health Information (PHI)
Other information requiring explicit CUI designation
FOUO
Official Use Only
Law Enforcement Sensitive
Critical Infrastructure Information
Sensitive Security Information

26. Overall Rec. Primary Rationale
Overview of Program Cloud Transition Requirements / Provider Capability Comparison
Capability
Overview of Capability/Terminology
AWS*
DISA MilCloud*
Derived Requirements / Explanations / Discussion
Cloud Capabilities
NIST-defined characteristics: Resource Pooling / Self-Service on Demand / Broad Network Access / Elasticity / Metered Usage 5 4
AWS: well beyond the five cloud characteristics that NIST has identified
DISA: four of the five cloud characteristics; not available to customers is metered usage
Security
Ability to demonstrate security/IA requirements 3
Neither AWS or DISA provided full security capabilities.
Derived: Is security Integrated with RMF?What Physical, Logical security controls in place? Which security controls must be implemented by the organization?
AWS: Logical security seems to be pushed to the customer level. Physical security controls in place.
DISA: JIE compliant. Shared responsibility of VDCs as specified in overview document.
Ease of Migration
Ability to provide a mechanism and defined process to migrate apps and data into the HP infrastructure. 2
Derived: Do migration paths both to and from the HP exist? What support areas are in place?
AWS: Significant challenges due to proprietary format. See derived requirements for exit/support strategy questions.
DISA: Multiple options using VMWare and standardized tools/processes
Management / Monitoring
Ability to manage and monitor network and storage infrastructure. 1
Derived: Ability to Manage/Monitor to Application level? What percentage of tools are automated? Manual?
AWS: Monitoring not included in IL4. Options available, but at additional (unrealized?) cost
DISA: Responsibility of customer. New toolsets, processes, and capabilities would all need to be developed and implemented.
DR / COOP
Demonstrated and proven DR/COOP process with secondary site.
Derived: Distance from primary to secondary site, Hot or Cold site
AWS: Substantial DR/COOP processes, but secondary site location is an issue requiring waiver.
DISA: DR and COOP functionality in place. We would have to learn CONS3RT in order to fully take advantage of COOP.
Governance
Governance and RM processes in place. CC SRG impact level 4 or above.
AWS: provides a great governance, risk management, and compliance solution - Host operating system up is our responsibility. IL4 - CAP required
DISA: IL5 – more governance/risk transferred to customer
Total (less Cost) 19 21
Costs
Include Up Front and Ongoing for Development, Test, and Production. Minimized cost with maximized capabilities. Pay-for-use model desired
$110k
$327k
Derived: What Development and test requirements beyond FY18 exist?
AWS: many additional costs may be required, which may drive this cost up.
DISA: does not include COOP cost and some additional services.
Section 5
SEE: Section 5 Cloud BCA
Cloud BCA Scenario and Instructions
Overall Rec.
Primary Rationale
*1: No Capability; 2: /Low Capability; 3: Some Capability 4: Most Capability 5: Full Capability

27. Provider Capability Comparison Cost (Back-up slide)
AWS Cost Overview:
Environment
Upfront Annual Payment
Monthly Payment
Total Annual Costs
Development
$9,650.01
$1,948.95
$33,037.41
Test
$11,788.39
$1,890.74
$34,477.27
Production
$13,071.79
$2,461.85
$42,613.99
Total
$34,510.19
$6,301.54
$110,128.67
DISA Cost Overview
Environment
Connection Fee
Monthly Payment
Total Annual Costs
Development
$8,580
$7,584.04
$99,588.48
Test
$8,970
$7,715.87
$101,560.44
Production
$10,530
$9,663.49
$126,491.88
Total
$28,080
$24,963.40
$327,640.08
Cost Comparison and Overview
Section 5
SEE: Section 5 Cloud BCA
Costs
Overview
AWS
DISA MilCloud
Explanations / Comment / Discussion
Include Up Front and Ongoing for Development, Test, and Production. Minimized cost with maximized capabilities. Pay-for-use model desired
$110,128.67
$327,620.08
Derived: What Development and test requirements beyond FY18 exist?
AWS: many additional costs may be required, which may drive this cost up.
DISA: does not include COOP cost and some additional services.

28. Overall Recommendation (use as needed)
Based upon the scores alone, AWS looks like the preferred option. Two areas that might impact an eventual decision might be:
The AWS initial migration: AWS’ proprietary processes and virtual instances practically guarantee that the initial migration costs associated with the effort will be substantial. Furthermore, should we choose to migrate *away* from AWS in the future, moving from the AWS proprietary environment to another would also prove to be substantial.
DISA/milCloud does have challenges of its own, especially in the areas of management and monitoring. However, we feel that we can leverage not only our own experience in this area, but also the experiences of other providers and some of the available tools to overcome these challenges.
For these reasons, a recommendation of AWS may be offered based on similar capability and the costs provided. The DISA/milCloud recommendation may be the choice if the as yet undefined cost risks (combined with other factors discussed earlier) are seen as involving too much risk.
Section 5
SEE: Section 5 Cloud BCA
Use the Cloud BCA Scenario and Instructions

29. Risk Analysis – choosing DISA
Identify 3 risks to the program (based on your selected provider). Present risk statements for your three risks and present your risks using a risk cube. Focus your risk analysis on the following categories: Security, Support, Staffing, and Governance. 1p
1. Security Risk: If the gap between hosting and program security and IA support is not closed then the program will loose it Authority To Operate (ATO). 2c
3c,s
2. Data Migration Risk: If data migration cannot be completed without significant redesign, then migration to the cloud may not be possible within cost and schedule. 1 2 3 4 5
Section 5
SEE: Section 5 Cloud BCA
3. Monitoring Risk: If the CSP’s management and monitoring services are not readily available and automated, then external tools may be required to provide management and monitoring services. P C S P C S P C S
Yellow = Medium Risk
Green = Low Risk
Red =
High Risk

30. Risk Analysis— choosing AWS
Identify 3 risks to the program (based on your selected provider). Present risk statements for your three risks and present your risks using a risk cube. Focus your risk analysis on the following categories: Security, Support, Staffing, and Governance.
1) PM JTAMS desires a PaaS capability. IF the cloud provider is unable to offer a PaaS offering, THEN the program will have to continue to manage more than just the application level, which will have a cost impact. 1C
3C,S
2C,S,P
2) IF PM JTAMS is unable to develop a tested migration path, THEN the environment will not be successfully migrated without operational impact. 1 2 3 4 5
Section 5
SEE: Section 5 Cloud BCA P C S P C S
3) IF the hosting environment does not provide security controls that can be inherited, THEN additional products and services will be needed to ensure adequate security controls are in place, which will have a cost and schedule impact. P C S
Yellow = Medium Risk
Green = Low Risk
Red =
High Risk

31. Cloud Section Summary/Conclusion
1. Summarize the Requirement
The program desires to move to a cloud provider who provides the greatest benefit to the government, while meeting defined program and project requirements related to IIL.
2. Summarize the Cloud Provider Capability Comparison Results
Overall, while the providers are nearly “equal” in terms of capabilities – technically DISA/milCloud exceeds AWS in terms of capability.
3. Summarize the risks and impacts to the program:
The AWS initial migration: AWS’ proprietary processes and virtual instances practically guarantee that the initial migration costs associated with the effort will be substantial. Furthermore, should we choose to migrate *away* from AWS in the future, moving from the AWS proprietary environment to another would also prove to be substantial.
DISA/milCloud does have challenges of its own, especially in the areas of management and monitoring. However, we feel that we can leverage not only our own experience in this area, but also the experiences of other providers and some of the available tools to overcome these challenges.
4. Provide an overall Recommendation/Conclusion: a recommendation of AWS may be offered based on similar capability and the costs provided. The DISA/milCloud recommendation may be the choice if the as yet undefined cost risks (combined with other factors discussed earlier) are seen as involving too much risk.
Section 5
SEE: Section 5 Cloud BCA