1. Architecture Review Boards Foundations Commitment Review
Including Past Experiences
October 22, 2012

2. Outline AT&T/Lucent ARB Concept -Overview -Results -Recommendations
USC CS577 ARB Concept
-Participants
-Procedures

10. [CSCI 577a FCR]
[CSCI 577a DCR and 577b RDCR]

24. Outline AT&T/Lucent ARB Concept -Overview -Results -Recommendations
USC CS577 ARB Concept
-Participants
-Procedures

25. The Incremental Commitment Spiral Model (ICSM)
4 Key Principles:
Stakeholder value-based system definition and evolution
Incremental commitment and accountability
Concurrent system and software definition and development
Evidence and risk-based decision making
First of all, ICSM or the Incremental Commitment Spiral Model, which is a process model. One of the most important ingredient of the ICSM is the opportunity and risk that will determine course of action in the development.

26. ICSM for 24-week e-services projects26

27. USC CS577 ARB Participants
Project Team
-Everybody presents something
Reviewers
-Clients
-Instructors and TA’s
-Industry participants
80 minute time slots

28. ARB/milestones for two-semester team
FCR ARB: October 29th to November 2nd
Based on preliminary FC package
Focus on FCR success criteria
DCR ARB: December 3rd to 7th
Based on draft DC package
Focus on DCR success criteria

29. ARB/milestones for one-semester AA team
FCR/DCR ARB: October 29th to November 2nd
Based on DC package
Focus on DCR success criteria
CCD: November 19th
Core Capability Drive-through
Client(s) will have hands-on experience on your core capabilities
TRR: November 26st
Informal review with TA; readiness for transition
OCR: November 28th to December 2nd
Based on AsBuilt package

30. ARB Review Success Criteria
FCR
For at least one architecture, a system built to arch. will:
Support the Ops Concept
Satisfy the Requirements
Be faithful to the Prototype(s)
Be buildable within the budgets and schedules in the Plan
Show viable business case
Most major risks identified and resolved or covered by risk management plan
Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR)
DCR
For the selected architecture, a system built to the arch. will:
Be buildable within the budgets and schedules in the Plan
All major risks resolved or covered by risk management plan
Key stakeholders committed to support full life cycle

31. Commitment Review Success Criteria
CCD
TRR / OCR
Determine whether client needs anything further to ensure successful Transition and Operation
Changes in priorities for remaining features?
Changes being made to operational procedures?
More materials needed for training?
Changes in system data or environment we should prepare for?
Anyone else who should experience CCD?
Show value
Product works as expected (or not with noted exceptions)
Product will help users do job
Show quality development
e.g. relevant parts of your
IOC documentation
Process
Show sustainability
e.g. support requirements/plans
Transition plan & status
Show confidence that product is/will be ready to be used
e.g. Transition plan & status
See also value

32. Team Preparation for ARB Reviews
Week-1
Within-team Dry run of presentations and demo
Further dry runs as necessary
ARB Week
ARB Presentation and discussion
Follow-up team discussions, client discussions
Week+1
Monday: FC packages due
Monday: DC packages due

33. FCR ARB Session Outline Architected Agile Team
(x,y): (presentation time, total time)
(5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations
(10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; desired capabilities and goals
(10,10) Prototype. Most significant capabilities [buying information](especially those with high risk if gotten wrong)
(5, 10) Requirements. Most significant requirements
(5, 10) Architecture. Top-level physical and logical architecture; status of NDI/reuse choices
(5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation
(5, 10) Feasibility Evidence. Business case (beginnings, including benefits analysis); NDI analysis results; major risks;
(5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix
(10) Things done right; issues to address (Instructional staff)
Plan on 2 minutes per briefing chart, except title
Each chart MUST have information specific to your project

34. FCR ARB Session Outline NDI/ NCS Team (2 semesters)
(x,y): (presentation time, total time)
(5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations
(10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals
(5 , 5) WinWin Agreements. Agreed Win conditions in each category
(10,10) Prototype. Most significant capabilities, NDI/NCS integration
(5, 10) Architecture. Top-level physical and logical architecture;
(5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation
(10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence
(5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix
(10) Things done right; issues to address (Instructional staff)
Plan on 2 minutes per briefing chart, except title
Each chart MUST have information specific to your project

35. DCR ARB Session Outline NDI/ NCS Team (1 semester)
(x,y): (presentation time, total time)
(5, 5) Remote Team Member(s) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions; S/P Engineer observations
(10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals
(5 , 5) WinWin Agreements. Agreed Win conditions in each category
(10,10) Prototype/ Product Demo. Most significant capabilities, NDI/NCS integration
(5, 5) Architecture. Top-level physical and logical architecture; (if applicable)
(5, 10) Life Cycle plan. Life cycle strategy; focus on Development phase & transition increment; key stakeholder responsibilities; project plan; resource estimation
(10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence
(5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix
(5, 5) Acceptance Test Plan and Cases
(10) Things done right; issues to address (Instructional staff)
Plan on 2 minutes per briefing chart, except title
Each chart MUST have information specific to your project

36. Specfics for DEN students Team’s strong points & weak points
List at least one item for each of the following
List your team’s strong points
Operational view
Technical view
List your team’s weak points
Identify specific technical concerns & possible solutions
Identify operational risks & possible mitigation
Sources of observations
Team activities, package evaluation, WinWin negotiation, and etc.

37. Specfics for DEN students System/Project Engineer at ARB
WinWin Shaping Status:
Open WinCs: Identified
Agreed WinCs with Issues & without Issues
Overall Project Evaluation consideration
All SCS(s) CRACKness
Complexity
Precedentedness (for team!)
Communication & use of communication tools between on-campus team, S/PE and client
Skills/needs match
Knowledge/experience mis-matches

38. QFP – Defect Identification Review
For each document section, UML model, and etc. identify the following
type of review you used (peer review, agile artifact review, and etc. )
Other form of defect identification, e.g., grading, client feedback, etc.
Current Defect Injection and Removal Matrix
Current, total defect information from your progress report

39. ARB Session Outline DCR Similar format to FCR, different focus:
Less time for OCD, Prototype
More time for Architecture, Plans
Details TBD based on FCR ARB experience
General rule on focus: emphasize your project's high risk areas
At FCR (in most cases) this will involve establishing the operational concept (including system analysis)
At DCR (in most cases) this will involve the system design and development plan (especially schedule)

40. Results To Date * Reasons: - Poor performance Poor team management
Poor communication (within team and with client)

41. ARB Packages
If you would like to have your ARB presentation and FC/DC package printed for you at the ARB meeting,
the documents (in a zip file) to 24 hours in advance, unless your ARB is on Monday, in which case, the documents 5 hours in advance.
Otherwise, bring 4 copies of your ARB presentation and 2 copies of your FC package to the meeting.

42. Demos in ARB
For those teams doing a live demo in the ARB meeting, please include screenshots of your demo in your presentation
for your IV&Vers to see the demo in case video connection is a problem
for reviewers to make notes on

43. ARB Presentation slides
Upload your ARB presentation slides (before your ARB) on your team website for off-campus students

44. Webex & Teleconf
Off-campus students who can not attend the ARB in-person, will be connecting through Webex

45. Past FCR Experiences and General Guidelines

46. Outline FCR ARB 2011 Feedback Summary
Examples of Good and Bad Practices seen at ARBs
ARB Chartsmanship & Presentation

47. Overall FCR Feedback – Fall 2011
Generally done well: presentations, time management, client rapport
Reconcile FCR content with ARB Success Criteria
Define key terms, acronyms in SID glossary
If you’re deferring or skipping a normally-included artifact, explain why (e.g. COTS internals unavailable), and note in SID-document tailoring section
Occassional order changes in presentation without telling us in a modified agenda at the beginning
Role of primary DEN/remote student often mis-stated
Only one IS System/Project Engineer (S/PE)
Includes IIV&V (also done by second DEN/remote student)
Includes shaping, and re-shaping throughout semester(s)

48. Overall FCR Feedback 2011 - 2 When asked a question:
Give the answer in brief, this will help your time management and the Review Board will get the desired information
Do NOT answer back while Review Board attempts to provide guidance
Many had poor time management that indicated that presentation(s) had NOT been practiced
Occassional pointing at laptop screen, not projected image (even better over Webex, use mouse)
Very occasionally, slides with NO value added 48 48

49. OCD Feedback (1) System boundary diagram
Generally done well: Organizational goals, Operational concept, System boundary and organizational environment.
Some benefits chains need rework:
Added stakeholders: users, clients, developers, IIV&Vers, database administrators, maintainers, interoperators, suppliers
Assumptions are about environment not about outcome
Involvement/use of system before system is built
Some organization goal(s) are Benefits Chain end outcomes
System boundary diagram
If you are using the component in/for your system, remove it from environment, e.g. PHP, .NET framework. 49 49

50. OCD Feedback (2) Organization Goals
Are Benefits Chain End Outcomes (or maybe a subset)
Are NOT project Initiative contributions
Identify Levels of Service properly
100% availability, 100% reliability - not feasible!
Make sure you can measure LOS goals
Prototypes and System are NOT the same (usually)
Business Workflow
Use activity-type diagram
Illustrate business activities
Not technical/system activity
May not even “see” system explicitly 50 50

51. Prototype Feedback
Generally done well: GUI Prototypes, Good understanding of client’s needs
Prototype all high-risk elements, not just GUI’s
COTS interoperability, performance, scalability
Use user/client-friendly terms
“John Doe, 22 Elm St.” not generic substitutions like “Name1, Addr1”
Use as an opportunity to gather more information and/or examples
Identify end users and try to get feedback from end users
Focus on important and high priority requirements (initially) 51 51

52. SSRD Feedback
Generally done well: Project and Capability requirements, OCD-SSRD traceability
Prioritize all the requirements
Propagate LOS goals from OCD into SSRD or drop LOS requirements from SSRD (and SSAD)
Distinguish between client imposed requirements (SSRD) and developer choice solution (SSAD)
Make sure all requirements are testable
Qualify “24/7 Availability" with exceptions
Update the new requirements in WinBook tool
There is no such thing as an “implicit requirement” 52 52

53. SSAD Feedback Generally done well: Overall views
Follow UML conventions (arrows, annotations, etc.)
Generalization of actors
Uncommon mistakes in use-case diagrams
Two actors-one use case (means BOTH present)
Arrow direction for <extends> or <includes>
Devil is in the detail; simple is the best
Only two teams had an adequate start on Information & Arctifacts Diagram
Read the exit criteria for the milestone carefully 53 53

54. SSAD Feedback Generally done well: Overall views
Follow UML conventions (arrows, annotations, etc.)
Generalization of actors
Uncommon mistakes in use-case diagrams
Two actors-one use case (means BOTH present)
Arrow direction for <extends> or <includes>
Devil is in the detail; simple is the best
Only two teams had an adequate start on Information & Arctifacts Diagram
Read the exit criteria for the milestone carefully 54 54

55. Team 3 Signup New User

56. Team 3 Information & Artifacts

57. Team 14 Information & Artifacts

58. LCP Feedback - 1
Generally done well: overall strategy, roles and responsibilities
Too many 577b TBDs
Identify required skills for NN new team member(s) (577b; if needed to meet "team size")
Show (concentrate on) your future plan; not the past
Full Foundations [nee Elaboration] phase plan
Don’t plan ONLY for documentation
Include Modeling
Include Prototyping; coding; executable architecture 58 58

59. LCP Feedback - 2 COCOMO drivers
Often differ per the module (type)
PMATs rationale was often wrong:
CS577 projects' process maturity should be between 2 and 3
NOBODY did the detailed check list in the SwCEwCII book!
Some driver rationales were "ridiculous"
Add DEN Student interactions to Gantt Chart
IIV&V
System/Project Engineer (includes Shaping)
Add maintainer’s responsibilities 59 59

60. FED Feedback
Generally done well: Business case framework, risk analysis
Specify LOS feasibility plans
Include training, operations, maintenance, opportunity costs/effort
Few had developers hours as cost (which is correct)
Try to quantify benefits, show return on investment
Change ROI to reflect on-going costs (possibly savings)
Distinguish one-time from annual costs in business case
Benefits start in mid 2010 (go at 6 months granularity); Costs start mid 2009
Elaborate process rationale
Complete section 6 – COTS Analysis 60 60

61. SID Feedback Requirements traceability Update Glossary!
OCD  WinC  SSRD  SSAD
Update every time there is a change
Update Glossary!
Glossary MUST have all system/project specific terms
Non-standard (unusual uses) 61 61

62. QFP Generally done well
Some missing traceability injection-removal matrix
Some seemed to try to "snow us with data", not present just a quick summary
Did NOT specify type of "Peer Review"
Pass around or Buddy Check (NOT desirable: no record of concerns)
Agile Artifact Review
Agile Internal/Informal (Role Based) Peer Review
Will be expected to say lots more to say at DCR! 62 62

63. Things to improve Presentation – communication skill
One word wrong could lead to billion $ loss.
Practice in front of others
Be concise and precise
Consistencies among each artifact
Team work vs. integrated individual works
Prepare your client:
Tell them what an ARB is (use agenda, success criteria)
Tell them what to expect from ARB
Time management
Get in and set-up ASAP
Have documents & client present 63 63