1. University of Southern California Center for Systems and Software Engineering Milestone Reviews CS 577b Software Engineering II Supannika Koolmanojwong

2. University of Southern California Center for Systems and Software Engineering Outline Milestone Reviews –Core Capability Drive thru –Transition Readiness Review –Design Code Review Individual Research Presentation 2©USC-CSSE

3. University of Southern California Center for Systems and Software Engineering Milestone review ©USC-CSSE3

4. University of Southern California Center for Systems and Software Engineering ©USC-CSSE4 Waterfall Model Spiral Model Iterative and Incremental Model Agile Model

5. University of Southern California Center for Systems and Software Engineering The Incremental Commitment Spiral Model 5©USC-CSSE

6. University of Southern California Center for Systems and Software Engineering 6©USC-CSSE

7. University of Southern California Center for Systems and Software Engineering Milestone review tasks Prepare for milestone review –Set scope Schedule review meeting Distribute meeting materials Conduct review meeting –Progress, Quality, risk profile, feasibility Record decision ©USC-CSSE7

8. University of Southern California Center for Systems and Software Engineering ©USC-CSSE8 ICSM –Class Milestones 02/11 04/03 04/1505/01

9. University of Southern California Center for Systems and Software Engineering ARB Review Success Criteria 9 FCR For at least one architecture, a system built to arch. will: Support the Ops Concept Satisfy the Requirements Be faithful to the Prototype(s) Be buildable within the budgets and schedules in the Plan Show viable business case Most major risks identified and resolved or covered by risk management plan Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR) DCR For the selected architecture, a system built to the arch. will: Support the Ops Concept Satisfy the Requirements Be faithful to the Prototype(s) Be buildable within the budgets and schedules in the Plan All major risks resolved or covered by risk management plan Key stakeholders committed to support full life cycle

10. University of Southern California Center for Systems and Software Engineering Commitment Review Success Criteria 10 TRR / OCR Show value Product works as expected (or not with noted exceptions) Product will help users do job Show quality development e.g. relevant parts of your IOC documentation Process Show sustainability e.g. support requirements/plans Transition plan & status Show confidence that product is/will be ready to be used e.g. Transition plan & status See also value Determine whether client needs anything further to ensure successful Transition and Operation Changes in priorities for remaining features? Changes being made to operational procedures? More materials needed for training? Changes in system data or environment we should prepare for? Anyone else who should experience CCD? CCD

11. University of Southern California Center for Systems and Software Engineering Core Capabilities Drive-thru ©USC-CSSE11

12. University of Southern California Center for Systems and Software Engineering ©USC-CSSE12 http://justincaseyouwerewondering.com/wp-content/uploads/2012/01/UsabilityTest.png

13. University of Southern California Center for Systems and Software Engineering CCD vs Testing ©USC-CSSE13 http://www.digitalcunzai.com/usability_testing.php

14. University of Southern California Center for Systems and Software Engineering CCD Purpose Improve likelihood of successful transition Improve operational stakeholder communication & motivation –Sense of what they’ll be getting Hands-on usage opportunity Product will soon be theirs to manage –Determine whether developers are on right track Use real operational scenarios (preferred) ©USC-CSSE14

15. University of Southern California Center for Systems and Software Engineering CCD Purpose Determine whether client needs anything further to ensure successful Transition and Operation –Changes in priorities for remaining features? –Changes being made to operational procedures? –More materials needed for training? –Changes in system data or environment? –Anyone else who should experience CCD? ©USC-CSSE 15

16. University of Southern California Center for Systems and Software Engineering Collaboration Problem ©USC-CSSE 16 Exploration & Valuation Foundations Development Operations ConstructionTransition

17. University of Southern California Center for Systems and Software Engineering Solution: CCD ©USC-CSSE 17 Exploration & Valuation Foundations Development Operations ConstructionTransition

18. University of Southern California Center for Systems and Software Engineering Developer Preparation ©USC-CSSE 18 Schedule drive–through time with client –40-60 minutes generally OK –Place: SAL 322, SAL 329 –Discuss with client Agenda Core Capabilities Scenarios (acceptance test sub-set) Drive–through users –Coordinate with 577b staff schedule prior to CCD Specify hardware/software required (if needed) http://blog.zilicus.com/enhancing-user-experience-of-zilicuspm/

19. University of Southern California Center for Systems and Software Engineering Developer Preparation ©USC-CSSE 19 Acceptance Test Subsets Prepare draft User’s Manual –Bring hard copies for clients & others –Minimally: describe how to use core capabilities Outline form –1 high-level per capability –Sublevels describe steps to perform capability Index cards –1-2 cards per capability –Steps to perform capability on cards

20. University of Southern California Center for Systems and Software Engineering Developer Preparation ©USC-CSSE 20 Prepare & dry run context presentation –Bring hard copies for clients & others Concern Logs –Can be in any form 577 template OR Your own –Included in the report

21. University of Southern California Center for Systems and Software Engineering Client Preparation ©USC-CSSE 21 Communicate with client Not just limited to client(s), but user(s) as well Plan “user” test scenario(s) of core capabilities –High-level description of typical usage –Should exercise capabilities in way user would May want to discuss with –Intended users –Acceptance Test developers Data, usage scenarios, users, etc.

22. University of Southern California Center for Systems and Software Engineering CCD Presentation: Baseline Agenda ©USC-CSSE 22 Summary of Core Capability content –Prioritized capabilities Review example Core Capability usage scenario Hands-on client usage –Most of time should be spent here Discussion of IOC priorities Tailor agenda to your project

23. University of Southern California Center for Systems and Software Engineering Client’s “Hands-on” Usage ©USC-CSSE 23 Imagine the reality once software is delivered Let the clients play with the system Use of user’s guide/manual –DO NOT tell the clients what to do Observe and listen –Usability –Reactions –Etc.

24. University of Southern California Center for Systems and Software Engineering CCD Products ©USC-CSSE 24 Concern logs (include things customer liked) –Core capabilities –User’s Manual –Tutorial –Test Cases As appropriate –Re–prioritized list of remaining features –List of changes Operational procedures System data or environment developers

25. University of Southern California Center for Systems and Software Engineering CCD Report ©USC-CSSE 25 Gather and submit –As-Is user’s manual –Concern Logs –Record of demonstration as performed Summarize Core Capabilities driven–through Include suggestions and positive feedbacks –Be specific –Break down by capability –New risks, if any, and mitigation plans Things that are Core Capabilities, but were NOT exercised: Mitigation = repeat CCD? Core capabilities not ready: Mitigation = do afterwards, coordinated with Client Changes in understanding –Reprioritized capabilities, if any –COCOMO Estimation + Code Count reports

26. University of Southern California Center for Systems and Software Engineering CCD Motivation Effort for remaining work More data for analysis Past estimation –Overestimate? –Underestimate? Analyze past estimations 26©USC-CSSE

27. University of Southern California Center for Systems and Software Engineering CCD Summary ©USC-CSSE 27 CCD is an opportunity to –Set customer expectations –Get positive feedbacks and suggestions –Validate core capabilities –Validate development directions and understandings –Ease transition –Identify new risks and mitigations

28. University of Southern California Center for Systems and Software Engineering Grading Guidelines Total = 20 points (3) Preparation for the CCD (10) Progress of your work (5) Quality of the core capabilities (2) Overall 4/2/201228(C) USC-CSSE

29. University of Southern California Center for Systems and Software Engineering Outline Core Capability Drive thru Transition Readiness Review Design Code Review 29©USC-CSSE

30. University of Southern California Center for Systems and Software Engineering 30 TRR Package Overview Transition Set –Transition plan –User manual Support Set –Support plan –Training materials inc. Tutorials & sample data –Test plan, cases, and results ©USC-CSSE

31. University of Southern California Center for Systems and Software Engineering 31 Transition Plan “Ensure that system’s operational stakeholders are able to successfully operate & maintain system” Plans for change from development mode to operational mode –Site installation & test –Load data –Pilot Operations –Preparations for roll-out ©USC-CSSE

32. University of Southern California Center for Systems and Software Engineering 32 User Manual Teach & guide user on how to use product i.e., describe Steps –For running SW –Performing operations Expected –Inputs –Output(s) Measures to be taken if errors occur ©USC-CSSE

33. University of Southern California Center for Systems and Software Engineering 33 Support Plan Guide system’s support stakeholders (administrators, operators, maintainers, …) on successful –Operation –Maintenance [and Enhancement?] ©USC-CSSE

34. University of Southern California Center for Systems and Software Engineering 34 Training Materials Identify training –Objectives –Schedule –Participants Prepare –Lectures –Examples –Exercises Prepare any sample data need for training ©USC-CSSE

35. University of Southern California Center for Systems and Software Engineering 35 TRR Presentation Summary Specific requirements for your presentation: –Your product! i.e., fully working IOC version –Salesman-like discussion of your project’s usefulness Base on your business case, etc Why is system going to be really great for customer –Transition issues & transition plan if you delivered your product how did it go? –(you should have by presentation) If not, when? –Support issues How will you support product, once deployed? –E.g. next term, for instance –OK to say that »You will never touch it ever again »Everything’s up to customer ©USC-CSSE

36. University of Southern California Center for Systems and Software Engineering 36 TRR Agenda (80 Minutes) 10 min.Introduction –Operational concept overview, TRR specific outline, transition objective & strategy 25 min.Demo of IOC (Product status demonstration) 5 min.Support Plan 10 min.Data Reporting & Archiving 15 min.Summary of Transition Plan (as appropriate) –HW, SW, site, staff preparation –Operational testing, training, & evaluation –Stakeholder roles & responsibilities –Milestone plan –Required resources –Software product elements (code, documentation, etc.) 15 min.Feedback *** Times are approximate *** ©USC-CSSE

37. University of Southern California Center for Systems and Software Engineering Goals & ObjectivesEntry ConditionsInputsSteps (concurrent) Develop, V&V, and transition the n th increment of capability Deliver on schedule, defer low-priority features if necessary Prepare rebaselined specifications, plans, FED for increment n+1 Execute next phase of manufacturing plans Acceptable responses to change requests SCSH commitment to life-cycle plans and approach Stabilized and prioritized requirements, specifications, and plans Adequate staffing; funding of development, rebaselining, and V&V teams; and manufacturing capabilities Technology Development work products Increment development and V&V plans risk resolution, infrastructure, plans, staff, resources Change traffic from previous-increment users, management, technology, and competition Stabilized build-to- specifications development of increment Continuous V&V of build-to-specifications artifacts Next-increments rebaselining, FEDs based on change traffic inputs Development progress monitoring and scope adjustment Increment installation, operational test, training, and acceptance Increment test preparations ©USC-CSSE37 Key phase elements (Development Phase)

38. University of Southern California Center for Systems and Software Engineering Key phase elements (Development Phase) Exit ConditionsWork productsPitfalls Accepted increment n operational capability Satisfactory disposition of change traffic Validated, rebaselined next- increments Capability Increment n test plans and preparations Committed to by SCSHs Accepted increment operational capability Satisfactory disposition of change traffic Validated, rebaselined next- increments Capabilities Incrementn test plans and preparations Committed to by SCSHs SCSH Commitment Inadequate phase budgets, schedules Neglecting SCSHs Destabilizing increment n development Inadequate development monitoring and rescoping Inadequate test and transition plans and preparations Inadequate change-source monitoring and response Unvalidated and unprioritized next-increment capabilities Weak manufacturing process and quality controls Inadequate next-phase budgets, schedules ©USC-CSSE38

39. University of Southern California Center for Systems and Software Engineering Grading Guidelines Total = 60 points (25) Progress of your work (10) Presentation (10) Risk Management (15) Quality 4/2/201239(C) USC-CSSE

40. University of Southern California Center for Systems and Software Engineering Outline Core Capability Drive thru Transition Readiness Review Design Code Review 40©USC-CSSE

41. University of Southern California Center for Systems and Software Engineering ©USC-CSSE41

42. University of Southern California Center for Systems and Software Engineering Internal Code Review 42©USC-CSSE

43. University of Southern California Center for Systems and Software Engineering Outline Problems & Motivation Faithful vs. Unfaithful Preparation and Review Process 4/2/201243(C) USC-CSSE

44. University of Southern California Center for Systems and Software Engineering Motivation To aid with successful transition Ensure that implementation is faithful to the design –Or at least consistent Sufficient documentation for maintenance period –Ease software understanding –Enable evolutionary development 4/2/201244(C) USC-CSSE

45. University of Southern California Center for Systems and Software Engineering Problems When implementation and design are different –The only way to understand implementation is to read through code –Documented artifacts appear useless –Wasted efforts and time Ultimately, Lose-Lose situation –Unhappy clients –Unhappy developers bugged by clients –Unhappy maintainers 4/2/201245(C) USC-CSSE

46. University of Southern California Center for Systems and Software Engineering Architectural Drift Classic problem in software engineering and software maintenance Often happen slowly during implementation –Developers feel changes are minor –Effects of changes multiply 4/2/201246(C) USC-CSSE

47. University of Southern California Center for Systems and Software Engineering Requirements-to-Code Elaboration Impact factor from one step to the next Increase in number of “statements” Clearly, significant impact when changes made to early stages Objective Shall Statements Use-Case Use-Case Steps SLOC x 2.46 x 0.89 x 7.06 x 66.91 * Ali Afzal Malik, Supannika Koolmanojwong, Barry Boehm. “An Empirical Study of Requirements-to-Code Elaboration Factors 4/2/201247(C) USC-CSSE

48. University of Southern California Center for Systems and Software Engineering Faithful Implementation The definition Structural elements  Source code –All should be there Source code must not utilize new major computational elements not specified in architecture Source code must not contain new connections not found in architecture Can we deviate from this? 4/2/2012(C) USC-CSSE48

49. University of Southern California Center for Systems and Software Engineering Unfaithful Implementation The implementation has its own architecture –Implementation is the architecture Impacts of failure to recognize distinction between designed and implemented architecture –Ability to reason about application’s architecture in future –Misleading (what they think vs. what they have) –Evolution strategies based on documented architecture doomed to failure 4/2/2012(C) USC-CSSE49

50. University of Southern California Center for Systems and Software Engineering Two-Way Mapping This is what should have been done Changes can be discovered during design or implementation phases –What to do? –How to effectively handle? 4/2/201250(C) USC-CSSE

51. University of Southern California Center for Systems and Software Engineering Design to Implementation Decide to first visit the design –Update the design –Review the changes –Implement according to the new design Always have a “blue print” to refer to Architects may have more understanding on impact of design changes –Easier to foresee future impacts 4/2/201251(C) USC-CSSE

52. University of Southern California Center for Systems and Software Engineering Implementation to Design Changes made to the implementation immediately –Then trace back to design –Update design to reflect new implementation Easier from developers perspective Many changes may have been missed in design update Difficult to foresee future effects –Unless highly experienced 4/2/201252(C) USC-CSSE

53. University of Southern California Center for Systems and Software Engineering Preparation Source code files –As implemented SSAD and UML models Personnel –Must be present: Developer (coder) Architect(s) –Optional Other team members 4/2/201253(C) USC-CSSE

54. University of Southern California Center for Systems and Software Engineering Review Process Mainly done with code tracing and walkthrough Design patterns / Architecture frameworks –Which is specified in SSAD? –Meet the standards? Design classes vs. Implemented classes –Consistent attributes –Consistent operations Use-case tracing –Consistency between use-case behavior and implementation 4/2/201254(C) USC-CSSE

55. University of Southern California Center for Systems and Software Engineering Grading Guidelines Total = 15 points (3) Faithfulness to design patterns / architecture frameworks (5) Faithfulness in design classes to implemented classes mapping (5) Accuracy of implemented Use-Case behaviors (2) Overall 4/2/201255(C) USC-CSSE

56. University of Southern California Center for Systems and Software Engineering Other things about milestone reviews Phase shifting –The milestones are being made, but the work hasn’t actually been completed, hence shifting it to subsequent phase Project signoff / Project close-out –Signifies completeness or business acceptance of the deliverable –Prepare list of deliverables in advance –Phase signoff – check point Conditional signoff – specific conditions that will need to be met in the succeeding phases; spike in Scrum; need more feasibility evidence Acceptable variance signoff – not completely satisfy, but deliver with acceptable variance Postponed signoffs – move to next checkpoint, should not happen –Closeout – sometimes include lesson learned, post-implementation report, recognize outstanding work, archive project records ©USC-CSSE56 Ref: www.pmhut.com

57. University of Southern California Center for Systems and Software Engineering Outline Milestone Reviews –Core Capability Drive thru –Transition Readiness Review –Design Code Review Individual Research Presentation 57©USC-CSSE

58. University of Southern California Center for Systems and Software Engineering Individual Research Presentation 15 minutes presentation April 26-May3 Off-campus students – select your schedule from the link provided. 2 minutes Q & A Powerpoint / vdo / audio / demo / prototype Peer review as in-class quizzes ©USC-CSSE58

59. University of Southern California Center for Systems and Software Engineering Overview 15 minutes presentation 2 minutes Q & A Powerpoint / vdo / audio / demo / prototype Peer review as in-class quizzes ©USC-CSSE59

60. University of Southern California Center for Systems and Software Engineering Presentation Criteria Interesting Soundness Contribution to 577 Benefit to SE students Quality of Work Quality of Presentation ©USC-CSSE60