Download presentation
Presentation is loading. Please wait.
Published byRudolph Wilson Modified over 7 years ago
1
Lecturer: Dr. AJ Bieszczad Chapter 1211-1 Evaluating products, processes and resources how do we decide which technique or tool to use? hoe do we evaluate the efficiency of what we are already doing? when is one technique more appropriate than another? how doe we demonstrate that our products, processes and resources have the characteristics (e.g., quality) as we want them to have?
2
Lecturer: Dr. AJ Bieszczad Chapter 1211-2 Categories of evaluation Feature analysis: rate and rank attributes Survey: document relationships Case study Formal experiment
3
Lecturer: Dr. AJ Bieszczad Chapter 1211-3 Example feature analysis Table 12.1. Design tool ratings FeatureTool 1: t-OO-l Tool 2: ObjecTool Tool 3: EasyDesign Importance Good user interface4543 Object-oriented design5555 Consistency checking5313 Use cases4442 Runs on Unix5455 Score857773
4
Lecturer: Dr. AJ Bieszczad Chapter 1211-4 Case study types Sister projects: each is typical and has similar values for the independent variables Baseline: compare single project to organizational norm Random selection: partition single project into parts
5
Lecturer: Dr. AJ Bieszczad Chapter 1211-5 Formal experiment Controls variables Uses methods to reduce bias and eliminate confounding factors Often replicated Instances are representative: sample over the variables (whereas case study samples from the variables)
6
Lecturer: Dr. AJ Bieszczad Chapter 1211-6 Evaluation steps Setting the hypothesis Maintaining control over variables Making investigation meaningful –generalizing the results
7
Lecturer: Dr. AJ Bieszczad Chapter 1211-7 Investigation and evaluation: Non-trivial pursuits
8
Lecturer: Dr. AJ Bieszczad Chapter 1211-8 Table 12.2. Common pitfalls in evaluation. Adapted with permission from (Liebman 1994) PitfallDescription 1. ConfoundingAnother factor is causing the effect. 2. Cause or effect?The factor could be a result, not a cause, of the treatment. 3. ChanceThere is always a small possibility that your result happened by chance. 4. HomogeneityYou can find no link because all subjects had the same level of the factor. 5. MisclassificationYou can find no link because you cannot accurately classify each subject’s level of the factor. 6. BiasSelection procedures or administration of the study inadvertently bias the result. 7. Too shortThe short-term effects are different from the long-term ones. 8. Wrong amountThe factor would have had an effect, but not in the amount used in the study. 9. Wrong situationThe factor has the desired effect, but not in the situation studied.
9
Lecturer: Dr. AJ Bieszczad Chapter 1211-9 Boehm’s quality model
10
Lecturer: Dr. AJ Bieszczad Chapter 1211-10 ISO 9126 quality model
11
Lecturer: Dr. AJ Bieszczad Chapter 1211-11 Table 12.4. ISO 9126 quality characteristics. Quality characteristicDefinition FunctionalityA set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. ReliabilityA set of attributes that bear on the capability of software to maintain its performance level under stated conditions for a stated period of time. UsabilityA set of attributes that bear on the effort needed for use and on the individual assessment of such use by a stated or implied set of users. EfficiencyA set of attributes that bear on the relationship between the software’s performance and the amount of resources used under stated conditions. MaintainabilityA set of attributes that bear on the effort needed to make specified modifications (which may include corrections, improvements, or adaptations of software to environmental changes and changes in the requirements and functional specifications). PortabilityA set of attributes that bear on the ability of software to be transferred from one environment to another (including the organizational, hardware or software environment).
12
Lecturer: Dr. AJ Bieszczad Chapter 1211-12 Dromey quality model Identify a set of high-level quality attributes. Identify product components. Identify and classify the most significant, tangible, quality-carrying properties for each component. Propose a set of axioms for linking product properties to quality attributes. Evaluate the model, identify its weaknesses, and refine or recreate it.
13
Lecturer: Dr. AJ Bieszczad Chapter 1211-13 Linking product properties to quality in Dromey’s model
14
Lecturer: Dr. AJ Bieszczad Chapter 1211-14 Products properties and their effect on quality
15
Lecturer: Dr. AJ Bieszczad Chapter 1211-15 Baselines and targets for quality evaluation baseline – typical result in an organization or category target – minimally acceptable behavior from guidelines of US Department of Defense
16
Lecturer: Dr. AJ Bieszczad Chapter 1211-16 Software reuse Producer reuse: creating components for someone else to use Consumer reuse: using components developed for some other product Black-box reuse: using component without modification Clear- or white-box reuse: modifying component before reusing it
17
Lecturer: Dr. AJ Bieszczad Chapter 1211-17 More on reuse Compositional reuse: uses components as building blocks; development done from bottom up Generative reuse: components designed specifically for a domain; design is top- down Domain analysis: identifies areas of commonality that make a domain ripe for reuse
18
Lecturer: Dr. AJ Bieszczad Chapter 1211-18 Table 12.6. Aspects of reuse. (adapted from Prieto-Díaz 1993) SubstanceScopeModeTechniqueIntentionProduct Ideas and concepts VerticalPlanned and systematic CompositionalBlack-box, as isSource code Artifacts and components HorizontalAd hoc, opportunistic GenerativeClear-box, modified Design Procedures, skills, experience Requirements PatternsObjects ArchitectureData Processes Documentation Tests
19
Lecturer: Dr. AJ Bieszczad Chapter 1211-19 Example of hierarchical classification scheme in a component retrieval system
20
Lecturer: Dr. AJ Bieszczad Chapter 1211-20 Reuse lessons Reuse goals should be measurable. Management should resolve reuse goals early. Different perspectives may generate different questions about reuse. Every organization must decide at what level to answer reuse questions. Integrate the reuse process into the development process. Let your business goals suggest what to measure.
21
Lecturer: Dr. AJ Bieszczad Chapter 1211-21 Reuse issues: conflicting interpretation of goals
22
Lecturer: Dr. AJ Bieszczad Chapter 1211-22 Postmortem analysis Design and promulgate a project survey to collect relevant data. Collect objective project information. Conduct a debriefing meeting. Conduct a project history day. Publish the results by focusing on lessons learned.
23
Lecturer: Dr. AJ Bieszczad Chapter 1211-23 Schedule predictability chart
24
Lecturer: Dr. AJ Bieszczad Chapter 1211-24 Table 12.9. When post-implementation evaluation is done. Time periodPercentage of respondents (of 92 organizations) Just before delivery27.8% At delivery4.20% One month after delivery22.20% Two months after delivery6.90% Three months after delivery18.10% Four months after delivery1.40% Five months after delivery1.40% Six months after delivery13.90% Twelve months after delivery4.20%
25
Lecturer: Dr. AJ Bieszczad Chapter 1211-25 Capability Maturity Model from SEI
26
Lecturer: Dr. AJ Bieszczad Chapter 1211-26 Table 12.10. Required questions for level 1 of process maturity model. Question numberQuestion 1.1.3Does the Software Quality Assurance function have a management reporting channel separate from the software development project management? 1.1.6Is there a software configuration control function for each project that involves software development? 2.1.3Is a formal process used in the management review of each software development prior to making contractual commitments? 2.1.14Is a formal procedure used to make estimates of software size? 2.1.15Is a formal procedure used to produce software development schedules? 2.1.16Are formal procedures applied to estimating software development cost? 2.2.2Are profiles of software size maintained for each software configuration item over time? 2.2.4Are statistics on software code and test errors gathered? 2.4.1Does senior management have a mechanism for the regular review of the status of software development projects? 2.4.7Do software development first-line managers sign off on their schedule and cost estimates? 2.4.9Is a mechanism used for controlling changes to the software requirements? 2.4.17Is a mechanism used for controlling changes to the code?
27
Lecturer: Dr. AJ Bieszczad Chapter 1211-27 Table 12.11. Key process areas in the CMM (Paulk et al. 1993) CMM levelKey process areas Initialnone RepeatableRequirements management Software project planning Software project tracking and oversight Software subcontract management Software quality assurance Software configuration management DefinedOrganization process focus Organization process definition Training program Integrated software management Software product engineering Intergroup coordination Peer reviews ManagedQuantitative process management Software quality management OptimizingFault prevention Technology change management Process change management
28
Lecturer: Dr. AJ Bieszczad Chapter 1211-28 A repeatable process
29
Lecturer: Dr. AJ Bieszczad Chapter 1211-29 A defined process
30
Lecturer: Dr. AJ Bieszczad Chapter 1211-30 A managed process
31
Lecturer: Dr. AJ Bieszczad Chapter 1211-31 An optimizing process T 0, T 1, T 2, …, T n - timeframes
32
Lecturer: Dr. AJ Bieszczad Chapter 1211-32 Evaluation of CMM conformance Each process area comprises of key practices The key practices are organized by these common features: –Commitment to perform practices on policy, leadership –Ability to perform practices on resources, training, orientation, tools, organizational structure –Activities performed practices on plans, procedures, work performed, corrective action, tracking –Measurement and analysis practices on process measurements and analysis –Verifying implementation practices on management reviews and audits
33
Lecturer: Dr. AJ Bieszczad Chapter 1211-33 Software Process Improvement and Capability dEtermination (SPICE) Base practices (essential activities) and general practices (institutionalize or implement the process) Ratings in terms of –customer-supplied processes –engineering processes –project processes –support processes –organizational processes 0 – not performed 1 – performed informally 2 – planned and tracked 3 – well defined 4 – quantitatively controlled 5 – continuously improving
34
Lecturer: Dr. AJ Bieszczad Chapter 1211-34 ISO 9001 clauses. Clause numberSubject matter 4.1Management responsibility 4.2Quality system 4.3Contract review 4.4Design control 4.5Document and data control 4.6Purchasing 4.7Control of customer-supplied product 4.8Product identification and traceability 4.9Process control 4.10Inspection and testing 4.11Control of inspection, measuring and test equipment 4.12Inspection and test status 4.13Control of nonconforming product 4.14Corrective and preventive action 4.15Handling, storage, packaging, preservation and delivery 4.16Control of quality records 4.17Internal quality audits 4.18Training 4.19Servicing 4.20Statistical techniques
35
Lecturer: Dr. AJ Bieszczad Chapter 1211-35 People capability maturity model. LevelFocusKey practices 5: optimizingContinuous knowledge and skills improvement Continuous workforce innovation Coaching Personal competency development 4: managedEffectiveness measured and managed, high performance teams developed Organizational performance alignment Organizational competency management Team-based practices Team-building Mentoring 3: definedCompetency-based workforce practices Participatory culture Competency-based practices Career development Competency development Workforce planning Knowledge and skills analysis 2: repeatableManagement takes responsibility for managing its people Compensation Training Performance management Staffing Communication Work environment 1: initial
36
Lecturer: Dr. AJ Bieszczad Chapter 1211-36 Return on investment Use net present value (NPV) –the present value of the benefits minus the value of the initial investments Example: Cash flowsCOTSReuse Initial investment-9000-4000 Year 15000-2000 Year 260002000 Year 370004500 Year 4-40006000 Sum of all cash flows50006500 NPV at 15%22002162
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.