2 Until you can measure something and express it in numbers, you have only the beginning of understanding.- Lord Kelvin
3 Until you can measure something and express it in numbers, you have only the beginning of understanding.- Lord KelvinNon-software metrics you use everyday:- Gas tank scale- Speedometer- Thermostat in your house- Battery monitor in your laptopWhat would happen if instead these were not numeric?What other examples do you have?Gas Tank☐A Lot☐SomeA little
4 Until you can measure something and express it in numbers, you have only the beginning of understanding.- Lord KelvinThe problem is non-numeric measurements are subjective… they mean different things to different people. Numbers are objective… they mean the same thing to everyone!When your friend says “oh yeah, John/Jane Doe is super attractive, you should go out with him/her”… that is a subjective measurement… so, you ask “Send me a photo”. Why? Because the subjective term “super attractive” has vastly different meanings for different people!Coming up: Metrics for software
5 Metrics for softwareWhen asked to measure something, always try to determine an objective measurement. If not possible, try to get as close as you can!Coming up: A Good Manager Measures
6 A Good Manager Measures processprocess metricsproject metricsmeasurementproduct metricsproductWhat do weuse as abasis?• size?product measurements are things like code complexity, number of function, classes, coupling, etc…process/project metrics are how well your process works• function?Coming up: We need a basis to say 20 defects per X lines of code. Why is this important?
7 We need a basis to say 20 defects per X lines of code We need a basis to say 20 defects per X lines of code. Why is this important?A Because lines of code equals costB We want our metrics to be valid across projects of many sizesC Because you just caused me to die in God of War III… stop asking these questions!D Because this helps up understand how big our program isComing up: Why Do We Measure?
8 Why Do We Measure? assess the status of an ongoing project track potential risksuncover problem areas before they go “critical,”adjust work flow or tasks,evaluate the project team’s ability to control quality of software work products.Coming up: Process versus Project Metrics
9 Process versus Project Metrics Process Metrics - Measure the process to help update and change the process as needed across many projectsProject Metrics - Measure specific aspects of a single project to improve the decisions made on that projectProcess may aggregate % delay of deliverables (schedule conformance) across many projects to determine how good our scheduling/planning process isProject would use the same measurement to make project level decisionsFrequently the same measurements can be usedfor both purposesComing up: Process Measurement
10 Process MeasurementWe measure the efficacy of a software process indirectly.That is, we derive a set of metrics based on the outcomes of the processOutcomes includemeasures of errors uncovered before release of the softwaredefects delivered to and reported by end-userswork products delivered (productivity)human effort expendedcalendar time expendedschedule conformancemany others…We also derive process metrics by measuring the characteristics of specific software engineering tasks.Efficacy - does the process do what is intendedSpecific tasks - measure any of the above for a specific phase (communication or analysis, design, construction)Coming up: Process Metrics Guidelines
11 Process Metrics Guidelines Use common sense and organizational sensitivity when interpreting metrics data.Provide regular feedback to the individuals and teams who collect measures and metrics.Don’t use metrics to appraise individuals.Work with practitioners and teams to set clear goals and metrics that will be used to achieve them.Never use metrics to threaten individuals or teams.Metrics data that indicate a problem area should not be considered “negative.” These data are merely an indicator for process improvement.Don’t obsess on a single metric to the exclusion of other important metrics.Coming up: If I calculate the number of defects per developer and rank them, then using that rank assign salary raises based on that.
12 If I calculate the number of defects per developer and rank them, then using that rank assign salary raises based on that.A. This is goodB. This is badComing up: Software Process Improvement
13 Software Process Improvement Process modelSPIProcess improvementrecommendationsImprovement goalsProcess metricsMake your metrics actionable!Coming up: Typical Process Metrics
14 Typical Process Metrics Quality-relatedfocus on quality of work products and deliverablesProductivity-relatedProduction of work-products related to effort expendedStatistical SQA dataerror categorization & analysisDefect removal efficiencypropagation of errors from process activity to activityReuse dataThe number of components produced and their degree of reusabilityWithin a single project this can also be a “project metric”. Across projects this is a “process metric”.CorrectnessMaintainabilityIntegrityUsabilityEarned Value AnalysisDefects found in this stageThis Stage + Next StageSeverity of errors (1-5)MTTF (Mean time to failure)MTTR (Mean time to repair)SQA - types of errors (0-5), MTTF (failure), MTTR (Repair),Quality - Correctness-adherence to rqmts, Maintainability-easy to fix?, Integrity-attack vulnerability, Usability-training time, number of screens, etc…Coming up: Can you calculate a metric that records the number of ‘e’ that appear in a program? A. Yes B. No
15 Can you calculate a metric that records the number of ‘e’ that appear in a program? A. Yes B. No Should you calculate the number of ‘e’ in a program?A. YesB. NoComing up: Effective Metrics (ch 16)
16 Effective Metrics (ch 16) Simple and computableEmpirically and intuitively persuasiveConsistent and objectiveConsistent in use of units and dimensionsProgramming language independentShould be actionablePersuasive - Be what you would naturally think about a metricComing up: Effective Metrics- Baseball On Base Percentage
17 Effective Metrics- Baseball On Base Percentage Simple and computableEmpirically and intuitively persuasiveConsistent and objectiveConsistent in use of units and dimensionsProgramming language independentShould be actionableNot effective for “general audience”Persuasive - Be what you would naturally think about a metricComing up: Effective Metrics- Baseball Runs Batted In
18 Effective Metrics- Baseball Runs Batted In Count of times when the outcome of player’s at-bat results in a run being scoredSimple and computableEmpirically and intuitively persuasiveConsistent and objectiveConsistent in use of units and dimensionsProgramming language independentShould be actionableMuch more effective for “general audience”Persuasive - Be what you would naturally think about a metricComing up: Actionable Metrics
19 Actionable MetricsActionable metrics (or information in general) are metrics that guide change or decisions about somethingActionable: Measure the amount of human effort versus use cases completed.Too high: more training, more design, etc…Very low: maybe we can shorten the scheduleNot-Actionable: Measure the number of times the letter “e” appears in codeThink before you measure. Don’t waste people’s time!Coming up: Project Metrics
20 Project Metricsused to minimize the development schedule by making the adjustments necessary to avoid delays and mitigate potential problems and risksused to assess product quality on an ongoing basis and, when necessary, modify the technical approach to improve quality.every project should measure:Inputs —measures of the resources (e.g., people, tools) required to do the work.Outputs —measures of the deliverables or work products created during the software engineering process.Results —measures that indicate the effectiveness of the deliverables.Coming up: Typical Project Metrics
21 Typical Project Metrics Effort/time per software engineering taskErrors uncovered per review hourScheduled vs. actual milestone datesChanges (number) and their characteristicsDistribution of effort on software engineering tasksDist of effort on SWE == Effort on each phase in the lifecycle (then see where you have problems and correlate them)Actionable: What do you do if it’s too high/low?Coming up: Metrics Guidelines
22 Metrics Guidelines Same as process metrics guidelines Use common sense and organizational sensitivity when interpreting metrics data.Provide regular feedback to the individuals and teams who have worked to collect measures and metrics.Don’t use metrics to appraise individuals.Work with practitioners and teams to set clear goals and metrics that will be used to achieve them.Never use metrics to threaten individuals or teams.Metrics data that indicate a problem area should not be considered “negative.” These data are merely an indicator for process improvement.Don’t obsess on a single metric to the exclusion of other important metrics.Same as process metrics guidelinesComing up: Typical Size-Oriented Metrics
23 Typical Size-Oriented Metrics errors per KLOC (thousand lines of code)defects per KLOC$ per LOCpages of documentation per KLOCerrors per person-monthErrors per review hourLOC per person-month$ per page of documentationTTH Class got here.Coming up: Typical Function-Oriented Metrics
24 Typical Function-Oriented Metrics errors per Function Point (FP)defects per FP$ per FPpages of documentation per FPFP per person-monthComing up: But.. What is a Function Point?
25 But.. What is a Function Point? See Book section for more detailFunction points (FP) are a unit measure for software size developed at IBM in 1979 by Richard AlbrechtTo determine your number of FPs, you classify a system’s features into five classes:Transactions - External Inputs, External Outputs, External InquiresData storage - Internal Logical Files and External Interface FilesEach class is then weighted by complexity as low/average/highMultiplied by a value adjustment factor (determined by asking questions based on 14 system characteristicsEI - Info coming into the systemEO - provides derived info out of a system (use ILF and EIF to create information)ExtInq - provides non-derived information out of the system (echoes back EI)ILF - Think structures in RAM, datafiles ONLY updated based on EIExtIF - Think database, datafile updated by anythingComing up: But.. What is a Function Point?
26 But.. What is a Function Point? CountLowAverageHighTotalExternal Inputx3x4x6External Outputx5x7External InquiriesInternal Logic Filesx10x15External Interface FilesBe wary of statistics!Unadjusted Total:Value Adjustment Factor:Total Adjusted Value:Coming up: Function Point Categories
27 Function Point Categories External Inputs (EI) - Info coming into the systemExternal Outputs (EO) - provides derived info out of a system (use ILF and EIF to create information)External Inquiries - provides non-derived information out of the system (echoes back EI)Internal Logical Files (ILF) - Think structures in RAM, datafiles ONLY updated based on EIExternal Files - Think database, datafile updated by this code, but also other systemsComing up: Function Point Example
28 Function Point Example Coming up: Comparing LOC and FP
29 Comparing LOC and FP Representative values developed by QSM Coming up: At IBM in the 70s or 80s (I don’t remember) they paid people per line-of-code they wrote
30 At IBM in the 70s or 80s (I don’t remember) they paid people per line-of-code they wrote What happened?A. The best programmers got paid the mostB. The worst programmers got paid the mostC. The sneakiest programmers, got paid the mostD. The lawyers got paid the mostComing up: Why opt against LOC?
31 Why opt against LOC? LOC is not programming language independent LOC cannot use readily countable characteristics that are determined early in the software processLOC “penalizes” inventive (short) implementations that use fewer LOC that other more clumsy versionsOther metrics makes it easier to measure the impact of reusable components (screen, widgets, etc…)Other options: COCOMO, Planning Poker, SLIM, Story Points (remember Scrum?), many others…Coming up: Object-Oriented Metrics
32 Object-Oriented Metrics Number of scenario scripts (use-cases)Number of support classes (required to implement the system but are not immediately related to the problem domain)Average number of support classes per key class (analysis class)Number of subsystems (an aggregation ofclasses that support a function that is visible tothe end-user of a system)Coming up: WebEngineering Project Metrics
33 WebEngineering Project Metrics Number of static Web pages (the end-user has no control over the content displayed on the page)Number of dynamic Web pages (end-user actions result in customized content displayed on the page)Number of internal page links (internal page links are pointers that provide a hyperlink to some other Web page within the WebApp)Number of persistent data objectsNumber of external systems interfacedNumber of static content objectsNumber of dynamic content objectsNumber of executable functionsGot here April 13, 2010Coming up: Measuring Quality
34 Measuring QualityCorrectness — the degree to which a program operates according to specificationMaintainability—the degree to which a program is amenable to changeIntegrity—the degree to which a program is impervious to outside attackUsability—the degree to which a program is easy to useVerified non-conformancewith reqmtsKLOCMTTCMean time to change:time to analyze, design,implement and deploya changet=threat probabilitys=security = likelihood of repelling attackIntegrity = 1-(threat*(1-security)) E.g. t=0.25, s= > I=0.99Correctness - number ofMaintainability - MTTC (meant time to change) - given an incoming change req, what is time to design, implement and test the changeIntegrity – T=Liklihood of threat occuring, S = likelihood of repelling the attackMany options. See ch 12Coming up: Defect Removal Efficiency
35 Defect Removal Efficiency DRE = E /(E + D)E is the number of errors found before delivery of the software to the end-userD is the number of defects found after delivery.Provides a measure of how well your team removed defects during a phase. If that DRE is 33%, you found 33% of the defects that you should have found.Coming up: Defect Removal Efficiency
36 Defect Removal Efficiency DRE = E /(E + D)Defects found during phase:Requirements (10)Design (20)ConstructionImplementation (5)Unit Testing (50)TestingIntegration Testing (100)System Testing (250)Acceptance Testing (5)By Customer (10)10 / ( ) = 33%What are the rest?10 / ( ) = 33%20 / ( ) = 28%5 / (5 + 50) = 9%50 / ( ) = 33%100 / ( ) = 28%250 / ( ) = 98%5 / (5 + 10) = 33%Coming up: Metrics for Small Organizations
37 Metrics for Small Organizations time (hours or days) elapsed from the time a request is made until evaluation is complete, tqueue.effort (person-hours) to perform the evaluation, Weval.time (hours or days) elapsed from completion of evaluation to assignment of change order to personnel, teval.effort (person-hours) required to make the change, Wchange.time required (hours or days) to make the change, tchange.errors uncovered during work to make change, Echange.defects uncovered after change is released to the customer base, Dchange.Coming up: Establishing a Metrics Program
38 Establishing a Metrics Program Set GoalsIdentify your business goals.Identify what you want to know or learn.Identify your subgoals.Identify the entities and attributes related to your subgoals.Formalize your measurement goals.Determine indicators for goalsIdentify quantifiable questions and the related indicators that you will use to help you achieve your measurement goals.Identify the data elements that you will collect to construct the indicators that help answer your questions.Define MeasurementsDefine the measures to be used, and make these definitions operational.Identify the actions that you will take to implement the measures.Prepare a plan for implementing the measures.Coming up: Metrics give you information!
39 Metrics give you information! Metrics about your process help you determine if you need to make changes or if your process is workingMetrics about your project do they same thingMetrics about your software can help you understand it better, and see where possible problems may lurk. Let’s see the complexity measurement (after a few questions…)Coming up: Questions
40 QuestionsWhat are some reasons NOT to use lines of code to measure size?What do you expect the DRE rate will be for the implementation (or construction) phase of the software lifecycle?What about for testing?Give an example of a usability metric?According to the chart, Smalltalk is much more efficient than Java and C++. Why don’t we use it for everything?Coming up: References
41 Code MetricsPreviously we have been discussing product and process metrics.However, code metrics are used to make your code betterComplexity - See CyclomaticComplexity slidesDynamic measurements – Profile JkaboomMemory – garbage collector, object crerationsCPU performanceThreadingComing up: Questions
42 References http://www.lansing.lib.il.us/images/baseball_player.gif End of presentation