1. Value Based Software Engineering
What, Why, How & The Mysterious
Nupul Kukreja
SoCal SPIN
May 6 ‘11

2. About Me
Pursuing PhD in Computer Science under Dr. Barry Boehm. Focus Area: Value Based Software Engineering (VBSE)
MS in CS from USC and BE in CS from Mumbai University, India
Worked as a Software Developer at Capgemini Consulting India Pvt. Ltd.
Lecturer for CS Undergraduates at Watumull Institute (Mumbai University) teaching Computer Graphics and Operating Systems
10/6/2010
© USC-CSSE

3. Agenda Section 1* : Overview
Understanding “Theory W”
VBSE’s 4+1 Theory
VBSE Agenda
VBSE: 7 Key Elements (Overview)
Section 2: Applying 7 Key Elements in practice
Identifying company’s strategic goals and their overall value
Benefits Realization Analysis
Measurements/Estimation and Value of Information
Creating a Value Based Business Case
Elicitation/Reconciliation of Stakeholder Win Conditions
Prioritization of Win Conditions/Requirements using Kano Analysis
Tracking Value Earned
*Content courtesy: Dr. Barry Boehm
10/6/2010
© USC-CSSE

4. Tools & Techniques Involved
Application of VBSE in practice involves understanding practices from various domains:
Decision Theory
Statistics
Management/People Engineering
Software Engineering Principles
We shall see how to effectively combine them to help practice VBSE with minimal overhead
10/6/2010
© USC-CSSE

5. Theory W*: Enterprise Success Theorem – An informal proof
Theorem: “Your enterprise will succeed if and only if it makes winners of your success-critical stakeholders”
Proof of “if”: Everyone that counts is a winner. Nobody significant is left to complain.
Proof of “only if”:
Nobody wants to lose. Prospective losers will refuse to participate, or will counterattack. The usual result is lose-lose.
So, we’ll start right away with something that we call the Enterprise Success Theorem. So if you are a manager of a project or a product line and you want to be successful, you’d like to know what it is that will make you successful. And what we’ve found on lots and lots of projects is that this theorem tends to be true: That your project or product line will succeed if, and only if, it makes winners of your Success-Critical Stakeholders. And a little bit later we’ll go into techniques for helping you determine who your Success-Critical Stakeholders are. But, again, we can go through the proof of this. Suppose you’ve made winners of your Success-Critical Stakeholders. Is there any way your project can fail? Well, who’s left to complain? There’s nobody left to complain unless they are computer scientists who don’t like your assertions, or something like that, and if they’re not success-critical, then you still succeed. So, basically, that’s the proof of “if.”
The proof of “only if” is a little more complicated. Suppose that you have made winners of all your Success-Critical Stakeholders, except one. So, this might be your users, it might be your maintainers, it might be a COTS vendor that you’re depending on, or a strategic partner, or a venture capitalist. In general, whoever it is, is not going to want to lose. So, frequently, the losers will refuse to participate. They won’t put up their venture capital; they won’t use your product. Or, they will counter-attack and try to collude with one of the other stakeholders and make somebody else a loser. Usually, what happens in those cases is that everybody loses and the project gets cancelled and nobody ends up winning. So, you would like to make winners of your Success-Critical Stakeholders, which in some ways seems sort of like a tautology--that, certainly, “this” and “this” seem to be sort of equivalent. But if you parse what it means to make winners of your Success-Critical Stakeholders, you find that there are some significant, useful implications there.
*An Initial Theory of VBSE – Barry Boehm & Apurva Jain
10/6/2010
© USC-CSSE

6. Theory W: WinWin Achievement Theorem
Making winners of your success-critical
stakeholders (SCSs) requires:
Identifying all of the SCSs
Understanding how the SCSs want to win
Having the SCSs negotiate a win-win set of product and process plans
Controlling progress toward SCS win-win realization, including adaptation to change.
Which is reflected in the next theorem, which is the Win-Win Achievement Theorem, which says that to make winners of your Success-Critical Stakeholders requires, for one, to identify all of the ones that are success-critical. So, again, if you don’t identify them, then you won’t involve them, and it’s likely they’ll turn out losers, and it’s likely they’ll refuse to participate and you won’t succeed. Once you’ve identified them, it’s important to understand how they want to win. So, again, you may be a computer scientist or a programmer and how you want to win is to build a programmer-friendly user interface. But if you’re building a system for doctors, they probably won’t like a programmer-friendly user interface, so you may need to do a bunch of prototyping to see what kind of user interface the doctors really want. So, again, this requires some investment into learning what’s important to them. Once you do that, frequently, as you will see in some of the charts to follow, you’ll find that the stakeholders have win conditions that conflict with each other. So the users will want lots and lots of features, the customers will only have a finite amount of resources, which may not be enough to build all the features, so, there’s a conflict. You may contract with a developer who has a way to build the features, but the features are not compatible with (yours). Their infrastructure is not compatible with your computing facility. So, again, they think they are winning, but they are building something that doesn’t work in your facility, and everybody turns out to lose.
So, frequently, what you have to do is get all the stakeholders to understand each other’s win conditions and determine where are their conflicts, and if there are conflicts to determine, there’s a technique that’s called “Inventing Options for Mutual Gain,” that says, well, if we have too many features maybe we can win by prioritizing the features and building this incrementally. So, we’ll build what’s the core capability that you want, within the available resources, and based on that, if that succeeds, then more resources will become available and you can get some of the rest of your features.
So, again, this is something that requires a lot of mutual understanding and creativity and innovation in determining how to invent these options for mutual gain. What comes out of that, then, is a set of product plans and process plans, and in the old days that was enough. Basically, it used to be that the computing field was relatively stable, and once you had some requirements, you could spend three years building the system to requirements, and everybody would pick it up and use it. Nowadays, if what you deliver is what you planned to deliver three years ago, you’re at least eighteen months out-of-date. So one of the things you need to do is, also, to control progress toward this win-win realization and find out when is the COTS product you were betting on becoming obsolete and unsupported, and you need to go to another one, or go to a new release, or adjust your previous plans and specifications to address succeeding under these new changes.
So, that’s basically what the Achievement Theorem provides to you. So, it turns out that you can configure this.
10/6/2010
© USC-CSSE

7. Initial VBSE Theory: 4+1
Engine: Theory W (stakeholder win-win): What values are important?
Enterprise Success Theorem
Theory of Justice
Win-Win Equilibrium and Negotiation
Four Supporting Theories
Dependency Theory: How do dependencies affect value realization?
Results chains; value chains; cost/schedule/performance tradeoffs
Utility Theory: How important are the values?
Multi-attribute utility; Maslow’s need hierarchy
Decision Theory: How do values determine decisions?
Investment theory; game theory; statistical decision theory
Control Theory: How to monitor and control value realization?
Feedback control; adaptive control; spiral risk control
Well, here’s another way of looking at it, that says, “what does this theory involve?” First, it has a sort of an engine, which is the Stakeholder Win Win Theorem, that says, what really matters is what values are important. So, again, the success theorem tells you a lot about that. If people’s values include being fair to all of the participants, then things like the Theory of Justice and Fairness becomes important. Another thing that becomes important is determining: “When are you in a win-win state?” And we have a slide (in a previous lecture) that shows what it means to be in a Win-Win Equilibrium, and what to do if you get out of equilibrium.
Again, the Enterprise Success Achievement Theorem says (the) first thing you need to do is to say: “Who does success depend on?” So this gets into something called Dependency Theory, that says, “how do dependencies effect the realization of value?” Who do you depend on, and what do you need to promise to them, to make them collaborate and cooperate and get you what you want? And things like results chains and value chains help you determine that. In other cases, what you want to do is, to do various kind of business cases and look at trade-offs between cost and schedule and performance to determine: “How does the cost of something depend on the architecture, and the re-use products, and the capabilities of the people?” So eventually we’ll talk about models that estimate cost and schedule and also see how those trade-off with performance.
In determining how important the values are, this depends a lot on Utility Theory, and since there are multiple stakeholders, the users want usability, they want lots of features, they want instant response time. The acquirers have different attributes, they want something that is very controllable, that can be done within the available resources, and is relatively stable. Users want to change their mind any time they see a new attractive feature. Another thing that is important is that, in some cases, there’s something called a “mass-load-need hierarchy,” that says, once you’ve satisfied a particular low-level need, it’s not a motivator any more. So, once you get people a basic set of infrastructure, they don’t want any more infrastructure, they want applications that will add value and give them competitive advantages and things like that. But if you try to do the application features before you have the infrastructure, you’ll find the people are unsatisfied because you don’t have something that you’re building on top of that’s reliable.
Another theory that’s important is Decision Theory, which basically is what helps people generate options and evaluate them, and determine when the options provide a mutual gain. So, Investment Theory, Game Theory, Statistical Decision Theory, because a lot of things are uncertain, help you do this. What helps you adapt to change is basically Control Theory. Feedback control is basically saying how do you control within what you started with as a set of goals and milestones. As things change, then you’d need to change the plans and the milestones and adapt the system to your new objectives. And, again, one of the things we find is important, in Spiral development or Incremental Commitment, is assessing risks as they emerge and making sure that they are under control.
10/6/2010

8. VBSE Theory 4+1 Structure
Slide has hidden content. Please view it as a slideshow
So, let’s go on from this, and see how this 4+1 Theory translates into a sequence of things that you can do to achieve success on the project. So, you start by saying: “Here I am, I’m trying to start a project and I need to make it succeed, and I know I need to make winners out of my Success-Critical Stakeholders.” I need to find out: Who are they, who am I depending on? So Dependency Theory helps you to determine: How do these dependencies effect realizing your values? Do you need some strategic partners? Do you need some more marketing to determine what the customers really want? Do you need a business case that says how much investment you need to make, and how much capital that’s going to require, and things like that?
Again, as you find out who it is you’re depending on, you need to find out what is important to them, and what will make the strategic partners interested in making you a winner. So, again, if you choose a supplier who is your low bidder, frequently they’re going to be more interested in putting together something that satisfies the contract at minimum cost and it may not be very useable and maintainable, and things like that. So, you might want to look at what would incentivize them to be more cooperative as suppliers, and giving them a share of the profits might be a way of getting them to, not only build something, but build something that is easy to maintain and use and build upon.
Once you know what people want, basically you need to find out where the conflicts are and determine how to resolve them. Once you’ve done that, then you need to put the project in place and start executing the negotiated plans and resources and specifications, and monitor what’s happening with your environment, what’s happening with the marketplace, what’s happening with technology, and adaptively control your original plans so that it continues to track what it means to succeed.
10/6/2010

9. Initial VBSE Theory: 4+1 Process – With a great deal of concurrency and backtracking
So, what this works out to, in terms to a particular set of steps, one thing the theory tells you is how to get something started. And, fundamentally, no project gets started unless there is some protagonist who is sufficiently unhappy with the current state of the world, that it needs some effort to change it. So, this might be a CEO, who says: “Here’s something that is keeping us from increasing our market share and we need to do something about it.” It may be an innovated person in your organization, who has a neat idea like Google Maps, or something like that, and says, “we aught to do this because I think it will generate a lot of value for us downstream.” And, again, the inventor doesn’t have a lot of resources, so he’s got to convince other people that he depends on that supply the resources, supply a number of the underlying technologies and things like that, to enable him to go forward and to create something that will add value to the company. So once the protagonist determines, “who does success depend on?” that he can’t supply himself or she can’t supply herself, then they need to go and explore various pathways to success.
And one of the pathways to success that we’ll look at is something called a Results Chain, that says: “What initiatives are needed besides building software to make this thing succeeding?” So, in some cases, it might be migrating databases, it may be training users, it may be creating strategic partnerships. So, again, we’ll look at what kind of things are important there.
Once you’ve determined who your stakeholders are that you’re depending on, then you need to go off to Utility Theory and find out: What are their value propositions or their win conditions? And, again, frequently what you’ll find… One of the cases we’ll look at is a system where the users wanted three-and-a-half-million-lines-of-code worth of features. And, the acquirers only had twenty-two million dollars, and three-and-a-half-million-lines-of-code worth of features is not buildable within twenty-two million dollars. And so, in a lot of cases, what you need to do is something called Expectations Management that says, “let’s put this into a well-calibrated cost-estimation model and see how much software can be built for twenty-two million dollars,” and, basically, saying, “if it’s only two-hundred-and-fifty (thousand) lines of code, then let’s prioritize what you want, and build the most-important two-hundred-and-fifty-K worth of things that you want or need, and we’ll go from there, once we have that.”
So, Expectations Management is something that requires people to be confronted with objective representations of reality, such as cost models, or performance models, or laws, or other constraints on success. So, again, once you’ve done this, you’re in better shape to negotiate a win-win agreement among your Success-Critical Stakeholders and decide on a win-win set of specifications and plans for the system you want to build. Once you have that you can go to Step Six, which basically says, “execute the plans and monitor and control your progress with respect to plans, and monitor and control what is happening to the world outside, in terms of, if we build this to what we originally planned, is it going to give us the benefits we thought?” If not, then you need to do some adaptation to change, which frequently will involve a fair amount of backtracking.
And, I should have mentioned, a lot of these things are going to require a fair amount of backtracking. So once you get to Decision Theory and find that you’re short of a win-win, and you start inventing options, you need to evaluate the options which take you back to Dependency Theory, that says, “how much are we going to save in costs, if we decide to reuse some software, even though it’s not perfect, or endorse/adopt a COTS product, even though it’s not perfect?” So, again, this will require a lot of cost-schedule-performance trade-offs that are going on after you get into the middle of Decision Theory. So, there is a bunch of backtracking that can go on, there’s a lot of concurrency that is going on that some people will be negotiating, while other people are doing analysis, and so forth. And certainly when you get down here and you see that some new technology is coming along or your competitors have a feature that is going to out-do you in the marketplace, then you need to rebaseline what you are doing and, again, that will require a lot of backtracking into some of these other things. It might involve finding another strategic partner, and finding out how they want to win, and rebaselining a lot of that. So, again, this thing looks a little more simple than it is, and again there’s a lot of judgment involved in navigating it.
10/6/2010

10. VBSE Agenda
Objective: Integrating value considerations into the full range of existing & emerging Software Engineering principles in a manner so that they ‘compatibly’ reinforce one another
Major Elements:
VB* Requirements Engineering
VB Architecting
VB Design and Development
VB Verification And Validation
VB Planning and Control
VB Risk Management
VB Quality Management
VB People Management
*VB = Value-Based
10/6/2010
© USC-CSSE

11. VBSE: Seven Key Elements
Benefits Realization Analysis
Stakeholder Value Proposition Elicitation & Reconciliation
Business Case Analysis
Continuous Risk and Opportunity Management
Concurrent System & Software Engineering
Value-Based Monitoring & Control
Change as Opportunity
10/6/2010
© USC-CSSE

12. 1. Benefits Realization Analysis
DMR/BRA* Results Chain
Assumption(s):
-Order to delivery time is
an important buying criterion
Accountable Stakeholder(s)
OUTCOME
Reduced order processing cycle
(intermediate outcome)
OUTCOME
Increased sales
INITIATIVE
Implement a new order
entry system
Contribution
Reduce time to process order
Contribution
Reduce time to deliver product
*DMR Consulting Group’s Benefits Realization Approach
© USC-CSSE

13. Key ‘Benefits’ of Doing BRA/Results Chain
Explicitly maps out the intended benefits of the system to be acquired (or any initiative to be undertaken)
Identifies what all initiatives need to be taken to actually realize the benefits (A.K.A. Work Breakdown Structure (WBS) at times i.e., “within” the initiatives)
Identifies the ‘key stakeholders’ accountable for the above initiatives
NOTE:
Must be refined as more is learnt about the initiatives or an unforeseen benefit is uncovered
Done at multiple levels of granularity and stabilizes earlier into the SDLC than most artifacts
Subtle but valuable: Lays foundation for the relevant metrics required to ‘track’ the benefits
10/6/2010
© USC-CSSE

14. 2. Stakeholder Value Proposition Elicitation & Reconciliation
Myth: ALL SCSs have readily expressible and compatible value propositions that can be easily turned into a set of objects for each initiative and the overall ‘portfolio’ of initiatives
The following figure is also known as a “Model Clash”
10/6/2010
© USC-CSSE

15. Techniques
There are a myriad techniques that can be applied for stakeholder value proposition reconciliation:
Expectations Management: Just awareness of potential conflicts could help SCSs relax their less critical levels of desire
Visualization & Trade-off Analysis Techniques: Prototyping, scenarios, estimation models help SCSs to mutually understand most important & achievable aspects of the system
Prioritization: Helps identify which combination of capabilities best satisfies the stakeholders’ most critical needs within available resource constraints
Groupware: Some groupware tools have support for collaborative brainstorming, discussions, prioritization etc.,
10/6/2010
© USC-CSSE

16. 3. Business Case Analysis (BCA)
Another extremely valuable technique for stakeholder value proposition reconciliation
Helps determine where is the best bang for the buck – i.e., capabilities providing the best ROI (We’ll see how to create a VB-Business Case later)
Can directly influence capability prioritization
TWO Major factors influencing BCA
Unquantifiable benefits (a.k.a. intangibles)
Uncertainties & risk
10/6/2010
© USC-CSSE

17. Techniques
ROI – most commonly used to justify a business case. Difficult to apply for intangibles
Deriving variables/attributes from an objectives hierarchy – closer to value based but we can derive them straight from the benefits chain!
Each of the variables need to be ‘estimated’ either using subjective judgment, past data, expert advice…
…OR we could ‘calibrate’ the estimator to provide better estimates!
We’ll see how to do this in Section 2
10/6/2010
© USC-CSSE

18. 4. Continuous Risk & Opportunity Management
Understanding & Addressing People’s Utility Functions
Risk Averse
Risk Neutral
Risk Seeking
Implication(s):
Very people oriented process
Continuous and Iterative
Must have plans/processes in place to identify, manage and mitigate risks
1,1
Utility
It is possible to have negative utilities too  losses
0,0
Benefit
10/6/2010
© USC-CSSE

19. Central Tenet of Risk Management RE = Probability (Loss) * Size (Loss)
Metric Risk Exposure:
RE = Probability (Loss) * Size (Loss)
“Loss” of stakeholders’ value
Ex.: profits, reputation, quality of life etc.,
Prioritizing Risks using Risk Exposure (may be misleading!)
Check
Utility -
Loss Estimate
High
Major
Risk
Risk Probability
Check
Probability
Estimate
Little
Risk
Low
Low
Loss of Utility
High
10/6/2010
© USC-CSSE

20. 5. Concurrent System & Software Engineering
Increasing pace of change in IT marketplace  traditional sequential approach to software development (i.e., Waterfall model) is increasingly risky to use!
Preferable: Concurrently engineer the product’s operational concept, requirements, architecture, life cycle plans and key sections of code
Concurrent engineering further preferable if:
Software system requirements emergent from usage and/or prototyping rather than prespecifiable
Costs, benefits & risks of COTS software and/or outsourcing decisions affects requirements, architectures, cost, schedules etc.,
10/6/2010
© USC-CSSE

21. Choose Relevant Process Models
Anchor point milestones – of the chosen process model
Lifecycle Objective (LCO)
Lifecycle Architecture (LCA)
Initial Operational Capability (IOC)
For at least one architecture, a system built to that architecture will:
For a specific detailed architecture a system built to that architecture will:
An implemented architecture, an operational system that has:
Support core Operational concept
Satisfy core requirements
Be faithful to prototype(s)
Buildable in budget/schedule in the plan
Show a viable Business case
Have key SCSs committed to support the ‘next phase’
Support elaborated operational concept
Satisfy elaborated requirements
Be faithful to prototype(s)
Buildable in budget/schedules in the plan
Have key SCSs committed to support the full lifecycle
All Major risks resolved/covered by a risk management plan
Realized the operational concept
Implemented the initial operational requirements
Prepared a system operation & support plan
Identified the initial site(s) of deployment for initial transition
Identified users, operators & maintainers to assume operational roles
This slide has hidden content. Please view it as a slide show.
Milestone Pass/Fail Criteria an important aspect of concurrent process models (a.k.a. anchor point milestones)
LCO = Life cycle objectives
LCA = Life cycle Architecture
IOC = Initial Operational Capability

22. Techniques
There are a myriad software process models – choose the one that fits best to your organization
Feasibility evidence MUST be provided at each milestone to support the claims as shown on the previous slide
10/6/2010
© USC-CSSE

23. 6. Value-Based Monitoring & Control
Use project’s business case for monitoring the actual business value of the capabilities to be delivered
YES
Develop/update business case; time phased cost; benefit flows; plans
Value being realized?
YES
Perform to plans
Assumptions still valid? NO NO
Determine Corrective Actions
10/6/2010
© USC-CSSE

24. Techniques
Forms the bulk of the ‘detailed’ section on VBSE in Practice
What metrics to use? Are those the right metrics?
Are we gaining benefits? Are we doing the right things and doing them correctly?
We shall answer these questions in Section 2.
10/6/2010
© USC-CSSE

25. Opportunities can become risks if nothing is done about it!
7. Change as Opportunity
Today’s world – a not so good Present
Expending resources to adapt to change is often stated as “rework costs”
Changes are treated as defects in change tracking systems
The real world:
Continuous ongoing changes in
Technology
Marketplace
Organizational
Stakeholders’ value propositions and priorities
Opportunities can become risks if nothing is done about it!
10/6/2010
© USC-CSSE

26. A Brave New World
Organizations that can adapt to change more rapidly will succeed better in the their mission
Developing change anticipatory modular design can be considered a good investment to enhance system value in the future
Two techniques for enhancing adaptability to change:
Architecture based
Refactoring based
Example of “Change as Opportunity”
Internet and the Web and their effect on ecommerce

27. Section 2: Applying VBSE in Practice
10/6/2010
© USC-CSSE

28. Agenda Identify the need
Identify the motive/goal/objective of the need to be satisfied
Check alignment with company/organization’s goals and objectives:
Find/Derive the ‘value’ of the initiative
Identify initiatives to realize value
Identify validate/assumptions when articulating value
Identify success critical stakeholders (SCSs)
Elicit value propositions (win conditions) of SCSs
Create a value-based business case with information from above and check for business alignment
‘Percolate’ the value to all phases of the SDLC (software development lifecycle)
Track/adjust the business case over time to ascertain value gained/created and validate assumptions

29. Company Objectives
Before we start – we MUST know on what basis does the organization decide to pursue any endeavor
Profit/Revenue is not the only variable…
…Competitive advantage, Brand projection, Enhanced customer loyalty etc., to name a few
The current undertaking must align with the strategic goals of the company
10/6/2010
© USC-CSSE

30. Ranking of Strategic Goals
Say you have two programs each satisfying (or satisficing) the organization’s goals in varying degrees, which program should you choose?
Layout the full ‘results chain’ and see what benefits can be gained – disadvantage? There is no inherent ranking
Use Multi-Attribute Utility Theory MAUT (could also use Analytical Hierarchy Process (AHP) but we’ll focus on the former)
10/6/2010
© USC-CSSE

31. WHY MAUT?? Mathematically sound
Easier to capture the risk attitude in the form of utility functions
Strategic goals don’t tend to have too many attributes and thus attribute space is well managed and easily understood
Easier to lay coarse grained benchmarks for various attributes
10/6/2010
© USC-CSSE

32. MAUT Example Let’s capture a single utility function for Revenue
The more revenue the better – increasing utility
Gamble
Sure thing:
Best: $10M_
_____
Worst: $0.5M
Best: $10M
50%
Worst: $0.5M
For which value of the sure thing are you indifferent between the sure thing and the gamble? _______
10/6/2010
© USC-CSSE

33. Graphing Utility Utility Risk Averse Risk Seeking Revenue (1, $10M)
0.5
0.75
Risk Averse
$1.5M
$3M
$4.7M
$7.5M
$8.1M
$8.6M
Risk Seeking
0.25
The out put of this slide is based as per the input of someone from the audience who answered the question(s) on the previous slide. The utility function would be piecewise linear. The previous slide would give ONLY the 50th percentile utility. To know the 75th percentile you change the sure thing’s worst limit to that of 50th percentile and for the 25th Percentile change sure thing’s best limit to that of 50th percentile and repeat the process. 5 point utility is enough for even the most critical problems. Can go further if need to, but wouldn’t be all that valuable. These are preferential approximations any which ways.
You either get a risk seeking or a
10/6/2010
© USC-CSSE

34. Finding Utility
Q: Given a value of the ‘attribute’ what is its utility? How do we find that from the graph?
This second step involves understanding the ranking of the attributes:
“If you can change one attribute from its worst state to the best state which one would you choose? Ties are allowed.”
Repeat this question till all attributes are ranked
For each of the ranked attributes repeat the Gamble vs Sure thing lottery to find the ‘P’ i.e., probability at which you’ll be indifferent!
10/6/2010
© USC-CSSE

35. Calculating the Attribute Tradeoff Scaling Constant (ki)
Gamble
Sure thing:
Reference System:
All attributes at worst state
Chosen attribute at best state
100%
_____ 0%
P =
Best System All attributes at best state P
1-P
Worst System All attributes at worst state
There is no 75th or 25th percentile in this case. Done only once to find the indifference point
For what value of ‘P’ are you indifferent between the sure thing and the gamble? ________
10/6/2010
© USC-CSSE

36. Finding Utility (Finally!)
10/6/2010
© USC-CSSE

37. MAUT (Cont’d) Advantages:
Simple question answering and choosing the outcomes of a lottery (sure thing vs. gamble)
Can easily be ‘computerized’
Understanding a lottery comes naturally to most people and the risk attitude is hidden within the choices
Difficult to ‘game’ the system with false attitudes
Disadvantages:
May be non-intuitive to start with
Time consuming to get 5 point utility functions and attribute tradeoff scaling constants
Sometimes it’s just difficult to get utility functions (Arrow’s impossibility theorem)
10/6/2010
© USC-CSSE

38. Next: Benefits Analysis
Understanding the Value of ‘what you are doing’
10/6/2010
© USC-CSSE

39. Before We Begin…
The previous application of MAUT is optional if your organization already has prioritized it’s strategic goals and objectives (continually updated)
Still may be valuable to ‘mathematically’ affirm the choices!
There are a myriad of methods available to help you with this: Balanced Score Cards, AHP etc., They are more ‘management’ i.e., no mathematical basis
Ideally before using any technique you should ask the ‘value’ of that technique. You can apply VBSE principles from the benefits analysis process to determine the value of the technique or do a simple value of information analysis. Ex.: If you knew the future what would be the ‘value’ of your decision. Subtract from it the best guesstimate you’ve come up with using your analysis – that’s the theoretical maximum you should be willing to pay for the extra information. It is also known as Expected Value of Perfect Information (EVPI). More commonly though it’s EVSI (Expected Value of Sampled Information).
10/6/2010
© USC-CSSE

40. Approach
Use a live example from anyone in the audience (could be any example even something you are currently working on) or use a case study example of Sierra Mountain Bikes (SMB) (Former preferable)
Walk through the steps of applying VBSE
ONE such way of applying VBSE
Has research viewpoints
Open to discussion
Combines techniques from Statistical Decision Theory, Management and Software Engineering
*Taken from: An Initial Theory of VBSE – Barry Boehm & Apurva Jain
10/6/2010
© USC-CSSE

41. SMB – Problem Description
Susan Swanson, CEO of SMB when developing a shared constructive vision of SMB’s primary opportunities and problems, determined a significant opportunity of growth, since they produced top quality bikes at a competitive price. Major problem area: old manual order processing system.
Distributors, retailers and customers were dissatisfied with high rates of late/wrong deliveries, poor synchronization between order entry, confirmation & fulfillment; and disorganized responses to problem solutions.
It was decided to outsource the development of the new system, since it wasn’t their area of expertise.
10/6/2010
© USC-CSSE

42. What all to do to realize the benefits/ outcomes??
The initiative identified by SMB is to ‘implement a new order entry system’ to get rid of the manual order processing
Assumptions: - Increasing market size
Continuing consumer satisfaction with product
Relatively stable e-commerce infrastructure
Continued high staff performance
Distributors, retailers, customers
New Order Fulfillment processes, outreach, training
New Order- Fulfillment System
Safety, fair ness of inputs
Inter - operability inputs
Increased customer satisfaction, decreased operation costs
Less time, fewer errors in order processing
Increased profit, growth
Lesser time, fewer errors/order fulfillment system
Faster, better order fulfillment system
Increased sales, profitability, customer satisfaction
New Order- Entry System
Developers
What all to do to realize the benefits/ outcomes??
Faster order entry steps, fewer errors
On-time assembly
Under what assumptions are these benefits realizable?
New order entry processes, training and outreach
SO WHAT?
Improved supplier coordination
What are the ‘contributions’ of the initiatives?
Who are the accountable stakeholders to help realize the benefits?
Suppliers
Sales personnel, distributors
10/6/2010
© USC-CSSE

43. Benefits Analysis
The previous slide lays out the expected benefits of having a new order entry/fulfillment system
The ‘initiative’ of having a new system is linked to the final outcome ‘increased profitability/growth’ (strategic goal?) through intermediate outcomes!
Each initiative must be followed by an outcome or to realize the benefits of an outcome some initiative(s) need to be taken (may not be so for the final outcome(s))
Key question to ask at each stage: “So what?” to get the subsequent outcome/initative.
Stakeholders who are/would be accountable for realizing/reporting the benefits should be identified
10/6/2010
© USC-CSSE

44. Key Questions for performing Benefits Analysis
Q1: Why do you want to know the value of “<initiative>”? A: You must know the why to know the purpose of performing the benefits analysis i.e., the value of knowing the value of the initiative  It’ll help understand the alignment with the organization’s strategic goals (will also help with identifying the need) Ex: “Why do you want to know the value of having a new order entry processing system?” A: So that the company has a viable case to know/believe if and whether a new system would/could increase productivity, growth, profit and customer satisfaction and by how much
10/6/2010
© USC-CSSE

45. Key Questions (Cont’d)
Q2: What IS the value of “<initiative>”? A: Perform the prior benefits analysis to see the value!! Q2.1: How do you define “value”? A: You just did! Questions 1 and 2 help answer this question itself. The “value” in this case is alignment with company goals and ‘how much’ does the said initiative align with it!! Q3: All this is good but ‘what exactly is the value’? What do I track? There are tons of intangibles to complicate the problem further?
10/6/2010
© USC-CSSE

46. Epiphany: Measurement!
WHAT to measure? We already have standard metrics that we track for every project
Are those ‘standard metrics’ really tracking overall project value or just some numbers?
What about intangibles? They are almost impossible to track at best. We have a 1-N rating system to help us with that
They may be hard, but not impossible!
Douglas Hubbard has even published a book “How to Measure Anything – Finding the Value of Intangibles in Business”
…Let’s see how we can use some of that wisdom to alleviate our measurement problem. But first…
10/6/2010
© USC-CSSE

47. WHAT to track/measure?
This is where we’ll build a value-based business case
Directly derivable from our prior benefits analysis!
Yes, we’ll also track/measure intangibles – probably won’t be a 1-N scale but actual numbers
Risks! We didn’t forget that. We’ll also see how to better identify ‘strategic risks’ and measure them too!
10/6/2010
© USC-CSSE

48. What all measurable items can we find from this diagram?
Safety, fair ness of inputs
Faster, better order fulfillment system
Increased sales, profitability, customer satisfaction
Faster order entry steps, fewer errors
On-time assembly
Inter - operability inputs
Lesser time, fewer errors/order fulfillment system
New Order- Entry System
Less time, fewer errors in order processing
Increased customer satisfaction, decreased operation costs
Increased profit, growth
New order entry processes, training and outreach
Improved supplier coordination
New Order- Fulfillment System
New Order Fulfillment processes, outreach, training
Distributors, retailers, customers
Developers
Suppliers
Sales personnel, distributors
Assumptions: - Increasing market size
Continuing consumer satisfaction with product
Relatively stable e-commerce infrastructure
Continued high staff performance
What all measurable items can we find from this diagram?
Something is missing…
For the system under development there could be various metrics to track. Those are very project specific. At this stage the metrics/outcomes that matter the most (to the developers) are client satisfaction, process improvement, resource use, work backlog etc., which MUST be taken into account when developing a ‘value model’
Associated costs of having a new system (make/buy/customize…)
Something is still missing…
10/6/2010
10/6/2010
© USC-CSSE
© USC-CSSE 48

49. But still…What exactly should be measured?
The previous exercise should promote the discussion of what is concretely meant by the terms – i.e., how can it be quantified?
But this is exactly the problem with measuring coarse grained criteria!
And that’s where we start to think
If you can observe a difference then it can probably be measured!
In fact the same benefits chain diagram can be used to simulate the discussion!!
Let’s see how…
REALLY HARD!
If there is an observable ‘delta’ difference between the current state and the future state then you can probably know what to look for to discern/quantify the difference! If you think hard you can probably come up the what to measure! Even if they are just proxies at least they’ll be relevant! Immeasurable wouldn’t be a valid excuse any more
10/6/2010
© USC-CSSE

50. Example: Growth Increased sales Increase profits Increase growth
Increase market share
Increase growth
Cost savings
You could augment this diagram with any other notations. However a key point to note is the possibility of ‘another’ stakeholder at this stage! For example marketing personnel wasn’t identified as a stakeholder in the benefits chain!! Now that you know you may have to measure it you may have to bring him/her on board – but must also incorporate their value propositions – we’ll see that in the later slides
SO WHAT?
This approach probably looks familiar – something from classical decision theory?
10/6/2010
© USC-CSSE

51. Objectives Hierarchy
You could replace the previous analysis with that of ‘objectives’ hierarchy
In fact you could replace the entire benefits chain analysis with an objectives hierarchy (You already know the top most goal “Why do you want to know the value of “_____”?)
You will however lose the connections between outcomes, stakeholders accountable, initiatives to be taken etc., of the benefits chain approach
Suggestion: You can use it at this stage when ‘drilling down’ the ambiguous metrics to more specific attributes! OR
Transform the benefits chain into an objectives hierarchy if you are comfortable with that!
10/6/2010
© USC-CSSE

52. How to Measure/Estimate
Once you’ve drilled down both, tangibles and intangibles to concretely observable (measurable) facts we can proceed with how best to estimate/measure them
The prior analysis from benefits chain all the way to measurable attributes should provide enough dimensions on which to capture the value of the project
But aren’t the estimates just guesstimates for the ‘value based business case’?
Yes, to some extent. Let’s see how to solidify these estimates…
10/6/2010
© USC-CSSE

53. That’s where we take Douglas Hubbard’s help. Remember him?
Are you 90% Confident?
When was the last time you were 100% confident with your estimates?
Were you confident 100% of the time ?
100% confident estimates 100% of the time would make you a priceless asset!!
Can you at least be 90% confident 90% of the time?
Wait a minute! How can you measure this confidence level??
That’s where we take Douglas Hubbard’s help. Remember him?
10/6/2010
© USC-CSSE

54. Getting a 90% Confidence Interval (CI)
Remember CIs from your statistics class?
A 90% CI is a range that has a 90% chance of containing the ‘correct answer’
Let’s try a simple test: (No need to Google )
Provide a range wide enough so that you believe there is a 90% chance that the value will be within this range (Units don’t matter. We are more interested in the range)
Question
Lower Bound
Upper Bound
What is the circumference of Mars?
_____________
10/6/2010
© USC-CSSE

55. Would you Gamble? Let’s say we gave you the following proposition:
You win $1000 if the actual answer is within your range
You win $1000 by spinning the wheel below:
What would you NOW CHOOSE?
Win $1000 $0
10%
90%
Actual circumference of Mars is about 21343km or miles
10/6/2010
© USC-CSSE

56. Choices & Inferences Choice Inference Gamble Your initial estimate
You think the gamble has a higher chance of payoff! Implies your 90% CI is NOT really your 90% CI but lower – 50%, 60%, 80%. Basically an overconfident estimate
Your initial estimate
This means that you think there is more than a 90% chance your range contains the answer! An underconfident estimate
Indifference
You are indifferent between your initial estimate and the gamble  an equivalent 90% chance  90% CI
This slide is animated…view it as slide show.
10/6/2010
© USC-CSSE

57. Sure Thing vs Gamble
If you are still with me you’ll realize that the previous exercise is similar to the one we did for capturing stakeholder utility!
We just replace the ‘sure thing’ by your estimate and the gamble is a 90% Win and 10% loss!
You ARE using utility theory concepts to make a 90% confident estimate that factors in uncertainty and probable prior knowledge of experts!
10/6/2010
© USC-CSSE

58. The Value Based Business Case
The prior analyses of benefits, initiatives, stakeholder identification, metrics can provide a great first cut value based business case
For quantities that need to be estimated you could apply the 90% CI calibration exercise to get ‘better’/more reliable estimates
However, one question is not enough to help you calibrate. You may have to spend some time practicing calibrating yourself*
You may need some sample information to help you get started…
*Please refer “How to Measure Anything…” by Douglas Hubbard for more details/exercises
10/6/2010
© USC-CSSE

59. Value of Information
How much time/money should you spend (if at all) gathering information ‘about’ the metric before estimating?
That depends on the expected value (EV) of information – Commonly applied in statistical decision theory but uncommonly so in software engineering!
Defined as Expected Value of Perfect Information (EVPI):
EVPI = EV(perfect info) – EV(no info)
EVPI is the ‘theoretical maximum’ you should be willing to pay for procuring further information
Since perfect information is not obtainable, EVSI (expected value of sample information) is usually employed. Thus the net expected value of information is gained by ‘subtracting’ cost of sampling/procuring information!
This was elucidated way back in 1981 by Dr. Boehm when he published the Blue Book*: “Software Engineering Economics”!
You must perform Value of Information Analysis on EVERY attribute before deciding to gain additional information about it. It’d save a lot of time/money since a lot of attributes turn out to have VI = 0!!
The interested reader should read the Blue Book mentioned above. Further insight can be gained by referring Doug Hubbard’s book “How to measure anything…”
*Chapter 20: Statistical Decision Theory: Value of Information
10/6/2010
© USC-CSSE

60. Next: Eliciting Stakeholders’ Win Conditions
(a.k.a., Value Propositions)
10/6/2010
© USC-CSSE

61. Stakeholders’ Value Propositions
We started our discussion with Theory-W…
…we now build on how to ‘implement’ it in action
Value Propositions are also known as Win Conditions
Theory-W involves eliciting, understanding and reconciling SCS (Success Critical Stakeholders) win conditions with achievable solutions
10/6/2010
© USC-CSSE

62. Win-Win Negotiation Process*
Collect SCS win conditions (WC) – basically brainstorming what the ‘expect’ from the system
Converge on WCs – since it’s captured in natural language effort is taken to converge on a concisely worded, non-redundant, unambiguous list of Win Conditions
Define glossary of key terms – basically the domain glossary used within a project
Prioritize WCs – Business Value vs. Ease of Realization
Reveal Issues/Constraints – the prioritization helps facilitate discussion about issues that may arise with the stakeholder’s expectations
Identify Issues and Options – the team posts issues (if any) for each of the identified win conditions and identifies the options to deal with the issues
Negotiate Agreements – negotiate/agree to the above, using oral discussion and entering it into the ‘online tool’. We say WinWin Equilibrium is reached when all WCs and Options have been agreed to.
*Stakeholder Value Proposition Elicitation and Reconciliation – Grunbacher et. al.
10/6/2010
© USC-CSSE

63. How/Where to Capture Win Conditions
I’m currently working on an online collaboration tool, similar to Facebook to help capture stakeholder’s win conditions, raise issues, supply options, prioritize win conditions and indicate the current statue of equilibrium
Attached are the screenshots of the tool. The tool will be deployed on USC’s servers for the upcoming class of Software Engineering CS577a in Fall ’11 (The tool will also be available for download for use by anyone)
10/6/2010
© USC-CSSE

64. Winbook

65. Prioritization
These are two primary approaches that I’ve narrowed down from a value based perspective
Incremental Funded Methodology (IFM)
Advocated by the book “Software by Numbers” – Mark Denne, Jane Cleland-Huang.
Deals with grouping features into minimum marketable features and performing financial analysis using net present values and using that to predict ROI
Kano Analysis
Commonly applied in Quality Function Deployment for prioritizing voice of customer data. Also advocated by Agile Mentor Mike Cohn for use in agile software development.
We’ll go with the latter since the former is mathematically involved. Both could be applied in tandem, but we’ll go with the latter
10/6/2010
© USC-CSSE

66. So, how do we know which features fall in which category?
Kano Analysis
Linear: The more of the feature the better
Exciter: If absent customer wouldn’t care but would be delighted to have them
Must be: Absence implies severe dissatisfaction. Expected to be present
So, how do we know which features fall in which category?
Image From:
10/6/2010
© USC-CSSE

67. Kano Questionnaire*
You ask the following question(s) to the prospective users:
Functional Form of Question
If feature X were present how would you feel?
I like it that way
I expect it to be that way X
I am neutral
I can live with it that way
I dislike it that way
Dysfunctional Form of Question
If feature X were absent how would you feel?
*Further details can be found in Agile Estimating and Planning – Mike Cohn.
10/6/2010
© USC-CSSE

68. Cross-referencing the Responses
Customer Requirements
Dysfunctional Question
Like
Expect
Neutral
Live with
Dislike
Functional Question Q E L R I M
E Exciter
L Linear
M Must Have
Q Questionable
I Indifferent
R Reverse
10/6/2010
© USC-CSSE

69. Percolation of Value into the SDLC
The prioritized win conditions give an idea of how the customers values the features on a scale of desirability
The win conditions are the set of value propositions of the stakeholders – their expectation(s) from the system
Win conditions can be ‘mapped’ to benefits – either in the form of a matrix or by visual inspection
You can even ‘rank’ the benefits in order of importance using MAUT too!
Decisions about various win conditions are then taken with respect to the benefits to be achieved
Decisions at the project level must factor in the impact on strategic goals
This ‘framework’ of expected benefits should always be visible to the development team!
10/6/2010
© USC-CSSE

70. Value Based Tracking
I haven’t explored this area in it’s totality but what seems to hold a lot of promise is “Multivariate Statistical Analysis” especially Multivariate Regression Analysis
We can pick out key ‘independent variables’ from the intermediate outcomes of the benefits chain and let the ‘dependent variable’ be that from the final intended goal/outcome
We can run a regression analysis to know if we really gained the benefits. Statisticians have been doing it all along!
This implies that if we DON’T have relevant data we may need to perform sampling to have enough data points to warrant this analysis.
Is the sampling effort worth it? It depends on the value of information!
10/6/2010
© USC-CSSE

71. Wrapping Up!
…finally
10/6/2010
© USC-CSSE

72. Conclusion
Presented ONE way of applying VBSE in practice using ONE set of tools and techniques.
There are myriads of possible combinations of tools/techniques that could be used. Use the one best suited to your organization…
…The ideas in this presentation should help you get started
We’ve covered a lot of ground but it’s only possible to present a limited amount. I’ve attached a set of references for further reading!
Hope you gained some ‘value’ out of this presentation!
10/6/2010
© USC-CSSE

73. References Value Based Software Engineering – Stefan Biffl et. al.
How to Measure Anything - Douglas Hubbard
The Information Paradox – John Thorp
Software Engineering Economics – Barry Boehm
Making the Software Business Case – Donald Reifer
Decision with multiple objectives – Keeney, Raiffa
Agile Estimation and Planning – Mike Cohn
Software by Numbers – Mark Denne, Jane Cleland-Huang
Lean Software Development - Poppendieck
Agile Software Requirements – Dean Leffingwell
Yet to finalize/refine/clean…
10/6/2010
© USC-CSSE