Modeling Uncertainty Farrokh Alemi, Ph.D. Saturday, February 21, 2004

Why Make a Model? To forecast likelihood for complex events To forecast likelihood for complex events To predict events that rarely occur To predict events that rarely occur To understand and communicate the nature of uncertainty To understand and communicate the nature of uncertainty

Online HMO MD – Pt and phone contact MD – Pt and phone contact –Triage before visit  Lab done before visit  Visits postponed –Follow up automated No historical precedence

Step 1. Select Target Event Usually two events Usually two events –Die/live –Purchase/not purchase –Join HMO/Not join Mutually exclusive events Mutually exclusive events Exhaustive Exhaustive

Step 2. Divide & Conquer Posterior Odds Likelihood Ratios Prior Odds

Step 3. Identify Clues Would you tell me a little about your experience with employee choice of health plans? Would you tell me a little about your experience with employee choice of health plans? –Suppose you were to decide whether an employee is likely to join but you could not contact the employee. I was chosen to be your eyes and your ears. What should I look for?  What is an example of a characteristic that would increase the chance of joining the HMO?  Describe an employee who is unlikely to join the proposed HMO

Example of Clues Identified Age Age Income and value of time to the employee Income and value of time to the employee Gender Gender Computer literacy Computer literacy Current membership in an HMO Current membership in an HMO

Step 4. Describe Levels of Clues Measures the extent to which clue is present Measures the extent to which clue is present –Usually present or absent Combine levels that are similar Combine levels that are similar Analyst: What age would favor joining the HMO? Expert: Young people are more likely to join Analyst: How do you define young employees? Expert: It all depends. Below 30 is different from above 30.

Levels for HMO Example Age (younger than 30, 31 ‑ 40, older than 41) Age (younger than 30, 31 ‑ 40, older than 41) Value of time to the employee (income over $50,000, income between $30,000 and $50,000, income less than $30,000) Value of time to the employee (income over $50,000, income between $30,000 and $50,000, income less than $30,000) Gender (male, female) Gender (male, female) Computer literacy (programs computers, frequently uses a computer, routinely uses output of a computer, has no interaction with a computer) Computer literacy (programs computers, frequently uses a computer, routinely uses output of a computer, has no interaction with a computer) Tendency to join existing HMOs (enrolled in an HMO, not enrolled in an HMO) Tendency to join existing HMOs (enrolled in an HMO, not enrolled in an HMO)

Step 5. Test for Independence If you have data: If you have data: –Reduction of sample size –Correlation analysis If you do not have data: If you do not have data: –Draw a causal model –Ask experts to group clues if one tells us a lot about another with specific populations

Step 6. Estimate Likelihood Ratios Of 100 people who do join, how many are younger than 30? Of 100 people who do not join the HMO, how many are younger than 30? Of 100 people who do join, how many are younger than 30? Of 100 people who do not join the HMO, how many are younger than 30? Imagine two employees, one who will join the HMO and one who will not. Who is more likely to be younger than 30? How many times more likely? Imagine two employees, one who will join the HMO and one who will not. Who is more likely to be younger than 30? How many times more likely?

How to Improve Estimates of Likelihoods? Ask experts not novices Ask experts not novices Ask only in format familiar with the expert. Ask only in format familiar with the expert. Provide access to tools and available data, if relevant Provide access to tools and available data, if relevant Train experts in probability concepts Train experts in probability concepts –Likelihood ratio of 1, less, or more than 1 –Relationship between odds and probability (Odds of 2 ‑ to ‑ 1 mean a probability of 0.67; odds of 5 ‑ to ‑ 1 mean a probability of 0.83) Rely on more than one expert and discuss first estimate as well as any estimate with large differences Rely on more than one expert and discuss first estimate as well as any estimate with large differences

Step 7. Estimate Prior Odds Out of 100 employees, how many will join? Out of 100 employees, how many will join? Odds for joining = p(Joining) / [1 - p(Joining)] Odds for joining = p(Joining) / [1 - p(Joining)] Probability of Joining = Odds of Joining / (1 + odds of joining) Probability of Joining = Odds of Joining / (1 + odds of joining)

Step 8. Develop Scenarios Select one level for each clue Select one level for each clue Organize on one piece of paper Organize on one piece of paper Ask the expert to rate on a scale from 0 1o 100 the chances that target event will happen Ask the expert to rate on a scale from 0 1o 100 the chances that target event will happen

Optimistic Scenario for HMO Example A 29 ‑ year ‑ old male employee A 29 ‑ year ‑ old male employee Earns more than $60,000. Earns more than $60,000. He is busy and values his time; He is busy and values his time; He is familiar with computers, using them both at work and at home. He is familiar with computers, using them both at work and at home. He is currently an HMO member, though not completely satisfied with it. He is currently an HMO member, though not completely satisfied with it. On a scale from 0 1o 100, how likely do you think is for this person to join the proposed HMO?

Pessimistic Scenario for HMO Example A 55 ‑ year ‑ old female employee earning less than $85,000. She has never used computers and has refused to join the firm's existing HMO. A 55 ‑ year ‑ old female employee earning less than $85,000. She has never used computers and has refused to join the firm's existing HMO. On a scale from 0 1o 100, how likely do you think is for this person to join the proposed HMO?

A More Realistic Scenario for HMO Example A 55 ‑ year ‑ old female employee earning more than $60,000 has used computers but did not join the firm's existing HMO. A 55 ‑ year ‑ old female employee earning more than $60,000 has used computers but did not join the firm's existing HMO. On a scale from 0 1o 100, how likely do you think is for this person to join the proposed HMO?

Scenario Planning Helps decision makers understand possible futures Helps decision makers understand possible futures Helps decision makers work to realize alternative futures Helps decision makers work to realize alternative futures

Step 9: Validate the Model

Step 10. Make a Forecast Likelihood ratios Likelihood ratios –1.2 for being young, –1.1 for being male, –1.2 for having a high hourly rate, –3.0 for being computer literate, –0.5 for not being a member of an HMO. Odds of joining = 1.1 x 1.2 x 3 x 0.5 x 1 = 1.98 Odds of joining = 1.1 x 1.2 x 3 x 0.5 x 1 = 1.98 Probability of joining = 1.98 / ( ) = 0.66 Probability of joining = 1.98 / ( ) = 0.66

Take Home Lessons Forecasts of unique events are possible Forecasts of unique events are possible Known clues and relationships can be used to forecast an event Known clues and relationships can be used to forecast an event Bayes model needs likelihood ratios and prior odds Bayes model needs likelihood ratios and prior odds –Experts can supply these estimates We can validate model against experts’ judgments We can validate model against experts’ judgments