Download presentation

Presentation is loading. Please wait.

Published byKaley Ramos Modified over 2 years ago

1
INFO 631 Prof. Glenn Booker Week 9 – Chapters 24-26 INFO631 Week 9

2
Decisions Under Risk Ch. 24 INFO631 Week 9

3
**Decisions Under Risk Outline**

Introducing decisions under risk Different techniques Expected value decision making Expectation variance Monte Carlo analysis Decision trees Expected value of perfect information INFO631 Week 9

4
Decisions Under Risk When you know the probabilities of the different outcomes and will incorporate them Expected value decision making Expectation variance Monte Carlo analysis Decision trees Expected value of perfect information When you don’t know (or can’t say) the probabilities, use Decisions Under Uncertainty (next chapter) INFO631 Week 9

5
**Expected Value Decision Making**

The value of an alternative with multiple outcomes can be thought of as the average of the random individual outcomes that would occur if that alternative were repeated a large number of times Can use PW(i), FW(i), or AE(i) INFO631 Week 9

6
**Expected Value of a Single Alternative**

Denali project at Mountain Systems Imagine 1000 parallel universes where the Denali project could be run at the same time Should expect most favorable outcome would happen in 15% or 150 of those universes Fair outcome would happen in 650 Least favorable outcome would happen in 200 Least Most favorable Fair favorable PW(MARR) $ $ $9012 Probability INFO631 Week 9

7
**Expected Value of a Single Alternative**

Total PW(i) income generated Average PW(i) income in each universe Notice 200 * -$1234 = ,800 650 * $5678 = $3,690,700 150 * $9012 = $1,351,800 $4,795,700 $4,795,700 / 1000 = $ (0.20 * -$1234) + (0.65 * $5678) + (0.15 * $9012) = $ INFO631 Week 9

8
**Expected Value of a Single Alternative**

General formula Can be used to help decide between multiple alternatives INFO631 Week 9

9
**Expected Value of Multiple Alternatives**

Same probability Several projects at Mountain Systems Expected values Choose Shasta, it has the highest expected value Least Most favorable Fair favorable Alternative % % % Denali $ $ $9012 Shasta Washington The concept can be extended to help when choosing from among several alternatives Denali (0.20 * -$1234) + (0.65 * $5678) + (0.15 * $9012) = $ Shasta (0.20 * -$1201) + (0.65 * $6601) + (0.15 * $9282) = $ Washington (0.20 * -$3724) + (0.65 * $4104) + (0.15 * $9804) = $ INFO631 Week 9

10
Expectation Variance What if probabilities were different for each alternative? Comparing projects Lassen has higher expected value but win big-lose big Moana Loa has lower expected value but more probability of profit Outcome Probability AE(i) Least favorable % $3494 Nominal % Most favorable % Expected value = $665 Least favorable % $200 Low nominal % High nominal % Most favorable % Expected value = $466 Lassen Moana Loa Depending on how risk-averse you are. Lower risk tolerance would take Moana Loa project (better choice in lean times?), higher risk tolerance would likely take Lassen (better choice in boom times?) INFO631 Week 9

11
Monte Carlo Analysis Randomly generate combinations of input values and look at distribution of outcomes Named after gambling resort in Monaco Use [a variant of] Zymurgenics project (different data) Least favorable Fair Most favorable estimate estimate estimate Initial investment $500, $400, $360,000 Operating & maintenance $ $ $800 Development staff cost / month $49, $35, $24,500 Development project duration months months months Income / month $24, $40, $56,000 Zymurgenics project was in last chapter but data here is actually wrong data from book. Need to rerun simulation… INFO631 Week 9

12
**Monte Carlo Analysis Simulation run results**

Income range Number of occurrences -$75,000 to -$50, -$50,000 to -$25, -$25,000 to -$ $0 to $24, $25,000 to $49, $50,000 to $74, $75,000 to $99, $100,000 to $124, $125,000 to $149, $150,000 to $174, $175,000 to $199, $200,000 to $224, $225,000 to $249, $250,000 to $274, Zymurgenics project was in last chapter but data here is actually wrong data from book. Need to rerun simulation… INFO631 Week 9

13
**Monte Carlo Analysis INFO631 Week 9**

Zymurgenics project was in last chapter but data here is actually wrong data from book. Need to rerun simulation… Point out EV, probability of – outcome, … INFO631 Week 9

14
Decision Trees Maps out possible results when there are sequences of decisions and future random events Useful when decisions can be made in stages Basic Elements Decision nodes – points in time where a decision maker makes a decision (square) Chance nodes – points in time where the outcome is outside the control of the decision maker (circles) Node sequencing INFO631 Week 9

15
Sample Decision Tree INFO631 Week 9

16
**Decision Tree Analysis, Part 1**

Add the financial consequences for each arc (PW(i), FW(i), or AE(i)) Properly adjust for time periods as required Sum financial consequences from the root node to all leaf nodes INFO631 Week 9

17
Sample Decision Tree INFO631 Week 9

18
**Decision Tree Analysis, Part 2**

Write probabilities for each arc out of each chance node Probabilities out of a chance node must = 1.0 Roll back values from leaf nodes to root If node is chance node, calculate expected value at that node based on values on all nodes to its right If node is decision node, select the maximum profit (or minimum cost) from nodes to its right INFO631 Week 9

19
Sample Decision Tree INFO631 Week 9

20
**Expected Value of Perfect Information**

Value at root node is expected value of decision tree based on current information Current information is known to be imperfect Reasonable follow-on question Research, experimentation, prototyping, … Might even be able to eliminate one or more paths through the tree because you may discover them to be impossible Analyzed decision tree provides information that will help answer that question “Would there be any value in taking actions that would reduce the probability of ending up in an undesirable future state?” Just introducing topic here—need to see the details in the book… INFO631 Week 9

21
**Expected Value of Perfect Information**

If we had a crystal ball and knew outcomes for chance nodes, we could find which path would be best Finding best path can be repeated for all possible combinations of random variables Probabilities for random variables are known Can calculate probability for each combination of outcomes For each combination of outcomes, multiply its best value by probability of that combination Sum the results of (value * probability) for all combinations of outcomes Sum is expected value given perfect information Difference between sum and expected value given current information is expected value of perfect information Just introducing topic here—need to see the details in the book… INFO631 Week 9

22
**Expected Value of Perfect Information**

EVPI is upper limit on how much to spend to gain further knowledge Probably impossible to actually get perfect information, organization should plan on spending less Just introducing topic here—need to see the details in the book… INFO631 Week 9

23
Key Points Value of an alternative with multiple outcomes is the average of the random individual outcomes that would occur if that alternative were repeated a large number of times (expected value) The alternative with the highest expected value is best With expectation variance, differing probabilities could influence the decision Alternative with lower expected value might be a better choice if it also has a much lower probability of a negative outcome Monte Carlo analysis generates random combinations of the input variables and calculates results under those conditions Repeated many times and statistical distribution of outcomes is analyzed Decision trees map out possible results when there are sequences of decisions together with a set of future random events that have known probabilities Useful with many possible future states and decisions can be made in stages The Expected value of perfect information provides answer to, “Would there be any value in taking actions that would reduce the probability of ending up in an undesirable future state?” INFO631 Week 9

24
**Decisions Under Uncertainty**

Ch. 25 Slides adapted from Steve Tockey – Return on Software INFO631 Week 9

25
**Decisions Under Uncertainty Outline**

Introducing decisions under uncertainty Different Techniques Payoff matrix Laplace Rule Maximin Rule Maximax Rule Hurwicz Rule Minimax Regret Rule INFO631 Week 9

26
**Decisions Under Uncertainty**

Used when impossible to assign probabilities to outcomes Can also be used when you don’t want to put probabilities on outcomes, e.g., safety-critical software system where a failure could threaten human life People may not react well to an assigned probability of fatality If probabilities can be assigned, Decision Making under Risk should be used INFO631 Week 9

27
**Payoff Matrix Shows all possible outcomes to consider**

One axis lists mutually exclusive alternatives Other axis lists different states of nature Each state of nature is a future outcome the decision maker doesn’t have control over Cells have PW(i), FW(i), AE(i), … Alternative State1 State2 State3 A A A A A INFO631 Week 9

28
**Reduced Payoff Matrix One alternative may be “dominated” by another**

Another alternative has equal or better payoff under every state of nature Reduced payoff matrix has no dominated alternatives Less work if dominated alternatives are removed Alternative State1 State2 State3 A A A A A INFO631 Week 9

29
**Laplace Rule Assumes each state of nature is equally likely**

Sometimes called “principle of insufficient reason” Calculate average payoff for each alternative across all states of nature Same as expected value analysis for multiple alternatives with equal probabilities INFO631 Week 9

30
Laplace Rule Example Alternative A4 is chosen; the highest payoff always wins! Alternative State1 State2 State Average payoff A A A A INFO631 Week 9

31
**Maximin Rule Assumes worst state of nature will happen Formula**

Most pessimistic technique Pick alternative that has best payoff from all worst payoffs Formula INFO631 Week 9

32
**Maximin Rule Example Alternative A5 is chosen**

Alternative State1 State2 State Worst payoff A A A A INFO631 Week 9

33
**Maximax Rule Assumes best state of nature will happen Formula**

Most optimistic technique Pick alternative that has best payoff from all best payoffs Formula INFO631 Week 9

34
**Maximax Rule Example Alternative A3 is chosen**

Alternative State1 State2 State Best payoff A A A A INFO631 Week 9

35
Hurwicz Rule Assumes that without guidance people will tend to focus on extremes Blends optimism and pessimism using a selected ratio Index of optimism, a, between 0 and 1 a = 0.2 means more pessimism than optimism a = 0.1 means more pessimism than a = 0.2 a = 0.85 means lots of optimism but a small amount of pessimism (15%) remains INFO631 Week 9

36
**Hurwicz Rule Formula Example a = 0.2 Alternative A2 is chosen**

Alternative State1 State2 State Blended payoff A (0.2 * 4021) + (0.8 * 948) = 1563 A (0.2 * 6004) + (0.8 * -2005) = -403 A (0.2 * 5104) + (0.8 * 0) = 1021 A (0.2 * 3014) + (0.8 * 1005) = 1407 INFO631 Week 9

37
**Hurwicz Rule .25 A3 6000 A4 A2 4000 A5 2000 0.5 -2000 INFO631 Week 9**

-2000 0.5 A2 A3 A4 A5 .25 Blended payoffs for each alternative as a function of alpha INFO631 Week 9

38
Minimax Regret Rule Minimize regret you would have if you chose wrong alternative under each state of nature If you selected A1 and state of nature happened where A1 had the best payoff then you would have no regrets If you selected A1 and state of nature happened where another alternative was better, you can quantify regret as difference between payoff you chose and best payoff under that state of nature Regret matrix Need to calculate Difference between payoff you chose and best payoff under that state of nature INFO631 Week 9

39
**Minimax Regret Rule – Calculate Regret matrix**

Difference between payoff you chose and best payoff under that state of nature For State 1 – A2 1005 – 948 = 57 For State 1 – A3 1005 – (-2005) = 3010 Etc. NOTE: use numbers from original matrix Alternative State1 State2 State3 A A A A INFO631 Week 9

40
**Minimax Regret Rule Choose alternative with smallest maximum regret**

Alternative A4 is chosen Alternative State1 State2 State Maximum regret A A A A INFO631 Week 9

41
**Summary of Uncertainty Rules**

Decision rule Alternative selected Optimism or pessimism Laplace A Neither Maximin A Pessimism Maximax A Optimism Hurwicz (a=0.2) A Blend Minimax regret A Pessimism INFO631 Week 9

42
Key Points Uncertainty techniques used when impossible, or impractical, to assign probabilities to outcomes Payoff matrix shows all possible outcomes to consider Laplace rule assumes each state of nature is equally likely Essentially expected value with equal probabilities Maximin rule is most pessimistic Pick alternative with best payoff from all worst payoffs Maximax rule is most optimistic Pick alternative with best payoff from all best payoffs Hurwicz Rule assumes that without guidance people will tend to focus on the extremes Blend optimism and pessimism using selected ratio Minimax Regret rule minimizes regret you would have if you chose the wrong alternative under each state of nature Choose alternative with smallest maximum regret INFO631 Week 9

43
**Multiple Attribute Decisions**

Ch. 26 INFO631 Week 9

44
**Multiple Attribute Decisions Outline**

Introducing multiple attribute decisions Case study: Fly-by-Night Air Different kinds of “value” Choosing attributes Measurement scales Non-compensatory techniques Compensatory techniques INFO631 Week 9

45
**Introducing Multiple Attribute Decisions**

Previous chapters explained how to make decisions using a single criterion, money Alternative with best PW(i), AE(i), incremental IRR, incremental benefit-cost ratio, etc. is selected Aside from technical feasibility, money is almost always the most important decision criterion But not the only one Often, other criteria (“attributes”) must be considered and can’t be cast in terms of money If you were responsible for buying a new laptop computer for use on a development project, price would probably be the most important decision criterion. But other attributes like processor speed, memory capacity, disk capacity, size of the screen, and reliability would probably also be factored into your decision. This chapter presents several techniques for making decisions when there is more than one attribute to consider. INFO631 Week 9

46
**Case Study: Fly-by-Night (FBN) Airlines**

10-year old regional airline with above average growth Moving into nationwide market as no-frills carrier As part of strategic planning, IT department charged with examining airline reservations systems 10 year planning horizon, effective income tax rate=37%, after-tax MARR=15% Research has identified five technically-viable alternatives Keep existing software Buy Jupiter commercial system Buy Sword commercial system Buy Guppy commercial system Develop new software in-house Develop new software offshore Assumed to be service alternatives—income is expected to be essentially same on all INFO631 Week 9

47
**Different Kinds of “Value”**

Decision process is all about maximizing value Choose from available alternatives the one that maximizes value When value is expressed as money, decision process may be complex but is straightforward Money isn’t the only kind of value Money is really only a way to quantify value Two kinds of value Use-value - the ability to get things done, the properties of the object that cause it to perform Esteem value - the properties that make it desirable Two kinds of value Use-value—the ability to get things done, the properties of the object that cause it to perform Esteem value—the properties that make it desirable INFO631 Week 9

48
Choosing Attributes Decisions should be based on appropriate attributes Each attribute should capture a unique dimension of decision Set of attributes should cover important aspects of decision Differences in attribute values should be meaningful in distinguishing among alternatives Each attribute should distinguish at least two alternatives Selection of attributes may be subjective Too many attributes is unwieldy Too few attributes gives poor differentiation Potential for better decisions needs to be balanced with extra effort of more attributes INFO631 Week 9

49
**FBN Air: Decision Attributes**

Total cost of ownership In-service availability Liffey performance index From Liffey Consultancy, Ltd in Dublin, Ireland Alignment with existing business processes Liffey index is entirely mythical—made up to support an interval scale INFO631 Week 9

50
Measurement Scales Each alternative will be evaluated on each attribute Many ways to measure things In fact, different “classes” of measurements Within a class, some manipulations make sense and others don’t So it’s important for you to know what the different classes of measurements are, how to recognize them, and what can and can’t be done with them. A given multiple attribute decision, such as the case study, often includes attributes that are in different classes. So it’s important for you to know what the different classes of measurements are, how to recognize them, and what can and can’t be done with them. INFO631 Week 9

51
**Measurement Scales Scale type Description Example Operations Nominal**

Two things are assigned the same symbol if they have the same value House style (Colonial, Contemporary, Ranch, Craftsman, Bungalow, …) =, <> Ordinal The order of the symbols reflects an order defined on the attribute Letter grades in school (A, B, C, ...) =, <>, <, >, <=, => Interval Differences between the numbers reflect differences in the attribute Temperature in degrees Fahrenheit or Celsius, Calendar date <, >, <=, =>, +, - Ratio Differences and ratios between the numbers reflect differences and ratios of the attribute Length in centimeters, Duration in seconds, Temperature in Kelvin +, -, *, / Here are four different scale types that can be used in measuring things. Nominal -> is existence. Put things in buckets. Ordinal -> One item is higher, lower, or equal to another. Buckets have order. Interval -> Same as ordinal but the space between them is a fixed amount. Consistent gap between buckets. Ratio -> Same as interval but zero is meaningful. Note the types of comparisons allowed. INFO631 Week 9

52
**FBN Air: Evaluation and Attribute Scales**

Cost Availability Liffey index Alignment Alternative PW(i) Months [ ] [Ex, Vg,Ok,Pr, Vpr] Existing $1.8M Excellent Jupiter $15.4M Poor Sword $21.6M Ok Guppy $16.7M Very poor New in-house $30.3M Excellent New off-shore $17.5M Very good Attribute Scale Cost Ratio Availability Ratio Liffey index Interval Alignment Ordinal INFO631 Week 9

53
**Dimensionality of Decision Techniques**

Two families of decision techniques Differ in how attributes used Non-compensatory, or fully dimensioned, techniques Each attribute treated as separate entity No tradeoffs among attributes Compensatory, or single-dimensioned, techniques Collapse attributes onto single figure of merit Lower score in one attribute can be compensated by—or traded off against—higher score in others The different families of techniques are not mutually exclusive. You could, for example, use non-compensatory techniques to reduce the number of alternatives then use compensatory techniques to make the final selection. INFO631 Week 9

54
**Non-compensatory Decision Techniques**

Three will be described Dominance Satisficing Lexicography INFO631 Week 9

55
Dominance Compare each pair of alternatives on attribute-by-attribute basis Look for one alternative to be at least as good in every attribute and better in one or more When found, no problem deciding One alternative is clearly superior to the other, inferior can be discarded May not lead to selecting one single alternative Good for filtering alternatives and reducing work using other techniques In FBN Air, Jupiter dominates Guppy Dominance technique is unlikely to select a single alternative but in some situations it could. If UFO’s choice were only between A3 and A5, the dominance technique would be sufficient. At the very least, dominated alternatives can be dropped from further consideration to reduce the amount of work to be done using the other techniques. INFO631 Week 9

56
**Satisficing Sometimes called “method of feasible ranges”**

Establish acceptable ranges of attribute values Alternatives with any attributes outside acceptable range are discarded May not lead to selecting one single alternative Good for filtering alternatives and reducing work using other techniques If FBN Air were to define a maximum cost at $25M and Alignment better than Poor, would eliminate alternatives Guppy and In-house A decision that considers several alternatives is almost certain to yield a more optimal choice than a decision that stops at the first minimally acceptable one INFO631 Week 9

57
Satisficing Can lead to selecting one alternative when used with an iterative propose-then-evaluate process Iterative version is appropriate when satisfactory performance, rather than optimal performance, is good enough If optimal performance needed, always identify several alternatives that meet satisficing criteria then do further decision analysis with one of other techniques Repeat Propose a new solution Evaluate that solution against the decision attributes Until the solution is within the acceptable range for all decision attributes Note: Stops when 1st acceptable solution is proposed If FBN Air were to define a maximum cost at $25M and Alignment better than Poor, would eliminate alternatives Guppy and In-house A decision that considers several alternatives is almost certain to yield a more optimal choice than a decision that stops at the first minimally acceptable one INFO631 Week 9

58
Lexicography Two previous techniques assume attributes have equal importance If one attribute is far more important than others, final choice could be made on that one attribute alone If alternatives have identical values for most-important attribute, use next-most-important attribute to break tie If still tied, compare next most important attribute, … Continue until a single alternative chosen or all alternatives evaluated FBN Air Alignment might be #1, eliminates all but Existing and In-house Cost might be #2, eliminates in-house What if the process ends without selecting a single alternative? On the assumption that every important attribute has been included in the decision analysis, then it doesn’t matter which of the remaining alternatives is chosen. Any one of them is, by definition, as good as the other remaining alternatives. In this case you can choose one at random, go with your gut feel, etc. Otherwise, you’ll need to introduce new attributes into the decision analysis. INFO631 Week 9

59
**Compensatory Decision Techniques**

Attribute values converted into common “figure of merit” Units for common scale are usually arbitrary If common scale is at least interval scale then scores can be compared meaningfully Two will be presented Nondimensional scaling Additive Weighting Analytical Hierarchy Process (see text) Remember, worse performance in one attribute can be compensated by—traded off against—better performance in another Analytic Hierarchy Process (AHP) described in book—read on your own INFO631 Week 9

60
**Non-Dimensional Scaling**

Convert attribute values into common scale so they can be added together to make composite score for each alternative Alternative with best composite score is selected All attributes are defined to have equal importance Common scale needs same range for all attributes Must also follow same trend on desirability; most-preferred value needs to always be biggest or always be smallest common scale value Formula for converting attributes, as long as interval or ratio-scaled, into the common scale Let’s entirely arbitrarily chose the common scale to be In FBN’s case, lower cost is better so lowest cost alternative would need to be given, say, the highest common rating for the cost attribute. The cost attribute of the Existing System alternative could be given 50 as its common scale rating. But higher Liffey Index is better so the highest Liffey Index alternative would be given the highest common rating for the L-I attribute. The L-I attribute of the Sword would be given 50 as its common scale rating. INFO631 Week 9

61
**FBN Air: Scaled Attributes**

Cost Availability Liffey index Alternative [0..50] [0..50] [0..50] Total Existing Jupiter Sword Guppy New in-house New off-shore Note: Let’s entirely arbitrarily chose the common scale to be In FBN’s case, lower cost is better so lowest cost alternative highest common rating higher Liffey Index (LI) is better so the highest LI alternative highest common rating. Best = Sword Best is Swords INFO631 Week 9

62
**Non-Dimensional Scaling and Ordinal Attributes**

When decision includes ordinal scaled attributes, you will need to: Ignore ordinal-scaled attributes Refine ordinal-scaled attributes to use interval or ratio scales and include them in nondimensional scaling Do nondimensional scaling for all interval- and ratio-scaled attributes then finish using a non-compensatory technique Alternative Total Alignment Existing Excellent Jupiter Poor Sword Ok Guppy Very poor New in-house Excellent New off-shore Very good Let’s entirely arbitrarily chose the common scale to be In FBN’s case, lower cost is better so lowest cost alternative would need to be given, say, the highest common rating for the cost attribute. The cost attribute of the Existing System alternative could be given 50 as its common scale rating. But higher Liffey Index is better so the highest Liffey Index alternative would be given the highest common rating for the L-I attribute. The L-I attribute of the Sword would be given 50 as its common scale rating. INFO631 Week 9

63
Additive Weighting Identical to non-dimensional scaling except attributes have different “weights” or degrees of influence on the decision An attribute that’s more important will have more influence on outcome Most popular Step 1: select common scale and convert all interval and ratio-scaled attribute values into that scale Just like non-dimensional scaling Step 2: assign weights based on relative importance Many different approaches to this Recommended approach is Each attribute given “points” corresponding to importance Weight for each attribute is its points divided by sum of points across all attributes Most popular compensatory technique This one separates dimensionalizing attribute values from weighting attributes & reduces decision complexity and allows for more precise analysis Attribute with twice the importance is given twice as many points INFO631 Week 9

64
**FBN Air: Weighting the Attributes**

Suppose FBN Air gives point values as shown for ratio and interval-scaled attributes Attribute Points Weight Cost / ( ) = 0.588 Availability / ( ) = 0.118 Liffey index / ( ) = 0.294 INFO631 Week 9

65
Additive Weighting Step 3: calculate each alternative’s total weighted score Example Existing = (0.588*50)+(0.118*50)+(0.294*0) = 35.3 Same as non-dimensional scaling, decision is made on total score if there are no relevant ordinal-scaled attributes Cost Availability Liffey index Alternative (0.588) (0.118) (0.294) Total Existing Jupiter Sword Guppy New in-house New off-shore Ignoring the ordinal-scaled attributes in the FBN Air example, the existing system is the best under additive weighting. If there are important ordinal-scaled attributes they will need to be addressed as explained in the non-dimensional scaling section. INFO631 Week 9

66
Key Points Aside from technical feasibility, money is almost always the most important decision criterion but it’s not always the only one Use values can usually be quantified in terms of money Esteem values can't be quantified in terms of money Decisions involving more than one attribute are almost inevitable Choose decision attributes to cover all relevant use values and esteem values Several different classes of measurement Nominal, Ordinal, Interval, and Ratio Within each class, some comparisons will make sense and others won’t Non-compensatory techniques treat each attribute as a separate entity Dominance, Satisficing, Lexicography Compensatory techniques allow better performance on one attribute to compensate for poorer performance in another Nondimensional Scaling, Additive Weighting INFO631 Week 9

Similar presentations

OK

1 1 © 2003 Thomson /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s University.

1 1 © 2003 Thomson /South-Western Slide Slides Prepared by JOHN S. LOUCKS St. Edward’s University.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google

Ppt on recycling of waste in india Ppt on common man made disasters Ppt on types of motion Ppt on accounting and finance Ppt on blood stain pattern analysis problems Ppt on meta analysis journal club Ppt on ozone layer depletion and its effects Convert pdf ppt to ppt online student Ppt online application form 2016 Ppt on job evaluation sample