1. Mark W. Rosegrant Siwa Msangi Liangzhi You Assessing Climate Forecast Impacts Advancing Ex Post Methodologies

2. The Growing Importance of Climate Forecast Information Increasing frequency of extreme weather events and changing global trends in climate characteristics The vulnerability of increasingly complex global economic and food systems to environmental factors In the face of increasing uncertainty, policy makers are demanding better information

3. How To Measure Forecast Value? Growing body of literature examining the economic value of forecast information Theoretical underpinnings are grounded in the theory of decision-making under uncertainty The majority of this literature employs ex ante methods

4. Ex Ante or Ex Post ? A (prior beliefs) B (update beliefs) (model possible response) C (observe outcome of event) (also observe agents actions) Receive Forecast Signal Realized Climate Outcome Ex AnteEx Post Modeled behavior Simulated Benefit Measured behavior Realized Benefit (simulated counter-factual)

5. Measuring Forecast Value Within an Ex Ante framework, behavior is modeled and response is simulated to evaluate the net benefits of forecast Within an Ex Post framework actual behavior is observed and underlying structural relationships driving response must be inferred to estimate the net benefits (comparing to without forecast information)

6. Overview of the Presentation Look at Traditional Impact Assessment Discuss the Challenges of Ex Post Evaluation Look at some promising directions Draw conclusions and recommendations for advancing research in this area

7. Ex Post Assessment in Ag Research Long history of application in empirical literature Looks mostly at benefits of new technology Evaluates net benefits with and without innovation Employs a variety of empirical methods

8. Two Approaches in Ex Post Assessment One approach tries to econometrically measure the impact of Ag Research on productivity or production costs with reduced form relationships Another approach uses consumer welfare theory to relate technology improvements to benefits received by consumers and producers of the agricultural goods within the economy

9. Econometric Methods Treating technology as an input, estimate production function, cost function or total factor productivity (TFP) Conventional Inputs (i.e. land) Unconventional Inputs (i.e. infrastructure) Technical knowledge (i.e.R&D investment) Uncontrollable factors (i.e. weather) Ag. Output

10. Econometric Methods Estimated research coefficients are then used to calculate the value of additional output attributable to the lagged research expenditures ( marginal rate of return to the research investment) Growth Accounting: contributions by the components in the above equation to the rate of growth of aggregated output

11. Basic Economic Surplus Model e Technology-induced supply shift (S 0 to S 1 ) Total benefits (consumer and producer benefits) are Area of I 0 I 1 ab Basic model can be extended to incorporate multi-markets, to accommodate spillover, to adjust for market distortions etc.

12. Lessons to Draw from the Literature Scale (project, program, institution or the whole system) Attribution (proper accounting for benefits and costs) Selection bias (random sampling or cherry-picking) Time lags (long lag between R&D investment and final impact)

13. Lessons to Draw from the Literature Econometric methods rely on good- quality time series/panel data. More appropriate for entire research system rather than individual projects. Economic surplus method requires limited data and flexible. It is widely used.

14. Challenges of Ex Post Assessment Harder to take a descriptive rather than prescriptive approach Non-excludable nature of climate information makes valuation harder Must infer relevance of forecast from observed behavior which could be driven by a variety of factors Dis-entangling the underlying structural relationships is non-trivial

15. Undertaking a Descriptive Analysis 1. Define decision alternatives and determine that the decision is weather-information-sensitive 2. Identify users goals 3. Identify all decision-relevant information available to user 4. Develop a model describing the relationship between available information and the decision 5. Evaluate the model. Does it adequately describe the users behavior? 6. Use the model to determine the impact of forecast on criteria

16. Economic Valuation Methods Observed Behavior (ex post) Hypothetical Behavior (ex ante) Direct Valuation Experiments that measure response to better information in a laboratory Observed behavior under improved information from collected data Questions on willingness-to-pay for better forecasts Simulated behavior under improved information, assuming a behavioral model Indirect Valuation Hedonic values generated by actual behavior within a related market that can be tied to forecast information Contingent ranking of attributes that can be indirectly related to forecast

17. Direct Valuation Methods An Ex Ante approach would rely on questioning the information consumer directly An Ex Post approach relies on the revelation of this value through behavior directly tied to climate shocks Making the structural link between the climate shock and observed action is the challenge

18. Ex Post Approaches to Direct Valuation Direct Valuation can be done using reduced-form relationships that derive statistical relationships between forecast information to behavior Surplus values can also be computed An alternative is to look more closely at the underlying structural determinants of behavior and estimate models that can link those to climate information

19. Reduced-Form Methods An econometric Ex Post analysis relies on the statistical inference of forecast value from observables in both the physical and economic environment (e.g. land value and climate characteristics) Controlling for non-forecast related factors that affect observed reaction to climate shocks (and information) is the principal challenge

20. Possible Reduced-Form Model Relationship tying Farm profits (P) to climate information (K) and other on- farm characteristics Conventional Inputs (i.e. land) Unconventional Inputs (i.e. infrastructure) Climate Information (knowledge sources) Uncontrollable factors (i.e. weather)

21. Controlling for Behavioral Factors Since the on-farm input levels are endogenous, they are instrumented The ability of the farmer to adjust also should be accounted for On-farm characteristics (i.e. credit/labor constraints, farmer experience) Exogenous factors (environmental, etc. ) Error term

22. Structural Estimation Approach Is better able to connect the underlying drivers of response to climate information and environmental shocks than a reduced-form approach Better able to represent the constraints to agent behavior Comes at a great computational cost as behavioral as well as environmental relationships must be estimated

23. Promising Trends Despite relatively thin literature covering ex post methods, recent examples of innovative applications have surfaced Cover a variety of settings, from pastoral management to crop production Some methods empirical while others are experimental

24. A Few Case Studies Solow et al. (1998) estimate net welfare from the use of ENSO-based climate information to range between $240 and $320 million annually for the U.S. agriculture sector alone Bradford and Kelajian (1978) looked at the benefit-to-cost ratio of reducing sampling error in government-collected crop and livestock statistics

25. Livestock Management Luseno et al. (2003) look at pastoralists in Northern Kenya/Southern Ethiopia Compare pastoralists own perceptions with climate forecasts and observe how it influences their beliefs They note the importance of taking flexibility into account, when measuring forecast value, to avoid underestimation

26. Experimental Approach Sonka et al. (1988) use agribusiness example to show how decision experiments can be used to test response to varying levels of climate information By using a controlled setting they are able to measure the impact of information more accurately and directly observe agent behavior

27. Comments and Critique Despite the novelty of experimental methods, one may not be entirely sure that real world behavior is being observed Much depends on the design of the experiment and the framing of the problem But much insight can still be gained, and methods are apt to keep improving

28. Conclusions Surplus-based methods might be applicable if designed properly Structural estimation approaches give the most detail of the underlying relationships, but are the most challenging to apply, hard to generalize Experimental techniques have increasing appeal and potential utility

29. Conclusions Econometric methods would be valuable if reliable time series/panel data are available Need to design panel or cross-sectional data collection efforts for evaluating climate forecast information (i.e. ASTI in agricultural R&D). Needs collaborative effort and long-term commitment.