Analysing the reliability of forecast information provided by UNECE member states Master’s thesis by Markus Stolze

Who am I? And what am I doing here? Name: Markus Stolze From Finland Forestry student in University of Helsinki M.Sc. in Forest economics 6/2019 I was working summer 2018 here with FPAMR As part of my master’s thesis conducting research for UNECE about reliability of forest products forecast

United Nations Economic Commission for Europe What is UNECE? Organisation founded in 1947 and has currently 56 member states The UNECE region covers more than 47 million square kilometres, around 17% of world’s population and more than 40% of world’s forests

Background of the research And data used in the research Data used in this research is provided by member states of UNECE JFSQ data collected each year by Forestry and Timber section in Geneva Forecast data of forest products have been collected in database since 2002 Estimates (year n) and forecasts (year n+1) are produced annually Haven’t been used yet, as nobody knows how reliable it is Past study (2005) covered one product Research was done in co-operation with UNECE and Department of Forestry in University of Helsinki

Objective of research & Research Questions Objective of this research is to understand how reliable these predictions are As part of thesis I will also see, what are the elements that affect data quality Are some products more reliable than others? Goal of prediction is to be as similar as possible to JFSQ-value provided More specifically, this study aims to answer following questions: What are the main elements of forest product market data quality? Are forecast data about forest product markets presented by UNECE reliable? And if so, how reliable?

What products are included All together 12 products were chosen 8 are forest products and 4 are removals Product JFSQ-code HS2012-code What information is collected  Coniferous sawlogs and veneer logs 1.2.1.C   Only harvest Non-Coniferous sawlogs and veneer logs 1.2.1.NC Coniferous pulpwood 1.2.2.C Non-Coniferous Pulpwood 1.2.2.NC Coniferous sawnwood 5.C 4407.10 Exports, Imports and production Non-Coniferous sawnwood 5.NC 4407.21/22/25/26/27/28/29/91/92/93/94/95/99 Plywood 6.2 4412.31/32/39/94/99 Particle board (including OSB) 6.3 44.10 OSB 6.3.1 4410.12 Fibreboard 6.4 44.11 Wood Pulp 7 47.01/02/03/04/05 Paper and Paperboard 10 48.01/02/03/04/05/06/08/09/10, 4811.51/59 48.12/13

How is the analysis structured And other information about research Countries with response rate for estimates and forecasts of 66.6% are included in the research Out of possible 1116 data points (12 products & 15 years) 744 data points have to be filled actual estimates or forecasts This allows to observe real trends from year to year 27 countries are included in the research Estimates and forecasts are compared to historical data to see how far off they are A repeated estimate is created by repeating historical data from prior year (year n- 1) and comparing it to historical outcome (year n) Analysis includes all products, flows (imports, exports, production, removals) and countries, that have a high enough response rate Results presented are averages as %-difference from historical value of that year E.g. forecast of 500 is 10% smaller than historical value of 550

Results Part 1: Products Product Product code Estimate Forecast Repeated Paper and paperboard 10 10.0% 15.4% 8.6% Coniferous sawlogs and veneer logs 1.2.1.C 12.4% 16.8% 11.5% Coniferous sawnwood 5.C 12.6% 21.3% 13.4% Particle board (including OSB) 6.3 13.5% 21.1% 14.7% Fibreboard 6.4 18.4% 37.2% 25.5% Coniferous pulpwood 1.2.2.C 19.6% 25.1% 15.3% Non-Coniferous pulpwood 1.2.2.NC 22.5% 29.8% Plywood 6.2 39.8% 25.4% Non-Coniferous sawlogs and veneer logs 1.2.1.NC 27.5% 36.0% 22.0% Non-Coniferous sawnwood 5.NC 28.1% 46.9% 18.0% Wood Pulp 7 53.4% 70.9% 71.0% OSB 6.3.1 56.0% 69.1% 42.2%   Averages 24.9% 35.8% 23.8%

Results Part 2 As seen, repeated data are better than estimates 7 out of 12 times There are clear differences between different products Products with bigger volumes are more accurate If 5 countries with lowest accuracy are ignored: Estimates: 24.9% -> 14.6% Forecasts: 35.8% -> 20.6% Repeated: 23.8% -> 12.9%

Rank (based on estimate) Results Part 3: Countries As expected, there are big differences between countries Below is a table with averages from all years, products and flows Countries with large volumes are more likely to be more accurate Rank (based on estimate) Country estimate forecast repeated 1 Austria 6.3 % 9.3% 7.2% 2 Poland 7.8% 11.4% 7.7% 3 France 9.5% 18.3% 7.1% 4 Germany 9.8% 11.9% 9.2% 5 United States 9.9% 13.9% 7.4% 6 Canada 13.3% 15.6% 7 Italy 14.0% 22.5% 8 Lithuania 21.8% 16.1% 9 Spain 14.8% 32.9% 18.1% 10 United Kingdom 15.4% 30.9% 16.8%

Results Part 4: Estimates or repeated data When all countries, products and flows are added together for products, repeated data are the most accurate 172 times, while estimates are 138 and forecasts 11 times Product Product code Estimate Forecast Repeated Paper and paperboard 10 16 11 Coniferous sawlogs and veneer logs 1.2.1.C 2 14 Coniferous sawnwood 5.C 12 Particle board (including OSB) 6.3 6 1 18 Fibreboard 6.4 13 Coniferous pulpwood 1.2.2.C 9 Non-Coniferous pulpwood 1.2.2.NC 7 20 Plywood 6.2 4 Non-Coniferous sawlogs and veneer logs 1.2.1.NC 15 Non-Coniferous sawnwood 5.NC Wood Pulp OSB 6.3.1   Total 138 172

Results Part 5: Flows Out of 4 flows, production is the most accurate, followed by removals, imports and exports Removals were the only flow, where estimates were the most accurate Exports and imports are more likely to be affected by outside influences

Conclusion What to make of all this 53.6% of times repeated data are more accurate than others As repeated data were the most accurate 172 times when all products and countries were added up together If nothing unexpected happens, estimates are rarely better than repeated data Forecasts suffer from unexpected events more than others Financial crisis is clearly visible in the results Large volumes indicate accurate forecasts, both with countries and products There are some exceptions to this rule: E.g. Wood Pulp

How different countries produced forecasts Finding similarities between countries In addition to percentage values, also absolute values were calculated Different flows were transformed into graphs and visually compared All flows, exports, imports, production and removals Countries were divided into 4 groups based on how graphs of different flows looked like and selected countries interviewed Main methods included talking with outside experts and using data from first 2 quarters of each year Usually 1 or 2 people are making the final decisions No clear connection between different groups of countries and methods were identified

Limitations of the research And suggestions for the future Historical data could be edited afterwards, if new information was provided This creates a possibility that repeated data looks better than it is… …and estimates/forecasts are not compared to ”original version” of historical data E.g. removals from the Russian Federation, where forecasts are 90% off from actual values If research is re-done in the future, unedited historical data should be used Products with small volumes often have big effect on overall results Drop of 2 to 0.1 is 200% difference, but doesn’t matter in the reality Use of time series analysis could offer new information Deeper dive into reasons why some products are harder to predict

Acknowledgements I would like to thank following parties for crucial help with this project UNECE FAO University of Helsinki

Markus Stolze markus.stolze@helsinki.fi University of Helsinki 26 March 2019, Geneva