Presentation on theme: "1 Now-casting Food Consumer Price Indexes with Big Data: Public-Private Complementarities Pietro Gennari, Director, Statistics Division and Chief Statistician."— Presentation transcript:
1 Now-casting Food Consumer Price Indexes with Big Data: Public-Private Complementarities Pietro Gennari, Director, Statistics Division and Chief Statistician Sangita Dubey, Senior Statistician Food and Agriculture Organization of the United Nations IAOS 2014, 8-10 October, Da Nang, Vietnam Parallel Session 1 - Theme 2: Price and Economic Statistics
2 “Big Data” and Official Statistics Opportunities More timely & more frequent More detailed Cost-effective & lower response burden Issues Differences in target population and implications for representativity Data quality and reliability Privacy and confidentiality Equal access and transparency
3 “ Big Data” and Official Statistics – Key Questions Can these issues be addressed? Can big data contribute to the production of official statistics? For which indicators? As new private sector big data producers appear, should the role of NSOs be redefined? Is there a role for International Organizations? FAO publishes monthly Food CPIs using ILO + UNSD + NSO websites Added value: Harmonization, regional and global Aggregates, Now-casting (4 months time lag of ILO estimates) Now-casting essential for food security monitoring/early warning The use of big data in compiling official Food CPIs can inform this discussion
AFCAS 23, 2013 4 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Use of Big Data in compiling Food CPIs Used to compile Food CPIs in Netherlands, Norway, Switzerland, and other Eurostat member countries Requires legal agreements with retailers/data providers to ensure timely data delivery Omits mom-and-pop outlets and outdoor markets Retail, point- of-sale Scanner Data Used to compile CPIs by private firms (PiceStats, Premise) Permits real-time CPI compilation; low-cost and trans- national Data access not free; sub-indices not available Expenditure weights not used Omits outlets/products without online presence Internet scraped food prices Used by private firms (Premise) and government ministries for market information/food security monitoring Permits near real-time CPI compilation/production Pure crowd-sourcing may entail partial coverage Controlled crowd-sourcing can approximate NSO data collection if used as a CAPI-type application Crowd- sourced mobile app food price collection
5 The Premise Food Staples Index (FSI) Premise is trans-national company, collecting food price statistics in US, Argentina, Brazil, India, China, and now expanding into Africa. It adopts international guidelines (CPI Manual) and official IBGE expenditure weights as a starting point It publishes both key food price levels as well as indexes, which expands use of their data to include price monitoring for food security Key advantage = provides daily indices in near real-time Business model = sell data to international banks and hedge funds interested in monitoring real-time price movements for purposes of lending and investment decisions. It combines crowd-sourced data with internet-scraped prices = potentially covers a broader range of countries Not strictly crowd sourcing = field workers are screened and paid, obtain field training & are assigned locations for collecting food prices.
6 Private vs Public “food CPIs” – an example from Brazil Brazil’s NSO – IBGE: Approach Monthly Laspeyre’s-type CPI with IBGE expenditure weights Covers 12 major cities with 3-stage sampling: city, outlet, product COICOP-type product classes Expert judgement to select main outlets and products IBGE enumerators PAPI data collection Publicly available, for free, shortly after month end, with long time series Premise (private firm): Approach Daily Laspeyre’s-type CPI, with IBGE expenditure weights (updated) Covers 5 major cities with 3-stage sampling: city, outlet, product COICOP-type product classes Expert judgement to select main outlets and products Crowd-sourced mobile app type CAPI data collection + web-scraped data Available only for subscribers. Short- time series (May 2013). Lead time varies from 25 (7day FSI) to 2 days (30day FSI)
AFCAS 23, 2013 7 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Brazil’s food-at-home CPI vs Premise Food Staples Index
AFCAS 23, 2013 8 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Food Price Inflation: IBGE vs Premise
AFCAS 23, 2013 9 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS The Premise FSI as predictor of the food-at-home CPI: Mean Absolute Percentage Errors and Lead Times Predicted food inflation using daily average Premise indices (Ft) IBGE value 7day15day21day30day(At) April0.810.600.550.501.52 May0.090.110.170.120.41 June-0.44-0.47-0.49-0.43-0.60 July0.280.340.320.30-0.51 August0.69 0.830.88-0.61 MAPE, April - Aug0.97 0.950.97 MAPE, April - June0.150.08 0.07 Lead Time25 days17 days10 days2 days
AFCAS 23, 2013 10 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Correlations indexes: Premise FSI vs IBGE food-at-home CPI 7day15day21day30day FSI vs food-at-home CPI 0.920.91 0.92 Meats 0.920.910.90 Fish & Seafood 0.910.930.92 Fruits 0.93 0.92 Vegetables 0.790.77 0.75 Oils & Fats 0.60*0.680.730.76 Beverages 0.920.91 Salts & Condiments 0.880.91 0.92 * indicates p-value in (0.01, 0.05); else where p-value ≤ 0.01
AFCAS 23, 2013 11 FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Correlations indexes: Premise FSI vs IBGE food-at-home CPI Month-on-month Inflation 7day15day21day30day FSI vs food-at-home CPI 0.51*0.50*0.51*0.50* Meats 0.50*0.48**0.49*0.45** Fish & Seafood 0.610.75 0.74 Fruits 0.630.650.630.62 Vegetables 0.700.730.750.74 Oils & Fats 0.33***0.48**0.55*0.64 Beverages 0.06***0.03***-0.07***-0.15*** Salts & Condiments 0.33***0.40***0.42***0.49** *** indicates p-value>0.10; ** p-value in (0.05, 0.10); * p-value in (0.01, 0.05); else p-values ≤ 0.01
12 The Premise FSI and the IBGE food CPI: some conclusions Premise data can now-cast food CPI up to 25 days before official data Quality of prediction improves with additional information. Marginal improvement is largest moving from the first 7-day average of daily price indices to the first 15-day average in a month. July and August 2014 inflation deviations between Premise FSI and Brazilian food CPI reduce quality of Premise’s now-cast. More investigation on cause of deviation required Quality of now-cast much higher when omitting July/Aug 2014 Current length of Premise time series reduces sophistication of now- cast methodology, and increases the standard error. A longer time series will also help validate if July/August 2014 were aberrations.
13 Implications for Public-Private Complementarities International methodologies and guidelines have been essential for private sector production of (food) CPIs “comparable” to official statistics Private sector may be more agile in producing low cost, high- frequency real-time statistics than NSOs, particularly important to monitor food security Business model may render private producers unable to provide impartial access to data in order to maintain their clients. NSOs may benefit from adopting private sector innovations NSOs may benefit from use of private sector data to validate official statistics Governments and IOs may benefit from use/purchase of private sector food price data for monitoring food security and now-casting official statistics
14 Implications for International Organizations IOs could develop and implement international standards, methodologies and guidelines in the use of new Big Data and its tools No current guidelines exist on use of crowd-sourced, mobile app- based food price collection, despite its growth in use by private firms and national and regional government organizations IOS can coordinate globally across statistical systems. Important for food security statistics, given national inter- dependencies and the growing role of private production and trans-national data IOs and regional organizations can serve as interim producers till national capacities are developed The African Development Bank and European Commission contracting a private firm to undertake food price collection in 40 African countries using mobile applications.
15 Pietro Gennari, Chief Statistician, Director of Statistics Division: ESS-Director@fao.org Sangita Dubey, Senior Statistician, Economic Statistics Team: Sangita.Dubey@fao.org Sangita.Dubey@fao.org For more information, please contact Thank you