Modelling sample data from smart-type meter electricity usage Susan Williams NTTS Conference, March 2015, Brussels

Introduction Big Data Project in the UK’s Office For National Statistics Practical pilot on the potential of smart-type meter data Pilot objectives: To investigate applications of big data sources within Official Statistics To develop capability in methods and processing of big datasets

Smart-type meter pilot: Background Smart meters: electronic devices taking high frequency energy readings Gas and electric smart meters to be rolled out to all households in UK by 2020 (UK policy) Readings (min 30 min frequency) to be wirelessly transmitted to central body for purposes covered by legislation. Various levels of consumer opt-out built in.

Privacy/Ethics High public concern over data security and access Theoretically: data might be used to identify individual households and real-time occupancy Data access legislation being devised ONS has approached UK privacy groups to gain support for this research

Research Question Benefit Validate Census returns or optimise Census follow up Investigate the potential of smart-type meter electricity data (high frequency – 30 mins) to model likelihood of household occupancy

What does an unoccupied household look like?

Research data Consumer behaviour trials of smart-type meters conducted by the Commission for Energy Regulation in Ireland and held in the Irish Social Science Data Archive 4,225 domestic smart-type meters 30 minute frequency 18 month trial (14 th July 2009 to 31 Dec 2010) Pre and post survey assigned

Approach Initial methods developed to automatically identify days where a household is unoccupied Key variables of interest include: ratio of day time to night time consumption, variance, average consumption, difference to usual consumption etc. Thresholds set for classification Manual checking (visual verification) – restricted to sample of 10 meters Performance visualised by contingency tables (confusion matrices): assessed using sensitivity and specificity measures

Confusion matrix Method 1 (low variance over 24 hours) Examined by eye Days unoccupiedDays occupied Days unoccupied18924 Days occupied35144 Sensitivity (true positive rate) = 189/192 = 98 per cent Specificity (true negative rate) = 5144/5168 = 99.5 per cent

Continuing research Writing more efficient code and using big data tools – now running methods on all meters, iterations to improve thresholds. Testing combinations of methods, longer term vacants Machine learning algorithms: e.g. logistic regression, cluster analysis Ultimately consider how to process circa 20 million meters. New research data sourced from trials conducted in Great Britain – comparison with Irish data ongoing

Summary ONS Big Data project has pilot project on the use of smart meter data Huge privacy and ethical concerns on access to smart meter data Methods to identify an unoccupied day using smart-type meter trial data have given promising results Big data processing capability has increased

A. Timed appliance in night and activity during day A more challenging profile B. Unoccupied C. Timed appliance in night ….but..unoccupied? A B C