PROFILING USERS BY ESTIMATING COMPOSITE AND MULTI-VALUED ATTRIBUTES FROM BIG DATA SOURCES FOR SOCIAL STATISTICS PURPOSES NTTS 2017, Brussels, March 14-16, 2017 Jacek Maślankowski CENTRAL STATISTICAL OFFICE, Statistical office in gdańsk, poland UNIVERSITY OF GDAŃSK, POLAND

AGENDA NTTS 2017, Brussels, March 14-16, 2017 Prerequisites Framework Results of analysis Conclusions NTTS 2017, Brussels, March 14-16, 2017

OVERVIEW AND THE GOAL OF THE STUDY Show the methodology of extracting big data sources for social statistics purposes Provide information on population with detailed attributes that can be extracted from the data or at least estimated Attributes can be composite (e.g., address, name, etc.) as well as multi-valued (e.g., phone numbers) H1: The usability of the data can be increased by estimating values for specific entities H2: The representativeness of the web data does not allow applying it directly for social statistics purposes NTTS 2017, Brussels, March 14-16, 2017

METHODOLOGY AT A GLANCE Set of combined methods used to extract users profiles from both social media as well as webpages Attributes available for analysis: Screen name Full name Geographic location URL Description Estimation of attributes: Gender Different forms of verbs Language markers Find phone/e-mail address Regular expressions NTTS 2017, Brussels, March 14-16, 2017

METHODOLOGY IN DETAILS METHODS: (1) Machine Learning tools (2) Text Mining methods STEPS: Analyse the readiness of the data source Social Big Data – Social Mining – Web Mining: profiling users Identification of demographic attributes POPULATION Group of social media users and Internet users that make comments on selected web portals NTTS 2017, Brussels, March 14-16, 2017

USE CASES WHO WHAT Entity is a person who is active on social media as well as persons that are making comments on various events in the country Three different cases were used to make analysis and enhance the social statistics: intentions to vote (mostly covered in statistics from OECD) media education – how people trust in media social confidence NTTS 2017, Brussels, March 14-16, 2017

RESULTS – ENTITY CLASSIFICATION (IDENTIFY GENDER) BASED ON THE VERB FORM BASED ON THE VERB FORM AND TRAINING DATASET Based on suffix of the verb form: F1-score varies from 0.25 to 0.57 in specific datasets with precision 0.33 to 0.5. Cannot be included as the primary method of identification. MultinomialNB and Linear SVM From 0.8075-0.866 Supported by different methods NTTS 2017, Brussels, March 14-16, 2017

CONCLUSIONS (1/2) Several useful and reliable attributes can be extracted to enhance social statistics surveys Can enrich social surveys, e.g., on social confidence and intention to vote Results are presented using geographic and demographic attributes of the entities NTTS 2017, Brussels, March 14-16, 2017

CONCLUSIONS (2/2) The hypothesis H1 has been confirmed by comparing the results of analysis with the data from official statistics The hypothesis H2 has also been confirmed – we have to expect some differences in the results Each data source must be treated individually NTTS 2017, Brussels, March 14-16, 2017

THANK YOU! Jacek Maślankowski STATISTICAL OFFICE IN GDAŃSK POLAND NTTS 2017, Brussels, March 14-16, 2017