1. Big Data ESSNet WP 1: Web scraping / Job Vacancies Pilot

2. WP1 – Web scraping / job vacancies pilot
Aim:
To demonstrate which approaches are most suitable for the production of statistical estimates in the domain of job vacancies
To establish how these approaches could be used within the ESS
Potential Sources:
Job portals
Jobs advertised on enterprise websites
Third party data providers

3. WP1 – Web scraping / job vacancies pilot
Participants:
UK (work package lead)
Germany
Greece
Italy
Netherlands
Sweden
Slovenia

4. Eurostat Job Vacancy Statistics
EU policies in the area of job vacancies aim to improve the functioning of the labour market by trying to more closely match supply and demand. 
Current statistics
Based on business surveys
Quarterly job vacancy rates by member state
Totals by enterprise sector and enterprise size (for some member states only)
Issues:
No statistics on job vacancies by occupation or geography
Inconsistent definitions and coverage

5. Web scraping job portals
Benefits:
Data relatively easy to collect.
Feasible to build robots to scrape specific websites
Issues:
Duplication of vacancies (both within and between portals)
Many jobs advertised through employment agencies (that do not mention the employing enterprise)
Challenges linking back to the business register
Many jobs not advertised on job portals
Legal and Ethical Issues / Relationship with owners of job portals

6. Web scraping enterprise websites
Benefits:
Eliminate problem of duplicate job vacancies
Easy integration with existing job vacancy survey data
Issues:
Very technically challenging. A “crawling” approach is required that can cope with variable website structures (WP2 is focused on developing the approach)
Need a list of enterprise URLs (linked to business register)
Some enterprises may not advertise job vacancies through their website
Legal Issues

7. Third party data suppliers
Benefits:
Outsourcing of legal / ethical risks
Eliminate technical risk of non-delivery
Could help fast-track quality assessment of what is possible with on-line job vacancy data
Issues:
Cost / Procurement
Lack of methodological transparency

8. Third Party Supplier: Wanted Analytics
Combines real-time, global business intelligence with hiring demand and talent supply data to help make better strategic business decisions
ONS (UK) have already had some preliminary discussions about obtaining data.
Data available (at a cost) for four countries participating in WP1 (UK, Germany, Netherlands, Sweden)
Data provider for The Conference Board’s Help- Wanted OnLine Data Series™, the monthly economic indicator of Hiring Demand in the United States.
Published outputs useful for assessing what might be possible using web data……

9. Wanted Analytics Sample Output: Time series of job advertisement volumes

10. Wanted Analytics Sample Output: Regional Labour Market Indicators

11. Wanted Analytics Sample Output: Supply/Demand Index

12. Overall Approach
Initial focus on job portals and third party data (2016)
Review of recent research (e.g. UNECE pilot) and legal issues
Review and qualitative assessment of major job portals
Design and implement some experimental data collections using simple “point and click” web scraping tools
Develop methods for de-duplicating, cleaning, coding and storing job portal data
Explore option of obtaining data direct from job portal owners
Gain access to third party data (possibly for Germany?)
Produce experimental outputs from job portals and third party data in 2016
Explore approach of web scraping enterprise websites in 2017 once approach is further developed

13. Challenges and Issues
Overlaps and dependencies with WP2 – Work package leads for WP1 and WP2 are participating in both pilots
Different levels of technical experience. Training offer for Greece
A large number of participating countries
Managing expectations. This pilot is about feasibility, not the delivery of a full set of methods and tools for producing job vacancy statistics.