1. Classifying enterprises by economic activity
Cedefop/CRISP/ESSNet
21 March 2018, Milan

2. Outline What is this about Few small presentations (~7 min) Discussion
UK – Fero Hajnovic
Germany – Chris Gabriel Islam
France – Maxime Bergeat
Belgium – Thomas Delclite
Discussion
- Think about what to get out of this session in discussion!

3. What is this about?
Methods for extracting economic activity from job ads
Assessment of the algorithms (accuracy, speed, applicability)

4. UK
Fero Hajnovic

5. Data Adzuna, 10th of Jan. 2018 Survey (has SIC) Matching Ads Comps
Before matching ~
~45 000
After matching ~
~ 1000

6. Machine learning 3. Word selection 1. NLP 4. Classifier + grid search
- low IDFs
words
- in some…
- ...but not all
SICs
Machine learning
1. NLP
The Skills People Group are the largest provider of...
skills people group largest provider funded commercial...
4. Classifier + grid search
- Multinomial NB
- SVM
2. TF-IDF
1051x27808 sparse matrix with stored elements

7. Results Accuracy 66% (NB) 60% (SVC)
55% (Combined - NB + Random forest)
Similar recall
Todo
More data
Equal distribution
Word selection

8. Germany
Chris Gabriel Islam

9. Source: Wikipedia, Adrichel
Using R on CEDEFOP data
Used Package: Class
Algorithm: K-Nearest-Neighbour, Accurarcy: 78 % A C D F G H I J K M N Q R
317 1 4 2
98% 5
758
183
367 50
55% 3 0%
1804 7 44 27 48
93%
109 26
806 22
131 12 13
506
49% 10
4626 19 83
97%
366
171
181 67
23% 29 30
38% 9 39 54
35% 77 47 24 32 8
940
224 15
68% 57
358
112 34 92
2999 42
80% 11 17
482
94% 66
96%
57%
92%
51%
95%
24%
40%
74%
88%
99%
Source: Wikipedia, Adrichel
During this time only CEDEFOP-data was available, so we had to measure the quality of the date by itself
Variables for Prediction: YearGrabDate + YearExpireDate + Experience + Salary
In R there were some problems reading the CSV: Only entries out of 2.1 millions
© Federal Statistical Office of Germany (Destatis)

10. Using Python on Federal Employment Agency data
Used Packages: NLTK and Sklearn
Algorithm: Multinomial Naive Bayes Classifier, Accuracy: 85 % A B C D E F G H I J K L M N O P Q R S T U
446 19 21 48 14 17 1 5 33 4 2
73% 9 16 6 3
17% 15
8866
539
597 98
221
172 95
728 7
78% 24 51 11
36% 37 45 10 30
55%
231
6711 88 52 8 90
220
90%
673
294
8598
261
223
254
112
400 53 23
79% 69
152
3103 40
115
201
82% 22 39
5747 18 29 67
96% 79 28 46
1879 44
87% 32
478
77% 38
194
51%
435 12
154
399 97
319
4352
242
299
66% 55
524
695
260
190
240
286 27
267
25618 13
347 20
954
89%
848
74% 35
107
9747
105
304
58% 71 64 60
111
146
1393
69%
12% 31
100%
80%
84%
83%
61%
92%
88%
Good data since it is provided manually
Only variable was job description, but we want to try the job title as well. Problem: Sometimes no job title
Used options: Stopwords in German, alpha=0.01, fit to prior was true
Tokenizing and discarding punctuation as well as counting only words that appears at least n-times didn‘t increase the accuracy
© Federal Statistical Office of Germany (Destatis)

11. Outlook Focus on signal words or dictionaries
ML in order to detect ads from employment agencies
Try out other packages and tools: FastText, Tensorflow, etc.
Cooperate with collleagues in our institute on our newly built ML platform
© Federal Statistical Office of Germany (Destatis)

12. France
Maxime Bergeat

13. Methodology
Data received from National Employment Agency (Pôle emploi)
~140 websites
Subset of job ads
Always including job description and coded occupation (no job title )
Variables used as predictors
Words from job description (input is basically a document-term matrix)
Selection of words is based on term frequencies of words.
Mostly Python sklearn library
Goal: classification of occupation (22 categories)
March 2018
Cedefop meeting

14. Application #1
~1500 words used as predictors (words present in > 0.5% of all documents)
Different methods tried for classification
Logistic Regression
Gradient boosting
Random Forests with bagging
Basic neural network
Support vector machine
Here: results presented for
Logistic regression
With L1 penalty
Evaluation of results: train and test datasets (80% / 20%)
Global accuracy: 70%
March 2018
Cedefop meeting

15. Application #2 – Precision matrix
March 2018
Cedefop meeting

16. Application #3 – Recall matrix
March 2018
Cedefop meeting

17. Future work Improve selection of words used as predictors
Choose words which are occupation-specific
Improve tuning of parameters used in predictive methods
Improve evaluation process -> Cross-validation
First results seem good…
… Global accuracy of 80% with selection of occupation-specific words for predictors and basic tuning of parameter (for penalty) with random forests
(words present in > 2% of documents for at least one occupation category)
First experiments show that cross-validation does not improve a lot robustness of evaluation results (k-fold cross validation).
To be continued…
March 2018
Cedefop meeting

18. Belgium
Thomas Delclite

19. Machine Learning and prediction of NACE code by job description
ESSnet Big Data, Milan, 03/2018

20. Using R package on administrative data
Sensitivity
Specificity
Package RTextTools

21. Using R package on administrative data
Package RTextTools

22. Issues Predictions for three languages Application to webscraping data
Quality of webscraping data to assess variations of job vacancy

23. Discussion SIC classification in Cedefop data? Why is this useful?
Boundary till which this works
BG – companies house,

24. Thank you