1. Meeting of TF1 "Input Harmonisation" 24-25 April 2017
Agenda Item 2.1
Main conclusions from the test results
Eurostat

2. Overview Tests run before 2015 Results from 2015 tests
Conclusions
Eurostat

3. Tests run before 2015
2012 – first tests on the Model Questionnaire (MQ) proposed by TF HMEU: in DE, RO and CZ
2013 – second test exercise on modified MQ
Main problems related to translation and wording
ES: length and complexity of 1st question of "At work"
AT: results positive; "At work" similar to proposed FC
HU: any particular problems with "At work"; better catch marginal employment
NL: proposals for splitting answers, adding explanatory texts and rephrasing
SE: problems with the 1st question of "At work"; "work" difficult to be fully covered by just 1 question
Eurostat

4. 2015 EU grant - Romania
Test on: V2 vs. V3 vs. current national questionnaire
Method: field test
Main results:
Adjusting the results for the new classification of self-producers, comparison shows that V3 better catchs marginal employment
“without doubts, V3 was the preferred version due to its simplicity, shortness and ease of administration. Questionnaire V2 was considered too complicated, with too many filters and questions that are perceived as useless or even annoying”
Eurostat

5. 2015 EU grant - Spain
Test on: V2 vs. current national questionnaire (V3)
Method: cognitive interviewing, focus group
Main results:
“we can conclude that no evidence has been found that this formulation from the “At work” section works less effectively than the EPA formulation”
In the other sub-modules modifications introduced after former tests work generally better
Translation and wording should be very well studied at national level
Eurostat

6. 2016 without EU grant - France
Test on: V2 vs. current national questionnaire (V3)
Method: field test
Main results:
Even if no significant differences in terms of resulting employment rate, clear preference for the V3 approach
Question on small jobs works better if used as catching-up question with V3
Better to add response items to ‘Absences’ and ‘Job search methods’
French questionnaire works better for the order of the first 2 questions of ‘Search for work’
Eurostat

7. 2016 EU grant – version to be tested
The version tested in the sub-action 4 of the EU grant exercise is the one developed after the discussion that took place at the June 2016 LAMAS meeting (V4)
Questions sequence determined through a written consultation after the June 2016 LAMAS meeting
This version has the form of a flowchart (FC)
The FC is anyway complemented by a MQ that should not be considered as compulsory, but as guidelines for the translation in national language
All NSIs decided to test the MQ as provided
Eurostat

8. 2016 EU grant - Germany
Method: cognitive interviewing and expert review
Main results:
Sub module "At work": main issue concerned the 1st question The term “pay or profit” very difficult to translate; preferred “no matter if employee or self-employed”. In contrast to earlier pretests, no signs that this question could fail to cover small or side jobs. All people indicated the correct status according their situation. The 2nd question shows no problem. The 3rd question should be complemented by the word “paid” in German.
Sub module "Absences", no major problems were found
Eurostat

9. 2016 EU grant - Finland Method: cognitive interviewing and field test
Main results:
Cognitive test: 1st question of ‘At work’: all respondents seemed to understand the question correctly. Recommendation is to change even if for one hour into at least one hour
Field test: 1st question of ‘At work’: confusing mainly due to the sentence “even if it was only for one hour”. No feedback from interviewers asked for AW2 and no problems reported with AW3. Question AW4 often felt as burdening, but 0.8% of population has been retrieved to employment thanks to this question.
Eurostat

10. 2016 EU grant - Latvia Method: cognitive and personal interviewing
Main results:
1st question of ‘At work’ showed language problems, translation of pay and profit is problematic; rewording needed. Terms 'pay or profit 'and time reference should be better explained; some explanations help in receiving correct answers. But, AW1 works better than the current LV question.
AW2 works well. AW3 is too long. AW4 is clear but often considered redundant.
Suggestions: add examples; split some questions to get more information on occasional and short-term works; mention for the reference period “during the week from [date] to [date]” (without "Monday" and "Sunday")
Eurostat

11. Conclusions Outcomes are generally positive
Flexibility is sometimes needed as regards the number of questions to cover one concept
Other problems can be solved at national level by fine tuning the wording
Suggestions include examples and explanations, given by interviewers or written in questionnaire
Sequence of questions, information contents, routing in questionnaire and filtering did not show any problems
Recommendations taken into account to build the new simplified FC approach
Eurostat

12. TF1 is invited to take note of:
the results of the tests, which can serve as working tool for the detailed discussion on the sub modules in following agenda items
Eurostat