23-11-2018

Alerk Amin Marika Puumala 23-11-2018 Questasy A web application for online data dissemination based on DDI 3 Alerk Amin Marika Puumala 11/23/2018

Contents Background & requirements Solution: Questasy & DDI 3 23-11-2018 Contents Background & requirements Solution: Questasy & DDI 3 Demonstration 11/23/2018

Background CentERdata LISS panel Online panel of ~8000 individuals 23-11-2018 Background CentERdata LISS panel Online panel of ~8000 individuals Free survey data & data collection Diverse questionnaires each month (~30 mins in total) Disseminate all data CentERdata is a research institute located in Tilburg in the Netherlands. We’re not a data-archive, but a couple of years ago we received a large subsidy from the Dutch science foundation (NWO) to establish a new online panel, called the LISS panel (Longitudinal Internet studies in the Social Sciences). This panel is based on a probability sample (it’s not a free access panel) and it consists of about 8000 individuals. The main idea of the panel is to make it easier for researchers to obtain survey data. Researchers, not only from the Netherlands but also abroad, can either use existing data from the panel or they can apply for conducting their own survey in the panel. In both cases everything is free of charge. Each month, we conduct several surveys in the panel, for about 30 minutes per month per respondent in total. In principle, we disseminate all data that are collected in the panel. 11/23/2018

Diversity of Surveys Various topics Single wave studies 23-11-2018 Diversity of Surveys Various topics Single wave studies e.g. European Value Survey (partial) Longitudinal studies e.g. LISS Core Study Varying data collection frequency The topics of the surveys can vary a great deal, and we have both single-wave and longitudinal studies. The frequency of the waves of the longitudinal studies can vary a lot. For example, we collect the background variables every month. But on the other hand, the LISS Core Study, which consists of 8 different modules (Health; Religion and Ethnicity; Politics and Values; Social Integration and Leisure; Personality; Work and Schooling; Economic Situation), is repeated once a year. 11/23/2018

Objectives for Data Dissemination 23-11-2018 Objectives for Data Dissemination Previously No dissemination tool or standard Data delivered ‘ad hoc’ in SPSS files + Word codebooks Goal Disseminate all data collected in the panel via a website (www.lissdata.nl) Before we started with this project, we had no earlier dissemination tool or standard in use. We usually just delivered SPSS files and codebooks to our clients. Since one of the goals of the new panel was to disseminate all data to the international research community, we wanted to be able to publish our data via a website (www.lissdata.nl) and provide all related documentation via this website. 11/23/2018

Dissemination Requirements 23-11-2018 Dissemination Requirements Complete documentation in database Studies, questions, variables, … Longitudinal/multi-measure studies Public and restricted access + administrator views Advanced search Most importantly, we wanted to provide full documentation on the website, more than we usually documented in codebooks. We needed a storage for not only the data files but also all metadata, and wanted to publish metadata not only on study level, but also about questions and variables. Next to this we wanted to include information about the data collection, concepts that were being studied, etc. Since we have all kinds of surveys in the panel, the system had to support also longitudinal and multi-measure studies. While all metadata could be freely accessibly to the public, we wanted to restrict the access for downloading the actual data files to people who had registered for this. We wished to create an intuitive interface for internal administrators to manage the system. Since we expected to publish a lot of studies, it had to be easy to search within the database. While all surveys are conducted in Dutch, we want the data to be accessible for international researchers and thus document everything in English as well. Therefore the system needed to support multiple languages. We wanted to use an existing, international standard to structure all the metadata. Multiple languages Use an international standard -> DDI 3 11/23/2018

Solution: Questasy Developed by CentERdata Web application 23-11-2018 Solution: Questasy Developed by CentERdata Web application Manage and distribute survey data and metadata Web application – back-end database. Web pages are generated using data from the database and delivered to the end user’s web browser. Works stand-alone or integrates into another website (www.lissdata.nl) Manage & disseminate survey metadata 11/23/2018

Questasy Website Integration 23-11-2018 Questasy Website Integration www.lissdata.nl /lissdata /dataarchive Static Web Pages (External) Researcher Interface (External) Administrator Interface (Internal) Questasy integrates seamlessly into the LISS Data Website Static web pages are handled by a Content Management System Information about the board, panel recruitment process, submitting proposals, etc. The Data Archive pages are handled by Questasy Hyperlinks allow easy navigation between all of the pages Database Questasy 11/23/2018

Questasy Implementation 23-11-2018 Questasy Implementation PHP / MYSQL MVC Architecture Model based on DDI 3 Search CakePHP Questasy written in PHP Works with many different operating systems, web servers & databases We use Apache on Windows and MySQL on Linux. Future move to Linux webserver Model-View-Controller framework Database Model Database tables correspond to DDI elements Relationships between tables correspond to DDI heirarchy and references 11/23/2018

DDI 3 Relationships in Model 23-11-2018 DDI 3 Relationships in Model Simplified example of how we have mapped the DDI XML to DB tables We looked at the relationships between all DDI elements Normal DDI hierarchy (eg. Question Items belong to a Question Scheme) References (eg. Question Constructs contain references to Question Items) Question Item is part of a Question Scheme, by DDI hierarchy Question Item has a Response Domain. This might be a simple domain, or a reference to a Code Scheme or Category Scheme Question Constructs have a single Question Reference Variables and Concepts can refer to multiple Question Items 11/23/2018

23-11-2018 DDI 3 in Database The various DDI elements are now shown as database tables The relationships are all captured n:1 and 1:n relationships use a foreign key n:m relationships use a join table 11/23/2018

Search Researcher interface Sphinx search engine 23-11-2018 To improve the researcher interface, we need a powerful search engine Researchers might be searching for specific variables, concepts, or words/phrases in question text We have integrated the Sphinx search engine into Questasy Full-text indexing, directly against the database. Bypasses the DDI model Stemming Breaks down words into their base form Jump, jumps, jumped, jumping all map to the stem “jump” in English Ranking & relevance of search results Incredibly fast. Much faster than native SQL searching 11/23/2018

23-11-2018 Demonstration www.lissdata.nl 11/23/2018

23-11-2018