2. Alerk Amin Marika Puumala
Questasy
A web application for online data dissemination based on DDI 3
Alerk Amin
Marika Puumala
11/23/2018

3. Contents Background & requirements Solution: Questasy & DDI 3
Contents
Background & requirements
Solution: Questasy & DDI 3
Demonstration
11/23/2018

4. Background CentERdata LISS panel Online panel of ~8000 individuals
Background
CentERdata
LISS panel
Online panel of ~8000 individuals
Free survey data & data collection
Diverse questionnaires each month (~30 mins in total)
Disseminate all data
CentERdata is a research institute located in Tilburg in the Netherlands. We’re not a data-archive, but a couple of years ago we received a large subsidy from the Dutch science foundation (NWO) to establish a new online panel, called the LISS panel (Longitudinal Internet studies in the Social Sciences). This panel is based on a probability sample (it’s not a free access panel) and it consists of about 8000 individuals.
The main idea of the panel is to make it easier for researchers to obtain survey data. Researchers, not only from the Netherlands but also abroad, can either use existing data from the panel or they can apply for conducting their own survey in the panel. In both cases everything is free of charge.
Each month, we conduct several surveys in the panel, for about 30 minutes per month per respondent in total. In principle, we disseminate all data that are collected in the panel.
11/23/2018

5. Diversity of Surveys Various topics Single wave studies
Diversity of Surveys
Various topics
Single wave studies
e.g. European Value Survey (partial)
Longitudinal studies
e.g. LISS Core Study
Varying data collection frequency
The topics of the surveys can vary a great deal, and we have both single-wave and longitudinal studies. The frequency of the waves of the longitudinal studies can vary a lot.
For example, we collect the background variables every month. But on the other hand, the LISS Core Study, which consists of 8 different modules (Health; Religion and Ethnicity; Politics and Values; Social Integration and Leisure; Personality; Work and Schooling; Economic Situation), is repeated once a year.
11/23/2018

6. Objectives for Data Dissemination
Objectives for Data Dissemination
Previously
No dissemination tool or standard
Data delivered ‘ad hoc’ in SPSS files + Word codebooks
Goal
Disseminate all data collected in the panel via a website (
Before we started with this project, we had no earlier dissemination tool or standard in use. We usually just delivered SPSS files and codebooks to our clients.
Since one of the goals of the new panel was to disseminate all data to the international research community, we wanted to be able to publish our data via a website ( and provide all related documentation via this website.
11/23/2018

7. Dissemination Requirements
Dissemination Requirements
Complete documentation in database
Studies, questions, variables, …
Longitudinal/multi-measure studies
Public and restricted access + administrator views
Advanced search
Most importantly, we wanted to provide full documentation on the website, more than we usually documented in codebooks.
We needed a storage for not only the data files but also all metadata, and wanted to publish metadata not only on study level, but also about questions and variables. Next to this we wanted to include information about the data collection, concepts that were being studied, etc.
Since we have all kinds of surveys in the panel, the system had to support also longitudinal and multi-measure studies.
While all metadata could be freely accessibly to the public, we wanted to restrict the access for downloading the actual data files to people who had registered for this.
We wished to create an intuitive interface for internal administrators to manage the system.
Since we expected to publish a lot of studies, it had to be easy to search within the database.
While all surveys are conducted in Dutch, we want the data to be accessible for international researchers and thus document everything in English as well. Therefore the system needed to support multiple languages.
We wanted to use an existing, international standard to structure all the metadata.
Multiple languages
Use an international standard -> DDI 3
11/23/2018

8. Solution: Questasy Developed by CentERdata Web application
Solution: Questasy
Developed by CentERdata
Web application
Manage and distribute survey data and metadata
Web application – back-end database. Web pages are generated using data from the database and delivered to the end user’s web browser.
Works stand-alone or integrates into another website (
Manage & disseminate survey metadata
11/23/2018

9. Questasy Website Integration
Questasy Website Integration
/lissdata
/dataarchive
Static
Web Pages
(External)
Researcher
Interface
(External)
Administrator
Interface
(Internal)
Questasy integrates seamlessly into the LISS Data Website
Static web pages are handled by a Content Management System
Information about the board, panel recruitment process, submitting proposals, etc.
The Data Archive pages are handled by Questasy
Hyperlinks allow easy navigation between all of the pages
Database
Questasy
11/23/2018

10. Questasy Implementation
Questasy Implementation
PHP / MYSQL
MVC Architecture
Model based on DDI 3
Search
CakePHP
Questasy written in PHP
Works with many different operating systems, web servers & databases
We use Apache on Windows and MySQL on Linux. Future move to Linux webserver
Model-View-Controller framework
Database Model
Database tables correspond to DDI elements
Relationships between tables correspond to DDI heirarchy and references
11/23/2018

11. DDI 3 Relationships in Model
DDI 3 Relationships in Model
Simplified example of how we have mapped the DDI XML to DB tables
We looked at the relationships between all DDI elements
Normal DDI hierarchy (eg. Question Items belong to a Question Scheme)
References (eg. Question Constructs contain references to Question Items)
Question Item is part of a Question Scheme, by DDI hierarchy
Question Item has a Response Domain. This might be a simple domain, or a reference to a Code Scheme or Category Scheme
Question Constructs have a single Question Reference
Variables and Concepts can refer to multiple Question Items
11/23/2018

12.
DDI 3 in Database
The various DDI elements are now shown as database tables
The relationships are all captured
n:1 and 1:n relationships use a foreign key
n:m relationships use a join table
11/23/2018

13. Search Researcher interface Sphinx search engine 23-11-2018
To improve the researcher interface, we need a powerful search engine
Researchers might be searching for specific variables, concepts, or words/phrases in question text
We have integrated the Sphinx search engine into Questasy
Full-text indexing, directly against the database. Bypasses the DDI model
Stemming
Breaks down words into their base form
Jump, jumps, jumped, jumping all map to the stem “jump” in English
Ranking & relevance of search results
Incredibly fast. Much faster than native SQL searching
11/23/2018

14.
Demonstration
11/23/2018