1. Using Shiny to Efficiently Process Survey Data Carl Ganz, Akbar Akbari Esfahani, Hongjian Yu & Ninez Ponce UCLA Center for Health Policy Research
Company or University Logo
Problem
Example 1: Public Use File
Example 2: Upcoding
Example 3: Metadata
Content experts must thoroughly review our data for sensitive information before public use files (PUF) are created.
This requires creating new, less sensitive variables by collapsing, top/bottom coding, and grouping variables.
These new variables need to be generated in SAS, but content experts are not comfortable with SAS.
In the past, programmers would generate frequencies for content expert who in turn would recommend new variables via .
With Shiny we have custom GUI that allows content experts to interactively generate new variables, and the SAS code required to create them.
At the UCLA Center for Health Policy Research, we continuously process the annual California Health Interview Survey (CHIS). As is the case with many large surveys, CHIS relies on the work of statisticians along with a variety of content experts from public health, sociology, epidemiology, and other areas. Many tasks require extensive back and fourth between the statisticians, and the content experts. The majority of this work is done via Excel, because it is accessible to a variety of stakeholders. Typically, any input required from the content expert (i.e. the names of the requested variables) is inputted to excel. A Statistician then writes a SAS program to read the excel table, and generate the required output. This non-interactive workflow is slow, and error-prone, because the content experts get delayed feedback.
Many questions from our survey are open response meaning there are unaccountably many possible answers.
Content experts must review the open responses, and categorize the answers into one of finitely many categories.
Many responses can be matched to similar responses in the past, but others require human review.
In the past, content experts would manually review, and categorize in Excel.
With Shiny we created a custom interface where content experts can harness R’s text-mining capabilities to suggest similar upcodes.
For each year’s survey, we generate metadata for each variable, including label, formats, variable type, etc.
In the past, the metadata was managed in Excel because it is accessible to a wide audience.
Excel presented problems because data was untidy, and there were referential integrity issues across years.
SQL is a logical alternative for solving these issues, but it is much less user friendly for non-programmers.
With Shiny, we developed an interface that allows content experts to download SQL data to Excel, make changes in Excel, and then upload those changes back to SQL.
App A
App B
App C
Solution
Ideally, content experts would work in an interactive environment that validates their work and gives them the results immediately. Luckily, R’s web-framework Shiny makes it easy to create such tools. To see examples visit: