Presentation is loading. Please wait.

Presentation is loading. Please wait.

Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,

Similar presentations

Presentation on theme: "Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,"— Presentation transcript:

1 Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director, Survey Research Operations Survey Research Center Institute for Social Research University of Michigan

2 SRO – SRC – ISR Outline Whats different about an academic survey research center? Introduction to survey research operations Project management 101 Communication The triple constraints in project management A note about quality Communication Can we make it more complex? 2

3 SRO – SRC – ISR Whats different about an academic survey research center? 3

4 SRO – SRC – ISR Institute for Social Research 4

5 SRO – SRC – ISR Survey Research Center Established in 1946 Productivity: In the past ten years faculty and staff have: produced 59 books, 232 book chapters, and 1116 scientific articles produced datasets that have been downloaded by thousands of researchers worldwide resulting in a new scientific article every three days submitted ~3 proposals per week achieved higher than average success rate on proposal funding from NIH and NSF 5

6 SRO – SRC – ISR Survey Research Center Research Volume: ISR approximately ~$100M + per year in sponsored research; SRC about 70-80% of volume Smallest of large-scale survey research organizations but largest of academic-based ~80% from US Federal government, predominately National Institutes of Health, National Science Foundation Investigator initiated grants (very few contracts) Increased flexibility in design and execution 6

7 SRO – SRC – ISR Survey Research Center Faculty and Staff 70 faculty, most with joint appointments in academic departments 275 staff ~up to 2000 interviewers ~100 students Teaching mission of University 7

8 SRO – SRC – ISR Educational Programs Summer Institute in Survey Research Techniques Michigan Program in Survey Methodology Masters PhD Certificate Joint Program in Survey Methods Joint Institute with Peking University 8

9 SRO – SRC – ISR Introduction to Survey Research Operations 9

10 SRO – SRC – ISR Survey Research Operations Complete about ~100 projects per year with staff of ~130 Process ~100 proposals per year Historically, ~ half of SRO clients are outside of ISR 10

11 SRO – SRC – ISR Types of Projects Interviewer-administered CAPI CATI (both centralized and decentralized) CASI and A-CASI as components of CAPI instruments Self-administered Web Mail Many projects use a combination of modes, attempting to optimize quality and cost 11

12 SRO – SRC – ISR Types of Projects Our volume is comprised of: large, national CAPI studies regional CAPI studies smaller telephone, mail or web-based studies Our organizational infrastructure accommodates large projects that can be modified for smaller or more specialized efforts 12

13 SRO – SRC – ISR Project Management 101 13

14 SRO – SRC – ISR Project Manager/Team Roles The project manager is the person responsible for accomplishing the project objectives The project team is comprised of the people who have assigned roles and responsibilities for completing aspects of the project 14


16 SRO – SRC – ISR Communication... 75 to 90 percent of the project manager's time on a project is spent communicating with one or more stakeholders (Corpedia) 16

17 SRO – SRC – ISR Project Management Managing a project includes: Identifying requirements Establishing clear and achievable objectives Adapting/balancing approach with various stakeholders Managing the team Balancing the competing demands (scope (includes quality), time & cost) 17

18 SRO – SRC – ISR Project Management Application of knowledge, skills, tools and techniques to project activities with following phases: Initiation Planning Risk assessment/ Plan B Example Executing and controlling Closing 18

19 SRO – SRC – ISR Stakeholders Individuals and organizations actively involved in the project, or whose interest may be affected by the project Project management teams job is to: Identify stakeholders Determine requirements and expectations Manage stakeholders influence 19

20 SRO – SRC – ISR Key Stakeholders Sponsor Client (if different from sponsor) Users (if different from above) Project management organization Performing organization Project manager and team members Respondents Example 20

21 SRO – SRC – ISR Stakeholder Influence 21

22 SRO – SRC – ISR Management Plan Purpose: Documents actions necessary to define, prepare, integrate, and coordinate project activities May include several subsidiary plans Progressively elaborated Multiple authors Version control important (hard to do after the fact) 22

23 SRO – SRC – ISR Management Plan Broad Outline Background Initiation activities Project planning/risk assessment Implementation and control Closing 23

24 SRO – SRC – ISR Launching a Project Project design also involves: Creating a team Developing a team Managing a team 24

25 SRO – SRC – ISR Team Management Complexities of managing resources: Delineate roles & responsibilities Acquire staff with needed skills Develop and build the team Manage individual performance 25

26 Task Team Model 26

27 SRO – SRC – ISR The Triple Constraints 27

28 SRO – SRC – ISR Constraint #1: Scope (read: Design) Scope planning Scope definition Work breakdown structure Scope control 28

29 SRO – SRC – ISR Design Specifications (1) Study Purpose Objectives Funding source(s) Timing Production plan milestones/deliverables Overall design summary Project Coordination Plan Lead team Task teams Working groups Communication/coordination 29

30 SRO – SRC – ISR Design Specifications (2) Sample Design Type (sample frame availability and coverage properties) Size Accessibility of population Goals (response rate, number of cases in various domains, etc.) Replicate design Responsive design 30

31 SRO – SRC – ISR Design Specifications (3) Mode Feasibility of mode Infrastructure Social desirability bias (what is sensitive) Literacy Interviewer safety Mixed/multi-mode – increasing in use Sequential – cheapest first Driven by coverage/cost 31

32 SRO – SRC – ISR Design Specifications (4) Data Collection Ethical consideration/review Confidentiality/privacy Informed consent Sensitive topics or vulnerable populations Age of majority Literacy Incentives/coercion Local laws, regulations Special cases: biomarkers 32

33 SRO – SRC – ISR Design Specifications (5) Data Collection (continued) Field Staffing Plan Assess local infrastructure Languages Sample assignment model Supervisory structure/reporting Production schedule Interviewer remuneration guidelines Mechanisms for reporting progress Result code definitions and response rate calculation guidelines 33

34 SRO – SRC – ISR Design Specifications (6) Instrument /Application Development Mode Languages Special printing requirements Hardware and software requirements Local adaptation requirements 34

35 SRO – SRC – ISR Design Specifications (7) Testing Pretests Applications/systems testing Sample requirements Training Plans Supervisor/Team Leader training Interviewer training Special training, e.g., biomarkers 35

36 SRO – SRC – ISR Design Specifications (8) Quality Checks Application requirements Training Paradata: dashboards, control charts Verifications/validations Editing and coding Data review Site visits 36

37 SRO – SRC – ISR Design Specifications (9) Data Processing Plans Coding Data entry Staffing Production schedule Mechanism for reporting Sample Post-processing Weights, nonresponse or other adjustments, imputation, variance estimation 37

38 SRO – SRC – ISR Design Specifications (10) Data Processing Quality Control Testing Training plans Supervisor training Editor guide DDE training Coding/Special function training 38

39 SRO – SRC – ISR Design Specifications (11) Materials Development (Deliverables) Datasets Sampling variables (weighting, nonresponse adjustments) Sample file (result codes to check response rate calculations) Call records Data/Application Respondent materials Coversheets Questionnaires/ applications Questionnaire supplements/aids (e.g., respondent booklets, event history calendars) Contact materials 39

40 SRO – SRC – ISR Design Specifications (12) Materials Development (Deliverables) - continued Training Materials (e.g., PowerPoint slides, exercises) Manuals (supervisor/interviewer, other) Agendas Location/organization Protocols Incident/problem reporting Technical support/back-up 40

41 SRO – SRC – ISR Design Specifications (13) Materials Development (Deliverables) - continued Codebook Reports/Quality Profile Sample design description Ethics review verification Translation reports Pretest reports Field progress reports Quality control audits Field procedures reports 41

42 SRO – SRC – ISR Constraint #2: Time Project Time Management Activity Definition Activity Sequencing Activity Resource Estimation Schedule Development Schedule Control 42

43 SRO – SRC – ISR Schedule Development and Control Identify schedule activities Establish optimum project work flow Determine resources needed Determine time needed for each activity Lay out activities in calendar time Keep track of progress, take corrective action 43

44 SRO – SRC – ISR Project Work Plan Example 44

45 SRO – SRC – ISR Constraint #3: Cost Project Cost Management Primarily concerned with the cost of resources needed to complete schedule activities 45

46 SRO – SRC – ISR Cost Monitoring Cost management processes include Cost estimating Cost budgeting Cost control 46

47 SRO – SRC – ISR Design Considerations in Tenders, Bids, and Contracts Purpose: to thoroughly document and communicate roles, responsibilities, design details, expectations and outcomes in order to adequately budget the survey effort Organizational Considerations Locus of control Balancing standardization and localization Balancing funding versus contractual obligations versus expertise Methodological, substantive, local expertise Legal expertise One Voice/Point of contact 47

48 SRO – SRC – ISR Budgeting and Bid Assessment Benchmark, where possible Seek multiple bids, if feasible Ask for project experience and references Ask for sample materials Ask for staff and organizational capacity Ask for facilities and equipment capacity Allow time for clarification and several iterations of bids 48

49 SRO – SRC – ISR Budgeting and Bid Assessment Potential Problems Sample frame access Difficulty with concept of probability sampling Inadequate quality control procedures Not familiar/sensitive to/barrier to nonresponse reduction techniques (number of attempts to contact households, refusal conversion techniques, inconsistent contact result coding schemes) Inadequate interviewer training/interviewer incentives Inadequate facilities (hardware, software, Web access, etc.) Inadequate budget detail Research traditions 49

50 SRO – SRC – ISR Budget Template 50

51 SRO – SRC – ISR Budget Template (continued) 51

52 SRO – SRC – ISR Contracting Specific to: Funding source Contracting organizations Employee contracts and remuneration policies Countries Type of contract Must have local legal advice Commercial versus academic versus government pros / cons 52

53 SRO – SRC – ISR One Mechanism to Control Scope and Costs: Responsive Design Interviewer labor dominates costs on most surveys Surveys inherently uncertain (e.g. estimated costs and completed interviews) Maximize cost benefit by adapting design features based on incoming survey process data (make data driven decisions) 53

54 SRO – SRC – ISR Responsive Design Implementation Process (Groves et al., 2009) 1) Pre-identify features potentially affecting costs and errors (risks) 2) Identify indicators 3) Wherever possible, devise benchmarks for each indicator 4) Create decision rules that alter features of the design if benchmarks are not met 5) Develop effective visual displays of indicators 6) When benchmarks are not met, alter the survey design based on the decision rules 54

55 SRO – SRC – ISR Creating decision rules: Example Risk: Interviewers unable to convince eligible respondents to participate, potentially increasing non response bias Indicator: Cumulative average interview cooperation rate by day by interviewer by area Benchmark: Similar study had a 71% interview cooperation rate after 3 weeks of data collection Decision rule: If not at a 70% coop rate by the end of the 3rd week, identify source of problem: is it systematic across all interviewers and sample types or limited to a few interviewers or sample types, area 55

56 SRO – SRC – ISR A note about quality 56

57 SRO – SRC – ISR Quality Management Components Quality Planning and Assurance Quality Control Quality Profile 57

58 SRO – SRC – ISR


60 SRO – SRC – ISR Communication Communications Planning Information Distribution Performance Reporting Manage Stakeholders 60

61 SRO – SRC – ISR Communication Frequent Detailed Mixed (face-to-face, e-mail, conference call, newsletters, Webcasts) Document Sharing Critical for timely review Version control 61

62 SRO – SRC – ISR Data Documentation eRoom 62

63 SRO – SRC – ISR Can we make this more complex? 63

64 SRO – SRC – ISR YES! We could try to do this across multiple countries… 64

65 SRO – SRC – ISR Mode of Administration 65

66 SRO – SRC – ISR Cross-national Considerations (1) Structure dynamics Leadership Levels Number of organizations Locus of control Level of control Experience at various levels Funding 66

67 SRO – SRC – ISR Cross-national Considerations (2) Degree of standardization: Often determined by level, flow, and source of funding Experience and research infrastructure at central and local levels Control of data Decisions need to be made at every stage of the survey lifecycle 67

68 SRO – SRC – ISR Cross-national Considerations (3) Laws/ethical standards Lingua Franca Time zones Terminology Research traditions and infrastructure Technology (Web: e-mail, VOIP, web casts, document sharing) 68

69 SRO – SRC – ISR Danke! 69

Download ppt "Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an Academic Survey Organization Beth-Ellen Pennell Director,"

Similar presentations

Ads by Google