Presentation is loading. Please wait.

Presentation is loading. Please wait.

A RIZONA IN THE G REEN E CONOMY Lessons Learned in Designing an Effective Business Survey Session 4 April 26, 2011.

Similar presentations

Presentation on theme: "A RIZONA IN THE G REEN E CONOMY Lessons Learned in Designing an Effective Business Survey Session 4 April 26, 2011."— Presentation transcript:

1 A RIZONA IN THE G REEN E CONOMY Lessons Learned in Designing an Effective Business Survey Session 4 April 26, 2011

2 C REDITS AND D ISCLAIMER 2 This report was funded by a grant awarded by the U.S. Department of Labors Employment and Training Administration. The report does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to such information, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership. State of Arizona has the copyright to this report. Internal use by an organization and/or personal use by an individual for non-commercial purposes is permissible. All other uses require the prior authorization of the copyright owner.

3 S ESSION O VERVIEW : P REPARING FOR A S URVEY Defining the survey purpose Identifying survey recipients Defining key concepts Designing the survey instrument Preparing the survey cover letter


5 4 C OMMON A NALYTIC R ESEARCH M ETHODS 1. Archival Use of available administrative records to analyze the past or forecast the future Observations used from a time period before a research question is posed 2. Experiment/ Quasi-experiment Pilot project compared against the real world Controls for a single variable Helps to factor out other explanations for the occurrence of a few observations 3. Case study In-depth examination of a particular story Review multiple influences Develop theories about possible causal explanations 4. Survey Primary data collection Surveys typically focus gathering a larger number of responses that can be analyzed using statistical methods Interviews and focus groups that can be de-constructed and easily compared (often provide in-depth companion info for surveys) 5

6 D EFINING THE SURVEY PURPOSE : A NSWERING EXPLORATORY QUESTIONS Gathering and reporting facts or opinions Examples from the Arizona Green Survey: How many Arizona companies have green economic activity? What proportion of Arizonas employment is dedicated to green economic activity? How rapidly are employers planning to add green jobs? What are the wages provided to workers in these green jobs?

7 D EFINING THE SURVEY PURPOSE : T ESTING SPECIFIC HYPOTHESES Assessing whether patterns emerge as expected Examples from the Arizona Green Survey: In which industries does Arizona have a high concentration of green activities? How much faster do companies with green jobs grow relative to all other companies? How much more do jobs requiring green skills pay than other jobs in the same occupation?

8 B IASES THAT MUST BE UNDERSTOOD The selection of questions reflect the biases and perspectives of the researchers The questions being posed often include inherent biases Answers to survey questions reflect the perspective of respondent In business surveys, multiple respondents may answer differently The selected sample may not necessarily reflect the general population Difficulty in getting accurate responses from larger and smaller businesses

9 I DENTIFYING S URVEY R ECIPIENTS Who are the businesses that you are surveying? 9

10 D ATABASES FOR BUSINESS LISTS Quarterly Census of Employment and Wages Availability Suitability of contacts Partners and stakeholders Availability Reliability Proprietary vendors (e.g., InfoGroup; D&B) Cost Accuracy and completeness 10

11 S ELECTING A VALID SURVEY FRAME Determining how to narrow the list Geography, industry, size, business age, available contact info, etc. Cleaning up the list Time, access to information, accuracy of alternative data sources Determining appropriate contacts within the company Related to questions being asked 11

12 G ETTING THE R IGHT S AMPLE S IZE Identify sub-groups of survey recipients (stratification) Identify potential biases within sub-groups and the need for oversampling some groups Estimate expected response rates and overall sample size required Establish quotas for sub-groups in the sample (balancing resources with desire for of statistical validity) Adjust the sample size to reflect sub-group quotas as well as expected overall response rate 12


14 W HAT ARE YOU TRYING TO MEASURE ? Operationalizing key concepts Firm vs. establishment Jobs Employment Wages Identifying terms with shared definitions Understanding the limits What can you expect respondent to know? Selecting appropriate time periods The power of memory The limits of projecting the future (OR the problem with hypothetical questions) 14

15 EXAMPLE: D EFINING G REEN J OBS Examples Of Green Activities Making Goods or Providing Services Improving Processes in Existing Goods or Services Energy generationProducing renewable energy Improving fuel efficiency Energy efficiency in buildings, equipment, appliances, or vehicles Building long-lasting batteries Implementing smart grid technologies Pollution reduction/removal, green-house gas reduction, recycling and reuse Producing industrial scrubbers or generating nuclear energy Managing energy proactively Natural resources conservationOrganic farmingConserving soil, water, or wildlife Environmental compliance, education and training, and public awareness Training in regulatory compliance 15

16 A N E XAMPLE : R ENEWABLE E NERGY What are key elements of the renewable energy industry? Which segments do we focus on? How do we handle nuclear energy or other segements? What are related industries? …activities? …occupations? How do you handle suppliers that might not be unique to solar such as… Maintenance and repair? Accountants and attorneys? Educational institutions Utilities? Investors? What occupations would want to include? 16

17 A NOTHER EXAMPLE: O PERATIONALIZING I NNOVATION IN R ENEWABLE E NERGY Defining innovation Innovationintroducing something newmostly derives from technological advances that ultimately lead to productivity and prosperity gains. Communities must provide an appropriate foundation to realize those gains, including: A strong educational system A solid physical infrastructure, A base of research and development activities, Access to financial resources, and other factors. Science and engineeringin the educational system and in the workforceare of particular importance. 17 Source: Adapted from Arizona Indicators, Morrison Institute for Public Policy

18 S ELECTED W AYS TO M EASURE I NNOVATION Human capital University entrance exam scores Graduate students and postdoctorates Employed science, engineering & health doctorate holders Patents granted Financial capital Research and development Innovation grants Venture capital High tech employment High tech industries High tech occupations 18 Source: Arizona Indicators, Morrison Institute for Public Policy

19 H OW COULD FURTHER RESEARCH HELP US BETTER UNDERSTAND R ENEWABLE E NERGY RELATED TO …? …Arizonas educational system? …the states physical infrastructure? …research and development activities? …access to financial resources? …the workforces preparation for emerging jobs in this area? 19

20 D ESIGNING A S URVEY I NSTRUMENT Creating and aggregating information about a large number of respondents 20

21 5 A LTERNATIVE I NTERVIEW M ETHODS Interview type AdvantagesDisadvantages Personal interview Stronger commitment to completing the interview. Greater confidence in finding the right recipient Longer interviews tolerated More costly Greater potential for creating a non-representative sample Telephone surveys Quicker turn-around Wide availability of telephone numbers Availability of CATI to address complex questions Role skilled interviewers to elicit more complete answers Negative perception of surveyors as telemarketers Challenges of getting past gatekeepers (e.g., administrative assistants, caller ID) Mail surveys Relative expense per response Ability to use only names and address to reach respondent Can use photos Answer at respondents leisure Viewed as less intrusive Survey takes longer Response rates lower Respondents are more likely to have something to say Less ability to correct misunderstanding of questions 21

22 5 A LTERNATIVE I NTERVIEW M ETHODS ( CONTINUED ) Interview type AdvantagesDisadvantages surveys Inexpensive with quick turnaround Simple Allows attachments Cannot handle complex logic Requires having addresses Perceptions about viruses and spam Potential missed messages due to junk mail filters Difficult to manage sample due to ease of forwarding Loss of anonymity Response bias/inability to generalize Web surveys High speed in getting responses Low cost Use of to generate responses Flexibility in creating complex logic More responsive to sensitive questions Greater ability to use open-ended questions Respondent inputs data Increase survey quitting Restricting response to the sample Even greater difficulty in generalizing results 22

23 T IPS IN SURVEY DESIGN Keep it Short and Simple (KISS) Keep questions simple and specific Start with an interesting question What makes a question interesting? Ask answerable questions Taking care with question skipping Recognize that surveys reveal a story about the research Encourage a logic flow for questions Understand how question order might affect response 23

24 T IPS IN W RITING Q UESTIONS Allow dont know, not applicable, or OTHER options Avoids skipping and early termination Use forced-choice questions whenever possible Create appropriate answer categories How many categories? Ensure that answer options are complete AND mutually exclusive When to use open-ended questions Provide context for specific questions or survey You get 1 or 2 shots at most Recognize respondents desire to please Will affect responses Omit the middle alternative and focus on intensity Provide option for no opinion or not applicable Watch for double-barreled or negative questions? About people or policies that might be controversial Using words that unintentionally evoke emotion 24


26 K EY G OALS IN P RE -T ESTING THE S URVEY I NSTRUMENT Variation Meaning Task difficulty Respondent interest and attention Flow and naturalness of the sections The order of questions Impact of skip patterns Timing Overall respondent interest and attention Respondent reaction Specific QuestionsInstrument as a Whole 26


28 T IPS FOR A G REAT C OVER L ETTER Keep the letter short Personalize it to an individual whom you want to respond Tell the recipient how they could benefit directly (or pay them for their time) Tell them how long it will take and BE HONEST Give them options for responding Provide a deadline within a week of receipt Say something memorable (funny is good, but not offensive) Use the right logo Careful about who signs the letter 28



31 W ORKING G ROUP EXERCISE : D ESIGNING A B USINESS S URVEY ON I NNOVATION IN R ENEWABLE E NERGY 1. Issues to explore: How many Arizona renewable energy companies are innovative? What are the key characteristics of innovative renewable energy companies? 2. Hypotheses to test: Innovative companies were much more likely to add jobs likely to add jobs during the recession. Innovative companies need different kinds of skills and training than do other renewable energy companies. What are the key talking points in the letter? Who should sign it and why? Design an Instrument – 2 survey design groups Prepare a Cover Letter -- for each group 31 Who should get the survey? What subgroups are there? How should the survey be adapted to the subgroups? Determine Sample Frame -- for each group


Download ppt "A RIZONA IN THE G REEN E CONOMY Lessons Learned in Designing an Effective Business Survey Session 4 April 26, 2011."

Similar presentations

Ads by Google