Presentation is loading. Please wait.

Presentation is loading. Please wait.

Source:

Similar presentations


Presentation on theme: "Source:"— Presentation transcript:

1 Source: Best Practices in Survey Development for the Evaluation of Student Gains from Entrepreneurship Programs and Classes Presented by: Jessica Menold

2 2000 entrepreneurship courses are offered nationwide and evaluating the impact of this education is critical Source:

3 Today we will review a set of nine metrics, used to evaluate eleven entrepreneurship assessment instruments

4 Content relevance and representativeness refer to the range and limits of content coverage Content to be assessed Content relevance and representativeness set the boundaries of the construct domain to be assessed by an instrument or measure

5 According to Messick’s theory, validity can be differentiated into six aspects Content Substantive Structural Generalizability External Consequential Source:

6 Downing and Haladyna advocate for the use of qualitative evidence in the development test items Content definition Item TryoutItem Verification Test Specifications

7 Content definition links the construct that is to be measured with the test specifications and items Three metrics were derived for Content Definition: 1.Clearly defined boundary of the construct(s) of the instrument or survey 2.Definition of the construct(s)based on literature, job task analysis, and/or expert review 3.Documentation of methods used to select the content presented 5

8 Test specifications tie the content domain with the test items and define the segments of the domain to be assessed Three metrics were derived for Test Specifications: 4.Detailed rationale or process for breakdown of proportions of content domain to be represented by test items 5.Content areas and more specific content within these areas clearly stated 6.Agreement on proportional breakdown by panel of researchers or content experts

9 Item verification can help ensure that items reflect content that is representative and relevant to the domain of interest Two metrics were derived for Item Verification: 7.Initial items based on literature review, previous instruments/survey, field experience 8.Items verified through panel of content experts on both relevance and representativeness to the content domain relative to the pre-defined construct definition

10 Item “tryout”, or pretesting, is a method used to evaluate the cognitive processes that items evoke from test takers One was derived for Item Tryout: 9.Pretesting of items using think aloud or other protocols to ensure the test takers engage in the expected cognitive processes

11 We used these metrics as a lens to examine eleven instruments used to evaluate entrepreneurial constructs 1.Entrepreneurial self-efficacy scale 2.NCIIA Entrepreneurship Inventory 3.Proactive Personality Scale 4.Entrepreneurial self-efficacy scale 5.General enterprising tendency 6.Individual entrepreneurial orientation 7.Student entrepreneurial orientation 8.Entrepreneurial behavior inventory 9.Tolerance for Ambiguity Scale 10.Engineering Entrepreneurship Survey 11.Entrepreneurial mindset index

12 We chose the NCIIA Entrepreneurship Inventory as an illustrative example to review the use of each metric Clearly defined boundary of the construct(s) of the instrument or survey Definition of the construct(s)based on literature, job task analysis, and/or expert review Detailed rationale or process for breakdown of proportions of content domain to be Content areas and more specific content within these areas clearly stated Items verified through panel of content experts Initial items based on literature review, previous instruments/survey, field experience Agreement on proportional breakdown by panel of researchers or content experts Documentation of methods used to select the content presented5 Pretesting of items using think aloud or other protocols to ensure the test takers engage in the expected cognitive processes

13 The selected instruments are strongest in content definition and weakest in test specifications/blueprint creation Met Unmet Met Content definition Item TryoutItem Verification Test Specifications

14 In summary, two key improvements for entrepreneurship instruments include: The adoption and use of test specifications and test blueprints The use of content experts to verify item representativeness and relevance

15 Thank you


Download ppt "Source:"

Similar presentations


Ads by Google