1. The Test of Economic Literacy Content, Development,Uses William Walstad, Professor of Economics University of Nebraska-Lincoln Webinar: November 13, 2013

2. Objectives for Webinar Describe the TEL content Explain the TEL development Present the TEL national results Suggest ways to use TEL Help you obtain a TEL copy  Examiner’s Manual  Test forms

4. TEL Content For high school (grades 10-12) Multiple-choice test  45 items on each form  two parallel forms (A and B) Classroom test  Basic economics  AP/honors economics Based on CEE content standards

6. Form AForm B Standard with selected key conceptsItemsTotalItemsTotal 1.Scarcity, choice, productive resources1, 2, 33 3 2.Decision-making, marginal analysis4141 3.Economic systems and allocation5, 62 2 4.Economic incentives—prices, wage, profits,7, 82 2 5.Voluntary exchange and trade9, 102 2 6.Specialization and comparative advantage11, 122 2 7.Markets and prices13, 142 2 8.Supply and demand15, 16, 173 3 9.Competition18, 19, 203 3 10.Economic institutions21, 222 2 11.Money and inflation23, 24, 253 3 12.Interest rates26, 272 2 13.Labor markets and income28, 292 2 14.Entrepreneurship301 1 15.Physical and human capital investment31, 322 2 16.Economic role of government33, 342 2 17.Government failure, special interest groups351 1 18.Output, income, employment, and price lev. 36, 37, 38, 39, 40 5 5 19.Unemployment and inflation41, 422 2 20.Fiscal and monetary policy43, 44, 453 3 Total Number of Questions 45 TABLE 1: Distribution of TEL Items by Economics Standards

7. Item & Rationale ITEMRATIONALE 1.The opportunity cost of a new city park is the A.cost of staff and maintenance for the park. B.increased congestion from traffic around the park. C.best alternative use of resources given up for the park. D.lack of personal incentive for people to take care of a public park. The opportunity cost of producing a good or service is the next best alternative good or service that might have been produced with the same resources. In other words, opportunity cost refers to what is forgone once money or resources are used for a specific purpose. [1/4/5] [Code for bracket item: Standard / Grade Level / Benchmark (CEE, 2010)]

8. 1.The opportunity cost of a new public high school is the A.money cost of hiring teachers for the new school. B.cost of constructing the new school at a later date. C.change in the annual tax rate to pay for the new school. D.other goods and services that must be given up for the new school. The opportunity cost of a new city park is the A.cost of staff and maintenance for the park. B.increased congestion from traffic around the park. C.best alternative use of resources given up for the park. D.lack of personal incentive for people to take care of a public park. Parallel Items (A & B)

9. Cognitive Levels Knowledge (recognition and recall) Comprehension (understand the meaning) Application (apply concepts or doing analysis)

10. Cognitive Level Distribution Knowledge……….13% Comprehension……31% Application………56%

11. Table 1: Content & Cognitive LevelsForm A Standard with selected key conceptsKnowCompAppTotal 1.Scarcity, choice, productive resources 2, 313 2.Decision-making, marginal analysis 41 3.Economic systems and allocation mechanisms 5, 6 2 4.Economic incentives—prices, wage, profits, etc.7 82 5.Voluntary exchange and trade 9, 102 6.Specialization and comparative advantage 11, 122 7.Markets and prices 13, 142 8.Supply and demand 15, 16, 17 3 9.Competition 2018, 193 10. Economic institutions2221 2 11. Money and inflation2523243 12. Interest rates 26, 272 13. Labor markets and income 28292 14. Entrepreneurship 30 1 15. Physical and human capital investment 32312 16. Economic role of government 33342 17. Government failure, special interest groups 351 18. Output, income, employment, & price level36, 3738, 39405 19. Unemployment and inflation 41422 20. Fiscal and monetary policy44 43, 453 Total Number of Questions…………….......... Percent of Total…………………………… 6142545 13.331.155.6100.0

12. Test Development: Preparation review and rate existing items (good base– 3 editions) develop new test items conduct a field test prepare norming version

13. Timeline (2011-2012)  Spring-Summer 2011 Prepare new test questions  Fall 2011 Field test Prepare final norming version Administer norming version  Winter 2012 Administer norming version  Spring 2012 Administer norming version-posttest  Summer 2012 Data analysis and write manual

14. Test Results Overall statistics  Total (with and without econ) Basic (with and without econ) Advanced (with and without econ) Percentile norms Item difficulty & discrimination Construct validity data

15. Aggregate Statistics for TEL Norming Sample Form AForm B Sample Size Number of Students3,6823,686 Percent with Economics5049 Reliability Coefficient alpha.91.90 Means Overall23.3223.17 [A = 3,682; B = 3,686](9.70)(9.29) With Economics27.03 [A = 1,829; B = 1,816](9.77)(9.30) Without Economics19.0919.43 [A = 1,853; B = 1,870](8.12) (7.60) Notes:(1) Sample sizes are in brackets. (2) Standard deviations are in parentheses. (3) Also data for basic and advanced

16. With Economics Without Economics Raw Score Basic (n = 1,494) Advanced (n = 335) Basic (n= 1,702) Advanced (n = 151) 45 99 44 98 43 92 429985 99 419879 98 409771 97 399564 97 389359 9996 379054 9894 368747 9792 358443 9787 348139 9683 337735 9478 327333 9376 317027 9272 306725 9068 Percentile Norms

17. Item Correct Answer Corrected Item–Total Correlation (n = 3,682) Percent Correct With Economics (n = 1,829) Without Economics (n = 1,853) 1D.46 61.834.5 2B.46 57.432.5 3C.47 72.049.0 4D.40 50.036.9 5†5† B.37 55.639.7 6C.45 77.650.2 7†7† A.47 70.444.6 8C.39 54.242.3 9A.31 55.245.8 10C.43 70.057.4 Item Discrimination and Percentage of Correct Responses

18. ItemABCDBlank 115132048*4 21845*20135 381260*155 48123443*4 5†5† 848*17207 6121064*104 7†7† 57*818124 8141748*184 950*39543 10 1564*75 Percentage Response to Each Alternative

19. Test Validity Content validity: Based on Voluntary National Content Standards in Economics (CEE, 2010) Construct validity: Expected differences with and without economics

20. Descriptive Statistics (with and without econ) Gender Grade level Race/ethnicity Verbal ability Communication skills School size Student/teacher ratio Percent free lunch Type of community Region

21. With Economics Without Economics MeanStd. Dev.Number MeanStd. Dev.Number By gender Female26.179.43893 18.897.70942 Male27.8210.02936 20.518.46911 By grade level Grade 925.659.62178 18.467.86194 Grade 1024.2610.0087 17.927.27241 Grade 1125.9010.03391 19.207.67475 Grade 1227.809.601,773 20.638.48943 By race/origin African American/black 21.349.7283 15.986.60278 Asian33.459.13131 22.198.2878 Caucasian/white27.289.551,343 20.788.241,219 Hispanic24.618.95171 17.797.44191 Native American17.929.0613 15.507.9914 Pacific Islander26.8710.0415 18.094.1311 Other24.349.7473 18.637.9262 By verbal ability level Low19.096.73434 14.895.47575 Middle22.248.32602 19.026.87631 High28.909.31793 24.598.45647 By communication skills Best in English27.159.841,690 19.898.121,683 Best in another language 26.328.9828 17.568.5336 Equal in English and another language 25.198.70111 17.737.74134 Descriptive Statistics Within the Total Norming Sample: TEL Form A

22. With Economics Without Economics MeanStd. Dev. Number MeanStd. Dev. Number By school size Fewer than 1,500 students 26.549.59939 19.517.981,054 1,500 or more students 28.189.68824 20.268.46688 By student/teacher ratio 17 to 1 or less 27.879.79642 20.188.29690 More than 17 to 1 27.009.581,121 19.638.101,041 By % free lunch 17 or less 28.459.55976 21.608.12591 More than 17 24.719.14616 18.467.81950 By type of community City 27.708.85371 20.678.34544 Suburb 27.1210.55638 19.828.11487 Town 26.718.83311 18.517.75305 Rural 26.759.17413 19.388.15436 By region Northeast 27.249.0791 16.116.0163 Midwest 26.329.88995 20.628.23947 South 29.179.65312 18.377.70538 West 27.039.55431 19.838.41305 All students27.029.771,829 19.688.121,853

23. Uses of the TEL Overall test  Pretest only (knowledge)  Posttest only (knowledge)  Posttest—Pretest (learning) Comparative Norms—national data Item analysis  Right and wrong answers Item rationales  Explain why item is correct Reports and evaluations

24. For more test information: TEL Examiner’s manual Thank You!