Presentation is loading. Please wait.

Presentation is loading. Please wait.

Test Construction and Evaluation Lizamarie Campoamor-Olegario.

Similar presentations

Presentation on theme: "Test Construction and Evaluation Lizamarie Campoamor-Olegario."— Presentation transcript:

1 Test Construction and Evaluation Lizamarie Campoamor-Olegario

2 Test or Scale Development Procedure Step 1. Definition of Construct. Definition of Construct Step 2. Table of Test SpecificationsTable of Test Specifications Step 3. Item WritingItem Writing Step 4. Content Validation including Factor AnalysisContent Validation including Factor Analysis Step 5. Pilot Testing to Equivalent GroupPilot Testing to Equivalent Group Step 6. Item-analysis, reliability analysisItem-analysis, reliability analysis Step 7. Final AdministrationFinal Administration Lizamarie Campoamor-Olegario

3 DEFINITION OF THE CONSTRUCT Step 1. Lizamarie Campoamor-Olegario

4 Step 1. Definition of Construct Literature review Scientifically defined constructs Popular definition Syllabus back Lizamarie Campoamor-Olegario

5 TABLE OF TEST SPECIFICATIONS Step 2. Lizamarie Campoamor-Olegario

6 Step 2. Table of Test Specifications Table of Specifications Subject: _______________ Grade/ Year Level: _____ Type of Test: Cognitive (Knowledge), Psychomotor (Skills), or Affective (Attitudes)? Testing Time: _____ Type of Item/s: Likert? Or rating? Number of items: ____ Lizamarie Campoamor-Olegario

7 Knowledge (Cognitive) (Source: Bloom, B., Taxonomy of Educational Objectives, 1956) Lizamarie Campoamor-Olegario

8 Levels of Cognitive Domain (Knowledge) Knowledge: The remembering of previously learned material (recall of facts) Comprehension: The ability to grasp the meaning of the knowledge being learned Application: The ability to use learning materials in a new way Analysis: The ability to break material down into its parts so that its organizational structure may be understood Synthesis: The ability to combine previous experiences with new material to form a whole new structure Evaluation: The ability to judge the value of material for a given purpose Lizamarie Campoamor-Olegario

9 Knowledge – Can students RECALL information? Key Questions: Who, What, When, Where, How, How much, Describe, Define, Memorize, Literal questions, Which one, Name, Label, List, Reproduce, Recall Example of Knowledge Question Which of the following are raw materials or photosynthesis? a)Water, heat, sunlight b)Carbon dioxide, sunlight oxygen c)Water, carbon dioxide, sunlight d)Sunlight, oxygen, carbohydrates e)Water, carbon dioxide, carbohydrates Lizamarie Campoamor-Olegario

10 Comprehension – Can students EXPLAIN ideas? Key Questions: Explain, Describe in your own words, Inferential questions, Summarize, What would go better, Select the definition, Read the graph table, This represents¸ Condense this paragraph, What part doesnt fit, What are they saying, Explain what is happening, Give an example, State in 5 words, Explain what is meant, What restriction would you add, Translate, Outline, Locate, Match Lizamarie Campoamor-Olegario

11 Comprehension – Can students EXPLAIN ideas? Example of Comprehension Question If living cells similar to those found on earth were found on another planet where there was no molecular oxygen, which cell part would most likely be absent? a)Cell membrane b)Nucleus c)Mitochondria d)Ribosome e)Chromosomes Lizamarie Campoamor-Olegario

12 Application – Can students USE ideas? Key Questions: What is this used for?, How would you use, Make a model, Tell what would happen, If…how, Demonstrate how, Construct how, Show how, How much would there be if…, Design a lesson, Choose the statements that dont apply Lizamarie Campoamor-Olegario

13 Application – Can students USE ideas? Example of Application Question Phenylketonuria (PKU) is an autosomal recessive condition. About one in every fifty Individuals is heterozygous for the gene but shows no symptoms of the disorder. If you select a symptom-free male and a symptom –free female at random, what is the probability that they would have a child afflicted with PKU? –(.02)(.02)(.25) = = 0.01%, or about 1/10,000 –(.02)(.02) = = 0.04%, or about 1/2,500 –(1)(50)(0) = 100% = all –(1)(50)(0) = 0 = none –1/50 = 2%, or 2/100 Lizamarie Campoamor-Olegario

14 Analysis – Do students SEE relationships? Whole into parts, Analyse, Research, Survey, Group, Categorise, Compare and Contrast, What inconsistencies, fallacies, Arrange, What is the relationship, Chart, What is the function of, Diagram, What conclusions, Reason for…, What does the author believe, Investigate, Make a distinction, Cause for, What motive is there, Conclude, State the point of view, Separate, What relationship, Similar, Graph, Like, Differentiate, Dissect, Categorize, Distinguish fact from fiction, fact and inference, fact from opinion, advantage from disadvantage, good from poor reason, What persuasive technique Lizamarie Campoamor-Olegario

15 Analysis – Do students SEE relationships? Example of Analysis Question Mitochondria are called the powerhouses of the cell because they make energy available for cellular metabolism. Which of the following observations is most cogent in supporting this concept of mitochondrial function? a)ATP occurs in the mitochondria b)Mitochondria have a double membrane c)The enzymes of the Krebs cycle, and molecules required for terminal respiration, are found n mitochondria d)Mitochondria are found in almost all kinds of plant and animal cells e)Mitochondria abound in muscle tissue Lizamarie Campoamor-Olegario

16 Synthesis – Can students combine ideas and CREATE a new entity? Key Questions: New ways of doing, Take risks, Consider the unexpected, Pose an alternative, Hypothesis, Create, Compose, Solve, Design, Blend, Construct, How else would you, Build, Combine, Solve the following, Imagine, Plan, Predict, Link concepts in an unusual and flexible way, Make, What if, Make a film, Invent, Propose an alternative Lizamarie Campoamor-Olegario

17 Evaluation – Can students make JUDGEMENTS and support them? Evaluate quality, relevance, reliability, truth, Which is best, Accuracy and effectiveness, Choose and explain why¸ Rate, Rank, Defend, Choose, Grade, Order, Verify, Dispute, Criticize, Defend, Find the errors, Editorialize, Appraise, Judge, What fallacies, consistencies, inconsistencies appear, Which is more important, better, moral, appropriate, inappropriate, useful, clearer, suits the purpose, achieves the goal, logical, valid Lizamarie Campoamor-Olegario

18 Evaluation – Can students make JUDGEMENTS and support them? Example of Evaluation Question Disregarding the relative feasibility of the following procedures, which of these lines of research is likely to provide us with the most valid and direct evidence as to revolutionary relations among different species? a)Analysis of the chemistry of stored food in female gametes b)Analysis of the form of the Krebs cycle c)Observation of the form and arrangement of the endoplasmic reticulum d)Comparison of details of the molecular structure of DNA e)Determination of the total protein in the cell Sources: – – Lizamarie Campoamor-Olegario

19 Skills (Psychomotor) Lizamarie Campoamor-Olegario

20 Sample Practice Questions 1. Do you eat breakfast? ____ always ____ sometimes _____never 2. How often do you eat breakfast during the week? ____ 1 – 2 days per week ____ 3 – 4 days per week ____ 5 – 6 days per week ____ everyday ____ never Lizamarie Campoamor-Olegario

21 Sample Practice Questions 3. What do you usually take to have a good breakfast? Give reason. (Please indicate number below). FOODS REASONS 1. _____________________________________________ 2. ____________________________________________ 3. ____________________________________________ Reason:1 – nutritious 2 – tastes good 3 – always available 4 – affordable 5 – others, specify ___________________ Lizamarie Campoamor-Olegario

22 Sample Practice Questions 4. Why do you eat breakfast? (Please check all that apply to you.) ____ I feel better if I eat breakfast ____ I am hungry in the morning ____ my parents told me ____ I enjoy eating ____ I feel it is important to eat breakfast ____ others, specify ___________________ Lizamarie Campoamor-Olegario

23 Sample Practice Questions 5. If you skip breakfast, how do you feel at mid-morning (9:30-10:00)? Check one that best describes how you feel. _____ Fine _____ Angry/bored _____ Hungry/starving _____ Headache/sick _____ Nervous _____ Easily irritated _____ Weak _____ Others, specify: ____________ Lizamarie Campoamor-Olegario

24 Attitudes (Affective) Lizamarie Campoamor-Olegario

25 Sample Attitude Items I cringe when I have to go to math class. I am uneasy about going to the board in a math class. I am afraid to ask questions in math class. I am always worried about being called on in math class. I understand math now, but I worry that it's going to get really difficult soon. I tend to zone out in math class. I fear math tests more than any other kind. I don't know how to study for math tests. It's clear to me in math class, but when I go home it's like I was never there. I'm afraid I won't be able to keep up with the rest of the class. Lizamarie Campoamor-Olegario

26 Put a check to the number which best corresponds to your answer: 1 – strongly agree4 - disagree 2 – agree5 – strongly disagree 3 – neutral Adolescent who eats a wholesome nutritious breakfast do better in school. 2. I prefer to eat breakfast at home to have quality time with my family 3. Drinking milk is of much important as part of adolescent breakfast. 4. Skipping breakfast could lose out on an important nutritional contribution to adolescents total daily food intake. 5. Adolescents who take enough exercise will reduce the danger for excessive intake of calories. Lizamarie Campoamor-Olegario

27 6. I know that eating breakfast will improve my attendance and punctuality in class. 7. I believe that if I am able to consume a balanced meal during breakfast, I will have a healthy day. 8. If I eat breakfast, it is more likely that I will get the recommended dietary allowance for vitamin A, C, D, and B6, calcium, iron, zinc & folate. 9. If I have instant noodle, biscuit or coffee for breakfast, it will be enough to start my day. 10. Washing my hands before eating will keep me safe from food borne disease. Put a check to the number which best corresponds to your answer: 1 – strongly agree4 - disagree 2 – agree5 – strongly disagree 3 – neutral Lizamarie Campoamor-Olegario

28 Table of Specifications Content areaRecall / Recognition Skills Comprehens ion/ Application Critical Thinking/ Problem Solving Total Allocation of marks Biochemistry Cells/Tissues Genetics Reproduction Invertebrates46010 Vertebrates Plant life26210 Ecological07310 Total Lizamarie Campoamor-Olegario

29 ITEM WRITING Step 3. Lizamarie Campoamor-Olegario

30 Step 3. Item Writing 1. Keep the Test Short –Try out a longer test on a small number of subjects, and then cut those sections that don't result in useful data. 2. Plan to Quantify Your Results. 3. Good tests must be based on the objectives of a course. 4. Each test should meet four criteria: RVDN –Reliability –Validity –Discrimination –Non-triviality Lizamarie Campoamor-Olegario

31 RVDN R: Does not generate random answers and consistently results in marks that reflect the skill level of each student. V: Measures what it intends to measure. D: Shows clearly the differences in the levels of achievement of the students to whom it is administered. N: Focuses on that which the students should know and not on that which is irrelevant to evaluation. Lizamarie Campoamor-Olegario

32 Step 3. Item Writing 5.Questions must be clearly written. 6.Least complex questions should appear first to encourage students' confidence in their knowledge. 7.Choose the test format. Commonly used are: a)essay b)true-or-false tests c)multiple choice tests d)completion-and-short answer tests e)matching tests 8.Reread all the items from the standpoint of the students Lizamarie Campoamor-Olegario

33 Essay Test Questions should clearly delimit the scope of the answer required Avoid words such as what, who, when, which trigger closed responses Avoid letting them to answer a choice of questions Give definitive task to student-compare, analyze, evaluate, etc. Use checklist point system to score with a model answer: write outline, determine how many points to assign to each part Score one question at a time-all at the same time. Lizamarie Campoamor-Olegario

34 True-or-false tests Avoid having more than one idea in a question. Avoid absolute terms: never, only, all, none, always Avoid indefinite terms: in most cases, great, sometimes, generally, some, few Avoid double negatives. Use exact quantitative language Don't lift items straight from the book. Make more false than true (60/40). Lizamarie Campoamor-Olegario

35 Multiple choice tests In order to measure insight, avoid quoting from the textbook. Avoid none of these or all of the above Stem should be in simple, understood language; delete extraneous words. Make all distracters plausible Make all alternatives homogeneous (in subject, content, form, length, explicitness, and grammatical structure) Don't use double negatives. Present alternatives in logical or numerical order. Lizamarie Campoamor-Olegario

36 Multiple choice tests Make each item independent of others on test. Need more than 3 alternatives, 4 is best. Ask for the best answer and use terms such as most and primary if more than one answer is partially correct. The reading and linguistic difficulty of items should be low. Use care in the repetition of words or phrases between the stem and the correct answer. Avoid items that reveal the answer to another item. Ordinarily, distracters should not overlap, subsume, or be synonymous with one another. Lizamarie Campoamor-Olegario

37 Completion and short answer tests/ Identification Limit the number of blanks in each question. Generally, blanks are not inserted at the beginning of the question. Make sure there is only one unambiguous response for the blank. In numerical problems, indicate the type of units in which the answer should appear. State the item so that only one answer is correct State the items with words that students understand Avoid items in which the correct answer is a matter of opinion. Lizamarie Campoamor-Olegario

38 Matching tests Organize questions together in logical groups. In your instructions, indicate whether or not some answers may be used more than once. Need 15 items or less. Use homogenous material in each exercise. Make all responses plausible. Put all items on a single page. Put response in some logical order (chronological, alphabetical, etc.). Lizamarie Campoamor-Olegario

39 CONTENT VALIDATION Step 4. Lizamarie Campoamor-Olegario

40 Step 4. Content Validation including Factor Analysis Validity denotes the extent to which an instrument is measuring what is supposed to measure. –Forms of Validity –Face Validity –Content Validity –Criterion-related Validity –Construct Validity Lizamarie Campoamor-Olegario

41 Face Validity Does the test appear valid? Subjective judgment of a group generally composed of non-experts Includes good layout and format, clarity of directions/ instructions, non- ambiguity of items, correct grammar and sentence structure(syntax), appropriate scales, appropriate number of items, clarity of printing Lizamarie Campoamor-Olegario

42 Exercises on Face Validity Other than infancy, nutrition needs are higher in ________________ than at any other time in the life cycle because this is the period of rapid growth. a)Infancy b)Adolescence c)Adulthood d)no response e)do not know Lizamarie Campoamor-Olegario

43 Exercises on Face Validity __________ is the meal that should not be missed. Because it breaks the fast of not eating 8 to 12 hours. a)Breakfast b)Lunch c)Supper d)no response e)do not know Lizamarie Campoamor-Olegario

44 Exercises on Face Validity The following food groups would consist of a good breakfast. a)coffee, biscuit, instant noodles b)fried rice, fried dilis, tomato, banana, milk c)peanut butter, pan de sal, coffee d)no response e)do not know Lizamarie Campoamor-Olegario

45 Exercises on Face Validity Eating breakfast has many benefits. It improves concentration, increase alertness, and helps one function more efficiently. a)true b)false c)no response d)do not know Lizamarie Campoamor-Olegario

46 Exercises on Face Validity Adolescents who skip breakfast have no better academic performance and have negative effects on mood, alertness and contentment. a)true b)false c)no response d)do not know Lizamarie Campoamor-Olegario

47 Exercises on Face Validity Adolescents need to consume at least ________ piece of fruit during breakfast.. a)1 piece b)2 pieces c)3 pieces d)No response e)Do not know Lizamarie Campoamor-Olegario

48 Exercises on Face Validity Uriin ang mga sumusunod na pagkain kung ito ay isda o pagkaing dagat, uri ng prutas, o uri ng gulay. Ilagay lamang sa patlang ang letra ng tamang sagot at lagyan ng (/) ang hanay kung ang pagkaing ito ay Yamang Lupa o Yamang Tubig. Lizamarie Campoamor-Olegario

49 Exercises on Face Validity Upang maging malakas at masigla, dapat ay maging aktibo. Lagyan ng star (*) ang mga gawain na nakapagpapalakas ng katawan at (x) kung ito ay hindi masyadong nakapagpapalakas ng katawan. Lizamarie Campoamor-Olegario

50 Oo o Hindi. Ilagay sa puwang ang Oo kung ito ay tumutugon sa kakulangan sa pagkain o Hindi kung ito ay hindi makakatulong upang matugunan ang pangangailangan ng mag-anak sa pagkain ______1.Ang tatay ay nangingisda sa dagat. ______2.Ang mag-anak ay naglilinis ng bahay ______3.Ang tatay ay nagpuputol ng kahoy upang gamitin sa pagluluto. ______4. Ang mag-anak ay nagtatanim ng gulay sa paligid ______5. Ang bata ay tumatakbo ng matulin ______6. Ang pamilya ay nag-aalaga ng hayop _____10. Ang mag-anak ay nagsisimba tuwing Linggo Lizamarie Campoamor-Olegario

51 Exercises on Face Validity Multiple Choice. Read the questions carefully and write the correct letter of your answer on the blank. (Basahin ang sumusunod na mga katanungan at ilagay sa blanko ang letra ng tamang sagot) ______ 1. Needed in the delivery of nutrients and removal of waste from the body (Ito ay kailangan sa pag-deliver ng sustansiya at pagtanggal ng mga dumi sa katawan) a)Rice (kanin) b)Chicken (manok) c)Fluids (tubig) d)Fruits (prutas) Lizamarie Campoamor-Olegario

52 Exercises on Face Validity ______2. To be more active, we should eat plenty of Go foods such as: (Upang maging aktibo, dapat tayo ay kumain ng go foods tulad ng): a)Bread, rice, cereals (tinapay, kanin at cereals b)fruits and vegetables (prutas, at gulay) c)meat, fish, and chicken (karne, isda at manok) Lizamarie Campoamor-Olegario

53 Exercises on Face Validity ______3. Grow Foods are important for growth and development, examples of grow foods are: (Ang grow foods ay importante para sa paglaki at pag- developng muscles) a)Bread, rice, cereals (tinapay, kanin at cereals) b)fruits and vegetables (prutas at gulay) c)meat, fish, and chicken (karne, isda at manok) Lizamarie Campoamor-Olegario

54 Exercises on Face Validity ______5. Breakfast is an important meal. Choose a good breakfast from the following. (Mahalaga na kumain ng agahan. Alin sa mga ito ang mainam na agahan?) a)Rice, dilis, banana and a glass of milk b)Bread and coffee c)Noodles only Lizamarie Campoamor-Olegario

55 Exercises on Face Validity ______9. Height and weight of a growing child should be taken every: (Ang taas at sukat ng katawan ng lumalaking bata ay dapat kunin kada): a)Month (buwan) b)Year (taon) c)Week (Linggo) Lizamarie Campoamor-Olegario

56 Matching Type. Match item A with Item B. Write your answers on the blank provided. AB _____ 1. Needed for proper hydrationa) banana _____ 2. Organ which produces insulinb) stomach _____ 3. Where solid waste in the body is stored c) digestive system _____ 4. High potassium contentd) water and other fluids _____ 5. Cause of diabetese) lack of insulin Lizamarie Campoamor-Olegario

57 _____ 6. The long tube between the mouth and stomach f) mouth _____ 7. The first part of the digestive system, where food enters the body g) pancreas _____ 8. Sac-like muscular organ that is attached to the esophagus h) esophagus _____ 9. Promotes healthi) proper diet and physical activity _____ 10. System that consists the organs of digestion j) rectum Lizamarie Campoamor-Olegario

58 II.Put a check (/) on the diseases of the digestive system if it is caused by the intake of dirty foods or (x) if not ______ 1. Flu ______ 6. Diabetes ______ 2. Diarrhea ______ 7. Sore throat ______ 3. Cough ______ 8. Dysentery ______ 4. Fever ______ 9. vomiting ______ 5. Stomach pain Lizamarie Campoamor-Olegario

59 Exercises on Face Validity What principle of healthy eating is indicated in the following sentences? Put your answer on the blank provided before the number. (Anong mga prinsipyo, na nakasalungguhit, sa pagkain ng wasto ang isinasaad ng mga pangungusap sa ibaba. Ilagay ang iyong sagot sa patlang. a)Balance b)Variety c)Moderation Lizamarie Campoamor-Olegario

60 1. Eating different kinds of food everyday is important to get all the essential nutrients needed by the body. (Ang pagkain ng ibat ibang pagkain ay nakapagbibigay ng kumpletong sustansiya na kailangan ng katawan), 2. Eating the right amount of go, grow and glow foods (Ang pagkain ng tamang dami mula sa go grow and glow foods) 3. Put less sodium such as vetsin, toyo, patis in food to prevent high blood pressure. (Maglagay lamang ng kaunting dami ng vetsin, toyo,o patis sa pagkain upang maiwasan ang pagkakaroon ng mataas na presyon ng dugo) 4. A meal should consist of rice, protein source such as fish/meat, a piece or slice of fruit, at least half cup of vegetable and a glass of milk (Ang pagkain ay dapat na mayroong kanin, piraso ng karne, isang piraso ng prutas, kalahating tasa ng gulay at isang basong gatas. 5. Too much candies, cakes and chocolates may cause you to have tooth ache. (Ang pagkain ng sobrang matatamis tulad ng kendi, keyk at tsokolate ay nakapagdudulot ng sakit sa ngipin) Lizamarie Campoamor-Olegario

61 Identify the correct locomotor movement _________ 1.A regular succession of leapsa. Run _________ 2.To advance or move forward on foot with one foot always on the ground floor b.Jump _________ 3. To move fast with both feet off the ground c.gallop _________ 4. To slip or pass along over a surfaced. Slide _________ 5. to spring from the floor or groundf. Leap Lizamarie Campoamor-Olegario

62 Multiple Choice. Write your answer on the blank provided before the number. ___________ 1. These are signs of stress? a)healthy glowing skin b)fatigue, dizziness and tension c)mental alertness ____________ 2. Gives the red color of the blood a)Oxygen b)Hemoglobin c)Carbon dioxide ____________ 5. Diabetes is caused by lack of: a)Exercise b)Insulin c)Food Lizamarie Campoamor-Olegario

63 ____________ 6. This carries nutrients and oxygen to the body a) b) c) Blood Water Fats ____________ 7. High blood pressure is caused by: a)Excessive intake of fat b)Excessive intake of salt c)Excessive intake of sugar ___________ 8. High intake of fat increases the risk of: a) b) c) juvenile delinquency heart disease urinary tract infection ____________ 9. Over consumption of sodium leads to a high blood pressure. Foods high in sodium are: a) b) c) cakes table salt, patis and bagoong fruits Lizamarie Campoamor-Olegario

64 Content Validity The degree to which the sample of items, tasks, or questions on a test are representative of some defined universe or domain of content. It therefore, indicates whether or not the test items adequately represent that domain. Usually a table of specifications is built in order to ensure that the entire domain is represented by the items in a test. Expert judgment Lizamarie Campoamor-Olegario

65 Content Validity Is the instrument comprehensive? What is not there? Is it logical How adequately does the sample of items or questions represent the content to be assessed? Consult some experts who rate the items, eliminating or changing those that do not meet the specified content. Repeat until all raters agree on the questions and answers. Lizamarie Campoamor-Olegario

66 Criterion-related Validity The relationship between the scores obtained using the instrument and the scores obtained using one or more other instruments or measures Lizamarie Campoamor-Olegario

67 Two Types of Criterion-Related Validity Concurrent Validity Predictive Ability Lizamarie Campoamor-Olegario

68 Concurrent Validity Measure and criterion are assesses at about the same time Looks at information gathered from two sources at the same time. Example New intelligence test and established test, math ability and aptitude for engineering, student evaluation instrument and peer- evaluation instrument, teacher-made test and standardized test on the same subject area Lizamarie Campoamor-Olegario

69 Predictive Ability Used to look at validity over time (months/ years) Administer one assessment, allow time to lapse, then compare the earlier score to another measure of performance. Examples The Scholastic Aptitude Test (SAT) scores and GPA SAT and Achievement Test IQ test, GPA, and Achievement Test Lizamarie Campoamor-Olegario

70 Construct Validity Concerned with the extent to which an instrument may be said to measure a theoretical construct or latent trait Encompasses other elements of validity. Construct Validity Methods –Convergent validity –Divergent validity –Discriminant validity Lizamarie Campoamor-Olegario

71 Convergent Validity Correlating with other theoretical measure with which the developing instrument should correlate Example: A new instrument to measure professionalism in teaching and Teaching Commitment Scale (expect high correlation/ validity coefficient) Lizamarie Campoamor-Olegario

72 Divergent Validity Correlating with other theoretical measure with which the developing instrument should not correlate Example A new instrument to measure professionalism in teaching and Attitude Toward Movies Scale (expect low or negative correlation/ validity coefficient) Lizamarie Campoamor-Olegario

73 Discriminant Validity Examine the difference in test scores arising from groups known to differ on the construct Known-groups technique: comparison between known high and known low performers; difference must be significant; t-test Lizamarie Campoamor-Olegario

74 Construct Validation by Factor Analysis (Factorial Validity) Tests that purports to measure the same construct should load on the same factor, and that different factors should emerge for different constructs Items that load on the wrong factor are deleted or changed. Lizamarie Campoamor-Olegario

75 Factor Analysis – SPSS To run a factor analysis in SPSS, select Analyze (or Statistics) in the top menu bar. Move down to Data Reduction and select Factor Analysis. In the Factor Analysis dialog box, enter all the variables listed above in the "Variables" box. Lizamarie Campoamor-Olegario

76 Factor Analysis – SPSS Lizamarie Campoamor-Olegario

77 Factor Analysis – SPSS Click on the Descriptives button and check Coefficients, and Significance Levels. Click on the Extraction button and under Display check Unrotated Factor Matrix and Scree Plot. Leave as defaults the settings for Analyze Correlation Matrix and Extract Eigenvalues over 1. Lizamarie Campoamor-Olegario

78 Factor Analysis – SPSS Lizamarie Campoamor-Olegario

79 Factor Analysis – SPSS How many factors? Perform factor analyses with various values. Choose the one that gives the most appealing structure. Click on the Rotation button and select Varimax. Under Display, check Rotated Solution and Loading Plots. Lizamarie Campoamor-Olegario

80 Factor Analysis – SPSS Lizamarie Campoamor-Olegario

81 Factor Analysis – SPSS Click on the Scores button and check Display factor score coefficient matrix". Lizamarie Campoamor-Olegario

82 Factor Analysis – SPSS Click on the Options button and check "Coefficient Display Format, Sorted by Size". Click on OK to run the procedure. Lizamarie Campoamor-Olegario

83 Factor Analysis – SPSS When you have a highly interpretable solution, which represents almost 90% of the data, name the factors. Look for patterns of similarity between items that load on a factor. better interpretation. Lizamarie Campoamor-Olegario

84 Example: Oblique Promax rotation of 4 factors of 24 mental ability variables (Gorsuch, 1983) Factors1234 General information Paragraph comprehension Sentence completion Word classification Word meaning Add Code Counting groups of dots Straight & curved capitals Woody-McCall mixed Visual perception Cubes Paper form board Flags Deduction Numerical puzzles Problem reasoning Series completion Word recognition Number recognition Figure recognition Object-number Number-figure Figure-word Lizamarie Campoamor-Olegario

85 Factor Analysis – SPSS High values (above about.4 in absolute value) Variables within each block all seem to measure the same general kind of mental ability Note: Criterion and Construct Validation can be done after Pilot- testing Lizamarie Campoamor-Olegario

86 PILOT-TESTING TO EQUIVALENT GROUP Step 5. Lizamarie Campoamor-Olegario

87 Step 5. Pilot-testing to equivalent group If achievement test, to higher level Number of items times three Lizamarie Campoamor-Olegario

88 ITEM ANALYSIS, RELIABILITY Step 6. Lizamarie Campoamor-Olegario

89 Step 6. Item-analysis, reliability To improve the effectiveness of test items and the validity of test scores Select and rewrite items on the basis of item performance data Item analysis provides information about: –The difficulty of the item: How easy or hard the item was –The discriminating power of the item: How well it discriminated between high and low scorers in the test –The effectiveness of the alternative: Whether all the alternatives served their purpose Lizamarie Campoamor-Olegario

90 Steps of an Item Analysis S1. Order the N papers by score, placing the one with the highest score on top and continuing sequentially until on with the lowest score is on the bottom. –Note: Where two or more examinees made the same score, order them alphabetically by surname. Lizamarie Campoamor-Olegario

91 Steps of an Item Analysis S2. Multiply N, the total number of tests, by 0.27 and round off the result to the nearest whole number, the number is represented by n. If N is 30, n would be 8 (8.1 rounded) S3. Count off the n best papers from the top of the stack. This is the high group. S4. Count off the n poorest papers from the bottom of the stack. This is the low group. Lizamarie Campoamor-Olegario

92 Steps of an Item Analysis S5. Determine the proportion in the high group (pH) answering a particular item correctly by dividing the number of correct answers for the high group by n, that is: Number of correct responses to the item pH = n Repeat the procedure for the low group to obtain pL for each item Lizamarie Campoamor-Olegario

93 Steps of an Item Analysis S6. To obtain an estimated item difficulty index, p (that is, the proportion of the total group that answered the item correctly), add pH to pL and divide the resulting sum by 2: pH + pL p = Lizamarie Campoamor-Olegario

94 S7. To obtain a measure of item discrimination (i.e., how well this item distinguishes between the students who understand the content universe of the test and those who do not), subtract pL from pH: D = pH-pL Item Number High or Low Number of Correct Responses for Groups (n=8) Proportion of Correct Responses Item Discriminat ion D = pH-pL Item Difficulty pH+pL p = H7 L3 2H8 L6 3H4 L4 4H2 L6 n = (0.27) N n=0.27 (30) = 8.1 (rounded off to 8) students in the high group and 8 in the low group Lizamarie Campoamor-Olegario

95 Item Number GroupNumber of Correct Responses for Groups (n=8) Proportion of Correct Responses Item Discriminati on D = pH-pL Item Difficulty pH+pL p = H7pH=7/8= = =0.63 L3pL=3/8=0.38 2H L H L4 4H L60.75 Answer Lizamarie Campoamor-Olegario

96 Steps of an Item Analysis For discrimination, the following table may be used: –0.40 and above = very discriminating item – = discriminating item – = reasonably discriminating item – = marginally discriminating item, usually subject to improvement –Below 0.10 = unproductive item for test reliability purposes Lizamarie Campoamor-Olegario

97 Difficulty Lizamarie Campoamor-Olegario

98 Interpreting Item Analysis Good item Potential miskey Ambiguous items Equal distribution to all alternatives Alternative/s not working Distracter too attractive Question not discriminating Negative discrimination Too easy Omit Lizamarie Campoamor-Olegario

99 Good item * keyed correct answer O omit or no answer GroupABC*DO High00100 Low4372 Lizamarie Campoamor-Olegario

100 Potential Miskey GroupA*BCDO High11103 Low4353 Lizamarie Campoamor-Olegario

101 Ambiguous Items GroupABC*DO High6171 Low5253 Lizamarie Campoamor-Olegario

102 Equally to all alternatives GroupABC*DO High4353 Low3444 Students are guessing Lizamarie Campoamor-Olegario

103 Alternatives arent working Each distracter has to attract at least 5% of the students If two alternatives don't draw any students - -> might consider redoing as true/false GroupAB*CDO High11400 Low5720 Lizamarie Campoamor-Olegario

104 Distracter too attractive Not one distracter should pull more than about half of students GroupABCD*O High7115 Low10212 Lizamarie Campoamor-Olegario

105 Question not discriminating GroupA*BCDO High7323 Low7215 Lizamarie Campoamor-Olegario

106 Negative Discrimination GroupA*BCDO High7323 Low10311 usually means an ambiguous question Lizamarie Campoamor-Olegario

107 Too Easy GroupAB*CDO High01401 Low11311 If you dropping too many questions, you may lose content validity (specs) Lizamarie Campoamor-Olegario

108 Omit Possibilities –If near end of the test, they didn't find it because it was on the next page (format problem) –Test is too long –Way too difficult GroupAB*CDO High01401 Low11311 Lizamarie Campoamor-Olegario

109 Reliability Analysis Reliability is consistency of test scores The difference of test scores is attributed to true differences Not to –testing environment –instruction –time limits –rapport –fatigue –practice –examinees mood –and other similar factors Lizamarie Campoamor-Olegario

110 How can these be avoided Careful planning of the test Good sampling of the items Lizamarie Campoamor-Olegario

111 Reliability Coefficient Used in determining the consistency of tests The larger the number is, the more consistent the assessment.90 or higher means high reliability.80 to.89 good reliability.60 to.79 means low to moderate reliability.40 to.59 means doubtful reliability Lizamarie Campoamor-Olegario

112 Internal Consistency Cronbachs Alpha a measure of the here-and-now, on the spot reliability Limitations of internal consistency include –1) there needs to be sufficient number of items (around five items per trait or skill) –2) it is not appropriate for speeded tests. Lizamarie Campoamor-Olegario

113 Other Things to Look at Number of items –Cronbach's alpha increases as the number of items in the scale increases. Alpha if deleted –The researcher may wish to drop items where the alpha if deleted is higher than the overall alpha as another way to improve the alpha level. Lizamarie Campoamor-Olegario

114 Other Things to Look at The item-total correlation –A low item-total correlation means the item is little correlated with the overall scale (ex., <.3 for large samples or not significant for small samples) and the researcher should consider dropping it. –A negative correlation indicates the need to recode the item in the opposite direction. Lizamarie Campoamor-Olegario

115 Remember The reliability analysis should be re-run if an item is dropped or recoded. Lizamarie Campoamor-Olegario

116 In SPSS, Cronbach's alpha is found under Analyze, Scale, Reliability Analysis Lizamarie Campoamor-Olegario

117 Then in the Statistics button, check Scale to get alpha. You can also check Scale if deleted Lizamarie Campoamor-Olegario

118 R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P H A) Item-total Statistics Scale Scale Corrected Mean Variance Item- Alpha if Item if Item Total if Item Deleted Deleted Correlation Deleted ITEM ITEM ITEM ITEM ITEM ITEM ITEM ITEM ITEM ITEM N of Cases = N of Items = 80 Alpha =.9045 Lizamarie Campoamor-Olegario

119 FINAL ADMINISTRATION Step 7. Lizamarie Campoamor-Olegario

120 Step 7. Final Administration Usually, the run is five years Lizamarie Campoamor-Olegario

121 Thank you for listening! Lizamarie Campoamor-Olegario

Download ppt "Test Construction and Evaluation Lizamarie Campoamor-Olegario."

Similar presentations

Ads by Google