1. What is the key to Ukrainian teachers? Teachers’ learning needs in the area of “educational measurement” Tempus JEP IB_JEP-24141-2003 TRUE ASSESSMENT. Distance learning network for teachers

2. Survey on educational measurement  3 parts: information about respondent; content of the course; DL and computer issues.  Considerations: –Anonymous study –Language: as simple as possible, avoid special terms (limited knowledge in the area)  Problems: –Language: Low discrimination between terms (assessment, evaluation, grade, mark); absent terminology (scoring, scaling, equating, standard setting)

3. Respondents  452 persons. 18% male, 81% female  Different age (54% - 41-50 y.o.)  7+ educational areas –29% natural history, 23% language & literature  Student age: 44% - middle, 66% - high school  Position: –0,5% methodologist; 55% teachers; 43% administrators  Various lengths of service (45% - over 20 years)  Place of work: –49% general school, 33% lyceum, gymnasium  Geographical place of work: –31% big city, 28% village, 27% Kyiv

4. Content of the course (suggested)  Eight 1-day modules. 6 modules are detailed. –Selection of methods of educational diagnostics –Learning objectives. Domains of competencies of students and pupils –Measurement of academic achievement of pupils with the help of written methods –Interpretation of measurement results –Use of assessment (evaluation) results –Standardizing of assessment on school, district, country, international levels –Modern methods of assessment (evaluation) of senior pupils abroad –Basics of statistics relevant for educational measurement

5. Assessment of modules and topics  1 st set of data: indication whether the theme was ever studied or not  2 nd set of data: whether the theme is important for person in his/her work or not  Selection of 6 themes that are of the largest interest for respondent  Suggestion of new topics

6. 1 st set of data: 1  Theme was studied in formal education: –15-50% of respondents (depends upon theme) –“Best” themes: 1.1. Main concepts and categories of educ’l diagnostics (50%) 2.3.Formulating of learning objectives. (43%) 3M. Measurement of academic achievement of pupils with the help of written methods (43%), 1M. Selection of methods of educational diagnostics (42%) 6.3. National exams (41%),  Theme was studied individually (self) –25-63% of respondents  Theme was never studied –7-41% of respondents

7. 1 st set of data: 2  Theme was studied individually (self) –25-63% of respondents –The “best” mostly studied themes: 5.2.Monitoring of teacher work (63%) 4.2. Scales and converting of results between scales (62%) 4.1. Establishing of pass/fail and other criteria for grading. (57%) 3.1. Development of measurement instrument (test). Item- writing (56%) 5.3. Management of school using objective assessment (55%) 3.3.Procedure of written assessment. (53%) 3.6. Evaluation (assessment) of test item quality based upon test results (52%) 1.5. Stages of test development. Strength and weaknesses of testing. (51%)

8. 1 st set of data: 3  Theme was never studied –7-41% of respondents –“Worse”, least studied themes 1.6. History of testing (41%) 7M. Modern methods of assessment abroad (37%) 1.4. Characteristics of measurement methods (validity, reliability) (36%) 8M. Basics of statistics for educ’l measurement (34%) 2M. Learning objectives (31%) Problems with 1 st set of data: Not very consistent data between modules and subtopics in the modules. Fragmental knowledge (within the module)

9. 2 nd set of data: 1  Theme is very important, necessary: –33-85% of respondents (depends upon theme) –The most important themes: 3.6. Evaluation (assessment) of test item quality based upon test results (85%) 3.5. Processing, analysis, scoring of written assessment (83%) 3.1. Development of measurement instrument (test). Item- writing (80%) 3.4. Processing of the test exams (Scanning and scoring) (77%)  Theme is somewhat important –9-48% of respondents  Theme is not important –0-13% of respondents  Cannot evaluate/do not know –0-14 % of respondents

10. 2 nd set of data: 2  Theme is not important –0-13% of respondents –The least important themes: 1.6. History of testing (13%) 8M. Basics of statistics for educ’l measurement (12%) 2.1. Classification of learning objectives (11%) 2M. Learning objectives (10%) 2.4 domains of competencies of pupils (8%) 1.3. Methods of educ’l measurement (8%)  Cannot evaluate/do not know –0-14 % of respondents

11. Selection of themes that are of the largest interest for respondents M1. Selection of methods of educational diagnostics 148 M3. Measurement of academic achievement of pupils with written assessment methods 130 M4. Interpretation of measurement results 102 M5. Use of assessment (evaluation) results 94 M2. Learning objectives. Domains of competencies 91 M6. Standardizing of assessment on different levels 84 2.3. Formulating of learning objectives. 117 3.1. Development of measurement instrument (test). Item-writing (item types, rules, defects). Selection of items. Test blueprint. 105 6.1. Evaluation of quality and validity of assessment method 91 3.6. Evaluation (assessment) of items quality 84 Presented those topics that were chosen by more than 20% of respondents

12. Selection of themes that are of the least interest for respondents 4.2. Scales and converting of results between scales 21 1.3. Methods of educational measurement (testing, written assessment, practical assignments, orals, observation, projects etc). 20 3.7. Evaluation (assessment) of written instrument quality.20 1.4. Characteristics of measurement methods (validity, reliability etc)19 1.6. Overview of history of testing. 19 3.5. Processing, analysis and scoring of written assessment works. 12 6.3. National exams (school-leaving, university entrance, certification)2 1.1. Main concepts and categories of educational diagnostics. Goal and function of measurement and assessment (evaluation). Content, method, instrument, procedures. 1 3.2. Development of measurement instrument (written assessments)1 1.2. Concept of learning objectives (educational goals)0 2.1. Classification of learning objectives. 0 3.3. Procedure of written assessment.0 4.1. Establishing of pass/fail and other criteria for grading. 0 Presented those topics that were chosen by less than 5% of respondents

13. Suggestion of new topics Only 8 respondents suggested. Total 13 new themes. Relevant !!! Comparative analysis of different systems of assessment !!! Psychological problems of assessment (evaluation) !!! Practice on development of measurement instruments