Arengu-uuringud Arengu mõõtmine, hindamine, võrdlemine Mati Heidmets 2013 sügissemester

Sotsiaalne mõõtmine ja hindamine Küsimus: kuidas suhtuda arenguhindamisse, kuivõrd on see tõsiseltvõetav, mil määral äri, mäng, PR? Mida tähendab mõõtmine ja hindamine inimese, organisatsiooni, ühiskonna puhul? Mis on mõõtmistulemused, kuidas neid seletada ja mõtestada? Mõisteaparaat: mõõtmine, hindamine, standardid, kontekst, võrdlus. Sotsiaalse mõõtmise eripärad Kuidas mõõta Eestit!

Mõõtmine (measurement) Mõõtmine: isiku või nähtuse suvalise tunnuse hindamine kokkulepitud skaala suhtes. Tulemust üritatakse väljendada numbriliselt. Rikkus, pikkus, tervis, tarkus, areng … Mõõtmine – üldmõiste. Siia alla suvalise omaduse, tunnuse, ka näit kvaliteedi hindamine väga erinevate protseduuride abil. Vajalik mõõtmisinstrument – kaal, IQ test jne. Measurement is the process of estimating the magnitude of some attribute of an object, such as its length, weight, or depth relative to some standard (unit of measurement), such as a meter or a kilogram. The act of measuring usually involves using a measuring instrument, such as a ruler, weighing scale, thermometer, speedometer, or voltmeter, which is calibrated to compare the measured attribute to a measurement unit. Any kind of attributes can be measured.

Hindamine A (assessment) Hindamine (assessment): hinnangu - numbrilise, sõnalise - andmine protsessile või tulemusele. Koolihinne, hinnang MA tööle, projekti lõpphinnang, arengutaseme hindamine … Instrument sisaldab sageli subjektiivset momenti – kirjandi hindamine! Hindamise tunnused: avalik või varjatud positiivne – negatiivne skaala, hindaja ja hinnatav, varem teadaolevad protseduurid ja hindamiskriteeriumid Assessment is the process of documenting, usually in measurable terms, knowledge, skills, attitudes and beliefs. There are two main types of assessment: Summative assessment - Summative assessment is generally carried out at the end of a course or project. In an educational setting, summative assessments are typically used to assign students a course grade. Formative assessment - Formative assessment is generally carried out throughout a course or project. Formative assessment, also referred to as educative assessment, is used to aid learning. In an educational setting, formative assessment might be a teacher (or peer) or the learner, providing feedback on a student's work, and would not necessarily be used for grading purposes. Summative and formative assessment are referred to in a learning context as "assessment of learning" and "assessment for learning" respectively

Hindamine B (evaluation) Evaluation – väärtustav hindamine Asja, isiku või nähtuse väärtuse, kvaliteedi, olulisuse, meeldivuse määratlemine. Subjektiivsem, kvalitatiivsem kui “assessment”. Evaluation is systematic determination of merit, worth, significance of something or someone using criteria against a set of standards. Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts, criminal justice, government, health care, and other. The American Evaluation Association has created a set of Guiding Principles for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows: Systematic Inquiry: Evaluators conduct systematic, data-based inquiries about whatever is being evaluated. Competence: Evaluators provide competent performance to stakeholders. Integrity / Honesty: Evaluators ensure the honesty and integrity of the entire evaluation process. Respect for People: Evaluators respect the security, dignity and self-worth of the respondents, program participants, clients, and other stakeholders with whom they interact. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of interests and values that may be related to the general and public welfare Hindamine A ja hindamine B: hinde omistamine ja väärtustamine! Erinevus tinglik, eesti keeles sama mõistega tähistatav tegevus

Sotsiaalse mõõtmise ja hindamise elemendid 1-Mõõtja ja mõõdetav, hindaja ja hinnatav 2-Instrument (mille abil mõõtmine toimub - mõõdulint, skaala, küsimustik) 3-Standard, kontekst (mille suhtes mõõtmistulemust mõtestatakse) Eripärad: –Mõõtja pole alati neutraalne –Instrumente palju, skaalad kokkuleppelised –Standardid väga erineval arenguastmel Tulemus: sotsiaalse mõõtmise tulemused reeglina tõenäosuslikud, ka numbriliselt väljendatud tulemus tähendab sageli (tugevalt subjektiivset) hinnangut, mõõtmise ja hindamise piirid ähmased, mõtestamine toimub võrdluse kaudu Kas toimub avaliku arvamuse mõõtmine või hindamine, kas mõõtmine, mille täpsusaste on +/- 3% on hindamine või mõõtmine? Mõõtmine = protseduur, hindamine = tulemuste konteksti asetamine, standardiga või teistega võrdlemine!?

Sotsiaalse hindamise vormid Väljakujunenud hindamisskeemid organisatsioonis, ühiskonnas: Eelhinnang – tulevikuvõimaluste hinnang. Tööintervjuu (kandidaadi sobivus ametikohale), sisseastumisvestlused, turunõudluse prognoos Enesehinnang – eneseanalüüsi raport, motivatsioonikiri Tulemushindamine – riigieksamid, hinnang võistkonna x hooajale Eksperthinnang – kollektiivse tarkuse kasutamine nähtuse x hindamiseks Auditeerimine – finantsaudit (protseduuride järgmine) Akrediteerimine – näit õppekava akrediteerimine (nõuetele vastavuse hindamine) Kvaliteedihindamine - EFQM Uurimistöö – isiksuse omaduste mõõtmine, organisatsioonikultuuri hindamine, majandusarengu mõõtmine. IAI ! ……………………………………………………………

Näiteid: auditeerimine Auditeerimine: isiku, organisatsiooni, protsessi või tulemuse süsteemne hindamine. Eesmärk – määratleda tegevuse nõuetele vastavuse ja usaldusväärsus. Finantsaudit, keskkonnaaudit, kvaliteediaudit … The most general definition of an audit is an evaluation of a person, organization, system, process, project or product. Audits are performed to ascertain the validity and reliability of information, and also provide an assessment of a system's internal control. The goal of an audit is to express an opinion on the person/organization/system etc. under evaluation based on work done on a test basis. Due to practical constraints, an audit seeks to provide only reasonable assurance that the statements are free from material error. Hence, statistical sampling is often adopted in audits. In the case of financial audits, a set of financial statements are said to be true and fair when they are free of material misstatements - a concept influenced by both quantitative and qualitative factors. Traditionally audits were mainly associated with gaining information about financial systems and the financial records of a company or a business (see financial audit). However recently auditing has begun to include other information about the system, such as information about environmental performance. As a result there are now professions that conduct environmental audits.

Sise- ja välisaudit Internal auditors are employees of a company hired to assess and evaluate its system of internal control. To maintain independence, they present their reports directly to the Board of Directors or to Top Management. They provide functional operation to the concern. Internal auditors are employees of the company so that they can easily find out the frauds and any mishappening. External auditors are independent staff assigned by an auditing firm to assess and evaluate financial statements of their clients or to perform other agreed upon evaluations. Most external auditors are employed by accounting firms for annual engagements. They are called upon from the out side of the company An audit is an educational term in the United States for the completion of a course of study for which no assessment is made or grade awarded. The grade of "audit" is awarded to those who have elected not to receive a letter grade for a course in which they are typically awarded. In this case, 'audit' indicates that the individual has merely received teaching, rather than achieved a given standard. The term 'audit' is Latin, translating as 'he hears'; in other words, the student has experienced the course but not been assessed. This technique is often employed by individuals who wish to take a specific course without the risk of under-performance resulting in a poor or failing grade. This can be helpful when reviewing a long-unstudied subject, when first beginning the study of a discipline wherein one has little experience or confidence, when taking a course merely for enjoyment with no need or desire of academic credit, et cetera.

Akrediteerimine Akrediteerimine: kehtestatud nõuetele vastavuse hindamine, sellekohaste tunnistuste väljastamine Accreditation is a process in which is certification of competency, authority, or credibility is presented. Organizations that issue credentials or certify third parties against official standards are themselves formally accredited by accreditation bodies (such as ); hence they are sometimes known as "accredited certification bodies". The accreditation process ensures that their certification practices are acceptable, typically meaning that they are competent to test and certify third parties, behave ethically, and employ suitable quality assurance. One example of accreditation is the accreditation of testing laboratories and certification specialists that are permitted to issue official certificates of compliance with established standards, such as physical, chemical, forensic, quality, and security standards Educational accreditation is a type of quality assurance process under which services and operations of an educational institution or program are evaluated by an external body to determine if applicable standards are met. Should standards be met, accredited status is granted by the agency Erijuht: Diplomatic accreditation is the process in which an ambassador is certified as one country's official representative to another

Kvaliteedihindamine Kvaliteedihindamine: hinnang organisatsiooni, protsessi või produkti omaduste vastavuse kohta kehtestatud (kvaliteedi)nõuetele Quality assurance, or QA for short, refers to planned and systematic production processes that provide confidence in a product's suitability for its intended purpose. It is a set of activities intended to ensure that products (goods and/or services) satisfy customer requirements in a systematic, reliable fashion. QA cannot absolutely guarantee the production of quality products, unfortunately, but makes this more likely. Two key principles characterise QA: "fit for purpose" (the product should be suitable for the intended purpose) and "right first time" (mistakes should be eliminated). QA includes regulation of the quality of raw materials, assemblies, products and components; services related to production; and management, production and inspection processes.quality It is important to realize also that quality is determined by the intended users, clients or customers, not by society in general: it is not the same as 'expensive' or 'high quality'. Even lowly bottom-of- the-range goods can be considered quality items if they meet a market need. Whereas Quality Control emphasises testing and blocking the release of defective products, Quality Assurance is about improving and stabilizing production and associated processes to avoid or at least minimize issues that led to the defects in the first place. However, QA does not necessarily eliminate the need for QC: some product parameters are so critical that testing is still necessary just in case QC fails.

Standard, norm Standard – kokkuleppeline norm, nõuete kogum, hindamiskriteerium. See, mille suhtes asja, isikut või nähtust hinnatakse. Formaliseeritud, kirjapandud, avalikustatud. Tehnilised standardid, kullastandard, kvaliteedistandardid, sotisaalabi standard … Standard is an established norm or requirement. It is usually a formal document that establishes uniform engineering or technical criteria, methods, processes and practices. The standardization process may be by edict or may involve the formal consensus experts Standard sotsiaalse mõõtmise puhul: –Täpselt määratletud tase, omadus, tunnus (punktide arv autokoolist läbisaamiseks, lävend ülikooli sissesaamiseks) –Kehtestatud normid ja nende järgimine (kvaliteediaudit) –Tunnuse jaotus populatsioonis (standardiseeritud test, keskmine eluiga) –Kontekst (võrdlus seniste hindamistulemustega, uuringutega) –Sisemine standard (eksperdi hinnang korruptsioonitasemele) Sotsiaalse hindamise standard vs tehniline standard!

Võrdlus Sotsiaalse hindamise päästerõngas – hindamistulemused konteksti, võrdlusse. Võrdlus kui orientiir, võrdlemine kui mõtestamine Võrdluses “standardiks” teised analoogilised asjad, isikud, protsessid … Hea õpilane (vaid võrdluses teistega!), arenenud riik, pikk eluiga, tark laps, tubli Eesti … Võrdluspõhine MA töö!

Mõõtmise/hindamise eesmärk – usalduse loomine Otsime: kindlust, püsivaid teadmisi, püsivaid seoseid. Kardame määramatust ja ettearvamatust. Kuidas seda teha sotsiaalses mõõtmises, mis olemuslikult tõenäosuslik ja mitmetähenduslik. Mida saame uskuda, kus on tõde? Sotsiaalses maailmas on tõde konstrueeritud, kokku lepitud, ajutine usalduse objekt?! Social constructivism holds that truth is constructed by social processes, is historically and culturally specific, and that it is in part shaped through the power struggles within a community. Constructivism views all of our knowledge as "constructed," because it does not reflect any external "transcendent" realities (as a pure correspondence theory might hold). Consensus theory holds that truth is whatever is agreed upon, or in some versions, might come to be agreed upon, by some specified group. Such a group might include all human beings, or a subset thereof consisting of more than one person.

Sotsiaalse mõõtmise usaldusväärsus Usaldus - teadmine “tõe” tekkimise protseduuride kohta. Kuigi lõplikku tõde pole olemas, toodetakse seda iga päev juurde. Usaldusväärsed tootjad on enesekriitilised ja refleksiivsed. Välja mõelnud protseduurid sotsiaalse mõõtmise usaldusväärsuse suurendamiseks Valiidsus – kas mõõdame, seda, mida arvame mõõtvat. Konstruktid ja operatsionaliseeringud Reliaablus – kuivõrd püsiv (nähtust püsivalt kirjeldav) on mõõtmistulemus? The considerations of validity and reliability typically are viewed as essential elements for determining the quality of any assessment. However, professional and practitioner associations frequently have placed these concerns within broader contexts when developing standards and making overall judgments about the quality of any assessment as a whole within a given context

Sotsiaalse mõõtmise ja hindamise loogika Mõõdame/hindame konstrukti ehk kindlas populatsioonis käibivat ja mõistetavat nähtust (organisatsioonikultuuri, pühendumus, hoiak) Mõõtmisinstrument kokkuleppeline ja üks paljudest Tulemused mõtestatud suhetes standardiga või võrdluses Nimetamise võim, klassifitseerimise võim! (sooline ebavõrdsus, vaesuspiir, eliitkoolid) Iga tulemus on vihje asjade tegeliku sisu kohta