Presentation on theme: "Outline Ethics vs. Law Ethical use of tests – issues: Competence"— Presentation transcript:
1Outline Ethics vs. Law Ethical use of tests – issues: Competence Informed ConsentKnowledge of ResultsConfidentialityTest securityDivided Loyalties
2Outline Professional Issues Social Issues Theoretical issues Test adequacyActuarial vs. clinical predictionSocial IssuesDehumanizationAccess to benefits of psychological testing
3Outline 6. Two examples of bad tests Handwriting analysisTOEFL7. The CPA Code of Ethics (synopsis)Respect for the dignity of personsResponsible caringIntegrity in relationshipsResponsibility to society
4Ethics vs. LawEthics: what one should or should not do, according to principles or norms of conductEthics codes are not produced by democratically-elected legislaturesEnforcement mechanisms are usually informal, may be complex, even unconscious
5Ethics vs. LawLaw: what one must or must not do, according to legal dictatesLaws are created by democratically-elected legislatures in democracies such as CanadaLaws come with explicit penalties for infractions and a formal enforcement systemLegislatures that are not democratically-elected, in dictatorships, do not pass real laws of this sort
6Ethical use of tests – issues: CompetenceInformed ConsentKnowledge of ResultsConfidentialityTest securityDivided Loyalties
71. Competence develop competence in assessment concepts and methods recognize boundaries of your competencecompetence is very important if test result affects someone’s lifeAny one engaged in psychological testing has a very important responsibility: to be competent. The importance of this responsibility varies with the importance of the test to the test-taker’s life. Thus, if the test is provided in a popular magazine essentially for entertainment purposes, a sensible person would not take it seriously and would not make life-altering decisions on the basis of the test result. In that case, the people developing or publishing the test have less responsibility, so long as they are careful to explain that the test is for entertainment purposes only.In contrast, if the test is given as part of a meaningful process, such as an employee selection procedure, a university admissions procedure, or an assessment of an accused person’s ability to understand legal proceedings against them, then the responsibility is significant. It includes competence in relevant technical areas. The test-developer and the test-giver must be competent in assessment concepts and methods and they must be able to recognize the boundaries of their competence (that is, to know what they know and don’t know).
82. Informed ConsentGenerally, the client/subject must voluntarily consentpsychologist must inform client about nature and purpose of assessment in understandable language
9Informed consent - exceptions Sometimes, it is acceptable to test without getting consent – e.g., you have given implied consent to be tested by registering in this course, but I have never sought your explicit consentSometimes it is necessary to test without getting consent, even when consent is explicitly refused by person to be tested – e.g., when mandated by law
103. Knowledge of Resultsmust fully disclose test results in understandable languagebut do not do so in a way that reveals the content of test items, because that content is proprietary information (e.g., somebody owns it – it’s not yours to reveal)From the joint CPA / test publishers’ statement: “To provide clients with test items, scoring criteria, and other test protocols would be to reveal confidential commercial information on which the scores are based and would render the Test Materials useless.”
11Constraints on client access to results PIPEDA Principal 4.9:Upon request, a person must be informed of the existence, use, and disclosure of his or her personal information and must be given access if requested.Section 9.3b says that access can be denied if it would mean revealing confidential commercial information (unless latter is severable).
12Constraints on client access to results Ontario Personal Health Information Act: access rights do not apply to a record that contains raw data from standardized psychological tests or assessments, unless those data are reasonably severable.CPA and leading Canadian test publishers say that Test Materials are not “personal information” so should not be released.CPA and test publishers also argue that their Test Materials are proprietary, copyrighted, confidential commercial information, analogous to trade secrets, and so refuse to release them – these materials fall under a PIPEDA exemption. Similar approach taken by test publishers in the US.
134. Confidentiality Test results are confidential information Release of results should only be made to another qualified professional after client’s consent
145. Test Security Test materials must be kept secure Test items are not revealed except in training programs and when mandated by law, to protect test integrityTest items are private property
156. Automated Scoring/Interpretation Systems Psychologist is still responsible for proper interpretation of test resultsPsychologists are professionals – they take personal responsibility for their workThe main difference between a professional and a non-professional is that professionals take personal responsibility for their work.
167. Divided loyalties Who is the client? The person being tested, or the institution you work for?What if these parties have conflicting interests?How do you maintain test security but also explain an adverse decision?
17Divided loyaltiesSuppose you are asked to test a child who might have disorder X or YSuppose X is very common and so the government provides resources to help schools deal with itSuppose Y is not common and no resources are available in school to deal with itIf your testing suggests Y, will you report Y – or X, so the child gets some sort of help?
18Professional Issues Theoretical issues Test adequacy Actuarial vs. clinical prediction
191. Theoretical IssuesAre you measuring a stable characteristic of the person being tested?If so, differences in scores over time reflect measurement error or subject variables such as fatiguePersonality is about things that don’t change – but humans are great learners, and learning should change our responses.
201. Theoretical IssuesAre you measuring a stable characteristic of the person being tested?In fact, we have little evidence that this is true, so what is the value of your test result – will it still be true next year?Especially problematic for personality testsPersonality is about things that don’t change – but humans are great learners, and learning should change our responses.
211. Theoretical issuesAre you measuring a stable characteristic of the person being tested?Suppose you test someone as part of a hiring process.If the test result says that person does not have the characteristic you’re looking for, does that mean they could never acquire that characteristic?This is a thorny problem – simply stopping the use of tests is no solution because we would still have to make decisions about people. The solution will be developed over time, through development of better theories of psychology in the various domains in which we use tests.
222. Adequacy of tests How do we measure test adequacy? (We’re supposed to be good at measuring things!)What should go into an assessment of test adequacy?
232. Adequacy of testsSo far, in considering the relative merits of various tests, we have asked whether they are psychometrically adequateWe haven’t asked: is the best test available good enough?Society uses that standard when testing becomes a legal issueBut the answer is not psychometric
243. Actuarial vs. clinical judgment U.S. Supreme Court Justice Potter Stewart said in a 1964 case that he didn’t know how to define pornography, but he knew it when he saw it.Is it possible for us to make good judgments on a question when we cannot articulate the basis for the judgment?
253. Actuarial vs. clinical judgment Actuarial judgment occurs when we feed test scores into statistical formulas to diagnose a psychological condition or predict future performance.Clinical judgment occurs when we have a trained psychologist interpret test scores to diagnose a psychological condition or predict future performance.
263. Actuarial vs. clinical judgment In actuarial judgment, we cannot make accurate predictions tailored to individualsInstead, our conclusion will be the same for every person with a given set of test scoresIn clinical judgment, the claim is that you can determine “what caused what” in an individual’s person’s life (Dawes, 1994)But clinical judgment does not improve with experienceA person giving a clinical judgment, in contrast, is saying, “I see something particular in this case, and I can use that particular bit of information to make an accurate diagnosis or prediction of outcome for this case which will be different from the diagnosis or prediction for other people who have the same set of test scores.”As Dawes (1994) put it, the clinician claims “to be able to analyze “what caused what” in an individual’s life rather than to state what is in general true.” (p.80; emphasis in original.) That claim is not true (Dawes, Faust, & Meehl, 1989; Faust, Hart, Guilmette, & Arkes, 1988; Goldberg, 1991; Ruscio, 2002).Dawes, R.M. (1994). House of Cards: Psychology and Psychotherpay Built on Myth. New York: Free Press.Dawes, R.M. Faust, D. & Meehl, P.E. (1989). Clinical versus actuarial judgment. Science, 243,
273. Actuarial vs. clinical prediction Meehl’s work in the 1950s: feed test scores into statistical formula to predict future performance – or have trained psychologist interpret scoresstatistical formula does a better job of predictionstill, someone has to be legally responsible – that has to be a trained professionalFaust, D., Hart, T., Guilmette, T., & Arkes, H. (1988). Neuropsychologists’ capacity to detect adolescent malingerers. Professional Psychology: Research and Practice, 19,Goldberg, L.R. (1991). Human mind versus regression equation: Five contrasts. In D. Chicchetti & W. Grove (Eds.), Thinking Clearly about Psychology: Essays in Honor of Paul E. Meehl. 1, Minneapolis: University of Minnesota Press
28Social issues Dehumanization Access to benefits of psychological testing
291. DehumanizationDoes computerized testing and analysis of test results create a danger of minimizing human uniqueness?humans are very complex – which allows us to be individuals, different from each otherbut testing and interpretation generalize
301. Dehumanization Is generalization a problem? Computer-scored and administered psychological tests might determine important aspects of your lifeIf you think this is a problem, why do you think so? What evidence do you have that this hurts people?What alternative approach would you use?Computer-administered and –scored tests might determine your future creditworthiness, educational opportunity, employment, psychological health, and much more
312. Access to psychological testing We’ve considered possibility of being hurt by tests – but what about possible benefits of tests? Who gets those benefits?WAIS-III kit costs $775 per person – for tester to buy the kit. Cost must be passed on to someone. Who should that be?
322. Access to psychological testing Psychological services are not covered by OHIP; Psychiatric services arePsychiatrists have opposed extending OHIP coverage to psychological testingIs that just turf protection?There is rivalry between the professions – but we could make a stronger case for billing OHIP if we used only actuarial judgment
33Two examples of bad tests Graphology (handwriting analysis)TOEFL
34Graphology (handwriting analysis) Used in hiring decisions in businessUsed to make assessments of personality type and to predict on-the-job performanceSimner & Goffin (2003): very little evidence of validity for graphologystudies using non- autobiographical writing are the most useful and show the least evidence of validity.Simner & Goffin: neither applicant nor firm has much legal recourse if graphologist’s report is wrong.
35Graphology (handwriting analysis) Dean (1992) meta-analysis of handwriting studies showed an average correlation of .16 between graphological judgment and job performance, using autobiographical writing.Correlation drops to .09 when using non- autobiographical writing.
36Graphology (handwriting analysis) Simner & Goffin: both general mental ability testing and structured employment interviews correlate around .50 with job performance. When used together, the two correlate .63 with job performance.Thus, GMA testing and interviewing are much more likely to lead to sensible hiring decisions.
37Test of English as a Foreign Language Widely used to select among students who are not native speakers of English for admission to Canadian universities.TOEFL manual presents a distorted picture of validity studiesSimner (1998) presents CPA’s recommendation – don’t use TOEFL for university admission decisions
38TOEFL – Criterion Validity TOEFL manual leaves out studies that show low to moderate correlations between TOEFL scores and teachers’ ratings of English language proficiency (.34 to .55)TOEFL manual leaves out studies that show low to moderate correlations between TOEFL scores and oral interviews (.30 to .60)Plus some of the evidence for criterion validity comes from correlations with tests such as MIchigan Test of English Language Proficiency which are not all that different from TOEFL. Thus, Simner says that these validity studies should be seen as parallel forms reliability studies instead.
39TOEFL – Criterion validity Generally, these studies assess concurrent validity, not more relevant predictive validitySimner: some criterion validity tests for TOEFL involve the Michigan TELP, very similar to TOEFL – so this is really parallel forms reliability, not validity evidencePlus some of the evidence for criterion validity comes from correlations with tests such as MIchigan Test of English Language Proficiency which are not all that different from TOEFL. Thus, Simner says that these validity studies should be seen as parallel forms reliability studies instead.
40TOEFL – Construct validity ETS: Native speakers of English do much better than non-natives on TOEFL.Simner: in these studies, native speakers are a more highly-selected group.Non-natives were unselected examinees from a range of backgrounds.Natives were selected from among entering freshmenTest anxiety differencesAnxiety – for the native speakers in these studies, nothing was at stake in the testing.
41TOEFL – Construct validity Unpublished ETS study: English speakers had an average score of 590, below the 600 cutoff used for non- natives at many Canadian universities.If native English speakers can succeed with scores < 600 why can’t non-natives also succeed with scores below 600?
42Test of English as a Foreign Language Simner: Since 1995, 10 Ontario universities have raised their TOEFL cutoff from around 550 to This means that only the top 10% of examinees are accepted, instead of top 30%.Taylor-Russell tables: r = .20, now 10% accepted, before 30%Net gain in percentage likely to be successful is 4%, at a cost of rejecting an additional 20%.
43TOEFL: Simner’s recommendations make admissions decisions on the basis of all relevant information, not just TOEFL scoresdo not use rigid cutoff scoresconsider the kinds and levels of English proficiency required in different fields and levels of studyconsider resources available for improving English language skills
44CPA Code of EthicsThe following slides show the four basic principles advanced in the CPA Code of Ethics. All of the material on the next 15 slides is quoted directly from that Code.There is significantly more information on each principle at this web address: dian%20Code%20of%20Ethics%20for%20Psycho.pdf(Note: scroll down if the page looks blank)
45CPA Principle I: Respect for the Dignity of Persons. In the course of their work as scientists, practitioners, or scientist-practitioners, psychologists come into contact with many different individuals and groups, including: research participants; clients seeking help with individual, family, organizational, industrial, or community issues; students; trainees; supervisees; employees; business partners; business competitors; colleagues; employers; third party payers; and, the general public…
46CPA Principle I: Respect for the Dignity of Persons. In these contacts, psychologists accept as fundamental the principle of respect for the dignity of persons; that is, the belief that each person should be treated primarily as a person or an end in him/herself, not as an object or a means to an end…
47CPA Principle II: Responsible Caring. A basic ethical expectation of any discipline is that its activities will benefit members of society or, at least, do no harm. Therefore, psychologists demonstrate an active concern for the welfare of any individual, family, group, or community with whom they relate in their role as psychologists. This concern includes both those directly involved and those indirectly involved in their activities.
48CPA Principle II: Responsible Caring. However, as with Principle I, psychologists’ greatest responsibility is to protect the welfare of those in the most vulnerable position. Normally, persons directly involved in their activities (e.g., research participants, clients, students) are in such a position. Psychologists’ responsibility to those indirectly involved (e.g., employers, third party payers, the general public) normally is secondary.
49CPA Principle III: Integrity in Relationships. Psychologists are expected to demonstrate the highest integrity in all of their relationships. However, in rare circumstances, values such as openness and straightforwardness might need to be subordinated to the values contained in the Principles of Respect for the Dignity of Persons and Responsible Caring.
50CPA Principle III: Integrity in Relationships. The relationships formed by psychologists in the course of their work embody explicit and implicit mutual expectations of integrity that are vital to the advancement of scientific knowledge and to the maintenance of public confidence in the discipline of psychology. These expectations include: accuracy and honesty; straightforwardness and openness; the maximization of objectivity and minimization of bias; and, avoidance of conflicts of interest. Psychologists have a responsibility to meet these expectations and to encourage reciprocity.
51CPA Principle III: Integrity in Relationships. In addition to accuracy, honesty, and the obvious prohibitions of fraud or misrepresentation, meeting expectations of integrity is enhanced by self-knowledge and the use of critical analysis.
52CPA Principle IV: Responsibility to Society. Psychology functions as a discipline within the context of human society. Psychologists, both in their work and as private citizens, have responsibilities to the societies in which they live and work, such as the neighborhood or city, and to the welfare of all human beings in those societies.
53CPA Principle IV: Responsibility to Society. Two of the legitimate expectations of psychology as a science and a profession are that it will increase knowledge and that it will conduct its affairs in such ways that it will promote the welfare of all human beings.
54CPA Principle IV: Responsibility to Society. Freedom of enquiry and debate (including scientific and academic freedom) is a foundation of psychological education, science, and practice. In the context of society, the above expectations imply that psychologists will exercise this freedom through the use of activities and methods that are consistent with ethical requirements.
55CPA Principle IV: Responsibility to Society. The above expectations also imply that psychologists will do whatever they can to ensure that psychological knowledge, when used in the development of social structures and policies, will be used for beneficial purposes, and that the discipline’s own structures and policies will support those beneficial purposes. Within the context of this document, social structures and policies that have beneficial purposes are defined as those that more readily support and reflect respect for the dignity of persons, responsible caring, integrity in relationships, and responsibility to society.
56CPA Principle IV: Responsibility to Society. If psychological knowledge or structures are used against these purposes, psychologists have an ethical responsibility to try to draw attention to and correct the misuse. Although this is a collective responsibility, those psychologists having direct involvement in the structures of the discipline, in social development, or in the theoretical or research data base that is being used (e.g., through research, expert testimony, or policy advice) have the greatest responsibility to act. Other psychologists must decide for themselves the most appropriate and beneficial use of their time and talents to help meet this collective responsibility.
57CPA Principle IV: Responsibility to Society. When a person’s welfare appears to conflict with benefits to society, it is often possible to find ways of working for the benefit of society that do not violate respect and responsible caring for the person. However, if this is not possible, the dignity and well-being of a person should not be sacrificed to a vision of the greater good of society, and greater weight must be given to respect and responsible caring for the person.
58CPA’s Caveat…Even with the above ordering of the principles, psychologists will be faced with ethical dilemmas that are difficult to resolve. In these circumstances, psychologists are expected to engage in an ethical decision-making process that is explicit enough to bear public scrutiny. In some cases, resolution might be a matter of personal conscience.
59CPA’s Caveat…However, decisions of personal conscience are also expected to be the result of a decision-making process that is based on a reasonably coherent set of ethical principles and that can bear public scrutiny. If the psychologist can demonstrate that every reasonable effort was made to apply the ethical principles of this Code and resolution of the conflict has had to depend on the personal conscience of the psychologist, such a psychologist would be deemed to have followed this Code.