Psychological Assessment Seminar

Psychological Assessment Seminar
Rubik’s cubes of colour, shape and size: Ethical psychological testing in a multicultural work environment Department of Industrial & Organisational Psychology 18 August 2010

agenda Morning session: 08h30 - 13h00
• Welcome by Prof Dirk Geldenhuys • Introduction: Welcome to the Rubik’s Cube Adventure • A short history of psychological assessment • The areas of psychological assessment • Psychological assessment myth busters Tea break: 11h00 • Legislation and ethics • Ethical freeze role play Lunch: 13h h45 • Current topics in psychological assessment • Psychological assessment Crosswords • The future of psychological assessment Conclusion: 15h00 Dieter to do introduction and talk around the analogy of the rubik’s cube.

A short history of Psychological Assessment

Why look at the past? History influences current practice Relevance
Evolved and progressed History is important Helps explain current practice Strengths and weaknesses Prevent repetitions of the ‘wrongs of the past’

History of testing: An International Perspective
Over the years many authors, philosophers and scientist have explored various avenues in an attempt to assess human attributes. To mention a few; Astrology (positions of planets to describe the personality of individuals and predict what may happen in their lives) Physiognomy (judging the character of a person from external features of the body, especially the face.) Graphology (the systematic study of handwriting and the notion that it can be used as an expression of personality characteristics) Humorology Phrenology (Foxcroft, 2009)

Testing in South Africa: The Early Years
Psychology emerged as an academic discipline in South Africa after World War 1. Many of the influential figures in the development of intellectual tests in South Africa where trained at American institutes. Therefore tests which where typically used in the early phases where also American such as; Goddards 1911 revision of the Binet-Simons test, Termans 1916 Stanford revision, the Army Beta, the Porteus Maze Test etc. In the period between the two world wars, social and human sciences in South Africa became important contributors to debates on crucial social issues including ‘the Native question” (Louw, 1997; Foxcroft, 2009; Classen, 1997)

Testing in South Africa: The Early Years
Psychological knowledge was used as a tool legitimise a social order based on race as the perception was that psychological testing produced empirical data which supported certain explanations of this order. There was an increase in the use of science as a factor in regulating aspects of social economic life. Differential performance of black and white reinforced the idea of hierarchy of human societies and consequently differential treatment in terms of education, employment etc. (Louw, 1997)

Key notes

Psychometric Testing Psychometric testing is applied worldwide and used in various industries for recruitment, selection and counselling purposes (Gregory, 2000). For the purpose of this seminar four important areas for assessment have been identified. Cognitive assessment Personality assessment Behavioral assessment Interest

Range of tests and distributors available - Cognitive
MEASUREMENT CONSTRUCT CONCEPT TEST DISTRIBUTORS APPLICATION IN INDUSTRIAL/ PSYCHOLOGICAL FIELDS Cognitive IQ, General cognitive, Multiple aptitude, Learning potential WAIS, SSAIS, ASAT, RPM, DAT Forms R, S, K & L Mindmuzik Media Personnel screening and selection, Career guidance and counseling, Personnel development CPP, CTA Magellan Consulting APIL-B, TRAM 1 & 2, AproLab M&M Initiatives LPCAT TST Thomas International COPAS Integrity International WAIS – Weschler Adult Intelligence Scale SSAIS – Senior South African Individual Scale ASAT – RPM – Raven’s Progressive Matrices DAT – Differential Aptitude Test forms R, S, K and L. CPP – Cognitive Process Profile CTA – Cognitive Task Assessor APIL-B - TRAM 1 & 2 – LPCAT – Learning Potential Computerized Adaptive Test TST – Test for selection and Training COPAS – Cognitive and Potential Assessment

Cognitive Assessment Some consensus
IOPs agree that cognitive ability test are valid and fair Cognitive ability tests provide good but inadequate measures of intelligence Other attributes are necessary to account for multi dimensional nature of performance (90% of respondents) (Murphy, Cronin & Tam)

Cognitive Assessment Some controversy Societal concerns Cultural bias
Education Socio-economic status Education levels of parents Effects of negative stereotypes The g-Ocentric vs MI (Murphy, Cronin & Tam) (McKay, Doverspike, Bowen-Hilton & McKay)

Cognitive Assessment Unifying theoretical approach 3 approaches
Structural approach (factor analytic) Information-processing Dynamic (based on learning theory) “Learning as the critical factor underlying cognitive competence and the mastery of problems and challenges” (Taylor)

Structural - Spearman (g – one general factor) and Thurstone 7&9 Primary mental abilities
static - fluid & crystallised (Cattell) - mostly power tests Info processing – e.g. Ravens progressive matrices receiving, processing & retrieval speeds computer admin essential Learning / dynamic approach Vygotsky – Zone of proximal development (ZPD) less susceptible to cultural bias 3 phases of learning – conceptual understanding, compilation of execution procedures & automatization of processing. Test – teach – test or pre-test – mediation – post-test Emphasis on potential and not achievement

Range of tests available: Personality
MEASUREMENT CONSTRUCT CONCEPT TEST DISTRIBUTORS APPLICATION IN INDUSTRIAL/ PSYCHOLOGICAL FIELDS Personality Personality preferences, Management styles, Communication styles, Interpersonal styles, Coping styles MMPI, TAT, SORT, VS , WLQ Mindmuzik Media Personnel selection, Career counselling, Academic counselling, Personal development, Psychological growth, Candidate - job match, Management development, Stress management OPQ 32 SHL 15 FQ, OPP, Jung Psytech MBTI, Big Five, WPI, LSP MSP, CPI Jopie van Rooyen PAW Integrity International Names of assessments: MMPI – Minnesota Multiphasic Personality Inventory TAT – Thematic Apperception Test SORT – Structured-Objective Rorschach Test VS – Value Scale WLQ – Experience of Work and Life Circumstances Questionnaire OPQ 32 – Occupational Personality Questionnaire 15 FQ – Fifteen Factor Questionnaire OPP – Occupational Personality Profile Jung – The Jung Big Five – Big Five Model of Personality WPI – Work Personality Index LSP – Leadership Skills Profile MSP – Management Success Profile CPI – California Psychological Inventory PAW – Personality at Work

Range of tests available: Personality
MEASUREMENT CONSTRUCT CONCEPT TEST DISTRIBUTORS APPLICATION IN INDUSTRIAL/ PSYCHOLOGICAL FIELDS Personality Locus of control, motivation and autonomy LCI AMP Jopie van Rooyen Personnel selection Career counselling Academic counselling, Personal development Psychological growth, Candidate - job match, Management development, Stress management Emotional Intelligence MP SHL, Magellan Consulting EQI 15 FQ EI Psytech TEI Thomas International Integrity Giotto Mindmuzik Media PSI IP-200 IMI Integrity International LCI – Locus of Control AMP – Achievement Motivation Profile MP – Motivation 15 FQ Emotional Intelligence report EQI – Emotional Quotient Inventory (The new version of BARONEQ) 15 FQ – Fifteen Factor Questionnaire Emotional Intelligence Report TEI – Thomas Emotional Intelligence Giotto – Giotto PSI – Personnel Selection Inventory IP-200 – Integrity Profile with 200 items IMI – Integrity Measurement Instrument

Range of tests available: Behavioral
MEASUREMENT CONSTRUCT CONCEPT TEST DISTRIBUTORS APPLICATION IN INDUSTRIAL/ PSYCHOLOGICAL FIELDS Behavioural Underlying emotional issues, Stress, Anxiety, Depression, Conflict BESS Jopie van Rooyen Screening and selection Training and development Promotion Management potential Group-role analysis Interpersonal relationships Career psychology Personnel psychology Organizational psychology BAI BDI-II BDI Fast screen Mindmuzik Media SWSI SRP MBI SP TKI Conflict management Personal development BESS – Behavioral and Emotional Screening BAI –Beck Anxiety Inventory BDI-II - Beck Depression Inventory BDI Fast Screen – Beck’s Depression Inventory Fast Screen SWSI – Sources of Work Stress Inventory SRP – Stress Resilience Profile MBI – Maslach Burnout Inventory SP – Stress Profile TKI – Thomas Killman Conflict Mode

Range of tests available: Interest
MEASUREMENT CONSTRUCT CONCEPT TEST DISTRIBUTORS APPLICATION IN INDUSTRIAL/ PSYCHOLOGICAL FIELDS Interest Career SDS CVS CIP Jopie van Rooyen Career guidance and counselling Personal development and growth 19 FII CDQ SAVII Mindmuzik Media IDEAS workbook COP ?? SDS – Self Directed Search CVS – Career Values Scales CIP – Career Interest Profile 19FII – Nineteen Field Interest Inventory CDQ – Career Development Questionnaire SAVII – South African Vocational Interest Inventory IDEAS – Interest, Determination, Exploration and Assessment System COP – Career Orientations Inventory

Cost Pricing Analysis Option 1: Purchasing a “report”
Advised for non-accredited users and for individual testing on once-off basis Some test can be expensive (up to R3,000 per report) Suppliers include SHL, Jopie van Rooyen, and Thomas International Benefit: initial costs is low Computer generated reports is time efficient Downfall: ongoing high costs Potential misuse of reports due to lack of control

Cost Pricing Analysis Option 2: Purchasing Material: Software or Paper-and-Pencil version Advised for accredited users and group or large-scale testing on ongoing basis Initial purchasing of material and software can be expensive (need manuals, booklets, answer sheets, scoring stencils, etc – between R2,000 – R25,000) Suppliers include Minimize, M&M Initiatives, PsyTech Benefit: ongoing assessments at minimal fee Downfall: initial start-up fees can be high Software need ongoing purchasing of credits for reporting Manual interpretation, scoring and reporting is time consuming Ofentse

Key notes

Legal aspects of Psychological Assessment

The Constitution of the Republic of South Africa:
Chapter 2: Bill of Rights “is a cornerstone of democracy in South Africa - democratic values of human dignity, equality and freedom”. Section 9(3): “The state may not unfairly discriminate directly or indirectly against anyone on one or more grounds, including race gender, sex, and pregnancy, marital status, ethical or social origin, colour, sexual orientation, age, disability, religion, conscience, belief, culture, language and birth. Section 9 (4): “No person may unfairly discriminate directly or undirectly against anyone on one or more grounds in terms of subsection (3).

Employment Equity Act, No.55 (1998)
Section 8: “Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used – Has been scientifically shown to be valid and reliable; Can be applied fairly to all employees; Is not biased against any employee or group. South African law requires psychologists to be proactively involved in preventing unfair discrimination by requiring evidence that test are fair and unbiased.

EEA (continued) Section 11: Burden of proof.
Section 15: Affirmative action. Subsection 20(3): “capacity to acquire, within a reasonable time, the ability to do the job” Code of Good Practice on the Integration of Employment Equity into Human Resources Policies and Practices: Section 10.

The Labour Relations Act, No. 66 (1995)
Chapter VIII: Unfair discrimination and unfair labour practices – same grounds as the Constitution.

Legal Cases Association of Test Publishers of South Africa Saville & Holdsworth (SA) and the Chairperson of the Professional Board for Psychology Dispute: ATP: Any individual (unregistered) should be able to conduct psychometric assessments? HPCSA: Only registered professionals ! Ruling (19 February 2010): In favour of ATP SA. “The court order declares the notice published on the 10th November 2008, which stated that “it is not permissible to use unregistered persons to render psychological services including the administration of tests, instruments or techniques”, to be void and of no force and effect. Current status: This ruling will stand until further notice. WATCH THIS SPACE!!!!

Legal Cases (continued)
Buthelezi v Amalgamated Beverage Industries [1999] JOL 5086 (LC) Hendricks v SA Airways [2002] JOL (LC) NUM obo Moeng / Douglas Colliery [2007] 7 BALR 647 (CCMA)

What will the long arm of the law do to you?
Fines: Employment Equity Act Costs: CCMA, Labour Court, Constitutional Court

Closing statement Assessments play an important role in the measuring of performance, selection, recruitment, etc. in the workplace, schools and various other institutions. Assessment Practitioners should today more than ever be aware of the responsibilities that the legal use of these powerful instruments require. Our history has not been without bumps, hurdles and mountains, but it is our responsibility to promote a fair and unbiased future and set an example for the rest of the world. The current limitations and challenges that we face can only be achieved through co-operation, adhering to legislation and high ethical standards.

We rest our case…

The Regulation of Psychological Assessment
Regulation by Legislation Regulation by Professional Bodies Self-Regulation

LEGISLATION Employment Equity Act (1998) – different to all other countries, in that we are asked to proactively defend our position with regards to psychological assessment PROFESSIONAL BODIES (Responsibilities: Training, Accreditation, Test classification) HPCSA -- might not be in a position to efficiently regulate access to and use of tests Psychometric Committee of the board of Psychology – falls under HPCSA International Test Commission (2001) Society for Industrial and Organisational Psychology of South Africa People Assessment in Industry Human Sciences Research Council

PROFESSIONAL BODIES (cont.)
South African Board for People Practices Association of test publishers Test Developers/Publishers Any others? What about universities? SELF-REGULATION Strategy of self-regulation (Paterson & Uys, 2005) – practitioners empowered to make informed decisions through access to information and adequate training Training = better decision making with regards to selection of (registered) tests Continuing professional development Personal moral compass

ADDITIONAL ISSUES TO CONSIDER
Conflicting policies (e.g. Employment Equity Act (1998) – assessments must not discriminate against groups, but can be used to assist with addressing previously disadvantaged groups) Personal morals vs Ethical policies Canadian Psychological Association – ethical principle ranking to assist decision making

Competency Based Assessment -
One minute wonder Or Here to stay?

Competency Based Assessment Development
What is competency-based testing? Competencies Knowledge, Skills, Abilities Job Analysis Effective in Application Movement from intelligence testing

Defining Competency Based Assessment
A way to monitor and assess the competences a person has, no matter where or how these competences were gained. Three major principles differentiating competence-based assessment from other methods used to assess are: Assessment of evidence Current abilities Standards or competence

Benefits of Competency Based Assessment
An assessment of past behaviour can assist you in predicting future behaviour Multiple assessment techniques are used to verify the results obtained The competencies measured are related to successful job execution Objectivity of evaluation is ensured through the utilisation of trained assessors

Recognition of Prior Learning
“Recognition of prior learning is a process whereby, through assessment, credit is given to learning which has already been acquired in different ways”. – SAQA Discuss practical example of the RPL process (real life case study)

Benefits of RPL RPL is seen to have the capacity:
to contribute to redress equity by opening up more ways for people to attain qualified status; enable more people to reach higher levels of qualification and expertise by beginning with an acknowledgement of existing skills and knowledge; contribute to enhancing international economic competitiveness by building on often invisible and unacknowledged workforce skills; and offer the first step in attaining the goal of developing a multiskilled and flexible workforce by acting as an auditing tool to quantify existing competence.

Competency Based Assessment Series (CAS)
Designed by JvR professionals and associates Purpose is to provide their clients with a range of uniquely South African assessment centre exercises the exercises are designed to evaluate performance against certain job relevant competencies and can contribute to an informed selection decision. The exercises can be categorised into four groups, namely: In-basket Exercises Group Exercises Strategy and/or Presentation Exercises Role Play Exercises All the exercises are job relevant and assess current as well as future potential.

Examples of Competency Based Assessment
Competency Assessment Series (CAS) The aims of the CAS exercises include: To identify behavioural competencies needed for a specific job. To assist the assessor in gaining a comprehensive understanding of the candidates’ behavioural competencies. Provide additional, job-relevant, “tangible” information to supplement that gathered with traditional psychometric tools. To determine the participants’ strengths and development areas. To use the information in combination with collateral information for selection and development purposes in the workplace.

Examples of Competency Based Assessment
Case studies In-basket exercises Job sample tests/Skills Tests Direct Observation Oral/Written Tests Portfolios Simulation

Designing a Competency Based Battery
Step 1: Planning Purpose Step 2: Establish Competencies Job Analysis Step 3: Develop assessment battery Structure Step 4: Implement and evaluate battery

Uses of Competency Based Assessment
Within in the organisation In the South African context Advantages Disadvantages Fairness Unbiased, ethical

Key notes

Computerised Assessment
Introduction – Calum and Des

Before we start CAPA CBA CBT CAT CBTI IDT
Before we can drill into the details it is important to go through a few definitions first! This way we can ensure that we are all on the same page! (CAPA) – refers to the entire range of computer applications in psychological assessment. This includes administration, scoring and interpretation of tests; computerized adaptive resting; and sophisticated multimedia applications (Gregory, 2007). (CBA) Computer-based assessment - the term CBA may be considered to include any psychological assessment that involves the use of digital technology to collect process and report the results of that assessment. The figure that follows summarises four components of a CBA system for the delivery of psychological assessment. (CBT) A computer-based test refers to a test that is administered and scored using a computer. This type of testing is referred to as computer-based testing (CBT) (Foxcroft & Roodt, 2006). (CAT) Computerized-adaptive testing is CBT that uses a set of statistical procedures that make it possible to tailor the test for the individual being assessed on the basis of their responses and which allow for more accurate and efficient measurement (Foxcroft & Roodt, 2006). (CBTI) Computer-based test interpretation systems which aid the assessment practitioners in interpreting test performance and provide hypothesis related to understanding the test-taker better, making a diagnosis, and suggesting treatment possibilities (Foxcroft & Roodt, 2006). (IDT) Internet delivered testing represents a method of administering computer-based tests via the World Wide Web (Foxcroft & Roodt, 2006).

Computer-based Assessment
Assessment Generation Assessment Delivery Assessment Scoring and Interpretation Storage, Retrieval and Transmission

Taking a step back in History
International: 1962 1970’s 1980’s 1990’s 2000’s 1962: the first CBTI system was developed at the Mayo Clinic for the MMPI followed by the development of a CBTI system for the Rorschach Inkblot Test in 1964 1970’s: first automation of the entire assessment process at a psychiatric hospital in Utah and the introduction of CAT. first original applications of computerized testing involving the computerisation of the items of paper-and-pencil tests for both administration and scoring purposes 1980’s: emergence of advance computer technology with increased possibilities for design, administration, and scoring of tests. 1990’s : computer-based testing and it’s delivery over the Internet increased flexibility to testing in terms of being able to register for testing via or telephone, be tested by appointment in a designated testing centre and receive their scores at the end of the test. 2000’s : computers are being used developing tests and generating items. Item engines as well as software allow for the writing of tests at the touch of a key. Artificial intelligence has further enhanced item engines by enabling the software to ‘learn’ in the act of production, which enhances the production of subsequent items and other versions of a test.

Taking a step back in History
South-Africa 1970 Late 1970’s and early 1980’s 1979 1980’s 1986 1990 1970’s: National Institute for Personnel Research (NIPR) and the Council for Scientific and Industrial Research developed tests for selecting military airline pilots among other forerunner projects of computerized testing. Late 1970’s to early 1980’s: Malcom Coulter develops a computerized testing system at the NIPR on a VARIAN minicomputer. The system was later adapted for research project on basic cognitive abilities. 1979 : NIPR undertakes a contract to computerize tests for selection purposes at Eskom. Almost the full range of tests was computerized in this project using the PLATO based test system, and a sophisticated test management system was developed, enabling assessment practitioners at remote locations to request an assessment battery to be set up by sending their requirements via the network to the testing site. 1980’s: the PLATO based system consisting of 14 terminals eventually became too expensive to maintain due to sanctions placed on South Africa by other countries because of apartheid. 1986: NIPR and the Human Sciences Research Council (HSRC) developed a new PC based system called PsiTest. The system was used by organisations well into the 1990’s. 1990’s: Also saw the development of computerized report-writing systems for existing tests. : the first windows based test was developed by Nanette Tredoux in South Africa for Liberty Life. The system was called PsiSim.

Advantages and Disadvantages
More enjoyable Ultimate levels of standardization Biasing effect is eliminated Reduction of time More information about test-takers Spatial and perceptual measured to greater extend Advantages – More enjoyable for test takers than pen and paper Ultimate levels of standardization of assessment instructions are achieved The potential biasing effect of the assessment practitioner is eliminated as the computer administers and scores the measure in an objective way There is a reduction in the amount of time need for the assessment It provides the opportunity to obtain more information about test-takers (such as precisely recording and storing their response time to items), as well as providing instant scoring that allows for prompt feedback to both the assessment practitioners and test takers The graphic capabilities of computers make it possible to measure spatial and perceptual abilities to a far greater extent than is possible with pen and paper-based tests

Advantages (continued) Voice activated and touch screen responses are possible Assessments can be individually tailored Greater element of control Number of assessment practitioners and assistants needed Scoring by a computer Increased test security

Eish Eskom Hardware / software compatibility & capability Screen size Machine availability Socio-cultural and linguistic factors System failure Power conditions (EISH ESKOM) Hardware/software compatibility and capability Screen size and clarity – studies suggest that computer screens take longer to read than printed materials Machine availability – (RURAL AFRICA) Socio-cultural and linguistic factors influence the test takers ability to interact with the computer – given the diversity of cultures in South Africa, the role of the cultural background of test takers is particularly pertinent System failure – where the computers crash and the system is brought to a standstill. One is less likely to lose all the answer sheets of a pen and paper administered assessment

Disadvantages (continued) Copy right violation Lack of security Problems with confidentiality Skilled clinical judgement is overlooked Difficult to detect certain problems in software High costs in item development Costly Copy right violation is a concern for instruments available on the internet Lack of security with measures on the internet Problems with confidentiality which could arise when unauthorized people gain access to large amounts of information stored on computers Computer generated assessment reports still require skilled clinical judgement as far as interpretation is concerned – something which is often ignored by inexperienced assessment practitioners Computerized scoring routines may have errors or may be poorly validated; such problems are often difficult to detect in software Computerized testing involves high costs in item development as a much larger item pool is required in order to address the issues of security and over-exposure of items, as well as technical issues associated with computerized adaptive testing Computerized assessment packages are sometimes unnecessarily costly and the psychometric properties of the computerized measures have not always been researched adequately as failure to have the product on the market quickly places a financial burden on the test development company

Disadvantages (continued) Vital qualitative information not assessed Human-computer interface issues Lack of computer literacy Vital qualitative information about test taking behaviour and problem-solving strategies cannot be accessed readily during computerized assessment Human-computer interface issues arise in that test-takers, especially older adults, may have a phobia about using computers; this could raise their anxiety levels and, in turn, have a negative impact on their test performance Lack of computer literacy on the part of some test-takers could impact negatively on their performance; in this regard there is a need to expose test-takers to a practice session and a practice measure to provide them with sufficient time to familiarize themselves with the keyboard and using a computer before the assessment takes place (Foxcroft & Roodt, 2006).

Computer-based Interpretation
Four approaches to CBTI are recognized: Scoring reports Descriptive reports Actuarial reports Computer-assisted clinical reports Scoring reports: Consist of only scores and or profiles but may include statistical significance tests and confidence intervals plotted for the tests scores. These reports highlight meaningful scores and score differences at a glance (Gregory, 2007). Descriptive reports: Provide brief scale-by-scale interpretation of test results. These reports are especially useful when test findings are conveyed to mental health who have little knowledge of the test in question (Gregory, 2007). Actuarial reports: An empirically derived formula is used to diagnose, classify, or predict behaviour. This is in contrast to the clinical approach in which the psychologist processes information in his or her head to diagnose, classify, or predict behaviour (Gregory, 2007). Computer-assisted clinical reports: The interpretive statements are based upon the automated and computerized judgement of one or more expert clinicians. This approach allows for interpretation of all test profiles, not just those that fit certain actuarial patterns (Gregory, 2007).

CAT What is it Purpose Characteristics Measurement Advantages
CAT Computerized adaptive testing - is a family of procedures that allows for accurate and efficient measurement of ability. In this approach the computer guides item selection based upon prior examinee answers (Gregory, 2007). The purpose of CAT is to reach a predetermined level of measurement accuracy with as few test items as possible. A typical finding is that CAT reduces test length by about 50% with no loss in measurement accuracy (Gregory, 2007). Most forms of CAT share the following characteristics: Based on extensive pretesting, the item response characteristics of each item are appraised precisely. These item response characteristics and a CAT item-selection strategy are programmed into the computer. In selecting the next item for presentation, the computer uses the examinees total history of responses up to that point. The computer recalculates the examinees estimated ability level after each response. The computer also estimates the precision of measurement (standard error of measurement) after each response. Testing continues until a predetermined level of measurement precision is reached. The examinees score is based on the difficulty level and other measurement characteristics of items passed, not on the total number items correct. Measurement advantages of CAT Precision – testing continues until a predetermined degree of precision is met Efficiency – far fewer test items are needed than traditional tests Other advantages include: Improved test security Examinee’s work at their own pace Examinees are equally challenged Answer sheets pose no ambiguity Immediate scoring and feedback is possible Pretesting of new items can be included Faulty items can be eliminated immediately A variety of question types can be included

Dynamic Assessment What is it? Increased attention
What does it involve? South Africa WHAT IS IT? It is a specific approach to assessment which incorporates training into the assessment process in an attempt to evaluate not only the current level of cognitive ability, but also the potential future level of ability (Foxcroft & Roodt, 2005). INCREASED ATTENTION: In an attempt to provide more equitable cognitive assessment in the last few decades, dynamic testing and the measurement of learning potential have received increasing attention both locally and internationally. WHAT DOES IT INVOLVE? Measurement of learning potential typically involves a test-retest strategy with some form of help or training provided as part of the assessment process. The provision of a leaning opportunity in the test administration provides fairer assessment of disadvantaged groups in particular. By providing a learning opportunity in the assessment the focus is not only on the present level of performance, but also on the potential future levels of performance that can be achieved if relevant training can be provided (De Beer, 2006). SOUTH AFRICA? Measures have recently specifically been developed for the purpose of dynamic assessment. In South Africa, Taylor (1994) has developed a family of measures, namely the APIL-B, TRAM1 and TRAM2, based on his approach to measurement of learning potential. Professor Marie De Beer (1998, 2000a, 2000b) developed the Learning Potential Computerized Adaptive Test, LPCAT, a dynamic computerized adaptive measure for the assessment of learning potential.

Assessing Potential Effective management tool Time and cost effective
Usage The LPCAT is an effective management tool that provides accurate information regarding the potential levels of development of employees. The LPCAT is a time and cost effective assessment method which provides accurate and scientific information to help unlock cognitive potential in a multicultural context. In so doing, it provides excellent decision making information regarding levels at which present as well as future training and development should be aimed. Usage: Numerous companies use the LPCAT, in combination with other instruments for decision making in bursary student selection, personnel development and also in selection. Although the performance in the LPCAT is not reliant on language proficiency or prior educational qualification, the results of the LPCAT indicates the present and potential levels of general reasoning ability, which can be interpreted in terms of academic or educational level. If a person would be expected to undergo training at a prescribed academic level, the LPCAT results can indicate how difficult or easy this person is likely to find training at that level. In cases where a person does not have the required academic or formal training qualifications, the results can indicate to what level this person can be trained over time. Screening and selection Development Training Promotion Career planning

Assessing Potential Why will we use it? Present and potential
Opportunity to be measured fairly. Is suitable at all levels Scientifically proven valid and reliable Learning takes place during the assessment Is extremely time-saving Uses computerised adaptive testing Based on extensive multicultural research at various levels, the LPCAT was developed over a period of eight years. The LPCAT: Provides information regarding the present and potential - not yet developed - future levels of reasoning ability of an individual. Gives every person regardless of qualification, culture or gender the opportunity to be measured fairly. Is suitable for use at all levels of ability - tertiary, secondary as well as primary, not excluding illiterate adults. Provides scientifically proven valid and reliable information regarding the potential level at which individuals can cope with and benefit from training. Provides for learning to take place during assessment by making use of the dynamic test-teach-retest strategy. Is extremely time-saving, as assessment information is quick to obtain and easy to interpret (Pre-test, training and post-test all done within approximately one hour). Uses computerised adaptive testing, whereby items of appropriate difficulty level are selected during testing to match the estimated ability level of the individual.

Assessing Potential Advantages: Fair
Suitable for use at all levels of ability; Valid and reliable indication of potential to cope with training; Time saving; Uses CAT which allows for all ability levels.

Item Response Theory Definition
Higher an individual’s ability level, the greater the chance of getting an item correct. Each item is referenced IRT is a test theory used to develop and assemble test items, detect bias in measuring instruments, implement computerized adaptive tests, and analyse test data (Foxcroft & Roodt, 2005). The basic assumption is that the higher an individual’s ability level is, the greater the chances of the individual getting an item correct. In essence instead norming an entire instrument, each item is referenced according to the amount of ability needed to answer the item correctly, this makes computerized adaptive testing possible where each item is presented based on the history of items already attempted.

Item Response Theory The relationship between the probability of answering an item correctly and ability level is depicted above. The X-axis represents the ability level of test takers, while the Y-axis represents the probability of answering an item correctly. The curve depicted above is called the Item Characteristic Curve (ICC) – the slope of the curve indicates the discriminating power. The steeper the slope is, the higher the discriminating power of an item.

Good practice and Ethical considerations
The International Test Commission (ITC) Give due regard to technological Attend to quality issues in CBT and internet testing; Provide appropriate levels of control Make appropriate provision for security According to The International Test Commission (ITC) guidelines, assessment practitioners who use computer-based and Internet-delivered testing need to do the following: Give due regard to technological issues in computer-based and internet testing; Attend to quality issues in CBT and internet testing; Provide appropriate levels of control over the test conditions; and Make appropriate provision for security and safeguard privacy in CBT and Internet testing.

Minimum professional and ethical standards: Competence Potential utility Choose a technically (psychometrically) sound computer-based test Equivalence of paper and computer-based version Consider the human factors and issues of fairness The following good practice guidelines represent the minimum professional and ethical standards that South African assessment practitioners should adhere to when using computer-based and Internet delivered tests: Ensure that the practitioners themselves have the competence to use computer-based and Internet delivered tests. Establish the potential utility of the computer-based test. Choose a technically (psychometrically) sound computer-based test that has been evaluated and classified by the Psychometrics Committee of the Professional Board for Psychology. Check for equivalence of paper and computer-based version, especially if the two versions are to be used interchangeably. Give consideration to human factors and issues of fairness when using a computer-based test. Where test takers have low levels of computer familiarity or have certain neurological or physical problems that make computer-administered testing unsuitable, assessment practitioners should offer them an alternative paper-based option.

Prepare test takers Verify the identity of test-takers Closely supervise the administration Unsupervised psychological test Contingency Securely stored. Computer scoring system Interpretation of results Debrief test-takers Prepare test takers appropriately through practice tutorials or tests. Verify the identity of test-takers. This is especially important for Internet delivered testing. Closely supervise the administration of the computer-based test so as to provide appropriate support to the test taker during the session. Psychology professionals in South Africa are not permitted to allow test takers to complete unsupervised psychological tests via the internet. A professional needs to be in attendance at all times, even for internet-delivered testing, as it is illegal for a paper-based, computer-based, or internet delivered psychological test to be administered by anyone other than a psychology professional registered with the HPCSA. Ensure that contingency plans are in place if technology fails. For example switch to a paper-based test if is equivalent to the computer-based version. Ensure the computer-based test is securely stored. Check the computer scoring system and the accuracy of the classification system used to generate reports. Interpret results appropriately and be aware of limitations of computer-based test interpretation (CBTI) systems. Modify the report where necessary and complement it with additional information before signing (and thus professionally owning) it. Ensure that the results are securely stored. Debrief test-takers. This is especially important if test-takers became anxious as a result of having to use computer technology to complete the test.

The future – high definition video and virtual reality
Multimedia A five-dimension framework Innovations in item format Innovations in response Innovations in media inclusion Innovations in the extent of interactivity Innovations in scoring Authenticity 9) The future – high definition video and virtual reality – Multimedia includes realistic, interactive presentation of test stimuli via computer (e.g., video display of a work situation). Multimedia allows for the testing of complex, real-life problems, such as conflict resolution in the workplace (Gregory, 2007). A five-dimension framework for understanding and driving innovation in CBA: Innovations in item format – specifically with regard to the way the test taker responds to an item, e.g. multiple choice or essay Innovations in response action – the way the test-taker responds to an item, e.g. double click or drag and drop Innovations in media inclusion – audio, video, virtual reality simulations etc Innovations in the extent of interactivity – the extent to which the test architecture reacts to the response of the test-taker e.g. CAT Innovations in scoring methods – conversion into quantitative scores e.g. IRT used to score CAT’s, and sophisticated algorithms can be use to score essays Authenticity – item types should try to emulate real-world tasks which make them more authentic in the eyes of the test-taker, to construct items with face validity

The Future of Psychological Assessments

Research building on existing knowledge
Enhancing consumer psychology Reduced costs System generated reports Dynamic assessment of intelligence Construction of new adaptive tests

The Future Role of the HPCSA
Need for an all encompassing body to: Monitor test use Advise practitioners Research and review tests with information centrally available To monitor and coordinate test development, adaptation, and updating (Foxcroft, 2004) Control and regulation Increased availability of information via internet

Artificial Intelligence
Virtual Reality – a new tool for psychological assessment (Fernandez-Ballesteros, 2009) Computer assisted assessment Ethical considerations Control and access to electronic assessments Closer examination of the tester-testee relationship School of Psychology Virtual Reality Research Team Early in 1995, we opened a virtual reality (VR) laboratory within the School of Psychology to investigate the use of computer generated virtual environments in neurological assessment and rehabilitation. The laboratory is equipped with a fully immersive system, a 10-foot rear projection screen with stereoscopic facility, a head-mounted display, 3D motion trackers, a data glove, and a number of “desktop” systems. The research team is headed by Professor David Rose, who is assisted by Dr Paul Penn (Research Fellow), Tony Leadbetter (Programmer) and Ambi Ambihaipahan (Programmer). Our studies, which have encompassed vascular, traumatic, degenerative and developmental brain damage, suggest that VR has great potential in helping brain damage rehabilitation. We aim to develop the use of VR as a flexible, controllable, yet non-invasive method of directly manipulating brain activity in order to reduce the impact of brain damage. Research Interests We have already performed or collaborated in research investigating the following uses of VR in brain damage assessment and rehabilitation and in occupational training of people with learning disabilities: Cognitive assessment and retraining of attention, memory and spatial skills Assessment and training of motor skills Occupational training Training powered wheelchair use We are currently undertaking or collaborating in research using VR to: Assess the driving ability of patients following vascular or traumatic brain injury Train manual wheelchair use Assess the prospective memory ability (capacity to remember to perform future actions) of vascular brain injury patients Create standardised and validated psychometric assessments of rule breaking, strategy formation and prospective memory ability Assess egocentric and allocentric spatial memory using functional magnetic resonance imaging Develop an interactive multimedia package for training people with learning disabilities for employment in an office environment Profiling cognitive difficulties in patients with Dysexecutive syndrome

The Future of Assessments in South Africa
Locally developed tests appropriate for all cultures within SA Leadership tests appropriate for SA context Explore other intelligences – cosmic, spiritual, moral, emotional, transcendental contributions Stimulating use of assessments by smaller organisations and NGOs Industrial psychologists making significant contributions locally and internationally

Psychometric tests – most effective of predicting behaviour (Mittner, 1998) Tests 4 X more effective than screening interviews (Van der Walt, 1998) Employment Equity Act – definite move towards making assessment techniques more scientific (Eckstein, 1998) To be used as additional aid in decision making i.e. not in isolation Retain professional level for conducting psychometric assessment.

(Bartram & Coyne, 1998) in a recent world-wide survey found lack of psychologists involvement in testing and use of tests Found 60% of test users were not specifically trained in use of psychological tests. Therefore recommend training and increasing public understanding. Gregoire, (1999), recommend academic education, continued education, and more publications on tests and assessment methods.

Cook (1997), trend towards assessment centres and competency based assessment in changing socio-political and economic context. [some psychological factors that competence measures cannot provide, i.e. learning and potential] Consideration should be given to development of new tests with urgent special emphasis on culture-fairness. Validation of tests in different organisations that they are used in. Cook (1997),

Conclusion: History In the past the same tests were used for everyone.
Psychologists used international tests. Testing was for a long time viewed as discriminatory and unjust. Practitioners are becoming aware of the advantages of fair assessment. The implementation of cross-culturally fair tests. Jeannette van Dyk I would like to summarise the day and run through some main points as discussed today: History In the past the same tests were used for everyone despite culture and language differences. Psychologists used international tests as opposed to tests developed for South Africa’s culture. Testing was for a long time viewed as discriminatory and unjust, this perception is slowly changing. Practitioners are becoming more aware of the advantages of fair assessment. Improvements in test development have lead to the implementation of cross-culturally fair tests.

Conclusion: Assessment Areas
COGNITIVE PERSONALITY BEHAVIOURAL INTEREST Assessment Areas Earlier today we gave an overview of tests currently in use in South Africa: cognitive, personality, behavioural and interest assessments areas. Psychological tests must not be used in isolation but it should be part of an extensive assessment battery that measure different areas.

Conclusion: Legislation
Changes in legislation. The EEA has been established in 1998 to ensure more equitable and fair Psychological testing. Tests need to be scientifically valid and reliable; tests should be able to be applied fairly to all employees; and tests should not be biased against any employee or group. Legislation There have been changes in legislation and the workforce structure in South Africa after our first democratic election in 1994. As mentioned earlier today the EEA has been established in 1998 to ensure more equitable and fair Psychological testing. The EEA states that tests need to be scientifically valid and reliable; tests should be able to be applied fairly to all employees; and tests should not be biased against any employee or group.

Conclusion: Ethics Psychological tests can act as a disabling factor.
Practitioners therefore have to be more cautious and informed and should take responsibility to ensure that a test can be fairly applied. Practitioners should act in an ethical way before, during and after psychological tests. Ethics Psychological tests can act as a disabling factor if the test is inappropriately used in isolation without verifying the results against other measures. Practitioners therefore have to be more cautious and informed about the tests they apply and should take responsibility to ensure that a test can be fairly applied. Practitioners should act in an ethical way before, during and after psychological tests. Next, competency based assessment....

Conclusion: Competency Based Assessment
Focus on the essential behaviours required to perform a specific job. Focussing on individual differences in terms of work-related constructs relevant to job performance. This contributes to keep tests fair and relevant as required by the law. Competency based assessments The Focus is on the essential behaviours required to perform a specific job, focussing on individual differences in terms of work-related constructs relevant to job performance. contributes to keep tests fair and relevant as required by the law.

Conclusion: Computerised Assessment
In our modern society there is new technology on the market every day and everything is being computerised in one way or another as well as psychological testing. There is a lot of advantages to computerized testing we just need to make sure that it remains unbiased, ethical, valid and reliable for our rainbow nation. Computerised assessments Furthermore, there is a move towards computerised testing. In our modern society there is new technology on the market every day and everything is being computerised in one way or another as well as psychological testing. As mentioned earlier today, there is a lot of advantages to computerised testing we just need to make sure that it remains unbiased, ethical, valid and reliable for our rainbow nation. For example extra measures must be applied to keep test scores secure and confidential. My final comments concern some challenges that we currently experience in psychology test use and development.

Conclusion: Challenges
According to research the most frequently cited hindrance to the administration of psychological tests is language. When administering individual intelligence tests, psychologists often argue that it is justifiable to administer the measure in English, irrespective of whether English is the first or second language of the test-taker. Continue... Challanges regarding Test use & development My final comments concern some challenges that we currently experience in psychology test use and development. According to research the most frequently cited hindrance to the administration of psychological tests is language. In South Africa, 11 official languages are recognised. The dominant language in business and industry is English. Consequently, when administering individual intelligence tests, psychologists often argue that it is justifiable to administer the measure in English, irrespective of whether English is the first or second language of the test-taker; as it is important that test-takers can demonstrate their ability to perform test tasks in the language that will be used in the workplace.

It is important to determine whether the performance on the test reflects the testee’s actual ability and not the testee’s competence in the test language. Translation of a test is not a quick-fix. Changing some wording in an item/question can draw a question mark over the construct, score and predictive comparability. Continue... Therefore it is important to determine whether the performance on the test reflects the testee’s actual ability and not the testee’s competence in the test language. In addition, translation of a test is not a quick-fix for this problem. Even changing some wording in an item/question can draw a question mark over the construct, score and predictive comparability of the test.

Other factors that need to be taken into account: cultural and environmental factors, age and gender. Test anxiety has an effect on test resuls. Anxiety can hamper a person’s test performance. Employees may also start to view tests as unfair and invalid. Explain everything in simple terms. Other factors that need to be taken into account that can influence test performance are cultural and environmental factors, age and gender. Therefore there is a need in South Africa for test development with standardized test norms, with questions specifically designed for our multicultural country. Finally, I want close off with test anxiety. According to research test anxiety has a an effect on test results and practitioners should try to provide a relaxing and safe atmosphere. Anxiety can hamper a person’s test performance and this may result in unaccurate test scores. Furthermore, Employees may also start to view tests as unfair and invalid. In addition test practitioners should explain everything in simple terms and be available to answer any questions to help reduce tension and anxiety that the testee may experience.(Paterson & Uys, 2005) I will leave you now with a quote of Nelson Mandela and a quote of Theodore Roosevelt.

“If you talk to a man in a language he understands, that goes to his head. If you talk to him in his language, that goes to his heart.” Nelson Mandela “ The most important single ingredient in the formula of success is knowing how to get along with people” Theodore Roosevelt.

Crossword Competition

Psychological Assessment Seminar

Similar presentations

Presentation on theme: "Psychological Assessment Seminar"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Psychological Assessment Seminar

Similar presentations

Presentation on theme: "Psychological Assessment Seminar"— Presentation transcript:

Similar presentations

About project

Feedback