Measuring Human Intelligence with Artificial Intelligence Adaptive Item Generation Sangyoon Yi Susan E. Embretson

Introduction Adaptive Item Generation –Intelligence testing Generate optimally informative item for the examinee during the test –Optimally informative item Based on the previous pattern of the examinee’s response –Ex. Deep Blue (Chess Computer)

Introduction Adaptive Item Generation –Psychometric methods for adaptive testing Intelligence measurement Adaptive item selection leads to shorter and more reliable tests –A cognitive analysis of items Knowledge is required of how stimulus features in specific items impact the ability construct

Introduction Adaptive Item Generation {f_1, f_3, f5} Psychometric properties impact

Cognitive Design System Approach to Adaptive Item Generation –Theoretical Foundations for Cognitive Design Systems –Supporting Developments –Stages in Applying Cognitive Design Systems Supporting Data for Cognitive Design Systems –Initial Cognitive Model for Matrix Items –Algorithmic Item Generation and Reversed Cognitive Model –Item Generation by Artificial Intelligence –Empirical Tryout of Item Generation Related Approaches to Item Development Evolution of Approach : Advantage and Disadvantages Future

Cognitive Design System Approach to Adaptive Item Generation Matrix completion problems –Regard this item type as central to measuring intelligence

Cognitive Design System Approach to Adaptive Item Generation Cognitive processing model for the item type –It measures the construct For adaptive item generation –A conceptualization of construct validity –Psychometric models –A computer program

Cognitive Design System Approach to Adaptive Item Generation Theoretical Foundations for Cognitive Design Systems –Based on an information processing theory of the item type. Originated with cognitive component analysis of complex item types for measuring intelligence …

Cognitive Design System Approach to Adaptive Item Generation Theoretical Foundations for Cognitive Design Systems –Cognitive theory Specifies the impact of processes on performance, and the impact of stimulus features on processes Stimulus features ProcessesPerformance

Cognitive Design System Approach to Adaptive Item Generation Supporting Developments –Construct Validity and Cognitive Design Systems –Psychometric Models for Cognitive Design Systems –Computer programs for Adaptive Item Generation

Supporting Data for Cognitive Design Systems Initial Cognitive Model for Matrix Items –Advanced Progressive Matrices(Raven, et al. 1992) Algorithmic Item Generation and Reversed Cognitive Model

Supporting Data for Cognitive Design Systems Item Generation by Artificial Intelligence –Ex) ITEMGEN1 Randomly selects stimuli and their attributes to fulfill the structural specifications Empirical Tryout of Item Generation –Item generation has not been attempted with the full cognitive approach for the matrix completion items

Related Approaches to Item Development Traditional Approach –Item writing as an art –By human Item model Approach –Items are “variablized” –Item parameters are invariant over the cloned items –Ex) an existing mathematics word problem

Evolution of Approach : Advantages and Disadvantages Advantages –New items may be readily developed –Items may be developed to target difficulty levels and psychometric quality –New items may be placed in the item bank without empirical tryout –Construct validity is available at the item level –Tests may be redesigned to represent specifically targeted sources of item difficulty

Evolution of Approach : Advantages and Disadvantages Disadvantages –The approach requires substantial initial effort –The approach works best for item types that already have been developed

Future Item generation by artificial intelligence fulfills practical needs for new items The many correlates and relationships of intelligence measurements to other variables may be understood more clearly if the characteristic processing at different ability levels can be explicated