Biological and cognitive plausibility in connectionist networks for language modelling Maja Anđel Department for German Studies University of Zagreb
Connectionist networks AI networks for simulating cognitive processes – language implementing the “computer metaphor” architectural inspiration: human brain (network of artificial neurons)
Connectionist networks subsymbolic, not symbolic processing = no division in hardware/software – software “built in” to the hardware structure
Connectionist networks
Language modeling in connectionism Biology / neurological functioning Language processes (as cognitive processes) Computer implementation
Models in history (’80) Rumelhart & McClelland (1986): English past tense – modeling the U- shaped learning curve
Models in history (’80) McClelland & Kawamoto (1986): Thematic roles in sentences “How to represent sequences?” w i c k e l f e a t u r e s w i c i c k c k e k e l e l f l f e...
Models in history (’80) Elman (1990): Words of different length “How to represent sequences?” inputs hidden units outputs context units
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –biological realism –distributed representations –inhibitory competition –bidirectional propagation of activation (top-down and bottom-up) –error-driven task learning –Hebbian learning New algorithms that satisfy all the constraints!
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –biological realism
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –distributed representations
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –distributed representations
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –inhibitory competition
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –bidirectional propagation of activation (top-down and bottom-up) conceptual perceptual d o g
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –error-driven task learning desired outcome input signal error computation error backpropagation transfer function
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –error-driven task learning desired outcomeinput signalerror computation transfer function network state
Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –Hebbian learning "The general idea is an old one, that any two cells or systems of cells that are repeatedly active at the same time will tend to become 'associated', so that activity in one facilitates activity in the other." (Hebb 1949, p. 70) "When one cell repeatedly assists in firing another, the axon of the first cell develops synaptic knobs (or enlarges them if they already exist) in contact with the soma of the second cell." (Hebb 1949, p. 63)
Today – biological plausibility New algorithms – combination of new principles Complexity brings better results Old models sucessfully transposed –morphology acquisition –syntax processing –semantic categorization Computational neuroscience
Thank you!