1. Biological and cognitive plausibility in connectionist networks for language modelling Maja Anđel Department for German Studies University of Zagreb

2. Connectionist networks AI networks for simulating cognitive processes – language implementing the “computer metaphor” architectural inspiration: human brain (network of artificial neurons)

3. Connectionist networks subsymbolic, not symbolic processing = no division in hardware/software – software “built in” to the hardware structure

4. Connectionist networks

6. Language modeling in connectionism Biology / neurological functioning Language processes (as cognitive processes) Computer implementation

7. Models in history (’80) Rumelhart & McClelland (1986): English past tense – modeling the U- shaped learning curve

8. Models in history (’80) McClelland & Kawamoto (1986): Thematic roles in sentences “How to represent sequences?” w i c k e l f e a t u r e s w i c i c k c k e k e l e l f l f e...

9. Models in history (’80) Elman (1990): Words of different length “How to represent sequences?” inputs hidden units outputs context units

10. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –biological realism –distributed representations –inhibitory competition –bidirectional propagation of activation (top-down and bottom-up) –error-driven task learning –Hebbian learning New algorithms that satisfy all the constraints!

11. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –biological realism

12. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –distributed representations

13. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –distributed representations

14. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –inhibitory competition

15. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –bidirectional propagation of activation (top-down and bottom-up) conceptual perceptual d o g

16. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –error-driven task learning desired outcome input signal error computation error backpropagation transfer function

17. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –error-driven task learning desired outcomeinput signalerror computation transfer function network state

18. Today – biological plausibility O’Reilly (1998): Six principles of biological plausibility: –Hebbian learning "The general idea is an old one, that any two cells or systems of cells that are repeatedly active at the same time will tend to become 'associated', so that activity in one facilitates activity in the other." (Hebb 1949, p. 70) "When one cell repeatedly assists in firing another, the axon of the first cell develops synaptic knobs (or enlarges them if they already exist) in contact with the soma of the second cell." (Hebb 1949, p. 63)

19. Today – biological plausibility New algorithms – combination of new principles Complexity brings better results Old models sucessfully transposed –morphology acquisition –syntax processing –semantic categorization Computational neuroscience

20. Thank you!