1. Self-Assemblying Hypernetworks for Cognitive Learning of Linguistic Memory Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6-8, 2008, Sheraton Hotel, Cairo, Egypt Byoung-Tak Zhang and Chan-Hoon Park Biointelligence Laboratory School of Computer Science and Engineering Cognitive Science, Brain Science, and Bioinformatics Programs Seoul National University Seoul 151-744, Korea btzhang@bi.snu.ac.kr http://bi.snu.ac.kr/

2. Talk Outline A Language Game Learning the Linguistic Memory  The Hypernetwork Model of Language Sentence Recall Experiments Extension to Multimodal Memory Game (Language + Vision) Conclusion

3. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 3 The Language Game Platform

4. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 4 A Language Game ? still ? believe ? did this.  I still can't believe you did this. We ? ? a lot ? gifts.  We don't have a lot of gifts.

5. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 5 Text Corpus: TV Drama Series Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City 289,468 Sentences (Training Data) 700 Sentences with Blanks (Test Data) I don't know what happened. Take a look at this. … What ? ? ? here. ? have ? visit the ? room. …

6. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 6 Step 1: Learning a Linguistic Memory k = 2 k = 3 k = 4 … Hypernetwork Memory

7. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 7 Step 2: Recalling from the Memory Heismybestfriend bestfriendmybestismyHeis Heisa?friend HeisastrongboyStrongfriendlikesprettygirl Heis aastrong boyStrongfriend likes strongprettygirlStrongfriendastrongbestfriend Heisastrongfriend X7 X6 X5 X8 X1 X2 X3 X4 Recall Self-assembly Storage

8. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 8 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 4 sentences (with labels) x4x4 x 10 y=1x1x1 x4x4 x 12 y=1x1x1 x 10 x 12 y=1x4x4 x3x3 x9x9 y=0x2x2 x3x3 x 14 y=0x2x2 x9x9 x 14 y=0x3x3 x6x6 x8x8 y=1x3x3 x6x6 x 13 y=1x3x3 x8x8 x 13 y=1x6x6 1 2 3 1 2 3 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 4 x 11 x 15 y=0x8x8 4 Round 1 Round 2 Round 3

9. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 9 x1 x2 x3 x4 x5 x6 x7 x8x9 x10 x11 x12 x13 x14 x15 The Hypernetwork Memory [Zhang, DNA12-2006]

10. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 10 Molecular Self-Assembly of Hypernetworks xixi xjxj y X7 X6 X5 X8 X1 X2 X3 X4 Hypernetwork Representation Molecular Encoding DNA Computing

11. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 11 Experimental Setup The order (k) of an hyperedge  Range: 2~4  Fixed order for each experiment The method of creating hyperedges from training data  Sliding window method  Sequential sampling from the first word The number of blanks (question marks) in test data  Range: 1~4  Maximum: k - 1

12. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 12 Learning Behavior Analysis (1/3) The performance monotonically increases as the learning corpus grows. The low-order memory performs best for the one-missing-word problem.

13. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 13 Learning Behavior Analysis (2/3) The medium-order (k=3) memory performs best for the two-missing-words problem.

14. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 14 Learning Behavior Analysis (3/3) The high-order (k=4) memory performs best for the three-missing-words problem.

15. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 15 The Language Game: Results Why ? you ? come ? down ?  Why are you go come on down here ? think ? I ? met ? somewhere before  I think but I am met him somewhere before ? appreciate it if ? call her by ? ?  I appreciate it if you call her by the way I'm standing ? the ? ? ? cafeteria  I'm standing in the one of the cafeteria Would you ? to meet ? ? Tuesday ?  Would you nice to meet you in Tuesday and ? gonna ? upstairs ? ? a shower  I'm gonna go upstairs and take a shower ? have ? visit the ? room  I have to visit the ladies' room We ? ? a lot ? gifts  We don't have a lot of gifts ? ? don't need your ?  If I don't need your help ? ? ? decision  to make a decision ? still ? believe ? did this  I still can't believe you did this What ? ? ? here  What are you doing here ? you ? first ? of medical school  Are you go first day of medical school ? ? a dream about ? In ?  I had a dream about you in Copenhagen

16. Extension to Multimodal Memory Game

17. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Image Sound Text But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. Memorizing Images to Retrieve Texts Memorizing Texts to Retrieve Images Hint Text But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. Image Hint Image Generation Game Text Generation Game Machine Learner How can it be done? Scene1Heisbestfriend Scene2Sheisstrongboy Scene3friendlikesprettygirl Scene1a1a2a3a4 Scene2b1b2b3b4 Scene3c1c2c3c4TextImage Text: N sequential samples Image: N random samples Heisa1a3 a4 bestfrienda1 a3a2 Multimodal HN Sheisb1b3b5 isstrongb2 b3 b1 Heisbestfriend Text Query Heisa1a3 a4 bestfrienda1 a3a4 isbestb2 b3 b1 Matching Heisbestfriend Text Query Heisa1a3 a4 bestfrienda1 a3a4 isbestb2 b3 b1 Matching Generating an Image Map a1=2 b1=1 b2=1 a3=2 b3=1 a4=2 voting Mapa1b2 a3 a4 Mapa1b2 a3 a4 Scene1a1a2a3a4 Scene2b1b2b3b4 Scene3c1c2c3c4Image Hamming Distance Scene1a1a2a3a4 Mapa1b2 a3 a4

18. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 18 Image-to-Text Crossmodal Recall Text Learning by Viewing Image - Where am I giving birth - You guys really don't know anything - So when you guys get in there - I know it's been really hard for you - … User Text Corpus Question: Answer: Where am I giving birth Where ? I giving ?

19. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 19 Text-to-Image Crossmodal Recall Text Learning by Viewing Image Corpus User Question: Answer: Image You've been there - Where am I giving birth - You guys really don't know anything - So when you guys get in there - I know it's been really hard for you - …

20. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 20 The Multimedia (Movie) Corpus Dataset: 2 dramas  Images and the corresponding scripts  Titles  Friends, Prison Break  Training data: 2,808 images and scripts  Image size: 80 x 60 = 4800 pixels  Vocabulary: 2,579 words Where am I giving birth I know it's been really hard for you So when you guys get in there

21. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 21 Experimental Setup The order (k) of memory units  Text: k = 2, 3, 4  Image: k = 10, …, 340 Constructing hyperedges from training data  Text: Sequential sampling from a random position  Image: Random sampling from 4,800 pixel positions The number of repetitive samples from an image-text pair  N = 150, …, 300

22. Answer Query I don't know what happened There's a kitty in my guitar case Maybe there's something I can do to make sure I get pregnant Maybe there's something there's something I … I get pregnant There's a a kitty in … in my guitar case I don't know don't know what know what happened Matching & Completion Image-to-Text Recall Examples © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

23. Query Matching & Completion I don't know what happened Take a look at this There's a kitty in my guitar case Maybe there's something I can do to make sure I get pregnant Answer Text-to-Image Recall Examples © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

24. 24 Image to Text (Recall Rate) Image to Text (Recall Rate) Note: In the tolerant recall, the generated sentence is evaluated correct if the number of mismatches is within the specified tolerance level (here two words).

25. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 25 Text to Image (Recall Rate) Text to Image (Recall Rate) Note: The retrieved image is evaluated correct if its hamming distance to the target image is the smallest.

26. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 26 Conclusion Hypernetworks are a random graph model employing higher-order edges and allowing for a more natural representation for learning higher-order interactions. We introduce a linguistic memory model based on a self-organizing hypernetwork inspired by mental chemistry. The hypernetwork stores the sentences in random fragments and recalls a sentence by self-assemblying them given a partial, query sentence. Applied to a sentence corpus of 290K sentences, we obtain a recall performance of 90-100%, depending on the difficulty of the task. Cognitive plausibility:  “Multiple representations of partially overlapping micromodules which are partially active simultaneously” [Fuster, 2003]  Neural microcircuits [Grillner et al, 2006]  Cognitive schema or cognitive code [Tse et al., 2007]

27. Data Acquisition and Experimentation Text: Ha-Young Jang Image: Min-Oh Heo Sun Kim Joo-Kyoung Kim Ho-Sik Seok Kwonil Kim Sang-Yoon Lee Supported by - National Research Lab Program of Min. of Sci. & Tech. (2002-2007) - Next Generation Tech. Program of Min. of Ind. & Comm. (2000-2010) - BK21-IT Program of Min. of Education (2006-2009) - SK Telecom (2007-2008) More Information at - http://bi.snu.ac.kr/  Research  MMG (to be open soon)http://bi.snu.ac.kr/ Acknowledgements

29. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 29 The Hypernetwork Model of Learning [Zhang, 2006]

30. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 30 Deriving the Learning Rule

31. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 31 Derivation of the Learning Rule

32. Molecular Self-Assembly

33. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 33 Encoding a Hypernetwork with DNA z 1 : z 2 : z 3 : z 4 : b) x1x1 x2x2 x3x3 x4x4 x5x5 y 0 1 where z 1 : (x 1 =0, x 2 =1, x 3 =0, y=1) z 2 : (x 1 =0, x 2 =0, x 3 =1, x 4 =0, x 5 =0, y=0) z 3 : (x 2 =1, x 4 =1, y=1) z 4 : (x 2 =1, x 3 =0, x 4 =1, y=0) a) AAAACCAATTGGAAGGCCATGCGG AAAACCAATTCCAAGGGGCCTTCCCCAACCATGCCC AATTGGCCTTGGATGCGG AATTGGAAGGCCCCTTGGATGCCC GG AAAA AATT AAGG CCTT CCAA ATGC CC Collection of (labeled) hyperedges Library of DNA molecules corresponding to (a)

34. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 34 DNA Molecular Computing Self-assembly Heat Cool Polymer Repeat Self-replication Molecular recognitionNanostructure