Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Statistical Mechanical Analysis of Online Learning: Can Student be more Clever than Teacher ? Seiji MIYOSHI Kobe City College of Technology

Similar presentations


Presentation on theme: "A Statistical Mechanical Analysis of Online Learning: Can Student be more Clever than Teacher ? Seiji MIYOSHI Kobe City College of Technology"— Presentation transcript:

1 A Statistical Mechanical Analysis of Online Learning: Can Student be more Clever than Teacher ? Seiji MIYOSHI Kobe City College of Technology

2 2 Background (1) Batch Learning –Examples are used repeatedly –Correct answers for all examples –Long time –Large memory Online Learning –Examples used once are discarded –Cannot give correct answers for all examples –Large memory isn't necessary –Time variant teacher

3 3 Background (2) TeacherStudent

4 4 Simple Perceptron Output Inputs Connection weights +1

5 5 Background (2) TeacherStudent Learnable Case

6 6 Background (3) Teacher Student Unlearnable Case ( Inoue & Nishimori, Phys. Rev. E, 1997) ( Inoue, Nishimori & Kabashima, TANC-97, cond-mat/ , 1997)

7 7 Background (4) Hebbian Learning Perceptron Learning

8 8 Model (1) Moving Teacher Student True Teacher A

9 9 Model (2) Length of Student Length of Moving Teacher A B J

10 10 Model (3) A B J

11 11 Output Inputs Connection weights Simple Perceptron Linear Perceptron

12 12 Model (3) Linear Perceptrons with Noises A B J

13 13 f g Model (4) Squared Errors Gradient Method A B J

14 14 ErrorGaussian Generalization Error A B J

15 15 Differential equations for order parameters

16 16 f g Model (4) Squared Errors Gradient Method A B J

17 17 B m+1 = B m + g m x m + Nr B m+1 = Nr B m + g m y m Ndt Nr B m+2 = Nr B m+1 + g m+1 y m+1 Nr B m+Ndt = Nr B m+Ndt-1 + g m+Ndt-1 y m+Ndt-1 Nr B m+Ndt = Nr B m + Ndt N(r B +dr B ) = Nr B + Ndt dr B / dt =

18 18 Differential equations for order parameters

19 19 Sample Averages

20 20 Differential equations for order parameters

21 21 Analytical Solutions of Order Parameters

22 22 Differential equations for order parameters

23 23 ErrorGaussian Generalization Error A B J

24 24 Dynamical Behaviors of Generalization Errors η J = 1.2 η J = 0.3

25 25 Dynamical Behaviors of R and l η J = 1.2η J = 0.3

26 26 Analytical Solutions of Order Parameters

27 27 Steady State

28 28 ηJηJ 20

29 29 Conclusions Generalization errors of a model composed of a true teacher, a moving teacher, and a student that are all linear perceptrons with noises have been obtained analytically using statistical mechanics. Generalization errors of a student can be smaller than that of a moving teacher, even if the student only uses examples from the moving teacher.


Download ppt "A Statistical Mechanical Analysis of Online Learning: Can Student be more Clever than Teacher ? Seiji MIYOSHI Kobe City College of Technology"

Similar presentations


Ads by Google