1. Affective Computing and Intelligent Interaction Ma. Mercedes T. Rodrigo Ateneo Laboratory for the Learning Sciences Department of Information Systems and Computer Science Ateneo de Manila Univeristy

2. Page 2 Affective computing Computing that relates to, arises from or deliberately influences emotion - Picard, 1997, Affective Computing

3. Page 3 Affective computing  Emotion recognition  Emotion expression  Intelligent response to emotion

4. Page 4 Significance: Towards more humane interfaces How can we enable computers to better serve people’s needs--adapting to you, vs. treating you like some fictionalized ideal user, and recognizing that humans are powerfully influenced by emotion, even when they are not showing any emotion? - Picard, 2003

5. Page 5 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

6. Page 6 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

7. Page 7 Methods: Aplusix

8. Page 8 Methods: Ecolab / MEcolab

9. Page 9 Methods: Cognitive Tutor

10. Page 10 Methods: BlueJ

11. Page 11 Methods: The Incredible Machine

12. Page 12 Methods: Math Blaster

13. Page 13 SQL/EER Tutors

14. Page 14 SimStudent

15. Page 15 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

16. Page 16 Biometrics instruments

17. Page 17 Biometrics instruments

18. Page 18 Biometrics instruments

19. Page 19 Sample Log: Brainfingers [Header V2035] 7/31/2009 7:49:05 PM UserFile = C:\Nia Data\\__20090731194905.usr [Data] Sample,Event,GlanceMagJs,GlanceDirJs,A1Js,A2Js,A3Js,B1Js,B2Js,B3Js,MuscleJs 1,0,0.1,0.065,0.0195,0.20775,-1.165867E-07,0.5815,0.5311,0.6048,0.6782,0.7665,0.7074,0 1,0,-0.029,-0.122,0.021,0.2056,-1.165867E-07,0.5774,0.5251,0.5892,0.6723,0.7598,0.7125,0 1,0,0.015,-0.167,0.0205,0.203625,-1.165867E-07,0.5743,0.5187,0.5737,0.6683,0.7534,0.7157,0 1,0,-0.0595,-0.1555,0.0205,0.20165,-1.165867E-07,0.573,0.5118,0.5596,0.6656,0.7468,0.7159,0 1,0,-0.285,-0.163,0.0205,0.199625,-1.165867E-07,0.5733,0.5043,0.5477,0.6622,0.7398,0.7168,0 1,0,-0.3665,-0.206,0.022,0.197625,-1.165867E-07,0.5745,0.4966,0.5371,0.6567,0.7321,0.7221,0 1,0,-0.125,-0.158,0.0225,0.195675,-1.165867E-07,0.5772,0.4885,0.5268,0.6483,0.7242,0.7262,0

20. Page 20 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

21. Page 21 Sample Log: Aplusix %;ACTIONS;#Date=1/16/2007#Heure=14:57:59;#TypeProbleme=TpbDevelopper 0;0.0;structure;();0;();();();();();(); 1;0.0;enonce;();0;-7x(7x{@^[2]}-7x+4);();(devant);rien;;N1; 2;5.1;placerCurseur;();0;-7x(7x{@^[2]}-7x+4);();(0 2 derriere);rien;;N1; 3;0.8;dupliquer;();1;-7x(7x{@^[2]}-7x+4);();(0 2 derriere);rien;V1;N1; 4;1.9;selection;();1;-7x(7x{@^[2]}-7x+4);();rien;();V1;N1; 5;5.3;-;();1;7x(7x{@^[2]}-7x+4);();rien;();V0;N0; 6;1.5;BackSpace;();1;?;();(dedans);rien;V-;N-; 7;2.0;-;();1;-?;();(0 dedans);rien;V-;N-; 8;3.7;4;();1;-4;();(0 0 derriere);rien;V0;S0; 9;0.2;9;();1;-49;();(0 1 derriere);rien;V0;S0; 10;2.7;x;();1;-49x;();(0 1 derriere);rien;V0;S0; 11;1.3;{@^[?]};();1;-49x{@^[?]};();(0 1 1 dedans);rien;V-;N-;

22. Page 22 Sample Log: Ecolab New Activity Toolbar ButtonClick0 6 Activity 16 Activity Chosen: Food 46 Suggested Help 06 Suggested Challenge 16 Challenge Accepted18 ViewWeb change13 ViewWeb change14 ActionShow19

23. Page 23 Sample Log: Scatterplot Tutor *000:03:781 READY. *000:59:503 APPLY-ACTION WINDOW; ALGEBRA-2-TRANSLATOR::VARIABLE-TYPE-MODEL, CONTEXT; SPLOT-DB-C-0-10-0-10, SELECTIONS; (|var-1val-1|), ACTION; SUBSTITUTE-TEXT-INTO-BLANK, INPUT; ("Numerical"),. *000:59:503 UPDATE-P-KNOW META; META-VALUING-NUM-FEATURES, PRODUCTION; (CHOOSE-VAR-TYPE-NUM MIDSCH-VARIABLE-TYPING), SUCCESS?; T, P-KNOW; 0.33333333333333326,..

24. Page 24 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

25. Page 25 consolidation analysis Methods

26. Page 26 Methods

27. Page 27 Methods

28. Page 28 analysis Methods

29. Page 29 analysis consolidation analysis Methods

30. Affective states

31. Page 31 Boredom

32. Page 32 Confusion

33. Page 33 Delight

34. Page 34 Frustration

35. Page 35 Engaged concentration

36. Page 36 Others  Neutrality  Surprise

37. Behaviors

38. Page 38 On-task, solitary

39. Page 39 On-task, giving and receiving answers

40. Page 40 Other on-task conversation (probably)

41. Page 41 Off-task solitary

42. Page 42 Others  Off-task, conversation  Inactive  Gaming the system

43. Page 43 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

44. Page 44 Talk to me %;ACTIONS;#Date=1/16/2007#Heure=14:57:59;#TypeProbleme=TpbDevelopper 0;0.0;structure;();0;();();();();();(); 1;0.0;enonce;();0;-7x(7x{@^[2]}-7x+4);();(devant);rien;;N1; 2;5.1;placerCurseur;();0;-7x(7x{@^[2]}-7x+4);();(0 2 derriere);rien;;N1; 3;0.8;dupliquer;();1;-7x(7x{@^[2]}-7x+4);();(0 2 derriere);rien;V1;N1; 4;1.9;selection;();1;-7x(7x{@^[2]}-7x+4);();rien;();V1;N1; 5;5.3;-;();1;7x(7x{@^[2]}-7x+4);();rien;();V0;N0; 6;1.5;BackSpace;();1;?;();(dedans);rien;V-;N-; 7;2.0;-;();1;-?;();(0 dedans);rien;V-;N-; 8;3.7;4;();1;-4;();(0 0 derriere);rien;V0;S0; 9;0.2;9;();1;-49;();(0 1 derriere);rien;V0;S0; 10;2.7;x;();1;-49x;();(0 1 derriere);rien;V0;S0; 11;1.3;{@^[?]};();1;-49x{@^[?]};();(0 1 1 dedans);rien;V-;N-;

45. The problem: 5(-9x+7)+3(4x-3)

46. Step #Text typed by student 21-45x 22-45x=? 23-45x 24-45x+? 25-45x+3 26-45x+35 27-45x+35+? 28-45x+35+1 29-45x+35+12 30-45x+35+12x 31-45x+35+12x-? 32-45x+35+12x-9

47. The problem: 8x 2 -2x+6-(-5x 2 +8x+3)

48. Page 48 Step #Text typed by student 563c 564ch 565chr 566chri 567chris 568christ 569christi 570christin 571christine 572christine+? 573christine+c 574christine+cy 575christine+cyr

49. Step #Text typed by student 576christine+cyri 577christine+cyril 578christine+cyril=? 579christine+cyril=a 580christine+cyril=ab 581christine+cyril=abi 582christine+cyril=abig 583christine+cyril=abiga 584christine+cyril=abigai 585christine+cyril=abigail

50. Page 50 I’m not a math genius but I’m pretty sure that 8x 2 -2x+6-(-5x 2 +8x+3) != christine+cyril=abigail

51. Page 51 Analysis methods  Clean the data  Define the different features  Distill new features  Define desired range of values  Select an appropriate statistical test or data mining algorithm  Validate the findings

52. Page 52 Analysis techniques  Statistical methods  Data mining techniques

53. Page 53 Persistence of affective states  Regardless of software, boredom tends to persist  Affect and behavior detection

54. Page 54 Affect and behavior  Students who attempt the most difficult problems experience flow the most  Students who try the lowest levels experience more boredom and confusion.  Students who take the longest time in solving the problems experience confusion the most  Students who take the shortest time experience confusion the least.  Students who use the most number of steps to solve a problem experience confusion and boredom the most.  Students who take the least number of steps experience more flow.

55. Page 55 Carelessness  Creation of models that generalize across different datasets and countries  Students who are bored and confused are less likely to be careless  Students who are engaged are more likely to be careless

56. Page 56 Novice programmer behaviors  Students who are persistently confused do worse on the midterm  Students who are able to resolve their confusion do better  Students with low error quotients do better on the midterm

57. Page 57 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

58. Page 58 Example

59. Page 59 New version

60. Page 60 Affect-sensitive game

61. Page 61 BlueJ Browser

62. Page 62 BlueJ Browser

63. Page 63 EDM Workbench

64. Page 64 Closing the loop  We continue to create new models and integrate the models with software  We still have a long way to go

65. My thanks  Ryan Baker  Jessica Sugay  Thor Collin Andallaza  Rina Joy Jimenez  Jason King Li  Tricia Monsod  Diane Lee  Minmin Lagud  Sweet San Pedro  Thomas Dy  And all the “cast of thousands” who constitute the ALLS

66. Thank you! Questions? mrodrigo@ateneo.edu http://penoy.admu.edu.ph/~alls