1. Salah Taamneh Advisor: Prof. Ioannis Pavlidis

2. 2 Outline  Introduction  Related Work  Ontology Building-File Structure  Design  System Architecture  Future Work

3. 3 Introduction Ontology Building DesignSystem Architecture Outline Future Work

4. 4 Affective Science  Affective science is the study of emotions and of affective phenomena such as moods, affects and bodily feelings. It combines the perspectives of many disciplines, such as neuroscience, psychology and philosophy [1].  Affective science involves research on emotion recognition, emotionally driven behavior, emotion regulation, decision making, and the underlying physiology [1] R. Davidson, K. Scherer, and H. Goldsmith. Handbook of affective sciences. Oxford University Press, 2003.  Introduction  Ontology Building  Design  System Architecture  Future Work

5. 5 Why Affective Data?  There is a growing interest in collecting affective datasets[1][2][3]:  Many phenomena, ranging from individual cognitive processing to social and collective behavior, cannot be understood without taking into account affective determinants  The advent of inexpensive, yet efficient, devices capable of capturing human affect  The complexity and size of these data sets renders them unique intellectual products, for which reproducibility of test results and information sharing acquire paramount importance [1] G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schröder. "The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent." IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 5-17, 2012. [2] F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne. “Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions.” In 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition - FG 2013, Shanghai, China, April 2013. [3] M. Soleymani, J. Lichtenauer, T. Pun, and M Pantic. "A multimodal database for affect recognition and implicit tagging." IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 42-55, 2012  Introduction  Ontology Building  Design  System Architecture  Future Work

6. 6 Multimodal Affective Studies  A typical multimodal affective dataset includes:  Sensor data streams  Psychometrics  Performance  Observational data (e.g., videos)  Biographic information  Facial expressions  Introduction  Ontology Building  Design  System Architecture  Future Work

7. 7 Exploratory Data Analysis  Introduction  Ontology Building  Design  System Architecture  Future Work

8. 8 The Big Picture Session 1 Subject 1 Session 2 Session 3 Subject 2 Subject 3  Introduction  Ontology Building  Design  System Architecture  Future Work

9. 9 Prior Work- CPL  Avinash Wesley in his PhD dissertation entitled “Eustressed or distressed? combining physiology with observation in user studies” developed a stress model that aims to quantify arousal emotional states and disambiguate them into positive and negative instances via the combination of a physiological and an observational information channel [1]  This work was done manually!  SubjectBook aims to automate this process, and make it much easier for researchers to analyze and explore their multimodal data [1] A. Wesley, P. Lindner, and I. Pavlidis. "Eustressed or distressed? combining physiology with observation in user studies." In CHI'12 Extended Abstracts on Human Factors in Computing Systems, pp. 327-330. ACM, 2012.  Introduction  Ontology Building  Design  System Architecture  Future Work

10. 10 Prior Work– Literature ArticleAuthorToolIssues An interactive web-based tool for multiscale physiological data visualization Oengeret et al., CinC 2004 Web-based Tool - No support for other types of data  Introduction  Ontology Building  Design  System Architecture  Future Work

11. 11 Prior Work– Literature ArticleAuthorToolIssues Elan: A professional framework for multimodality research Wittenburg et al., LREC 2006 ELAN-No record keeping -No multiple sessions  Introduction  Ontology Building  Design  System Architecture  Future Work

12. 12 Prior Work– Literature ArticleAuthorToolIssues Chronoviz: A system for supporting navigation of time-coded data Fouse et al., ACM CHI’11 ChronoViz-No record keeping -No multiple sessions  Introduction  Ontology Building  Design  System Architecture  Future Work

13. 13 Prior Work– Literature ArticleAuthorToolIssues Beda: Visual analytics for behavioral and physiological data. Kim et al., VAHC 2013 Beda- No support for covariates  Introduction  Ontology Building  Design  System Architecture  Future Work

14. 14 Prior Work- Discussion  Direct visualizations are typically provided with excessive details that strongly limit our understanding of the complete phenomena  No abstraction offered at the subject and study levels  Sharing among collaborators is not supported Session 1 Subject 1 Session 2 Session 3 Subject 2 Subject 3  Introduction  Ontology Building  Design  System Architecture  Future Work Study

15. 15 Prior Work ArticleAuthorToolMinor issues Major issues An interactive web-based tool for multiscale physiological data visualization Oenger et al. 2004 Web-based Tool not designed for analysis or visualization of multiple types of data. 1. No abstraction offered at the subject and study levels. 2. Sharing among collaborato rs is not supported Elan: a professional framework for multimodality research. Wittenburg et al. 2006 ELAN No support for record keeping and analysis of multiple sessions in longitudinal studies. Chronoviz: a system for supporting navigation of time-coded data. Fouse et al. 2011 ChronoViz No support for record keeping and analysis of multiple sessions in longitudinal studies. Beda: Visual analytics for behavioral and physiological data. Kim et al. 2013 BEDA No support for biographic and psychometric information

16. 16 What is SubjectBook?  A web-based visualization tool aiming to facilitate analyzing and exploring multimodal affective dataset via:  Providing higher levels of abstraction that focus users’ attention on important trends, patterns and characteristics:  Select the right follow-up analysis  Perform quality assurance  Enabling data sharing, which promotes collaboration and helps research community  Introduction  Ontology Building  Design  System Architecture  Future Work

17. 17 Psychometrics STAI PANAS Observational data Biographic data Physiological sensor streams Performance data What is SubjectBook?  Introduction  Ontology Building  Design  System Architecture  Future Work SubjectBook

18. 18 Multilayer Visualization SubjectBook’s Layering The units upon which the hypothesis is tested The hypothesis The variables that help compute the hypothesis for these units Scientific Process’s Layering Study Portrait Subject Portrait Session Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

19. 19 Multilayer Visualization The hypothesis Scientific Process’s Layering Treatment 3 Subjects pool - Treatment 1 -Treatment 2 -Treatment 3 - Treatment 1 -Treatment 2 -Treatment 3 Cross-over Treatment 1 Treatment 2 Parallel Group SubjectBook’s Layering Study Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

20. 20 Multilayer Visualization The hypothesis Scientific Process’s Layering SubjectBook’s Layering Study Portrait The units upon which the hypothesis is tested The variables that help test the hypothesis for these units  Introduction  Ontology Building  Design  System Architecture  Future Work

21. 21 Case Study: Driving Behaviors  A central goal of this study was to understand how driver-based and vehicle- based data might be used in tandem to detect high risk driving scenarios  In a sample of 59 drivers, balanced in terms of age and gender, the effects of cognitive, emotional, sensorimotor, and mixed stressors on driver affect and performance with respect to baseline was studied in a simulation experiment  Each subject was asked to perform the following simulator drives:  Drive with No Stressor (LDN)  Loaded Drive with Cognitive Stressor (LDC)  Loaded Drive with Emotional Stressor (LDE)  Loaded Drive with Sensorimotor Stressor (LDM)  Failure Drive-Normal (FDN)  Failure Drive-Loaded (FDL)  Introduction  Ontology Building  Design  System Architecture  Future Work Cross-over Parallel group Baseline Treatments

22. 22 Introduction Ontology Building DesignSystem Architecture Outline Future Work

23. 23 Ontology Building EXPERIMENTAL DESIGN DATA SOURCE EXPLANATORY VARIABLES RESPONSE VARIABLES COVARIATES  Introduction  Ontology Building  Design  System Architecture  Future Work

24. 24 Ontology Building EXPERIMENTAL DESIGN DATA SOURCE EXPLANATORY VARIABLES RESPONSE VARIABLES COVARIATES  Introduction  Ontology Building  Design  System Architecture  Future Work

25. 25 Ontology Building EXPERIMENTAL DESIGN DATA SOURCE EXPLANATORY VARIABLES RESPONSE VARIABLES COVARIATES  Introduction  Ontology Building  Design  System Architecture  Future Work

26. 26 Ontology Building EXPERIMENTAL DESIGN DATA SOURCE EXPLANATORY VARIABLES RESPONSE VARIABLES COVARIATES  Introduction  Ontology Building  Design  System Architecture  Future Work

27. 27 Ontology Building EXPERIMENTAL DESIGN DATA SOURCE EXPLANATORY VARIABLES RESPONSE VARIABLES COVARIATES  Introduction  Ontology Building  Design  System Architecture  Future Work

28. 28 File Structure SubjectsSessions Variables Covariates Explanatory Var. Response Var. Stimuli Other variables Observational data  Introduction  Ontology Building  Design  System Architecture  Future Work

29. 29 Data Abstraction Variable Types:  Videos  Facial video  Signals  Heart rate  Rank  Questionnaire score(s)  Categorical  Gender  Introduction  Ontology Building  Design  System Architecture  Future Work

30. 30 Introduction Ontology Building DesignSystem Architecture Outline Future Work

31. 31 Multilayer Visualization Low level High level  Introduction  Ontology Building  Design  System Architecture  Future Work

32. 32 Session Portrait  Website for the study, presenting the covariates at the top, followed by the time- registered explanatory and response variables Covariates Explanatory variable Response variable Data streams are synchronized Color-coded annotation  Introduction  Ontology Building  Design  System Architecture  Future Work Explore sessions

33. 33  Synchronization between data streams and observational data in order to support cause effect reasoning Session Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

34. 34  SubjectBook helps users easily compare between Baseline and Interventions Session Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

35. 35 Multilayer Visualization Low level High level  Introduction  Ontology Building  Design  System Architecture  Future Work

36. 36 Multilayer Visualization Low level High level  Introduction  Ontology Building  Design  System Architecture  Future Work

37. 37 Subject Portrait  Summarizes context information along with the explanatory and response measurements in a construct reminiscent of an ID card  Second level of abstraction, enabling the investigator to appreciate phenomena at the subject level  Introduction  Ontology Building  Design  System Architecture  Future Work

38. 38  The key explanatory variable in each session is represented by a colored bar, where the proportion of red color reflects the subject's level of arousal with respect to baseline Mean LDN Subject Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

39. 39  Circles with different shapes are used to encode the key response variable in each session  We compute the percentage of the response signal that is higher than the mean of the equivalent signal in Baseline  The following percentiles are used to split them into four categories : [0%, 25%), [25%,50%), [50%, 75%), [75%, 100%]  Feedback received from subjects before and/or after each session is represented using color-coded semicircles that appear next to bars  All scores from all sessions are collected, and the same percentiles are used to split them into four categories Low Fair Medium High Subject Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

40. 40 Multilayer Visualization Low level High level

41. 41 Multilayer Visualization Low level High level  Introduction  Ontology Building  Design  System Architecture  Future Work

42. 42 Study Portrait  Grid visualization of the study’s significant outcomes Phases Sessions  Introduction  Ontology Building  Design  System Architecture  Future Work

43. 43  In a great number of experiments we have  The explanatory variable  The response variable  We run experiments to see if the explanatory or response variables are any different from the baseline. BL-RESP. Exper. 1 with intervention element P1 P2 P3 P4 P5 Expl. Resp. Exper. 2 with intervention element Expl. Resp. BL-EXPL. Study Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

44. 44 Study Portrait  Introduction  Ontology Building  Design  System Architecture  Future Work

45. 45 Study Abstraction What are the experimental sessions? What are the stimuli? What are the explanatory variables? What are the response variables? What are the covariates?

46. 46 Introduction Ontology Building DesignSystem Architecture Outline Future Work

47. 47 SubjectBook Architecture  Front End  Browsers (HTML, Java Script, D3.js, and JQuery)  Back End  HTTP Server (Play Framework, Java, Scala, and Akka)  Database (i.e., MySQL)  Cloud Storage (at present we support Google Drive)  Statistical tool (R)  Introduction  Ontology Building  Design  System Architecture  Future Work

48. 48 System Architecture  Introduction  Ontology Building  Design  System Architecture  Future Work

49. 49 Introduction Ontology Building DesignSystem Architecture Outline Future Work

50. 50 Future Work  Real-time streaming (Progress: 50% )  Visualize the data as it is being acquired  2 months  Connection the tool with R, allowing user to run statistical tests (Progress: 70 %)  1month  Introduction  Ontology Building  Design  System Architecture  Future Work

51. 51 PhD Progress Joined CPL PhD showcase award April 2014 Poster in SAS 15  A. Alkhder, M. Taamneh, and S. Taamneh. "Severity Prediction of Traffic Accident Using Artificial Neural Network." Journal of Forecasting - to appear in 2016 [IF: 0.930].  M. Taamneh, A. Alkhder, and S. Taamneh. "Data Mining Techniques for Traffic Accident Modeling and Prediction in the United Arab Emirates." Journal of Transportation Safety & Security - to appear in 2016 [IF: 0.45]  S. Taamneh, M. Dcosta, K. Kwon, and I. Pavlidis. “SubjectBook: Hypothesis-Driven Ubiquitous Visualization for Affective Studies”. CHI'16 Extended Abstracts on Human Factors in Computing Systems. ACM, 2016 [acceptance rate: 42%].  S. Taamneh, M. Dcosta, K. Kwon, and I. Pavlidis. “SubjectBook: Web-based Visualization Of Multimodal Affective Datasets Residing On The Cloud”, The Society for Affective Science Conference, SAS 2016, Chicago, IL, USA. - S.  S. Taamneh, D. Shastri, D. Currie, M. Dcosta, and I. Pavlidis; “What Sympathetic Responses Can Tell about Children’s’ Performance in Reading?” The Society for Affective Science Conferences, 9-14 April 2015, San Francisco, CA.  Y. Lu, S. Taamneh, and J. Ashley, “Implementation of Genetic K-means Algorithm on Iterative MapReduce Framework for Clustering Gene Expression Data”, the proceedings of 25th International Conference on Computer Applications in Industry and Engineering (CAINE- 2012), New Orleans, USA, November 14-16, 2012 April 2015 Best Poster Presentation: Awarded for the best poster presentation at the UHCS PhD Showcase. Poster in SAS 2016 Jan 2016 Publication in CHI Summer 2013 Feb 2016 Complete d course- work Spring 2013 Fall 2012 Pub. In CAINE

52. 52 Publications  S. Taamneh, M. Dcosta, K. Kwon, and I. Pavlidis. “SubjectBook: Web-based Visualization of Multimodal Affective Datasets”. CHI'16 Extended Abstracts on Human Factors in Computing Systems. ACM, 2016.  S. Taamneh, M. Dcosta, K. Kwon and I. Pavlidis “SubjectBook: Web-based Visualization Of Multimodal Affective Datasets Residing On The Cloud”, The Society for Affective Science Conference, SAS 2016, Chicago, IL, USA. - S.  S. Taamneh, D. Shastri, D. Currie, M. Dcosta, and I. Pavlidis; “What Sympathetic Responses Can Tell about Children’s’ Performance in Reading?” The Society for Affective Science Conferences, 9-14 April 2015, San Francisco, CA.

53. 53 In the pipeline  I. Pavlidis, M. Dcosta, S. Taamneh, M. Manser, T. Ferris, R. Wunderlich, P. Tsiamyrtzis. "Dissecting Driver Behaviors Under Cognitive, Emotional, Sensorimotor, and Mixed Stressors.“ Scientific Reports [IF ~ 6]-Under minor revision

54. 54 Acknowledgements Dr. Ioannis Pavlidis Karl Kwon Ashik Khatri Dinesh Majeti Muhsin Ugur Dr. Zhigang Deng Dr. David Francis Dr. Malcolm Dcosta This research was supported by a grant from the Texas A&M Transportation Institute Dr. Guoning Chen