1. Bite detection and differentiation using templates of wrist motion MS Defense Exam Soheila Eskandari Committee members: Dr. Adam Hoover (chair) Dr. John N. Gowdy Dr. Eric R. Muth December 5 th, 2013 Department of Electrical and Computer Engineering

2. Outline ◦ Motivation and Background ◦ Methods ◦ Results ◦ Conclusions 2

3. Motivation 3 One third of U.S. adults were overweight and another one third were obese in 2003-2004 (reported by NHANES) Cost associated with obesity was $117 billion in the US in 2000

4. Obesity treatments Weight maintenance goal is to achieve : EI=EE The problem is with the tools people use to measure EI

5. Mobile Health technologies Mobile monitoring of the human electrocardiogram (ECG) Heart rate, Breathing frequency, Blood pressure variations, Breathing amplitude. Detection of different sleep phases

6. Wrist motion tracking Dong et al. [7,8] developed a wrist-worn device to track wrist motion and measure the number of bites taken during a meal. Additional research showed that bites, automatically counted using this method, correlated with self-reported caloric intake at the meal level at 0.5. Amft [1] developed a wrist-worn device with the primary objective of detecting drinking activities, the container used, and the fluid level. Junker and Amft [1,2] presented a recognition system that used five inertial sensors located on the wrists, upper arms, and upper torso. Their research describes motion gestures based on the particular utensil used, establishing four gestures (cutlery, drink, spoon, hands). 6

7. Bite detection based on threshold method by wrist motion tracking T1 and T2 : The roll velocities T3 : Time interval between the first and second events of roll motion T4 : Time interval between the end of one bite and beginning of the next bite Tested on total of 276 subjects 22,383 bites True detection rate of 76% with a positive predictive value of 87% Adjusting the second timing threshold (T4): True detection rate of 82% and a positive predictive value of 82% Threshold algorithm: Let EVENT = 0 Loop Let V_t = measured roll vel. at time t if V_t > T1 and EVENT = 0 EVENT = 1 Let s=t if V_t T3 and EVENT = 1 Bite detected Let s=t EVENT = 2 if EVENT = 2 and T-s>T4 EVENT = 0 7

8. Template matching 8

9. Methods Data collection Bite templates Bite differentiation Bite detection 9

10. Data collection 10

11. Data collection tools: 11

12. Ground truth Total of 22,383 bites

13. Bite Templates Determine the overall pattern and variability pattern of wrist motion of a bite Created by : Using both the accelerometers and gyroscopes data Averaging the motion data across all the bites in the 22,383 total ground truth bites Over a six second window centered on the bite time Templates of food and drink bites Four different types of food bites:  bites taken with a fork,  bites taken with a spoon,  food bites eaten using one hand  food bites eaten using both hands

14. Bite differentiation Recognizing different types of bites using template matching against the typical motion pattern 14 ? ? ?

15. Algorithm:

16. Bite detection Detect the bites from other activities during a meal by template matching based on just roll motion Steps:  Sum of absolute difference between a bite template and the wrist motion data at every time step  Detecting local minima  Best template matched at the local minima position Detected bite

17. Ground truth bites Computer detected bites

18. Results Bite templates Bite differentiation Bite detection 18

19. Total bite templates 19

20. 17,166 ground truth food bites 3,185 bites drink bites 20 Food bites (17,166 bites) Drink bites (3,185 bites)

21. Food bites larger average motion in the Z and roll axes Drink bites larger average motion in the X and yaw axes 21 Food bites (17,166 bites) Drink bites (3,185 bites)

22. Drink bites longer (slower) motion than food bites in the yaw axis. Roll motion for drink bites is opposite to food bites, with negative roll preceding positive roll. 22 Food bites (17,166 bites) Drink bites (3,185 bites)

23. 23 Food bites (17,166 bites) Drink bites (3,185 bites) Food bites opposite average motion with drink bites in roll axes

24. 24 Fork (8,764 bites) Spoon (1,986 bites) Single hand (9,241 bites) Both hand (2,441 bites) Ax Ay Az Yaw Pitch Roll

25. Bite differentiation Ground truth Computer detected Food (Ax,Ay,Az,Yaw,Pitch,Roll)Drink(Ax,Ay,Az,Yaw,Pitch,Roll) Food75%,72%,68%,72%,43%,64%25%,28%,32%,27%,57%,36% Drink13%,10%,12%,40%,19%,5.6%87%,90%,88%,60%,81%,94% Accuracy81%,81%,78%,66%,62%,79% 25 Ground truth Computer detected FoodDrink Food70%30% Drink5%95% Accuracy83%  Bite differentiation of food and drink bites using all 6 motion axes.  Accelerometer and gyroscope motions confusion table for food & drink bites recognition.

26. Confusion matrices for the five types of bites according to utensil, for each axis Overall accuracy for recognizing for the 4 different types of utensils :19- 48% and Drink: 80%  Confusion Accelerometer motion axes. 26 Ground truth (Ax,Ay,Az) Computer detected (Ax,Ay,Az) %ForkSpoonDrinkBoth handSingle hand Fork23,20,2149,56,514,5,613,9,1012,10,1.4 Spoon19,14,1820,64,604.3,5,514,8.5,98,9,9 Drink1,1,15,4.5,681,84,826,6,6.57,5,5 Both hand8,6,730,36,3118,28,3728,18,1617,12,10 Single hand15,11,1021,27,2820,28,3519,15,1125,19,17 Accuracy42,41,39 %

27.  Confusion gyroscope motion axes. 27 Ground truth (Yaw, Pitch, Roll) Computer detected (Yaw,Pitch,Roll) %ForkSpoonDrinkBoth handSingle hand Fork42,14,4931,33,1714,25,86,13,127,14,14 Spoon40,9,3738,40,2011,25,135,11,147,15,16 Drink28,3,1.5 10,10,1.6 51,48,7110,27,181.5,12,8 Both hand41,6,417,19,628,31,409,31,326,13,19 Single hand36,8,3529,27,1220,31,187,16,1810,19,18 Accuracy30, 31, 38

28.  Confusion combining all 6 motion axes. 28 Ground truth (Ax, Ay, Az, Yaw, Pitch, Roll) Computer detected (Ax,Ay,Az,Yaw,Pitch,Roll) %ForkSpoonDrinkBoth handSingle hand Fork48273156 Spoon30386215 Drink1.5 80108 Both hand55284616 Single hand3313142219 Accuracy 46

29. Bite detection Tested on 22,383 total bites Detection rate: 48% Positive predictive value: 75% No higher performance for different axes and different combinations of axes 29

30. Conclusions Food and drink bites appear to have different wrist motion patterns Different types of utensils for food bites also appear to have different wrist motion patterns, however, they are not consistent enough to enable differentiation via template matching Original threshold-based algorithm: 77% true detections, 86% PPV Template matching algorithm: 46% true detection, 75% PPV Template matching is too rigid for detecting bites; there is too much variability in appearance; interestingly, it yielded the close PPV in the threshold-based algorithm suggesting it might be useful for suppressing false positives 30

31. References [1] O. Amft, H. Junker, and G. Troster, \Detection of eating and drinking arm gestures using inertial body-worn sensors," in Proceedings of the ninth IEEE International Symposium on Wearable Computers, 2005, pp. 160-163. [2] O. Amft and G. Troster, \On-body sensing solutions for automatic dietary monitoring,"IEEE Pervasive Computing, vol. 8, no. 2, pp. 62-70, 2009. [3] G. Billington, Epstein, \Overweight, obesity, and health risk." Arch Intern Med, vol. 160, pp. 898,904- 2000. [4] M. Boninsegna and M. Rossi, \Similarity measures in computer vision," Pattern Recognition Letters, vol. 15, no. 12, pp. 1255-1260, 1994. [5] C. Champagne, G. Bray, A. Kurtz, J. Monteiro, E. Tucker, J. Volaufova, and J. Delany, \Energy intake and energy expenditure: a controlled study comparing dietitians and non-dietitians," Journal of the American Dietetic Association, vol. 102, no. 10, pp. 1428-1432, 2002. [6] C. Ching, M. Jenu, and M. Husain, \Fitness monitor system," in Proceeding of Conference on Convergent Technologies for Asia-Pacic Region, vol. 4, 2003, pp. 1399-1403. 31

32. [7] Y. Dong, \Tracking wrist motion to detect and measure the eating intake of free- living humans," Ph.D. dissertation, Electrical and Computer Engineering Department, Clemson University, 2012. [8] Y. Dong, J. Scisco, M. Wilson, E. Muth, and A. Hoover, \Detecting periods of eating during free living by tracking wrist motion," IEEE of Biomedical Health Informatics, 2013. [9] K. Flegal, M. Carroll, B. Kit, and C. Ogden, \Prevalence of obesity and trends in the distribution of body mass index among us adults, 1999-2010," The journal of the American Medical Association, vol. 307, no. 5, pp. 491-497, 2012. [10] J. Foreyt and W. Poston II, Overview and the future of obesity treatment. Springer, 1999. [11] C. Gagnadre, M. Billon, and S. Thuillier, \Fibre optic sensor for physiological parameters," Electronics Letters, vol. 34, no. 21, pp. 1991-1993, 1998. [12] C. Harland, T. Clark, and R. Prance, \High resolution ambulatory electrocardiographic monitoring using wrist-mounted electric potential sensors," Measurement Science and Technology, vol. 14, no. 7, pp. 923-928, 2003. [13] B. Howard and S. Howard, \Lightglove: Wrist-worn virtual typing and pointing,“ in Proceedings of IEEE Fifth International Symposium on Wearable Computers,2001, pp. 172-173. 32

33. [14] Z. Huang, \An assessment of the accuracy of an automated bite counting method in a cafeteria setting," Master's thesis, Electrical and Computer Engineering Department, Clemson University, 2013. [15] H. Junker, O. Amft, P. Lukowicz, and G. Troster, \Gesture spotting with bodyworn inertial sensors to detect user activities," pattern Recognition, vol. 41, no. 6, pp. 2010-2024, 2008. [16] E. Kelly, Obesity: Health and Medical Issues Today. Greenwood Publishing Group, 2006. [17] S. Kumar, W. Nilsen, M. Pavel, and M. Srivastava, \Mobile health: Revolutionizing healthcare through transdisciplinary research." IEEE Computer, vol. 46, no. 1, pp. 28- 35. [18] J. Lementec and P. Bajcsy, \Recognition of arm gestures using multiple orientation sensors: gesture classication," in Proceedings of the 7th IEEE International Conference on Intelligent Transportation Systems, 2004, pp. 965-970. [19] A. Mikhail, C. Frederic, and D. Philippe, \An algorithm for estimating all matches between two strings," INRIA, Tech. Rep., 2001. [20] C. Ogden, M. Carroll, L. Curtin, M. McDowell, C. Tabak, and K. Flegal, \Prevalence of overweight and obesity in the united states, 1999-2004," The journal of the American Medical Association, vol. 295, no. 13, pp. 1549-1555, 2006. [21] G. Ogris, T. Stiefmeier, H. Junker, P. Lukowicz, and G. Troster, \Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures," in proceedings of the Ninth IEEE International Symposium on Wearable Computers, 2005, pp. 152-159. 33

34. [22] W. H. Organization, \Overweight and obesity," http://www.who.int/ mediacentre/factsheets/fs311/en/index.html, 2008.http://www.who.int/ [23] J. Salley, \Accuracy of a bite-count based calorie estimate compared to human estimates with and without calorie information available," Master's thesis, Psychology Department, Clemson University, 2013. [24] E. Sazonov and S. Schuckers, \The energetics of obesity: A review: Monitoring energy intake and energy expenditure in humans," IEEE Engineering in Medicine and Biology Magazine, vol. 29, no. 1, pp. 31-35, 2010. [25] D. Schmidt, R. Dannenberg, A. Smailagic, D. Siewiorek, and B. Biigge, \Learning an orchestra conductor's technique using a wearable sensor platform," in Proceeding of the 11th IEEE International Symposium on Wearable Computers, 2007, pp. 113-114. [26] A. Smailagic, D. Siewiorek, U. Maurer, A. Rowe, and K. Tang, \Ewatch: Context sensitive system design case study," in Proceedings of the IEEE Computer Society Annual Symposium on VLSI, 2005, pp. 98-103. [27] STMelectronics. (2013) Mems inertial sensor 3-axis linear accelerometer. http: //www.st.com/web/catalog/sense power/FM89/SC444/PF207281. [28] STMelectronics. (2013) Mems motion sensor 2-axis pitch and roll gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248621. 34

35. [29] STMelectronics. (2013) Mems motion sensor 2-axis pitch and yaw gyroscope. http://www.st.com/web/catalog/sense power/FM89/SC1288/PF248616. [30] C. Sugimoto, H. Ariesanto, H. Hosaka, K. Sasaki, N. Yamauchi, and K. Itao, \Development of a wrist-worn calorie monitoring system using bluetooth," Mi- crosystem technologies, vol. 11, no. 8-10, pp. 1028-1033, 2005. [31] L. Terre, W. Poston II, and J. Foreyt, 12 Overview and the Future of Obesity Treatment. Springer, 2005. [32] N.Wellman and B. Friedberg, \Causes and consequences of adult obesity: health, social and economic impacts in the united states," Asia Pacic journal of clinical nutrition, vol. 11(s8), pp. S705-S709, 2002. 35

36. Thank you! Q uestions? 36