1. 1 i-Care Project Dietary-Aware Dining Table: Observing Dietary Behavior over Tabletop Surface Keng-hao Chang, Shih-yen Liu, Toung Lin, Hao Chu, Jane Hsu, Polly Huang, (Cheryl Chen) i-space Laboratory National Taiwan University

2. 2 What is it? A dietary-tracker built into an everyday dining table –Track what & how much you eat over tabletop surface Motivation –We are what we eat –Food choices affect long-term & short-term health Show a demo video

3. 3 Smart Everyday Object Digital-enhanced everyday objects –Provide digital services Support natural human interactions –Natural human interactions = inputs to digital services Goals –Providing digital services without (users) operating digital devices → better usability –Human-centric computing: technology adapting to users rather than users adapting & learning about technology

4. 4 Outline for Reminder of Talk Related work Approach Assumptions & Limitations Design & Implementation Experimental Evaluation Future work

5. 5 Related Work Dietary trackers –Shopping receipt scanner (GaTech) –Chewing Sound (ETH) –My food phone (startup) Intelligent surfaces –Load sensing table (Lancester) –Smart floor (GaTech, NTU) –Posture Chair (MIT) What’s new here? –Accuracy –Fine-grained tracking –Simultaneous concurrent interactions

6. 6 Contribution claims It is a fine-granularity (automated) dietary tracker. –It can track multiple concurrent interactions from multiple individuals over the same tabletop surface. People usually don’t eat alone It is an enhanced loading sensing table.

7. 7 General Approach RFID tags on food containers Two sensor surfaces on table –Each surface is made of cells –RFID reader surface Detect RFID(s) in each cell –Weighting surface (load cells) Measure weight change in each cell Track the food path from container(s) → container(s) → mouth using these two sensor surfaces

8. 8 Assumptions (Limitations) Closed system rather than open system. –Food transfers among tabletop objects and mouths, no external objects and food sources Users identified by personal containers (personal plates and cups) Food containers tagged with RFID tags No cross-cell objects No leaning their hands on the table Not a mobile tracker

9. 9 Single Interaction Example Bob pours tea from the tea pot to his personal cup, and drinks it Detect tea transfer from one container to another container 1)Identify the presence & absence of containers RFID tags on containers tag-food mapping 2)Track tea transfer Weight change detection Weight matching algorithm

10. 10 Single Interaction Example Pour tea? Weight increases ∆ w 2. Bob pours tea from the tea pot to personal cup, and drinks it Pick up tea pot. RFID tag disappears Weight decreases ∆ w 1 Put on tea pot. RFID tag appears Weight increases ∆w 3 Pour tea! | ∆ w 3 - ∆ w 1 | ≈ ∆w 2

11. 11 Single Interaction Example Bob pours tea from the tea pot to personal cup, and drinks it Pick up cup. RFID tag disappears. Weight decreases ∆w 1. Put on cup. RFID tag appears. Weight increases ∆w2. Drink tea? (only if no match) Amount | ∆w2 - ∆w1 |

12. 12 Concurrent Interactions Example Bob pours tea & Alice cuts cake Pour tea? Cut cake? Weight change ∆ w Pour tea Weight increases ∆ w 1 Cut cake Weight decreases ∆w 2

13. 13 Concurrent Interactions Example Multiple, concurrent person-object interactions –The larger the cell, the higher the possibility of concurrent interactions over a cell –Cell size = average size of container –Reduce the possibility of concurrent interactions over one cell

14. 14 Design Architecture Tag-object mappings Behavior Inference Engine Event Interpreter Weight Change DetectorObject Presence Detector Weighing surface (weighing sensors) RFID Surface (readers) Applications (Dietary-aware Dining Table) Common sense semantics Sensor Events Intermediate Events Dietary Behaviors

15. 15 Inference Rule Dietary behaviorsBehavior Inference Rules Transfer(u, w, type)Weight-Change(Object-i1, Δ w1) ∩ ( Δ w1 0) ∩ Contains(Object-i1, type) ∩ Owner(Object-i2, u) ∩ (| Δ w1 + Δ w2 |< ε ) → Transfer (u, Δ w2, type) Eat(u, w, type)Weight-Change(Object-i, Δ w) ∩ ( Δ w<0) ∩ Contains(Object-i, type) ∩ Owner(Object-i, u) → Eat(u, - Δ w, type)

16. 16 Experimental setup 2 Dining settings –Afternoon tea –Chinese-style dinner 2 Parameters –# of participants –Predefined vs. Random Sequence A Willy Keng-hao

17. 17 Experimental Results ScenariosEvent StatisticsResults Dining Scenarios # Users Activity Sequence Time Duration (seconds) # of Dietary Behavior Average Activity Interval Behavior Recognition Accuracy #1 Afternoon tea 1Predefined73126.08100% #2 Afternoon tea 2Predefined162246.75100% #3 Afternoon tea 2Random9137811.7179.49% #4 Chinese- style dinner 3Random181116211.1885.8%

18. 18 Predefined Activity Sequence Afternoon Tea (Single User) 1.cut a piece of cake and transfer it to the personal plate; 2.pour tea from the tea pot to the personal cup; 3.add milk to the personal cup from the creamer; 4.eat the piece of cake from the personal plate; 5.drink tea from the personal cup; 6.add sugar to the personal cup from the sugar jar. Afternoon Tea (Multi-users) 1.A cuts cake and transfers it to A’s personal plate; 2.B pours tea from the tea pot to B’s personal cup; 3.A pours tea to A’s personal cup while B cuts a piece of cake and transfers it to B’s personal plate; 4.A adds sugar from the sugar jar to A’s personal cup while B adds milk from the creamer to B’s personal up; 5.A eats cake and B drinks tea; 6.B eats cake from B’s personal plate while A drinks tea from A’s personal cup; 7.A pours tea from the tea pot to both A’s and B’s personal cups.

19. 19 Activity Recognition Accuracy in Scenario #3 Dietary Behavior# of Actual EventsRecognition Accuracy Transfer event4170.73% Eat event3789.19%

20. 20 Causes of Misses in Scenario #3 Causes of misses# of misses of transfer events # of misses of eat events Total Event interference within the weighing cell’s weight stabilization time 628 Weight matching threshold202 Slow RFID sample rate303 Touching table123 Total of misses12416

21. 21 Dietary Behavior# of times Recognition AccuracyWeight Accuracy Transfer dish A events1973.68%68.42% Transfer dish B events2979.31%78.75% Transfer dish C events2382.61%79.19% Transfer rice events1283.33%81.88% Transfer soup events1984.21%80.16% Eat events6088.33%91.23% Activity Recognition Accuracy in Scenario #4

22. 22 Causes of Misses in Scenario #4 Causes of misses# of misses of transfer events # of misses of eat events Total Segmented weight-change events505 Eating before transferring food to personal containers 5510 Weight matching ambiguity707 Touching table325 Slow RFID sample rate303 Total of misses23730

23. 23 Conclusion It is a smart object and a smart surface It supports natural user interface It supports fine-grained dietary tracking at individual level It is about human-centric computing Accuracy can be improved further

24. 24 Future Work Improving recognition accuracy Removing constraints (assumptions) Persuasive computing –Encourage balanced diet –Encourage proper amount of diet

25. 25 Questions & Answers Thank You