1. Hybrid-ε-greedy for Mobile Context- Aware Recommender System Djallel Bouneffouf, Amel Bouzeghoub & Alda Lopes Gançarski Institut Télécom, Télécom SudParis, France 1

2. OUTLINE 1. Introduction 2. State of the art 3. Proposition 4. Experimental evaluation 5. Conclusion 2

3. OUTLINE 1. Introduction 2. State of the art 3. Proposition 4. Experimental evaluation 5. Conclusion 3

4. Software editor Access and navigation into the corporate data www.nomalys.com 4

5. 5 MOBILE INFORMATION SYSTEMS Context

6. 6 Context-based Recommender System To reduce search and navigation time To assist users in finding information

7. PROBLEMS IN CONTEXT-BASED RECOMMENDER SYSTEM 7 USER Contextual Recommender System algorithm: Selects item(s) to show Gets feedback (click, time spent,..) Refine the models Repeats (large number of times) with an Optimization of metrics of interest (Total number of clicks, Total rewards,…) How to recommend information to users taking into account their surrounding environment (location, time, near people)? How to follow the evolution of user’s interest? Item Inventory Articles, web page, documents, … Context location, time, …

8. OUTLINE 1. Introduction 2. State of the art 3. Proposition 4. Experimental evaluation 5. Conclusion 8

9. USER OR EXPERT SPECIFICATION Constraints  Laborious  Not a dynamic system  Not a personalized system Advantage  Context management Reference [Panayiotou, 2006] [Bila, 2008] [Bellotti, 2008] [ Dobson, 2005] [Lakshmish, 2009] [Alexandre de Spindler, 2006] [Mieczysław, 2009] [Wei, 2010] [Lihong, 2010] 9

10. CONTENT-BASED AND COLLABORATIVE FILTERING Dataset Situations Reward Action Social group Meeting Home Drive Offic e Constraints  Cold start problem  Slow training Reference [Panayiotou, 2006] [Bila, 2008] [Bellotti, 2008] [ Dobson, 2005] [Lakshmish, 2009] [Alexandre de Spindler, 2006] [Mieczysław,2009] [Wei,2010] [Lihong, 2010] Advantage  Context management  Automatic process 10

11. MACHINE LEARNING - The greedy strategy only exploitation; - The ε-greedy strategy adds some random action. Advantage  Solve cold start problem  Follow the evolution of user interest Constraints  No context management  Slow training Reference[Panayiotou,2006] [Bellotti,2008] [Bila,2008] [Dobson,2005] [Lakshmish, 2009] [Alexandre de Spindler, 2006] [Mieczysła,2009] [Wei,2010] [Lihong,2010] Reinforcement learning 11 DocumentsD1D2D3D4D5D6D8D9D10 Displays12 Number of Clicks 7 52 1 Exploration Exploitation 1 2 1 2 1 1 mean= 0.48 mean= 0.79

12. State of the art Learning Profile The user or The expert specificati- on Content and Collaborative filteringReinforcement learning Reference [Pana yiotou,2006] [Bila, 2008] [Bellotti, 2008] [Dobson, 2005] [Lakshmish, 2009] [Alexandre de Spindler, 2006] [Mieczysła, 2009] [Wei,2010] [Lihong,2010] Context management +++++ + --- Semantic Context representation ++++ ----- Content- based -- ++ ----- Automatic process - - +++++++ Follow the evolution of user interest ------ +++ Solve the cold start ------ +++ 12

13. OUTLINE 1. Introduction 2. State of the art 3. Proposition 4. Experimental evaluation 5. Conclusion 13

14. MULTI-ARMED BANDITS (MAB) A (basic) MAB problem has: A set D of possibilities (documents) A CTR(d) ∈ [0,1] of expected rewards for each d ∈ D In each round: algorithm picks document d ∈ D based on past history Reward: independent sample in [0,1] with expectation CTR (d) Classical setting that models exploration/exploitation trade-off 14 DocumentsD1D2D3D4D5D6D8D9D10 CTR0.60.40.30.5

15. CONTEXTUAL BANDITS X is a set of situations, D is a set of arms, CTR: X x D  [0,1] expected rewards Situation x ∈ X occurs In each round: Algorithm picks arm d ∈ D Rewards: independent sample in [0,1] with expectation CTR(x, d) 15 x1x1 x2x2 x3x3 DocumentsD1D2D3D4D5D6D8D9D10 CTR0.20.40.30.4 DocumentsD1D2D3D4D5D6D8D9D10 CTR0.60.40.30.5 DocumentsD1D2D3D4D5D6D8D9D10 CTR0.20.10.30.7 Situations Meeting Home Drive Offic e 1 2

16. GET SITUATION FROM CONTEXT SENSING Mon Oct 3 12:10:00 2011 GPS "38.868143, 2.3484122" NATIXIS 16

17. GET SITUATION FROM CONTEXT THINKING ABSTRACTION Mon Oct 3 12:10:00 2011 GPS "38.868143, 2.3484122" NATIXIS Time Ontology Location Ontology Social Ontology situationtimelocationsocial x1x1 workdayParisBank 17

18. GET SITUATION FROM CONTEXT RETRIEVING THE RELEVANT SITUATION IDSUsersTimePlaceClient 1PaulWorkdayParisBNP 2FabriceHolydayEvryMGET 3PaulWorkdayGentillyAMUNDI IDSUsersTimePlaceClient 1PaulWorkdayParisNATIXIS RetrieveSituation IDSUsersTimePlaceClient 1PaulWorkdayParisBNP Location Ontology Time Ontology Social Ontology

19. SELECT DOCUMENTS 19 Documentsd1d2d3d4d5d6d8d9d10 CTR0.60.20.4 Hybrid-ε-greedy argmax d (CTR(d)) p(1-ε) d t = Random(D) p(ε) CBR-ε-greedy argmax d (CTR(d)) p(1-ε) d t = CBF (d) p(z) Random(D) p(k) ε is the probability of exploration CBF (d) gives documents similar to document d ε = z+k Content-Based filtering (CBF)

20. OUTLINE 1. Introduction 2. State of the art 3. Proposition 4. Experimental evaluation 5. Conclusion 20

21. IDSUsersTimePlaceClient 1Paul11/05/2011ParisAFNOR 2Fabrice15/05/2011EvryMGET 3Paul19/05/2011GentillyAMUNDI IdDocIDSClickTimeInterestDocuments 1122 min3/5Demand 2133 min1/5Contact 32150 secnullPerson Diary navigation entries Diary situation entries 21 Data from Nomalys 16 286 diary situations 342 725 navigation entries EXPERIMENTAL DATASETS

22. RECOMMEND DOCUMENTS ε variation on learning ε variation on deployment ε- Variation 22 ε is the probability of exploration CTR argmax d (CTR(d)) p(1-ε) d t = Random(D) p(ε)

23. RECOMMEND DOCUMENTS Data size on learning Data size on deployment Data size variation 23 CTR Data size

24. CONCLUSION Our experiments yield to the conclusion that: Considering the user’s context for the exploration/exploitation strategy significantly increases the performance of the recommender system. In the future: We plan to investigate methods that automatically learn the optimal exploitation and exploration trade-o ff. 24