1. Bayesian Metanetworks for Context-Sensitive Feature Relevance Vagan Terziyan vagan@it.jyu.fi Industrial Ontologies Group, University of Jyväskylä, Finland SETN-2006, Heraclion, Crete, Greece 24 May 2006

2. 2 Contents Bayesian Metanetworks Metanetworks for managing conditional dependencies Metanetworks for managing feature relevance Example Conclusions Vagan Terziyan Industrial Ontologies Group Department of Mathematical Information Technologies University of Jyvaskyla (Finland) http://www.cs.jyu.fi/ai/vagan This presentation: http://www.cs.jyu.fi/ai/SETN-2006.ppthttp://www.cs.jyu.fi/ai/SETN-2006.ppt

3. 3 Bayesian Metanetworks

4. 4 Conditional dependence between variables X and Y P(Y) =  X (P(X) · P(Y|X))

5. 5 Bayesian Metanetwork Definition. The Bayesian Metanetwork is a set of Bayesian networks, which are put on each other in such a way that the elements (nodes or conditional dependencies) of every previous probabilistic network depend on the local probability distributions associated with the nodes of the next level network.

6. 6 Two-level Bayesian C-Metanetwork for Managing Conditional Dependencies

7. 7 Contextual and Predictive Attributes Machine Environment Sensors X x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 predictive attributes contextual attributes air pressure dust humidity temperature emission

8. 8 Contextual Effect on Conditional Probability (1) X x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 predictive attributes contextual attributes xkxk xrxr Assume conditional dependence between predictive attributes (causal relation between physical quantities)… xtxt … some contextual attribute may effect directly the conditional dependence between predictive attributes but not the attributes itself

9. 9 Contextual Effect on Conditional Probability (2) X ={x 1, x 2, …, x n } – predictive attribute with n values; Z ={z 1, z 2, …, z q } – contextual attribute with q values; P(Y|X) = {p 1 (Y|X), p 2 (Y|X), …, p r (Y|X)} – conditional dependence attribute (random variable) between X and Y with r possible values; P(P(Y|X)|Z) – conditional dependence between attribute Z and attribute P(Y|X);

10. 10 Contextual Effect on Conditional Probability (3) xkxk xrxr xtxt X k 1 : order flowers X k 2 : order wine X r 1 : visit football match X r 2 : visit girlfriend P 1 (X r |X k ) Xk1Xk1 Xk2Xk2 Xr1Xr1 0.30.9 Xr2Xr2 0.40.5 P 2 (X r |X k ) Xk1Xk1 Xk2Xk2 Xr1Xr1 0.10.2 Xr2Xr2 0.80.7 X t 1 : I am in Paris X t 2 : I am in Moscow X k : Order present X r : Make a visit

11. 11 Contextual Effect on Conditional Probability (4) xtxt P 1 (X r |X k ) Xk1Xk1 Xk2Xk2 Xr1Xr1 0.30.9 Xr2Xr2 0.40.5 P 2 (X r |X k ) Xk1Xk1 Xk2Xk2 Xr1Xr1 0.10.2 Xr2Xr2 0.80.7 X t 1 : I am in Paris X t 2 : I am in Moscow xrxr xkxk P( P (X r |X k ) | X t ) Xt1Xt1 Xt2Xt2 P 1 (X r |X k ) 0.70.2 P 2 (X r |X k ) 0.30.8

12. 12 Contextual Effect on Unconditional Probability (1) X x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 predictive attributes contextual attributes xkxk Assume some predictive attribute is a random variable with appropriate probability distribution for its values… xtxt … some contextual attribute may effect directly the probability distribution of the predictive attribute x1x1 x2x2 x3x3 x4x4 X P(X)

13. 13 Contextual Effect on Unconditional Probability (2)  X ={x 1, x 2, …, x n } – predictive attribute with n values; · Z ={z 1, z 2, …, z q } – contextual attribute with q values and P(Z) – probability distribution for values of Z; P(X) = {p 1 (X), p 2 (X), …, p r (X)} – probability distribution attribute for X (random variable) with r possible values (different possible probability distributions for X) and P(P(X)) is probability distribution for values of attribute P(X); · P(Y|X) is a conditional probability distribution of Y given X; · P(P(X)|Z) is a conditional probability distribution for attribute P(X) given Z

14. 14 Contextual Effect on Unconditional Probability (3) xkxk xtxt XkXkXkXk P 1 (X k ) Xk1Xk1 Xk2Xk2 0.2 0.7 XkXkXkXk P 2 (X k ) Xk1Xk1 Xk2Xk2 0.5 0.3 X t 1 : I am in Paris X t 2 : I am in Moscow X k 1 : order flowers X k 2 : order wine X k : Order present P( P (X k ) | X t ) Xt1Xt1 Xt2Xt2 P 1 (X k ) 0.40.9 P 2 (X k ) 0.60.1

15. 15 Causal Relation between Conditional Probabilities xkxk xrxr xmxm xnxn P 1 (X n |X m ) P(X n | X m ) P(P(X n | X m )) P 2 (X n |X m )P 3 (X n |X m ) P 1 (X r |X k ) P(X r | X k ) P(P(X r | X k )) P 2 (X r |X k ) P(P(X r | X k )|P(X n | X m )) There might be causal relationship between two pairs of conditional probabilities

16. 16 Two-level Bayesian C-Metanetwork for managing conditional dependencies

17. 17 Example of Bayesian C-Metanetwork The nodes of the 2 nd -level network correspond to the conditional probabilities of the 1 st -level network P(B|A) and P(Y|X). The arc in the 2 nd - level network corresponds to the conditional probability P(P(Y|X)|P(B|A))

18. 18 Two-level Bayesian R-Metanetwork for Modelling Relevant Features’ Selection

19. 19 Feature relevance modelling (1) We consider relevance as a probability of importance of the variable to the inference of target attribute in the given context. In such definition relevance inherits all properties of a probability.

20. 20 Feature relevance modelling (2) X: {x 1, x 2, …, x nx }

21. 21 Example (1) Let attribute X will be “state of weather” and attribute Y, which is influenced by X, will be “state of mood”. X (“state of weather”) ={“sunny”, “overcast”, “rain”}; P(X=”sunny”) = 0.4; P(X=”overcast”) = 0.5; P(X=”rain”) = 0.1; Y (“state of mood”) ={“good”, “bad”}; P(Y=”good”|X=”sunny”)=0.7; P(Y=”good”|X=”overcast”)=0.5; P(Y=”good”|X=”rain”)=0.2; P(Y=”bad”|X=”sunny”)=0.3; P(Y=”bad”|X=”overcast”)=0.5; P(Y=”bad”|X=”rain”)=0.8; P(X) P(Y|X) Let:  X =0.6

22. 22 Example (2) Now we have: One can also notice that these values belong to the intervals created by the two extreme cases, when parameter X is not relevant at all or it is fully relevant: !

23. 23 General Case of Managing Relevance (1) Predictive attributes: X1 with values {x1 1,x1 2,…,x1 nx1 }; X2 with values {x2 1,x2 2,…,x2 nx2 }; … XN with values {xn 1,xn 2,…,xn nxn }; Target attribute: Y with values {y 1,y 2,…,y ny }. Probabilities: P(X1), P(X2),…, P(XN); P(Y|X1,X2,…,XN). Relevancies:  X1 = P(  (X1) = “yes”);  X2 = P(  (X2) = “yes”); …  XN = P(  (XN) = “yes”); Goal: to estimate P(Y).

24. 24 General Case of Managing Relevance (2) Probability P(XN)

25. 25 Example of Relevance Bayesian Metanetwork (1) Conditional relevance !!!

26. 26 Example of Relevance Bayesian Metanetwork (2)

27. 27 Example of Relevance Bayesian Metanetwork (3)

28. 28 When Bayesian Metanetworks ? 1. Bayesian Metanetwork can be considered as very powerful tool in cases where structure (or strengths) of causal relationships between observed parameters of an object essentially depends on context (e.g. external environment parameters); 2. Also it can be considered as a useful model for such an object, which diagnosis depends on different set of observed parameters depending on the context.

29. 29 Conclusion We are considering a context as a set of contextual attributes, which are not directly effect probability distribution of the target attributes, but they effect on a “relevance” of the predictive attributes towards target attributes. In this paper we use the Bayesian Metanetwork vision to model such context-sensitive feature relevance. Such model assumes that the relevance of predictive attributes in a Bayesian network might be a random attribute itself and it provides a tool to reason based not only on probabilities of predictive attributes but also on their relevancies.

30. 30 Read more about Bayesian Metanetworks in: Terziyan V., A Bayesian Metanetwork, In: International Journal on Artificial Intelligence Tools, Vol. 14, No. 3, 2005, World Scientific, pp. 371-384. http://www.cs.jyu.fi/ai/papers/KI-2003.pdf Terziyan V., Vitko O., Bayesian Metanetwork for Modelling User Preferences in Mobile Environment, In: German Conference on Artificial Intelligence (KI-2003), LNAI, Vol. 2821, 2003, pp.370-384. http://www.cs.jyu.fi/ai/papers/IJAIT-2005.pdf Terziyan V., Vitko O., Learning Bayesian Metanetworks from Data with Multilevel Uncertainty, In: M. Bramer and V. Devedzic (eds.), Proceedings of the First International Conference on Artificial Intelligence and Innovations, Toulouse, France, August 22- 27, 2004, Kluwer Academic Publishers, pp. 187-196. http://www.cs.jyu.fi/ai/papers/AIAI-2004.ps