Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 A self-organizing map for adaptive processing of structured data Advisor ： Dr. Hsu Reporter ： Chun Kai Chen Author ： Markus Hagenbuchner and Alessandro Sperduti 2003 IEEE TNN

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2 Outline Motivation Objective Introduction Supervised Learning: Recursive Neural Networks Unsupervised Learning within an SOM Framework Experimental Evaluation Conclusions Personal Opinion

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 3 Motivation  Recent developments in the area of neural networks produced models capable of dealing with structured data.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 4 Objective  Here, we propose the first fully unsupervised model ─ an extension of traditional self-organizing maps (SOMs) ─ for the processing of labeled directed acyclic graphs (DAGs)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 5 Introduction(1/3)  MANY natural and artificial systems are more appropriately modeled using data structures ─ This structured representation conveys much more information than a “flat” one, e.g., a vector of numerical features extracted from the image ─ Supervised: Use feedback from knowledge of correct classification. ─ Unsupervised: No knowledge of correct classification needed.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 6 Introduction(2/3)  In recent years, supervised neural networks have been developed ─ able to deal with structured data encoded as labeled directed acyclic graphs (DAGs) ─ Supervised information, however, either may not be available or very expensive to obtain  Thus it is very important to develop models which are able to deal with structured data in an unsupervised fashion

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 7 Introduction(3/3)  In this paper ─ we will show how self- organizing maps (SOMs) [22] can be extended to treat complex objects represented by data structures ─ we extend the standard SOM model [22], so as to allow the mapping of structured objects into a topological map

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 8 Fundamental Concepts of Data Structures  Given a DAG D ─ v ─ denote by ch[v] the set of children of v ─ the kth child of v by ch k [v]  The data structures ─ we consider are labeled DAGs ─ labels are tuples of variables  In the following ─ denote by # (C) the class of DAGs with maximum outdegree c

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 9 Supervised Learning: Recursive Neural Networks(1/3)  Recursive neural networks described in [29] are neural networks capable of performing mappings from a set of labeled graphs to a set of real vectors

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 10 Supervised Learning: Recursive Neural Networks(2/3)  Recursive neural networks described in [29] are neural networks capable of performing mappings from a set of labeled graphs to a set of real vectors ─ IR m where denotes the label space, ─ while the remaining domains represent the encoded subgraphs spaces up to the maximum outdegree of the input domain I #, ─ c is the maximum out-degree of DOAGs in I #, ─ s = source(D), ─ y s is the label attached to the super-source of D, and ─ D (1) …D (c) are the subgraphs pointed by s

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 11 Supervised Learning: Recursive Neural Networks(3/3)  The function τ ─ maps from a higher dimensional space (i.e., m+c ˙ n) to a lower dimensional space (i.e., n). ─ the role of consists of compressing the information about a node (i.e., the label, and the compressed information about the children of the node) in a vector of dimension  This observation is ─ fundamental to understand how (1) can be adapted to unsupervised learning within a SOM approach. ─ In fact, the aim of the SOM learning algorithm is to learn a feature map ─ which given a vector in the spatially continuous input space I ─ returns a point in the spatially discrete output display space A

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 12 Unsupervised Learning within an SOM Framework  We are interested in generalizing (4) to deal with the case, i.e., the input space is a structured domain with labels in. ─ nil Α is a special coordinate vector into the discrete output space Is represented by the coordinates (-1 -1).

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 13 Model of M node

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 14 Training Algorithm for M node  Initialize SOM  For each input data ─ Step 1—Competitive Step: Identify its Best Matching Unit (BMU) ─ Step 2—Cooperative Step: Adjust BMU and its neighborhood  Repeat Step 2 till stop criteria met Data x1=[8, 5, 9] x2=[7, 4, 2] … x1=[8, 5, 9] m i =[7, 5, 8] BMU training After training

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 15 Training Algorithm For M #

Intelligent Database Systems Lab N.Y.U.S.T. I. M Experimental Evaluation  For the experiments, we used an extended version of the Policemen Benchmark, ─ is a set of labeled DOAGs extracted from images produced ─ the dataset consists of visual patterns and associated graph structures from three different domains

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 17 d=2 d=1 overlap

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 18

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 19 Fig. 14. Network performance versus size of the training set. Fig. 15. Network performance versus the total number of iterations trained. over fitting Only 3% neurons are utilized over fitting problem

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 20 Fig. 16. Network performance versus initial neighborhood radii

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 21 Fig. 17. Network performance versus the initial learning parameter

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 22 Conclusion  In this paper ─ we have described a new way of addressing unsupervised learning problems involving structured objects using an SOM technique  The experiments ─ clearly show that the model is able to encode structural features in a coherent way ─ optimal parameters a network of size 114×87 (number of neurons approximately 1/3 the number of nodes in the training set) the best set of parameters is the number of training iterations is greater than 450

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 23 Personal Opinion  Advantage ─ Specifically, we showed how an SOM network can be used recursively in order to learn a map defined on DAGs ─ The major innovation is to treat each leaf node as a standard input for an SOM, and the coordinates of the winning neuron as pointer within a parent node  Disadvantage ─ The information on the labels, on the other hand, can be encoded finely only if the map is sufficiently large and training parameters are chosen appropriately