10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather.

10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather than the "please find" end. To say it another way, data mining queries are on the ad hoc or unstructured end of the query spectrum rather than standard report generation or "retieve all records matching a criteria" or SQL side). Still, Data Mining queries ARE queries and are processed (or will eventually be processed) by a Database Management System the same way queries are processed today, namely: 1. SCAN and PARSE (SCANNER-PARSER): A Scanner identifies the tokens or language elements of the DM query. The Parser check for syntax or grammar validity. 2. VALIDATED: The Validator checks for valid names and semantic correctness. 3. CONVERTER converts to an internal representation. |4. QUERY OPTIMIZED: the Optimzier devises a stategy for executing the DM query (chooses among alternative Query internal representations). 5. CODE GENERATION: generates code to implement each operator in the selected DM query plan (the optimizer-selected internal representation). 6. RUNTIME DATABASE PROCESSORING: run plan code. Developing new, efficient and effective DataMining Query (DMQ) processors is the central need and issue in DBMS research today (far and away!). These notes concentrate on 5,i.e., generating code (algorithms) to implement operators (at a high level): Association Rule Mining (ARM), Clustering (CLU), Classification (CLA)

Machine Learning is almost always based on Near Neighbor Set(s), NNS. Clustering, even density based, identifies near neighbor cores 1 st (round NNS s,  about a center). Classification is continuity based and Near Neighbor Sets (NNS) are the central concept in continuity  >0  >0  : d(x,a)<   d(f(x),f(a))<  where f assigns a class to a feature vector, or   -NNS of f(a),  a  -NNS of a in its pre-image. f(Dom) categorical  >0  : d(x,a)<  f(x)=f(a) Data Mining can be broken down into 2 areas, Machine Learning and Assoc. Rule Mining Machine Learning can be broken down into 2 areas, Clustering and Classification. Clustering can be broken down into 2 types, Isotropic (round clusters) and Density-based Classification can be broken down into to types, Model-based and Neighbor-based Database analysis can be broken down into 2 areas, Querying and Data Mining. Caution: For classification, boundary analysis may be needed also to see the class (done by projecting?). 1234 Finding NNS in lower a dimension may still the 1st step. Eg, 12345678 are all  from a 5 a 6 (unclassified sample); 1234 are red-class, 5678 are blue-class. 7 8 Any  that gives us a vote gives us a tie vote (0-to-0 then 4-to-4). But projecting onto the vertical subspace, then taking  /2 we see that  /2 about a contains only blue class (5,6) votes. ****** Using horizontal data, NNS derivation requires ≥1 scan (O(n)). L  ε-NNS can be derived using vertical-data in O(log 2 n) (but Euclidean disks are preferred). (Euclidean and L  coincide in Binary data sets).

Query Processing and Optimization: Relational Queries to Data Mining Most people have Data from which they want information. So, most people need DBMSs whether they know it or not. A major component of any DBMS is the DataMining Query Processor. Queries can range from structure to unstructured: SQL: SELECT FROM WHERE Complex SQL queries (nested, EXISTS.. ) FUZZY queries (e.g., BLAST searches,.. OLAP (rollup, drilldown, slice/dice.. Machine LearningData Mining Relational querying Simple Searching and aggregating Supervised - Classification Regression Unsupervised- Clustering Association Rule Mining On the Data Mining end, we have barely scratched the surface. But those scratches have already made the difference between becoming the world’s biggest corporation (Walmart - got into DM for supply chain mgmt early) and filing for bankruptcy (KMart - didn't!).

Recall the Entity-Relationship (ER) Model's notion of a Relationship Relationship: Association among 2 [or more, the # of entities is the degree] entities. The Graph of a Relationship: A degree=2 relationship between entity T and I generates a bipartite undirected graph (bipartite means that the node set is a disjoint union of two subsets and that all edges must run from one subset to the other). lot name Employee ssn since Works_In dname budget did Department Degree=2 relationship between entities, Employees and Departments. subor- dinate super- visor Reports_To lot name Employee ssn To distinguish roles in a unipartite graph, can specify “role” of each entity. A degree=2 relationship between an entity and itself, e.g., Employee Reports_To Employee, generates a uni-partite undirected graph. Relationships can have attributes too!

Association Rule Mining (ARM) In a relationship between entities, T is the set of Transactions an enterprise performs and I is the set of Items on which those transactions are performed. In Market Basket Research (MBR) a transaction is a checkout transaction and an item is an Item in that customer's market basket which gets checked out) An I-Association Rule, A  C, relates 2 disjoint subsets of I (I-temsets) has 2 main measures, support and confidence (A is called the antecedent, C is called the consequent) There are also the dual concepts of T-association rules (just reverse the roles of T and I above). Examples of Association Rules include: The MBR, relationship between customer cash-register transactions, T, and purchasable items, I (t is related to i iff i is being bought by that customer during that cash-register transaction.). In Software Engineering (SE), the relationship between Aspects, T, and Code Modules, I (t is related to i iff module, i, is part of the aspect, t). In Bioformatics, the relationship between experiments, T, and genes, I (t is related to i iff gene, i, expresses at a threshold level during experiment, t). In ER diagramming, any “part of” relationship in which i  I is part of t  T (t is related to i iff i is part of t); and any “ISA” relationship in which i  I ISA t  T (t is related to i iff i IS A t)... The support of an I-set, A, is the fraction of T-instances related to every I-instance in A, e.g. if A={i 1,i 2 } and C={i 4 } then supp(A)= |{t 2,t 4 }|/|{t 1,t 2,t 3,t 4,t 5 }| = 2/5 Note: | | means set size or count of elements in the set. I.e., T 2 and T 4 are the only transactions from the total transaction set, T={T 1,T 2,T 3,T 4,T 5 }. that are related to both i 1 and i 2, (buy i 1 and i 2 during the pertinent T-period of time). support of rule, A  C, is defined as supp{A  C} = |{T 2, T 4 }|/|{T 1,T 2,T 3,T 4, T 5 }| = 2/5 confidence of rule, A  C, is supp(A  C)/ supp(A) = (2/5) / (2/5) = 1 DM Queriers typically want STRONG RULES: supp≥minsupp, conf≥minconf (minsupp and minconf are threshold levels) Note that Conf(A  C) is also just the conditional probability of t being related to C, given that t is related to A). T I A t1t1 t2t2 t3t3 t4t4 t5t5 i1i1 i2i2 i3i3 i4i4 C

Finding Strong Assoc Rules The relationship between Transactions and Items can be expressed in a Transaction Table where each transaction is a row containing its ID and the list of the items that are related to that transaction: T IDABCDEF 2000111000 1000101000 4000100100 5000010011 If minsupp is set by the querier at.5 and minconf at.75: To find frequent or Large itemsets (support ≥ minsupp) PseudoCode: Assume the items in L k-1 are ordered: Step 1: self-joining L k-1 insert into C k select p.item 1, p.item 2, …, p.item k-1, q.item k-1 p from L k-1, q from L k-1 where p.item 1 =q.item 1,..,p.item k-2 =q.item k-2, p.item k-1 <q.item k-1 Step 2: pruning forall itemsets c in C k do forall (k-1)-subsets s of c do if (s is not in L k-1 ) delete c from C k Or Transaction Table Items can be expressed using “Item bit vectors” FACT: Any subset of a large itemset is large. Why? (e.g., if {A, B} is large, {A} and {B} must be large) APRIORI METHOD: Iteratively find the large k-itemsets, k=1... Find all association rules supported by each large Itemset. C k denotes the candidate k-itemsets generated at each step L k denotse the Large k-itemsets. 3 2 2 1 1 1 1-itemset supp 3 2 2 Large (supp  2) Start by finding large 1-ItemSets.

6. 1 st half of 1 st of 2 nd is  1 0 0 1 1 4. 1 st half of 2 nd half not  0 0 2. 1 st half is not pure1  0 0 0 1. Whole file is not pure1  0 Horizontal structures (records) Scanned vertically P 11 P 12 P 13 P 21 P 22 P 23 P 31 P 32 P 33 P 41 P 42 P 43 0 0 0 0 1 10 0 1 0 0 1 0 0 0 0 0 0 1 01 10 0 1 0 0 1 0 0 0 0 1 0 01 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 10 01 5. 2 nd half of 2 nd half is  1 0 0 1 R 11 0 1 0 1 then process using multi-operand logical AND s. Vertical basic binary Predicate-tree (P-tree): vertically partition table; compress each vertical bit slice into a basic binary P-tree as follows 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 R( A 1 A 2 A 3 A 4 ) Ptree Review: A data table, R(A 1..A n ), containing horizontal structures (records) is processed vertically (vertical scans) The basic binary P-tree, P 1,1, for R 11 is built top- down by record truth of predicate pure1 recursively on halves, until purity. 3. 2 nd half is not pure1  0 0 7. 2 nd half of 1 st of 2 nd not  0 0 0 1 10 0 1 0 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 0 0 0 0 0 1 1 0 0 R 11 R 12 R 13 R 21 R 22 R 23 R 31 R 32 R 33 R 41 R 42 R 43 R[A 1 ] R[A 2 ] R[A 3 ] R[A 4 ] 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 Eg, Count number of occurences of 111 000 001 100 0 2 3 -level P 11 ^P 12 ^P 13 ^P’ 21 ^P’ 22 ^P’ 23 ^P’ 31 ^P’ 32 ^P 33 ^P 41 ^P’ 42 ^P’ 43 = 0 0 2 2 -level =2 01 2 1 -level But it is pure (pure0) so this branch ends

R 11 0 1 0 1 Top-down construction of basic binary P-trees is good for understanding, but bottom-up is more efficient. Bottom-up construction of P 11 is done using in-order tree traversal and the collapsing of pure siblings, as follow: 0 1 0 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 0 0 0 0 0 1 1 0 0 R 11 R 12 R 13 R 21 R 22 R 23 R 31 R 32 R 33 R 41 R 42 R 43 P 11 0 0 0 0 0 0 1 0 0 0 0 1 1 1 0

0 1 0 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 0 0 0 0 0 1 1 0 0 R 11 R 12 R 13 R 21 R 22 R 23 R 31 R 32 R 33 R 41 R 42 R 43 R[A 1 ] R[A 2 ] R[A 3 ] R[A 4 ] 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 To count occurrences of 7,0,1,4 use pure111000001100 : 0 P 11 ^P 12 ^P 13 ^P’ 21 ^P’ 22 ^P’ 23 ^P’ 31 ^P’ 32 ^P 33 ^P 41 ^P’ 42 ^P’ 43 = 0 0 01 ^ 7 0 1 4 P 11 P 12 P 13 P 21 P 22 P 23 P 31 P 32 P 33 P 41 P 42 P 43 0 0 0 0 1 10 0 1 0 0 1 0 0 0 0 0 0 1 01 10 0 1 0 0 1 0 0 0 0 1 0 01 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 10 01 ^ ^^^ ^ ^ ^ ^^ R (A 1 A 2 A 3 A 4 ) 2 7 6 1 6 7 6 0 2 7 5 1 2 7 5 7 5 2 1 4 2 2 1 5 7 0 1 4 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 = This 0 makes entire left branch 0 These 0 s make this node 0 These 1 s and these 0 s make this 1 2 1 -level has the only 1-bit so the 1-count = 1*2 1 = 2 Processing Efficiencies? (prefixed leaf-sizes have been removed)

Database D Scan D C1C1 TID 12345 10010110 20001101 30011101 40001001 C2C2 Scan D C2C2 L3L3 P 1 2 //\\ 1010 P 2 3 //\\ 0111 P 3 3 //\\ 1110 P 4 1 //\\ 1000 P 5 3 //\\ 0111 Build Ptrees: Scan D L 1 ={1,2,3,5} P 1 ^P 2 1 //\\ 0010 P 1 ^P 3 2 //\\ 1010 P 1 ^P 5 1 //\\ 0010 P 2 ^P 3 2 //\\ 0110 P 2 ^P 5 3 //\\ 0111 P 3 ^P 5 2 //\\ 0110 L 2 ={13,23,25,35} P 1 ^P 2 ^P 3 1 //\\ 0010 P 1 ^P 3 ^P 5 1 //\\ 0010 P 2 ^P 3 ^P 5 2 //\\ 0110 L 3 ={235} L1L1 C3C3 L2L2 {123} pruned since {12} not large {135} pruned since {15} not Large Example ARM using uncompressed P-trees (note: I have placed the 1-count at the root of each Ptree)

L3L3 L1L1 L2L2 1-ItemSets don’t support Association Rules (They will have no antecedent or no consequent). Are there any Strong Rules supported by Large 2-ItemSets (at minconf=.75)? {1,3}conf{1}  {3} = supp{1,3}/supp{1} = 2/2 = 1 ≥.75 STRONG conf{3}  {1} = supp{1,3}/supp{3} = 2/3 =.67 <.75 {2,3}conf{2}  {3} = supp{2,3}/supp{2} = 2/3 =.67 <.75 conf{3}  {2} = supp{2,3}/supp{3} = 2/3 =.67 <.75 {2,5}conf{2}  {5} = supp{2,5}/supp{2} = 3/3 = 1 ≥.75 STRONG! conf{5}  {2} = supp{2,5}/supp{5} = 3/3 = 1 ≥.75 STRONG! {3,5}conf{3}  {5} = supp{3,5}/supp{3} = 2/3 =.67 <.75 conf{5}  {3} = supp{3,5}/supp{5} = 2/3 =.67 <.75 Are there any Strong Rules supported by Large 3-ItemSets? {2,3,5}conf{2,3}  {5} = supp{2,3,5}/supp{2,3} = 2/2 = 1 ≥.75 STRONG! conf{2,5}  {3} = supp{2,3,5}/supp{2,5} = 2/3 =.67 <.75 conf{3,5}  {2} = supp{2,3,5}/supp{3,5} = 2/3 =.67 <.75 No subset antecedent can yield a strong rule either (i.e., no need to check conf{2}  {3,5} or conf{5}  {2,3} since both denominators will be at least as large and therefore, both confidences will be at least as low. No need to check conf{3}  {2,5} or conf{5}  {2,3} DONE! 2-Itemsets do support ARs.

Ptree-ARM versus Apriori on aerial photo (RGB) data together with yeild data Scalability with support threshold 1320  1320 pixel TIFF- Yield dataset (total number of transactions is ~1,700,000). P-ARM compared to Horizontal Apriori (classical) and FP-growth (an improvement of it).  In P-ARM, we find all frequent itemsets, not just those containing Yield (for fairness)  Aerial TIFF images (R,G,B) with synchronized yield (Y). Scalability with number of transactions  Identical results  P-ARM is more scalable for lower support thresholds.  P-ARM algorithm is more scalable to large spatial datasets.

P-ARM versus FP-growth (see literature for definition) Scalability with support threshold 17,424,000 pixels (transactions) Scalability with number of trans  FP-growth = efficient, tree-based frequent pattern mining method (details later)  For a dataset of 100K bytes, FP-growth runs very fast. But for images of large size, P-ARM achieves better performance.  P-ARM achieves better performance in the case of low support threshold.

Other methods (other than FP-growth) to Improve Apriori’s Efficiency (see the literature or the html notes 10datamining.html in Other Materials for more detail) Hash-based itemset counting: A k-itemset whose corresponding hashing bucket count is below the threshold cannot be frequent Transaction reduction: A transaction that does not contain any frequent k-itemset is useless in subsequent scans Partitioning: Any itemset that is potentially frequent in DB must be frequent in at least one of the partitions of DB Sampling: mining on a subset of given data, lower support threshold + a method to determine the completeness Dynamic itemset counting: add new candidate itemsets only when all of their subsets are estimated to be frequent  The core of the Apriori algorithm: –Use only large (k – 1)-itemsets to generate candidate large k-itemsets –Use database scan and pattern matching to collect counts for the candidate itemsets  The bottleneck of Apriori: candidate generation 1. Huge candidate sets: 10 4 large 1-itemset may generate 10 7 candidate 2-itemsets To discover large pattern of size 100, eg, {a 1 …a 100 }, we need to generate 2 100  10 30 candidates. 2. Multiple scans of database: (Needs (n +1 ) scans, n = length of the longest pattern)

3 steps: Build a Model of the TDS feature-to-class relationship, Test that Model, Use the Model (to predict the most likely class of each unclassified sample). Note: other names for this process: regression analysis, case-based reasoning,...) Other Typical Applications: –Targeted Product Marketing (the so-called classsical Business Intelligence problem) –Medical Diagnosis (the so-called Computer Aided Diagnosis or CAD) Nearest Neighbor Classifiers (NNCs) use a portion of the TDS as the model (neighboring tuples vote) –finding the neighbor set is much faster than building other models but it must be done anew for each unclasified sample. (NNC is called a lazy classifier because it get's lazy and doesn't take the time to build a concise model of the relationship between feature tuples and class labels ahead of time). Eager Classifiers (~all other classifiers) build 1 concise model once and for all - then use it for all unclassified samples. The model building can be very costly but that cost can be amortized over all the classifications of a large number of unclassified samples (e.g., all RGB points in a field). Classification Unclassified sample Predicted class of unclassified sample Using a Training Data Set (TDS) in which each feature tuple is already classified (has a class value attached to it in the class column, called its class label.), 1. Build a model of the TDS (called the TRAINING PHASE). 2. Use that model to classify unclassified feature tuples (unclassified samples). E.g., TDS = last year's aerial image of a crop field (feature columns are R,G,B columns together with last year's crop yeilds attached in a class column, e.g., class values={Hi, Med, Lo} yeild. Unclassified samples are the RGB tuples from this year's aerial image 3. Predict the class of each unclassified tuple (in the e.g.,: predict yeild for each point in the field.) OUTPUT conveyor INPUT hopper 1. Build a MODEL of the feature_tuple  class relationship tuple# feature 1 f 2 f 3... f n class tuple 1 value 1,1 val 2,1 val 3,1 val n,1 class 1... tuple m value 1,m val 2,m val 3,m val n,m class m

Unclassified sample herb, professor, 3 Unclassified sample clyde, professor,8 yes Unclassified sample tim, assoc. professor, 4 yes no Training Data Eager Classifiers TRAINING PHASE Classification Algorithm (creates the Classifier or Model during training phase) IF rank = ‘professor’ OR years > 6 THEN tenured = ‘yes’ e.g., Model (as a rule set) INPUT hopper OUTPUT conveyor CLASSIFICATION or USE PHASE

Classifier Testing Data NAMERANKYEARSTENURED TomAssistant Prof2no MerlisaAssociate Prof7no GeorgeAssociate Prof5yes JosephAssistant Prof7no % correct classifications? Test Process (2): Usually some of the Training Tuples are set aside as a Test Set and after a model is constructed, the Test Tuples are run through the Model. The Model is acceptable if, e.g., the % correct > 60%. If not, the Model is rejected (never used). Correct=3 Incorrect=1 75% Since 75% is above the acceptability threshold, accept the model!

Classification by Decision Tree Induction Decision tree (instead of a simple case statement of rules, the rules are prioritized into a tree) –Each Internal node denotes a test or rule on an attribute (test attribute for that node) –Each Branch represents an outcome of the test (value of the test attribute) –Leaf nodes represent class label decisions (plurality leaf class is predicted class) Decision tree model development consists of two phases –Tree construction At start, all the training examples are at the root Partition examples recursively based on selected attributes –Tree pruning Identify and remove branches that reflect noise or outliers Decision tree use: Classifying unclassified samples by filtering them down the decision tree to their proper leaf, than predict the plurality class of that leaf (often only one, depending upon the stopping condition of the construction phase)

Algorithm for Decision Tree Induction Basic ID3 algorithm (a simple greedy top-down algorithm) –At start, the current node is the root and all the training tuples are at the root –Repeat, down each branch, until the stopping condition is true At current node, choose a decision attribute (e.g., one with largest information gain). Each value for that decision attribute is associated with a link to the next level down and that value is used as the selection criterion of that link. Each new level produces a partition of the parent training subset based on the selection value assigned to its link. –stopping conditions: When all samples for a given node belong to the same class When there are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf When there are no samples left

Bayesian Classification (eager: Model is based on conditional probabilities. Prediction is done by taking the highest conditionally probable class) A Bayesian classifier is a statistical classifier, which is based on following theorem known as Bayes theorem: Bayes theorem: Let X be a data sample whose class label is unknown. Let H be the hypothesis that X belongs to class, H. P(H|X) is the conditional probability of H given X. P(H) is prob of H, then P(H|X) = P(X|H)P(H)/P(X)

Naïve Bayesian Classification Given training set, R(f 1..f n, C) where C={C 1..C m } is the class label attribute. A Naive Bayesian Classifier will predict the class of unknown data sample, X=(x 1..x n ), to be the class, C j having the highest conditional probability, conditioned on X. That is it will predict the class to be C j iff (a tie handling algorithm may be required). P(C j |X) ≥ P(C i |X), i  j. From the Bayes theorem; P(C j |X) = P(X|C j )P(C j )/P(X) –P(X) is constant for all classes so we need only maximize P(X|C j )P(C j ): –P(C i )s are known. –To reduce the computational complexity of calculating all P(X|C j )s, the naive assumption is to assume class conditional independence: P(X|Ci) is the product of the P(X i |C i )s.

Neural Network Classificaton A Neural Network is trained to make the prediction Advantages –prediction accuracy is generally high –it is generally robust (works when training examples contain errors) –output may be discrete, real-valued, or a vector of several discrete or real-valued attributes –It provides fast classification of unclassified samples. Criticism –It is difficult to understand the learned function (involves complex and almost magic weight adjustments.) –It makes it difficult to incorporate domain knowledge –long training time (for large training sets, it is prohibitive!)

The input feature vector x=(x 0..x n ) is mapped into variable y by means of the scalar product and a nonlinear function mapping, f (called the damping function). and a bias function,  kk - f weighted sum Input vector x output y Activation function weight vector w  w0w0 w1w1 wnwn x0x0 x1x1 xnxn A Neuron

The ultimate objective of training –obtain a set of weights that makes almost all the tuples in the training data classify correctly (usually using a time consuming "back propagation" procedure which is based, ultimately on Neuton's method. See literature of Other materials - 10datamining.html for examples and alternate training techniques). Steps –Initialize weights with random values –Feed the input tuples into the network –For each unit Compute the net input to the unit as a linear combination of all the inputs to the unit Compute the output value using the activation function Compute the error Update the weights and the bias Neural Network Training

Output nodes Input nodes Hidden nodes Output vector Input vector: x i w ij Neural Multi-Layer Perceptron

These next 3 slides treat the concept of Distance it great detail. You may feel you don't need this much detail - if so, skip what you feel you don't need. For Nearest Neighbor Classification, a distance is needed (to make sense of "nearest". Other classifiers also use distance.) A distance is a function, d, applied to two n-dimensional points X and Y, is such that d(X, Y) is positive definite: if (X  Y), d(X, Y) > 0; if (X = Y), d(X, Y) = 0 d(X, Y) is symmetric: d(X, Y) = d(Y, X) d(X, Y) holds triangle inequality: d(X, Y) + d(Y, Z)  d(X, Z) n xxxxX,,,, 321 …  n yyyyY,,,, 321 …  Minkowski distance or L p distance, Manhattan distance,(P = 1) Euclidian distance,(P = 2) Max distance, (P =  ) Canberra distance Squared cord distance Squared chi-squared distance

An Example Manhattan, d 1 (X,Y) = XZ+ ZY = 4+3 = 7 X (2,1) Y (6,4) Z d 1  d 2  d  always In fact, for any positive integer p, A two-dimensional space: Max, d  (X,Y) = Max(XZ, ZY) = XZ = 4 Euclidian, d 2 (X,Y) = XY = 5

Neighborhoods of a Point A Neighborhood (disk neighborhood) of a point, T, is a set of points, S,  : X  S iff d(T, X)  r 2r2r T X Manhattan If X is a point on the boundary, d(T, X) = r T 2r2r X Euclidian 2r2r T X Max

Classical k-Nearest Neighbor Classification Select a suitable value for k (how many Training Data Set (TDS) neighbors do you want to vote as to the best predicted class for the unclassified feature sample? ) Determine a suitable distance metric (to give meaning to neighbor) Find the k nearest training set points to the unclassified sample. Let them vote (tally up the counts of TDS neighbors that for each class.) Predict the highest class vote (plurality class) from among the k-nearest neighbor set.

Closed-KNN T Example assume 2 features (one in the x-direction and one in the y T is the unclassified sample. using k = 3, find the three nearest neighbor, KNN arbitrarily select one point from the boundary line shown Closed-KNN includes all points on the boundary Closed-KNN yields higher classification accuracy than traditional KNN (thesis of MD Maleq Khan, NDSU, 2001). The P-tree method always produce closed neighborhoods (and is faster!)

k-Nearest Neighbor (kNN) Classification and Closed-k-Nearest Neighbor (CkNN) Classification 1) Select a suitable value for k 2) Determine a suitable distance or similarity notion. 3) Find the k nearest neighbor set [closed] of the unclassified sample. 4) Find the plurality class in the nearest neighbor set. 5) Assign the plurality class as the predicted class of the sample T That's 1 ! CkNN yields higher classification accuracy than traditional kNN. At what additional cost? Actually, at negative cost (faster and more accurate!!) T is the unclassified sample. Use Euclidean distance. k = 3: Find 3 closest neighbors. Move out from T until ≥ 3 neighbors kNN arbitrarily select one point from that boundary line as 3 rd nearest neighbor, whereas, CkNN includes all points on that boundary line. That's 2 ! That's more than 3 !

The slides numbered 28 through 93 give great detail on the relative performance of kNN and CkNN, on the use of other distance functions and some exampels, etc. There may be more detail on these issue that you want/need. If so, just scan for what you are most interested in or just skip ahead to slide 94 on CLUSTERING. Experimented on two sets of (Arial) Remotely Sensed Images of Best Management Plot (BMP) of Oakes Irrigation Test Area (OITA), ND Data contains 6 bands: Red, Green, Blue reflectance values, Soil Moisture, Nitrate, and Yield (class label). Band values ranges from 0 to 255 (8 bits) Considering 8 classes or levels of yield values

Performance – Accuracy (3 horizontal methods in middle, 3 vertical methods (the 2 most accurate and the least accurate) 1997 Dataset: 40 45 50 55 60 65 70 75 80 256102440961638465536262144 Training Set Size (no. of pixels) Accuracy (%) kNN-ManhattankNN-Euclidian kNN-MaxkNN using HOBbit distance P-tree Closed-KNN0-maxClosed-kNN using HOBbit distance

Performance – Accuracy (3 horizontal methods in middle, 3 vertical methods (the 2 most accurate and the least accurate) 1998 Dataset: 20 40 45 50 55 60 65 256102440961638465536262144 Training Set Size (no of pixels) Accuracy (%) kNN-ManhattankNN-Euclidian kNN-MaxkNN using HOBbit distance P-tree Closed-KNN-maxClosed-kNN using HOBbit distance

Performance – Speed (3 horizontal methods in middle, 3 vertical methods (the 2 fastest (the same 2) and the slowest) 1997 Dataset: both axis in logarithmic scale 0.0001 0.001 0.01 0.1 1 256102440961638465536262144 Training Set Size (no. of pixels) Per Sample Classification time (sec) kNN-ManhattankNN-Euclidian kNN-MaxkNN using HOBbit distance P-tree Closed-KNN-maxClosed-kNN using HOBbit dist Hint: NEVER use a log scale to show a WIN!!!

Performance – Speed (3 horizontal methods in middle, 3 vertical methods (the 2 fastest (the same 2) and the slowest) 1998 Dataset : both axis in logarithmic scale 0.0001 0.001 0.01 0.1 1 256102440961638465536262144 Training Set Size (no. of pixels) Per Sample Classification Time (sec) kNN-ManhattankNN-Euclidian kNN-MaxkNN using HOBbit distance P-tree Closed-kNN-maxClosed-kNN using HOBbit dist Win-Win situation !! (almost never happens) P-tree CkNN and CkNN-H are more accurate and much faster. kNN-H is not recommended because it is slower and less accurate (because it doesn't use Closed nbr sets and it requires another step to get rid of ties (why do it?). Horizontal kNNs are not recommended because they are less accurate and slower!

Key a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 8 a 9 a 10 =C a 11 a 12 a 13 a 14 a 15 a 16 a 17 a 18 a 19 a 20 t12 1 0 1 0 0 0 1 1 0 1 0 1 1 0 1 1 0 0 0 1 t13 1 0 1 0 0 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 t15 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 0 1 1 0 0 t16 1 0 1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 0 1 0 t21 0 1 1 0 1 1 0 0 0 1 1 0 1 0 0 0 1 1 0 1 t27 0 1 1 0 1 1 0 0 0 1 0 0 1 1 0 0 1 1 0 0 t31 0 1 0 0 1 0 0 0 1 1 1 0 1 0 0 0 1 1 0 1 t32 0 1 0 0 1 0 0 0 1 1 0 1 1 0 1 1 0 0 0 1 t33 0 1 0 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 1 1 t35 0 1 0 0 1 0 0 0 1 1 0 1 0 1 0 0 1 1 0 0 t51 0 1 0 1 0 0 1 1 0 0 1 0 1 0 0 0 1 1 0 1 t53 0 1 0 1 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 t55 0 1 0 1 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 t57 0 1 0 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 t61 1 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 1 0 1 t72 0 0 1 1 0 0 1 1 0 0 0 1 1 0 1 1 0 0 0 1 t75 0 0 1 1 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 t12 0 0 1 0 1 1 0 2 t13 0 0 1 0 1 0 0 1 t15 0 0 1 0 1 0 1 2 a 5 a 6 a 10 =C a 11 a 12 a 13 a 14 distance The 3 nearest neighbors 0 0 0 d=2, don’t replace 0 0 0 d=4, don’t replace 0 0 0 d=4, don’t replace 0 0 0 d=3, don’t replace 0 0 0 d=3, don’t replace 0 0 0 d=2, don’t replace 0 0 0 d=3, don’t replace 0 0 0 d=2, don’t replace 0 0 0 d=1, replace t53 0 0 0 0 1 0 0 1 0 0 0 d=2, don’t replace 0 0 0 d=2, don’t replace 0 0 0 d=3, don’t replace 0 0 0 d=2, don’t replace 0 0 0 d=2, don’t replace 0 1 C=1 wins! WALK THRU : 3NN CLASSIFICATION of an unclassified sample, a=(a 5 a 6 a 11 a 12 a 13 a 14 )=(000000). HORIZONTAL APPROACH ( relevant attributes are Note only 1 of many training tuple at a distance=2 from the sample got to vote. We didn’t know that distance=2 was going to be the vote cutoff until the end of the 1 st scan. Finding the other distance=2 voters (Closed 3NN set or C3NN) requires another scan. a 5 a 6 a 11 a 12 a 13 a 14 )

Key a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 8 a 9 a 10 =C a 11 a 12 a 13 a 14 a 15 a 16 a 17 a 18 a 19 a 20 t12 1 0 1 0 0 0 1 1 0 1 0 1 1 0 1 1 0 0 0 1 t13 1 0 1 0 0 0 1 1 0 1 0 1 0 0 1 0 0 0 1 1 t15 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0 0 1 1 0 0 t16 1 0 1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 0 1 0 t21 0 1 1 0 1 1 0 0 0 1 1 0 1 0 0 0 1 1 0 1 t27 0 1 1 0 1 1 0 0 0 1 0 0 1 1 0 0 1 1 0 0 t31 0 1 0 0 1 0 0 0 1 1 1 0 1 0 0 0 1 1 0 1 t32 0 1 0 0 1 0 0 0 1 1 0 1 1 0 1 1 0 0 0 1 t33 0 1 0 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 1 1 t35 0 1 0 0 1 0 0 0 1 1 0 1 0 1 0 0 1 1 0 0 t51 0 1 0 1 0 0 1 1 0 0 1 0 1 0 0 0 1 1 0 1 t53 0 1 0 1 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 t55 0 1 0 1 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 t57 0 1 0 1 0 0 1 1 0 0 0 0 1 1 0 0 1 1 0 0 t61 1 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 1 0 1 t72 0 0 1 1 0 0 1 1 0 0 0 1 1 0 1 1 0 0 0 1 t75 0 0 1 1 0 0 1 1 0 0 0 1 0 1 0 0 1 1 0 0 WALK THRU of required 2 nd scan to find Closed 3NN set. Does it change vote? 0 0 0 d=2, include it also 0 0 0 d=4, don’t include 0 0 0 d=4, don’t include 0 0 0 d=3, don’t include 0 0 0 d=3, don’t include 0 0 0 d=2, include it also 0 0 0 d=3, don’t include 0 0 0 d=2, include it also 0 0 0 d=1, already voted 0 0 0 d=2, include it also 0 0 0 d=2, include it also 0 0 0 d=3, don’t replace 0 0 0 d=2, include it also 0 0 0 d=2, include it also 0 0 0 d=2, include it also 0 0 0 d=2, already voted 0 0 0 d=1, already voted 0 1 Vote after 1 st scan. YES! C=0 wins now! t12 0 0 1 0 1 1 0 2 t13 0 0 1 0 1 0 0 1 a 5 a 6 a 10 =C a 11 a 12 a 13 a 14 distance t53 0 0 0 0 1 0 0 1 Unclassified sample: 0 0 0 0 0 0 3NN set after 1 st scan

C00000000001111111C00000000001111111 C11111111110000000C11111111110000000 WALK THRU: Closed 3NNC using P-trees a 20 1 0 1 0 1 0 1 0 1 0 key t 12 t 13 t 15 t 16 t 21 t 27 t 31 t 32 t 33 t 35 t 51 t 53 t 55 t 57 t 61 t 72 t 75 a111110000000000100a111110000000000100 a200001111111111000a200001111111111000 a311111100000000111a311111100000000111 a400000000001111011a400000000001111011 a500001111110000100a500001111110000100 a600001100000000000a600001100000000000 a711110000001111011a711110000001111011 a811110000001111011a811110000001111011 a900000011110000100a900000011110000100 C11111111110000000C11111111110000000 a 11 0 1 0 1 0 1 0 1 0 a 12 1 0 1 0 1 0 1 a 13 1 0 1 0 1 0 1 0 a 14 0 1 0 1 0 1 0 1 0 1 a 15 1 0 1 0 1 0 1 0 1 0 a 16 1 0 1 0 1 0 a 17 0 1 0 1 0 1 0 1 0 1 a 18 0 1 0 1 0 1 0 1 0 1 a 19 0 1 0 1 0 1 0 1 0 Ps00000000000000000Ps00000000000000000 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 No neighbors at distance=0 First let all training points at distance=0 vote, then distance=1, then distance=2,... until  3 For distance=0 (exact matches) constructing the P-tree, P s then AND with P C and P C’ to compute the vote. (black denotes complement, red denotes uncomplemented

C00000000001111111C00000000001111111 C11111111110000000C11111111110000000 a 20 1 0 1 0 1 0 1 0 1 0 key t 12 t 13 t 15 t 16 t 21 t 27 t 31 t 32 t 33 t 35 t 51 t 53 t 55 t 57 t 61 t 72 t 75 a111110000000000100a111110000000000100 a200001111111111000a200001111111111000 a311111100000000111a311111100000000111 a400000000001111011a400000000001111011 a500001111110000100a500001111110000100 a600001100000000000a600001100000000000 a711110000001111011a711110000001111011 a811110000001111011a811110000001111011 a900000011110000100a900000011110000100 a 10 =C 1 0 a 11 0 1 0 1 0 1 0 1 0 a 12 1 0 1 0 1 0 1 a 13 1 0 1 0 1 0 1 0 a 14 0 1 0 1 0 1 0 1 0 1 a 15 1 0 1 0 1 0 1 0 1 0 a 16 1 0 1 0 1 0 a 17 0 1 0 1 0 1 0 1 0 1 a 18 0 1 0 1 0 1 0 1 0 1 a 19 0 1 0 1 0 1 0 1 0 P D(s,1) 0 1 0 1 0 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a500001111110000100a500001111110000100 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a600001100000000000a6000011000000000000 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 0 1 0 1 0 1 0 1 0 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 121 1 0 1 0 1 0 1 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 1 0 0 1 0 1 0 1 0 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 140 1 0 1 0 1 0 1 0 1 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 Construct Ptree, P S(s,1) = OR P i = P |s i -t i |=1; |s j -t j |=0, j  i = OR P S(s i,1)   S(s j,0) OR P5P5 P6P6 P 11 P 12 P 13 P 14 j  {5,6,11,12,13,14}-{i} 0 1 i= 5,6,11,12,13,14 WALK THRU: C3NNC distance=1 nbrs:

key t 12 t 13 t 15 t 16 t 21 t 27 t 31 t 32 t 33 t 35 t 51 t 53 t 55 t 57 t 61 t 72 t 75 a500001111110000100a500001111110000100 a600001100000000000a600001100000000000 a 10 C 1 0 a 11 0 1 0 1 0 1 0 1 0 a 12 1 0 1 0 1 0 1 a 13 1 0 1 0 1 0 1 0 a 14 0 1 0 1 0 1 0 1 0 1 0 1 OR{all double-dim interval-Ptrees}; P D(s,2) = OR P i,j P i,j = P S(s i,1)  S(s j,1)  S(s k,0) k  {5,6,11,12,13,14}-{i,j} i,j  {5,6,11,12,13,14} a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a600001100000000000a6000011000000000000 a500001111110000100a500001111110000100 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 0 1 0 1 0 1 0 1 0 a611110011111111111a611110011111111111 a500001111110000100a500001111110000100 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 121 1 0 1 0 1 0 1 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a500001111110000100a500001111110000100 a 14 1 0 1 0 1 0 1 0 1 0 a 13 1 0 0 1 0 1 0 1 0 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a500001111110000100a500001111110000100 a 140 1 0 1 0 1 0 1 0 1 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a500001111110000100a500001111110000100 P 5,6 P 5,11 P 5,12 P 5,13 P 5,14 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 0 1 0 1 0 1 0 1 0 a600001100000000000a6000011000000000000 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 1 0 1 0 1 0 1 a 11 1 0 1 0 1 0 1 0 1 a600001100000000000a6000011000000000000 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a600001100000000000a6000011000000000000 a511110000001111011a511110000001111011 a 14 0 1 0 1 0 1 0 1 0 1 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a600001100000000000a6000011000000000000 a511110000001111011a511110000001111011 P 6,11 P 6,12 P 6,13 P 6,14 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 1 0 1 0 1 0 1 a 11 0 1 0 1 0 1 0 1 0 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 1 0 1 0 1 0 1 0 a 12 0 1 0 1 0 1 0 a 11 0 1 0 1 0 1 0 1 0 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 0 1 0 1 0 1 0 1 0 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 P 11,12 P 11,13 P 11,14 a 14 1 0 1 0 1 0 1 0 1 0 a 13 1 0 0 1 0 1 0 1 0 a 12 1 0 1 0 1 0 1 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 a 140 1 0 1 0 1 0 1 0 1 a 13 0 1 0 1 0 1 0 1 a 12 1 0 1 0 1 0 1 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 P 12,13 P 12,14 We now have 3 nearest nbrs. We could quite and declare C=1 winner? a 140 1 0 1 0 1 0 1 0 1 a 13 1 0 1 0 1 0 1 0 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 P 13,14 We now have the C3NN set and we can declare C=0 the winner! WALK THRU: C3NNC distance=2 nbrs:

In the previous example, there were no exact matches (dis=0 neighbors or similarity=6 neighbors) for the sample. There were two neighbors were found at a distance of 1 (dis=1 or sim=5) and nine dis=2, sim=4 neighbors. All 11 neighbors got an equal votes even though the two sim=5 are much closer neighbors than the nine sim=4. Also processing for the 9 is costly. A better approach would be to weight each vote by the similarity of the voter to the sample (We will use a vote weight function which is linear in the similarity (admittedly, a better choice would be a function which is Gaussian in the similarity, but, so far, it has been too hard to compute). As long as we are weighting votes by similarity, we might as well also weight attributes by relevance also (assuming some attributes are more relevant neighbors than others. e.g., the relevance weight of a feature attribute could be the correlation of that attribute to the class label). P-trees accommodate this method very well (in fact, a variation on this theme won the KDD-cup competition in 02 ( http://www.biostat.wisc.edu/~craven/kddcup/ )

Association of Computing Machinery KDD-Cup-02 NDSU Team

Closed Manhattan Nearest Neighbor Classifier (uses a linear fctn of Manhattan similarity) Sample is (000000), attribute weights of relevant attributes are their subscripts) key t 12 t 13 t 15 t 16 t 21 t 27 t 31 t 32 t 33 t 35 t 51 t 53 t 55 t 57 t 61 t 72 t 75 a500001111110000100a500001111110000100 a600001100000000000a600001100000000000 C11111111110000000C11111111110000000 a 11 0 1 0 1 0 1 0 1 0 a 12 1 0 1 0 1 0 1 a 13 1 0 1 0 1 0 1 0 a 14 0 1 0 1 0 1 0 1 0 1 a 14 1 0 1 0 1 0 1 0 1 0 a 13 0 1 0 1 0 1 0 1 a 12 0 1 0 1 0 1 0 a 11 1 0 1 0 1 0 1 0 1 a611110011111111111a611110011111111111 a511110000001111011a511110000001111011 black is attribute complement, red is uncomplemented. The vote is even simpler than the "equal" vote case. We just note that all tuples vote in accordance with their weighted similarity (if the a i values differs form that of (000000) then the vote contribution is the subscript of that attribute, else zero). Thus, we can just add up the root counts of each relevant attribute weighted by their subscript. Class=1 root counts: rc(P C ^P a 5 )=4 C=1 vote is: 343 =4*5 + 8*6 + 7*11 + 4*12 + 4*13 + 7*14 rc(P C ^P a 6 )=8rc(P C ^P a 11 )=7 rc(P C ^P a 12 )=4rc(P C ^P a 13 )=4rc(P C ^P a 14 )=7 C=1 vote is: 343 Similarly, C=0 vote is: 258= 6*5 + 7*6 + 5*11 + 3*12 + 3*13 + 4*14

We note that the Closed Manhattan NN Classifier uses an influence function which is pyramidal It would be much better to use a Gaussian influence function but it is much harder to implement. One generalization of this method to the case of integer values rather than Boolean, would be to weight each bit position in a more Gaussian shape (i.e., weight the bit positions, b, b-1,..., 0 (high order to low order) using Gaussian weights. By so doing, at least within each attribute, influences are Gaussian. We can call this method, Closed Manhattan Gaussian NN Classification. Testing the performance of either CM NNC or CMG NNC would make a great paper for this course (thesis?). Improving it in some way would make an even better paper (thesis).

Machine Learning is based on Near Neighbor Set(s), NNS. Clustering, even density based, identifies near neighbor cores 1 st (round NNS s,  about a center). Classification is continuity based and Near Neighbor Sets (NNS) are the central concept in continuity  >0  >0  : d(x,a)<   d(f(x),f(a))<  where f assigns a class to a feature vector, or   -NNS of f(a),  a  -NNS of a in its pre-image. f(Dom) categorical  >0  : d(x,a)<  f(x)=f(a) Data Mining can be broken down into 2 areas, Machine Learning and Assoc. Rule Mining Machine Learning can be broken down into 2 areas, Clustering and Classification. Clustering can be broken down into 2 types, Isotropic (round clusters) and Density-based Classification can be broken down into to types, Model-based and Neighbor-based Review of slide 2 (with additions): Database analysis can be broken down into 2 areas, Querying and Data Mining. Caution: For classification, boundary analysis may be needed also to see the class (done by projecting?). 1234 Finding NNS in lower a dimension may still the 1st step. Eg, 12345678 are all  from a 5 a 6 (unclassified sample); 1234 are red-class, 5678 are blue-class. 7 8 Any  that gives us a vote gives us a tie vote (0-to-0 then 4-to-4). But projecting onto the vertical subspace, then taking  /2 we see that  /2 about a contains only blue class (5,6) votes. ****** Using horizontal data, NNS derivation requires ≥1 scan (O(n)). L  ε-NNS can be derived using vertical-data in O(log 2 n) (but Euclidean disks are preferred). (Euclidean and L  coincide in Binary data sets). Solution (next slide): Circumscribe desired Euclidean-  NNS with a few intersections of functional-contours, (f -1 ([b,c] ) sets, until the intersection is scannable, then scan it for Euclidean-  -nbrhd membership. Advantage: intersection can be determined before scanning - create and AND functional contour P-trees.

and  S  Y, contour(f,S) = f -1 (S) Equivalently, contour(A f,S) = SELECT A 1..A n FROM R* WHERE x.A f  S. Graphically: A Weather map, f = barometric pressure or temperature, {S i }=equi-width partion of Reals. f = local density (eg, OPTICS: f = reachability distance, {S k } = partition produced by intersection points of {graph(f), plotted wrt to some walk of R} and a horizontal threshold line. A grid is the intersection of dimension projection contour partitions (next slide for more defintions). A Class is a contour under f:R  ClassAttr wrt the partition, {C i } of ClassAttr (where {C i } are the classes). An L   -disk about a is the intersection of all  -dimension_projection contours containing a. Functional Contours:  function, f:R(A 1..A n )  Y A 1 A 2 A n A f x 1 x 2 x n f(x 1..x n ) :... R* Equivalently,  derived attribute, A f, with DOMAIN(A f )  Y (The equivalence is x.A f = f(x)  x  R ) A 1 A 2 A n x 1 x 2 x n : :... f R Y xx  partition, {S i } of Y, the contour set, {f -1 (S i )}, is a partition of R (clustering of R): f(x) If S={a}, Isobar(f,a)= contour(f,{a}) S A 1 A 2 A n : :... f R R Y S graph(f) = {(x, f(x)) | x  R} f-contour(S) Y S1S2S3S1S2S3

f:R  Y,  partition S={S k } of Y, {f -1 (S k )}= S,f-grid of R ( grid cells=contours) If Y=Reals, the j.lo f-grid is produced by agglomerating over the j lo bits of Y,  fixed (b-j) hi bit pattern. The j lo bits walk [isobars of] cells. The b-j hi bits identify cells. ( lo=extension / hi=intention) Let b-1,...,0 be the b bit positions of Y. The j.lo f-grid is the partition of R generated by f and S = {S b-1,...,b-j | S b-1,...,b-j = [(b-1)(b-2)...(b-j)0..0, (b-1)(b-2)...(b-j)1..1)} partition of Y=Reals. If F={f h }, the j.lo F-grid is the intersection partition of the j.lo f h -grids (intersection of partitions). The canonical j.lo grid is the j.lo  -grid;  ={  d :R  R[A d ] |  d = d th coordinate projection} j-hi gridding is similar ( the b-j lo bits walk cell contents / j hi bits identify cells). If the horizontal and vertical dimensions have bitwidths 3 and 2 respectively: 000 001 010 011 100 101 110 111 11 10 01 00 2.lo grid 1.hi grid Want square cells or a square pattern? 000 001 010 011 100 101 110 111 11 10 01 00 GRIDs

2.lo grid 1.hi grid 111 110 101 100 j.lo and j.hi gridding continued The horizontal_bitwidth = vertical_bitwidth = b iff j.lo grid = (b-j).hi grid e.g., for hb=vb=b=3 and j=2: 000 001 010 011 100 101 110 111 011 010 001 000 111 110 101 100 000 001 010 011 100 101 110 111 011 010 001 000

Similarity NearNeighborSets (SNNS) Given similarity s:R  R  PartiallyOrderedSet (eg, Reals) ( i.e., s(x,y)=s(y,x) and s(x,x)  s(x,y)  x,y  R ) and given any C  R The Cardinal disk, skins and rings are (PartiallyOrderedSet = Reals) disk(C,r)  {x  R | s(x,C)  r} also = functional contour, f -1 ([r,  ), where f(x)=s C (x)=s(x,C) skin(C,r)  disk(C,r) - C ring(C,r 2,r 1 )  disk(C,r 2 )-disk(C,r 1 )  skin(C,r 2 )-skin(C,r 1 ) also = functional contour, s C -1 (r 1,r 2 ] For C = {a} a r1r1 r2r2 C r1r1 r2r2 The Ordinal disks, skins and rings are: disk(C,k)  C  : |disk(C,k)  C'|=k and s(x,C)  s(y,C)  x  disk(C,k), y  disk(C,k) skin(C,k) = disk(C,k)-C ( skin comes from s k immediate neighbors and is a kNNS of C.) ring(C,k) = cskin(C,k)-cskin(C,k-1) closeddisk(C,k)  all disk(C,k); closedskin(C,k)  all skin(C,k) L  skins: skin  (a,k) = {x |  d, x d is one of the k-NNs of a d } (a local normalizer?) A distance, d, generates a similarity many ways, e.g., s(x,y)=1/(1+d(x,y)): (or if the relationship various by location, s(x,y)=  (x,y)/(1+d(x,y)) s d 1 Note: closeddisk(C,r) is redundant, since all r-disks are closed and closeddisk(C,k) = disk(C,s(C,y)) where y = k th NN of C s(x,y)= a*e -b*d(x,y) 2 : s d a 0 : d(x,y)>  s(x,y)= ae -bd(x,y) 2 -ae -b  2 : d(x,y)  (vote weighting IS a similarity assignment, so the similarity-to-distance graph IS a vote weighting for classification) s d a-ae -b  2 

Ptrees Vertical, compressed, lossless structures that facilitates fast horizontal AND-processing Jury is still out on parallelization, vertical (by relation) or horizontal (by tree node) or some combination? Horizontal parallelization is pretty, but network multicast overhead is huge Use active networking? Clusters of Playstations?... Formally, P-trees are be defined as any of the following; Partition-tree: Tree of nested partitions (a partition P(R)={C 1..C n }; each component is partitioned by P(C i )={C i,1..C i,n i } i=1..n; each component is partitioned by P(C i,j )={C i,j 1..C i,j n ij }... ) Partition tree R / … \ C 1 … C n / … \ … / … \ C 11 …C 1,n 1 C n1 …C n,n n... Predicate-tree: For a predicate on the leaf-nodes of a partition-tree (also induces predicates on i-nodes using quantifiers) Predicate-tree nodes can be truth-values (Boolean P-tree); can be quantified existentially (1 or a threshold %) or universally; Predicate-tree nodes can count # of true leaf children of that component (Count P-tree) Purity-tree: universally quantified Boolean-Predicate-tree (e.g., if the predicate is, Pure1-tree or P1tree) A 1-bit at a node iff corresponding component is pure1 (universally quantified) There are many other useful predicates, e.g., NonPure0-trees; But we will focus on P1trees. All Ptrees shown so far were: 1-dimensional (recursively partition by halving bit files), but they can be; 2-D (recursively quartering) (e.g., used for 2-D images); 3-D (recursively eighth-ing), …; Or based on purity runs or LZW-runs or … Further observations about Ptrees: Partition-tree: have set nodes Predicate-tree: have either Boolean nodes (Boolean P-tree) or count nodes (Count P-tree) Purity-tree: being universally quantified Boolean-Predicate-tree have Boolean nodes (since the count is always the “full” count of leaves, expressing Purity-trees as count-trees is redundant. Partition-tree can sliced at a given level if each partition at that level is labeled with very same label set (e.g., Month partition of years). A Partition-tree can be generalized to a Set-graph when the siblings of a node do not form a partition.

0 P f,S = equivalently, the existential R*-bit map of predicate, R*.A f  S The Compressed Ptree, s P f,S is the compression of 0 P f,S with equi-width leaf size, s, as follows 1. Choose a walk of R (converts 0 P f,S from bit map to bit vector) 2. Equi-width partition 0 P f,S with segment size, s (s=leafsize, the last segment can be short) 3. Eliminate and mask to 0, all pure-zero segments(call mask, NotPure0 Mask or EM) 4. Eliminate and mask to 1, all pure-one segments(call mask, Pure1 Mask or UM) Compressing each leaf of s P f,S with leafsize=s 2 gives: s 1,s 2 P f,S Recursivly, s 1, s 2, s 3 P f,S s 1, s 2, s 3, s 4 P f,S... (builds an EM and a UM tree) BASIC P-trees If A i Real or Binary and f i,j (x)  j th bit of x i ; { (*) P f i,j,{1}  (*) P i,j } j=b..0 are basic (*) P-trees of A i, *= s 1..s k If A i Categorical and f i,a (x)=1 if x i =a, else 0; { (*) P f i,a,{1}  (*) P i,a } a  R[A i ] are basic (*) P-trees of A i Notes: The UM masks (e.g., of 2 k,...,2 0 P i,j, with k=roof(log 2 |R| ), form a (binary) tree. Whenever the EM bit is 1, that entire subtree can be eliminated (since it represents a pure0 segment), then a 0-node at level-k (lowest level = level-0) with no sub-tree indicates a 2 k -run of zeros. In this construction, the UM tree is redundant. We call these EM trees the basic binary P-trees. The next slide shows a top-down (easy to understand) construction of and the following slide is a (much more efficient) bottom up construction of the same. We have suppressed the leafsize prefix. ( EM=existential aggregation UM=universal aggregation) The partitions used to create P-trees can come from functional contours (Note: there is a natural duality between partitions and functions, namely a partition creates a function from the space of points partitioned to the set of partition components and a function creates the pre-image partition of its domain). In Functional Contour terms (i.e., f -1 (S) where f:R(A 1..A n )  Y, S  Y), the uncompressed Ptree or uncompressed Predicate-tree 0 P f, S = bitmap of set containment-predicate, 0 P f,S (x)=true iff x  f -1 (S)

Example functionals: Total Variation (TV) functionals TV(a) =  x  R (x-a)o(x-a) If we use d for a index variable over the dimensions, =  x  R  d=1..n (x d 2 - 2a d x d + a d 2 ) i,j,k bit slices indexes =  x  R  d=1..n (  k 2 k x dk ) 2 - 2  x  R  d=1..n a d (  k 2 k x dk ) + |R||a| 2 =  x  d (  i 2 i x di )(  j 2 j x dj ) - 2  x  R  d=1..n a d (  k 2 k x dk ) + |R||a| 2 =  x  d  i,j 2 i+j x di x dj - 2  x,d,k 2 k a d x dk + |R||a| 2 =  x,d,i,j 2 i+j x di x dj - |R||a| 2 2  d a d (  x,k 2 k x dk ) + TV(a) =  i,j,d 2 i+j |P di^dj | - |R||a| 2  k 2 k+1  d a d |P dk | + Note that the first term does not depend upon a. Thus, the derived attribute, TV-TV(  ) (eliminate 1 st term) is much simpler to compute and has identical contours (just lowers the graph by TV(  ) ). We also find it useful to post-compose a log to reduce the number of bit slices. The resulting functional is called the High-Dimension-ready Total Variation or HDTV(a). =  x,d,i,j 2 i+j x di x dj +  d a d a d ) (equation 7) |R|( -2  d a d  d +=  x,d,i,j 2 i+j x di x dj - |R|  d a d a d 2|R|  d a d  d + - 2  d a d (  x (  k 2 k x dk ) ) - and  x x d = |R|μ d so, 2  d a d (  x (x d ) )

The length of LNTV(a) depends only on the length of a- , so isobars are hyper-circles centered at  The graph of LNTV is a log-shaped hyper-funnel: From equation 7, Normalized Total Variation, NTV(a)  TV(a)-TV(  ) =  d (a d a d -  d  d ) ) = |R| ( -2  d (a d  d -  d  d ) + TV(a) =  x,d,i,j 2 i+j x di x dj + |R| ( -2  d a d  d +  d a d a d ) +  d  d 2 )= |R|(  d a d 2 - 2  d  d a d LNTV(a) = ln( NTV(a) ) = ln( TV(a)-TV(  ) ) = ln|R| + ln|a-  | 2 = |R| |a-  | 2 go inward and outward along a-  by  to the points; inner point, b=  +(1-  /|a-  |)(a-  ) and outer point, c=  -(1+  /|a-  |)(a-  ).  -contour (radius  about a) a For an  -contour ring (radius  about a) Then take g(b) and g(c) as lower and upper endpoints of a vertical interval. Then we use EIN formulas on that interval to get a mask P-tree for the  -contour (which is a well-pruned superset of the  -neighborhood of a) b c g(b) g(c)  x1x1 x2x2 g(a)=LNTV(x) Thus there is a simpler function which gives us circular contours, the Log Normal TV function<

use circumscribing A  d -contour (Note: A  d is not a derived attribute at all, but just A d, so we already have its basic P-trees).   -contour (radius  about a) a LNTV(b) LNTV(c) b c As pre-processing, calculate basic P-trees for the LNTV derived attribute (or another hypercircular contour derived attribute). To classify a 1. Calculate b and c (Depend on a,  ) 2. Form mask P-tree for training pts with LNTV-values  [LNTV(b), LNTV(c)] 3. User that P-tree to prune out the candidate NNS. 4.If the count of candidates is small, proceed to scan and assign class votes using Gaussian vote function, else prune further using a dimension projections). contour of dimension projection f(a)=a 1 x1x1 x2x2 LNTV(x) If the LNTV circumscribing contour of a is still too populous, We can also note that LNTV can be further simplified (retaining same contours) using h(a)=|a-  |. Since we create the derived attribute by scanning the training set, why not just use this very simple function? Others leap to mind, e.g., h b (a)=|a-b| (Use voting function, G(x) = Gauss(|x-a|)-Gauss(  ), where Gauss(r) is (1/(std*  2  )e -(r-mean) 2 /2var (std, mean, var are wrt set distances from a of voters i.e., {r=|x-a|: x a voter} )

1 2 3 TV(x 15 )- TV(  ) 1 2 3 4 5 X Y TV- TV(  ) 4 5 TV(  )=TV(x 33 ) TV(x 15 ) 1 2 3 4 5 X Y TV 1 2 3 4 5 Graphs of functionals with hyper-circular contours  h b (a)=|a-b| b  h(a)=|a-  |  LNTV

Angular Variation functionals: e.g., AV(a)  ( 1/|a| )  x  R xoa d is an index over the dimensions, = (1/|a|)  x  R  d=1..n x d a d = (1/|a|)  d=1..n (  x x d ) a d COS  (and AV) has hyper-conic isobars center on  = |R|/|a|  d=1..n ((  x x d )/|R|) a d = |R|/|a|  d=1..n  d a d = ( |R|/|a| )  o a COS  (a)  AV(a)/(|  ||R|) =  oa/(|  ||a|) = cos(  a  ) COS  and AV have  -contour(a) = the space between two hyper-cones center on  which just circumscribes the Euclidean  -hyperdisk at a.  COS  (a)  a = (1/|a|)  d (  x x d a d ) factor out a d COS b (a)?  a b Intersection (in pink) with LNTV  -contour. Graphs of functionals with hyper-conic contours: E.g., COS b (a) for any vector, b

f(a) x = (x-a)o(x-a) d = index over dims, =  d=1..n (x d 2 - 2a d x d + a d 2 ) i,j,k bit slices indexes =  d=1..n (  k 2 k x dk ) 2 - 2  d=1..n a d (  k 2 k x dk ) + |a| 2 =  d (  i 2 i x di )(  j 2 j x dj ) - 2  d=1..n a d (2 k x dk ) + |a| 2 =  d  i,j 2 i+j x di x dj - 2  d,k 2 k a d x dk + |a| 2 f(a) x =  i,j,d 2 i+j (P di^dj ) x - |a|2 |a|2  k 2 k+1  d a d (P dk ) x + β exp( -  f(a) x ) = β exp(-  i,j,d 2 i+j (P di^dj ) x ) *exp( -  |a| 2 ) * exp(  k 2 k+1  d a d (P dk ) x ) Adding up the Gaussian weighted votes for class c: β exp( -  f(a) x ) = β ( exp(-  i,j,d 2 i+j (P di^dj ) x ) exp( -  |a| 2 ) * exp(  k 2 k+1  d a d (P dk ) x ) )  x  c β exp( -  f(a) x ) = β  x  c ( exp(-  i,j,d 2 i+j (P di^dj ) x ) exp( -  |a| 2 ) * exp(  k 2 k+1  d a d (P dk ) x ) )  x  c exp ( (-  i,j,d 2 i+j (P di^dj ) x ) +  k,d 2 k+1 a d (P dk ) x )  x  c exp (  i  j,d -  2 i+j (P di^dj ) x +  i=j,d (  a d 2 i+1 -  2 2i ) (P di ) x ) Collecting diagonal terms inside exp  i,j,d inside exp, coefs are multiplied by 1|0-bit (depends on x). For fixed i,j,d either coef is x-indep (if 1bit) or not (if 0bit)  x  c (  i  j,d exp ( -  2 i+j (P di^dj ) x ) *  i=j,d exp (  (a d 2 i+1 -2 2i )(P di ) x ) )  (  i  j,d  :P dijx =1 exp ( -  2 i+j ) *  i=j,d  :P dijx =1 exp (  (a d 2 i+1 -2 2i ) ) ) (eq1) =  d ( (  i 2 i x di )(  j 2 j x dj ) - 2(  i 2 i a di )(  j 2 j x dj ) + (  i 2 i a di ) (  j 2 j a dj ) ) =  d (  i,j 2 i+j x di x dj -  i,j 2 i+j+1 a di x dj +  i,j 2 i+j a di a dj ) =  d,i,j 2 i+j ( x di x dj - 2a di x dj + a di a dj ) =  d,i,j 2 i+j ( x di -a dj )(x dj -a dj ) Some additiona formulas:

f d (a) x = |x-a| d = |  i 2 i ( x di – a di )|= |  i: a di =0 2 i x di -  i: a di =1 2 i x' di | Thus, for the derived attribute, f d (a) = numeric distance of x d from a d, if we remember that: when a di =1, subtract those contributing powers of 2 (don't add) and that we use the complement dimension=d basic Ptrees, then it should work. The point is that we can get a set of near basic or negative basic Ptrees, nbPtrees, for derived attr f d (a) directly from the basic Ptrees for A d for free. Thus, the near basic Ptrees for f d (a) are the basic A d Ptrees for those bit-positions where a di = 0 and they are the complements of the basic A d Ptrees for those bit-positions where a di = 1 (called f d (a)'s nbPtrees) Caution: subtract the contribution of the nbPtrees for positions where a di =1 Note: nbPtrees are not predicate trees (are they? What's the predicate?) The EIN ring formulas are related to this, how? If we are simply after easy pruning contours containing a (so that we can scan to get the actual Euclidean epsilon nbrs and/or to get Guassian weighted vote counts, we can use Hobbit-type contours (middle earth contours of a?). See next slide for a discussion of hobbit contours.

A principle : A job is not done until the Mathematics is completed. The Mathematics of a research job includes 0. Getting to the frontiers of the area (researching, organizing, understanding and integrating everything others have done in the area up to the present moment and what they are likely to do next). 1. developing a killer idea for a better way to do something. 2. proving claims (theorems, performance evaluation, simulation, etc.), 3. simplification (everything is simple once fully understood), 4. generalization (to the widest possible application scope), and 4. insight (what are the main issues and underlying mega-truths (with full drill down)). Therefore, we need to ask the following questions at this point: Should we use the vector of medians (the only good choice of middle point in mulidimensional space, since the point closest to the mean definition is influenced by skewness, like the mean). We will denote the vector of medians as h (a)=|a- | is an important functional (better than h  (a)=|a-  |?) If we compute the median of an even number of values as the count-weighted average of the middle two values, then in binary columns,  and coincide. (so if µ and are far apart, that tells us there is high skew in the data (and the coordinates where they differ are the columns where the skew is found).

Are they easy P-tree computations? Do they offer advantages? When? What? Why? E.g., do they automatically normalize for us? What about the vector of standard deviations,  ? (computable with P-trees!) Do we have an improvement of BIRCH here? - generating similar comprehensive statistical measures, but much faster and more focused?) We can do the same for any rank statistic (or order statistic), e.g., vector of 1 st or 3 rd quartiles, Q 1 or Q 3 ; the vector of k th rank values (k th ordinal values). If we preprocessed to get the basic P-trees of, and each mixed quartile vector (e.g., in 2-D add 5 new derived attributes;, Q 1,1, Q 1,2, Q 2,1, Q 2,2 ; where Q i,j is the i th quartile of the j th column), what does this tell us (e.g., what can we conclude about the location of core clusters? Maybe all we need is the basic P-trees of the column quartiles, Q 1..Q n ?) Additional Mathematics to enjoy: L  ordinal disks: disk  (C,k) = {x | x d is one of the k-Nearest Neighbors of a d  d}. skin  (C,k), closed skin  (C,k) and ring  (C,k) are defined as above.

The Middle Earth Contours of a are gotten by ANDing in the basic Ptree for a d,i =1 and ANDing in the complement if a d,i =0 (down to some bit-position threshold in each dimension, bpt d. bpt d can be the same for each d or not). Caution: Hobbit contours of a are not symmetric about a. That becomes a problem (for knowing when you have a symmetric nbrhd in the contour) expecially when many lowest order bits of a are identical (e.g., if a d = 8 = 1000 ) If the low order bits of a d are zeros, one should union (OR) take the Hobbit contour of a d - 1 (e.g., for 8 also take 7=0111) If the low order bits of a d are ones, one should union (OR) the Hobbit contour of a d + 1 (e.g, for 7=111 also take 8=1000) Some need research: Since we are looking for an easy prune to get our mask down to a scannable size (low root count) but not so much of a prune that we have too few voters within Euclidean epsilon distance of a for a good vote, how can we quickly determine an easy choice of a Hobbit prune to accomplish that? Note that there are many Hobbit contours. We can start with pruning in just one dimension and with only the lowest order bit in that dimension and work from there, how though? THIS COULD BE VERY USEFUL?

Suppose there are two classes, red and green and they are on the cylinder shown. Then the vector connecting medians (vcm) in YZ space is shown in purple. s x y z Then the unit vector in the direction of the vector connecting medians (uvcm) in YZ space is shown in blue. The vector from the midpoint of the medians to s is in orange. The inner product of the blue and the orange is the same as the inner product we would get by doing the same thing in all 3 dimensions! The point is that the x-component of the red vector of medians and that of the green are identical so that the x component of the vcm is zero. Thus, when the small vcm comp in a given dimension is very small or zero, we can eliminate that dimension! That's why I suggest a threshold for the innerproduct in each dimension first. It is a feature or attribute relevance tool.

DBQ versus MIKE (DataBase Querying vs. Mining through data for Information and Knowledge Extraction Why do we call it Mining through data for Information & Knowledge Extraction and not just Data Mining? We Mine Silver and Gold! We don't just Mine Rock (The emphasis should be on the desired result, not the discard. The name should emphasize what we mine for, not what we mine from.) Silver and Gold are low-volume, high-value products, found (or not) in the mountains of rock (high-volume, low- value). Information and knowledge are low-volume, high-value, hiding in mountains of data (high-volme, low-value) In both MIKE and MSG the output and substrate are substantially different in structure (chemical / data structure) Just as in Mining Silver and Gold, we extract (hopefully) Silver and Gold from raw Rock, in Mining through data for Information and Knowledge, we extract (hopefully) Information and Knowledge from raw Data. So Mining through data for Information and Knowledge Extraction is the correct terminology and MIKE is the correct acronym, not Data Mining (DM). How is Data Base Querying (DBQ) different from Mining thru data for Info & Knowledge (MIKE)? In all mining (MIKE as well as MSG) we hope to successfully mine out something of value, but failure is likely, whereas in DBQ, valuable results are likely and no result is unlikely. DBQ should be called Data Base Quarrying, since it is more similar to Granite Quarrying (GQ), in that what we extract has the same structure as that from which we extract it (the substrate). It has higher value because its detail and specificity. I.e., the output records of a DBQ are exactly the reduce size set of records we demanded and expected from our query and the output grave stones of GQ are exactly the size and shape we demanded and expected, and in both cases what is left is a substance that is the same as what is taken). In sum, DBQ = Quarrying (highly predictable output and the output has same structure as the substrate (sets of records)). MIKE = Mining (unpredictable output and the output has different structure than the substrate (e.g., T/F or partition).

Some good Dataset for classification 1.KDDCUP-99 Dataset (Network Intrusion Dataset) –4.8 millions records, 32 numerical attributes –6 classes, each contains >10,000 records –Class distribution: –Testing set: 120 records, 20 per class –4 synthetic datasets (randomly generated): -10,000 records (SS-I) -100,000 records (SS-II) -1,000,000 records (SS-III) -2,000,000 records (SS-IV) Normal972,780 IP sweep12,481 Neptune1,072,017 Port sweep10,413 Satan15,892 Smurf2,807,886

Speed (Scalability) Comparison (k=5, hs=25) Algorithm x 1000 cardinality 10100100020004891 SMART-TV0.140.332.013.889.27 P-KNN0.891.063.9412.4430.79 KNN0.392.3423.4749.28 NA Speed and Scalability Machine: Intel Pentium 4 CPU 2.6 GHz, 3.8GB RAM, running Red Hat Linux Note: these evaluations were done when we were still sorting the derived TV attribute and before we used Gaussian vote weighting. Therefore both speed and accuracy of SMART-TV have improved markedly! 1000200030004891 0 10 20 30 40 50 60 70 80 90 100 Training Set Cardinality (x1000) Time in Seconds Running Time Against Varying Cardinality SMART-TV PKNN KNN

Dataset (Cont.) 2.OPTICS dataset –8,000 points, 8 classes (CL-1, CL-2,…,CL-8) –2 numerical attributes –Training set: 7,920 points –Testing set: 80 points, 10 per class

3.IRIS dataset –150 samples –3 classes (iris-setosa, iris- versicolor, and iris- virginica) –4 numerical attributes –Training set: 120 samples –Testing set: 30 samples, 10 per class Dataset (Cont.)

Overall Classification Accuracy Comparison Datasets SMART- TVPKNNKNN IRIS0.970.710.97 OPTICS0.960.990.97 SS-I0.960.720.89 SS-II0.920.910.97 SS-III0.940.910.96 SS-IV0.920.910.97 NI0.930.91NA Overall Accuracy

As you probably know, Taufik used a heap process to get the k nearest neighbors of unclassified samples (requiring one scan through the well-pruned nearest neighbor candidate set). This means that Taufik did not use the closed kNN set, so accuracy will be the same as horizontal kNN (single scan, non-closed) Actually, accuracy varies slightly depending on the k th NN picked from the ties). Taufik is planning to leave the thesis that way and explain why he did it that way (over-fairness) ;-) A great learning experience with respect to using DataMIME and a great opportunity for thesis exists here - namely showing that when one uses closed kNN in SMART-TV, not only do we get a tremendously scalable algorithm, but also a much more accurate result (even slightly faster since no heaping to do). A project: Re-run Taufik's performance measurements (just for SMART-TV) using a final scan of the pruned candidate Nearest Neighbor Set. Let all candidates vote using a Gaussian vote drop-off function as: More Mathematics to enjoy: If candidate lies Euclidean distance >  from a, vote weight = 0, else, we define Gaussian drop-off function, g(x)= Gauss(r=|x-a|)=1/(std*  2  ) * e -(r-mean) 2 /2var where std, mean, var refer to the set of distances from a of non-zero voters (i.e., the set of r=|x-a| numbers), but use the Modified Gaussian, MG(x) = g(x) - e -  2 so the vote weight function drops smoothly to 0 (right at the boundary of the  -disk about a and then stays zero outside it).

Meegeum has claimed the study of how much the above improves accuracy in various settings (and how various parameterization of g(x) (ala statistics) affect it)? More enjoyment!: If there is a example data set where accuracy goes down when using the Gaussian and closed NNS, that proves the data set is noise at that point? (i.e.,  no continuity relationship between features and classes at a). This leads to an interesting theory regarding the continuity of a training set. Everyone assume the training set is what you work with and you assume continuity! In fact, Cancer studies forge ahead with classification even when p>>n, that is there are just too few feature tuples for continuity to even makes good sense! So now we have a method of identifying those regions of the training space where there is a discontinuity in the feature-to-class relationship, FC. That will be valuable (e.g., in Bioinformatics). Has this been studied? (I think, everyone just goes ahead and classifies based on continuity and lives with the result! More Mathematics to enjoy:

More enjoyment (continued): I claim, when the class label is real, then with a properly chosen Gaussian vote-weight function (chosen by domain experts) and with a good method of testing the classifier, if SMART-TV miss-classifies a test point, then it is not a miss-classification at all, but there is a discontinuity in FC there (between feature and class)! In other words, that proves the training set is wrong there and should not be used. I am claiming (do you back me up, Dr. Abidin?) SMART-TV, properly tuned, DEFINES CORRECT! It will be fun to see how far we can go with this point of view. Be warned - it will cause quite a stir! Thoughts: 1. Choose a vote drop-off Gaussian carefully (with domain knowledge in play) and designates it as "right". What could be more right? - if you agree that classification has to be based on FC continuity. 2. Analyze (very carefully) SMART-TV vote histograms for  1 <  2 <... <  h If all are inconclusive then the Feature-to-Class function (FC) is discontinuous at a and classification SHOULD NOT BE ATTEMPTED USING THAT TRAINING SET! (This may get us into Krigging). If the histograms get more and more conclusive as the radius increases, then possibly one would want to rely on the outer ring votes, but one should also report that there is class noise at a! 3. We can get into tuning the Modified Gaussian, MG, by region. Determine the subregion where MC gives conclusive histograms. Then for each inconclusive training point, examine modifications (other parameters or even other dropoff functions) until you find one that is conclusive. Determine the full region where that one is conclusive...

More enjoyment (continued): CAUTION! Many important Class Label Attributes (CLAs) are real (e.g., level of ill intent in homeland security data mining, level of ill intent in Network Intrusion analysis, probability of cancer in a cancer tissue microarray dataset), but many important Class Label Attributes are categorical (e.g., bioinformatic anotation, Phenotype prediction, etc.). When the Class label is categorical, the distance on the CLA becomes the characteristic distance function (distance=0 iff the 2 categories are different). Continuity at a becomes:  >0  :d(x,a)<   f(x)=f(a). Possibly boundary determination of the training set classes is most important in case? Is that why SVM works so well in those situations? Still, For there to be continuity at a, it must be the case that some NNS of a maps to f(a). However, if the (CLA) is real: Has anyone every seen analysis of "what is the best definition of correct" done (do statistician do this?). Maybe we need a good statistician to take a look, but let's pursue it anyway so we know what we can claim. Step 1: re-run Taufik's SMART-TV with the Modified Gaussian, MG(x), and closed kNN. We should also take several standard UCI ML classification datasets, randomize classes in some particular isotropic neighborhood (so that we know where there is definitely an FC discontinuity) Then show (using matlab?) that SVM fails to detect that discontinuity (i.e., SVM gives a definitive class to it without the ability to detect the fact that it doesn't deserve one? or do the Lagrange multipliers do that for them?) and then show that we can detect that situation. Does any other author do this? Other fun ideas?

P 11 P 12 P 13 P 21 P 22 P 23 P 31 P 32 P 33 P 41 P 42 P 43 0 0 0 0 1 10 0 1 0 0 1 0 0 0 0 0 0 1 01 10 0 1 0 0 1 0 0 0 0 1 0 01 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 10 01 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 S(B 0 B 1 B 2 B 3 ) 0 1 0 1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 1 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 0 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 1 1 1 0 0 0 0 0 1 1 0 0 S 11 S 12 S 13 S 21 S 22 S 23 S 31 S 32 S 33 S 41 S 42 S 43 001 11 0 011 10 1 100 01 1 101 01 0 110 11 0 111 00 0 R(A 0 A 1 A 2 ) To data mine this PK multi-relation, (R.A 0 is ordered ascending primary key and S.B 2 is the foreign key), scan S building (basic P-trees for) the derived attributes B n+1..B n+m (here B 4,B 5 ) from A 1..A m using the bottom up approach (next slide)? Note: Once the derived basic P-trees are built, what if a tuple is added to S? If it has a new B 2 -value then a new tuples must have been added to R also (with that value in A 0 ). Then all basic P-trees must be extended (including the derived). Classify on a foreign key table (S) when some of the features need to come from the primary key table (R)? Classifying with multi-relational (heterogeneous) training data

S.B 4,1 1 1 1 1 0 0 1 0 0 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 S(B 0 B 1 B 2 B 3 ) 001 11 0 011 10 1 100 01 1 101 01 0 110 11 0 111 00 0 R(B 2 A 1 A 2 ) S.B 4,2 1 1 1 1 1 1 1 1 1 S.B 5 0 0 0 0 0 0 0 0 0 P 01 P 02 P 03 P 11 P 12 P 13 P 21 P 22 P 23 P 31 P 32 P 33 P 4,1 P 4,2 P 5 0 0 0 0 1 10 0 1 0 0 1 0 0 0 0 0 0 1 01 10 0 1 0 0 1 0 0 0 0 1 0 01 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 10 01 0 0 1 1 0 0 1 The cost is the same as an indexed nested loop join (reasonable to assume there is a primary index on R). When an insert is made to R, nothing has to change. When an insert is made to S, the P-tree structure is expanded and filled using the values in that insert plus the R-attribute values of the new S.B 2 value (This is one index lookup. The S.B 2 value must exists in R.A 0 by referential integrity). Finally, if we are using, e.g. 4 P i,j P-trees instead of the (4,2,1) P i,j P-trees shown here, it's the same: The basic P-tree fanout is /\, the left leaf is filled by the first 4 values, the right leaf is filled with the last 4.

S.B 4,1 1111 010 111 110 001 011 111 110 000 010 110 101 001 010 111 101 111 101 010 001 100 010 010 001 101 111 000 001 100 S(B 0 B 1 B 2 B 3 ) 001 11 0 011 10 1 100 01 1 101 01 0 110 11 0 111 00 0 R(B 2 A 1 A 2 ) If we are using, e.g. 4 P i,j P-trees instead of the (4,2,1) P i,j P-trees shown here, it's the same: The basic P-tree fanout is /\, the left leaf is filled by the first 4 values, the right leaf is filled with the last 4. S.B 4,2 S.B 5 1111 0000 0000 1111 0000 11111111 11111111 00000000 S.B 4,1 1111 S.B 4,2 S.B 5 0000 1,1 = EM-S.B 4,1 1,1 = EM-S.B 4,2 0,0 = EM-S.B 5 0,1 = UM-S.B 4,1 1,1 = UM-S.B 4,2 0,0 = UM-S.B 5 0 1 0 or

VertiGO (Vertical Gene Ontology) The GO is a data structure which needs to be mined together with various valuable bioinformatic data sets. Biologists waste time searching for all available information about each small area of research. This is hampered further by variations in terminology in common usage at any given time, and that inhibit effective searching by computers as well as people. E.g., In a search for new targets for antibiotics, you want all gene products involved in bacterial protein synthesis, that have significantly different sequence or structure from those in humans. If one DB says these molecules are involved in 'translation' and another uses 'protein synthesis', it is difficult for you - and even harder for a computer - to find functionally equivalent terms. GO is an effort to address the need for consistent descriptions of gene products in different DBs. The project began in 1988 as a collaboration between three model organism databases: FlyBaseFlyBase (Drosophila), Saccharomyces Genome DatabaseSaccharomyces Genome Database (SGD) Mouse Genome DatabaseMouse Genome Database (MGD). Since then, the GO Consortium has grown to include several of the world's major repositories for plant, animal and microbial genomes. See the GO web page for a full list of member orgs.GO web page

VertiGO (Vertical Gene Ontology) The GO is a DAG which needs to be mined in conjunction with the Gene Table (one tuple for each gene with feature attributes). The DAG links are IS-A or PART-OF links. (Description follows from the GO website). If we take the simplified view that the GO assigns annotations of types ( Biological Process (BP); Molecular Function (MF); Cellular Component (CC)) to genes, with qualifiers ( "contributes to", "colocalizes with", "not" ) and evidence codes: IDA=InferredfromDirectAssay; IGI=InferredfromGeneticInteraction, IMP=InferredfromMutantPhenotype; IPI=InferredfromPhysicalInteraction, TAS=TraceableAuthorStatement; IEP=InferredfromExpressionPattern, RCA=InferredfromReviewedComputationalAnalysis, IC=InferredbyCurator IEA=InferredbyElectronicAnnotation ISS=InferredfromSequence/StructuralSimilarity, NAS=NontraceableAuthorStatement, ND=NoBiologicalDataAvailable, NR=NotRecorded Solution-1: For each annotation (term or GOID) have a 2-bit type code column GOIDType BP=11 MF=10 CC=00 and a 2-bit qualifier code column GOIDQualifier with contributesto=11, co-localizeswith=10 and not=00 and a 4-bit evidence code column GOIDEvidence: e,g,: IDA=1111, IGI-1110, IMP=1101, IPI=1100, TAS=1011, IEP=1010, ISS=1001, RCA=1000, IC=0111, IEA=0110, NAS=0100, ND=0010, NR=0001 (putting DAG structure in schema catalog). (Increases width by 8-bits * #GOIDs to losslessly incorporate the GO info). Solution-2: BP, MF and CC DAGs are disjoint (share no GOIDs? true?), an alternative solution is: Use a 4-bit evidencecode/qualifier column, GOIDECQ: For evidence codes: IDA=1111 IGI-1110 IMP=1101 IPI=1100 TAS=1011 IEP=1010 ISS=1001 RCA=1000 IC=0111 IEA=0110 NAS=0101 ND=0100 NR=0011. Qualifiers: 0010=contributesto 0001=colocalizeswith 0000=not (width increases 4-bits*#GOID lossless GO). Solution-3: bitmap all 13 evidencecodes and all 3 qualifiers (adds 16 bit map per GO term). Keep in mind that genes are assumed to be inherited up the DAGs but are only listed at the lowest level to which they apply. This will keep the bitmaps sparse. If a GO term has no attached genes, it need not be included (many such?). It will be in the schema with its DAG links, and will be assumed to inherit all downstream genes, but it will not generate 16 bit columns in Gene Table). Is the not qualifier the complement of the term bitmap?

biological processes (BP), cellular components (CC) molecular functions (MF). There are three separate aspects to this effort: The GO consortium 1. writes and maintains the ontologies themselves; 2. makes associations between the ontologies and genes / gene products in the collaborating DBs, 3. develops tools that facilitate the creation, maintainence and use of ontologies. The use of GO terms by several collaborating databases facilitates uniform queries across them. The controlled vocabularies are structured so that you can query them at different levels: e.g., 1. use GO to find all gene products in the mouse genome that are involved in signal transduction, 2. zoom in on all the receptor tyrosine kinases. This structure also allows annotators to assign properties to gene products at different levels, depending on how much is known about a gene product. GO has 3 structured, controlled vocabularies (ontologies) describing gene products (the RNA or protein resulting after transcription) by their species-independent, associated GO is not a database of gene sequences or a catalog of gene products GO describes how gene products behave in a cellular context. GO is not a way to unify biological databases (i.e. GO is not a 'federated solution'). Sharing vocabulary is a step towards unification, but is not sufficient. Reasons include: Knowledge changes and updates lag behind.

Curators evaluate data differently (e.g., agree to use the word 'kinase', but not to support this by stating how and why we use 'kinase', and consistently to apply it. Only in this way can we hope to compare gene products and determine whether they are related. GO does not attempt to describe every aspect of biology. For example, domain structure, 3D structure, evolution and expression are not described by GO. GO is not a dictated standard, mandating nomenclature across databases. Groups participate because of self-interest, and cooperate to arrive at a consensus. The 3 organizing GO principles: molecular function, biological process, cellular component. A gene product has one or more molecular functions and is used in one or more biological processes; it might be associated with one or more cellular components. E.g., the gene product cytochrome c can be described by the molecular function term oxidoreductase activity, the biological process terms oxidative phosphorylation and induction of cell death, and the cellular component terms mitochondrial matrix, mitochondrial inner membrane. Molecular function (organizing principle of GO) describes e.g., catalytic/binding activities, at molecular level GO molecular function terms represent activities rather than the entities (molecules / complexes) that perform actions, and do not specify where or when, or in what context, the action takes place. Molecular functions correspond to activities that can be performed by individual gene products, but some activities are performed by assembled complexes of gene products. Examples of broad functional terms are catalytic activity, transporter activity, or binding; Examples of narrower functional terms are adenylate cyclase activity or Toll receptor binding. It is easy to confuse a gene product with its molecular function, and for thus many GO molecular functions are appended with the word "activity". The documentation on gene products explains this confusion in more depth.gene products

A Biological Process is series of events accomplished by 1 or more ordered assemblies of molecular fctns. Examples of broad biological process terms: cellular physiological process or signal transduction. Examples of more specific terms are pyrimidine metabolism or alpha-glucoside transport. It can be difficult to distinguish between a biological process and a molecular function, but the general rule is that a process must have more than one distinct steps. A biological process is not equivalent to a pathway. We are specifically not capturing or trying to represent any of the dynamics or dependencies that would be required to describe a pathway. A cellular component is just that, a component of a cell but with the proviso that it is part of some larger object, which may be an anatomical structure (e.g. rough endoplasmic reticulum or nucleus) or a gene product group (e.g. ribosome, proteasome or a protein dimer). What does the Ontology look like? GO terms are organized in structures called directed acyclic graphs (DAGs), which differ from hierarchies in that a child (more specialized term) can have many parent (less specialized term). For example, the biological process term hexose biosynthesis has two parents, hexose metabolism and monosaccharide biosynthesis. This is because biosynthesis is a subtype of metabolism, and a hexose is a type of monosaccharide. When any gene involved in hexose biosynthesis is annotated to this term, it is automatically annotated to both hexose metabolism and monosaccharide biosynthesis, because every GO term must obey the true path rule: if the child term describes the gene product, then all its parent terms must also apply to that gene product.

It is easy to confuse a gene product and its molecular function, because very often these are described in exactly the same words. For example, 'alcohol dehydrogenase' can describe what you can put in an Eppendorf tube (the gene product) or it can describe the function of this stuff. There is, however, a formal difference: a single gene product might have several molecular functions, and many gene products can share a single molecular function, e.g., there are many gene products that have the function 'alcohol dehydrogenase'. Some, but by no means all, of these are encoded by genes with the name alcohol dehydrogenase. A particular gene product might have both the functions 'alcohol dehydrogenase' and 'acetaldehyde dismutase', and perhaps other functions as well. It's important to grasp that, whenever we use terms such as alcohol dehydrogenase activity in GO, we mean the function, not the entity; for this reason, most GO molecular function terms are appended with the word 'activity'. Many gene products associate into entities that function as complexes, or 'gene product groups', which often include small molecules. They range in complexity from the relatively simple (for example, hemoglobin contains the gene products alpha-globin and beta-globin, and the small molecule heme) to complex assemblies of numerous different gene products, e.g., the ribosome. At present, small molecules are not represented in GO. In the future, we might be able to create cross products by linking GO to existing databases of small molecules such as Klotho, LIGAND cross productsKlothoLIGAND

How do terms get associated with gene products? Collaborating databasesCollaborating databases annotate their gene products (or genes) with GO terms, providing references and indicating what kind of evidence is available to support the annotations. More info in GO Annotation Guide.GO Annotation Guide. If you browse any of the contributing databases, you'll find that each gene or gene product has a list of associated GO terms. Each database also publishes a table of these associations, and these are freely available from the GO ftp site.GO ftp site You can also browse the ontologies using a range of web-based browsers. A full list of these, and other tools for analyzing gene function using GO, is available on the GO Tools page.GO Tools page In addition, the GO consortium has prepared GO slims, 'slimmed down' versions of the ontologies that allow you to annotate genomes or sets of gene products to gain a high-level view of gene functions. Using GO slims you can, for example, work out what proportion of a genome is involved in signal transduction, biosynthesis or reproduction. See the GO Slim Guide for more information.GO Slim Guide All GO data is free. Download the ontology data in a number of different formats, including XML and mySQL, from the GO Downloads page (more info on syntax of these formats, GO File Format Guide.GO Downloads pageGO File Format Guide If you need lists of the genes or gene products that have been associated with a particular GO term, the Current Annotations table tracks the number of annotations and provides links to gene association files for each of collaborating DBs is available.Current Annotations table GO allows us to annotate genes and their products with a limited set of attributes. e.g., GO does not allow us to describe genes in terms of which cells or tissues they're expressed in, which developmental stages they're expressed at, or their involvement in disease. It is not necessary for GO to this since other ontologies are doing it. The GO consortium supports the development of other ontologies and makes its tools for editing and curating ontologies available. A list of freely available ontologies that are relevant to genomics and proteomics and are structured similarly to GO can be found at the Open Biomedical Ontologies website. A larger list, which includes the ontologies listed at OBO and also other controlled vocabularies that do not fulfil the OBO criteria is available at the Ontology Working Group page of the Microarray Gene Expression Data Society (MGED). Open Biomedical OntologiesOntology Working Group

Cross-products: The existence of several ontologies will also allow us to create 'cross-products' that maximize the utility of each ontology while avoiding redundancy. For example, by combining the developmental terms in the GO process ontology with a second ontology that describes Drosophila anatomical structures, we could create an ontology of fly development. We could repeat this process for other organisms without having to clutter up GO with large numbers of species-specific terms. Similarly, we could create an ontology of biosynthetic pathways by combining biosynthesis terms in the GO process ontology with a chemical ontology. Mappings to other classification systems: GO is not the only attempt to build structured controlled vocabularies for genome annotation. Nor is it the only such series of catalogs in current use. We have attempted to make translation tables between these catalogs and GO. We caution that these mappings are neither complete nor exact; they are to be used as a guide. One reason for this is absence of definitions from many of the other catalogs and of a complete set of definitions in GO itself. More information on the syntax of these mappings can be found in the GO File Format Guide.translation tablesGO File Format Guide Contributing to GO: The GO project is constantly evolving, and we welcome feedback from all users. If you need a new term or definition, or would like to suggest that we reorganize a section of one of the ontologies, please do so through our online request-tracking system, which is hosted by SourceForge.net.SourceForge.net What is a GO term? The purpose of GO is to define particular attributes of gene products. A term is simply the text string used to describe an entry in GO, e.g. cell, fibroblast growth factor receptor binding or signal transduction. A node refers to a term and all its children. GO does not contain the following: Gene products: e.g. cytochrome c is not in GO; attributes of it, e.g., oxidoreductase activity, are. Processes, functions or components that are unique to mutants or diseases: e.g. oncogenesis is not a valid GO term because causing cancer is not the normal function of any gene. Attributes of sequence such as intron/exon parameters: these are not attributes of gene products and will be described in a separate sequence ontology (see OBO web site for more information).OBO Protein domains or structural features. Protein-protein interactions.

Conventions when adding a term: These stylistic points should be applied to all aspects of the ontologies. Spelling conventions: Where there are differences in accepted spelling between English and US, use US form, e.g. polymerizing, signaling, rather than polymerising, signalling. A dictionary of 'words' used in GO terms at GODict.DATGODict.DAT Abbreviations: Avoid abbreviations unless they're self-explanatory. Use full element names, not symbols. Use hydrogen for H+. Use copper and zinc rather than Cu and Zn. Use copper(II), copper(III), etc., rather than cuprous, cupric, etc. For biomolecules, spell out the term in full wherever practical: use fibroblast growth factor, not FGF. Greek symbols: Spell out Greek symbols in full: e.g. alpha, beta, gamma. Case: GO terms are all lower case except where demanded by context, e.g. DNA, not dna. Singular/plural: Use singula, except where a term is only used in plural (eg caveolae). Be descriptive: Be reasonably descriptive, even at the risk of verbal redundancy. Remember, DBs that refer to GO terms might list only the finest-level terms associated with a particular gene product. If the parent is aromatic amino acid family biosynthesis, then child should be aromatic amino acid family biosynthesis, anthranilate pathway, not anthranilate pathway. Anatomical qualifiers: Do not use anatomical qualifiers in the cellular process and molecular function ontologies. For example, GO has the molecular function term DNA-directed DNA polymerase activity but neither nuclear DNA polymerase nor mitochondrial DNA polymerase. These terms with anatomical qualifiers are not necessary because annotators can use the cellular component ontology to attribute location to gene products, independently of process or fctn. Synonyms: When several words or phrases that could be used as the term name, one form will be chosen as term name whilst the other possible names are added as synonyms. Despite the name, GO synonyms are not always 'synonymous' in the strictest sense of the word, as they do not always mean exactly the same as the term they are attached to. Instead, a GO synonym may be broader or narrower than the term string; it may be a related phrase; it may be alternative wording, spelling or use a different system of nomenclature; or it may be a true synonym. This flexibility allows GO synonyms to serve as valuable search aids, as well as being useful for apps such as text mining and semantic matching. Having a single, broad relationship between a GO term and its synonyms is adequate for most search purposes, but for other applications such as semantic matching, the inclusion of a more formal relationship set is valuable. Thus, GO records a relationship type for each synonym, stored in OBO format flat file. relationship typeOBO format flat file Synonym types: The synonym relationship types are: term is an exact synonym ( ornithine cycle is an exact synonym of urea cycle) terms are related (cytochrome bc1 complex is a related to ubiquinol-cytochrome-c reductase activity) synonym is broader than the term name (cell division is a broad synonym of cytokinesis) synonym is narrower or more precise (pyrimidine-dimer repair by photolyase is a narrow synonym of photoreactive repair) synonym is related to, but not exact, broader or narrower (virulence has synonym type of other related to term pathogenesis)

These types form a loose hierarchy: related [i] exact synonym [i] broad synonym [i] narrow synonym [i] other related synonym The default relationship is related to, as all synonyms are in some way related to the term name, but more specific relationships are assigned where possible. The synonym type other related is used where the relationship between a term and its synonym is NOT exact, narrower or broader. In some cases, broader and narrower synonyms are created in the place of new parent or child terms because some synonym strings may not be valid GO terms but may still be useful for search purposes. This may be because the synonym is the name of a gene product e.g. ubiquitin-protein ligase activity has the narrower synonym E3, as E3 is a specific gene product with ubiquitin-protein ligase activity. Adding synonyms: When you add a synonym using DAG-Edit, choose a type from the pull-down selector (see the DAG- Edit user guide for more information). DAG-Edit will incorporate the synonym type into the OBO format flat file when you save. The default synonym type is the broadest, 'synonym' (equivalent to 'related' above). Number of synonyms for a term is not limited, and the same text string can be used for more than 1 GO term. Add synonyms if you edit a term name but the old name is still a valid synonym; for example, if you change respiration to cellular respiration, keep respiration as a synonym. This helps other users find familiar terms. Add synonyms if the term has (or contains) a commonly used abbreviation. For example, FGF binding could be used as a synonym for fibroblast growth factor binding. Do not add a synonym if the only difference is case (e.g. start vs. START). Synonyms, like term names, are all lower case except where demanded by context (e.g. DNA, not dna). Rules for Synonyms: Acronyms are exactly synonymous with full name (if acronym is not used in any other sense elsewhere) 'Jargon' type phrases are exactly synonymous w full name (if phrase is not used in any other sense elsewhere) proton is exactly synonymous with hydrogen in most senses EXCEPT where hydrogen means H 2 (i.e. gas) include implicit information when making decision; take into account which ontology the term is in - e.g. an entry term that ends in 'factor' is not synonymous with a molecular function. ligand is NOT exactly synonymous with binding (ligand is an entity, binding an action) XXX receptor ligand is NOT exactly synonymous with XXX (1 potential ligands so XXX receptor ligand broader than XXX XXX complex is NOT exactly synonymous with XXX (XXX is ambiguous - could describe activity of XXX) porter and transporter are NOT exactly synonymous (transporter is broader) symporter/antiporter and transporter are NOT exactly synonymous (transporter is broader)

General database cross references (general dbxrefs) should be used whenever a GO term has an identical meaning to an object in another database. Some ex. of common general dbxrefs in GO: Ontology DB Sample dbxref Fctn Enzyme Commission EC:3.5.1.6 Transport Protein Database TC:2.A.29.10.1 Biocatalysis/Biodegradation DB UM-BBD_enzymeID:e0310 Biocatalysis/Biodegradation DB UM-BBD_pathwayID:dcb MetaCyc Metabolic Pathway DB MetaCyc:XXXX-RXN Process MetaCyc Metabolic Pathway DB MetaCyc:2ASDEG-PWY Component None The GO.xrf_abbs file is maintained by the BioMOBY project, so to make changes to the file, you need to use their web form.Enzyme CommissionTransport Protein DatabaseBiocatalysis/Biodegradation DBBiocatalysis/Biodegradation DBMetaCyc Metabolic Pathway DBMetaCyc Metabolic Pathway DBBioMOBYweb form Understanding relationships in GO: GO ontologies are structured as a directed acyclic graph (DAG), which means that a child (more specialized) term can have multiple parents (less specialized terms). This makes GO a powerful system to describe biology, but creates some pitfalls for curators Keeping the following guidelines in mind should help you to avoid these problems. A child term can have one of two different relationships to its parent(s): is_a or part_of. The same term can have different relationships to different parents; for example, the child 'GO term 3' may be an is_a of parent 'GO term 1' and a part_of parent, 'GO term 2': In GO, an is_a relationship means that the term is a subclass of its parent. For example, mitotic cell cycle is_a cell cycle, not confused with an 'instance' which is a specific example. E.g., clogs are a subclass or is_a of shoes, while the shoes I have on my feet now are an instance of shoes. GO, like most ontologies, does not use instances. The is_a relationship is transitive, which means that if 'GO term A' is a subclass of 'GO term B', and 'GO term B' is an subclass of 'GO term C', 'GO term A' is also a subclass of 'GO term C', E.g., Terminal N-glycosylation is a subclass of terminal glycosylation. Terminal glycosylation is a subclass of protein glycosylation. Terminal N-glycosylation is a subclass of protein glycosylation. part_of in GO is more complex. There are 4 basic levels of restriction for a part_of relationship: 1st type has no restrictions - no inferences can be made from the relationship between parent and child other than that parent may have child as a part, and the child may or may not be a part of the parent.

2nd type, 'necessarily is_part', means that wherever the child exists, it is as part of the parent. To give a biological example, replication fork is part_of chromosome, so whenever replication fork occurs, it is as part_of chromosome, but chromosome does not necessarily have part replication fork. 3rd type, 'necessarily has_part', is the exact inverse of type two; wherever the parent exists, it has the child as a part, but the child is not necessarily part of the parent. For example, nucleus always has_part chromosome, but chromosome isn't necessarily part_of nucleus. 4th type, is a combination of both two and three, 'has_part' and 'is_part'. An example of this is nuclear membrane is part_of nucleus. So nucleus always has_part nuclear membrane, and nuclear membrane is always part_of nucleus. The part_of relationship used in GO is usually type two, 'necessarily is_part'. Note that part_of types 1 and 3 are not used in GO, as they would violate the true path rule. Like is_a, part_of is transitive, so that if 'GO term A' is part_of 'GO term B', and 'GO term B' is part_of 'GO term C', 'GO term A' is part_of 'GO term C':true path rule E.g., Laminin-1 is part_of basal lamina. Basal lamina is part_of basement membrane. Laminin-1 is part_of basement membrane. The ontology editing tool DAG-Edit, from version 1.411 on, allows you to specify the necessity of relationships. The part_of relationship used in GO, necessarily is_part, would correspond to part_of, [inverse] necessarily true. For more information, see the DAG-Edit user guide.DAG-EditDAG-Edit user guide For info on how these relationships are represented in the GO flat files, see the GO File Format Guide.GO File Format Guide For technical info on the relationships used in GO and OBO, see the OBO relationships ontology.OBO relationships ontology The true path rule states that "the pathway from a child term all the way up to its top-level parent(s) must always be true". One of the implications of this is that the type of part_of relationship used in GO, outlined more fully in the part_of relationship section above, is restricted to those types where a child term must always be part_of its parent.the part_of relationship Often, annotating a new gene product reveals relationships in an ontology that break the true path rule, or species specificity becomes a problem. In such cases, the ontology must be restructured by adding more nodes and connecting terms such that any path upwards is true. When a term is added to the ontology, the curator needs to add all of the parents and children of the new term. This becomes clear with an example: consider how chitin metabolism is represented in the process ontology. Chitin metabolism is a part of cuticle synthesis in fly and is also part of cell wall organization in yeast. This was once represented in process ontology as: cuticle synthesis, [i]chitin metabolism, cell wall biosynthesis, [i]chitin metabolism, ---[i]chitin biosynthesis, ---[i]chitin catabolism

IllustrationIllustration The problem with this organization becomes apparent when one tries to annotate a specific gene product from one species. A fly chitin synthase could be annotated to chitin biosynthesis, and appear in a query for genes annotated to cell wall biosynthesis (and its children), which makes no sense because flies don't have cell walls. This is revised ontology structure which ensures that the true path rule is not broken: chitin metabolism, [i]chitin biosynthesis, [i]chitin catabolism, [i]cuticle chitin metabolism ---[i]cuticle chitin biosynthesis, ---[i]cuticle chitin catabolism [i]cell wall chitin metabolism, ---[i]cell wall chitin biosynthesis, ---[i]cell wall chitin catabolism IllustrationIllustration The parent chitin metabolism now has the child terms cuticle chitin metabolism and cell wall chitin metabolism, with the appropriate catabolism and synthesis terms beneath them. With this structure, all the daughter terms can be followed up to chitin metabolism, but cuticle chitin metabolism terms do not trace back to cell wall terms, so all the paths are true. In addition, gene products such as chitin synthase can be annotated to nodes of appropriate granularity in both yeast and flies, and queries will yield the expected results. Dependent ontology terms: Some GO terms imply presence of others. Examples from process ontology include: If either X biosynthesis or X catabolism exists, then parent X metabolism must also exist. If regulation of X exists, then the process X must also exist. Potentially any process in the ontology can be regulated. Note: X may refer to a phenotype (for example cell size in regulation of cell size; in these cases, X should not be added to ontology. GO nodes must avoid using species-specific defs. Nevertheless, many functions, processes and components are not common to all life forms. Our current convention is to include any term that can apply to more than one taxonomic class of organism. Within ontologies, are cases where a word/phrase has different meanings for organisms. E.g., embryonic development in insects is very different from embryonic development in mammals. Such terms are distinguished from one another by their definitions and by the sensu designation (sensu means 'in the sense of'), as in the term embryonic development (sensu Insecta). Nodes should be divided into sensu sub-trees where the children are or are likely to be different. Using sensu designation in a term does not exclude that term from being used to annotate species outside that designation. e.g., a 'sensu Drosophila' term might reasonably used to annotate a mosquito gene product. A GO node should never be more species-specific than any of its children. Child nodes can be at the same level of species specificity as the parent node(s), or more specific. When adding more species-specific nodes, curators should make sure that non-species-specific parents exist (or add them if necessary). E.g., take the process of sporulation. This occurs in both bacteria and fungi, but bacterial sporulation is quite a different process to fungal sporulation, so we therefore add two children to sporulation, sporulation (sensu Bacteria) and sporulation (sensu Fungi). If we now want to add a term to represent the assembly of the spore wall in fungi, we cannot just add spore wall assembly as a direct child of sporulation (sensu Fungi) as such a term could conceivably refer to the assembly of spore walls in bacteria. Name child term spore wall assembly (sensu Fungi) to ensure that it is as species-specific as parent term.

References and Evidence Every annotation must be attributed to a source, which may be a literature reference, another database or a computational analysis. The annotation must indicate what kind of evidence is found in the cited source to support the association between the gene product and the GO term. A simple controlled vocabulary is used to record evidence: IMP inferred from mutant phenotype IGI inferred from genetic interaction IPI inferred from physical interaction [w ] ISS inferred from sequence similarity [with ] IDA inferred from direct assay IEP inferred from expression pattern IEA inferred from electronic annotation [with ] TAS traceable author statement NAS non-traceable author statement ND no biological data available RCA inferred from reviewed computational analysis IC inferred by curator [from ] Annotation File Format: Collaborating databases export to GO a tab delimited file, known informally as a "gene association file" of links between database objects and GO terms. Despite jargon, the db object may represent a gene or a gene product (transcript or protein). Columns in file are described below, a table showing columns in order, with examples, is available.table The entry in the DB_Object_ID field (see below) of the association file is the identifier for the database object, which may or may not correspond exactly to what is described in a paper. For example, a paper describing a protein may support annotations to the gene encoding the protein (gene ID in DB_Object_ID field) or annotations to a protein object (protein ID in DB_Object_ID field). The entry in the DB_Object_Symbol field should be a symbol that means something to a biologist, wherever possible (gene symbol, for example). It is not an ID or an accession number (the second column, DB_Object_ID, provides the unique identifier), although IDs can be used in DB_Object_Symbol if there is no more biologically meaningful symbol available (e.g., when an unnamed gene is annotated). The object type (gene, transcript, protein, protein_structure, or complex) listed in the DB_Object_Type field MUST match the database entry identified by DB_Object_ID. Note that DB_Object_Type refers to the database entry (i.e. does it represent a gene, protein, etc.); this column does not reflect anything about the GO term or the evidence on which the annotation is based. For example, if your database entry represents a gene, then 'gene' goes in the DB_Object_Type column, even if the annotation is to a component term relevant to the localization of a protein product of the gene. The text entered in the DB_Object_Name and DB_Object_Symbol can refer to the same database entry (recommended), or to a "broader" entity. For example, several alternative transcripts from one gene may be annotated separately, each with a unique transcript DB_Object_ID, but list the same gene symbol in the DB_Object_Symbol column.

DB refers to the database contributing the gene_association file the value must be present in the file of database abbreviations. [Database abbreviations explanation] this field is mandatory, cardinality 1 DB_Object_ID unique identifier in DB for the item being annotated this field is mandatory, cardinality 1 DB_Object_Symbol (unique and valid) symbol to which DB_Object_ID is matched can use ORF name for otherwise unnamed gene or protein if gene products are annotated, can use gene product symbol if available, or many gene product annotation entries can share a gene symbol this field mandatory, card 1 Qualifier flags that modify interpretation of an annotation 1 (or more) of NOT, contributes_to, colocalizes_with field not mandatory; cardinality 0, 1, >1; for cardinality >1 use a pipe to separate entries (e.g. NOT|contributes_to) GOid GO identifier for the term attributed to the DB_Object_ID this field is mandatory, cardinality 1 DB:Reference one or more unique identifiers for a single source cited as an authority for the attribution of the GOid to the DB_Object_ID. This may be a literature reference or a database record. The syntax is DB:accession_number. Note that only one reference can be cited on a single line in the gene_association file. If a reference has identifiers in more than one database, multiple identifiers for that reference can be included on a single line. For example, if the reference is a published paper that has a PubMed ID, we strongly recommend that the PubMed ID be included, as well as an identifier within a model organism database. Note that if the model organism database has an identifier for the reference, that idenitifier should always be included, even if a PubMed ID is also used. this field is mandatory, cardinality 1, >1; for cardinality >1 use a pipe to separate entries (e.g. SGD:8789|PMID:2676709).database abbreviations[Database abbreviations explanation] The flat file format comprises 15 tab-delimited fields. Blue denotes required fields: Evidence: IMP, IGI, IPI, ISS, IDA, IEP, IEA, TAS, NAS, ND, IC, RCA this is mandatory, cardinality 1 With (or) From one of: DB:gene_symbol; DB:gene_symbol[allele_symbol]; DB:gene_id; DB:protein_nam; DB:sequence_id; GO:GO_id. this field is not mandatory (except in the case of IC evidence code), cardinality 0, 1, >1; for cardinality >1 use a pipe to separate entries (e.g. CGSC:pabA|CGSC:pabB). Note: This field is used to hold an additional identifier for annotations using certain evidence codes (IC, IEA, IGI, IPI, ISS). For example, it can identify another gene product to which the annotated gene product is similar (ISS) or interacts with (IPI). More information on the meaning of 'with/from' column entries is available in the evidence documentation entries for the relevant codes. Cardinality = 0 is not recommended, but is permitted because cases can be found in literature where no database identifier can be found (e.g. physical interaction or sequence similarity to a protein, but no ID provided). Annotations where evidence is IGI, IPI, or ISS and 'with' cardinality = 0 should link to an explanation of why there is no entry in 'with.' Cardinality may be >1 for any of the evidence codes that use 'with'; for IPI and IGI cardinality >1 has a special meaning. For cardinality >1 use a pipe to separate entries (e.g. FB:FBgn1111111|FB:FBgn2222222). Note that a gene ID may be used in the 'with' column for a IPI annotation, or for an ISS annotation based on amino acid sequence or protein structure similarity, if the database does not have identifiers for individual gene products. A gene ID may also be used if the cited reference provides enough information to determine which gene ID should be used, but not enough to establish which protein ID is correct. 'GO:GO_id' is used only when the evidence code is 'IC', and refers to the GO term(s) used as the basis of a curator inference. In these cases the entry in the 'DB:Reference' column will be that used to assign the GO term(s) from which the inference is made. This field is mandatory for evidence code IC. The ID is usually an identifier for an individual entry in a database (such as a sequence ID, gene ID, GO ID, etc.). Identifiers from the Center for Biological Sequence Analysis (CBS), however, represent tools used to find homology or sequence similarity; these identifiers can be used in the 'with' column for ISS annotations. 'with' col may not be used with evidence codes IDA, TAS, NAS, NDevidence documentation

The flat file format comprises 15 tab-delimited fields. Blue denotes required fields: Aspect one of P (biological process), F (molecular function) or C (cellular component) this field is mandatory; cardinality 1 DB_Object_Name name of gene or gene product. not mandatory, cardinality 0, 1 [white space allowed] Synonym Gene_symbol [or other text]. Strongly recommend gene synonyms are included in the gene association file, as this aids the searching of GO. this field is not mandatory, cardinality 0, 1, >1 [white space allowed]; for cardinality >1 use a pipe to separate entries (e.g. YFL039C|ABY1|END7|actin gene) DB_Object_Type what kind of thing is being annotated one of gene, transcript, protein, protein_structure, complex this field is mandatory, cardinality 1 Taxon taxonomic identifier(s). For cardinality 1, the ID of the species encoding the gene product. For cardinality 2, to be used only in conjunction with terms that have the term 'interaction between organisms' as an ancestor. The first taxon id should be that of the organism encoding the gene or gene product, and the taxon id after the pipe should be that of the other organism in the interaction. mandatory, cardinality 1, 2; for cardinality 2 use a pipe to separate entries (e.g. taxon:1|taxon:1000) Date: on which the annotation was made; format is YYYYMMDD this field is mandatory, cardinality 1 Assigned_by The database which made the annotation one of the values in the table of database abbreviations. [Database abbreviations explanation] Used for tracking the source of an individual annotation. Default value is value entered in column 1 (DB). Value will differ from column 1 for any that is made by one database and incorporated into another. this field is mandatory, cardinality 1database abbreviations[Database abbreviations explanation] Note that several fields contain database cross-reference (dbxrefs) in the format dbname:dbaccession. The fields are: GOid (where dbname is always GO), DB:Reference, With, Taxon (where dbname is always taxon). For GO ids, do not repeat the 'GO:' prefix (i.e. always use GO:0000000, not GO:GO:0000000)

Computational Annotation Methods This section includes descriptions of automated annotation methods used by participating databases (descriptions have been provided by each group listed). EBI | MGI | TIGREBIMGITIGR EBI GOA Electronic Annotation The large-scale assignment of GO terms to UniProt Knowledgebase entries involves electronic techniques. This strategy exploits existing properties within database entries including keywords and Enzyme Commission (EC) numbers and cross-reference to InterPro (a database of protein motifs) which are manually mapped to GO. SWISS-PROT keyword and InterPro to GO mappings are maintained in-house and shared on the GO home page for local database updates. Electronically combining these mappings with a table of matching Uniprot Knowledgebase entries generates a table of associations. For each GOA association, an evidence code, which summarizes how the association is made is provided. Associations are made electronically are labeled as 'inferred from electronic annotation' (IEA). Evelyn Camon, 2002-09-03Enzyme Commission (EC)InterPro MGI Electronic Annotation Methods Every object in the MGI databases (markers, seqids, references, etc.) has an MGI: accession ID. See details in GO TIGR ISS Annotation (Arabidopsis, T. brucei) For TIGR Arabidopsis or T. brucei annotations using 'Inferred from Sequence Similarity' (ISS) evidence, the reference is usually 'TIGR_Ath1:annotation' for Arabidopsis (author: TIGR Arabidopsis annotation team) and TIGR_Tba1:annotation for T. brucei (author: TIGR Trypanosoma brucei annotation team), which are defined as follows: name: TIGR annotation based upon multiple sources of similarity evidence description: TIGR_Ath1:annotation or TIGR_Tba1:annotation denotes a curator's interpretation of a combination of evidence. Our internal software tools present us with a great deal of evidence based domains, sequence similarities, signal sequences, paralogous proteins, etc. The curator interprets the body of evidence to make a decision about a GO assignment when an external reference is not available. The curator places one or more accessions that informed the decision in the "with" field. What this says is that we have used many sequence similarity hits, etc., to make our decision. However, we choose only 1-3 pieces of information as "with" information, as it is not practical to enter and submit many entries for each annotation. We also have internal calculations of paralogy and new domains we are identifying which have not yet been published, but which help inform our decisions.

Clustering Methods Clustering is partitioning into mutually exclusive and collectively exhaustive subsets, such that each point is: very similar to (close to) the other points in its component and very dissimilar to (far from) the points in the other components. A Categorization of Major Clustering Methods Partitioning methods (K-means | K-medoids...) Hierarchical methods (Agnes, Diana...) Density-based methods Grid-based methods Model-based methods

The K-Means Clustering Method Given k, the k-means algorithm is implemented in 4 steps (assumes partitioning criteria is: maximize intra-cluster similarity and minimize inter-cluster similarity. Of course, a heuristic is used. Method isn’t really an optimization) 1.Partition objects into k nonempty subsets (or pick k initial means). 2.Compute the mean (center) or centroid of each cluster of the current partition (if one started with k means, then this step is done). centroid ~ point that minimizes the sum of dissimilarities from the mean or the sum of the square errors from the mean. Assign each object to the cluster with the most similar (closest) center. 3.Go back to Step 2 4.Stop when the new set of means doesn’t change (or some other stopping condition?)

k-Means Clustering annimation 0 1 2 3 4 5 6 7 8 9 10 0123456789 0 1 2 3 4 5 6 7 8 9 0123456789 Step 1 Step 2 Step 3 Step 4 Strength relatively efficient: O(tkn), n is # objects, k is # clusters t is # iterations. Normally, k, t << n. Weakness Applicable only when mean is defined (e.g., a vector space). Need to specify k, the number of clusters, in advance. It is sensitive to noisy data and outliers.

The K-Medoids Clustering Method Find representative objects, called medoids, (must be an actual object in the cluster, where as the mean seldom is). PAM (Partitioning Around Medoids, 1987) –starts from an initial set of medoids –iteratively replaces one of the medoids by a non-medoid –if it improves the aggregate similarity measure, retain the swap. Do this over all medoid-nonmedoid pairs –PAM works for small data sets. Does not scale for large data sets CLARA (Clustering LARge Applications) (Kaufmann,Rousseeuw, 1990) Sub-samples then apply PAM CLARANS (Clustering Large Applications based on RANdom Search) (Ng & Han, 1994): Randomized the sampling

Hierarchical Clustering Methods: AGNES (Agglomerative Nesting) Introduced in Kaufmann and Rousseeuw (1990) Use the Single-Link (distance between two sets is the minimum pairwise distance) method Other options are complete link (distance is max pairwise); average,... Merge nodes that are most similarity Eventually all nodes belong to the same cluster

DIANA (Divisive Analysis) Introduced in Kaufmann and Rousseeuw (1990) Inverse order of AGNES (initially all objects are in one cluster; then it is split according to some criteria (e.g., maximize some aggregate measure of pairwise dissimilarity again) Eventually each node forms a cluster on its own

Contrasting hierarchical Clustering Techniques Partitioning algorithms: Partition a dataset to k clusters, e.g., k=3  Hierarchical alg: Create hierarchical decomposition of ever-finer partitions. e.g., top down (divisively). bottom up (agglomerative)

Hierarchical Clustering a b Step 1 d e Step 2 c d e Step 3 a b c d e Step 4 b d c e a Step 0 Agglomerative

Hierarchical Clustering (top down) In either case, one gets a nice dendogram in which any maximal anti- chain (no 2 nodes linked) is a clustering (partition). b a Step 4 d e Step 3 c d e Step 2 a b c d e Step 1 a b c d e Step 0 Divisive

Hierarchical Clustering (Cont.) Recall that any maximal anti-chain (maximal set of nodes in which no 2 are chained) is a clustering (a dendogram offers many).          

Hierarchical Clustering (Cont.) But the “horizontal” anti-chains are the clusterings resulting from the top down (or bottom up) method(s).

Data Mining Summary Data Mining on a given table of data includes Association Rule Mining (ARM) on Bipartite Relationships Clustering Partitioning methods (K-means | K-medoids...), Hierarchical methods (Agnes, Diana...), Model-based methods (K-Means, K-Medoids..),.... Classification Decision Tree Induction, Bayesian, Neural Network, k-Nearest-Neighbor,...) But most data mining is on a database, not just one table, that is, often times, first one must apply the appropriate SQL query to a database to get the table to be data mined. The next slides discuss vertical data methods for doing that. You may wish to skip this material if not interested in the topic.

Vertical Select-Project-Join (SPJ) Queries A Select-Project-Join query has joins, selections and projections. Typically there is a central fact relation to which several dimension relations are to be joined (standard STAR DW) E.g., Student(S), Course(C), Enrol(E) STAR DB below (bit encoding is shown in reduced font italics for certain attributes) S|s____|name_|gen| C|c____|name|st|term| E|s____|c____|grade | |0 000|CLAY |M 0| |0 000|BI |ND|F 0| |0 000|1 001|B 10| |1 001|THAIS|M 0| |1 001|DB |ND|S 1| |0 000|0 000|A 11| |2 010|GOOD |F 1| |2 010|DM |NJ|S 1| |3 011|1 001|A 11| |3 011|BAID |F 1| |3 011|DS |ND|F 0| |3 011|3 011|D 00| |4 100|PERRY|M 0| |4 100|SE |NJ|S 1| |1 001|3 011|D 00| |5 101|JOAN |F 1| |5 101|AI |ND|F 0| |1 001|0 000|B 10| |2 010|2 010|B 10| |2 010|3 011|A 11| |4 100|4 100|B 10| |5 101|5 101|B 10| Vertical bit sliced (uncompressed) attrs stored as: S.s 2 S.s 1 S.s 0 S.gC.c 2 C.c 1 C.c 0 C.tE.s 2 E.s 1 E.s 0 E.c 2 E.c 1 E.c 0 E.g 1 E.g 0 0000000000000110 0010001100000011 1000100100101100 1011101000100010 0101010101101111 0111011001101100 01001010 01000111 10010010 10110110 Vertical (un-bit-sliced) attributes are stored: S.name C.name C.st |CLAY | |BI | |ND| |THAIS| |DB | |ND| |GOOD | |DM | |NJ| |BAID | |DS | |ND| |PERRY| |SE | |NJ| |JOAN | |AI | |ND|

O :o c r |0 000|0 00|0 01| |1 001|0 00|1 01| |2 010|1 01|0 00| |3 011|1 01|1 01| |4 100|2 10|0 00| |5 101|2 10|2 10| |6 110|2 10|3 11| |7 111|3 11|2 10| C:c n cred |0 00|B|1 01| |1 01|D|3 11| |2 10|M|3 11| |3 11|S|2 10| Vertical preliminary Select-Project-Join Query Processing (SPJ) R:r cap |0 00|30 11| |1 01|20 10| |2 10|30 11| |3 11|10 01| SELECT S.n, C.n FROM S, C, O, R, E WHERE S.s=E.s & C.c=O.c & O.o=E.o & O.r=R.r & S.g=M & C.r=2 & E.g=A & R.c=20; S:s n gen |0 000|A|M| |1 001|T|M| |2 100|S|F| |3 111|B|F| |4 010|C|M| |5 011|J|F| E:s o grade |0 000|1 001|2 10| |0 000|0 000|3 11| |3 011|1 001|3 11| |3 011|3 011|0 00| |1 001|3 011|0 00| |1 001|0 000|2 10| |2 010|2 010|2 10| |2 010|7 111|3 11| |4 100|4 100|2 10| |5 101|5 101|2 10| S.s 2 0 1 0 E.s 2 0 1 C.c 1 0 1 R.r 1 0 1 S.s 1 0 1 S.s 0 0 1 0 1 0 1 S.n A T S B C J S.g M F M F C.c 0 0 1 0 1 C.n B D M S C.r 1 0 1 C.r 0 1 0 R.r 0 0 1 0 1 R.c 1 1 0 R.c 0 1 0 1 E.s 1 0 1 0 1 0 E.s 0 0 1 0 1 E.o 2 0 1 E.o 1 0 1 0 1 0 E.o 0 1 0 1 0 1 0 1 E.g 1 1 0 1 E.g 0 0 1 0 1 0 In the SCORE database (Students, Courses, Offerings, Rooms, Enrollments), numeric attributes are represente vertically as P-trees (not compressed). Categorical are projected to a 1 column vertical file O.o 2 0 1 O.o 1 0 1 0 1 O.o 0 0 1 0 1 0 1 0 1 O.c 1 0 1 O.c 0 0 1 0 1 O.r 1 0 1 O.r 0 1 0 1 0 1 0 decimal binary.

SM 1 0 1 0 For selections, S.g=M=1 b C.r=2=10 b E.g=A=11 b R.c=20=10 b create the selection masks using ANDs and COMPLEMENTS. S.s 2 0 1 0 S.s 1 0 1 S.s 0 0 1 0 1 0 1 S.n A T S B C J S.g 1 0 1 0 E.s 2 0 1 E.s 1 0 1 0 1 0 E.s 0 0 1 0 1 E.o 2 0 1 0 1 E.o 1 0 1 0 1 0 E.o 0 1 0 1 0 1 0 1 E.g 1 1 0 1 E.g 0 0 1 0 1 0 C.c 1 0 1 C.c 1 0 1 0 1 C.n B D M S C.r 1 0 1 C.r 2 1 0 O.o 2 0 1 O.o 1 0 1 0 1 O.o 0 0 1 0 1 0 1 0 1 O.c 1 0 1 O.c 0 0 1 0 1 O.r 1 0 1 O.r 0 1 0 1 0 1 0 R.r 1 0 1 R.r 0 0 1 0 1 R.c 1 1 0 R.c 0 1 0 1 SELECT S.n, C.n FROM S, C, O, R, E WHERE S.s=E.s & C.c=O.c & O.o=E.o & O.r=R.r & S.g=M & C.r=2 & E.g=A & R.c=20; C.r 1 0 1 C.r’ 2 0 1 Cr2 0 1 E.g 1 1 0 1 E.g 0 0 1 0 1 0 EgA 0 1 0 1 0 R.c 1 1 0 R.c’ 0 0 1 0 Rc20 0 1 0 Apply these selection masks (Zero out numeric values, blanked out others). S.s 2 0 S.s 1 0 1 0 S.s 0 0 1 0 S.n A T C E.s 2 0 E.s 1 0 1 0 1 0 E.s 0 0 1 0 E.o 2 0 1 0 E.o 1 0 1 0 E.o 0 0 1 0 1 0 C.c 1 0 1 C.c 0 0 1 C.n S O.o 2 0 1 O.o 1 0 1 0 1 001010101001010101 O.c 1 0 1 O.c 0 0 1 0 1 O.r 1 0 1 O.r 0 1 0 1 0 1 0 R.r 1 0 R.r 0 0 1 0

SELECT S.n, C.n FROM S, C, O, R, E WHERE S.s=E.s & C.c=O.c & O.o=E.o & O.r=R.r & S.g=M & C.r=2 & E.g=A & R.c=20; S.s 2 0 S.s 1 0 1 0 S.s 0 0 1 0 S.n A T C E.s 2 0 E.s 1 0 1 0 1 0 E.s 0 0 1 0 E.o 2 0 1 0 E.o 1 0 1 0 E.o 0 0 1 0 1 0 C.c 1 0 1 C.c 0 0 1 C.n S O.o 2 0 1 O.o 1 0 1 0 1 O.o 0 0 1 0 1 0 1 0 1 O.c 1 0 1 O.c 0 0 1 0 1 O.r 1 0 1 O.r 0 1 0 1 0 1 0 R.r 1 0 R.r 0 0 1 0 For the joins, S.s=E.s C.c=O.c O.o=E.o O.r=R.r, one approach is to follow an indexed nested loop like method. (Noting that attribute P-trees ARE an index for that attribute). The join O.r=R.r is simply part of a selection on O (R doesn’t contribute output nor participate in any further operations) Use the Rc20-masked R as the outer relation Use O as the indexed inner relation to produce that O-selection mask. Rc20 0 1 0 Get 1 st R.r value, 01 b (there's only 1) Mask the O tuples: P O.r 1 ^P’ O.r 0 O.r 1 0 1 O’.r 0 0 1 0 1 0 1 OM 0 1 0 1 This is the only R.r value (if there were more, one would do the same for each, then OR those masks to get the final O-mask). Next, we apply the O-mask, OM to O O.o 2 0 1 0 1 O.o 1 0 1 O.o 0 0 1 0 1 O.c 1 0 1 0 1 O.c 0 0 1

SELECT S.n, C.n FROM S, C, O, R, E WHERE S.s=E.s & C.c=O.c & O.o=E.o & O.r=R.r & S.g=M & C.r=2 & E.g=A & R.c=20; S.s 2 0 S.s 1 0 1 0 S.s 0 0 1 0 S.n A T C E.s 2 0 E.s 1 0 1 0 1 0 E.s 0 0 1 0 E.o 2 0 1 0 E.o 1 0 1 0 E.o 0 0 1 0 1 0 C.c 1 0 1 C.c 0 0 1 C.n S For the final 3 joins C.c=O.c O.o=E.o E.s=S.s the same indexed nested loop like method can be used. O.o 2 0 1 0 1 O.o 1 0 1 O.o 0 0 1 0 1 O.c 1 0 1 0 1 O.c 0 0 1 Get 1 st masked C.c value, 11 b Mask corresponding O tuples: P O.c 1 ^P O.c 0 O.c 1 0 1 0 1 O.c 0 0 1 OM 0 1 Get 1 st masked O.o value, 111 b Mask corresponding E tuples: P E.o 2 ^P E.o 1 ^P E.o 0 E.o 1 0 1 0 E.o 0 0 1 0 1 0 Get 1 st masked E.s value, 010 b Mask corresponding S tuples: P’ S.s 2 ^P S.s 1 ^P’ S.s 0 S’.s 2 1 0 1 0 S.s 1 0 1 0 S’.s 0 1 0 1 0 SM 0 1 0 Get S.n-value(s), C, pair it with C.n-value(s), S, output concatenation, C.n S.n There was just one masked tuple at each stage in this example. In general, one would loop through the masked portion of the extant domain at each level (thus, Indexed Horizontal Nested Loop or IHNL) E.o 2 0 1 0 EM 0 1 0 S C

Vertical Select-Project-Join-Classification Query Given previous SCORE Training Database (not presented as just one training table), predict what course a male student will register for, who got an A in a previous course in Room with a capacity of 20. This is a matter of applying the previous complex SPJ query first to get the pertinent Training table and then classifying the above unclassified sample (e.g., using, 1-nearest neighbour classification). The result of the SPJ is the single row Training Set, (S,C) and so the prediction is course=C.

10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather.

Similar presentations

Presentation on theme: "10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather.

Similar presentations

Presentation on theme: "10. Data Mining Data Mining is one aspect of Database Query Processing (on the "what if" or pattern and trend querying end of Query Processing, rather."— Presentation transcript:

Similar presentations

About project

Feedback