Applications of one-class classification -- searching for comparable applications for negative selection algorithms

background Purpose: looking for real world applications that demonstrates the usage of V-detector (a negative selection algorithm) One-class classification problem: Different from conventional classification: only information of one of the classes (target class) is available Original application: anomaly (outliner) detection More about the application than whatever classification methods involved

One-class Classification Basic concept of classification: A classifier is a function which outputs a class label from each input object. It cannot be constructed from known rules. In pattern recognition or machine learning: inferring a classifier (a function) from a set of training examples. Usually, the type of function is chosen beforehand and parameters are to be determined. Line classifier, mixture of Gaussians, neural networks, support vector classifiers

One-class Classification Basic concept of classification: Assumptions: continuity, enough information (amount of samples, limited noise), etc. Multi-class classification can be decomposed into two-class classifications

One-class classification Same problems as conventional classification Definition of errors Atypical training data Measuring the complexity of a solution The curse of dimensionality The generalization of the method

A conventional and a one-class classifier applied to an example dataset containing apples and pears, represented by 2 features per object. The solid line is the conventional classifier which distinguishes between the apples and pears, while the dashed line describes the dataset. This description can identify the outlier apple in the lower right corner, while the classifier will just classify it as an pear.

One-class classification Additional problems Most conventional classifier’s assumption that more or less balanced data. Hard to decide on the basis of on class how tightly the boundary should fit in each direction around the data. Hard to find which features should be used to find the best separation. Impossible to estimate false positives. More prominent curse of dimension. Extra constraints: closed boundary etc.

Various techniques Generate outliner detection Some methods requires near-target objects; Density method: directly estimating the density of target objects some works requires density estimate in the complete feature space Typical sample is assumed Reconstruction methods: based on prior knowledge Boundary methods Well defined distance

Application 1: texture classification Problem: classification of texture images polished granite (or ceramic) tiles that are widely used as construction elements The polished granite tiles are usually inspected by a human expert using a chosen master tile as the reference. Such inspection is subjective and qualitative One-class classifier is suitable: Outliners cannot be used to train any methods Recent development quasi-statistical representation of binary images used as a feature space for texture image classification

Based on CCR feature space (coordinated cluster representation) Outline of the method: Given a master texture image of a class, estimate statistics of CCR histogram Use parameters of the statistics to define a closed decision boundary.

Master images

CCR feature space A binary image intensity: Sa={sa{l,m}}, where l=1, 2, …L and m=1, 2, …, M A rectangular window W = I X J Scan all over the image with one pixel steps using that window The number of all possible state of the window is 2w Coordinated clusters representation consists of a histogram Ha(I,J)(b) a is the index of the image (I,J) indicated the size of the window b = 1, 2, …, 2w

When a histogram is normalized, it is considered as a proability distribution function of occurrence Fa(I,J)(b) = 1/A Ha(I,J)(b) Where A = (L-I+1)X(M-J+1) Histogram H contains all the information about n-point correlation moments of the image if and only if the separation vectors between n pixels fit between the scanning window In general, when the order of statistics is higher, more structural information is available There is a structural correspondence between a gray level image and its thresholded counterpart Provided that the binary image keeps enough structural information about a primary gray level image to be classified, the CCR of a binary image is highly suitable for recognition and classification of gray level texture image

Framework of classification Training phase a set of gray level image from each texture class Each threshold Calculate CCR distribution function Recognition phase Input test image Thresholded CCR distribution Compare with prototypes and assign to the class of best match One-class classification Define the limits of feature variations Establish the criterion

Thresholding (binarization) Because CCR is defined for binary image Fuzzy C-Means clustering method

Training phase assuming Q images of a class are available, a random set of P subimages is sampled If only one image is available, Q independent random sets are sampled Five measurements are calculated from distribution function Fa F: mass center of subimages (not a value, still a function or histogram) D: mean of distance (“distance” refers to the mean distance within a set) s: mean of standard deviation (“standard deviation” refers to the standard deviation of a set) D: mean of q-th sample center to the center of all samples s2: variance of each sets with regard to center of samples

C is the emprical adjustment parameter Criterion d(Ftest, F) < D+Cs D-2s<Dtest<D+2s C is the emprical adjustment parameter Ftest and Dtest are the mean of K ransom subimage of the texture image to be classified L1 distance is used as the measures of distinction d(Fa,Fb) = Sb|Fa(b)-Fb(b)|

Results C should be in the range of 1, 2, …, 20 Based on observation that s is approximately ten times less than D 8 master images (training data) plus 16 testing images are used (128X128). For C=1 or 2, only the master images are recognized More images are recognized for larger C For C<19, no mis-classification Proper C depends on the size of subimage (32, 24, 64 are discussed)

Application 2: authorship Problem: authorship verification Different from standard text categorization problem No realistic to train with negative samples Difference from other one-class classification Negative samples are not lacked – hard to choose to represent the entire class The object texts are long We can chunk to multiple samples – a set instead of single instance

New idea: Depth of difference between two sets Test the rate of degradation of accuracy as the best features are iteratively dropped

Standard method Choose a feature set: frequencies of function words, syntactic structures, parts-of-speech n-grams, complexity and richness measure, syntactic and orthographic idiosyncrasies Note: very different from text categorization by topic Having constructed feature vectors, use learning algorithm to construct distinguishing model Similar to categorization by topic Liner separators are believed to work well Assessment: k-fold cross-validation or bootstrapping

One-class scenario Naïve approach New approach: unmasking Chunk two works to generate two sufficient large sets Test if we can distinguish using cross-validation with high accuracy Failed in experiment (different works are just different enough to tell) New approach: unmasking In the above approach, a small number of features are doing all the work. They are likely to be from thematic differences, difference in genre or purpose, chronological shift of style, deliberate attempt to mask identity Unmasking: removing features that are most useful to distinguish Hypothesis: if they are by the same author, difference will be refelected in only a relative small number of features Sudden degradation shows the same author

Results: Corpus: 21 19th century English iterature Baseline: one-class SVM Extension: using negative samples to eliminate false positive Solution to a literary mystery: the case of the bashful rabbi

bibliography D. M. J. Tax, “One-class classification”, PhD thesis, 2001 D. M. J. Tax, “Data description toolbox: A Matlab toolbox for data description, outlier and novelty detection”. 2005 M. Koppel and J. Schler, Authorship verification as a one-class classification problem, in Proceedings of 21st International Conference on Machine Learning, 2004. R.E.Sanchez-Yanez et al, One-class texture classifier in the CCRfeature space, Pattern Recognition Letter, 24, 2003. R.E.Sanchez-Yanez et al, A framework for texture classification using the coordinated clusters representation, Pattern Recognition Letter, 24, 2003.