1. Data Science Input: Concepts, Instances and Attributes WFH: Data Mining, Chapter 2
Rodney Nielsen
Many/most of these slides were adapted from: I. H. Witten, E. Frank and M. A. Hall

2. Input: Concepts, Instances, Attributes
Preparing the input
Missing values
Getting to know data

3. Metadata Information about the data that encodes background knowledge
Can be used to restrict search space
Examples:
Dimensional considerations (i.e. expressions must be dimensionally correct)
Circular orderings (e.g. degrees in compass)
Partial orderings (e.g. generalization/specialization relations)

4. Preparing the Input Denormalization Other Issues
Problem: different data sources (e.g. sales department, customer billing department, …)
Differences: styles of record keeping, conventions, time periods, data aggregation, primary keys, errors
Data must be assembled, integrated, cleaned up
“Data warehouse”: consistent point of access
External data may be required (“overlay data”)
Critical: type and level of data aggregation

5. The ARFF Format %
% ARFF file for weather data with some numeric features
@relation weather
@attribute outlook {sunny, overcast, rainy}
@attribute temperature numeric
@attribute humidity numeric
@attribute windy {true, false}
@attribute play? {yes, no}
@data
sunny, 85, 85, false, no
sunny, 80, 90, true, no
overcast, 83, 86, false, yes
...

6. Sparse Data
In some applications most attribute values in a dataset are zero
E.g.: word counts in a text categorization problem
ARFF supports sparse data
0, 26, 0, 0, 0 ,0, 63, 0, 0, 0, “class A”
0, 0, 0, 42, 0, 0, 0, 0, 0, 0, “class B”
{1 26, 6 63, 10 “class A”}
{3 42, 10 “class B”}

7. Missing Values Frequently indicated by out-of-range entries
Types: unknown, unrecorded, irrelevant
Reasons:
Malfunctioning equipment
Changes in experimental design
Collation of different datasets
Measurement not possible
Missing value may have significance in itself (e.g. missing test in a medical examination)
Most schemes assume there are no missing values
Might need to be coded as additional value

8. Inaccurate Values Reason: data has not been collected for mining
Result: errors and omissions that don’t affect original purpose of data (e.g. age of customer)
Typographical errors in nominal attributes  values need to be checked for consistency
Typographical and measurement errors in numeric attributes  outliers need to be identified
Errors may be deliberate (e.g. wrong zip codes)
Other problems: duplicates, stale data

9. Getting to Know the Data
Simple visualization tools are very useful
Nominal attributes: histograms (Distribution consistent with background knowledge?)
Numeric attributes: graphs (Any obvious outliers?)
2-D and 3-D plots show dependencies
Need to consult domain experts
Too much data to inspect? Take a sample!

10. Data Transformations Attribute selection Dirty data
Scheme-independent, scheme-specific
Dirty data
Data cleansing, robust regression, anomaly detection

11. Just Apply a Learner? Scheme/parameter selection
treat selection process as part of the learning process
Modifying/creating the input:
Feature engineering to make learning possible or easier

12. Attribute Selection
Adding a random (i.e. irrelevant) attribute can significantly degrade C4.5’s performance
Problem: attribute selection based on smaller and smaller amounts of data
IBL very susceptible to irrelevant attributes
Number of training instances required increases exponentially with number of irrelevant attributes
Naïve Bayes doesn’t have this problem
Relevant attributes can also be harmful

13. Scheme-independent Attribute Selection
Filter approach: assess based on general characteristics of the data
One method: find smallest subset of attributes that separates data
Another method: use different learning scheme
E.g., Use attributes selected by c4.5, or coefficients of linear model, possibly applied recursively (recursive feature elimination)
IBL-based attribute weighting techniques
Can’t easily find redundant attributes
Correlation-based Feature Selection (CFS)
Correlation between attributes measured by symmetric uncertainty
Goodness of subset of attributes measured by (breaking ties in favor of smaller subsets)
Add formulas

14. Attribute Subsets for Weather Data

15. Searching Attribute Space
Number of attribute subsets is exponential in number of attributes
Common greedy approaches:
forward selection
backward elimination
More sophisticated strategies:
Bidirectional search
Best-first search: can find optimum solution
Beam search: approximation to best-first search
Genetic algorithms

16. Scheme-specific Selection
Wrapper approach to attribute selection
Implement “wrapper” around learning scheme
Evaluation criterion: cross-validation performance
Time consuming
greedy approach, k attributes  k2  time
prior ranking of attributes  linear in k
Can use significance test to stop cross-validation for subset early if it is unlikely to “win” (race search)
can be used with forward, backward selection, prior ranking, or special-purpose schemata search

17. Student Questions: Feature Selection
In attribute selection, less attributes means a smaller space to search during model construction and less opportunities to make wrong decisions and arrive at misleading, insufficiently justified generalizations. Computationally, wouldn't the actual model building in big data be demanding? Since this model is motivated by computational savings.

18. Student Questions: Feature Engineering
Throughout section 7.1 selecting which attributes to use in the machine learning algorithm is discussed in detail but I am curious if there is a specific way to generate attributes or if that is wholly decided upon by the data scientist and specialists in the field.

19. Automatic Data Cleansing
To improve a decision tree:
Remove misclassified instances, then re-learn!
Better (of course!):
Human expert checks misclassified instances
Attribute noise vs class noise
Attribute noise should be left in training set
When? Pros and Cons?
Don’t train on clean set and test on dirty one!
Systematic class noise (e.g. one class substituted for another): leave in training set?
Pros and Cons?
Unsystematic class noise: eliminate from training set, if possible

20. Robust Regression
“Robust” statistical method  one that addresses problem of outliers
To make regression more robust:
Minimize absolute error, not squared error
Remove outliers (e.g. 10% of points farthest from the regression plane)
Minimize median instead of mean of squares (copes with outliers in x and y direction)
Finds narrowest strip covering half the observations

21. Detecting Anomalies Visualization can help to detect anomalies
Automatic approach: committee of different learning schemes
E.g.
decision tree
nearest-neighbor learner
linear discriminant function
Conservative approach: delete instances incorrectly classified by all of them
Problem: might sacrifice instances of small classes

22. One-Class Learning Usually training data is available for all classes
Some problems exhibit only a single class at training time
Test instances may belong to this class or a new class not present at training time
One-class classification
Predict either target or unknown
Some problems can be re-formulated into two-class ones
Other applications truly don't have negative data

23. Outlier Detection
Outlier/novelty detection is sometimes called one-class classification
Generic approach: identify outliers as instances that lie beyond distance d from percentage p of the training data
Alternatively, estimate density of the target class and mark low probability test instances as outliers
Threshold can be adjusted to obtain a suitable rate of outliers

24. Generating Artificial Data
Another possibility is to generate artificial data for the outlier class
Can then apply any off-the-shelf classifier
Can tune rejection rate threshold if classifier produces probability estimates
Generate uniformly random data
Curse of dimensionality – as # attributes increase it becomes infeasible to generate enough data to get good coverage of the space

25. Questions

26. Student Questions
7.1 mentions that there have been attempts to come up with universally acceptable measures or terms for relevance. What would be an example?
How can you systematically add noise to the training set so that it correctly models noise in the test (and hopefully real-world) data?
In robust regression why is the least squares regression line affected by anomalous data affected the most?
If there a way to remove classification errors in the training set due to noise other than by going through each entry manually?
In regards to sparse data, can't a high enough amount of missing data invalidate any experiments or results gained from the data?
In outlier detection, how do you distinguish an outlier from noisy data?

27. Student Questions
What is the benefit of the ARFF format over other filer formats (ex. XML)?
What are some methods for editing the training set for misspelled words or synonyms?
What is the goal when choosing which attributes to branch on in a decision tree?
Naive Bayes works well with forward selection. How about backwards selection?

28. Student Questions
When preparing input why do in some cases is external data needed?
Although instances can be removed after learning from them to reduce overfitting and complexity of the algorithm couldn't leaving some of these incorrect instances in the data set cause the incorrect selection of attributes to use and therefor ruin the method of choosing incorrect instances. i.e. Wouldn't leaving incorrect instances in the training set cause the methods talked about in section 7.5 to be null.