1. Data, pre-processing and exploration
Chapter Three
Data, pre-processing and exploration

2. Chapter Overview Data, data types and operations
Properties of various data sets
Data source and data warehouse
Issues of data quality
Data pre-processing operations
Data summary and visualisation
Online analytic processing (OLAP)
Data exploration and visualisation in Weka

3. Data, Data Types and Operations
Data object and attributes
Data object or instance: individual independent recording of a real life object/event.
Characterised by its recorded values on a fixed set of features or attributes
Feature or attribute: a specific property or characteristic of the data object.
Measurement: assigning a valid value to an attribute according to an appropriate measurement scale.
Collection: collecting measurement results or recorded values

4. Data, Data Types and Operations
Data object and attributes (cont’d)
An example
123, “John Smith”, “03/02/1990”, 20, “male”, 1.82, 78
ID number, collected
Name
collected
Birthday collected
Age
calculated
Gender collected
Body height
measured
Body weight

5. Data, Data Types and Operations
Data object and attributes (cont’d)
Measurement and measurement errors
Precision: the closeness of measurements to one another, represented by the standard deviation of the measurements, e.g. repeated measure of body temperature
Bias: a systematic variation of measurements from the intended quantity measurement, only known when external reference available, e.g. bias in weight measure instrument
Accuracy: the closeness of the measure to the true value, indicated by the number of significant digits used in the measurement, e.g. measure of money: pound vs. penny
Collection errors
Incorrect data recording at the point of entry, e.g. “Hongpo Do” as for “Hongbo Du”

6. Data, Data Types and Operations
Attribute domain types and operations
Categorical/Qualitative types
Nominal, e.g. Gender (M, F)
A set of names: no concept of order nor difference
Operators applicable: =, 
1:1 transformation permissible, e.g. ID: 11  e901
Ordinal, e.g. Grade (A, B, C, D, E)
A set of names: with order but no concept of difference
Operator applicable: =, , <, >, , 
Order-preserving transformation permitted,
e.g. Grade: A  First, B  Second, C  Third, D  Pass, E  BarePass.

7. Data, Data Types and Operations
Attribute domain types and operations
Numeric/Quantitative types
Interval, e.g. Temperature in C
A set of numeric values: both order and difference exist
Operators applicable: =, , <, >, , , +, -
e.g. temperature (F and C), calendar year
Transformation new = a*old + b permitted, e.g. F  C
Ratio, e.g. Length
A set of numeric values: order, difference and ratio
The set has an absolute zero
Operator applicable: =, , <, >, , , +, -, , 
Transformation new = a*old permitted, e.g. meter  feet

8. Data Sets Various forms Table of records Ordered data Graph-based data
Relational table
Join of relational tables
Numerical spreadsheet (data matrix)
Boolean strings (document-term matrix)
Ordered data
Time series and temporal sequence
Data sequence
Spatial data
Graph-based data
Non record-based data

9. Data Sets Various forms (illustrated) GGTTCCGCCTTCAGCC CCGCGCCCGCAGGG…
Data Matrix
Relational Table
Transaction Database
Page1
link1
link2
Page2
link3
Page4
www
zzzz
Page3
xxxx
yyyy
Web Structure
Spatial Data
GGTTCCGCCTTCAGCC
CCGCGCCCGCAGGG…
Data Sequence

10. Data Sets
Properties
Type: file structure, e.g. ARFF for Weka, DAT for See5
Size: measured in terms of the total number of records or total number of bytes, e.g. small (MB), medium (GB) and large (TB)
Dimensionality: number of attributes
Sparsity:
Values are skewed to some extreme or sub-ranges
Asymmetric values (some are more important than others)
Resolution
Right level of data details
Related to the intended purpose

11. Data Sets Properties (example insurance data set) Type: ARFF
Dimensionality: 7
Asymmetric: Y/N
Skewed?
Resolution: detailed
Size: records

12. Data Source and Data Warehouse
Sources of data
Local data source available
Local operational systems from different departments
Third-party external data source
Enterprise/Organisational data warehouse
An organisational database for decision making
A central data repository separate from operational systems
Enforcing organisation-wide data consistency and integration
Providing data details as well as data summarisation
Providing data values as well as meta-data
Equipped with data analysis and reporting tools
As a data source for data mining

13. Data Source and Data Warehouse
Star schema for data warehouse
Central fact table
Dimension tables
Limited use of join operations
Part(p#, pname, weight, colour)
Supply(s#, p#, pj#, qty)
Supplier(s#, sname, city, status)
Project(pj#, jname, status, date)

14. Issues of Data Quality Main quality indicators
Accuracy: data recorded with sufficient precision and little bias
Correctness: data recorded without error and spurious objects
Completeness: any parts of data records missing
Consistency: compliance with established rules and constraints
Redundancy: unnecessary duplicates
Using the indicators to quantify quality of a data set
Improving quality if possible

15. Issues of Data Quality Some examples
Accuracy & correctness with the road accident reports in Exercise 1.3(c).
Completeness with the UK family expenditure surveys in Exercise 1.3(a).
Incompleteness introduced by data integration using outer join operation
Consistency in questionnaires, e.g. eating fruit & veg.
Q1: “give the fruit&veg portion consumed yesterday”: 2
Q2: “give the fruit&veg portion consumed today:” 3
Q3: “do you eat more today than yesterday?” No.
Redundancy in a local company’s database of 40,000 records about 15,000 client companies.

16. Issues of Data Quality Why is quality important?
“Garbage in, garbage out!”
Total data quality control requires a cultural change (comparing with total product quality control)
For data mining, tackling the quality issue at the data source cannot be always expected
By cleaning the data as much as possible
By developing and using more tolerate mining solutions
Data quality is relevant to the intended purpose of data mining, e.g. Do spelling errors in student names really matter when only the increase/decrease of student numbers in particular subject areas over the years is of interest?

17. Data Pre-processing Overview
Purpose: for speedy, cost-effective and high quality outcomes of data mining
Pre-processing tasks (not all are independent from each other)
Data aggregation
Data sampling
Dimension reduction
Feature selection
Feature creation
Discretisation/binarisation
Variable transformation
Dealing with missing values

18. Data Pre-processing Data aggregation
What: to summarise low level data details to higher level data abstraction
Why: to reduce the time of mining, to rescale data values, and to discover more stable patterns
How:
By generalisation using a given concept hierarchy
By applying aggregate functions (e.g. count, sum, average)
Dropping some attributes

19. Data Pre-processing Data sampling
What: selecting a subset of the given data set
Why: to make it possible to use sophisticated mining algorithms within a time limit.
Caution: the sample must be representative of the original data set
How:
Random sampling
Stratified sampling
Progressive sampling
With or without replacement
Data
population
Sampling
method
Selected
subset

20. Data Pre-processing Feature selection
What: reducing dimensionality by selecting a subset of attributes
Purposes:
To remove/reduce redundant features
To remove irrelevant features with no useful information for the mining task
How:
Manually with common sense and domain knowledge
Letting the mining solution to select suitable features (the embedded approach)
Filter and wrapper approaches
attributes
Subset
selection
One subset
evaluation
Stopping
criterion ok
Not ok
Selected
subset
Validate with
Mining task

21. Data Pre-processing Data dimension reduction
What: reduce redundancy implied among attributes
e.g. are all 9600 dimensions for a 120x80 pixel image necessary?
Curse of dimensions: as dimensionality increases
Data become more diverse, and any patterns are getting less significant and more peculiar.
The processing time may increase substantially.
Why: to reduce redundancy and effects of the curse
How:
Linear algebra techniques
Principal component analysis (PCA)
Independent component analysis (ICA)
Single value decomposition (SVD)
Feature selection (as described before)

22. Data Pre-processing Feature creation
What: to create a new set of features from the original features
Purpose: in the new feature space, meaningful and relevant patterns can be extracted more easily. The number of features may be reduced.
How:
Using feature extraction methods to extract new features from the existing ones, e.g. extracting colour, texture and shape from image of pixel values
Mapping data to a new space, e.g. wavelet transformation of pixel values of images to a frequency domain
Constructing new features from the existing ones using domain knowledge, e.g. using transaction dates to construct a new feature customer tenure that indicates the loyalty of the customer to the company

23. Data Pre-processing Data discretisation
What: to convert continuous attribute values to discrete categorical values
The purposes:
Requirement for some data mining solutions
Better data mining results (not always)
How:
Deciding how many categories to have and where split points should be
Mapping values to categories
Determine the number & locations of the split points  t1 t2 t3 t4
Mapping values within each sub-range to a category label 

24. Data Pre-processing Data discretisation (cont’d)
Discretisation methods:
Unsupervised: without concern to the outcome of a specific attribute, normally used for clustering and association rule mining
e.g. equal width, equal depth, clustering
Supervised: with respect to the outcome of the class attribute, normally used for classification
Simple methods: sorting according to the class attribute, and then discretising the attribute values for each class.
Sophisticated methods: the discretisation of the attribute values purifies the outcome of the class, e.g. using entropy to measure the degree of purity, and deciding the split points recursively, similar to decision tree induction
Merging methods, merging small intervals into a larger one with a stop criterion

25. Data Pre-processing Data binarisation
What: to convert discrete categorical values to binary Boolean attribute values
The purpose: the same as for discretisation
How:
Convert m categorical values to values in [0, m-1]
Convert each to binary number of n bits where n = log2m
Use m asymmetric binary variables to represent each of m values

26. Data Pre-processing Variable transformation
What: transform all values of an attribute to other values
The purposes:
Remove the effect of the outlier values
Make the result data visualisation more interpretable
Make the values more comparable
How:
Transformation using function
e.g. log(x)
Standardisation/normalisation
e.g. division-by-range

27. Data Pre-processing Handling missing values
What: to treat attributes with null values
The purposes:
Improve data quality
Better mining results
How:
Elimination (may not always be possible)
Using sensible default, e.g. Spending Amount is set to 0
By data imputation
Average, median, or mode of the whole data population
Average, median or mode of the nearest neighbours
Postponing the handling and making the mining methods adaptive to missing values

28. Data Exploration Exploring data before mining
Knowing data is essential for successful data mining
Purposes:
Better understanding of the characteristics of data
Better decision over data pre-processing tasks
Even being able to discover some hidden patterns
Categories of data exploration techniques
Summary statistics: using a small set of descriptors to describe the characteristics of a large data set
Data visualisation: using graphical or tabular forms to reveal hidden data patterns
Online Analytic Processing (OLAP)
Data exploration and exploratory data analysis (EDA)

29. Data Exploration Summary statistics
Frequency and mode for categorical attributes:
Frequency of value
Mode: the most frequently occurred value
Percentiles for ordinal or continuous attributes:
Given an attribute x and an integer p (0p100), the percentile xp is a value of x such that p% observed values of x are less than xp.
Mean and median for continuous attributes:
Mean and median
Median is a better indication of “average” when data distribution is skewed or outliers are present
Trimmed mean and median (after trimming top and bottom p%)

30. Data Exploration Summary statistics (cont’d) Measures of spread:
Range
Variance (2)
Standard Deviation ()
Absolute average deviation (AAD)
Multivariate summary statistics
Mean vector
Matrix of covariance
Correlation

31. Data Exploration Data visualisation
Rationale: human eyes are good at spotting patterns, particularly visual patterns.
Major ways of visualising data
Tabular form
Graphical form
Points and links
Visual representation must be related to the data types of the attributes
Visualising data as well as all its implicit relationships
The visualisation must be comprehensible
The visualisation of data must tell the truth

32. Data Exploration Data visualisation techniques Pie Chart
Parallel Dimension Chart
Stem & Leaf Plot
Bar Chart
Scatter Plot
Star Dimension Chart

33. Data Exploration Online analytic processing (OLAP)
Interactive reporting tool
Treating a data set as a multidimensional hypercube
Fast operation and fast result delivery
A typical OLAP query:
“For each product, find its market share in its category today minus its market share in its category in 1994”
Result of the OLAP query:

34. Data Exploration OLAP: Multidimensional hypercube Jan Feb Dec March
Buckingham
Milton Keynes
Northampton
1998
2000
1999
Total Customer = 5
Customer Names
March
Milton Keynes
1999

35. Data Exploration OLAP: Hierarchies winter spring summer Buckingham
Milton Keynes
Northampton
1998
2000
1999
autumn
January
February
March
winter
April
May
June
spring
July
August
September
summer
October
November
December
autumn
Jan
Feb
Dec
March
Buckingham
Milton Keynes
Northampton
1998
2000
1999

36. Data Exploration OLAP: Operations Pivoting Slicing and dicing
Selecting attributes to define the cube
Visually rotating the cube to show a face
Slicing and dicing
Selecting a part of a cube
Visually slicing a segment of a cube along a dimension
Rolling-up
Moving up along a hierarchy
Drilling-down
Moving down along a hierarchy
Performing aggregate functions while rolling-up or drilling-down

37. Data Exploration in Weka Explorer
ARFF file format
Data set name
Numeric attribute names and types
Schema section
Categorical attribute name and values
Data section
One data record per line;
Values separated by “,”;
“?” represents unknown.

38. Data Exploration in Weka Explorer
Glance of an opened data set
Summary
statistics
Visualisation of value distribution

39. Data Exploration in Weka Explorer
Visualisation in Weka (limited)

40. Data Exploration in Weka Explorer
Filters for pre-processing
Many filters
Supervised/unsupervised
Attribute/instance
Choose followed by parameter setting in command line

41. Chapter Summary
The domain types determine the validity of operations applied.
Transformation from one domain to another must preserve the domain characteristics.
Data sets can be of various forms and from different sources.
Data warehouse serves as a data source for data mining.
Data quality is relevant to the intended application purpose.
Data pre-processing operations are essential for good mining.
Knowing the data is important for good data mining.
Understanding of data is achieved via exploring, summarising and visualising data.
OLAP serves as a data exploration and summarisation tool.

42. References Read Chapter 3 of Data Mining Techniques and Application
Useful further references
Tan, P-N., Steinbach, M. and Kumar, V. (2006), Introduction to Data Mining, Addison-Wesley, Chapters 2 and 3