We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byBryan Barker
Modified over 2 years ago
© 2006 Intelliware Development Inc. An Introduction to Data Mining Concepts Tim Eapen and B.C. Holmes Intelliware Development
© 2006 Intelliware Development Inc. Agenda Introduction to data mining The typical steps What were we trying to accomplish Bayesian Categorization An example Data Clustering k-means clustering Interesting conclusions Other Stuff Java and Data Mining
© 2006 Intelliware Development Inc. What is Data Mining? Data mining is the discovery of useful information from data Data mining touches on many of the same problems as machine learning and artificial intelligence This is a huge topic, and we cant hope to do more than just touch on it, today
© 2006 Intelliware Development Inc. Some Crazy Examples Here are some interesting examples of useful information gleaned from data: Diapers and beer People who buy diapers are also likely to buy beer. Put potato chips in between them and the sales of all three items go up Google ad-words: digital cameras is worth more than digital camera Airline traveler behaviours Amazon.ca other people who bought this DVD liked such-and-such
© 2006 Intelliware Development Inc. The Data Mining Process Cle ans e Ext ract the Go od Stu ff Ide ntif y Pat ter ns Gather the Data Vet the res ults
© 2006 Intelliware Development Inc. What We Were Trying to Accomplish Tim, Tom and I were working on the WhatAmITaking.com project WhatAmITaking.com is a wiki / repository that collects information about medications Data is all available from public sources, including: Government drug reference database Wikipedia Open License publications available through the (U.S.) National Institute for Health News articles Concept: want to using data mining techniques on publications and news First steps: we wanted to try to emulate the Google news-style categorization and topic correlation
© 2006 Intelliware Development Inc. But Along the Way… We learned some interesting things about the field of Data Mining
© 2006 Intelliware Development Inc. News: Obtaining News How do we get news? Need to build a bot or a web crawler that goes out to a large number of web sites and GETs the interesting content. Nice additions: look for links to other pieces of news Some complications: Theres a Good Internet Citizen standard (the robots.txt file standard) that should be respected If the site has a robots.txt file that says bots keep out, you shouldnt crawl their site. How do you determine whats a story and whats not? Thats a hard problem: too big a topic for this presentation
© 2006 Intelliware Development Inc. Data Cleansing You would not believe how bad some news sites are with respect to their content. Poor formatting bad encoding problems Clear problems related to converting the content from another format (e.g. Word) Two interesting word-related cleansing problems The US spelling versus British spelling problem Root words Some of it looks deliberately obfuscated
© 2006 Intelliware Development Inc. Extracting Interesting Stuff Your typical web page news article has a lot of extra stuff on it: banner ads, menus, links to related stories, navigation widgets, etc. Almost all word manipulation problems talks about stop words: words that are so common they provide no significant meaning in analysis of text: the he she said it etc…
© 2006 Intelliware Development Inc. Two Interesting Topics Categorization I know what the groups are, and I want to assign a group to any particular data point E.g.: News is categorized: Sports, Health, Finance, World News, National, etc. Data Clustering I have a lot of data, and I want to find some mechanism for finding meaningful groups E.g.: News events
© 2006 Intelliware Development Inc. Bayesian Analysis A Delightful Example
© 2006 Intelliware Development Inc. The Problem HEALTH SPORTS TECHNOLOGY BUSINESSNEWS ENTERTAINMENT Given a random news article, how can we determine what category it belongs to?
© 2006 Intelliware Development Inc. In Light of New Evidence… Do some detective work! Start off with a hypothesis Collect evidence The evidence will be either consistent or inconsistent with a given hypothesis As more evidence is accumulated, the degree of belief in the initial hypothesis will change A hypothesis with a very high degree of belief may be accepted as true Likewise, a hypothesis with a very low degree of belief may be considered false How do we measure this degree of belief?
© 2006 Intelliware Development Inc. Bayes Theorem
© 2006 Intelliware Development Inc. Bayes Theorem
© 2006 Intelliware Development Inc. Bayes Theorem
© 2006 Intelliware Development Inc. An Edible Example 10 Chocolate Chip Cookies 30 Oatmeal Cookies 20 Chocolate Chip Cookies 20 Oatmeal Cookies
© 2006 Intelliware Development Inc. State a Hypothesis Little Johnny picks a bowl at random Little Johnny picks a cookie at random The cookie turns out to be an oatmeal cookie How probable is it that Johnny picked the cookie out of bowl #1?
© 2006 Intelliware Development Inc. Consider the Evidence Probability of selecting an Oatmeal cookie given Johnny chooses bowl #1 Probability of selecting an Oatmeal cookie given Johnny chooses bowl #2
© 2006 Intelliware Development Inc. An Edible Example Bayes Theorem gives the following result Notice that initially the prior probability that the cookie came from bowl #1 was P(H 1 ) = 0.5 In light of evidence E, the probability that the cookie came from bowl #1 increased to P(H 1 |E) = 0.6
© 2006 Intelliware Development Inc. Back to our problem… Given a random news article, how can we determine what category it belongs to? OF COURSE WE CAN!!! USE BAYESIAN ANALYSIS
© 2006 Intelliware Development Inc. Naïve Bayes Classifier To categorize a news article use a Naïve Bayes Classifier A simple probabilistic classifier based on some naïve independence assumptions Can be trained Naïve Probabilistic Model The probability model for a classifier is conditional: Given an news article with n words … Let C represent a category of news (i.e. Health) Let F n represent the frequency with which that n th word appears in articles from category C
© 2006 Intelliware Development Inc. Naïve Probabilistic Model We can express our probability model using Bayes Theorem Solving this is difficult so we make some simplifying assumptions: Denominator is constant Naively assume that each feature (word frequency) F i is conditionally independent of every other feature F j (i j)
© 2006 Intelliware Development Inc. Naïve Probabilistic Model Problems with our assumptions Words have context Assuming that the frequency (F i ) of word i is independent of the frequency (F j ) of word j is untrue For example the words War and Afghanistan are more likely to appear in the same article than the words War and Tuna Benefits of our assumptions It simplifies our math algorithm
© 2006 Intelliware Development Inc. Naïve Probabilistic Model We can approximate that the probability that an article belongs to category C as the product of a prior probability that the article belongs to that category multiplied by the product of individual word frequencies for that category
© 2006 Intelliware Development Inc. A Simple Algorithm for Classifying An Article Given a random article with n words to classify the article in one of several possible categories do the following: For each possible category Calculate the probability that article X belongs to that category by considering the prior probability and word frequencies Classify the article as belonging to the category with the highest probability
© 2006 Intelliware Development Inc. A Simple Example Consider this very simple article … hockey puck For simplicity consider that there are only two possible categories: Sports News
© 2006 Intelliware Development Inc. A Simple Example … Consider the following word frequencies: WordCategoryFrequency hockeySports98% puckSports96% hockeyNews2% puckNews4% 1.Let C = Sports: p(C)=0.5, p(F 1 |C)=0.98 and p(F 2 |C)=0.96 p(C|F 1,F 2 ) = 0.5x0.98x0.96= Let C = News: p(C)=0.5, p(F 1 |C)=0.02 and p(F 2 |C)=0.04 p(C|F 1,F 2 ) = 0.5x0.02x0.04=0.0004
© 2006 Intelliware Development Inc. Gathering the Evidence So where do the frequencies we use come from? To perform Bayesian analysis, it is important to have a large corpus of articles This corpus is what we use to determine the word frequencies used in categorizing a given article This corpus would grow over time This corpus is what we use to train our Bayesian classifier
© 2006 Intelliware Development Inc. What We Actually Did First step was to gather a corpus of articles This corpus would be used to train our Bayesian classifier Initially started by gathering 5000 articles Number of articles in the corpus would grow over time Built a simple, little NewsFinder utility that would regularly go to and gather articles Google has seven categories of news News Finder worldCanadaHealthbusinesssciencesports entertainment
© 2006 Intelliware Development Inc. Bayesian Classifier Started with an open-source package from sourceforge called classifier4j: available at Created a SimpleClassifier This classifier has an instance of our Bayesian classifier which does all the Bayesian analysis for us The classifier also has a WordDataSource: a simple map that correlates a frequency with a given word in a given category Used our corpus of articles to train the our classifier (fill up our word data source)
© 2006 Intelliware Development Inc. Issues To Consider Making sure that the corpus was clean This was part of cleansing the data as we gather it Had to actually tweak Classifier4j because the algorithm wasnt correct
© 2006 Intelliware Development Inc. Clustering What is a Cluster, anyway?
© 2006 Intelliware Development Inc. Data Clustering Data clustering is the process of taking points in some n- dimensional space, and grouping them into some understandable group. Thats kind of math-y sounding. How does that relate to news? This is the fundamental question: trying to decide good measures is the key success criteria I want to defer the answer for now There are two fundamental approaches: Centroid Guess certain centres of clusters, and iteratively refine them Hierarchical Assume that each point is a cluster, and iteratively merge them until good clusters emerge
© 2006 Intelliware Development Inc. Another Key Consideration The field of Data Mining spends a lot of time thinking about one special problem: Often, theres too much data to fit into memory; any algorithms that try to cluster information must think about the special problem of data not fitting into memory Im not going to say too much about this problem
© 2006 Intelliware Development Inc. k-Means Algorithm One of the fundamental centroid-based algorithms is called the k- means algorithm Assume you have a number of points of data and you want to cluster these points into some number of clusters (k) You dont really need to know what the clusters represent, just some arbitrary number of clusters
© 2006 Intelliware Development Inc. Step One: Pick k=3 objects
© 2006 Intelliware Development Inc. Step Two: Create initial Groupings Groups are based on distance from initial points
© 2006 Intelliware Development Inc. Step Three: Find the centres/means
© 2006 Intelliware Development Inc. Step Four: Re-jig the clusters
© 2006 Intelliware Development Inc. Repeat until the Clusters dont change
© 2006 Intelliware Development Inc. But How Do You Decide on k? A key question to ask is how many clusters is the right number? Try a bunch of different values, and map distance 12345
© 2006 Intelliware Development Inc. Converting from Words to Points One idea: There are about 100,000,000 English words. Consider an n-Dimensional space, where n = 100,000,000 Frequency of a particular word in an article can be considered a distance in one dimension of the n-Dimensional space.
© 2006 Intelliware Development Inc. Unintuitive Conclusions When dealing with points in n-Dimensional space, where n is very large (say > 100), most points are about as far away as average.
© 2006 Intelliware Development Inc. Determining a Good Measuring Stick So how do you deal with the problem of large dimensional spaces? Try to determine a smaller set of interesting dimensions. Try this: Pick an article In that article try to find 25 interesting words Whats interesting? Try 10 of the most common words in the article (excluding stop words) Pick 10 of the most significant classification words (e.g. certain words are strongly correlated with health articles. Find the 10 most strongly correlated, that also have high frequency of occurrence in the article) Pick 5 unusual words Now youve got some measuring stick. Now measure other articles according to this measuring stick, and figure out distance
© 2006 Intelliware Development Inc. Java and Data Mining There a few (but not many) Java initiatives relating to Data Mining Bayesian Classifier: - Classifier4J Used this initially, and discovered that the algorithm wasnt correctly implemented Weka Created by a number of Data Mining professors The same group has published a Data Mining book with some references to Weka (but its a heavy math book) YALE (Yet Another Learning Environment) Theres a Java Community Process around coming up with a consistent Java API for data mining JSR 73 and JSR 247 javax.datamining
© 2006 Intelliware Development Inc. Other Topics (Use Wikipedia) w-shingling Concept Mining
© 2006 Intelliware Development Inc. Crazy Ideas that Might Make Interesting Experiments Could you perform data mining on code? What if you parsed Camel Case variable and class names and performed text clustering on classes. Could you find interesting relationships between classes? In different projects? What could you learn if you tried to perform clustering on a bunch of open source web frameworks? How must similarity and/or difference do they have?
The. of and a to in is you that it he for.
A. as is a couldn’t does could has wouldn’t.
The people Look for some people. Write it down. By the water So there you are. Who will make it? You and I A long time What will they do?
Chapter 6 Three Simple Classification Methods The Naïve Rule Naïve Bayes k-Nearest Neighbor 1.
The Nature of Science Design of Controlled Experiments.
1 Ontologizing the Ontolog Content Protégé Workshop Denise A. D. Bedford, Ph.d. July 23, 2006.
Chapter 4 Slide 1 Original 33 slides by Prof. Anita Beecroft, Kwantlen Polytechnic University 16 slides added Feb 2011 by Prof. Tim Richardson, University.
Dolch Words the of and to a in that is was.
Copyright © 2010 Pearson Education, Inc. Slide
Introduction to Information Retrieval Introduction to Information Retrieval Lecture 10: Text Classification; The Naive Bayes algorithm.
A Publication of Bridgemark Solutions Six Keys to Generating More Sales Leads AND WINNING MORE MARKET RESEARCH PROJECTS.
Probability and Statistics Representation of Data Measures of Center for Data Simple Analysis of Data.
LOGIC AND CRITICAL THINKING Lecture 11 – Scientific Reasoning I.
Quality Tools and Techniques in the School and Classroom.
UNIT V: LEARNING. LEARNING Learning from Observation Inductive Learning Decision Trees Explanation based Learning Statistical Learning methods Reinforcement.
Teachers as researchers and the development of teacher professionalism Dr Ken Chow.
Chapter 3: Supervised Learning. CS583, Bing Liu, UIC 2 Road Map Basic concepts Decision tree induction Evaluation of classifiers Rule induction Classification.
Donna J. Scott Executive Vice President of Sales and Marketing.
Chapter 12 Technology. INTRODUCTION This chapter considers technology in general, with some limited emphasis on software. The life cycle and software.
Englewood Public Schools Englewood, Colorado 2009 A Compilation of E.W. Dolch 1936, The Reading Teachers Book of Lists, Fourth Edition,© 2000 by Prentice.
CS276B Text Retrieval and Mining Winter 2005 Lecture 2.
RESEARCH METHODOLOGY. Meaning Research is an endeavor to discover answers to intellectual and practical problems through the application of scientific.
The Joint Venture Partner Discovering the Win-Win Situation and How to Move Your Name From the Shadows to the Spotlight.
Measurement and Scaling. Measurement Measurement is the process observing and recording the observations that are collected as part of a research effort.
TESOL 2008 Creating and Maintaining A Classroom Community through Writing Online Mary Lee Wholey Miriam Horne Concordia University Continuing Education.
PHILOSOPHY OF SCIENCE: Bayesian inference Zoltán Dienes Thomas Bayes
Assessment for Learning Promoting Formative Strategies A quick reminder ….What does AfL involve in the classroom? Teachers having an understanding of the.
1 APO KM Tools and Techniques. 2 Objectives To present and discuss some of the key KM methods, tools, technologies and techniques to be considered for.
Experimental concepts. TheoryTheory. A theory can be defined as a "general principle proposed to explain how a number of separate facts are related." In.
Microsoft ® Office Outlook ® 2007 Training Get out of your Inbox Sweetwater ISD presents:
© 2016 SlidePlayer.com Inc. All rights reserved.