Presentation on theme: "An Introduction to Big Data Harry E. Pence 2013 Harry E. Pence TLTC Faculty Fellow for Emerging Technologies."— Presentation transcript:
An Introduction to Big Data Harry E. Pence 2013 Harry E. Pence TLTC Faculty Fellow for Emerging Technologies
Why Big Data Now? Big Data is produced by the collision of cheaper, faster computing with very large social networks. In 1980 a terabyte of disk storage cost $14 million; now it costs about $30. Amazon or Google will “rent” a cloud-based supercomputer cluster for only a few hundred dollars an hour. Sebastien Pierre’s Facebook Map
The annual growth rate of big data is 60%. Twitter generates more than 7 Terabytes (TB) a day; Facebook more than 10 TBs, and some enterprises already store data in the petabyte range.
1 gigabytes = 1E9 Bytes 1 terabytes = 1E12 Bytes 1 Petabyte = 1E15 Bytes or 250,000 DVDs Source Source http://tinyurl.com /a8zwman 1 Megabyte = 1E6 Bytes. Facebook currently stores more than 100 petabytes of data.
The amount of data available is growing much faster than the ability of most companies to process and understand it. Moore’s Law 2: Data expands to fill available storage. According to researchers at the UC-San Diego, Americans consumed about 3.6 zettabytes of information in 2008. David Weinberger (p.7) says digital War and Peace is about 2 megabytes, so one zettabyte equals 5E14 copies of War and Peace. It would take light 2.9 days to go from the top to the bottom of this stack. Harry E. Pence 2013 Or Garbage?
One definition is, “Big Data is when the size of the data itself becomes a problem.” But Big Data is important not just because of size but also because of how it connects data, people, and information structures. (It enables us to see patterns that weren’t visible before. But Big Data is important not just because of size but also because of how it connects data, people, and information structures. (http://tinyurl.com/ato2hbu ) It enables us to see patterns that weren’t visible before.http://tinyurl.com/ato2hbu Researchers at Harvard School of Public Health combined location data from over 14 million cell phone subscribers in Kenya with a malaria prevalence map to determine that a main source of malaria in Kenya. Anyone can use powerful, open, online databases, like http://data.worldbank.org/ and https://explore.data.gov/. (https://developers.google.com/bigquery/) Google’s Big Query allows anyone to query all of Wikipedia, Shakespeare, and weather stations from within your browser for less than $0.035 per GB. http://data.worldbank.org/https://explore.data.gov/.https://developers.google.com/bigquery/ http://data.worldbank.org/https://explore.data.gov/.https://developers.google.com/bigquery/ Harry E. Pence 2013
Gerd Leonhard John Wanamaker once said, “I know that half of my advertising doesn’t work. The problem is I don’t know which half.”
Did Big Data help to determine the 2012 election? NY times, 2/14/13 http://tinyurl.com/acj8hk5 Romney raised slightly more money from his online ads than he spent on them, Obama’s team more than doubled the return on its online-ad investment. Romney’s get-out-the-vote digital tool, Orca, crashed on Election Day; Obama’s Narwhal, gave every member of the campaign instant access to continuously updated voter information. Obama was the very first candidate to appear on Reddit, and the photo of the Obamas became the most popular image ever seen on Twitter or Facebook. Romney’s senior strategist, Stuart Stevens, may well be the last guy to run a presidential campaign who never tweeted. Harry E. Pence 2013
Microtrends Microtrends are developments among smaller groups that are counterintuitive to the mainstream, such as "soccer moms." Studying these subgroups can identify a tipping point in an election or a new business. Microtrends are developments among smaller groups that are counterintuitive to the mainstream, such as "soccer moms." Studying these subgroups can identify a tipping point in an election or a new business. http://tinyurl.com/cmjas4 http://tinyurl.com/cmjas4 A pharmaceutical group discovered a new use for a heart rate drug based on social media. People commented that when people took the drug they grew more hairy. They made more money from the new use of the drug. A multinational hotel group predict the occupancy rates of hotels in major cities three months in advance by monitoring comments about the city on social media. A multinational hotel group predict the occupancy rates of hotels in major cities three months in advance by monitoring comments about the city on social media. http://tinyurl.com/adrouf4 Harry E. Pence 2013
Commercial users seem more concerned with correlation, not causality By comparing the 50 million most common search terms with CDC data on the flu, Google's found a combination of search terms that strongly correlated with the spread and intensity of 2009 flu season. Both Amazon and Netflix use correlation- based recommendations to boost sales. Target assumes that if a 20-something female shopper purchases a unscented lotion, an supplements such as zinc and calcium, and a large purse, she is pregnant.
Cheap sensors constantly monitor personal data to allow for personalized support. Inexpensive ($100-200) devices, like Fitbit already will track your daily physical activity to a web page. General Motors (OnStar) and several insurance companies offer driving monitors. Harry E. Pence 2013
Inexpensive sensors are rapidly moving us towards an Internet of Things. http://tinyurl.com/aoev3x9 We are here! Physical World Web
Companies ranging from the NY Times to the UK Ordnance Survey are creating linked data to make the web more interconnected. Each entity is defined by a Uniform Resource Identifier (URI) which is machine readable. The hope is to attach metadata to each entity to show how they relate to each other, employees to companies, actors to motion pictures, etc. Google’s Information Graph is a step towards the Semantic Web. Harry E. Pence 2013
Some predict that the Internet of Things will soon produce a massive volume and variety of data at unprecedented velocity. http://tinyurl.com/ahytzdf Welcome to the new information age http://tinyurl.com/ahytzdf
Big Data and Big Science “The Fourth Paradigm” Massive Sensor Arrays are important in many areas of science, like meteorology, environmental monitoring, astronomy, and climate measurements. The Square Kilometre Array (SKA) under development in Australia and South Africa will collect one exabyte of data per day from 36 small antennas spread over more than 3000 km to simulate a single giant radio telescope. Bradley Voytek ( Big Data location 1186 ) argues that Big Data analysis allows researchers to identify patterns that were previously invisible; it is possible to automate critical aspects of the scientific method itself. Harry E. Pence 2013
The large Hadron Collider at CERN produces so much data that scientists must discard most of it, hoping they haven’t thrown away anything useful. Weather prediction combines data from multiple earth satellites with massive computing power. Most of the satellites belong to the U.S., but the Europeans have a more powerful computer. Our weather satellites are old. http://tinyurl.com/cvpz5qe Harry E. Pence 2013 17 miles
Big Data and the Future of Health Care A patient can use a smartphone to do many tests now done in the laboratory, such as an EKG or glucose test. The Quest for the $1000 Genome Analysis. In May 2007, the first personal genome cost $1 million. Recently, several companies are close to $1000 and some are predicting a $100 dollar cost soon. Knowing an individual’s genome should allow treatment to be customized to the individual. As the cost plummets, the bottleneck will increasingly be the cost of interpreting the genomic data.
IBM defines big data as the combination of velocity, variety, and volume
The Three V’s The sheer volume of stored data is exploding. IBM predicts that there will be 35 zettabytes stored by 2020. This data comes in a bewildering variety of structured and unstructured formats. The velocity of data depends on not just the speed at which the data is flowing but also the pace at which it must be collected, analyzed, and retrieved. IBM claims that its software will adjust to changes in any of these three factors. Harry E. Pence 2013
Search engines, like Yahoo! and Google, were the first companies to work with datasets that were too large for conventional methods. (According to Big Data 2 location 177 Google has over a million servers.) In order to power its searches, Google developed a strategy called MapReduce. You map a task onto a multitude of processors then retrieve the results. Traditional data warehouses use a relational database (think rows and columns); Search engines need to handle non-relational databases, sometimes called NoSQL. Harry E. Pence 2013
The most popular software to search No-SQL databases is called Hadoop and several different versions are freeware. Hadoop is designed to collect data, even if it doesn’t fit nicely into tables, distribute a query across a large number of separate processors, and then combine the results into a single answer set in order to deliver results in almost real time. This is often paired with machine learning apps, like a recommendation engine, classification, error detection, or facial recognition. Personal Aside: I suggest that Google Analytics might be the best way to introduce students to crafting a query for a Big Data exercise. Harry E. Pence 2013 Named after his son’s pet elephant.
We are in the Golden Age of Data Visualization A streamgraph of the conversation around a brand. http://tinyurl.com/beqxuyl
Classify each of these as pro or con fracking: Don’t appoint a #fracking proponent to lead the Dept. of Energy. (often RT) Broome County Executive took $82,428 in pro- #fracking campaign contributions. The only politician I know that has backed up his promise to address #fracking is Gov. Cuomo. Natural gas is neither perfect nor perfectly evil. Businesses surprised to see their names on #fracking petition. Mmmm @fracking fluid bit.ly/ZmQVQj Harry E. Pence 2013
Amazon’s Mechanical Turk is one way to manually create a standard data template. https://www.mturk.com/mturk/welcome
Moving from Big Data to Smart Data is a multistep process. http://tinyurl.com/atcanjw “Smart Data” “The issue is not about the volume of data but the ability to analyze and act on data in real time.” http://tinyurl.com/atcanjw
Scraping, processing, and buying Big Data One can only copyright a specific arrangement of the data, but the metadata is often extremely important and may not follow the scrape. Companies, like InfoChimp, are scraping and cleaning selected data from Twitter and then selling access to these datasets. Pete Warden reports that it only cost him $120 to gather, analyze, and visualize 220 million public Facebook profiles (and 8olegs allowed him to download a million web pages for about $2.20. Pete Warden reports that it only cost him $120 to gather, analyze, and visualize 220 million public Facebook profiles (http://tinyurl.com/yb2q3dv) and 8olegs allowed him to download a million web pages for about $2.20.http://tinyurl.com/yb2q3dv Harry E. Pence 2013
Problems with Big Data – Selection bias ( Problems with Big Data – Selection bias (http://tinyurl.com/bdb7wgy) Selection bias occurs when the individuals or groups to take part in a study don’t represent the general population. According to a recent Pew survey, Twitter users are younger than the general public and more likely to lean toward the Democratic Party. Twitter reactions are often at odds (six out of eight times) with overall public opinion. http://tinyurl.com/cvqq5hz http://tinyurl.com/cvqq5hz A recent article in EJP Data Science says Twitter is actually comprised of modern-day tribes, groups of people who use a discrete language and are connected to a. A recent article in EJP Data Science says Twitter is actually comprised of modern-day tribes, groups of people who use a discrete language and are connected to a character, occupation or interest. http://preview.tinyurl.com/bc8gecu Harry E. Pence 2013
The partition of English speaking Twitter users into communities, annotated with words typical of those often used by members of each community The partition of English speaking Twitter users into communities, annotated with words typical of those often used by members of each community. http://tinyurl.com/ bmb9r9e
Problems with Big Data – Misclassification bias ( Problems with Big Data – Misclassification bias (http://tinyurl.com/bdb7wgy) Misclassification occurs when either the cause or the effect is not accurately recognized. What if a response is not correctly identified as intended by the participant. This is especially true when subjective interpretation is required to classify an answer. What if a response is not correctly identified as intended by the participant. This is especially true when subjective interpretation is required to classify an answer. http://tinyurl.com/cvqq5hz Remember: Correlation does not imply causation, But some projects, like targeting building inspections in NY City (, seem to work well on the basis of correlation only. But some projects, like targeting building inspections in NY City (http://tinyurl.com/clhmv3t), seem to work well on the basis of correlation only. Harry E. Pence 2013
Comments on Big Data for education The information in LMSs (i.e. time on task, number of log- ins, number of list posts, etc.), like Moodle or Angel, is well structured and is already being analyzed. In MOOCs, students interact entirely online, leaving behind a record of every page they visited. The Gates Foundation recently gave $100 million to InBloom to improve ways to transfer information among the many technology information silos where students records are currently stored. Nine states (including New York) are participating in this pilot project and plan to offer third-party vendors access to student data (without student or parental consent). The Gates Foundation recently gave $100 million to InBloom to improve ways to transfer information among the many technology information silos where students records are currently stored. Nine states (including New York) are participating in this pilot project and plan to offer third-party vendors access to student data (without student or parental consent). http://tinyurl.com/b8e4whh Unanswered questions include security, who owns learner produced data (TurnItIn?), who owns the data analysis, and what will be shared with the students? Harry E. Pence 2013
Chico State University (CA) has been using learning analytics to study how student achievement is related to (LMS) use and student characteristics. Chico State University (CA) has been using learning analytics to study how student achievement is related to (LMS) use and student characteristics. http://tinyurl.com/acz5f7o The project merges LMS data with student characteristics and course performance from the campus database. The study reports a direct positive relationship between LMS usage and the student ‘s final grade. Voytek’s Third Law: Any sufficiently advanced statistics can trick people into believing the results reflect truth. Harry E. Pence 2013 Dwell time is how long a student spent on a given activity.
Problems with Big Data – Confounding bias ( Problems with Big Data – Confounding bias (http://tinyurl.com/bdb7wgy) Confounding occurs when there is a failure to control for some other factor that is affecting the outcome. How do other factors, like attendance, reading the textbook, attending extra help sessions, etc. relate to the time spent on the LMS (and so the course grade)? “Low-income students spend more time on the LMS.” The article notes that, “No individually identifiable information is included in the data files.” Really??? Harry E. Pence 2013
Problems with Big Data – Privacy A sensible back-up strategy may create more than 100 copies. How can organizations protect the privacy of all this data from hackers? This can make it hard to protect individual privacy, and recent experiences suggest that there are now so many public datasets available for cross-referencing that it is difficult to assure that any Big Data records can be kept private. In a number of cases, information from “anonymous studies” has been tracked back to identify individuals and even their families. Harry E. Pence 2013
Privacy (cont.) In 2006, a study examined the Facebook profiles of 1,700 students from an unnamed northeastern college to measure how their interests changed with time. (Lewis et al, 2008, Social Networks, 30, 330-342). Zimmer soon demonstrated that he could identify the college in the dataset, compromising the student identities. (Zimmer, 2008, ). Zimmer soon demonstrated that he could identify the college in the dataset, compromising the student identities. (Zimmer, 2008, http://tinyurl.com/b87x7wq).http://tinyurl.com/b87x7wq Science, In January, a researcher had not only identified five people he randomly selected from a 1,000 person genetics study but also found their entire families, even though the relatives had no part in the study — identifying nearly 50 people. (Science, (2013) 339, 321-324, http://tinyurl.com/a9jw283) Harry E. Pence 2013
“Just as Ford changed the way we make cars – and then transformed work itself- Big Data has emerged as a system of knowledge that is already changing the objects of knowledge, while also having the power to inform how we understand human networks and community.” danah boyd ( “Just as Ford changed the way we make cars – and then transformed work itself- Big Data has emerged as a system of knowledge that is already changing the objects of knowledge, while also having the power to inform how we understand human networks and community.” danah boyd (http://tinyurl.com/bdb7wgy) “And finally, how will the harvesting of Big Data change the meaning of learning and what new possibilities and limitations may come from these systems of knowing?” Harry E. Pence 2013
http://soundcloud.com/tracks/search?q=acxiom That was That was Vienna Teng singing the song acxiom, sacred music for the age of big data.Vienna Teng Any questions? Thank you for listening. Thank you for listening.