1. Practical Big Data 2016 Use cases, trends and patterns
Hari Gottipati
Phoenix Data Conference 2016
October 29th, 2016

2. Hari Gottipati Who Am I? Architect @ one of the leading
financial company
Hari
Gottipati
Profession
Passion
Programming
Nature
Interest
Open Source Advocate
Focus
Speaker, Freelance writer, blogger
Web Scaling, Big Data, AI,
Mobile ( IoT, Augmented Reality)

3. Disclaimer
I am not representing any company and opinions expressed here are solely mine.

4. Agenda Real time Use cases AI Algorithms/ML
Bots (Chat bots, NLP, Conversational Commerce)
Mass customization/hyper personalization
Sentiment analysis
Business demands
Technology trends

5. Real-time

6. We often use Real-time to describe two different things - Data freshness and Faster response time

7. Process entire data and give answer for X in the end
Batch
Data at rest
After the event occurs
Process entire data and give answer for X in the end
Realtime processing/Streaming/Fast Data
Data in motion
As the event occurs
Process incoming stream of data and give answer for X at this moment
Data freshness
Push action
Realtime
Data at rest
Low latency access
Pull
Batch out
Faster response time

8. Use Cases

9. AI

12. Co-founder & CEO of Luka
Eugenia Kuyda
Co-founder & CEO of Luka
Image Source : Google Images

13. Friend of Eugenia Kuyda Co-founder & CEO of Stampsy
Proposed a new kind of cemetery “Taiga” - dead would be buried in biodegradable capsules, and their decomposing bodies would fertilize trees that were planted on top of them, creating what he called “memorial forests.” A digital display at the bottom of the tree would offer biographical information about the deceased
Roman Mazurenko
Friend of Eugenia Kuyda
Co-founder & CEO of Stampsy
Got killed in a road accident
Image Source : Google Images

14. Image Source : Google Images

15. Image Source : Google Images

16. Image Source : Google Images

17. Image Source : Google Images

18. Image Source : Google Images

19. Image Source : Google Images

20. @Roman on Luka chat platform Replika - mimics user personality
Image Source : Google Images

21. ML

22. Sunspring is a 2016 experimental science fiction short film entirely written by an AI

26. Algorithms

27. Image Source : Google Images
Person of Interest centers on John Reese (Jim Caviezel), a presumed-dead former CIA agent who is hired by a mysterious billionaire named Harold Finch (Michael Emerson), to prevent impending violent crimes predicted by the Machine, a mass-surveillance computer system that relays the identity of a single person predicted to be the crime victim or perpetrator.
Image Source : Google Images

29. Dozens of inputs, including whether workers skip compliance classes, violate personal trading rules or breach market-risk limits, will be fed into the algorithm.
s, chats and telephone transcripts can be analyzed electronically to determine if employees are trying to collude or conceal intentions.
Using the technology that was built for counter-terrorism and using it against human language.
If you want to be proactive, you have to get people before they act.

30. Bots

32. Georgia Tech, College of Computing
Spring 2016
Georgia Tech, College of Computing
Knowledge Based Artificial Intelligence (KBAI) class
300 students post roughly 10,000 messages in the online forums
Professor Ashok Goel hired Jill Watson as teaching assistant
Ashok Goel
Jill Watson

33. Image Source : Google Images
Their identity
Ashok Goel
Jill Watson
Image Source : Google Images

34. Chat bots

36. Image Source : Google Images

37. Image Source : Google Images

38. Image Source : Google Images

39. NLP

41. Conversational Commerce

42. Image Source : Google Images

43. Image Source : Google Images

44. Mass customization/hyper personalization

45. Image Source : Google Images

46. Image Source : Google Images
Walmart
Real-time semantic analysis of the different social media streams.
Uses spikes in social network chatter to predict demand for out-of-the-ordinary products.
Image Source : Google Images

47. Walmart changes its prices roughly 50,000 times a month.
Tapping into social media, it captures metadata containing customers, topics, products, locations and events. Combine these data streams with customer data, sales data, financial data and mobile data to obtain a complete overview of their customers.
As a result, the Social Genome project is a massive, constantly-changing, knowledge base with hundreds of millions of entities and relationships. Walmart is using it to become a me-tailer instead of a retailer. Focusing on the individual shopper, providing a personalized experience.
Having the right products in the right place at the right time, so the right people can buy them
Walmart changes its prices roughly 50,000 times a month.
Grocery team struggling to understand why sales of a particular produce were unexpectedly declining. Once their data was in the hands of the Café analysts, it was established very quickly that the decline was directly attributable to a pricing error.
Sales across different stores in different geographical areas can also be monitored in real-time. One Halloween, Peddamail recalls, sales figures of novelty cookies were being monitored, when analysts saw that there were several locations where they weren’t selling at all. This enabled them to trigger an alert to the merchandizing teams responsible for those stores, who quickly realized that the products hadn’t even been put on the shelves.
Led to a reduction in the time it takes from a problem being spotted in the numbers to a solution being proposed from an average of two to three weeks down to around 20 minutes.
Products have to be efficiently priced to the cent, to stay competitive. And if customers find they can’t get everything they need under one roof, they will look elsewhere for somewhere to shop that is a better fit for their busy schedule.

48. Image Source : Google Images
There is no other car in like the Opel Adam. With more than 61,000 variants for the exterior and nearly 82,000 for the interior there are in total 4 billion combinations possible.
Image Source : Google Images

49. Sentiment analysis

51. Research suggesting a direct link between positive posts and a reduced risk of heart disease.
Future insurance coverage based on “sentiment analysis,” in which Big Data and artificial intelligence make predictive models ever more accurate.
Swiss Re, the world’s second-largest reinsurer, has invested in digi.me, a startup aiming to let consumers store personal data culled across various social media channels and beyond and to exchange their data with businesses for personalized deals.

52. Business Demands

53. DATA VOLUMES ARE GROWING AT UNEXPECTED RATE
By the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet.
By then, data will grow from 4.4 zeta bytes today to around 44 zeta bytes, or 44 trillion gigabytes.
By 2020, there will be over 50 billion smart connected devices in the world, all developed to collect, analyze and share data.
Technologies will keep changing to handle the new load. Did you know Twitter abandoned Storm last year because of scaling issues (billion events per second) – Heron is the answer

54. From 3 V’s To 6 V’s Image Source : IBM Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : IBM

55. From Descriptive to Predictive/Prescriptive Analytics
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : Google Images

56. Image Source : Google Images
Automation demand
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : Google Images

57. Rise of real-time use cases
More and more use-cases demanding faster insight to data
Data in motion is becoming common
Real-time data processing is getting traction
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected

58. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

59. Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

60. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

61. Image Source : Kreditech
New demands - example
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : Kreditech

62. Technology trends

63. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

64. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

65. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

66. Image Source : O’Reilly eBook - The Big Data Market
Plan for Future generation
Our children and grand children are not going to have better life if we don’t save water and energy, if we don’t act now
Only possible with sensors
Only possible by connecting the unconnected
Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

67. Image source: https://databricks

68. Spark is dominating the processing world
Image source:

69. Cloud is emerging as a preferred deployment model
Appliance and cloud are driving next wave of Hadoop adoption to mainstream enterprises lacking the same depth of IT skills as early adopters.
Image source:

70. Cloud is emerging as a preferred deployment model
Image source:

71. Data Scientist is still the demanding job
Statistical analysis and data mining retained the second spot in LinkedIn list of top skills in 2017 (Cloud and distributed computing has remained in the #1 spot for the past two years).
Demand for deep analytical talent in the U.S. projected to be 50-60% greater than supply by 2018, leading to a shortage of 140,000 to 190,000 .
In Q1 2015, the number of job postings for data scientist grew 57% year-over-year while searches for data scientist grew 73.5% for the same period.
Data scientist was ranked the best job in America because of its high earning potential, abundant career opportunities, and number of job openings.
Related search queries have increased six times over the past five years
SQL, R, Python, SPSS, Tableau, and Hadoop are leading tools used by Data Scientists.

72. Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

73. Image Source : O’Reilly eBook - The Big Data Market
A Data-Driven Analysis of Companies Using
Hadoop, Spark, and Data Science
Aman Naimat

74. Paysa - Data scientist salary analysis
The average base salary for Data Scientists is $117K per year, ranging from $89.2K to $242K.
Based on recent job postings for Data Scientists, 65% need to know Python, 53% need to have R programming skills, 50% need to know SQL and have machine learning development skills.
Lyft and WalMart are actively looking for Data Scientists today, offering over base salaries starting at $117K.
Lyft is offering the highest base salary of $167K followed by Facebook at $159K.

75. Big Data has many branches and growing…
Image Source :

76. Image Source : Google Images
Challenge
Technology is getting outdated in 2 years unlike over a span of 10 years about a decade back.
Image Source : Google Images

77. Summary AI is the future AI first strategy is essential
Business demands are real-time
One tool/technology is not going to be sufficient to meet business demands or get a job
All the use cases discussed are not the future. If you think those are future…

78. #TheFutureIsHere Thank you
harigottipati}
@twitter @linkedin @facebook

79. We got a song too...