Girish Nathan Misha Bilenko Microsoft Azure Machine Learning How to Work with Large Datasets to Build Predictive Models

Agenda 1. How to Work with Large Datasets Sample Dataset: NYC Taxi HDInsight (Hadoop on Azure) iPython notebook and HDInsight 2. Building Predictive Models Azure ML Studio Learning with Counts 3. Putting it all together: Learning with Counts and HDInsight

Sample Data: NYC Taxi One year log of NYC taxi rides 60GB, publicly available at Trip (driver id, times, locations) and fare (fare, tip, tolls) Rest of tutorial: data wrangling and tip prediction Tools: AzCopy, HDInsight, iPython, Azure ML Studio

100% Apache Hadoop as an Azure service Can deploy on Windows or Linux Provides Map-Reduce capability over big data in Azure blobs Head node: job and cluster monitoring Hive: SQL-like queries as an alternative to writing code SELECT Col1, COUNT(*) AS Count_Col1 FROM Your_Table GROUP BY Col1 ORDER BY Count_Col1 DESC LIMIT 10; HD Insight : Hadoop on Azure

Web-based Python REPL environment Combines authoring, execution, visualization Can author and execute HDInsight Hive queries Sample query (python code snippet) def submit_hive_query(self): response=urllib2.urlopen(self.url, self.hiveParams) data = json.load(response) self.hiveJobID = data[‘id’] def query(self, queryString): self.submit_hive_query() Example query string: SELECT * FROM sample_table LIMIT 10; Ipython Notebook

Fully managed cloud service Browser based authoring of dataflow Best in class machine learning algorithms Support for R/Python/SQL Collaborative data science Quickly deploy models as web services/REST API’s Publish to a gallery for collaboration with community What is Azure ML Studio

( Distributed Robust Algorithm for CoUnt-based LeArning) Misha Bilenko Microsoft Azure Machine Learning Microsoft Research Learning with Counts a.k.a Dracula

adid = adText = K2 ski sale! adURL= Userid = 0xb dd9b IP = Query = powder skis QCategories = {skiing, outdoor gear} 8 Information retrieval Advertising, recommending, search: item, page/query, user Transaction classification Payment fraud: transaction, product, user spam: message, sender, recipient Intrusion detection: session, system, user IoT: device, location Large Scale learning in multi entity domains

adid: adText: Fall ski sale! adURL: userid 0xb dd9b IP query powder skis qCategories {skiing, outdoor gear} 9 Large Scale learning in multi entity domains

IP ……… REST Learning with Counts

IP ……… REST Learning with Counts

IP ……… REST query facebook dozen roses ……… REST Query × AdId facebook, ad facebook, ad dozen roses, ad ……… REST time T now Counting IP[2] *.* *.* *.* ……… 12 Learning with Counts : aggregation

IP ……… REST query facebook dozen roses ……… REST time T now Train predictor …. IsBackoff Aggregated features Original numeric features Counting Train non-linear model on count-based features Counts, transforms, lookup properties Additional features can be injected Query × AdId facebook, ad facebook, ad dozen roses, ad ……… REST Learning with Counts : combiner training

IP ……… REST query facebook dozen roses ……… REST URL × Country url 1, US url 2, CA url 3, FR ……… REST time T now …. IsBackoff Aggregated features Counts are updated continuously Combiner re-training infrequent T train Original numeric features Prediction with counts

State-of-the-art accuracy Good fit for map-reduce Modular (vs. monolithic) Learner can be tuned/monitored/replaced in isolation Monitorable, debuggable (this is HUGE in practice!) Temporal changes easy to monitor Easy emergency recovery (remove bot attacks, etc.) Decomposable predictions Error debugging (which feature can we blame…) 15 What is great about learning with Counts ?

Learning with Counts : in Azure ML

HDInsight: large data storage and map-reduce processing Azure ML: cloud ML and analytics accessible anywhere Learning with Counts: intuitive, flexible large-scale ML solution Putting it all together

Thanks for your time Useful Links: Sign up for your free Azure ML Trial - Free tutorial on how to use Azure ML Need Azure ML for teaching in classroom ? - Contact the speakers Other Questions ? - Contact the speakers Speakers :- Misha Bilenko : Girish Nathan –