1. Implementing with Machine Learning
Geoff Hulten

2. Example of an Implementation: Laugh Finder
Indicates if the web page is funny x
Whatever.com/whatever.html
Browser Plugin
funnyWords = [ ‘enim’, …, ‘fugiat’, … ]
every time a page loads:
page = GetPageContext()
isFunny = false
for word in page.words:
if( word in funnyWords ):
isFunny = true
UpdateUserExperience( isFunny )
Blah blah blah…
Sapien eget mi proin sed libero enim. Purus sit amet volutpat consequat.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Issues:
Heuristic Model
Update via some plugin/app store?
Measure performance via…feedback? Star ratings?

3. Basic Machine Learning
Example File Format: <𝑡>, <𝑛>,
< 𝑤𝑜𝑟𝑑 0 >, < 𝑤 0 >,
< 𝑤𝑜𝑟𝑑 1 >, < 𝑤 1 >, …,
< 𝑤𝑜𝑟𝑑 𝑛 > < 𝑤 𝑛 > x
Whatever.com/whatever.html
model.load(<dataFile>)
every time a page loads:
page = GetAppContext()
x = Featurize(model.words, page.words)
y = model.predict( [ x ] )
isFunny = y[0] > model.𝑡
UpdateUserExperience( isFunny )
Blah blah blah…
Construct feature vector in correct order by testing for presence of selected words…
Sapien eget mi proin sed libero enim. Purus sit amet volutpat consequat.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Call into inference engine to apply the model to the feature vector…
Apply tuned threshold to achieve desired (stable) operating point…
Issues:
Heuristic Model
Update via some plugin/app store?
Measure performance via…feedback? Star ratings?

4. ‘Run Time’ vs ‘Creation Time’
Param sweep
Model in Run Time
Intelligence Creation Environment
model.load(<dataFile>)
every time a page loads:
page = GetAppContext()
x = Featurize(model.words, page.words)
y = model.predict( [ x ] )
isFunny = y[0] > 𝑡
UpdateUserExperience( isFunny )
for every pageID in trainingSet:
( page, label ) = LoadPageFromLog( pageID )
pages.append( page ); Y.append( label )
selectedFeatures = FeatureSelect( xTrain, <params> )
for page in pages:
x = Featurize( selectedFeatures, page.words )
xTrain.append( x )
model = train( <more params>, xTrain , yTrain )
𝑡 = FindThreshold( thresholdSet )
model.save(selectedFeatures, 𝑡, model)
# Deploy it…
Must be in sync!
Must be in sync!
Evaluation
More param sweeps
Intelligence Management

5. Intelligence Management
Intelligence Creation Environment
Intelligence Management
Intelligence Runtime
Compile and push app
Periodically build new model
New Model
Verify new intelligence
Decide when to deploy
Manage Deployment
Host Model in update service
Check for new model
every time app starts:
UpdateModel( <server>, <dataFile> )
model.load( <dataFile> )
every time a page loads:
page = GetAppContext()
# etc…
Download and save
Hide latency (?)
Issues:
Heuristic Model
Update via some plugin/app store?
Measure performance via…feedback? Star ratings?

6. Monitoring Models every time a page loads: page = GetAppContext()
Verifying Model is Running as Expected
every time a page loads:
page = GetAppContext()
x = Featurize(model.words, page.words)
y = model.predict( [ x ] )
if random.random() < veryLowSamplingRate:
LogToServer(page, x, y, model.version)
Is it working? x
Verify in Creation Environment
Whatever.com/whatever.html
Report as:
Not funny…
Funny…
Blah blah blah…
Sapien eget mi proin sed libero enim. Purus sit amet volutpat consequat.
User Reports
Hand Label
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Verifying Model Quality is good
every time user clicks a report:
page = GetAppContext()
x = Featurize(model.words, page.words)
y = model.predict( [ x ] )
LogToServer(page, x, y, model.version, userReport)
Add to Training Set
Issues:
Heuristic Model
Update via some plugin/app store?
Measure performance via…feedback? Star ratings?
Of course things are more complex in practice…

7. Design Patterns for ML Corpus Centric Closed Loop
Where in the system do you use Machine Learning?
All the choices you have to make along the way…
We’ll get to more as the course goes on, but for now…
How do you get the data to do the learning?
Design Patterns for ML
Can be quite expensive…
Data Collection and Labeling Process
Intelligence Creation Environment
Intelligence Management
Intelligence Runtime
Data Collection
Corpus Centric
Training Corpus
Data Labeling
Intelligence Creation Environment
Intelligence Management
Intelligence Runtime
Explicit
Reporting UX
Survey
Implicit
Achieve the desired result
Forward to a friend
Laugh out loud (audio detection ?)
More discussion of this later…
Closed Loop
Training Corpus
Explicit and Implicit User Feedback
Can be tricky to get right…

8. ML Design Patterns Examples Best When Key Challenges Corpus Centric
Computer Vision, Xbox Kinect, Speech
Hard But Stable Problem
Can’t use data from customer interaction
Bootstrapping a new system
Collecting & Labeling Data
Sophisticated Modeling
Closed Loop
Self Driving Car,
Recommender Systems
Open-ended/Time-changing problems
Users and Intelligence Interact at Scale
Shaping User/ML Interactions
Orchestrating Evolving System
Scoring/Ranking
Ad Targeting, Search, Voice Assistants, Designer
Diverse at scale content creation
Find right Situation for Each Content
Exploring and Avoiding Feedback Loops
Adversarial
Spam Filtering, Malware detection, Account compromise, Anti-phishing
Hyper Time-changing (adversarial)
Buffer ML From Adversary Change Economics of Broader System
Reinforcement
Games (e.g. Alpha Go),
Robotics
Correct Labeling Hard for Human
High Scale Simulator to Learn From
Difficult to Converge
Expensive ( AlphaGo ~40 days 1 )
We’ll get to these as course progresses

9. Orchestration every time a page loads: page = GetAppContext()
Bad False Positive
every time a page loads:
page = GetAppContext()
if any of page.words in blockedWords:
isFunny = false # no matter what model says
Comedy Site that model always calls not funny (bad FN)
Lots of fake reports (abuse) x
Whatever.com/whatever.html
Report as:
Not funny…
Funny…
Blah blah blah…
Sapien eget mi proin sed libero enim. Purus sit amet volutpat consequat.
Could try hard to tune models for specific bad mistakes, but sometimes heuristics are cheaper and more robust.
( and sometimes heuristics turn your system into brittle-spaghetti gibberish… )
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Bad False Negative
More Reasons for Orchestration:
Driving Quality
Concept Changes
Costs Change
Users Change
Bugs and Issues
Etc.
every time a page loads:
page = GetAppContext()
if page.domain in whitelist:
isFunny = true # no matter what model says
Super non-funny text (bad FP)

10. Summary: Components of an Implementation
Intelligence Management
Verify new intelligence
Control rollouts:
Keep in sync
Clients/services
Support online evaluation
Intelligence Creation Environment
Intelligence Runtime
Intelligence Orchestration
Monitoring success
Inspect Interactions
Adapt as things change
Deal with mistakes
Updating thresholds
Drive the racecar
Program State -> Context
Execute Feature Code
Execute Model
Interpret Model Output
Control User Experience
Update:
Models
Feature Code
Telemetry -> Context
Feature code in sync
Computation & Data
All the training stuff…
Telemetry
Verifying outcomes
Training data
Selecting what to observe
Sampling
Summarizing