Structured Prediction Cascades Ben Taskar David Weiss, Ben Sapp, Alex Toshev University of Pennsylvania

Supervised Learning Learn from Regression: Binary Classification: Multiclass Classification: Structured Prediction:

Handwriting Recognition `structured’ xy

Machine Translation xy ‘Ce n'est pas un autre problème de classification.’ ‘This is not another classification problem.’

Pose Estimation xy © Arthur Gretton

Structured Models space of feasible outputs scoring function parts = cliques, productions Complexity of inference depends on “part” structure

Supervised Structured Prediction Learning Prediction Discriminative estimation of θ Data Model: Intractable/impractical for complex models Intractable/impractical for complex models 3 rd order OCR model = 1 mil states * length Berkeley English grammar = 4 mil productions * length^3 Tree-based pose model = 100 mil states * # joints

Approximation vs. Computation Usual trade-off: approximation vs. estimation – Complex models need more data – Error from over-fitting New trade-off: approximation vs. computation – Complex models need more time/memory – Error from inexact inference This talk: enable complex models via cascades

An Inspiration: Viola Jones Face Detector Scanning window at every location, scale and orientation

Classifier Cascade Most patches are non-face Filter out easy cases fast!! Simple features first Low precision, high recall Learned layer-by-layer Next layer more complex C1C1 C2C2 C3C3 CnCn Non-face Face

Related Work Global Thresholding and Multiple-Pass Parsing. J Goodman, 97 A maximum-entropy-inspired parser. E. Charniak, 00. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking, E Charniak & M Johnson, 05 TAG, dynamic pro- gramming, and the perceptron for efficient, feature-rich parsing, X. Carreras, M. Collins, and T. Koo, 08 Coarse-to-Fine Natural Language Processing, S. Petrov, 09 Coarse-to-fine face detection. Fleuret, F., Geman, D, 01 Robust real-time object detection. Viola, P., Jones, M, 02 Progressive search space reduction for human pose estimation, Ferrari, V., Marin-Jimenez, M., Zisserman, A, 08

What’s a Structured Cascade? What to filter? – Clique assignments What are the layers? – Higher order models – Higher resol. models How to learn filters? – Novel convex loss – Simple online algorithm – Generalization bounds F1F1 F2F2 F3F3 FnFn ?????? ‘structured’

Trade-off in Learning Cascades Accuracy: Minimize the number of errors incurred by each level Efficiency: Maximize the number of filtered assignments at each level Filter 1 Filter 2 Filter D Predict

Max-marginals (Sequences) a b c d a b c d a b c d a b c d Score of an output: Compute max (*bc*) Max marginal:

Filtering with Max-marginals Set threshold Filter clique assignment if a b c d a b c d a b c d a b c d

Filtering with Max-marginals Set threshold Filter clique assignment if a b c d a b c d a b c d a b c d Remove edge bc

Why Max-marginals? Valid path guarantee – Filtering leaves at least one valid global assignment Faster inference – No exponentiation, O(k) vs. O(k log k) in some cases Convex estimation – Simple stochastic subgradient algorithm Generalization bounds for error/efficiency – Guarantees on expected trade-off

Choosing a Threshold Threshold must be specific to input x – Max-marginal scores are not normalized Keep top-K max-marginal assignments? – Hard(er) to handle and analyze Convex alternative: max-mean-max function: max scoremean max marginal m = #edges

Choosing a Threshold (cont) Max marginal scores Mean max marginal Max score Score of truth Range of possible thresholds α = efficiency level

Example OCR Cascade abcdefghijkl… Mean ≈ 0 Max = 368 α = 0.55 Threshold ≈ 204 acdefgijl bhk… b h k

Example OCR Cascade b h k a e nu r a b dga c egnoua e mnuwb h k

b h k a e nu r a b dga c egnou ▪b▪b▪h▪h▪k▪k bahaka be he…aaabad ag ea…aaacae ag an… a e mnuw aaaeam an au…

Example OCR Cascade ▪b▪b▪h▪h▪k▪k bahaka be he…aaabad ag ea…aaacae ag an…aaaeam an au… Mean ≈ 1412 Max = 1575 α = 0.44 Threshold ≈ 1502

▪b▪b ba be aaab ag ea aa acae ag an aaaeam an au ▪b▪b ba be aaab ag ea aa acae ag an aaaeam an au Example OCR Cascade ▪h▪h▪k▪k hakahe… ad… … … Mean ≈ 1412 Max = α = 0.44 Threshold ≈ 1502

Example OCR Cascade ka ▪h▪h▪k▪k hahe… ad… … … Mean ≈ 1412 Max = 1575 ▪h▪h▪k▪k hakaheke aded kn do owom nd α = 0.44 Threshold ≈ 1502

Example OCR Cascade ▪h▪h▪k▪k hakaheke aded kn do owom nd … … ▪▪h▪▪k ▪ha▪ka▪he▪ke hadkadhedked adoedondo dowdom

10x10x1210x10x2440x40x2480x80x24110×122×24 Example: Pictorial Structure Cascade Upper arm Lower arm H H T T ULA LLA URA LRA k ≈ 320,000 k 2 ≈ 100mil

Filter loss  If score(truth) > threshold, all true states are safe Efficiency loss  Proportion of unfiltered clique assignments Quantifying Loss

Learning One Cascade Level Fix α, solve convex problem for θ Minimize filter mistakes at efficiency level α No filter mistakesMargin w/ slack

An Online Algorithm Stochastic sub-gradient update: Features of truth Convex combination: Features of best guess + Average features of max marginal “witnesses”

Generalization Bounds W.h.p. (1-δ), filtering and efficiency loss observed on training set generalize to new data Expected loss on true distribution

)  -ramp( Generalization Bounds W.h.p. (1-δ), filtering and efficiency loss observed on training set generalize to new data - Empirical  -ramp upper bound 0 1

Generalization Bounds W.h.p. (1-δ), filtering and efficiency loss observed on training set generalize to new data n number of examples m number of clique assignments number of cliques B

Generalization Bounds W.h.p. (1-δ), filtering and efficiency loss observed on training set generalize to new data Similar bound holds for L e and all ® 2 [0,1]

OCR Experiments Dataset:

Efficiency Experiments POS Tagging (WSJ + CONLL datasets) – English, Bulgarian, Portuguese Compare 2 nd order model filters – Structured perceptron (max-sum) – SCP w/ ® 2 [0, 0.2,0.4,0.6,0.8] (max-sum) – CRF log-likelihood (sum-product marginals) Tightly controlled – Use random 40% of training set – Use development set to fit regularization parameter ¸ and α for all methods

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) English

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) Portuguese

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) Bulgarian

English POS Cascade (2 nd order) The cascade is efficient. DT NN VRB ADJ

Pose Estimation (Buffy Dataset) “Ferrari score:” percent of parts correct within radius

Pose Estimation

Cascade Efficiency vs. Accuracy

Features: Shape/Segmentation Match

Features: Contour Continuation

Conclusions Novel loss significantly improves efficiency Principled learning of accurate cascades Deep cascades focus structured inference and allow rich models Open questions: – How to learn cascade structure – Dealing with intractable models – Other applications: factored dynamic models, grammars

Thanks! Structured Prediction Cascades, Weiss & Taskar, AISTATS10 Cascaded Models for Articulated Pose Estimation, Sapp, Toshev & Taskar, ECCV10

Learning to Filter Optimization problem: subject to Maximize efficiency Hard constraint on filtering accuracy Not convex due to coupling ofααand θ Learn θ for fixed ®

Safe filtering When does filtering make mistakes? Lemma: a b c d a b c d a b c d a b c d Clique assignments with max-score lower than the truth are safe to filter

Safe filtering When does filtering make mistakes? Lemma: To guarantee no mistakes, set score of ground truth > threshold + margin

Structured Prediction Underlying goal of learning large-scale graphical models: prediction of structured outputs Structured: multivariate, correlated, constrained

Object Segmentation Spatial structure xy

Natural Language Parsing The screen was a sea of red Recursive structure xy

a b l m n wx

Word Alignment What is the anticipated cost of collecting fees under the new proposal? En vertu des nouvelles propositions, quel est le coût prévu de perception des droits? xy What is the anticipated cost of collecting fees under the new proposal ? En vertu de les nouvelles propositions, quel est le coût prévu de perception de les droits ? Combinatorial structure

Handwriting Recognition brace xy

Structured Prediction Protein folding Machine translation ‘Ce n'est pas un autre problème de classification.’ InputOutput AVITGACERDLQCG KGTCCAVSLWIKSV RVCTPVGTSGEDCH PASHKIPFSGQRMH HTCPCAPNLACVQT SPKKFKCLSK ‘This is not another classification problem.’ huge!

Pose Estimation xy

Cascade Model Complexity Filter 1 Filter 2 Filter D Predict

Example: OCR task abcde…abcde… abcde…abcde… abcde…abcde… abcde…abcde… abcde…abcde… Filter states

Example: OCR task abdabd cenrcenr acdeoacdeo aceracer aceoaceo Expand to bigrams…

Example: OCR task abdabd ac bc dc ae be de … ca ea na ra cc bc … aa ca da ea oa ac … aa ca ea ra ac cc … Compute max-marginals Filter edges

Example: OCR task abab ac be de br ar … ca na ra cd rd … ca da ac dc … ea ac re … Expand to trigrams…

Example: OCR task  a  b Expand to trigrams…  ac  be  de  ar  br aca ana ara bca bna bra cca rca cda rda rac dac aea aac ace are Compute max-marginals Filter edges

Visualizing L e vs. L f trade-off Train w to minimize loss function on training set For each ² 2 [0,0.002]: – Choose ® to minimize L e subject to L f · ² on the development set – Compute L e on test set – Plot ( ²,L e ) 10 replicates to estimate variance Error Cap (%) on Development Set % 0.05% errors on devel 50% pruned on test (Chose ® =0.43)

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) English

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) Portuguese

POS Tagging Results Error Cap (%) on Development Set Max-sum (SCP) Max-sum (0-1) Sum-product (CRF) Bulgarian

Efficiency within cascade SPF outperforms SP and CRF baselines – SP unable to achieve very low error rate What about the rest of the cascade? – Train/test methods on sparse lattice output by a unigram SCP model