Rule Selection as Submodular Function Wentao Ding

Rule Selection Problem A combination of two weighted coverage functions. maximize 𝐻 𝑃𝑜𝑠 𝑌 ,𝑁𝑒𝑔 𝑌 s.t. 𝑗: 𝑡 𝑗 ∈𝑅 𝑖 𝑥 𝑖 > 𝑦 𝑗 𝑦 𝑗 ∈ 0,1 𝑥 𝑖 ∈ 0,1 R1 R2 R3 R4 : Positive : Negative

Submodular Function Submodular: Maximize a submodular function Def 1: ∀𝐴⊆𝐵. 𝑓 𝐴∪ 𝑥 −𝑓 𝐴 ≥𝑓 𝐵∪ 𝑥 −𝑓 𝐵 Def 2: ∀𝐴⊆𝐵. 𝑓 𝐴 +𝑓 𝐵 ≥𝑓 𝐴∪𝐵 +𝑓 𝐴∩𝐵 Coverage function is monotone submodular function. Maximize a submodular function usually NP-hard. ½-approx if symmetric, 𝑒−1 𝑒 -approx if monotone. Combination of two submodular functions Close under non-negative linear combinations. Difference of Submodular function: Inapproximable. Ratio of Submodular function: Depends

Rule Section Problem Let #𝑝𝑜𝑠𝑡𝑖𝑣𝑒 #𝑎𝑙𝑙 =𝐶, 𝑁𝑒𝑔 𝑌 #𝑎𝑙𝑙 =𝑓 𝑌 , 𝑃𝑜𝑠 𝑌 #𝑎𝑙𝑙 =𝑔 𝑌 . TP=𝑔 𝑌 FP=𝑓 𝑌 TN=1−𝐶−𝑓 𝑌 FN=𝐶−𝑔 𝑌 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= TP TP+FP = 𝑔 𝑌 𝑓 𝑌 +𝑔 𝑌 𝑅𝑒𝑐𝑎𝑙𝑙= TP TP+FN = 𝑔 𝑌 𝐶 𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚= TP+TN 1 =1−𝐶+𝑔 𝑌 −𝑓 𝑌 𝑭 𝟏 = 2TP 2TP+FP+FN = 2𝑔 𝑌 𝐶+𝑓 𝑌 +𝑔 𝑌 ≈ 𝑔 𝑌 𝑓 𝑌

Ratio of submodular function min 𝑁 𝑌 𝑃 𝑌 ∃𝐶.𝑃 𝑌 ≥𝐶 min 𝑁 𝑌 𝐶 max 𝑃 𝑌 𝑁 𝑌 ∃𝐵.𝑁 𝑌 ≤𝐵 max 𝑃 𝑌 𝐵 A Submodular Cover (SCSC) / Submodular Knapsack (SCSK) max 𝑓 𝑋 𝑋⊆𝑉∧𝑔 𝑋 ≥𝐶 𝑓,𝑔:submodular max 𝑔 𝑋 𝑋⊆𝑉∧𝑓 𝑋 ≤𝐵 𝑓,𝑔:submodular Can be approximated by ratio based greedy method

Curvature of submodular function 𝒦 𝑓 =1− min 𝑣∈𝑉 𝑓 𝑣 𝑉∖𝑣 𝑓 𝑣 ∈ 0,1 Approximate ratio with simple greedy algorithm 𝑓 𝑌 𝐺 𝑔 𝑌 𝐺 ≤ 1 1− exp 𝒦 𝑓 −1 ⋅ 𝑓 𝑌 ∗ 𝑔 𝑌 ∗ (if 𝒦 𝑃 =0, there exists an 𝜖-bounded scheme) R1 R2 R3 R4 𝒦 𝑔 = 1 2 𝒦 𝑓 =1 a (1+)-approximation for RS minimization in O(log(1/)) calls to the subroutine

Optimizing RS Ellipsoid Approximation：O 𝑛 log 𝑛 -approx. Related Learning Problem Submodular optimization Feature selection Data subset selection (Document Summarization) ……

Reference George L. Nemhauser, Laurence A. Wolsey, Marshall L. Fisher: An analysis of approximations for maximizing submodular set functions - I. Math. Program. 14(1): 265-294 (1978) Michele Conforti, Gérard Cornuéjols: Submodular set functions, matroids and the greedy algorithm: Tight worst-case bounds and some generalizations of the Rado-Edmonds theorem. Discrete Applied Mathematics 7(3): 251-274 (1984) Mukund Narasimhan, Jeff A. Bilmes: A Submodular-supermodular Procedure with Applications to Discriminative Structure Learning. UAI 2005: 404-412 Rishabh K. Iyer, Jeff A. Bilmes: Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints. NIPS 2013: 2436-2444 Rishabh K. Iyer, Jeff A. Bilmes: Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with Applications. UAI 2012: 407-417 Wenruo Bai, Rishabh K. Iyer, Kai Wei, Jeff A. Bilmes: Algorithms for Optimizing the Ratio of Submodular Functions. ICML 2016: 2751-2759

Thanks for listening Q & A