1. On Sample Based Testers
Oded Goldreich, Weizmann Institute
Dana Ron, Tel Aviv University

2. (Standard) Testing – A quick reminder
Let OBJ be an object (function) of size N
a Testing Algorithm for a (prespecified) property P is given a proximity parameter (0,1];
- If OBJ has P should accept with prob  2/3; - If OBJ is -far from having P should reject with prob  2/3.
Distance is normalized Hamming
To this end the algorithm is given query access to OBJ x1
f(x1)
Query complexity q(N,) should be sublinear in N. x2
OBJ=f
f(x2)
...
This model is as defined by Rubinfeld and Sudan, and most results in property testing are in this model.

3. Sample-Based Testing f
What if we don’t have query access to the object, but can only obtain (uniform) random samples?
Namely, if the tested object is a function f : [N]  R, and we can only
obtain pairs (x,f(x)), where x is uniformly distributed in [N].
(x1,f(x1))
(x2,f(x2)) f
...

4. Sample-Based Testing Background
Setting of sample-based testing is similar to the setting in Learning Theory (learning under the uniform distribution), and was defined in “Property testing and its connection to learning and approximation” [Goldreich, Goldwasser, R].
However, in [GGR] and most works since, results on sample-based testing were mainly negative, essentially establishing the necessity of queries.
Two exceptions: Decision Trees over [0,1]d [Kearns, R], Interval Functions (d=1) and Linear Threshold Functions under Gaussian distribution [Balkan, Blais, Blum, Yang] (as part of their study of Active Testing) 1 1

5. Sample-Based Testing Background
Note: Many works on testing and estimating properties of distributions (starting with [Batu, Fortnow, Rubinfeld, Smith, White]) where the object is a distribution D, the algorithm gets sample distributed according to D, and should test if D has property P. x1 x2 D
...
Differs from sample-based testing where: (1) Object is function; (2) Underlying distribution is fixed (uniform); (3) Get function labels; (4) Tested property is of function.

6. This Work
We were interested in understanding the relation between sample-based testing and other models of testing, as well as variants of sample-based testing (as just defined).

7. Our Results: 1. Relation to POTs
Proximity Oblivious Testers (POT) that are “fair” imply sublinear sample-based testers.
(q,)-POT: Performs (const) q queries; - If fP, accepts with prob  c; - If fP, accepts with prob  c- (P(f))
Fair: if each query (almost) uniformly distributed.
(q,)-POT that is fair
Sample-based tester with sample complexity O(N1-1/q/(())2+3/q).
Comments: (1) Stronger notion of fairness (e.g., pairs of queries unif. dist.) gives better sample complexity;
(2) Some notion of fairness is necessary in general
(3) Fairness not necessary for Boolean functions, q=2 and POT with 1-sided error (c=1) [Fischer, Goldhirsh, Lachish].
dist of f to P
Example, Linearity: query on x1,x2 and x1+x2

8. Our Results: 2. Dense-Graphs Model
Quasi-Canonical testers in dense-graphs model imply sublinear sample-based testers.
Dense-Graphs model: G=(V,E) represented by adjacency matrix: can query if (u,v) E and N = n2 (n=|V|).
Quasi-Canonical: Select v(N,) vertices uniformly at random; Query all pairs; Decide based on induced subgraph, possibly randomly.
Quasi-Canonical tester (using v(N,) vertices)
Sample-based tester with sample complexity O(N1-1/(v(N,)-1) ).
Comment: Better results for graph partitioning problems: e.g., for k-colorability O(N1/2log k/)
Not time-efficient, but (for k-colorability) can get time N1-(1/2k)g(k,) (with N1-1/2k(k/2) samples). v u 1
(N1/2 necessary even for k=2)

9. Our Results: 3. Distribution-Free Testing
Distribution-free sample-based testing is related to 1-sided error sample-based testing (under unif. dist.) non-trivially.
Distribution-free (sample-based)* testing: Unknown distribution D, algorithm get samples (x,f(x)) where xD, and distance (to property) is defined w.r.t. D.
For property P, DF(P): distribution-free sample complexity, OSE(P): 1-sided error sample complexity, consider const .
Every P: OSE(P) = Õ(DF(P)2).
Exist P s.t. OSE(P) = (DF(P)).
Exist P s.t. OSE(P) = (DF(P)).
Exist P s.t. OSE(P) = o(DF(P)).
(x1D,f(x1)) f
(x2D,f(x2))
P is natural
(E.g., OSE(P) = O(log(DF(P))), or even OSE(P) = O(1) when DF(P)= (N).)
(*) Distribution-free testing previously studied when queries are also allowed (e.g., by Halevi and Kushilevitz).

10. Our Results: 4. Testing Distributions
Distribution testing reduces to sample-based testing of symmetric properties (articulates [Sudan]).
Distribution testing: For unknown distribution D, algorithm gets samples xD and should decide whether D has property P or is -far from having P, where distance is L1 (statistical/variation).
Symmetric properties of functions f: X  R: Property is invariant under permutations over X (in other words, property defined by {Ny}yR where Ny=|{xX : f(x)=y}| ).
Let P= U Pm be property of distributions s.t. Pm has support that is a subset of Sm, |Sm|=m. Denote sample complexity of testing P by s(m,).
Exists symmetric property P’=U PN,m of functions, with domain [N] and range Sm such that for every N  cn2/4 s(m,) = O(s’(N,m,/2)), s(m,) = (s’(N,m,2)).

11. Proof Sketch of OSE(P) = Õ(DF(P)2)
Recall: DF(P) is distribution-free sample complexity, OSE(P) is 1-sided error (unif. dist.) sample complexity.
Let T’ be a distribution-free sample-based algorithm with complexity s = o(N1/2) that errs with prob < 1/6. (The case s = (N1/2) is trivial.)
Define T to be the algorithm that takes a uniform sample of size r=O(s2), labeled by f:
((x1,f(x1)),…,(xr,f(xr)), xi  U
and accepts i.f.f  g  P s.t. g(xi) = f(xi) for every 1  i  r.
- By definition of T, if f  P, then T always accepts (as required from 1-sided error testers).
- Remains to show that if f is -far from P, then T rejects with probability at least 2/3.

12. Proof Sketch of OSE(P) = Õ(DF(P)2), cont’
Assume in contradiction: Exists f that is -far from P, which T accepts with probability > 1/3 (recall: T uses uniform sample)
That is, for more than 1/3 of sample sets x = {x1,…,xr},  gx  P s.t. gx(xi) = f(xi)
For each such “bad” sample set x, consider distribution x that is uniform over x={x1,…,xr}.
Key: Given sample of size s distributed according to x, labeled by f, T’ must accept w.p.  5/6.
Implies: If select x = {x1,…,xr} uniformly, and give T’ subsample of size s, must accept w.p.  (1/3)(5/6) > 1/6 + 1/10.
But, this dist over sample of size s is 0.1-close to uniform, on which T’ must accept w.p.  1/6. X
Recall: T’ is dist-free
contradiction

13. Wrapping up
We study relations between sample-based testing and other models of testing, as well as variants of sample-based testing. In particular, showed:
Proximity Oblivious Testers (POT) that are “fair” imply sublinear sample based testers.
Quasi-Canonical testers in dense-graphs model imply sublinear sample-based testers.
Distribution-free sample-based testing is related to 1-sided error sample-based testing (under uniform dist) non-trivially (always have quadratic upper bound but other than that, varies).
Distribution testing reduces to sample-based testing of symmetric properties (articulates [Sudan]).

14. Thanks