A theory of reverse engineering N.Y. Louis Lee (1) & P.N. Johnson-Laird (2) (1)Department of Educational Psychology, Faculty of Education, The Chinese University of Hong Kong (2) Department of Psychology, Princeton University Presented at the 2 nd London Reasoning Workshop 29 August 2007

1. What is reverse engineering? Reverse engineering: process of inferring how to assemble known components so that they match the performance of a target system A major industry! No psychological theories of it exist

Test-bed: Boolean systems. Defined only by Boolean logic e.g., Boolean electric circuits. Two switches control a light – comes on when only one of the switches is on, not both (“A or-else B”) Cf. Reverse engineering of computer software

2. A theory Two principal strategies: –focus on the positive outcomes one at a time, e.g., the light comes on –focus on the input components one at a time, e.g., the switches Individuals apply constraints when they implement the strategies while constructing a circuit (e.g., do not connect two terminals of a switch with a wire)

Difficulty of reverse engineering A reverse engineering problem should be harder when there is: (1)a greater number of variable components that affect the performance of a system (2)a greater number of distinct states the system that yield positive outputs (3)greater dependence of components on one another in determining the output of the system

3. Experiments Expt 1: A basic test of the theory’s prediction of difficulty. What strategies do people use? Expt 2: How do the number of positive outcomes and dependence matter? Expt 3: Does the physical layout of the circuit matter?

Experiment 1 Two isomorphic blocks of three problems: AND, OR, OR-ELSE. First block in the electric domain, second block in water flow domain Ss to draw solutions and to think aloud

Expt 1: Interface (electric) Solution:

Expt 1: Interface (water flow) Solution:

Expt 1: Difficulty prediction AND: independent – either switch can turn light off independently. One positive outcome OR: independent – either switch can turn the light on. But, three positive outcomes OR-ELSE: dependent – a the effect of one switch is dependent on the position of the other Hence, predicted difficulty: AND < OR < OR-ELSE

Expt 1: Results (strategies) (1) S9 – ‘Or’ problem (2) S16 – ‘Or’ problem Bias for the first strategy (92%) Few violations of local constraints, e.g., connecting two terminals of a switch

Expt 1: Results (accuracy) Reliable trend in both domains 1, no reliable differences between domains 2 or blocks 3 1.Page’s L = 264.0, z = 3.79, p<< Wilcoxon, z = 0.0, p = Wilcoxon, z = 1.27, p =.21

Expt 1: Results (latency) Overall trend reliable 1, no reliable difference between domains 2 or blocks 3 1.Page’s L = 272.5, z = 5.14, p<<.001; s vs 327.1s, Wilcoxon, z = 1.36, p =.17, n.s s vs 302.6s, Wilcoxon, z = 1.53, p=.126, n.s.

Experiment 2 Tested the number of positive outcomes and dependence orthogonally 5 problems varying in no. positive outcomes (1 vs 3 vs 5) and dependence (independent vs dependent): No. of positive outcomes 135 IndependentA and (B and C)A and (B or C)A or (B and C) Dependent (Iff (A and B) then C) and (A or B) (Iff (A or-else B) then C) or (A and B)

Expt 2: Results (accuracy) Effect of no. of positive outcomes 1, and dependence 2 1.Page’s L = 210.0, z = 4.74, p<< Wilcoxon, z = 3.52, p<<.01

Expt 2: Results (latency) Effect of no. of positive outcomes 1 and dependence 2 1.Page’s L = 270.5, z = 4.82, p<< Wilcoxon, z = 3.92, p<<.01

Expt 2: Discussion Effect of no. of positive outcomes only attributable to the problem with one positive outcome Explanation: possible to ‘decompose’ an independent circuit by building it up part by part regardless of the number of positive instances. Not possible for dependent circuits A and (B or C) (Iff (A and B) then C) and (A or B) (A B C, A ¬B ¬C, ¬A B ¬C)

Expt 3: Congruence A congruent system: interconnections run physically in parallel. Easier to reverse engineer than an incongruent system: interconnections have to cross one another Compare the following two problems: A or-else BIff A then B CongruentIncongruent

Expt 3: Congruence Solutions to AND, OR, OR-ELSE in Expt 1: Results in Expt 1 attributable to congruence? Expt 3: Congruent vs Incongruent block of the three problems

Expt 3: Results Trend reliable for overall 1, congruent, and incongruent problems; Congruent problems easier than incongruent problems 2 1.Accuracy: Page’s L = 262.5, z = 3.56, p <<.001; Page’s L = 212.5, z = 4.35, p << Accuracy: Wilcoxon, z = 1.66, p <.05; Latency: Wilcoxon z = 2.50, p <.01 Accuracy (%):Latency:

4. Conclusions Reverse engineering: devising an underlying mechanism for a device of a known functionality Naïve individuals use two principal sorts of strategy when they reverse engineer Boolean systems: focus on each outcome at a time, or each switch at a time Four factors of difficulty: dependence (crucial!), no. of positive outcomes, no. of input components, congruence