1. A theory of reverse engineering N.Y. Louis Lee (1) & P.N. Johnson-Laird (2) (1)Department of Educational Psychology, Faculty of Education, The Chinese University of Hong Kong (2) Department of Psychology, Princeton University Presented at the 2 nd London Reasoning Workshop 29 August 2007

2. 1. What is reverse engineering? Reverse engineering: process of inferring how to assemble known components so that they match the performance of a target system A major industry! No psychological theories of it exist

3. Test-bed: Boolean systems. Defined only by Boolean logic e.g., Boolean electric circuits. Two switches control a light – comes on when only one of the switches is on, not both (“A or-else B”) Cf. Reverse engineering of computer software

4. 2. A theory Two principal strategies: –focus on the positive outcomes one at a time, e.g., the light comes on –focus on the input components one at a time, e.g., the switches Individuals apply constraints when they implement the strategies while constructing a circuit (e.g., do not connect two terminals of a switch with a wire)

5. Difficulty of reverse engineering A reverse engineering problem should be harder when there is: (1)a greater number of variable components that affect the performance of a system (2)a greater number of distinct states the system that yield positive outputs (3)greater dependence of components on one another in determining the output of the system

6. 3. Experiments Expt 1: A basic test of the theory’s prediction of difficulty. What strategies do people use? Expt 2: How do the number of positive outcomes and dependence matter? Expt 3: Does the physical layout of the circuit matter?

7. Experiment 1 Two isomorphic blocks of three problems: AND, OR, OR-ELSE. First block in the electric domain, second block in water flow domain Ss to draw solutions and to think aloud

8. Expt 1: Interface (electric) Solution:

9. Expt 1: Interface (water flow) Solution:

10. Expt 1: Difficulty prediction AND: independent – either switch can turn light off independently. One positive outcome OR: independent – either switch can turn the light on. But, three positive outcomes OR-ELSE: dependent – a the effect of one switch is dependent on the position of the other Hence, predicted difficulty: AND < OR < OR-ELSE

11. Expt 1: Results (strategies) (1) S9 – ‘Or’ problem (2) S16 – ‘Or’ problem Bias for the first strategy (92%) Few violations of local constraints, e.g., connecting two terminals of a switch

12. Expt 1: Results (accuracy) Reliable trend in both domains 1, no reliable differences between domains 2 or blocks 3 1.Page’s L = 264.0, z = 3.79, p<<.001 2.Wilcoxon, z = 0.0, p = 1.0 3.Wilcoxon, z = 1.27, p =.21

13. Expt 1: Results (latency) Overall trend reliable 1, no reliable difference between domains 2 or blocks 3 1.Page’s L = 272.5, z = 5.14, p<<.001; 2.313.8s vs 327.1s, Wilcoxon, z = 1.36, p =.17, n.s. 3.338.3s vs 302.6s, Wilcoxon, z = 1.53, p=.126, n.s.

14. Experiment 2 Tested the number of positive outcomes and dependence orthogonally 5 problems varying in no. positive outcomes (1 vs 3 vs 5) and dependence (independent vs dependent): No. of positive outcomes 135 IndependentA and (B and C)A and (B or C)A or (B and C) Dependent (Iff (A and B) then C) and (A or B) (Iff (A or-else B) then C) or (A and B)

15. Expt 2: Results (accuracy) Effect of no. of positive outcomes 1, and dependence 2 1.Page’s L = 210.0, z = 4.74, p<<.001 2.Wilcoxon, z = 3.52, p<<.01

16. Expt 2: Results (latency) Effect of no. of positive outcomes 1 and dependence 2 1.Page’s L = 270.5, z = 4.82, p<<.001 2.Wilcoxon, z = 3.92, p<<.01

17. Expt 2: Discussion Effect of no. of positive outcomes only attributable to the problem with one positive outcome Explanation: possible to ‘decompose’ an independent circuit by building it up part by part regardless of the number of positive instances. Not possible for dependent circuits A and (B or C) (Iff (A and B) then C) and (A or B) (A B C, A ¬B ¬C, ¬A B ¬C)

18. Expt 3: Congruence A congruent system: interconnections run physically in parallel. Easier to reverse engineer than an incongruent system: interconnections have to cross one another Compare the following two problems: A or-else BIff A then B CongruentIncongruent

19. Expt 3: Congruence Solutions to AND, OR, OR-ELSE in Expt 1: Results in Expt 1 attributable to congruence? Expt 3: Congruent vs Incongruent block of the three problems

20. Expt 3: Results Trend reliable for overall 1, congruent, and incongruent problems; Congruent problems easier than incongruent problems 2 1.Accuracy: Page’s L = 262.5, z = 3.56, p <<.001; Page’s L = 212.5, z = 4.35, p <<.001 2.Accuracy: Wilcoxon, z = 1.66, p <.05; Latency: Wilcoxon z = 2.50, p <.01 Accuracy (%):Latency:

21. 4. Conclusions Reverse engineering: devising an underlying mechanism for a device of a known functionality Naïve individuals use two principal sorts of strategy when they reverse engineer Boolean systems: focus on each outcome at a time, or each switch at a time Four factors of difficulty: dependence (crucial!), no. of positive outcomes, no. of input components, congruence