1. Nick Bloom, 149, 2015 The Modern Firm in Theory & Practice Nick Bloom (Stanford Economics and GSB) Lecture 8: Randomized control trials 1

2. Nick Bloom, 149, 2015 2 Experiments and IDinsight case Targets management practices

3. Nick Bloom, 149, 2015 Formal choice testing growing in importance in management practices Driven by a few factors Rise of “big data” (more information to test on) Rise of online firms (easier to code for online decisions) Increasing statistical sophistication of managers

4. Nick Bloom, 149, 2015 In hi-tech often called “A/B testing”

5. Nick Bloom, 149, 2015 And has been around for a while

6. Nick Bloom, 149, 2015 A/B testing is happening all the time Typically runs continuously, often with multiple versions (e.g. A/B/C.D) – Facebook, Google, Amazon, Expedia etc all openly acknowledge this Variety of new software packages being introduced to make this easy to do (e.g. “Max A/B”, Google “Content Experiments” etc) One operational note is think about test power – that is having enough sample to test differences in subgroups

7. Nick Bloom, 149, 2015 Differential responses in subgroups OverallDay-timeNight-time Total tests4m3m1m Treatment A3%1%9% Treatment B4%5%1% If you found this on the first test what would you do next?

8. Nick Bloom, 149, 2015 In pharmaceutical testing Pharmaceutical firms are required by regulators (e.g. the Federal Drug Administration in the US) to test drugs Long history – the first trial was carried out by the UK Medical Research Council between 1946-47 on TB Run through various phases building up in size (small scale tests, pilots and then full waves)

9. Nick Bloom, 149, 2015 Q1: Costs and benefits of a formal experiment (an RCT) for a firm making a decision?

10. Nick Bloom, 149, 2015 Q1: Costs and benefits of an RCT for a firm making a decision? Benefits Information –Magnitudes of outcomes –Unexpected results Easier to execute Costs Logistics Speed Need large sample of decisions Admitting “I don’t know”

11. Nick Bloom, 149, 2015 IDinsight Case

12. Nick Bloom, 149, 2015 Q2: Describe IDinsight – the firm, the idea and the three-stage of an engagement

13. Nick Bloom, 149, 2015 Q2: Challenges of starting up IDinsight?

14. Nick Bloom, 149, 2015 Q3: What should they do? - Take on Laos and Cambodia - Take on only one - Take on neither

15. Nick Bloom, 149, 2015 I’m not sure of the outcome, but they seemed to succeed!

16. Nick Bloom, 149, 2015 16 Experiments and Idinsight case Targets management practices

17. Nick Bloom, 149, 2015 Score(1): Goals are based purely on accounting figures (with no clear connection to shareholder value). (3): Corporate goals are based on shareholder value but are not clearly communicated down to individuals (5): Corporate goals focus on shareholder value. They increase in specificity as they cascade through business units ultimately defining individual performance expectations. (9) Target interconnection

18. Nick Bloom, 149, 2015 Target interconnection (9): all countries, manufacturing 18 All countries, manufacturing firms (100 to 5000 employees), 9813 observations Average 3.01

19. Nick Bloom, 149, 2015 Score(1): Top management's main focus is on short term targets. (3): There are short and long-term goals for all levels of the organization. As they are set independently, they are not necessarily linked to each other (5): Long term goals are translated into specific short term targets so that short term targets become a "staircase" to reach long term goals (10) Target time horizon

20. Nick Bloom, 149, 2015 Target time horizon (10): all countries, manufacturing 20 All countries, manufacturing firms (100 to 5000 employees), 9814 observations Average 2.99

21. Nick Bloom, 149, 2015 Score(1): Goals are either too easy or impossible to achieve; managers provide low estimates to ensure easy goals (3): In most areas, top management pushes for aggressive goals based on solid economic rationale. There are a few "sacred cows" that are not held to the same rigorous standard (5): Goals are genuinely demanding for all divisions. They are grounded in solid, solid economic rationale (11) Targets are stretching

22. Nick Bloom, 149, 2015 Targets are stretching (11): all countries, manufacturing 22 All countries, manufacturing firms (100 to 5000 employees), 9819 observations Average 3.00

23. Nick Bloom, 149, 2015 Score(1): Performance measures are complex and not clearly understood. Individual performance is not made public (3): Performance measures are well defined and communicated; performance is public in all levels but comparisons are discouraged (5): Performance measures are well defined, strongly communicated and reinforced at all reviews; performance and rankings are made public to induce competition (12) Performance clarity

24. Nick Bloom, 149, 2015 Performance clarity (12): all countries, manufacturing 24 All countries, manufacturing firms (100 to 5000 employees), 9821 observations Average 2.67

25. Nick Bloom, 149, 2015 Wrap-up: Experiments and IDinsight