TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer in Test - Google AdSense

Guessing Game How many test cases are executed per year at Google? 100 x 10^9 ~ years of test execution per year

TestIstanbul Conferences 2012 Agenda Case study: AdSense Frontend Google's Innovation Culture The Google Scale Developer Engagement Testing Infrastructure Web Testing at Google Key learnings Q&A

G OOGLE ' S I NNOVATION C ULTURE

TestIstanbul Conferences 2012 Google's Innovation Culture "Launch early, launch often" "The only true failure is not attempting the audacious." Take risks Launch and iterate

TestIstanbul Conferences 2012 Launch and iterate Google+: 100 new features in first 90 days Search: Idea-to-user in less than 48 hours Chrome: A new version every 6 weeks

T HE G OOGLE S CALE

TestIstanbul Conferences 2012 The "Google Scale" 60 offices, 30+ countries 1 giant code base everything built at latest check-in 11K people in R&D 10s of thousands of changed code lines / day 10s of products, 100s of feature teams Weekly to sub-daily release cycles How is this possible?

TestIstanbul Conferences 2012 Paradigm Shifts! Developer owns code Test engineering = quality assurance Reliance on people to scale Test to prevent bugs Spread thinly Developer owns code + test Test engineering = Productivity enablers Reliance on common test infrastructure to scale Capacity to fix bugs fast Staff high impact projects

D EVELOPER E NGAGEMENT

TestIstanbul Conferences 2012 Developer engagement

TestIstanbul Conferences 2012 Developer engagement

TestIstanbul Conferences 2012 Developer engagement

TestIstanbul Conferences 2012 Test Engineering "Engineering productivity" Test leadership Process automation Code health signals Test evangelism

TestIstanbul Conferences 2012 Test Leadership Make existing code testable Implement the first N tests, lead by example Use data to define testing priorities Remove testing bottlenecks Make it faster to write and run the tests

TestIstanbul Conferences 2012 Process automation Continuous integration Automated releases Release tracking Production monitoring Automated code quality checks

TestIstanbul Conferences 2012 Code health signals Latency (ms) Code coverage If you don't measure it, you can't control it.

TestIstanbul Conferences 2012 Metrics: Incremental Code Coverage

TestIstanbul Conferences 2012 Test Evangelism Codelabs & Techtalks ("yt: code health talks") Testing on the Toilet (TotT) Internal & external test automation events (GTAC) Testing intergrouplet Test certification Make testing part of the development culture

TestIstanbul Conferences 2012 Test Certified Programme An indication test automation maturity Show value every stage Test Certified Levels 1: Setup tools – get familiar with testing 2: Write tests for all new code 3: Get all code under test 4: Improve test coverage 5: Invest in testable code

A D S ENSE F RONTEND Case Study

TestIstanbul Conferences 2012 Case Study: AdSense Frontend Before Time until next release: months Build broken for weeks 5+ release candidates Reliance on manual testing for release confidence Near absent test automation coverage One man to release them all

TestIstanbul Conferences 2012 Case Study: AdSense Frontend Now Developers write tests for their features Affected tests run at every code change Manual testing: only exploratory Build gets fixed in hours weekly releases <2 release candidates per release 6 auto-staging releases per day Push-button releases

T EST I NFRASTRUCTURE

TestIstanbul Conferences 2012 Life of a code change (AdSense Frontend) minuteshoursdays code review Push to production

TestIstanbul Conferences 2012 Life of a code change (AdSense Frontend) code ready for submission Submit code through presubmit queue tests fail? minuteshoursdays Push to production code review

TestIstanbul Conferences 2012 Life of a code change (AdSense Frontend) code ready for submission Submit code through presubmit queue tests fail? code submitted impacted tests run minuteshoursdays Push to production builcops/build breakers notified code review

TestIstanbul Conferences 2012 Life of a code change (AdSense Frontend) code ready for submission Submit code through presubmit queue tests fail? code submitted impacted tests run minuteshoursdays Push to production builcops/build breakers notified push to production dry run code review

TestIstanbul Conferences 2012 Test enablers test engineers worked on optimizing this process makes it easier to create/maintain tests o developers are more likely to create more tests lots of these steps are using a common infrastructure o search for "continuous integration google"

W EB T ESTING

TestIstanbul Conferences 2012 Browser Automation Selenium 2.0/WebDriver Core browser automation platform (others exist) A collection of language specific bindings to control a browser Emulate true user actions Support for major browser manufacturers

TestIstanbul Conferences 2012 Browser Automation WebDriver driver = new FirefoxDriver(); driver.get(" WebElement element = driver.findElement(By.name("q")); element.sendKeys("Cheese!"); element.submit(); assertTrue(driver.getTitle().contains("Cheese")); How does this scale?

TestIstanbul Conferences 2012 Browser automation Test Runner Shard 1 Browser 1 TestBrowser Shard 1 Browser 2 TestBrowser Shard N Browser M TestBrowser Shard N Browser1 TestBrowser Output Aggregator

TestIstanbul Conferences 2012 Browser automation (AdSense Frontend) Test Runner Shard 1 Chrome Test(s)Browser(s) Server Database... Shard 1 Firefox Test(s)Browser(s) Server Database... Shard 1 Internet Explorer Test(s)Browser(s) Server Database...

TestIstanbul Conferences 2012 Browser Automation: Test enablers Test engineers worked on making it easy to write/maintain browser tests o developers are more likely to more tests Browser tests run at every affecting code change some costly tests are throttled

K EY L EARNINGS

TestIstanbul Conferences 2012 Key Learnings Push testing up-stream o Early developer testing brings biggest wins o Position ourselves as productivity enablers o Make/Hire developers passionate about testing Invest in a great testing infrastructure o Reduce the cost of writing tests to zero o Automate as much as possible

TestIstanbul Conferences 2012 Thanks "Google Testing Blog" "Google Test Automation Conference (GTAC)" "Google Engineering Tools" "WebDriver" or "Selenium" "GTAC" "Clean code talks" "Web Testing at Google GTAC" "Build Software at Google Scale GTAC"