1. Staff Software Engineer
Testing at LinkedIn
Sajid Topiwala
Staff Software Engineer
LinkedIn Test Tools

2. Agenda Overview of testing at LinkedIn
Software Development Life Cycle at LinkedIn
Sample of LinkedIn Test Tools

3. Source: https://www.express.co.uk/news/science/
BIG BANG

4. Support multiple platforms
Flagship introduces 3x3
Support multiple platforms
Pushing for 100% adoption of CI/CD for all of LinkedIn
Rewrite of Flagship App
Mobile apps launched
Test all features
Keep the lights on
Explosion of features and products
Launch CI/CD Pipeline
LinkedIn goes global
All of LinkedIn is in a single DB

5. LinkedIn CI/CD Pipeline
Develop
Review
Push Code CI
Publish
Staging
Canary
Production

6. linkedin.com
android/ios
frontend db
midtier
desktop

7. Integration with 3rd Party Cloud services
ATC Front Door S1 S2 S3 S4 S8 S7 S6 S5 S9 SA SB SC SD SE
Online Data Routing
Kafka Queue
Schema Registry
Service Discovery
ETL Pipeline
Distributed db
HADOOP
X-COLO
Mid Tier Services
Search
Graph
Zookeeper
Replication
linkedin.com
Integration with 3rd Party Cloud services
Fizzy
Auth/Seurity
API Servers
web
android/ios

8. Testing during Development
Review
Push Code CI
Publish
Staging
Canary
Production

9. Production Environment
QPROD
Alisa's Dev Box
 Browser
Foo*
Foo*
Alisa LI
Services
Foo
  Production Environment

10. Production Environment
QPROD
Service
Foo
 Browser
Dev Box
 Tunnel
 Portal LI
prod services
  Production Environment
Auth
 Tools Environment

11. Testing in CI Pipeline Push Code Develop Review CI Publish Staging
Canary
Production

12. Epsilon Performance testing happens very late in the release cycle
Can we make Performance testing part of our CI/CD pipeline

13. Epsilon Record / Replay Framework Records traffic in production
Replays earlier in the pipeline
Takes advantage of Linkedin’s microservice architecture
Epsilon core is built into every service at LinkedIn and is activated by turning on a config
Security/Privacy of member data is a prominent feature

14. Epsilon – Architecture (Record)…
Epsilon Recording Store R1
DS Req 1
DS Resp 1
DS Req 2
DS Resp 2
epsilonMode=RECORD
Service FOO
Downstream Service 1
Downstream Service 2
DS Req 1
DS Resp 1
DS Req 2
DS Resp 2 R1
Runs on regular production node
Sample at low QPS
Continuous recording / auto-expire

15. Epsilon – Architecture (Replay)
epsilonMode=REPLAY
Epsilon Recording Store
Service FOO
DS Req 1
DS Resp 1
DS Req 2
DS Resp 2
DS requests served by Epsilon … R1 R1
DS Req 1
DS Resp 1
DS Req 2
DS Resp 2 …

16. Epsilon In Action (Capacity Testing)
Epsilon Recording Store
QPS R1
Service FOO V1 R2
# returned responses R3
Service FOO V2 R4
QPS
Ramp up QPS to 2 versions of service and measure response throughput
Compare max QPS each version can handle

17. Testing in Canary Develop Push Code Canary Review CI Publish Staging
Production

18. Canary
When you think of canary analysis, think of canary in a coalmine...Coalminers used to bring canaries into a coalmine with them. They had much smaller lungs, so if the canary died it was a early warning system that something is wrong. We use the same technique when deploying to produciton.
Challenges
1.) Quality of data
2.) Difference in input requests so we have to run statistical model and that gives less confidence in data
3.) Improperly tuned rules
PRODUCTION

19. Testing in Production Develop Review Push Code CI Publish Staging
Canary
Production

20. Is a service resilient? LINKEDIN A LINKEDIN B

21. RESOURCE NOT FOUND (4xx)
LINKEDOUT
Service
Bar
Foo
Normal Traffic
TIMEOUTS
Induced Failure
HIGH LATENCY
SERVER ERROR (5xx)
RESOURCE NOT FOUND (4xx)

22. What is Working for LinkedIn
What we are trying to improve
 Record/Replay for microservices
Canary Analysis
Fast release cycles ( 3X3 )
Mobile testing with real devices
Capacity testing
Resiliency testing
What we have nailed
Unit Testing
High Code Coverage​
A/B Testing
Standard CI Pipeline
Trunk Development

23. Challenges for LinkedIn
What has not worked
End-to-End testing 
Staging Environment
Dedicated QA Org 

24. Testing Best Practices
Test at all stages of development lifecycle
Try to find most defects early in the lifecycle
Empower Developers
Build tooling and automation
Deploy Often
Faster defect isolation
Ability to fix forward

25. WE ARE HIRING!
careers.linkedin.com

26. Q & A