1. ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013

2. ©2013 CloudBees, Inc. All Rights Reserved 2 Logistics Where is …? Network

3. ©2013 CloudBees, Inc. All Rights Reserved 3 Agenda Part 1: Story time Learn from our collective experience Identify high-priority problems What is hurting serious users? Concrete details are good

4. ©2013 CloudBees, Inc. All Rights Reserved 4 Agenda Part 2: Discuss & Design Collectively pick a few topics Split into 2 tracks See if we can start shaping up solutions Please keep shared notes –Details!

5. ©2013 CloudBees, Inc. All Rights Reserved 5 Possible topics includes … CPU/memory/disk/network/… consumption Stability/diagnosability of slaves Access control of builds/UI Organizing jobs and build records Master to master communication Workflow / choreography User interface Stability of Jenkins releases Plugin compatibilities

6. ©2013 CloudBees, Inc. All Rights Reserved 6 Goals No one goes listen-only mode Produce notes to show to the broader community Figure out how to do this better next year

7. ©2013 CloudBees, Inc. All Rights Reserved 7 What We’ve Done & What We Can

8. ©2013 CloudBees, Inc. All Rights Reserved 8 Thread Reduction Per-slave consumption –SSH (down to 1) –Channel (down to 1) Executor thread on demand

9. ©2013 CloudBees, Inc. All Rights Reserved 9 Thread Reduction: TODO NIO Asynchronous job execution –More about this in workflow

10. ©2013 CloudBees, Inc. All Rights Reserved 10 Memory reduction Lazy loading of build records Database plugin

11. ©2013 CloudBees, Inc. All Rights Reserved 11 Design Choices That Constrain Us Compatibility –Adding is doable, changing is often hard Heterogeneousness in data model Thread-driven execution model

12. ©2013 CloudBees, Inc. All Rights Reserved 12 ©2013 CloudBees, Inc. All Rights Reserved CloudBees Scalability Efforts

13. ©2013 CloudBees, Inc. All Rights Reserved 13 Multi-master / Meta Jenkins Master-to-master channel Extensions on top of it –“Cloud” impl to lease slaves –Push security realm –Push update center –…

14. ©2013 CloudBees, Inc. All Rights Reserved 14 Multi-master Where we think we can go with this –Move jobs around –Sharding with UI mashup Where this wouldn’t take us –Loss of inflight builds –True horizontal scaling

15. ©2013 CloudBees, Inc. All Rights Reserved 15 Workflow Brand new job type Kill multiple birds in one stone Inspired by buildflow & jenkow

16. ©2013 CloudBees, Inc. All Rights Reserved 16 Ingredients No single thread to execute the whole thing –Analogous to NIO –Check pointing BPMN-like workflow execution model –Surface syntax independent Groovy DSL –With continuation-passing style execution model Post-execution visualization –But no pre-execution visualization

17. ©2013 CloudBees, Inc. All Rights Reserved 17 Ingredients Unified properties, build variables, and environments Open up direct access to file store Hands-free process forking Interop with existing job types