Using Atlassian Products in an Enterprise Environment Mark McCormack British Telecom Inlife Support Lead Development Services Organisation

Outline Using Atlassian products in an Enterprise Atlassian Product adoption Size of Confluence and Jira in our Enterprise Some Atlassian Use Cases Customisations Application Monitoring Platform What’s Next Product Wishlist Some Lessons Learnt Q & A

About London 2012 Olympics Network is available in 173 countries. Around 3,000 new customer sites every month Carry 80% of all credit card transactions in the UK. More than 90% of the UK’s financial institutions use our secure data networks Rolling out super-fast broadband to 40% of the UK by 2012. FTTC downloads of 40Mb/s and uploads of 2 Mb/s. FTTP downloads of 100 Mb/s London 2012 Olympics

Development Services Organisation We advise on best practices for all tools Inlife team looks after all hosted applications: We are an Atlassian shop Subversion, Jive Forums, Nexus Other in-house tools Authz, Teams, IPPI, Provisioning, Static Data & PALMS SDK Catalogue (will soon become BT Enterprise Repository)

Product History Dec 2003 June 2004 April 2006 February 2007 July 2007 5 people created a team (called code red) but there was no way of coordinating their efforts Bought Confluence and Jira and installed it on a PC and put it in a meeting room June 2004 Usage spread to a few development teams (200 users) Bought new hardware and installed it in a server room (engineering.intra.bt.com) April 2006 Another group installed a separate instance of Jira and Confluence to manage service improvements February 2007 Management Services and Engineering decided joining forces would be a good idea Userbase reached 300 and 2000 respectively http://collaborate.intra.bt.com was born and hosted by VS Group on VMWare. July 2007 A Virtual Administrator crew formed out of 35 volunteers from the userbase to perform end user support and administer the platform Two users in particular did most of the Admin work. Userbase grows to over 9000 people.

Product Maturity August 2007 Sponsors found to pay for new hardware & uplift plan created Team split started to disrupt progress to achieve this and ownership was contested September 2007 Userbase grows to 10,000 October 2007 I/O Capacity of the disks in the VSP environment started to be exceeded. Platform started to have outages with huge traffic increase January 2009: Moved to an enterprise hardware infrastructure, Oracle 10G RAC & 24x7 support Collaborate Wiki was formed, Confluence 2.10 and integrated with Crowd February 2009: Introduced Storm & Jira Issues July 2009: Started to assimilate other wikis (like twiki, rdrwiki, ProfComms)

Confluence Numbers 800,000 pages 102,000 attachments 3000 spaces 25,000 display requests per day 30,000 actions per day (eg login, edit, comments, rss feeds etc)

Numbers 3 flavours of Jira ACF – Architecture Conformance Framework Issue Tracking (100 projects) Story Management Operating Model BT is adding > 20,000 stories every month

Confluence Use Cases Tool Guides on how to get started Tool Access Request Forms We have a teams application we encourage before requesting a Space Early stages of a Provision Application Application Deployment Schedule Collaboration between Dev and Inlife Teams (geographically dispersed) Don’t send emails but rather send Confluence links Trying to have an FAQ before raising an issue for help Application Usage Statistics MyBT has a dashboard interface and makes use of the Confluence remote API and render as html inside a portlet. Most of the Ajax related actions don’t work

Confluence Customisations Implemented the functionality of security classification requirements in the scaffolding plugin. To implement this functionality the edit page / do edit page actions of the confluence needs to be overridden. Scaffolding plugin was already overriding the edit actions of confluence. Confluence doesn't allow two plugins to override a particular action. Extended the edit actions to include classification

More Confluence Customisations https://collaborate.bt.com/staticdata plugin to provide a LOB picker from LDAP Static Data Macro that retrieves platforms BT Space Admin Confluence plugin makes use of our eorg DB in order to get the Line of Business functionality associated with a Space Modified the Excel Plugin as it was hanging Confluence a lot We had one user with a page that had 48 references to the excel macro Every reference was an API call that forced a GC, so we disabled explicit GC Page went from 14 minutes to load to about 4 seconds

Customisations Hierarchy plugin Static Data plugin to strip LOB info out from LDAP Loads of constraints on Story creation and workflow All Stories must have at least one child and at least on Acceptance Criteria Organogram Added Active User Report under Users, Groups and Roles Roles for Storm Loads of different workflows (put issues On Hold whilst with Dev team or Vendor)

More Customisations https://collaborate.bt.com/fisheye plugin to allow the authorisation of SVN repositories (handled by hasPermissionToAccess method which is overrided to authorise against our Authz). Use https://collaborate.bt.com/crowd client libraries for apps like https://collaborate.bt.com/forum We have a Jira Project that helps us keeps track of Subversion Projects ensuring everyone has a Salsa ID. Unified Rolling Calendar Plugin

Application Monitoring ZXTM Nagios alerts via email for slow (or dead) apps. JMX is enabled for most of our java apps Disk space (e.g shell script to alert > 80%) Backup successes or failures uploaded to our wiki Access logs with IP tracking Usage statistics to figure out what our users are doing Perl script we run for snapshots of ugly apps YourKit Profiling from within IntelliJ kSar

Platform Currently 18 HP Blades mostly BL460s with 2 Quad cores and 32GB of RAM for our existing hosted application servers RHEL 5 (soon OEL) Were using Weblogic 10.3 but now Tomcat 6 Java 6.0.18 ZXTM 5.0 Oracle 10g in a Cluster hosted by another team on HP Blade 680s with 2 Quad cores and 64GB of RAM Crowd is our only clustered application (currently) We use Capistrano to help automate our deployments and allows easy rollback

What’s Next? Clustering of Confluence, Jira and SVN Triple our hardware for Disaster Recovery Archiving & stats analysis plugins (customised) Improve provisioning All the Atlassian 2s (Bamboo, Crowd, Fisheye, Crucible) although upgrading to Fisheye 2.0 will be expensive on disk space. Start charting long-term trends with better monitoring Assimilate all wikis (like LimsWiki for one)

Wishlist Confluence Wishlist Tighter integration with Jira Allow Jira Issues to be raised via Confluence and vice versa Allow profiles to have attributes (like team, group or project) Confluence attachment storage API. We want to store attachments away from Confluence and write our own adaptors to do this. Improve the edit permissions cascading security Customisations of actions (eg. Overrides to allow > 1 plugin) Jira Wishlist Allow two columns for attributes Allow picklists to show more values without scrolling Smart forms that can include wiki macros How can we hide Administrators from assignee list? Can Greenhopper scale? How compatible is it with existing 3.13.2 plugins/customisations?

More Wishes Jira … Would like to use our Issues as a source for FAQ before users raise an issue? Ought to be Confluence tightly coupled with Jira Issue labels Timebooking to allow resource Management? Others Fisheye 2 IE6 browser support Make Crucible a viable app for the enterprise (we have some open Atlassian issues for 10k+ users) Can we turn on “remember my login on this computer” checkbox during login?

Lessons Learnt Be wary of any plugin Excel Macro plugin We turned off the usage tracking plugin We trialled the stats analysis plugin but its currently off since we got hammered two lunchtimes in a row shortly after going live WebLogic seems to be a lot slower than tomcat Confluence 3.x is much better than 2.x We had problems with Crowd 1.6 client and currently use Crowd 1.3 client Tune the JVM heap Confluence: we use java –server –Xms:8G –Xmx:8G PermSize=512m MaxPermSize=512m +UseParallelGC +AggressiveHeap LargePageSizeInBytes=4M -DisableExplicitGC Jira: we use –server –Xms4G –Xmx4G PermSize=1G MaxPermSize=1G +UseParallelOldGC +AggressiveOpts Crowd: we use java –server –Xms:4G –Xmx:4G PermSize=512m MaxPermSize=512m Fisheye: not really tuned yet –Xms:4G –Xmx:4G MaxPermSize=512m

Questions mark.j.mccormack@bt.com British Telecom Inlife Support Lead Development Services Organisation