1. Using Atlassian Products in an Enterprise Environment
Mark McCormack British Telecom Inlife Support Lead Development Services Organisation

2. Outline Using Atlassian products in an Enterprise
Atlassian Product adoption
Size of Confluence and Jira in our Enterprise
Some Atlassian Use Cases
Customisations
Application Monitoring
Platform
What’s Next
Product Wishlist
Some Lessons Learnt
Q & A

3. About London 2012 Olympics Network is available in 173 countries.
Around 3,000 new customer sites every month
Carry 80% of all credit card transactions in the UK.
More than 90% of the UK’s financial institutions use our secure data networks
Rolling out super-fast broadband to 40% of the UK by 2012.
FTTC downloads of 40Mb/s and uploads of 2 Mb/s.
FTTP downloads of 100 Mb/s
London 2012 Olympics

4. Development Services Organisation
We advise on best practices for all tools
Inlife team looks after all hosted applications:
We are an Atlassian shop
Subversion, Jive Forums, Nexus
Other in-house tools
Authz, Teams, IPPI, Provisioning, Static Data & PALMS
SDK Catalogue (will soon become BT Enterprise Repository)

5. Product History Dec 2003 June 2004 April 2006 February 2007 July 2007
5 people created a team (called code red) but there was no way of coordinating their efforts
Bought Confluence and Jira and installed it on a PC and put it in a meeting room
June 2004
Usage spread to a few development teams (200 users)
Bought new hardware and installed it in a server room (engineering.intra.bt.com)
April 2006
Another group installed a separate instance of Jira and Confluence to manage service improvements
February 2007
Management Services and Engineering decided joining forces would be a good idea
Userbase reached 300 and 2000 respectively
was born and hosted by VS Group on VMWare.
July 2007
A Virtual Administrator crew formed out of 35 volunteers from the userbase to perform end user support and administer the platform
Two users in particular did most of the Admin work. Userbase grows to over 9000 people.

6. Product Maturity August 2007
Sponsors found to pay for new hardware & uplift plan created
Team split started to disrupt progress to achieve this and ownership was contested
September 2007
Userbase grows to 10,000
October 2007
I/O Capacity of the disks in the VSP environment started to be exceeded.
Platform started to have outages with huge traffic increase
January 2009:
Moved to an enterprise hardware infrastructure, Oracle 10G RAC & 24x7 support
Collaborate Wiki was formed, Confluence 2.10 and integrated with Crowd
February 2009:
Introduced Storm & Jira Issues
July 2009:
Started to assimilate other wikis (like twiki, rdrwiki, ProfComms)

7. Confluence Numbers 800,000 pages 102,000 attachments 3000 spaces
25,000 display requests per day
30,000 actions per day
(eg login, edit, comments, rss feeds etc)

8. Numbers 3 flavours of Jira ACF – Architecture Conformance Framework
Issue Tracking (100 projects)
Story Management Operating Model
BT is adding > 20,000 stories every month

9. Confluence Use Cases Tool Guides on how to get started
Tool Access Request Forms
We have a teams application we encourage before requesting a Space
Early stages of a Provision Application
Application Deployment Schedule
Collaboration between Dev and Inlife Teams (geographically dispersed)
Don’t send s but rather send Confluence links
Trying to have an FAQ before raising an issue for help
Application Usage Statistics
MyBT has a dashboard interface and makes use of the Confluence remote API and render as html inside a portlet.
Most of the Ajax related actions don’t work

10. Confluence Customisations
Implemented the functionality of security classification requirements in the scaffolding plugin.
To implement this functionality the edit page / do edit page actions of the confluence needs to be overridden. Scaffolding plugin was already overriding the edit actions of confluence.
Confluence doesn't allow two plugins to override a particular action.
Extended the edit actions to include classification

11. More Confluence Customisations
plugin to provide a LOB picker from LDAP
Static Data Macro that retrieves platforms
BT Space Admin Confluence plugin makes use of our eorg DB in order to get the Line of Business functionality associated with a Space
Modified the Excel Plugin as it was hanging Confluence a lot
We had one user with a page that had 48 references to the excel macro
Every reference was an API call that forced a GC, so we disabled explicit GC
Page went from 14 minutes to load to about 4 seconds

12. Customisations Hierarchy plugin
Static Data plugin to strip LOB info out from LDAP
Loads of constraints on Story creation and workflow
All Stories must have at least one child and at least on Acceptance Criteria
Organogram
Added Active User Report under Users, Groups and Roles
Roles for Storm
Loads of different workflows (put issues On Hold whilst with Dev team or Vendor)

13. More Customisations
plugin to allow the authorisation of SVN repositories (handled by hasPermissionToAccess method which is overrided to authorise against our Authz).
Use client libraries for apps like
We have a Jira Project that helps us keeps track of Subversion Projects ensuring everyone has a Salsa ID.
Unified Rolling Calendar Plugin

14. Application Monitoring
ZXTM
Nagios alerts via for slow (or dead) apps.
JMX is enabled for most of our java apps
Disk space (e.g shell script to alert > 80%)
Backup successes or failures uploaded to our wiki
Access logs with IP tracking
Usage statistics to figure out what our users are doing
Perl script we run for snapshots of ugly apps
YourKit Profiling from within IntelliJ
kSar

15. Platform
Currently 18 HP Blades mostly BL460s with 2 Quad cores and 32GB of RAM for our existing hosted application servers
RHEL 5 (soon OEL)
Were using Weblogic 10.3 but now Tomcat 6
Java
ZXTM 5.0
Oracle 10g in a Cluster hosted by another team on HP Blade 680s with 2 Quad cores and 64GB of RAM
Crowd is our only clustered application (currently)
We use Capistrano to help automate our deployments and allows easy rollback

16. What’s Next? Clustering of Confluence, Jira and SVN
Triple our hardware for Disaster Recovery
Archiving & stats analysis plugins (customised)
Improve provisioning
All the Atlassian 2s (Bamboo, Crowd, Fisheye, Crucible) although upgrading to Fisheye 2.0 will be expensive on disk space.
Start charting long-term trends with better monitoring
Assimilate all wikis (like LimsWiki for one)

17. Wishlist Confluence Wishlist Tighter integration with Jira
Allow Jira Issues to be raised via Confluence and vice versa
Allow profiles to have attributes (like team, group or project)
Confluence attachment storage API. We want to store attachments away from Confluence and write our own adaptors to do this.
Improve the edit permissions cascading security
Customisations of actions (eg. Overrides to allow > 1 plugin)
Jira Wishlist
Allow two columns for attributes
Allow picklists to show more values without scrolling
Smart forms that can include wiki macros
How can we hide Administrators from assignee list?
Can Greenhopper scale? How compatible is it with existing plugins/customisations?

18. More Wishes
Jira …
Would like to use our Issues as a source for FAQ before users raise an issue?
Ought to be Confluence tightly coupled with Jira Issue labels
Timebooking to allow resource Management?
Others
Fisheye 2 IE6 browser support
Make Crucible a viable app for the enterprise (we have some open Atlassian issues for 10k+ users)
Can we turn on “remember my login on this computer” checkbox during login?

19. Lessons Learnt Be wary of any plugin Excel Macro plugin
We turned off the usage tracking plugin
We trialled the stats analysis plugin but its currently off since we got hammered two lunchtimes in a row shortly after going live
WebLogic seems to be a lot slower than tomcat
Confluence 3.x is much better than 2.x
We had problems with Crowd 1.6 client and currently use Crowd 1.3 client
Tune the JVM heap
Confluence: we use java –server –Xms:8G –Xmx:8G PermSize=512m MaxPermSize=512m +UseParallelGC +AggressiveHeap LargePageSizeInBytes=4M -DisableExplicitGC
Jira: we use –server –Xms4G –Xmx4G PermSize=1G MaxPermSize=1G +UseParallelOldGC +AggressiveOpts
Crowd: we use java –server –Xms:4G –Xmx:4G PermSize=512m MaxPermSize=512m
Fisheye: not really tuned yet –Xms:4G –Xmx:4G MaxPermSize=512m

20. Questions mark.j.mccormack@bt.com
British Telecom Inlife Support Lead Development Services Organisation