1. Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE

2. KUALI RICE @ INDIANA UNIVERSITY Integrated with 26 enterprise applications Hosting 50 eDocLite applications Over 400 different KEW Document Types 24 x 7 x 365 availability Approximately 99.8% uptime in the last year (including maintenance)

3. EXPANDING IMPLEMENTATION

4. USAGE

5. ARCHITECTURE AND INFRASTRUCTURE Red Hat Enterprise Linux on VMware Load balancer with four clustered Kuali Rice standalone server instances Java 6 Tomcat servlet container Shared file system mount for attachments Oracle 11g database

6. IU KUALI RICE ARCHITECTURE

7. MAINTAINED BY A team of 3 FTE, who do… Production Support System Maintenance Upgrades Integration Projects Application and eDocLite Development

8. PRODUCTION SUPPORT Areas of concern: Database resources Service bus Ingestion Clustering/Load Balancing Tools we use

9. DATABASE RESOURCES Recently updated to Oracle 11g Clients primarily use embedded Concerns: Connection pool limit Database blocking Client proxy accounts Export/import job for refreshing test environments

10. MANAGING THE SERVICE BUS Bad service endpoints Clean shutdown of client applications Monitoring/maintaining the message queue

11. CLUSTERING/LOAD BALANCING Horizontally scalable Session affinity Session replication required for full failover Unique endpoints per machine

12. TOOLS - DYNATRACE Performance monitoring Error reporting Drill down into requests Take memory/thread dumps

13. DYNATRACE EXAMPLE

14. TOOLS - DATABASE Reporting: Stuck documents Database blocking Cleanup User options table Message queue

15. SYSTEM MAINTENANCE Rice customization/patches Environment configuration Load Testing Functional testing Data migration

16. RICE CUSTOMIZATION/PATCHES Two options: Override Spring bean (ex: IU Group Service) Override a particular file Process Copy source Make necessary changes Build the standard Rice source Overlay customized files Deploy or distribute changes

17. ENVIRONMENT CONFIGURATION Java environment managed by a J2EE group Build process deploys: Application code Settings/security files Most Java applications share the same build process http.url configuration parameter

18. LOAD TESTING Using Apache JMeter Perform standalone and coordinated load tests Load testing transactions include Document search Action list Taking mass actions Viewing route logs Creation and routing of documents

19. LOAD TESTING EXAMPLE OUTPUT

20. FUNCTIONAL TESTING Uses Selenium Runs nightly through Bamboo Functional test suite includes: Most eDocLite applications Maintenance documents (group, rule, etc) Document search Action List/Preferences

21. DATA MIGRATION Database job to move production data into testing environments Use the “export” feature DevTest Pre- production Production

22. UPGRADING RICE Process: 1.Bring in new Kuali Rice foundation jars 2.Make changes to the IU version of Rice which includes: Updating customizations Merging "patched" code Produce IU Rice client jars 3.Perform DB updates to test environment 4.Bring up standalone testing environments 5.Work with clients to bring in IU Rice client jars and update code 6.Testing before making the change to production

23. INTEGRATION Client application configuration Keystore Administration

24. CONFIGURING CLIENT APPLICATIONS Spring files (ex KIM overrides) http.url configuration parameter Document handler URL Namespace (if appropriate) Roles/permissions Security file deployment

25. KEYSTORE ADMINISTRATION Keystore needed for KSB security Keytool commands genkey selfcert export import list

26. KEYSTORE ADMINISTRATION Items to consider in a production setting: Keep certificate expiration dates in mind Testing and prod keystores should not be the same New applications must: Create a certificate Include a copy of the KR certificate in their keystore Give their certificate to be included in the KR keystore

27. EDOCLITE DEVELOPMENT Process: 1.Request comes in for a new eDocLite 2.Meet to review requirements 3.Develop initial eDocLite 4.Test and make changes until ready for prod 5.Ingest eDocLite into production

28. IU EDOCLITE EXAMPLES

30. EDOCLITE MANAGEMENT Currently have 50 eDocLite apps in production Reasons for changes to existing eDocLites Annual changes due to nature of form Addition of a new field Change to drop down contents Change to field validation

31. QUESTIONS?