1. #SummitNow The Art of the Upgrade Kyle Adams Solutions Engineer at Alfresco @kylefadams

2. #SummitNow Objectives Arm you with the strategy and tactics to perform a successful upgrade of the core Alfresco repository.

3. #SummitNow Strategy and Tactics "Strategy without tactics is the slowest route to victory.” “Tactics without strategy is the noise before defeat.” – Sun Tzu, The Art of War

4. #SummitNow Strategy and Tactics “...tactics are the actual means used to gain an objective, while strategy is the overall campaign plan…” [1] [1] http://en.wikipedia.org/wiki/Tactic_(method)http://en.wikipedia.org/wiki/Tactic_(method

5. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

6. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. 80% of Work Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

7. #SummitNow Prepare "To... not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.” [1] [1] Sun Tzu, The Art of War

8. #SummitNow Prepare – Current State Count of all database tables Size and number of files in content store Size of lucene or Solr indexes Custom Content Models Custom Process definitions Installed modules Customizations

9. #SummitNow Prepare – Upgrade Path http://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/concepts/up grade-path.htmlhttp://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/concepts/up grade-path.html

10. #SummitNow Prepare – Backup/Restore Backup Prod Database Restore into new instance Backup Prod lucene/solr indices Restore into new instance Backup Prod ContentStore Restore into new instance

11. #SummitNow Prepare – Backup/Restore Backup Prod ContentStore What if your ContentStore is too large to restoring into a lower environment for testing?

12. #SummitNow Prepare – Backup/Restore No excuse! We can mock up the ContentStore against the database using small sample files. https://github.com/kylefernandadams/alfre sco-mock-content-generatorhttps://github.com/kylefernandadams/alfre sco-mock-content-generator CAUTION: Only use this for testing purposes.

13. #SummitNow Prepare – Alfresco Instances Install an out-of-the-box Alfresco instance for each upgrade segment Current Version: 3.3.4 Target Version: 4.2.0 Path: 3.3.4 -> 3.3.5 -> 4.2.0 Don’t forget to request license files for each minor version from Alfresco support! Point each instance to common content store, database, and indices. Do not perform an in-place upgrade!

14. #SummitNow Prepare – Alfresco Instances Do not perform an in-place upgrade!

15. #SummitNow Prepare – Patches Know exactly which patches will run during each segment of the upgrade. alfresco/WEB- INF/classes/alfresco/patch/patch- services-context.xml Patch Java docs: http://dev.alfresco.com/resource/docs/java/reposi tory/org/alfresco/repo/admin/patch/impl/package- summary.html http://dev.alfresco.com/resource/docs/java/reposi tory/org/alfresco/repo/admin/patch/impl/package- summary.html

16. #SummitNow Prepare – Patches Still unsure? Perform a dry run of the upgrade with OOTB installation. Set logging to debug for the following packages org.alfresco.repo.admin.patch org.alfresco.repo.domain.patch Review the logging output

17. #SummitNow Prepare – Customizations Only include bare minimum extensions and overrides (Example: content model) Restore a Prod backup into the extensionRoot

18. #SummitNow Prepare – Tools for Monitoring Logs tail, text editor, Splunk JVM YourKit, JProfiler DB Queries Vendor-specific tools (MySQL Workbench, SQL Developer, PgAdmin, SQL Server Mgmt Studio) Quickly toggle log4j logging for java.sql.*

19. #SummitNow Prepare – Tools for Monitoring Hardware & OS (CPU, Memory, Disk, etc) top, nmon, ProcessExplorer

20. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

21. #SummitNow Execute Validate your newly restored instance is working as expected. Example: 3.3.4 Shutdown the instance Startup the instance for the first segment of the upgrade. Example: 3.3.4 -> 3.3.5

22. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

23. #SummitNow Monitor - Logs Based on logging output from your dry run, you should be able to determine the overall progress of your upgrade. What if the logging output appears to be stalled?

24. #SummitNow Monitor - Logs Stalled?

25. #SummitNow Monitor - JVM Leverage a Java profiling tool such as YourKit YourKit will tell you exactly which class and methods are being called. What if it appears that the upgrade has stalled on a given method call?

26. #SummitNow Monitor – JVM Still Stalled???

27. #SummitNow Monitor – SQL Queries Use your DB vendor-specific tools to monitor the queries being executed Don’t have MySQL Workbench, SQL Developer, PgAdmin, or SQL Server Mgmt Studio? Quickly toggle log4j logger for java.sql.* You must add java.sql to custom- log4j.properties

28. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

29. #SummitNow Validate – Smoke Tests Test basic Alfresco functionality. Test one or more of your major use cases. For the latest 4.x versions, leverage WebDrone.

30. #SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate Repeat for each segment

31. #SummitNow Gotchas – Cyclical ACL’s FixAclInheritancePatch could fail with StackOverflowException during 3.4 upgrade. Added cyclical ACL detection in 3.4.13 Jira: MNT-2580

32. #SummitNow Gotchas – Orphaned Nodes Orphaned Nodes and Associations to deleted nodes Reindex could fail with “InvalidNodeRefException: Node does not exist”

33. #SummitNow Gotchas – Orphaned Nodes Find orphaned nodes: SELECT n.* FROM alf_node n, alf_qname q, alf_namespace ns where q.id = n.type_qname_id and ns.id = q.ns_id and not (ns.uri = 'http://www.alfresco.org/model/system/1.0' AND q.local_name = 'deleted') AND NOT EXISTS (SELECT 1 FROM alf_store where root_node_id = n.id) AND NOT EXISTS (SELECT 1 FROM alf_child_assoc where child_node_id = n.id)

34. #SummitNow Gotchas – Orphaned Nodes Find associations to deleted nodes: SELECT a.* FROM alf_child_assoc a, alf_node n, alf_qname q, alf_namespace ns where (n.id = a.child_node_id or n.id = a.parent_node_id) and q.id = n.type_qname_id and ns.id = q.ns_id and ns.uri = 'http://www.alfresco.org/model/system/1.0' AND q.local_name = 'deleted'

35. #SummitNow Gotchas – Orphaned Nodes CAUTION: Do not attempt to fix these by hacking the database! Detection and fix via Java has been added as of 4.1.4 Jira: MNT-8531

36. #SummitNow Gotchas – Nodes without Txn’s May cause schema updates to fail with *ContraintViolationExceptions.

37. #SummitNow Gotchas – Nodes without Txn’s Find nodes without transactions: SELECT * FROM alf_node node WHERE node.transaction_id NOT IN ( SELECT id FROM alf_transaction );

38. #SummitNow Gotchas – Nodes without Txn’s CAUTION: Do not alter the database without the consent or the assistance of Alfresco Support or Alfresco Consulting. Fix: UPDATE alf_node set transaction_id = (SELECT MIN(id) from alf_transaction) WHERE transaction_id NOT IN (SELECT id FROM alf_transaction);

39. #SummitNow Gotchas – Long-running Patches MigrateVersionStorePatch (2.x  3.x) Run asynchronously as a scheduled job # Never: * * * * * ? 2099 # Once every thirty minutes: 0 0/30 * * * ? version.store.migrateVersionStore.cronExpression=* * * * * ? 2099 # Limit number of version histories to migrate per job cycle, where -1 = unlimited. Note: if limit > 0 then need to schedule job to run regularly in order to complete the migration. version.store.migrateVersionStore.limitPerJobCycle=-1 version.store.migrateVersionStore.runAsScheduledJob=false

40. #SummitNow Gotchas – Long-running Patches ContentUrlConverterPatch (...  3.2.x) Run asynchronously as a scheduled job # Never: * * * * * ? 2099 # Once every thirty minutes: 0 0/30 * * * ? system.content.contentUrlConverter.cronExpression=* * * * * ? 2099 system.content.contentUrlConverter.threadCount=2 system.content.contentUrlConverter.batchSize=500 system.content.contentUrlConverter.runAsScheduledJob=false

41. #SummitNow Gotchas – Long-running Patches NodeDeleted.sql (…  4.1.x) Renames alf_node to t_alf_node Recreate the alf_node table without the node_deleted column Insert all rows from t_alf_node into alf_node Drop the temporary t_alf_node table

42. #SummitNow Gotchas – Database Statistics Don’t forget to recompute your database statistics! Without histograms if you’re using Oracle

43. #SummitNow Summary Be meticulous! Have attention to detail! Leave nothing up to mystery! There should be no surprises. “Know thy self, know thy enemy. A thousand battles, a thousand victories” [1] [1] Sun Tzu, The Art of War

44. #SummitNow Summary “The supreme art of war is to subdue the enemy without fighting.” [1] [1] Sun Tzu – The Art of War

45. #SummitNow Contact Information Kyle Adams Solutions Engineer at Alfresco Email: kyle.adams@alfresco.comkyle.adams@alfresco.com Twitter: @kylefadams

46. #SummitNow