#SummitNow The Art of the Upgrade Kyle Adams Solutions Engineer at

#SummitNow Objectives Arm you with the strategy and tactics to perform a successful upgrade of the core Alfresco repository.

#SummitNow Strategy and Tactics "Strategy without tactics is the slowest route to victory.” “Tactics without strategy is the noise before defeat.” – Sun Tzu, The Art of War

#SummitNow Strategy and Tactics “...tactics are the actual means used to gain an objective, while strategy is the overall campaign plan…” [1] [1]

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. 80% of Work Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

#SummitNow Prepare "To... not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.” [1] [1] Sun Tzu, The Art of War

#SummitNow Prepare – Current State Count of all database tables Size and number of files in content store Size of lucene or Solr indexes Custom Content Models Custom Process definitions Installed modules Customizations

#SummitNow Prepare – Upgrade Path grade-path.htmlhttp://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/concepts/up grade-path.html

#SummitNow Prepare – Backup/Restore Backup Prod Database Restore into new instance Backup Prod lucene/solr indices Restore into new instance Backup Prod ContentStore Restore into new instance

#SummitNow Prepare – Backup/Restore Backup Prod ContentStore What if your ContentStore is too large to restoring into a lower environment for testing?

#SummitNow Prepare – Backup/Restore No excuse! We can mock up the ContentStore against the database using small sample files. sco-mock-content-generatorhttps://github.com/kylefernandadams/alfre sco-mock-content-generator CAUTION: Only use this for testing purposes.

#SummitNow Prepare – Alfresco Instances Install an out-of-the-box Alfresco instance for each upgrade segment Current Version: Target Version: Path: > > Don’t forget to request license files for each minor version from Alfresco support! Point each instance to common content store, database, and indices. Do not perform an in-place upgrade!

#SummitNow Prepare – Alfresco Instances Do not perform an in-place upgrade!

#SummitNow Prepare – Patches Know exactly which patches will run during each segment of the upgrade. alfresco/WEB- INF/classes/alfresco/patch/patch- services-context.xml Patch Java docs: tory/org/alfresco/repo/admin/patch/impl/package- summary.html tory/org/alfresco/repo/admin/patch/impl/package- summary.html

#SummitNow Prepare – Patches Still unsure? Perform a dry run of the upgrade with OOTB installation. Set logging to debug for the following packages org.alfresco.repo.admin.patch org.alfresco.repo.domain.patch Review the logging output

#SummitNow Prepare – Customizations Only include bare minimum extensions and overrides (Example: content model) Restore a Prod backup into the extensionRoot

#SummitNow Prepare – Tools for Monitoring Logs tail, text editor, Splunk JVM YourKit, JProfiler DB Queries Vendor-specific tools (MySQL Workbench, SQL Developer, PgAdmin, SQL Server Mgmt Studio) Quickly toggle log4j logging for java.sql.*

#SummitNow Prepare – Tools for Monitoring Hardware & OS (CPU, Memory, Disk, etc) top, nmon, ProcessExplorer

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

#SummitNow Execute Validate your newly restored instance is working as expected. Example: Shutdown the instance Startup the instance for the first segment of the upgrade. Example: > 3.3.5

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

#SummitNow Monitor - Logs Based on logging output from your dry run, you should be able to determine the overall progress of your upgrade. What if the logging output appears to be stalled?

#SummitNow Monitor - Logs Stalled?

#SummitNow Monitor - JVM Leverage a Java profiling tool such as YourKit YourKit will tell you exactly which class and methods are being called. What if it appears that the upgrade has stalled on a given method call?

#SummitNow Monitor – JVM Still Stalled???

#SummitNow Monitor – SQL Queries Use your DB vendor-specific tools to monitor the queries being executed Don’t have MySQL Workbench, SQL Developer, PgAdmin, or SQL Server Mgmt Studio? Quickly toggle log4j logger for java.sql.* You must add java.sql to custom- log4j.properties

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate

#SummitNow Validate – Smoke Tests Test basic Alfresco functionality. Test one or more of your major use cases. For the latest 4.x versions, leverage WebDrone.

#SummitNow Upgrade Strategy Strategy: Accurately predict how my Alfresco instance will behave throughout the upgrade process. Tactic: Prepare Tactic: Execute Tactic: Monitor Tactic: Validate Repeat for each segment

#SummitNow Gotchas – Cyclical ACL’s FixAclInheritancePatch could fail with StackOverflowException during 3.4 upgrade. Added cyclical ACL detection in Jira: MNT-2580

#SummitNow Gotchas – Orphaned Nodes Orphaned Nodes and Associations to deleted nodes Reindex could fail with “InvalidNodeRefException: Node does not exist”

#SummitNow Gotchas – Orphaned Nodes Find orphaned nodes: SELECT n.* FROM alf_node n, alf_qname q, alf_namespace ns where q.id = n.type_qname_id and ns.id = q.ns_id and not (ns.uri = ' AND q.local_name = 'deleted') AND NOT EXISTS (SELECT 1 FROM alf_store where root_node_id = n.id) AND NOT EXISTS (SELECT 1 FROM alf_child_assoc where child_node_id = n.id)

#SummitNow Gotchas – Orphaned Nodes Find associations to deleted nodes: SELECT a.* FROM alf_child_assoc a, alf_node n, alf_qname q, alf_namespace ns where (n.id = a.child_node_id or n.id = a.parent_node_id) and q.id = n.type_qname_id and ns.id = q.ns_id and ns.uri = ' AND q.local_name = 'deleted'

#SummitNow Gotchas – Orphaned Nodes CAUTION: Do not attempt to fix these by hacking the database! Detection and fix via Java has been added as of Jira: MNT-8531

#SummitNow Gotchas – Nodes without Txn’s May cause schema updates to fail with *ContraintViolationExceptions.

#SummitNow Gotchas – Nodes without Txn’s Find nodes without transactions: SELECT * FROM alf_node node WHERE node.transaction_id NOT IN ( SELECT id FROM alf_transaction );

#SummitNow Gotchas – Nodes without Txn’s CAUTION: Do not alter the database without the consent or the assistance of Alfresco Support or Alfresco Consulting. Fix: UPDATE alf_node set transaction_id = (SELECT MIN(id) from alf_transaction) WHERE transaction_id NOT IN (SELECT id FROM alf_transaction);

#SummitNow Gotchas – Long-running Patches MigrateVersionStorePatch (2.x  3.x) Run asynchronously as a scheduled job # Never: * * * * * ? 2099 # Once every thirty minutes: 0 0/30 * * * ? version.store.migrateVersionStore.cronExpression=* * * * * ? 2099 # Limit number of version histories to migrate per job cycle, where -1 = unlimited. Note: if limit > 0 then need to schedule job to run regularly in order to complete the migration. version.store.migrateVersionStore.limitPerJobCycle=-1 version.store.migrateVersionStore.runAsScheduledJob=false

#SummitNow Gotchas – Long-running Patches ContentUrlConverterPatch (...  3.2.x) Run asynchronously as a scheduled job # Never: * * * * * ? 2099 # Once every thirty minutes: 0 0/30 * * * ? system.content.contentUrlConverter.cronExpression=* * * * * ? 2099 system.content.contentUrlConverter.threadCount=2 system.content.contentUrlConverter.batchSize=500 system.content.contentUrlConverter.runAsScheduledJob=false

#SummitNow Gotchas – Long-running Patches NodeDeleted.sql (…  4.1.x) Renames alf_node to t_alf_node Recreate the alf_node table without the node_deleted column Insert all rows from t_alf_node into alf_node Drop the temporary t_alf_node table

#SummitNow Gotchas – Database Statistics Don’t forget to recompute your database statistics! Without histograms if you’re using Oracle

#SummitNow Summary Be meticulous! Have attention to detail! Leave nothing up to mystery! There should be no surprises. “Know thy self, know thy enemy. A thousand battles, a thousand victories” [1] [1] Sun Tzu, The Art of War

#SummitNow Summary “The supreme art of war is to subdue the enemy without fighting.” [1] [1] Sun Tzu – The Art of War

#SummitNow Contact Information Kyle Adams Solutions Engineer at Alfresco

#SummitNow