Refeng Wu CQ5 WCM System Administrator Repository Optimization and TAR PM index rebuild in clustered Weblogic servers Refeng Wu CQ5 WCM System Administrator

CQ5 - Repository Database -> Repository Huge size; maintenance issue Keeping the clean repository is one of the main task for CQ5 system administration Challenges: Third party application server Websphere 7, JBoss 5.1, Oracle Weblogic 10.3.5, Tomcat 6 Clustered environments LDAP integration

Challenges Installation and configuration adjustments Performance Tuning Very time-consuming Maintenance process - Repository is very big Slave repository out-of-syn with Master repository Server maintenance / catastrophic failure, and could not be restarted smoothly LDAP issue: Author instance cannot login, system frozen Publish instance directed to geometrixx login page Log Errors: seldom with right solutions author 8G, publish 6G

Database Maintenance Monitoring databases: Monitoring the state of the database ensures -- Availability of the Database; Better Performance. Backup and recovery: To prevent the potential loss of data, set up a backup and recovery plan -- scheduling regular backups; special backup after large data is imported/modified. Performance Tuning Reorganizing and generating database statistics: The DB optimizer requires statistical metadata info about tables & the best way to access data. Re-creating all indexes: to improve query performance when many existing indexes require reorganization. Managing database connections: define max connection pools for each of services to optimize the balance between resources and requests. Purging performance profiling data: to restore free disk space and speed up database utilities. Dropping temporary tables: drop all data from temporary tables/indexes that hold data due to failed job process. Old object versions: check for old object versions and delete from database tables to restore storage space, enhance performance, and speed up database utilities.

Repository maintenance (Database + file system maintenance) Proactive solution Check up the log file daily Final solution Repository/instance backup daily Daily Maintenance for Performance Improvement Indexing and Optimization Enable hierarchy check Validate change-set hierarchy Setup LDAP log Schedule consistency check Test clustered PROD repository in QA Rebuild Index Running Garbage Collection -Dorg.apache.jackrabbit.core.state.validatehierarchy=true

Repository Optimization Tar Files: append-only file, disk usage increases copies data being used from old tar files into new tar files; deletes the old tar files that contain only old or redundant data Node parent-child inconsistency Search index inconsistency Optimization process: Merging index files IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/version IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/tarJournal IndexSet: Merging index files for /app/CQ5/Author/crx- quickstart/repository/workspaces/crx.default Rename TarFile: Renaming /app/CQ5/Author/crx-quickstart/repository/version/optimize.tar to /app/CQ5/Author/crx-quickstart/repository/version/optimizeNow.tar Optimize TarFile: /app/CQ5/Author/crx-quickstart/repository/version/data_00331.tar id:331 length:144674816 append:144673792 Merging index files: Merging index files for /app/CQ5/Author/crx-quickstart/repository/version ……

Indexing Lucent Index: node content full-text search, resource-consuming, time-consuming /repository/repository/index Lucene Search Index for jcr:system (includes jcr:versionStorage) /repository/workspace/crx.default/index Lucene Search Index for crx.default workspace TAR PM Index: node index /repository/workspace/crx.default/index*.tar Tar PM Index for crx.default workspace  /repository/version/index*.tar Tar PM Index for version workspace /repository/tarJournal/index*.tar Tar PM Index for the cluster journal for Journal PM in 5.4 http://dev.day.com/docs/en/crx/current/administering/persistence_managers.html

Consistency Check Consistency Check: repository.xml, workspace.xml Add parameters to all <SearchIndex...> elements Add parameters to all <PersistenceManager...> elements Not solving all corrupted nodes couple nodes, or too many nodes to be fixed Index Rebuild Removing index.tar files from three folders in repository Do it together with Consistency Check during repository startup 8G takes less than 30 minutes.

Rebuild Clustered PROD Repository Moving Clustered repository to in QA or Dev environment Modify context configure in each server / domain Weblogic server bootstrap.properties Modify cluster configure change slave cluster.property ip address Test the whole process in QA or Dev with PROD Repository Be prepared for possible results: master fails, slave fails – copy from master, recovery from zip file Get a solution for each result Maintenance window and anticipated hours for use Moving PROD master/slave backup files to QA: restore the whole application rather than just repository. 0.Indentify PROD master/slave servers, so that QA could start/stop in the same order; 1.copy PROD master/slave backup files to QA; 2.unzip file in QA master and slave respectively: jar -xvf 3.change/copy deployment plan; modify weblogic server bootstrap.properties file for cluster environment. (actually no need to change, as both settings are same) 4.Rename master clusered.txt.20130301; change slave cluster.property ip address. slave 006q 172.25.108.15; master 007q 172.25.108.16; remove lock file 5.restart master/slave servers in QA.

LDAP setup Enable LDAP log info Author instance - ldap group filter: <logger><appender name="ldap" class="org.apache.log4j.FileAppender"> </logger> /crx-quickstart/server/etc/Ldap_login.config Author instance - ldap group filter: (&(objectclass=group)(cn=wcm*)) Publish instance: remove group synchronization vs Close User Group *WARN * ItemSaveOperation: /home/groups/COM/ABB/PROD/CN=Adobe Pro Users,OU=Corporate,OU=Groups,DC=PROD,DC=ABB,DC =COM/rep:lastsynced: failed to restore transient state (ItemSaveOperation.java, line 897)

Q&A