1. Refeng Wu CQ5 WCM System Administrator
Repository Optimization and TAR PM index rebuild in clustered Weblogic servers
Refeng Wu
CQ5 WCM System Administrator

2. CQ5 - Repository Database -> Repository
Huge size; maintenance issue
Keeping the clean repository is one of the main task for CQ5 system administration
Challenges:
Third party application server
Websphere 7, JBoss 5.1, Oracle Weblogic , Tomcat 6
Clustered environments
LDAP integration

3. Challenges Installation and configuration adjustments
Performance Tuning
Very time-consuming Maintenance process - Repository is very big
Slave repository out-of-syn with Master repository
Server maintenance / catastrophic failure, and could not be restarted smoothly
LDAP issue:
Author instance cannot login, system frozen
Publish instance directed to geometrixx login page
Log Errors: seldom with right solutions
author 8G, publish 6G

4. Database Maintenance
Monitoring databases: Monitoring the state of the database ensures -- Availability of the Database; Better Performance.
Backup and recovery: To prevent the potential loss of data, set up a backup and recovery plan -- scheduling regular backups; special backup after large data is imported/modified.
Performance Tuning
Reorganizing and generating database statistics: The DB optimizer requires statistical metadata info about tables & the best way to access data.
Re-creating all indexes: to improve query performance when many existing indexes require reorganization.
Managing database connections: define max connection pools for each of services to optimize the balance between resources and requests.
Purging performance profiling data: to restore free disk space and speed up database utilities.
Dropping temporary tables: drop all data from temporary tables/indexes that hold data due to failed job process.
Old object versions: check for old object versions and delete from database tables to restore storage space, enhance performance, and speed up database utilities.

5. Repository maintenance (Database + file system maintenance)
Proactive solution
Check up the log file daily
Final solution
Repository/instance backup daily
Daily Maintenance for Performance Improvement
Indexing and Optimization
Enable hierarchy check
Validate change-set hierarchy
Setup LDAP log
Schedule consistency check
Test clustered PROD repository in QA
Rebuild Index
Running Garbage Collection
-Dorg.apache.jackrabbit.core.state.validatehierarchy=true

6. Repository Optimization
Tar Files: append-only file, disk usage increases
copies data being used from old tar files into new tar files;
deletes the old tar files that contain only old or redundant data
Node parent-child inconsistency
Search index inconsistency
Optimization process:
Merging index files
IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/version
IndexSet: Merging index files for /app/CQ5/Author/crx-quickstart/repository/tarJournal
IndexSet: Merging index files for /app/CQ5/Author/crx- quickstart/repository/workspaces/crx.default
Rename TarFile:
Renaming /app/CQ5/Author/crx-quickstart/repository/version/optimize.tar to /app/CQ5/Author/crx-quickstart/repository/version/optimizeNow.tar
Optimize TarFile:
/app/CQ5/Author/crx-quickstart/repository/version/data_00331.tar id:331 length: append:
Merging index files:
Merging index files for /app/CQ5/Author/crx-quickstart/repository/version ……

7. Indexing
Lucent Index: node content full-text search, resource-consuming, time-consuming
/repository/repository/index
Lucene Search Index for jcr:system (includes jcr:versionStorage)
/repository/workspace/crx.default/index
Lucene Search Index for crx.default workspace
TAR PM Index: node index
/repository/workspace/crx.default/index*.tar
Tar PM Index for crx.default workspace
 /repository/version/index*.tar
Tar PM Index for version workspace
/repository/tarJournal/index*.tar
Tar PM Index for the cluster journal for Journal PM in 5.4

8. Consistency Check Consistency Check: repository.xml, workspace.xml
Add parameters to all <SearchIndex...> elements
Add parameters to all <PersistenceManager...> elements
Not solving all corrupted nodes
couple nodes, or too many nodes to be fixed
Index Rebuild
Removing index.tar files from three folders in repository
Do it together with Consistency Check during repository startup
8G takes less than 30 minutes.

9. Rebuild Clustered PROD Repository
Moving Clustered repository to in QA or Dev environment
Modify context configure in each server / domain
Weblogic server bootstrap.properties
Modify cluster configure
change slave cluster.property ip address
Test the whole process in QA or Dev with PROD Repository
Be prepared for possible results: master fails, slave fails – copy from master, recovery from zip file
Get a solution for each result
Maintenance window and anticipated hours for use
Moving PROD master/slave backup files to QA: restore the whole application rather than just repository.
0.Indentify PROD master/slave servers, so that QA could start/stop in the same order;
1.copy PROD master/slave backup files to QA;
2.unzip file in QA master and slave respectively: jar -xvf
3.change/copy deployment plan; modify weblogic server bootstrap.properties file for cluster environment. (actually no need to change, as both settings are same)
4.Rename master clusered.txt ; change slave cluster.property ip address. slave 006q ; master 007q ; remove lock file
5.restart master/slave servers in QA.

10. LDAP setup Enable LDAP log info Author instance - ldap group filter:
<logger><appender name="ldap" class="org.apache.log4j.FileAppender"> </logger>
/crx-quickstart/server/etc/Ldap_login.config
Author instance - ldap group filter:
(&(objectclass=group)(cn=wcm*))
Publish instance:
remove group synchronization vs Close User Group
*WARN * ItemSaveOperation: /home/groups/COM/ABB/PROD/CN=Adobe Pro Users,OU=Corporate,OU=Groups,DC=PROD,DC=ABB,DC =COM/rep:lastsynced: failed to restore transient state (ItemSaveOperation.java, line 897)

11. Q&A