Business Resumption From An Application Perspective Presented By: Steven J. Skidmore IT Director Martin Clearwater & Bell LLP Thomas Huson Director, Professional Services Advanced Legal Systems, Inc. Lawnet 2003 Networking & Education Applications Track August 21, :00am

Session Overview The session is based on a worst case scenario Total devastation to computer room facilities The session is based on a worst case scenario Total devastation to computer room facilities Core components of session Core components of session –Decision Process –Build –Rebuild

Disaster Recovery Disaster Recovery – The process of recovering from an event that has negatively affected the operation of an organization Disaster Recovery – The process of recovering from an event that has negatively affected the operation of an organization –Disaster has occurred –Management declares disaster –Activate disaster recovery plan

Key Players The Firm The Firm –Litigation Firm Main Office Located Midtown Manhattan Main Office Located Midtown Manhattan Two Satellite Offices Two Satellite Offices Medical Malpractice Defense Medical Malpractice Defense 80 Attorneys 80 Attorneys –IT Department IT Director IT Director 2 Support Staff Employees 2 Support Staff Employees –Consulting Team Sr. Engineer Sr. Engineer

Case Study Objective To find and implement a disaster recovery solution that will restore the Firms critical business applications within 72 hours. To find and implement a disaster recovery solution that will restore the Firms critical business applications within 72 hours.

The Decision Process Assessing Our Risk Assessing Our Risk –Changed Attitudes –Everything has changed since 9/11. –Could not afford not to Interruption to business can be costly Interruption to business can be costly –Responsibility to clients –Our location puts us at a high risk United Nations, Israeli Consulate, Grand Central Station United Nations, Israeli Consulate, Grand Central Station –Anticipate the worst case scenario

Factors determining a DR solution Factors determining a DR solution –Cost Factor Had a fixed budget to deal with Had a fixed budget to deal with –Comdisco Recovery Center –Work with Consulting team and keep project in house. Utilize satellite office Utilize satellite office The Decision Process

Time Factor Time Factor –Identify critical business processes –How fast did we need to be operational Connection Factor Connection Factor –Who was going to access the recovery system? Identify Tier levels Identify Tier levels –Where are they going to access from? Onsite Onsite Offsite Offsite –How were they access the system? Citrix Citrix The Decision Process

Application Factor Application Factor –Essential Applications (0-72hrs) = Disruption in service exceeding 72 hours would jeopardize the operation of the Firm Elite Elite –Time / Billing –Case Management / Calendar Docket System MSExhange MSExhange – –Calendar –Contacts iManage / WordPerfect iManage / WordPerfect –Document Management –Word Processing The Decision Process

Recommended Applications (72hrs – 1wk) = Disruption in service exceeding 1 week would jeopardize the operation of the Firm Recommended Applications (72hrs – 1wk) = Disruption in service exceeding 1 week would jeopardize the operation of the Firm –Microsoft Office Suite –Summation –Record Access –CD-Rom Applications –Internet Access The Decision Process

Non-Essential Applications ( > 1 week) Non-Essential Applications ( > 1 week) –Nice to have – but would not jeopardize the operation of the Firm if they were missing The Decision Process

Decision Process Result Warm Site Warm Site –Long Island Satellite Office –T1 Connection –Off Site Backup Tape Storage Facility –Employees who live 15mins away

Designing a Recovery Environment Application Functionality in the Live Environment Spans 9 servers in 2 Domains Application Functionality in the Live Environment Spans 9 servers in 2 Domains Compress basic functionality into other hardware at minimal cost Compress basic functionality into other hardware at minimal cost The Mission Critical apps the Firm Identified not conducive to simple tape restore. Specifically: The Mission Critical apps the Firm Identified not conducive to simple tape restore. Specifically: – Exchange – SQL – Elites Informix Challenges:

Building a Recovery Warm Site Communication Redundancy Communication Redundancy Data Replication Scheme Data Replication Scheme Active Directory Configuration Active Directory Configuration Solid Hardware at Recovery Site Solid Hardware at Recovery Site New Software Installations New Software Installations Solution

Building a Recovery Warm Site Communication configuration Communication configuration – T-1 frame between sites – T-1 at recovery site – Redundant link – Recovery site has own scope of valid ips Hardware configuration Hardware configuration – Single high performance server to house Majority of apps – Additional light-weight box for 2 nd domain Solution

Building a Recovery Warm Site Software Configuration Software Configuration – New install of mission critical software from factory media Server/Active Directory (Into existing forest) Server/Active Directory (Into existing forest) Citrix Citrix MS Exchange MS Exchange SQL/iManage SQL/iManage RSASecureID RSASecureID Elite Elite Arcserve Arcserve –Installed, Tested – Services disabled Solution:

Building a Recovery Warm Site Replication Replication – Arcserve copy utility Runs every 4 hours Runs every 4 hours Copies over all files with archive bit flagged Copies over all files with archive bit flagged Resets flag Resets flag Active Directory Configuration Active Directory Configuration – Disaster Recovery OU – Security Group Recovery For Different application pointers, drive mappings etc. For Different application pointers, drive mappings etc. Solution:

Rebuilding the Enterprise Assumptions Assumptions – Catastrophic Event – May not Have Access to the Most Recent Backup Set – Localized Event – We Utilize a Full Backup Strategy Rebuild Process Rebuild Process – Shares – SQL – Elite Informix – MS Exchange

Rebuilding the Enterprise Obtain Most Current Backup Set Obtain Most Current Backup Set – May be 48 Hrs old Begin Restore Process to Bring Recovery Environment Online Begin Restore Process to Bring Recovery Environment Online – Restore Shares Merge Tape Merge Tape Replace newer files only Replace newer files only (Should not actually lay down any data) (Should not actually lay down any data) Apps Apps Data Data Docs Docs Home Home Share Share StartMenus StartMenus

Rebuilding the Enterprise Restore SQL Database for iManage Restore SQL Database for iManage – Merge SQL Backup tape – Restore to...\SQLBACKUP directory – Note Backup (BAK) file – Note Transaction log files Newer than backup file – Restore Database in SQL Enterprise Manager Restore database by device Restore database by device Select Force restore over existing database Select Force restore over existing database Select Leave Database nonoperational but able to restore logs Select Leave Database nonoperational but able to restore logs – Apply Transaction Log Files in SQL Enterprise Manager Must be done individually if more than one Must be done individually if more than one Select Force restore over existing database Select Force restore over existing database When all Log Files Applied Set Database to Online Mode When all Log Files Applied Set Database to Online Mode Modify directory pointers in Database (File server/Index server) Modify directory pointers in Database (File server/Index server) Change application pointers in iManage Change application pointers in iManage Start iManage Services Start iManage Services

Rebuilding the Enterprise

Informix database for Elite Informix database for Elite – Restore from tape via command line in KORN Shell – Does Not Involve ARCServe Onstat – L Onstat – L Onstat - K Onstat - K Ontape – R Ontape – R Onmode – M Onmode – M Onstat – L (To Verify Online Status) Onstat – L (To Verify Online Status)

Rebuilding the Enterprise Microsoft Exchange Microsoft Exchange – Merge Tape – Start Exchange Services – Modify User Accounts via ADSI Launch ADSI Edit Application and CLEAR for Each User: Launch ADSI Edit Application and CLEAR for Each User: – homeMDB – homeMTA – mailNickname – All other msExch properties

Rebuilding the Enterprise

Microsoft Exchange (Cont...) Microsoft Exchange (Cont...) – Create New Mailbox in AD Users and computers – Send test mail to activate mailbox – Brick level restore to new mailboxes – Move user account into disaster recovery OU – Add account to recovery group

Conclusion Disaster Recovery Solution Disaster Recovery Solution –DR solution needs to be tailored to the Firms size, resources and geographic location –Anticipate worst case scenario –Buy as much as you can –Documentation is key –Stay honest –Update