Active Directory Disaster Recovery

Active Directory Disaster Recovery
Premier/Alliance Customer Workshop Johan Ohlén

Intruduction Name Company Title/Function Job Responsibility
AD Disaster Experience Expectations of this Workshop

Agenda Deploying a Backup Strategy
Windows 2003 Backup and Restore Utility ERD(ASR) and Recovery Console AD Database Architecture Active Directory – Best practises Backup Active Directory – Type of Disaster Recovering Active Directory Scenarios Recovering Sysvol and Group Policies Typical PSS issues Forest Recovery

Resources Active Directory Product Operation Guide
Windows Server 2003 Active Directory Diagnostics, Troubleshooting, and Recovery Chapter 10 - Disaster Recovery for Branch Office Environments (AD Planning and Deployment Guides)

Developing a Strategy For each OS and application you introduce, you should answer the following questions: What are the possible failure scenarios? Plan for the Worst scenario: HW, Power, Software failures. Deletion of objects and Files. What is critical data? How often should backup be performed? How can we ensure that the backups are useable? Assume failure and consequences A good documented plan ensures that you quickly recover your data if it’s lost.

Developing a Strategy Guidelines for an effective strategy
Develop backup and restore strategies Appropriate resources and personnel Test your Backup’s Assign backup responsibilities Back up entire volume (Disk Failures) Keep three copies of the backup media Keep one copy “offsite” Perform trial restoration periodically Verify that your files were properly backed up. Secure storage of device and Backup media Prevent an un-authorised restore to your server.

Deploying a Strategy New Enhanced AD,FRS and Sysvol Features
These features require the use of backup products that are aware of the new capabilities built into Windows 2000 and 2003. Running third-party backup products designed for Windows NT 4.0 could cause loss of data.

Deploying a Strategy Who can perform backups and restores?
Backups: Domain Administrators, and Backup Operators Restores: Domain Administrators Custom Group: “Back up files and directories” right assigned to a security principal

Deploying a Strategy Collect information before the disaster
Disk configuration Computer name IP addresses Video mode settings Domain information Local Admin password

Backup and Restore Utility
Ntbackup -New graphical utility Offers three wizards: Backup Restore Emergency Repair Disk – Automated System Recovery Backup Media Tape drive, a logical drive, a removable disk

Ntbackup What can I do Backup “System state” and Windows system files while DC is online Backup and Restore to hard disk or any other disk that the system can access Schedule regular back ups Create an Emergency Repair disk (ERD)

Backup and Restore tools
NTBACKUP NTDSUTIL LDP ADSIEDIT REPADMIN , REPLMON EVENT VIEWER

Ntbackup Limitations Support only ”Normal” backup of AD (Not incremental) You cannot back up AD by iteslf (Entire System state) ”System State” cannot be backed up from remote PC. ”System state” Restore can only be done when AD is offline Can ”only” be restored to the same DC The Backup tool does not encrypt the unencrypted backup contents during the backup process.

The Age of your Backup It is not possible to restore a backup image into a replicated enterprise that is older than the tombstone lifetime value for the enterprise. The tombstone lifetime value represents the number of days that the deleted object (or tombstone) must be retained before it can be permanently removed from the directory

Backup Limitations Backup life = tombstonelifetime value
Default = 60 days old Password Machine account change interval = 30 days Password history = 2 maximum Backup useful life = 60 days or 2 default password changes Group Policy - Domain member: Maximum machine account password age 30 days Group Policy - Domain member: Disable machine account password changes Why? Old backups can re-introduce tombstoned objects Schema Rollback Not supported in W2K or W2K3 Defaults Did passwords increment more since backup?

Creating an ERD or ASR Setup places backup registry files in %systemroot%\repair folder Backup of the registry files are stored in %systemroot%\repair\regback To create an ERD: 1. Click Start, point to Programs, point to Accessories, point to System Tools, and then click Backup. 2. Click Emergency Repair Disk on the Welcome screen, or click "Create an Emergency Repair Disk" on the Tools menu. 3. Insert a blank, formatted 1.44-MB floppy disk in the drive, and then click OK. You can also back up the registry information by clicking to select the "Also back up the registry..." check box. The ERD contains only the following files: Autoexec.nt Config.nt Setup.log These files are not sufficient by themselves to repair a problem with Windows 2000. For additional information about repairing Windows 2000, click the article numbers below to view the articles in the Microsoft Knowledge Base: Q Description of the Windows 2000 Recovery Console Q Differences Between Manual and Fast Repair in Windows 2000

Using an ERD Q238359 Two repair options in Windows 2000
Manual Repair: R= repair + M=manual [ ] Inspect the startup environment [ ] Verify Windows 2000 system files and replace missing or damaged files [ ] Inspect and repair the boot sector Fast Repair - Performs all repairs options Uses files in winnt\repair to replace corrupted or missing files in winnt\system32\config ERD contain: Autoexec.nt Config.nt Setup.log ASR - Automated System Recovery Q W2K3 Safe mode, Recovery Console, and an Emergency Repair disk. Windows includes two repair choices: Manual Repair or Fast Repair. To see these choices, boot from the Windows installation media, press R to repair, and then press R to use the Emergency Repair process. When you do this, you see the following options: - Manual Repair: To choose from a list of repair options, press M. - Fast Repair: To perform all repair options, press F. The two repair choices cause the Repair process to perform different tasks. MORE INFORMATION ================ IMPORTANT: Please do not perform a manual or fast repair on a domain controller without specific knowledge of how to back up the Active directory database. If you do these options on a Windows 2000 Server domain controller you run the risk of overwriting the Active directory database at \WINNT\NTDS\ntds.dit. The Ntds.dit file contains your Active Directory,including user accounts. Manual Repair The Manual Repair option provides the following choices: [X] Inspect startup environment [X] Verify Windows system files [X] Inspect Boot Sector Continue <perform selected tasks> Inspect Startup Environment: This option checks the ARC path in the boot.ini file for a path to the Windows boot partition and %SystemRoot% folder. It does this by using the Setup.log file on the Emergency Repair disk by reading the following values: [Paths] TargetDirectory = "\WINNT" TargetDevice = "\Device\Harddisk1\Partition1" SystemPartitionDirectory = "\" SystemPartition = "\Device\Harddisk1\Partition1" If the Boot.ini file is missing, a new one is created with a valid ARC path. If the Boot.ini file is present, the ARC path is checked and updated if needed. Verify Windows System Files: This selection verifies that each file in the Windows system/boot partition is good and matches the files that were originally installed. This includes the Ntldr, Ntdetect.com, Arcsetup.exe, and Arcldr.exe files that are used for booting various computers. The optional Ntbootdd.sys file is never checked. Repair performs this check by using the Setup.log file to compare cyclical redundancy check (CRC) values for each file. If files are missing or corrupted, you are prompted to replace or skip the file. If you choose to replace the file, you need the Windows installation CD-ROM or an OEM driver disk that contains the correct file(s). Inspect Boot Sector: This option repairs the active system partition boot sector and reinstalls the boot loader functionality. If the partition uses the FAT or FAT32 file system and contains a non-Windows boot sector, this repair option also creates a new Bootsect.dos file to be used to dual-boot MS-DOS, Microsoft Windows 95, or Microsoft Windows 98 if these operating systems were previously available to be booted. If you also select the Inspect Startup Environment option and a new Bootsect.dos file is created, Repair adds the following entry to the Boot.ini file: C:\ = "Microsoft Windows" Note that the Manual Repair option does not give you a choice to repair the Windows registry files. Fast Repair The Fast Repair option performs all the repairs as the Manual Repair option, but you are not prompted for choices. Additionally the Fast Repair option tries to load each Windows registry file (SAM, SECURITY, SYSTEM, and SOFTWARE). If a registry file is damaged or cannot be loaded, Repair copies the missing or corrupted registry file from the <SystemRoot>\Repair folder to the <SystemRoot>\System32\Config folder. Because the Fast Repair option can replace registry files with those from the <SystemRoot>\Repair folder, it may revert parts of your operating system configuration back to the time when Windows was first installed. If this occurs, you need to restore your last "system state" backup or manually copy a more recent version of the registry files from the <SystemRoot>\Repair\Regback folder to the <SystemRoot>\System32\Config folder by using Recovery Console. The files that are located in the Regback folder are from the last time you created an Emergency Repair Disk and choose the option to also back up the registry files to the repair folder. General Information Both the Manual Repair and Fast Repair options start by performing a system/boot partition file system check. If file system problems are detected and corrected during this portion of the Repair process, you may need to restart your computer and start another Repair process before the actual repair operations take place. Neither of the repair options replaces the <SystemRoot>\System32\Config.nt or Autoexec.nt files. Although these files are located on the Emergency Repair Disk, they are not checked or replaced during any Repair operations. For computers without a local CD-ROM drive attached (for example, if Windows was installed by using Remote Installation Service, or RIS), it is possible to repair system files by using one of the methods described in the following Microsoft Knowledge Base articles: Q Replacing System Files Using a Modified Emergency Repair Disk Q Description of the Windows 2000 Recovery Console

Recovery Console Allows administration of files on NTFS drives without completely loading OS Requires local Administrator password Can be run from the cd-rom or installed locally Winnt32 /cmdcons Q Description of the Windows 2000 Recovery Console and commands Q229716 When you use the Windows Recovery Console, you can obtain limited access to NTFS, FAT, and FAT32 volumes without starting the Windows graphical interface. In the Windows Recovery Console you can: - Use, copy, rename or replace operating system files and folders. - Enable or disable services or devices from starting when you next start your computer. - Repair the file system boot sector or the Master Boot Record (MBR). - Create and format partitions on drives. Note that only an administrator can obtain access to the Windows Recovery Console so that unauthorized users cannot use any NTFS volume.

Active Directory Database
Database Files includes: Ntds.dit. The AD database Edb.chk. The checkpoint file. recovery Edbxxx.log. The transaction logs; circular logging only Edb.log The current log file (10 MB) Res1.log & Res2.log. Reserved logs. shutdown Located in %systemroot%\ntds by default EDB.LOG is the current log file. All changes are written to the EDB.LOG file. When EDB.LOG file is full of transactions, it’s renamed to EDB00001.LOG

Circular Logging for AD
Detailed info in Q (LSASS.EXE) Datatbase transaction written to EDB.LOG Shortly thereafter writes the transaction to Memory. When System has time or a system shutdown, transaction are written to NTDS.DIT When EDB.LOG is full, it rename and creates a new file EDB0001.LOG. Circular logging purges/removes the oldest file when the transaction have been comitted to the Database EDB.CHK is a pointer to the last transaction whithin the log have been committed to the Database. If the checkpoint file is missing for any reason, every transaction within the log file is replayed.

Active Directory Database
Edb.log, edbnnnnn.log Changes written to logs Transaction logs Active Directory Database Active Directory is a transacted database system that uses log files to support rollback semantics to ensure that transactions are committed to the database. The files associated with Active Directory are: Ntds.dit – the database. Edbxxxxx.log – transaction logs. Edb.chk – checkpoint file. Res1.log & Res2.log – reserved log files. Ntds.dit grows as the database fills up. However, the logs are of fixed size (10 MB). Any change made to the database is also appended to the current log file, and its disk image is always kept up to date. Edb.log is the current log file. When a change is made to the database, it is written to the Edb.log file. When the Edb.log file is full of transactions, it is renamed to Edbxxxxx.log. (It starts at and continues to increment using hexadecimal notation.) Since Active Directory uses circular logging, old log files are constantly deleted, once they have been written to the database. At any point in time, you will find the edb.log file, and maybe one or more Edbxxxxx.log files. Res1.log and Res2.log are "placeholders" — designed to reserve (in this case) the last 20 MB of disk space on this drive. This is designed to give the log files sufficient room for a graceful shutdown if all other disk space is consumed. The Edb.chk file, stores the database checkpoint, which identifies the point where the database engine needs to replay the logs, generally at the time of recovery or initialization. For performance reasons, the log files should be located on a different disk than the database to reduce disk contention. At the time of taking a backup, a new log file may be created. This log file would be deleted (like regular old log files) due to circular logging, as stated above. Q Ntbackup.exe Does Not Truncate Active Directory Logs – PreSP2 there is an issue where the log files are not truncated when frequent backup of the AD are preformed. Dit file Checkpoint tracks log position of latest write to database file Memory Disk Edb.chk

Active Directory Backup
System State Backup System Startup Files – System files to boot System registry Class registration database of COM+ Sysvol Active Directory database files Active Directory-integrated DNS Certificate Services database (if installed) Cluster Service (if installed) System State Active Directory is backed up as part of System State, a collection of system components that depend on each other. These components must be backed up (and restored) together. Components that make up the System State on a domain controller include: System Start-up Files (boot files). These are the files required for Windows 2000 to boot. They are automatically backed up as part of the System State. System registry. The contents of the registry are automatically backed up when you back up System State data. In addition, a copy of your registry files are saved in the folder %SystemRoot%\Repair\Regback allowing you to restore the registry without doing a complete restore of the System State. Class registration database of COM+. The Component Object Model (COM) is a binary standard for writing component software in a distributed systems environment. The Component Services Class Registration Database is backed up and restored with the System State data. SYSVOL. The system volume provides a default Active Directory location for files that must be shared for common access throughout a domain. The SYSVOL folder on a domain controller contains the following: Net Logon shares. (These usually host logon scripts and policy objects for non-Windows 2000–based network clients.) File system junctions. User logon scripts for Windows 2000 Professional–based clients and clients that are running Windows 95, Windows 98, or Windows NT 4.0. Windows 2000 Group Policy. File replication service (FRS) staging directories and files that are required to be available and synchronized between domain controllers. Active Directory. This includes: Ntds.dit. The Active Directory database. Edb.chk. The checkpoint file. Edb*.log. The transaction logs; each 10 MB in size. Res1.log and Res2.log. Reserved transaction logs. Note: If you have an Active Directory-integrated DNS, the zone data will be backed up as part of the Active Directory database. If you do not have an Active Directory-integrated DNS, the zone files will have to be backed up explicitly. However if you backup the system disk along with the System State, this data will be backed up as part of the system disk.

Active Directory Backup
What is a good backup? Content At least System State and system disk Age Never older than tombstone age. Recomendation: Minimum two backups or more within tombstone lifetime. You cannot use a backup of one DC to restore another DC Only type of backup supported by AD is normal backup. What is a Good Backup? To ensure a successful restore from backup, it is important to know what defines a “good backup.” For Active Directory, two things must be considered: Contents Age The first important aspect of a backup is its contents. A good backup will include at least the System State, the contents of the system disk, and the SYSVOL folder (if not located on the system disk). As described above, the System State includes many key files and settings to restore a domain controller. Backing up the system disk and SYSVOL folder structure will ensure that all the required system files and folders are in place to initiate a successful restoration. Note: Best performance practice states that the Active Directory’s log and database files should be on separate spindles (disks). If you have configured your DCs in this manner you will have Active Directory components spread out on multiple drives, such as D:\Winnt\NTDS for your logs and E:\Winnt\NTDS for your database. Because the Active Directory log files and database are backed up as part of System State, you will still only have to backup the system disk and System State in order to ensure a good backup, even under this distributed installation. If the backup is older than the tombstone age set in Active Directory, then it is not considered to be a good backup. When an object is deleted in Windows 2000, the DC from which the object was deleted informs the other DCs in the environment about the deletion by replicating what is known as a tombstone. A tombstone is a representation of an object that has been deleted but not fully removed from the directory. The tombstone will eventually be removed based on the tombstone lifetime setting, which by default is set to 60 days. If a DC is restored to a state prior to the deletion of an object, and the tombstone for that object is not replicated to the restored DC before the tombstone expires, the object remains present only on the restored DC, resulting in an inconsistency. Thus it is important that the DC be restored prior to expiration of the tombstone, and that inbound replication from a DC containing the tombstone to the restored DC is completed prior to expiration of the tombstone. Active Directory protects itself from restoring data older than the tombstone lifetime by disallowing the restore. As a result, the useful life of a backup is equivalent to the "tombstone lifetime" setting for the enterprise. Given this, the backup interval should be at least once within the tombstone lifetime. However, Microsoft strongly recommends that administrators backup the System State and system disk more often to ensure, at any given time, a backup is available that holds a recent version of the data. Important: Backup data from a DC can only be used to restore that DC. You cannot use a backup of one DC to restore another. To have your environment completely backed up, you would need to have a backup of every domain controller. This should be kept in mind while developing your backup strategy. The minimum requirement should be to backup all the OM role holders and GCs. Also the first domain controller in the root domain should always be backed up.

BACKUP STRATEGY What to backup (minimum) Recomended
Backup all Operation masters role holders in your Forest. At least 1 GC in each Domain. All DC’s in your root domain (critical) All DC’s also working as application servers At least 2 backups within the tomstone life Recomended All DC’s in your Forest Quick recovery in the event of Active Directory failure or a domain controller hardware failure Every week (Trust Relationship password)

Tombstoning 60 days Deleting an object from AD 1. Object gets converted into a tombstone state ”IsDeleted property” (not fully removed) 2. Inform all other DC’s for deletion 3. Object is deleted when tombstone lifetime is reached. (Garbage Collection Process) Tombstone lifetime Article Q216993 After they are deleted by the garbage collection process, they no longer exist in the directory database Note! AD protects itself from restoring data older than the tombstone lifetime by disallowing the restore. As a result, the useful life of a backup is equivalent to the tombstone lifetime setting for the enterprise.

Garbage Collection Process running on every DC at regular intervals every 12 hours. Deletes Tombstone objects. Delete any unnecessary Log files. Defrag the Database file. ADSI Edit is used to change the interval of Tombstone Lifetime and Garbage Collection Interval.

More Garbage Collection
Windows 2000: Each pass removes 5000 objects every 12 hours. Removes max 10,000 objects a day. Windows Server 2003 Default still runs every 12 hours. Each pass removes 5000 objects Difference in 2003: Garbage collection will reschedule itself to run immediately until all of the objects are removed

Restore of AD if older than the tombstone lifetime setting
Recovery Options: Reinstall the server after confirming there is at least one surviving domain controller. If every server in the domain is destroyed, restore one server from an arbitrarily outdated backup, and replicate all other servers from the restored one.

Deleted Objects in AD on a DC
Deleted “Objects container” Use LDP.exe , Search dc=msft,dc=com Filter = (isdeleted=*) Scope options = Subtree Search Call Type = Extended Object Identifier = Control Type = Server Q258310

Reanimation of Tombstones
Recover Deleted Objects – Described in Q840001 Limited attributes preserved - SID, Object GUID and last know parent Best Suited for small scale recoveries Backup and authoritative restore preferred for most scenarios Process Search for Deleted Objects Replace IsDeleted attribute and RDN Manually recreate group memberships and other attributes Required Permissions Reanimate Tombstones ACE on root of domain

Restoring Deleted Objects in W2K3
An object can still not be restored when the tombstone lifetime for the object has expired Code example that shows how to restore a deleted object in W2K3 at MSDN: SID and GUID are retained Some system attributes get stripped, and there's no way of putting them back into the reanimated object

Reanimation

Best Practices for a Good Backup
To ensure a successful restore of Active Directory from a “Good” backup Contents Age For full disaster recovery, back up all of the drives and the System State data. “Traveling” DC’s – Offline max Tombstone age or (2x computer password change) Q – W2k-SP3 or later support adds support for removing lingering objects Use LDP to search for duplicate user, group, or distribution list (Standard = Loose Replication Consistency) “Strict Replication Consistency” was created to prevent unwanted replication of lingering objects (not implemented when upgrading from W2K to W2K3)

Lingering Objects One or more objects never considered for replication
HighestCommited USN lower than DCs Highwater mark Problems with Lingering Objects Difficult to remove from Global Catalogs Causes Name conflict Halt Replication

Lingering Objects- Strict replication consistency Event 1864, 2042, 1988
1. Demote or reinstall the machine(s) that were disconnected, or….. 2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects 3. Resume replication. You can continue replication by using the following registry key: HKLM\System\CurrentControlSet\Services\NTDS\Parameters\”Allow Replication With Divergent and Corrupt Partner” to 1 Once the systems replicate once, it is recommended that you remove the key to reinstate the protection.

Type of Disaster Database Corruption – Reinstall situations
Disks become corrupted –writeback cache is not saved to a power failure Hardware failure – Replace DC or HW Software failure - Prevents system boot Data Corruption – Restore from Backup Accidental deletion of objects or files and has been replicated to other DC’s Corruption of Active Directory data, which has replicated to other domain controllers Fast Growing Database – DSASTAT –S Compare different DC’s - DSASTAT -S Types of Disaster When faced with a disaster, you must first determine the type of disaster. This paper focuses on troubleshooting two possible types of disasters: Database corruption—a situation in which one of the following occurs: Disks become corrupted, such as when the writeback cache is not saved due to a power failure and bad batteries. The domain controller has suffered a severe hardware failure and needs to be replaced. Software failure prevents the machine from booting in normal mode. Data corruption—defined as a situation in which an administrator or someone with the appropriate permissions has accidentally deleted an object and the deletion has replicated to other DCs within the environment.

Type of Disaster Data Corruption – Repair
Esentutl.exe repair of database is a last resort Use integrity check to see if database is damaged High Risk this process will result in further loss of data Do Not spend time to Repair the Database on a single DC, if other DC’s are working fine!! INTEGRITY = Helhet, fullstendig, Integritet

AD Disaster Recovery Objectives:
To resolve problems on domain controllers that affect clients, the domain or forest operation in: Least amount of time Least amount of pain Best possible results First determine what kind of disaster you have.

Reality Administrators often NOT do:
Backup – Typical PSS case Test backups prior to disaster – Is my backup ok? Test your recovery plan Create labs mirroring production environment Monitor failure symptoms and events – JET errors Administrator with ONLY few problems Follow the steps above PSS experience shows that for the majority of disaster recovery calls, no backup is available to recovery deleted objects or return the domain controller to a known good state. Lab testing includes testing configurations that will be deployed in the field including Network Utilization, Optimization Scalability Recovery Hardware configurations and the drivers that support them Testing the integrity of backups and various recovery scenarios is critical. Possible scenario based on real life examples includes: Hardware Failure Deletion of File System portion of policy Deletion of OU containing live users, groups or computer accounts Deleting NTDS Settings objects for DC’s Deleting machine accounts for Windows domain controllers Deleing all FRS members under an FRS replica set FSMO role recoveries Failure and recovery of individual services on the machine without returning healthy services to earlier state The number of calls involving the deletion of critical objects from the Active Directory has been surprisingly high. To issues are (from most to least): File System portion of policy (Policy is critical to healthy domain controller operation) Machine accounts for Windows domain controllers Trust relationships Monitoring symptoms and events that can lead to is critical failures is imperative. The current set of tools forces administrators to poll all machines in the forest since there is no event collection process or filter to illustrate critical events or patterns. Jet errors involving the NTDS DIT or related components DC’s that haven’t replicated in large periods of time including tombstonelifetime

Tombstonelifetime Lab

Preferred Recovery Options
Reinstallation: Winnt32 + DCPROMO + Re-replicate Winnt32 + DCPROMO IFM (Install from media) ForcedRemoval + Metadata Cleanup + DCPROMO + Re-replicate ForcedRemoval + Metadata Cleanup + DCPROMO  IFM Restore NTBACKUP restore to last known good state Re-Replicate changes made from other DC’s Repair Not an recovery option for AD Use Integrity Check to see if database is corrupt Removed from Windows 2003 The preferred methods of recovery are: Reinstall: WINNT32 + DCPROMO then replicate Restore: Restore to known good state and replicate Repairs: Database repairs for Active Directory is not an option for recovery.

Core Recovery Tools NTBACKUP NTDSUTIL & ESENTUTIL
Snap shots of system as state changes made NTDSUTIL & ESENTUTIL Database validation Metadata cleanup for DC / Domain removal Esentutl – Database Validation and Repair WINNT32 & DCPROMO + (IFM) Rebuild a DC faster and gives better results NTBACKUP Supports backup of Active Directory and system state in Active Directory mode Backup images can be stored on tape or any logical drive letter Scheduled backup support Restores of system state and active directory require boot into offline restore mode Restores on remote servers can be driven over terminal services. See Q256588 Lacks per service granularity Restores put working components to earlier state in time NTDSUTIL includes options for Authoritative Restore: Mark entire database as authoritative Mark specified objects or containers as authoritative Metadata Cleanup Remove selected domain controllers from a domain Remove selected domains from a forest Database Validation and Repair NTDSUTIL is a wrapper around ESENTUTIL WINNT32 and DCPROMO Sometimes the recovery method that gives the best results in the least amount of time is rebuilding and re-promoting the server. Certain recovery options like recreating deleted Windows 2000 domain controller accounts using DCDIAG demand a demotion and re-promotion. Component specific recovery options Component or service specific recovery options may be preferred since NTBACKUP restores lack granularity FAZAM includes options to backup and restore policy DFSUTIL can build batch files of current DFS configuration which can be rebuild DFS tree

Dcpromo from Media Benefits
Reduced network utilization when additional DC's are promoted into an existing domain Faster sourcing of Active Directory and Global Catalog data to the new domain controller Improved recovery of domain controllers in the event of failure !!!!!!!!!!!!!!!!!!!!

Dcpromo from Media When can you use it
Can only be used on W2k3 servers Can only be used for additional Domain Controllers in the same domain. Cannot be used for the first DC in a domain.

Dcpromo from Media Wen can you use it.
The system state backup must be no more than 60 days old. (results in reintroduction of tombstoned objects) The system state backup must be taken from a Windows 2003 DC in the same domain. (GC can also be included) The server you want to promote from media must be on the network and must be able to communicate with other healthy DCs.

Using "Install from Media"
IFM promotions consist of 3 steps: Performing a system state backup from a Windows Server 2003 DC to a Media(DVD) Restoring the System State Backup .bkf to an "Alternate Location“ on target machine IFM Promotion of a Replica domain controller with DCPROMO /ADV

Windows Backup backup system state DC Target Server DCPROMO /ADV Store to media: DVD CDROM Tape File System Restore to an alternative location

Create Replica From Media in Windows Server 2003
Source initial replication while promoting DC from backed up files instead of the network Backup DS using regular backup s/w Restore/copy files to candidate DC DCPROMO: source replication from restored files Also works for GCs Network connectivity still required Not a general replication mechanism

INSTALL FROM MEDIA Lab

NTDSUTIL Metadata cleanup Integrity Check + Repair
Remove orphaned dc’s or child domains Integrity Check + Repair Wrapper around ESENTUTL Tells you if database is good or bad – Jet Error Power failure, Faulty Hardware Authoritative Restore Mark selected objects on DC as authoritative

More NTDSUTIL Move the Database, Log files to another Disk.
Offline Defragmentation. To release space back to the file system. Must be done on Per DC.

Remote Administration
Q Administrating remote servers in DS restore mode Create a new entry in the Boot.ini file with /SAFEBOOT:DSREPAIR /SOS Set the appropriate boot option in the arch path and reboot the system multi(0)disk(0)rdisk(0)partition(2)\WINNT=“W2K DC \\your server name” /fastdetect multi(0)disk(0)rdisk(0)partition(2)\WINNT=“W2K DC \\your server name“ /fastdetect /SAFEBOOT:DSREPAIR /SOS Some low-level maintenance of the Windows 2000 Active Directory requires that Windows 2000 domain controllers (DCs) boot to Directory Service Restore mode. Configuring Windows 2000 domain controllers with Terminal Services in Remote Administration mode permits administrators to perform operations requiring Directory Service Restore mode without having to be present at the console of the server. This article describes the use of Terminal Services to transition a Windows 2000 domain controller between online and Directory Service Restore mode. Create a new entry in the Boot.ini file (a hidden system file) for the Windows 2000 Domain Controller installation to permit Windows 2000 to be booted in Offline Repair mode. Add the following switch: /SAFEBOOT:DSREPAIR /SOS The /SAFEBOOT:DSREPAIR switch only works Windows 2000 DCs. For a sample Boot.ini file with the entry: multi(0)disk(0)rdisk(0)partition(2)\WINNT="W2K DC \\<your server name>" /fastdetect Create a second entry with the same ARC path and /SAFEBOOT:DSREPAIR switch so the Boot.ini file appears as: multi(0)disk(0)rdisk(0)partition(2)\WINNT="W2K DC \\your server name" /fastdetect /SAFEBOOT:DSREPAIR /SOS NOTE: This should be tested locally prior to being used in a Remote Administration capacity. If the Boot.ini file is not modified properly, the computer will not come back up for connection by a Terminal Services session. Additionally, when you restart the computer, make certain you select Restart so it will properly restart. Choosing "Shut down" leaves the server turned off until someone physically goes to the server and turns it back on. The Terminal Services session will generate the following message if the server has not come back up for connection yet: Terminal Services Client Disconnected The server could not be found. Check that you have specified the correct server or IP address, and then try connecting again. Click Close, and then connect again after a few moments to make the connection.

How to restore AD [Winnt32]
Reinstall Windows 2000 Winnt32+SP3+IP-config + DCPROMO Normal replication process from a healthy DC in the same domain Not used to restore AD to a known state Computer/Server object needs to be cleaned out of AD Bandwith considerations – Slow links

Re-installation Steps involved: In correct order
Cleanup operation, such as removing the failed DC object from Active Directory. Installing a fresh copy of Windows 2000 Server. Running DCpromo.exe to promote this machine to the domain controller role.

NTDSUTIL Re-installing using the same DC name
Remove the failed domain controller from AD If the new DC receives the same name as the failed DC, you must remove the ntdsDSA object of the failed DC: At the command line, type ntdsutil. At the prompt ntdsutil:, type metadata cleanup and press Enter. You need to now connect to an existing domain controller on which you want to remove the ntdsDSA object of the failed DC. At the metadata cleanup prompt, type connections and press Enter. Type connect to server <servername> and press Enter. Where <servername> is the DC that will be used to clean the metadata from (any functional DC in the same domain). Type quit and press Enter. This will return you to the metadata cleanup menu. Type select operation target and press Enter. Type list domains and press Enter. This lists all domains in the forest with a number associated with each. Type select domain <number> and press Enter where <number> is the number corresponding to the domain in which the failed server was located. Type list sites and press Enter. Type select site <number> and press Enter where <number> refers to the number of the site in which the DC was a member. Type list servers in site and press Enter. This will list all servers in that site with a corresponding number. Type select server <number> and press Enter where <number> refers to the DC to be removed. Type quit and press Enter. The Metadata cleanup menu is displayed. Type remove selected server and press Enter. At this point you should receive confirmation that the DC was removed successfully. If you receive an error that the object could not be found, it might have already been removed from the Active Directory. 16. Type quit, and press Enter repeatedly to return to the command prompt. Note: Because this procedure requires modifying the configuration naming context, it requires Enterprise Administrator permissions. If the new DC receives a different name than the failed DC, you should perform the following additional steps: Removal of the failed server object from the Sites & Services snap-in: Open the Sites & Services snap-in. Select the appropriate site. Delete the server object associated with the failed DC. Removal of the failed computer account from the Users & Computers snap-in: 1. Open the Users & Computers snap-in. Select the domain controllers container. Delete the computer object associated with the failed DC. WARNING: Do not perform the additional steps above if the new machine will have the same name as the failed machine. Make sure that hardware failure wasn’t the cause of the problem. If the faulty hardware isn’t changed, then restoring via reinstallation may not help.

Cleanup cont. Q216498 Delete the FRS member object.
Remove the cname record in the _msdcs.root domain of forest zone in DNS. If this was a DNS server, remove the reference to this DC under the Name Servers tab Delete the computer account. [Adsiedit] CN=domain controller, OU=Domain Controllers, DC=Your Domain Name, DC=COM, PRI, LOCAL, NET. Delete the FRS member object. CN=Domain System Volume (SYSVOL share), CN=File Replication Service CN=System, DC=Your Domain, DC=COM, PRI, LOCAL, NET If it was the last domain controller in a child domain and the child domain was also deleted: Delete the trustDomain object for the child: Trust Domain object, CN=System, DC=Your Domain, DC=COM, PRI, LOCAL, NET Use ”Active Directory Sites and Services” to remove the Domain controller

Re-installation consideration
Allow time for the deletion of Server object to replicate throughout the forest Check the DCPROMO.log to verify the promotion completed successfully Use “Repadmin /showreps” or Replmon to verify that connection objects have been re-established Disadvantage: Other Applications needs to be reinstalled/configured Allow the appropriate amount of time for the deletions to replicate throughout the domain. Take into account domain controllers that are located across WAN links and whose site link replication interval is set to a high value. Ensure that the domain naming master receives the replication of the server deletion before continuing with the promotion of a domain controller with the same name. REPADMIN /SYNCALL /P

NTDSUTIL Lab

Non-authoritative What is it? When to use: SYSVOL
Restore to known good point using NTBACKUP – (Maintain the version number from the backup) Reboot into AD mode to apply all updates after backup When to use: Local server problems Data or application loss from reinstall too expensive SYSVOL Will automatically be updated by a replication partner It is NOT possible to use the non-authoritative restore process to reinstate deleted objects from an older backup Q216993: Backup of the Active Directory Has 60-Day Useful Life [arnetwork]

Non-authoritative restore
Reboot into DS Restore mode Restore system state and/or system disk Choose ”Advanced restore mode” options Restart DC in ”Normal” mode AD will replicate new information Non-Authoritative restore of SYSVOL When you restore the System State data, the location of the system root must be the same as the location when you backed up the System State data.

LAB: Non-authoritative restore
Use NTBACKUP to backup the system state on DC2 Unplug the network cable on DC2 (simulate a Network failure) Create a new OU object + a user on DC2 Restart DC2 in DS restore mode (simulate a catastrophic event on DC2) Perform a non authoritative restore of the backup you created Plug in the network cable and restart the server in normal mode Question: What happen with your new objects created in step 3 ???? Results: In this case, the new objects or modifications that originated on the domian controller after the backup are lost because they were never replicated to your other domain controllers and therfore can’t be applied to the restored domain controller. Conclusion: Always fix replication problems as soon as possible.

LAB RESULTS What did happen with your new objects?
The new objects that originated on the DC after the backup are lost because they were never replicated to other DC’s, and therefore can’t be applied to the restored DC. Conclusion: Always fix replication problems as soon as possible.

Authoritative Restore
What is it? Restore to known good point using NTBACKUP Make objects on reference dc as “master copy” for DS When to use Accidental deletion or modification of objects or containers in the domain or configuration NC Ability to restore entire AD or a single object Performed in DS Restore mode Note that

What it isn’t Q241594 Will not overwrite objects created after the backup Only carried out on objects from the configuration and domain contexts Will not overwrite objects which are tombstoned (>60 days dafault)

What it isn’t Auth restores of schema naming context are not supported. (Recovering AD Forest) The schema cannot be authoritatively restored because it might endanger data integrity. For example, if the schema was modified and then objects of the new or modified class schema object were created, subsequent authoritative restore might replace the new or modified classes, thereby causing serious data consistency problems

Enforcing Functionality Levels
Backup and Restore issues Restore of Windows Server 2003 prior to mode increase, forest or domain Restore of prior OS after version increase (upgraded DCs) Ntdsutil Limitations Authoritative restore of msDS-Behavior-Version not allowed

Boot into offline restore mode Press F8 during boot phase Login with offline administrator account Restore System state Mark objects in NTDSUTIL as authoritative Find machine w/ objects or restore image that has them Restore (entire) database (rare) or SUBTREE USN, originating invocation IDs, Version Numbers on reference server are incremented to WIN replication Movie Breakout Auth Restore Process 1.>Boot into offline restore mode by pressing F8 key during boot phase 2.>Enter account and password for offline administrator account 3.>Start NTDSUTIL in “Authoritative Restore” mode 4.>Mark entire database(rare) or DN for selected subtree as authoritative 5.>Reboot machine into normal DS mode Remote Restores over Terminal Services Steps described in Q Remote Administration of DCs in Directory Service Restore Mode allow machine to be booted into offline restore mode then restore the backup image using Terminal Services.

Authoritative Restore of AD
NTDSUTIL Type ”Authoritative restore” ”Restore database” – Entire Directory Restore Subtree – Specific OU OU=market,DC=msft,DC=com DC Machine account Deletion Restore Subtree ”CN=DC3,OU=Domain Controllers,DC=msft,DC=com” User account Deletion Restore object (New option in W2K3) CN=User1,CN=Users,DC=msft,DC=com

Ensure that SYSVOL and AD remain synchronized. Always authoritatively restore the Sysvol folder when you authoritatively restore the entire Active Directory Database. Gpt.ini – Version number of the Group Policy object Gpotool.exe – List all policies for your Domain. Checks version conflict with sysvol.

Auth Restore of Specific AD Objects and Corresponding GPO Objects from SYSVOL
Restart the computer in directory service restore mode Restore the System State data to its original location and to an alternate location By using Ntdsutil, separately mark specific Active Directory objects as authoritative Restart the computer in normal mode After the SYSVOL share is published, copy only Policy folders (identified by the globally unique identifier [GUID]) corresponding to the restored Group Policy objects from the alternate location

Authoritative Restore of Sysvol
Advanced Restore Options: “When restoring replicated data sets, mark the restored data as the primary data for all replicas”. (Only Single DC in a domain) System state to ”Alternate location”: Restore system state also to an Alt location <Alternate Sysvol Location> Scripts and Policies

Advanced Options

Morphed Folders C:\Windows\SYSVOL\DOMAIN
Policies Policis_NTFRS_030800fd Scripts Scripts_NTFRS_0004c427 The last writer gives a non-conflicting name: Foldername_NTFRS_GUID The loser keeps the original name The Administrator must intervene to resolve the names

Verifying Authoritative restore
REPADMIN /SHOWMETA ”CN=DC3,OU=Domain Controllers,DC=msft,DC=com” Version number incremented with Version number replicated to other DC’s Restore Database Verinc %d Increments the version number by %d

Authoritative Restore Scenario
Day1: Backup of DC3 Day2: ”User Two” is created and replicated to other DC’s Day3: ”User One” is inadvertently deleted. Day4: Authoritative Restore on DC3. Result: All users exist in domain. Conclusion: Will not overwrite objects created after the backup

Repadmin /Showmeta with Incremented Version Numbers

Can’t remember your DS Restore Mode “Administrator Password”
Same password is also used by Recovery Console The SAM-based account and password are computer specific SP2: c:\winnt\system32\Setpwd Specify a new password. Q239803 Setpwd on a Remote DC Setpwd /s:<servername>

Lab

Scenarios Hardware Failures Event ID 1018s Event ID 1168
Recovering deleted objects Deleted printer objects FSMO’s Duplicate SIDS after restore Critical system objects Objects with passwords File System Policy Restores and FRS Impact on Group Membership Forcefully demote a DC

Hardware Failure Scenario: Goal: Test Ideal Scenario:
DC experiences catastrophic hardware failure Goal: Restore the DC to new hardware Test Restoring to similar and dissimilar hardware Ideal Scenario: Target hardware identical to source NICS + Video + HAL + Kernel + # of Processors

Dissimilar Hardware W2K
Reality Not a trivial process. Complex and time consuming. Use KB Q to improve results Requirements (few examples……..) Backup SYSVOL + SYSTEM STATE Higher probability if %SYSTEMROOT% backed up Target machine has same number of drives & drive letters for entities being restored. Disk Controller….. Incompatible Boot.ini File. If you backup and restore the boot.ini file, you might have some incompatibility with your new hardware configuration, resulting in a failure to start. Logical drives on destination same size or larger Uninstall NIC and Video card before you restore data, if it’s different. Let plug&play make necessary changes Q263532: Disaster Recovery of Active Directory on dissimilar hardware [idea]

Dissimilar Hardware Restoring AD to dissimilar Hardware
Perform Clean Install as a stand-alone server Restore System partition and system state Select “Always replace the file on disk” in advanced restore settings Reboot the server in normal mode

Dissimilar Hardware in Windows 2003
Backup the System State from the Source DC Use DCPROMO /ADV with ”Install from Media” option on target computer. Choose "Additional domain controller for an existing domain”

Jet Error 1018 Event 404 (1) Problem Symptoms Causes
Jet error 1018: possible data corruption from faulty hardware. Symptoms Successful write is reported, though write fails Events may not be logged All classes and price points of hardware affected Causes Can vary from faulty RAID & SCSI Firmware, Termination Bad Hard Drives Description: An Application Error 1018 may occur on any Compaq server configured with Microsoft Exchange Server Version 4.0, 5.0, or 5.5 and any Compaq RAID Array Controller (SMART, SMART-2, Smart Array 4250ES, Smart Array 4200, Smart Array 3100ES, Smart Array 3200, Smart Array 221, or RAID Array 4000). Application Error 1018 is reported by the JET database and indicates possible data corruption from faulty hardware. NOTE: The JET database is the database engine for Microsoft Exchange Server. Application Error 1018 does not specifically indicate data corruption has occurred. Rather, the error indicates that the read verification process has failed the verification test. Application Error 1018 can result from many conditions, such as faulty hardware devices, slow transfer rates, or slow device responses to relinquish the SCSI bus.

Jet Error 1018 (critical) Impact: Corrupts AD, and FRS jet databases
Backups may also be corrupt Resolution: Check with HW vendors for known issues Replace/test Physical drives Restore the database Defragment the database with NTDSUTIL Hard repair the database with NTDSUTIL – Can results in loss of data (if you not have an ok backup)

Event ID 1168s Problem Active Directory fails to boot with Event ID 1168s 1168s generic events: contact Premier Support Unable to start the Active Directory Cause Permissions too restrictive on NTDS.DIT & LOGS or the NTDS folder Unscheduled loss of power that can cause the Ntds.dit file or log files to become un-readable Resolution: Q265089 Define Default permissions Delete/rename EDB.CHK file – Soft Recovery non-auth restore to recover Reinstall the operating system on the failed computer Q265089: Troubleshooting Event 1168 in Windows [idea]

Recovering deleted objects
Problem Accidental deletion of OU or other objects Resolution Restore + Authoritative Restore in NTDSUTIL Restore recent backup Mark deleted objects as authoritative Authoritative Restore in NTDSUTIL Find replica dc that hasn’t received deletions Mark deleted DN as authoritative (no restore required) Reduce LDAPSRVWEIGHT and PRIORITY to prevent being considered for authentication

Revovering AD objects No Good System state Backup – Q237677
LDIFDE - Export and Import Directory Objects Export OUs, users, and groups from an entire forest Run LDIFDE export commands against each domain in the forest, or alternatively, run the query once against the global catalog (GC). No SID history

Recovering deleted DNS zone
Deleted a DNS Zone called: Test.sample.com and want to restore it back: Restore Sytem state from backup in “DS restore mode” Start NTDSUTIL Restore Subtree DC=test.sample.com,cn=MicrosoftDNS,cn=system,DC=MS,DC=COM Reboot DC in normal mode

Deleted Printer objects
Printer Pruning on DC’s (for it’s own site) A process which keeps printer information in Active Directory current. Controlled by GPO’s. Q234270 ”Allow pruning of published printers” ”Directory pruning interval” - Default 8hours Stop the Spooler service on the offline DC If a DC does not see the printers for a period of time, it may consider the printers orphaned when the DC come back online WAN LINKS

Deleted Objects in Active Directory
Protection Set replication schedule once every >four days on “backup domain controller” Mark objects as authoritative when deletion detected

FSMO Rules Transfer if role needed before current owner goes offline
Seize if current role owner offline & never coming back Roles and Dependencies PDC: Down level clients, Policy updates, DFS updates, RID: Inability to add any new security objects as users, groups, computers beyond local rid pools on each DC Domain Naming Master: Domains cannot be added/removed Seizure Rules Install of OS that held seized role can never come back online Reinstall with same names is ok. Do Not use old Backup’s. PDC: Downlevel client password changes, domain joins, object creation and modification, policy updates, DFS updates, some object picker enumeration RID: Bulk creation of security principals beyond local rid pools Domain Naming Master: Addition of 1st domain in forest, Removal of last dc in domain (removal of domain) Domain Naming Master

RID Master Considerations for performing a Seizure on a RID Master.
Risk of Duplicate RIDs. Original Master should NEVER come back online again if a Seizure is performed. Instead, the original role holder must be reinstalled before introduced in the Domain again.

Duplicate SIDS after restore
Problem after Backup and Restore of a DC. RID pool is set to a range that has already been used to allocate SIDs Duplicate SIDS are created as the customer creates new users & groups on restored DC May not be immediately detected NTDSUTIL: "check duplicate sid” "cleanup duplicate sid” Q315062 Resolution: Finally SP4 Q or W2K3 Duplicate relative ID pools can occur if the administrator seizes the relative ID master role while the original relative ID master is operational but temporarily disconnected from the network. In typical practice, after one replication cycle, the relative ID master role is assumed by just one domain controller. However, before the role ownership is resolved, two different domain controllers might each request a new relative ID pool and be allocated the same relative ID pool. How to Check for a Duplicate SID 1. At the Ntdsutil command prompt, type "security account management" (without the quotation marks), and then press ENTER. 2. At the Security Account Maintenance command prompt, type "connect to server <DNSNameOfServer>" (without the quotation marks), and then press ENTER. Connect to the server that stores your SAM database. 3. At the Security Account Maintenance command prompt, type "check duplicate sid" (without the quotation marks), and then press ENTER. A display of duplicates appears. How to Clean Up a Duplicate SID 3. At the Security Account Maintenance command prompt, type "cleanup duplicate sid" (without the quotation marks), and then press ENTER. Ntdsutil confirms the removal of the duplicate. 4. At the Security Account Maintenance command prompt, type "q" (without the quotation marks), and then press ENTER. 5. When you are finished with Ntdsutil, type "q" (without the quotation marks), and then press ENTER.

Restoring Operations Masters
Seize the role if you not intend to restore the original role holder from a backup. Seize a fsmo role ONLY as last resort. Active Directory continues to function when the operations master roles are not available. Operations Master Servers Active Directory supports multi-master updates. Each DC hosts a read/write version of its directory partition. Therefore, the Active Directory must allow for the possibility of conflicting changes, such as changes made simultaneously to the same object within the directory but on different DCs. The Active Directory uses a well-defined conflict resolution method and eventually all DCs converge to the same value. Even with this well-defined method, it is sometimes better to prevent conflicts than to resolve them after the event. Operations masters in Active Directory prevent conflicting updates in cases where conflict resolution is unsuitable. Active Directory defines five operations master roles: schema master domain naming master relative identifier (RID) master primary domain controller emulator (PDCE) infrastructure master The schema master and domain naming master are per-forest roles, meaning that there is only one schema master and one domain naming master in the entire forest. The other operations master roles are per-domain roles, meaning that each domain in a forest has its own RID master, PDCE, and infrastructure master. To check which DC owns the domain naming master role, open the Domain and Trusts snap-in. To check the schema master, open the Schema snap-in. For any of the per-domain roles, check the Users & Computers snap-in. At each snap-in, at the very top container (in the left pane), right click and select operations master.

Schema operations master
Isolate Schema additions made by adprep. Prevent a full Forest Recovery: Temporarily disable outbound replication repadmin /options +DISABLE_OUTBOUND_REPL X:\I386\>adprep /forestprep view the Adprep.log file in the %systemroot%\System32\Debug\Adprep\Logs\<Latest_log> folder Enable replication again on the Schema Master

Recommendations for Returning to service after Seizure
Schema Master Not recommended. Can lead to a corrupt Forest and require a Forest Recovery Domain Naming Master Not recommended. Can require rebuilding Domains. PDC Emulator Allowed. No Permanent damage occurs. Infrastructure Master Allowed. No damage occurs to the Directory RID Master Not recommended. Duplicate RID pools can be allocated to DC’s, leading to data corruption in the Directory.

Backup / Restore FSMO Rules
Document the machine names of FSMO role holders when you do the backup Include %windir%? Restore DC's that have not replicated for TSL # of days should not get fixed. They should get removed and rebuilt. Do not reduce TSL unless verifying end to end replication of forest. As a general rule, do not reduce TSL to low # of days to accelerate deletion of a tombstone Never restore FSMO's except as last restort - implies awareness of where FSMO was when backup made. Rules: Don't transfer or seize FSMO role of current owner offline unless you are performing dependent operation

Procedures for seizing fsmo’s
1. First replicate all AD changes to all DC’s in your forest. 2. Verify successfull replication for your DC which seize the role. 3. Seize the FSMO role. 4. View the current FSMO role holders. ”Netdom query fsmo”. Replmon

Critical Objects Machine Accounts for DC’s NTLM Trust Relationships
Machine account password and Trust relationship password is reset every 30 days Q257288:Recover from a Deleted DC Machine Account Dcdiag /s:localhost /repairmachineaccount Demote and then re-promote the server. (Services) Or, Dcpromo /forceremoval + clean up metadata + re-promote NTLM Trust Relationships Password is reset every 7 days Authoritative Restore - NOT older than 14 days To reset NTLM trust relationships to Windows 2000 or downlevel domains, the trust must be removed and re-created. Q257288: How to Recover from a Deleted Domain Controller Machine Account Q232538: Unsuccessful Replication Without Partner Listed

DC Computer Accounts Problem: Defaults Symptoms of mismatch?
AD replication uses locally held password for Kerberos authentication Defaults Password default change interval = 30 days (N) Backup useful life = 60 days Password history = N + N-1 Symptoms of mismatch? AD Replication fails with “access denied” Q216243: Impact of Authoritative Restore on Trusts and Computer Accounts[ntrelease]

Recovering File System Policy
Scenario: File system portion of policy has been deleted Cause Administrators deleted policy to rebuild from scratch Deleting a File/Directory from Sysvol on one DC, results in FRS replicated deletion to all members of set. If C:\Winnt\sysvol is copied to alternate location D:\sysvol = The junction points in sysvol are also transfered to the alternate location. Junction points still points to to original location C:\Winnt\sysvol Q How to Reset User Rights in the "Default Domain Controllers" GPO How to Reset User Rights in the Default Domain Controllers GPO

Recovering File System Policy
Recovery: Locate policy files Restore backup image of SYSVOL to alternate location Turn off “Mark restored data as primary for all replicas” Locate files on member with delayed schedule Locate files in pre-existing directory Copy files to member and let replication take place Q How to Reset User Rights in the "Default Domain Controllers" GPO How to Reset User Rights in the Default Domain Controllers GPO

Authoritative Restore of Sysvol

Rebuild Sysvol with Burflags
BURFLAGS = (backup / restore flags) NET STOP NTFRS HKLM\System\CCS\Services\NtFrs\Parameters\Backup/Restore\Process at Startup\D2 D2: Will perform a full synch of the replica set from it’s Upstream partner BURFLAGS registry key to D4 (authoritative mode) Also used as last resort to fix FRS replication related problems.

Authoritative FRS Restore
When to use: (rare) Q315457 SYSVOL replica set meltdown, requiring a complete rebuild from an Authoritative member to perform a full sync. Resolution: Stop NTFRS service on all DC’s Set D4 on primary machine Set D2 on all other members Start NTFRS service on primary machine Start NTFRS service on all other members Q289836: Generic FRStroubleshooting steps for DFS and SYSVOL [idea] Q266679: Pre-staging files on SYSVOL and DFS replica members for optimal[idea] Q279156: FRS: FileSystemPolicy causes excessive Group Policy updates and Q282791: FRS: Disk defragmentation causes excessive FRS replication traff[idea] Q284947: Norton Antivirus 7.5causes excessive replication by FRS [idea]

Group Policies GPOTool.exe Verifies the health of the GPO on a DC
DCGPOFix.exe Restores the Default Domain Controller and Default Domain Group Policies to the state when the DC was installed GPMC – Group Policy Management Console Backup/Restore, Import/Export of Group Policy objects. Help full during disaster recovery scenario if a Backup of GPOs exists White Paper: Administering Group Policy with the GPMC

Unable to open Group Policy Snap-In
Sysvol and NetLogon are not shared out Are all directories and files in Sysvol Netlogon and frs services running ACLs are correct for c:\winnt\sysvol Are AD replication working ok? Create a test.txt file in sysvol. Does it replicate? Initialization of the system volume: Value of SysvolReady to 1 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters W2K3 article: Q How to Troubleshoot Missing SYSVOL and NETLOGON Shares

Recommendation for GPO’s
Accept the defaults that are set within the Default Domain Policy and the Default Domain Controllers Policy. Create new policies instead. Default Domain Policy – Account Policies Password Policy Account Lockout Policy Kerberos Policy Default Domain Controllers Policy User Rights Assignment Ensure all DCs receive consistent Group Policy settings Do not filter policy settings on individual DCs All DCs should remain in the Domain Controllers OU

Forcefully demote a DC W2K
If DCPROMO fails, how to demote…. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ProductOptions Edit menu, click String, type ServerNT Promote the computer to a different forest demote the computer as standalone server Q Removing Active Directory Data After an Unsuccessful Demotion 1. Modify the ProductType value in the registry: a. Start Registry Editor (Regedt32.exe). b. Highlight the ProductType value under the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\ProductOptions c. On the Edit menu, click String, type "ServerNT" (without the quotation marks) (use the exact case), and then click OK. NOTE:If the value is not set correctly or is misspelled, you may get the following error: System Process - License Violation: The system has detected tampering with your registered product type. This is a violation of your software license. Tampering with product type is not permitted. d. Quit Registry Editor. 2. Restart the computer. 3. After the computer is restarted, log on with the Administrator account and password used for Directory Service Repair mode. 4. After you restart the computer, it will behave as a member server. However, there are still some remaining files and registry entries on the computer that are associated with the domain controller. To remove this data: a. Run Dcpromo.exe. b. Promote the computer to a domain controller for a new, temporary domain, such as "psstemp.deleteme". NOTE: Ensure to promote the computer to a different forest. c. After the promotion, run Dcpromo.exe again, and then demote the computer to a standalone server. 5. After forcefully demoting a domain controller, you need to remove some final metadata that is left in the domain. For additional information about how to do this, click the article number below to view the article in the Microsoft Knowledge Base: Q Removing Active Directory Data After an Unsuccessful Demotion

Forcefully demote a DC If re-installation is NOT an option and…
Parent domain DC is NOT accessible when attempting to demote last DC in a child. Replica DC in same domain is NOT accessible when trying to demote. Replication failure due to failed authentication or replication error. DC hasn’t replicated in Tombstone life number of days.

W2K3 and W2K-SP4 Forcefully
Force demotion W2K3 DCPROMO /FORCEREMOVAL W2K  DCPROMO /FORCEREMOVAL Need Windows 2000 SP4 Does not cleanup metadata from other DC’s in AD Use NTDSUTIL to cleanup the metadata Q Remove Old Metadata from the Active Directory Database Currently the DCPROMO /FORCEREMOVAL command doesn't work in DSREPAIR mode

Force Demotion Wizard 1 2 3 4

Remove a DC from a Domain
Standard/recommended method of removing a DC from a domain – demotion using DCPROMO. If a DC decommissioned incorrectly – need to clean up Stale metadata for that DC. Cleanup stale Metadata from Active Directory – NTDSUTIL Cleanup stale metadata from DNS Delete the A and CNAME Records. Optionally Cleanup metadata from DNS, WINS servers.

Domain Removal from a forest
GC maintains read-only partitions for every other domain in the forest Read-only partitions need to be destroyed when Corresponding domain removed from Forest. GC is demoted to non GC Tear down of these partitions done by KCC In windows 2000 KCC deletes 500 objects/NC every time it runs KCC runs every 15 minutes by default Cause long delays in destroying large NC In Windows server 2003 KCC runs asynchronously at full throttle to rip apart the NC faster Helps to Cleanup the removed the Domain faster

Restoring Groups and Users
If Groups and Users are athoritatively on one DC There is No guarantee that the users will replicate in advance of the group If a Group is replicated in advance of a User that is a member of the Group The receiving DC has no record of the User and deletes it from the Group

Impact on Group Membership
Problem: Group Membership information can be lost if groups are restored before users Cause: Backlinks to non-existent users are removed No way to define which objects replicate first after an authoritative restore If the group is restored prior to the member, the membership will be removed during replication, as you can't have a live group referring to a deleted member Resolution: Q (840001) All users and computers must be authoritatively restored and replicated out to all DC’s, and then all group objects must be authoritatively restored and replicated out to all DC’s again After you perform an authoritative restore of users and groups, the membership in the restored groups may be inconsistent across domain controllers. If the group is empty on the restored domain controller, but is populated on a replica domain controller, then when a user is added to the group on the restored domain controller, users are removed from the group on the replica domain controllers. CAUSE ===== This issue can occur because group membership is stored as the Member attribute on the group object. When a user is added to a group, a backlink is added to the user object as an entry in the MemberOf attribute. During an authoritative restore, if the group object is restored before the user object, then Active Directory removes the value from the Member attribute on the group because a user does not exist that has a matching backlink. After the authoritative restore, the version information on the Member attribute of the restored groups is consistent on each domain controller, even though the values in that attribute are not. Whenever the membership of the group is modified, the version number is incremented, and the contents of that group are replicated out to all domain controllers. If the group is modified on a domain controller that has a valid group membership, then the complete contents of the group are replicated, and data is not lost. However, if the group is modified on the restored domain controller, then only the added users are replicated, and users are removed from the group on the replica domain controllers.

Best Practices OU Structure - Auth Restore

Group Membership Lab

Restore of Schema failures
Examples Scenarios Schema corruption due to bug or user mistake replicated to all DC’s. Schema objects required by one application modified by other application. You need to restore the whole Forest to a point in time before corruption occurred. Use as a Last option, after determining the cause and possible remedies Applies only if, “All” DC’s in the Forest are affected

Forest Recovery Roadmap
Recover Forest Root Domain first Proceed to recover the remaining Domain’s. Rule of thumb, Restore the Parent Domain before Child Domain For each Domain Restore only one DC from a Good Backup!!! Promote remaining DC’s using DCPROMO For Windows 2003 DC’s use ”Install from media” This method is the recommended practice for Windows 2000 as well as Windows 2003 server forest.

Forest Recovery Timeline
Perform pre-recovery steps: determine a domain controller in each domain that you shall restore from backup and know the administrator password for that domain. SHUTDOWN ALL DOMAIN CONTROLLERS IN THE FOREST Perform an offline restore (including the recovery steps) on a single DC for each domain. Introduce the restored DC in the root domain back on the production network. Enable this DC to be a Global Catalog. Promote remaining DCs in the forest using the Active Directory Installation Wizard. Introduce the restored DC of each of the other domains on the production network (parents first). Force a replication sync between this DC and the first DC in the root domain and vice-versa. Install the operating system on all DCs that will be re-promoted using the Active Directory Installation Wizard Perform post-recovery steps

Recovering the Root Domain
Step 1: Restore AD marking SYSVOL primary Step 2: Verify Data on the restored DC Step 3: Configuring/Modifying DNS Server Step 3A: DNS server present prior to Failure OR Step 3B: DNS server absent prior to Failure Step 4: Disabling GC Flag if restored DC is a GC prior to failure Step 5: Seize FSMO roles Step 6: Clean up metadata of all other DC’s in the Domain Step 7: Delete Server and Computer objects for all other DC’s in the Domain Step 8: Raising the current RID pool Step 9: Reset Computer account password of the restored DC and LSA secret Step 10: Reset krbtgt password Step 11: Reset trust password Step 12: Introduce restored DC in Production Network Step 13: Enable the restored DC to be a GC

Recovering the Forest Detailed description in the following White Paper: Best Practices: Active Directory Forest Recovery

Best Practices The tombstone Lifetime Interval should not be reduced in a large environment A DC can not be off longer than the tombstone lifetime for the forest Separate the Database and Log files Backup ”System state” of DC’s frequently Perform Offline Defragmentation Only if you can recover a Significant amount of Hard Disk space DHCP and WINS databases requires special procedures to handle open files

Summary Common sense Take the least extreme method
Affect least number of objects possible Most deterministic results Least amount of time Determine Root Cause: Understand how you got here Learn from your mistakes Take steps to avoid repeat performances Make backups before you do anything so you can get back to where you started

Recover Root Domain Lab

Active Directory Disaster Recovery

Similar presentations

Presentation on theme: "Active Directory Disaster Recovery"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Active Directory Disaster Recovery

Similar presentations

Presentation on theme: "Active Directory Disaster Recovery"— Presentation transcript:

Similar presentations

About project

Feedback