SYSVOL Replication: FRS or DFS-R??? Rhonda Layfield Rhonda@Minasi.com Contents Copyright Rhonda J. Layfield 2009

Rhonda Layfield IT industry 25+ years NT/2000/2003 MCSE, MCT, MCSE+Security Contribute articles to Redmond and Windows IT Pro magazines Setup and Deployment MVP Desktop Deployment Product Specialist (DDPS) Co-Author of 2 Mastering Windows Server books (2003 & 2008)

SYSVOL LScript.vbs LScript.vbs Uptown DC Downtown DC

Prehistory to Today Always been a need for a tool to keep two or more server’s folders in sync NT -> LMRepl (Directory Replication) 2000 introduced FRS (2003 used this also) Sysvol DFS 2003 R2: a new Replication engine Sysvol uses “old” FRS R2’s DFS – which is called “DFS Namespace” – uses the “new” Replication engine called “DFS Replication” or “DFS-R” Server 2008 can be configured to use DFS-R for SYSVOL replication

Sysvol Created during dcpromo on domain controllers Automatically shared C:\Windows\SYSVOL\sysvol\Bigfirm.com (DNSDomainName) Group Policy Template (Settings) System Policies Scripts (NT & AD) Must reside on an NTFS volume

FRS Sysvol Terminology DC 1 Upstream Partner Direct Replication Transitive Replication DC 2 DC 3 Downstream Partner Upstream Partner Downstream Partner

FRS Process (10,000 foot view) NTFS logs changes to the “NTFS Change Journal” FRS monitors NTFS Change Journal FRS places changed files in staging area Replication partners notified “something’s changed” Partners request replication Files are transferred

NTFS Change Journal (aka USN Journal) What does it do? Logs all changes to an NTFS volume Separate log on each NTFS volume Doesn’t it take a lot of space? Sure; in fact it would eventually fill up a drive So Microsoft limits its size

So how large is it, and can I change it? W2K SP2 32 MB W2K SP3 512 MB 2003 pre-SP1 128 MB 2003 Hotfix 823230 or SP1/R2 512 MB Server 2008 512 MB Maximum size is 2 TB MS Recommends increasing by 128 MB for every 100,000 files/folders

What happens when the change journal fills up? Then NTFS just goes back up to the top and starts overwriting the oldest entries So… if FRS hasn’t checked in with the change journal in a while, then FRS may get lost This is called a “journal_wrap” Ooh No!

FRS monitors NTFS Change Journal 1 GUID1 2 GUID2 3 GUID3 4 GUID4 NTFRS FileIDTable 1 GUID1 2 GUID2 3 GUID3 4 GUID4 Received 1-4

FRS monitors NTFS Change Journal 1 GUID1 FileRef#: 0x000f000000003a6f USN: 0x000000000034cf40 NTFRS FileIDTable GUID1 FID 4222124650674799 USN 3460928

Tying the Change journal and FRS database together FileRef# & USN in the NTFS Change Journal – NTFS Utility: FSUtil FSUtil USN ReadData C:\WINDOWS\SYSVOL\sysvol\Domain\Policies\GUID1 Major Version: 0x2 Minor Version: 0x0 FileRef#: 0x000f000000003a6f Parent FileRef#: 0x0002000000002f45 Usn: 0x000000000034cf40 Time Stamp: 0x0000000000000000 12:00:00 AM 1/1/1601 Reason: 0x0 Source Info: 0x0 Security Id: 0x308 File Attributes: 0x20 File Name Length: 0xe File Name Offset: 0x3c FileName: GUID1

Dump GUID1 from FRS FileIDTable wmic /namespace:\\root\microsoftdfs path dfsridrecordinfo where filename=“GUID1" Attributes: 32 Clock: 20060906155126.906250-000 CreateTime: 20060906155119.203125-000 Fence: 3 Fid: 4222124650674799 FileHash: 0000000000000000 0000000000000000 FileName: GUID1 Flags: 1 GVsn: {55FDBBB9-0E2C-495C-8416-7CE2706D62C7}-v1468 Index: 729 ParentUid: {3AB160AD-E505-492F-9C3B-09382DDB0CCC}-v1 ReplicatedFolderGuid: 3AB160AD-E505-492F-9C3B-09382DDB0CCC Uid: {55FDBBB9-0E2C-495C-8416-7CE2706D62C7}-v1467 UpdateTime: 20060906155126.937500-000 Usn: 3460928 Volume: \\.\C:

And So The Journal Begins… NTFS Change Journal 1 GUID1 2 GUID2 3 GUID3 4 GUID4 NTFRS FileIDTable 1 GUID1 2 GUID2 3 GUID3 4 GUID4

Journal Wraps - Good NTFS Change Journal 5 GUID5 6 GUID6 7 GUID7 NTFRS FileIDTable 1 GUID1 2 GUID2 3 GUID3 4 GUID4

Journal Wraps - Good NTFS Change Journal NTFRS FileIDTable 1 GUID1 2 GUID2 3 GUID3 4 GUID4 5 GUID5 6 GUID6 7 GUID7 4 GUID4 5 GUID5 6 GUID6 7 GUID7 5 GUID5 6 GUID6 7 GUID7

This is a Journal Wrap Error Journal Wraps gone Bad NTFS Change Journal 5 GUID5 6 GUID6 7 GUID7 8 GUID8 NTFRS FileIDTable 1 GUID1 2 GUID2 3 GUID3 4 GUID4 FRS Database is lost! This is a Journal Wrap Error

Journal Wrap Error Likely Causes Fixes Turning off FRS for an extended period of time Disk intensive DCs Fixes Increase NTFS change journal size Self-Healing Non-Authoritative Restore

Non-Authoritative Restore “Flush everything in Sysvol and ask my upstream partner for its entire Sysvol” Stop ntfrs.exe HKLM\System\CCS\Services\ntfrs\Parameters\Backup/Restore\Process at Startup Modify existing REG_DWORD entry Burflags, set to hex D2 Start ntfrs.exe

Non-Authoritative Restores Automagically W2k or W2k SP1– only happens “manually” – when you set BurFlags=D2 W2k SP2 – occurs automatically whenever a journal wrap is detected, no Reg hacking required W2k SP3 – Back to manual-only but a Reg entry will make it happen automatically again HKLM\System\CCS\Services\ntfrs\Parameters “Enable journal wrap automatic restore” key to 1 2003/2008: MS says not to make automatic (KB 292438)

FRS keeps track of things via a database… Database lives in C:\Windows\ntfrs\jet\ntfrs.jdb Consists of 5 tables Connection Record table Version vector table File ID table (“IDtable”) Inbound log (“inlog”) Outbound log (“Ntfrs Outlog”) Viewed using Ntfrsutl

How FRS Handles New or Modified Files FRS classifies files as new, modified or deleted Handles new and modified files similarly FRS creates a Change Order (CO) in the “Inbound log” table FRS uses the Backup API to create a compressed copy of the file in a “staging area” folder FRS creates a Change Order (CO) in the “Outbound log” table FRS creates a new entry in the IDTable table… but only for new files

NTFS CJ FRS Database ntfrs.exe IDTable GPO created NTFS CJ GPO – (C) DC FRS Database ntfrs.exe In Log Out Log CO (C)GPO CO (C)GPO GPO IDTable GPO 123456748USN

Partner Notification Come and get it!!!! KCC AD Connection Objects & Site Links FRS polls AD at startup (and every 5 minutes after) to check for list of replication partners Ntfrsutl poll /now (forces polling) Ntfrsutl poll /now Computer (forces polling remotely) Intra-Site (immediate replication) Inter-Site (replication schedule)

GPO created DC 1 DC 2 NTFS CJ GPO - Create Change Notification FRS Database FRS Database In Log Out Log CO New GPO In Log Out Log CO New GPO CO New GPO CO New GPO Change Notification Psstt…I have changes GPO GPO Entire File is Copied!!

FRS Issues Relocating the FRS database & logs Relocating SYSVOL Authoritative Restore Morphed files/folders FRSDiag Ultrasound Sonar

Relocate FRS database & log files Stop FRS (net stop ntfrs) Copy ntfrs folder to new destination Modify Working Directory to reflect new path HKLM\SYSTEM\CCS\Services\ntfrs\Parameters Confirm Administrator/System accounts have full control to: ntfrs ntfrs\Jet ntfrs\Jet\Log ntfrs\Jet\Sys ntfrs\Jet\Temp Start FRS (net start ntfrs) Verify Inbound and Outbound replication with partners

Relocate Sysvol Confirm replication is working properly Dcpromo down (demoting) Wait for the removal of the DC from all DC’s (NTDS file system settings object deletion propagates, ADSS) Dcpromo back up (re-promoting) KB 842162 (manual – may the force be with you!)

Sysvol Authoritative Restore When would I do this? When many DC’s SYSVOL are inconsistent Multiple DCs have journal_wrap errors How do I do it? Stop FRS on ALL DCs Select one DC to be the reference machine (this copy of SYSVOL will be copied to all other DCs) On the reference machine copy all folders/files that should reside in SYSVOL to a temporary folder Restart FRS on the reference machine with the Burflag set to D4 HKLM\SYSTEM\CCS\Services\ntfrs\Parameters\Cumulative Replica Sets\GUID Restart all other DCs with the Burflag set to D2 On the reference machine copy files/folders from the temporary location into the root of SYSVOL Monitor that replication is consistent Detailed steps KB 315457

FRS debug logs C:\Windows\Debug ntfrs_0001.log - ntfrs_0005.log FRS Transactions & event details Settings range from 0-5 (5 highest – most information logged) Default setting is 2 Log detail controlled by HKLM\System\CCS\Services\ntfrs\Parameters Debug Log Severity Debug Log Files Can also be set using FRSDiag Must stop and start ntfrs.exe

FRS Conflicts! Morphed Files/Folders File/folder created on 2 different DC’s in the exact same folder with the same name prior to a replication cycle When replication occurs – the inbound file will be renamed Logon.vbs becomes Logon.vbs_ntfrs_0001ab39 How does this happen? Administrators are attempting to make data consistent with manual copies A server’s FRS was not stopped prior to an authoritative restore on another server During an authoritative restore ONE server did not have it’s BurFlag set to D2

Resolve Morphed files/folders Rename the original file/folder and the morphed file/folder to different names Wait for this to propagate to all servers After propagation – choose the file/folder you would like to keep and rename it back to the original name Delete all unwanted copies of the file/folder KB 328492

Tools FRSDiag Ultrasound Dump event logs Find members GUID numbers Requires a SQL database Installs WMI providers on each DC Polls DCs at defined intervals on FRS status and places information in a SQL database Configure Ultrasound to email or log an event in the application log whenever an error condition is detected

Sonar Sonar-d.htm installs with Sonar & is a great document on troubleshooting FRS Specific event ids which will help in resolving issues Find out which servers FRS service have been disabled or are not running SYSVOL not shared Staging area full Staging files being regenerated Burflags set NTFS change journal size Morphed folders/files

Replicating Sysvol via DFS-R All DC’s must be running Server 2008 DFS-Replication can be managed from: XP-Pro SP2 workstation Server 2008 Vista

DFS-Replication Terminology Replication group - A set of servers, called members, that participate in replicating one or more folders. Replicated folder - sysvol Connection topology - Which members replicate with other members. Schedule - When replication is available. Upstream partner - The partner who sends the notification that it has changes for a replicating partner. Downstream partner - The partner who received the notification from an upstream partner and initiates replication.

DFSR and Journal Wraps DFSR also monitors the NTFS change journal What’s different? DFSR always heals itself And here is how…

Self-Healing Journal Wraps DFS-R stops processing changes from the NTFS change journal Replication Stops All entries in the DFS-R FileIDTable receive a JWED (Journal Wrapped) flag The Directory Walker thread (DIRW) compares all Update Sequence Numbers (USNs) between the NTFS change journal and the DFS-R database

Upon Comparison - 3 Possible Conditions If the USNs in the NTFS change journal and the DFS-R Database are the same: DFS-R has the latest changes and removes the JWED flag If the USNs are NOT the same: The USN in the NTFS change journal is incremented The JWED flag is cleared from the DFS-R Database The file will be replicated normally Once all files in the DFS-R Database have been compared to the NTFS change journal and their JWED flags are cleared One last scan is performed to find any files still flagged JWED If a file was deleted from the NTFS change journal, but still exists in the DFS-R database. The files are tombstoned and will eventually be deleted from the replication process.

Do I have all the changes? Version Vectors (VVs) are compared Houston Dallas Server VV Houston 20 Dallas 30 Server VV Dallas 31 Houston 20

RDC in Action! Dallas DC Houston DC Data (16 Bytes) MD4 Row Row Row 27 your house 42 gently dow 17 Data (16 Bytes) MD4 Row Row Row 27 your house 42 gently dow 17

RDC in Action! Dallas DC Houston DC Data (16 Bytes) MD4 Row Row Row 27 your boat g 35 ently down 22 Data (16 Bytes) MD4 Row Row Row 27 your house 42 gently dow 17

What if there is a Conflict? What causes a conflict? The same file/folder (same UID) being modified on two different servers, within one replication cycle A file/folder is created on two different servers in the same folder with the same name (different UID), within one replication cycle

File and Folder Conflicts Resolved File with the Same Name & UID Last writer wins (based on time stamps) File with the Same Name but different UID First created wins (based on time stamps) Folder with the Same Name and UID Folder with the Same Name but different UID The contents are consolidated

Performance Monitor DFS Replicated Folders DFS Replication Connections Bandwidth Savings Number of conflicts that have occurred In bytes, or number of files/folders RDC information Staging Files DFS Replication Connections Number and size of files replicated DFS Replication Service Volume Number of entries read and accepted from the NTFS change journal (USN Journal)

Why Use DFSR vs FRS? FRS silently fails if the volume SYSVOL resides on < 1GB of free space Copies changes (RDC) not entire files Version Vector tables Journal Wraps are self-healing Morphed files and folders automagically taken care of

Migrating Sysvol Replication to DFS-R Pre-Migration Migration Demo PDC EM - Server 2003 that’s been upgraded to 2008 Or a 2008 / 2008-R2 Server Domain mode = Server 2008

Pre-Migration On 2008 Server (PDC) Pop in the Server 2008 DVD Adprep /forestprep from Sources/Adprep folder Raise domain functional level to Server 2003 Active Directory Domains and Trusts From Server 2008 DVD Adprep /domainprep Upgrade PDC & all DCs to Server 2008 Raise domain Functional level to Server 2008 Backup SYSVOL

Migration Process has 2 Types of States Stable states Processes are complete Can Rollback to a point Transitional states Processes are in a working state Stable State T

Migration Process 4 5 6 7 START ( State ) PREPARED 1 RE - DIRECTED 2 ) PREPARED 1 RE - DIRECTED 2 ELIMINATED 3 4 5 6 7

Meet DfsrMig.exe DfsrMig.exe lives in Windows\System32 on Server 2008 Run dfsrmig from the PDC Emulator State 0 - All DCs begin here Replicating SYSVOL via NTFRS State 1 “Prepared” Dfsrmig /SetGlobalState 1 To confirm a state has been set… Dfsrmig /GetGlobalState

What Happens… New Windows\SYSVOL_DFSR on all DC’s Contents of Windows\SYSVOL copied to new SYSVOL_DFSR folder Windows\SYSVOL_DFSR domain & sysvol folders Netlogon share still points to Windows\SYSVOL\sysvol\Bigfirm.Com\Scripts SYSVOL share still points to Windows\SYSVOL\sysvol Both SYSVOL and SYSVOL_DFSR are being replicated

ADUC Advanced View System DFSR-GlobalSettings is created Domain System Volume (Replication Group) DFSR-Replication Group DFSR-Content – SYSVOL Share DFSR-Topology – List DCs

Re-Directed State… Dfsrmig /SetGlobalState 2 Dfsrmig /GetGlobalState UptownDC DownTownDC SYSVOL_DFSR SYSVOL_DFSR DFS-R SYSVOL SYSVOL FRS Dfsrmig /GetGlobalState Current DFSR global state: Redirected

SYSVOL_DFSR & SYSVOL out of sync… The original copy of SYSVOL to SYSVOL_DFSR was performed by Robocopy This copy is only done once If you need to run it again you’ll have to do it manually

Copying SYSVOL ROBOCOPY c:\Windows\Sysvol\Domain c:\Windows\Sysvol_DFSR\Domain /Copyall /MIR /B /R:0 /XD “Do_Not_Remove_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_Prexisting__See_Eventlog” “ NTFRS_CMD_FILE_MOVE_ROOT” /XF “DO_NOT_REMOVE_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_PreExisting__See_Eventlog” “NTFRS_CMD_FILE_MOVE_ROOT” Windows\Debug\SYSVOL_DFSR-Robocopy.txt

Re-Directed State Redirects the SYSVOL share to the new SYSVOL_DFSR folder HKLM\Sys\CCS\Services\Netlogon\Params Sets SysvolReady to False Sets Sysvol = C:\WINDOWS\SYSVOL_DFSR\sysvol Sets SysvolReady to True

Migration Process X 6 4 5 7 START PREPARED RE - DIRECTED ELIMINATED ( State ) ( State 1 ) ( State 2 ) ( State 3 ) 4 5 7

The Eliminated State Dfsrmig /SetGlobalState 3 Deletes the NTFRS replica set from AD Deletes the old SYSVOL folder Leaves NTFRS on There is no going back!

Health Report

Create a Diagnostic Report Health Report Propagation Test __DFSR_DIAGNOSTICS_TEST_FOLDER__ FRSRIP@A13948E4-5E44-483A-B56B-65A075C446C0@Domain System Volume-SYSVOL Share.xml Propagation Report

Create a Diagnostic Report using The following example will generate a health report that: Gathers information ALL servers in the sysvol replication group. Houston is the Reference Member. The report will be named HealthReport.html stored in the C:\Reports folder of the local machine. DfsrAdmin Health New /RgName: ”Domain System Volume” /RefMemName:Bigfirm\Houston /RepName:C:\Reports\HealthReport.html /FsCount:true

More Diagnostic Reports The following example will generate a health report that: Gathers information only from Dallas Houston is the Reference Member for the sysvol replication group. The report will be named DallasHealth.html stored in the C:\Reports folder of the local machine. DfsrAdmin Health New /RgName:”Domain System Volume” /MemName:Bigfirm\Dallas /RefMemName:Houston /RepName:C:\Reports\DallasHealth.html /FsCount:true

The last Diagnostic Report… I promise The following example will generate a health report that: Gathers information from all Servers listed in the C:\Servers.txt file for the sysvol replication group. Houston is the Reference Member. The report will be named ServersHealth.html stored in the C:\Reports folder of the local machine. DfsrAdmin Health New /RgName:”Domain System Volume” /MemberListFile:C:\Servers.txt /RefMemName:Bigfirm\Houston /RepName:C:\Reports\ServersHealth.html /FsCount:true

In Summary FRS – RIP DFS-R Rocks! DFSRMig Works!

Hope you Enjoyed This Session Please Don’t forget to fill out your evaluations Rhonda@Minasi.Com Thank You!