1. SYSVOL Replication: FRS or DFS-R???
Rhonda Layfield
Contents Copyright Rhonda J. Layfield 2009

2. Rhonda Layfield IT industry 25+ years
NT/2000/2003 MCSE, MCT, MCSE+Security
Contribute articles to Redmond and Windows IT Pro magazines
Setup and Deployment MVP
Desktop Deployment Product Specialist (DDPS)
Co-Author of 2 Mastering Windows Server books (2003 & 2008)

3. SYSVOL
LScript.vbs
LScript.vbs
Uptown DC
Downtown DC

4. Prehistory to Today
Always been a need for a tool to keep two or more server’s folders in sync
NT -> LMRepl (Directory Replication)
2000 introduced FRS (2003 used this also)
Sysvol
DFS
2003 R2: a new Replication engine
Sysvol uses “old” FRS
R2’s DFS – which is called “DFS Namespace” – uses the “new” Replication engine called “DFS Replication” or “DFS-R”
Server 2008 can be configured to use DFS-R for SYSVOL replication

5. Sysvol Created during dcpromo on domain controllers
Automatically shared
C:\Windows\SYSVOL\sysvol\Bigfirm.com (DNSDomainName)
Group Policy Template (Settings)
System Policies
Scripts (NT & AD)
Must reside on an NTFS volume

6. FRS Sysvol Terminology
DC 1
Upstream Partner
Direct Replication
Transitive Replication
DC 2
DC 3
Downstream
Partner
Upstream Partner
Downstream
Partner

7. FRS Process (10,000 foot view)
NTFS logs changes to the “NTFS Change Journal”
FRS monitors NTFS Change Journal
FRS places changed files in staging area
Replication partners notified “something’s changed”
Partners request replication
Files are transferred

8. NTFS Change Journal (aka USN Journal)
What does it do?
Logs all changes to an NTFS volume
Separate log on each NTFS volume
Doesn’t it take a lot of space?
Sure; in fact it would eventually fill up a drive
So Microsoft limits its size

9. So how large is it, and can I change it?
W2K SP2 32 MB
W2K SP3 512 MB
2003 pre-SP1 128 MB
2003 Hotfix or SP1/R2 512 MB
Server MB
Maximum size is 2 TB
MS Recommends increasing by 128 MB for every 100,000 files/folders

10. What happens when the change journal fills up?
Then NTFS just goes back up to the top and starts overwriting the oldest entries
So… if FRS hasn’t checked in with the change journal in a while, then FRS may get lost
This is called a “journal_wrap” Ooh No!

11. FRS monitors NTFS Change Journal
1 GUID1
2 GUID2
3 GUID3
4 GUID4
NTFRS FileIDTable
1 GUID1
2 GUID2
3 GUID3
4 GUID4
Received 1-4

12. FRS monitors NTFS Change Journal
1 GUID1
FileRef#: 0x000f a6f
USN: 0x cf40
NTFRS FileIDTable
GUID1
FID
USN

13. Tying the Change journal and FRS database together
FileRef# & USN in the NTFS Change Journal – NTFS Utility: FSUtil
FSUtil USN ReadData C:\WINDOWS\SYSVOL\sysvol\Domain\Policies\GUID1
Major Version: 0x2
Minor Version: 0x0
FileRef#: 0x000f a6f
Parent FileRef#: 0x f45
Usn: 0x cf40
Time Stamp: 0x :00:00 AM 1/1/1601
Reason: 0x0
Source Info: 0x0
Security Id: 0x308
File Attributes: 0x20
File Name Length: 0xe
File Name Offset: 0x3c
FileName: GUID1

14. Dump GUID1 from FRS FileIDTable
wmic /namespace:\\root\microsoftdfs path dfsridrecordinfo where filename=“GUID1"
Attributes: 32
Clock:
CreateTime:
Fence: 3
Fid:
FileHash:
FileName: GUID1
Flags: 1
GVsn: {55FDBBB9-0E2C-495C CE2706D62C7}-v1468
Index: 729
ParentUid: {3AB160AD-E F-9C3B-09382DDB0CCC}-v1
ReplicatedFolderGuid: 3AB160AD-E F-9C3B-09382DDB0CCC
Uid: {55FDBBB9-0E2C-495C CE2706D62C7}-v1467
UpdateTime:
Usn:
Volume: \\.\C:

15. And So The Journal Begins…
NTFS Change Journal
1 GUID1
2 GUID2
3 GUID3
4 GUID4
NTFRS FileIDTable
1 GUID1
2 GUID2
3 GUID3
4 GUID4

16. Journal Wraps - Good NTFS Change Journal 5 GUID5 6 GUID6 7 GUID7
NTFRS FileIDTable
1 GUID1
2 GUID2
3 GUID3
4 GUID4

17. Journal Wraps - Good NTFS Change Journal
NTFRS FileIDTable
1 GUID1 2 GUID2 3 GUID3 4 GUID4
5 GUID5
6 GUID6
7 GUID7
4 GUID4
5 GUID5
6 GUID6
7 GUID7
5 GUID5 6 GUID6 7 GUID7

18. This is a Journal Wrap Error
Journal Wraps gone Bad
NTFS Change Journal
5 GUID5
6 GUID6
7 GUID7
8 GUID8
NTFRS FileIDTable
1 GUID1
2 GUID2
3 GUID3
4 GUID4
FRS Database is lost!
This is a Journal Wrap Error

19. Journal Wrap Error Likely Causes Fixes
Turning off FRS for an extended period of time
Disk intensive DCs
Fixes
Increase NTFS change journal size
Self-Healing
Non-Authoritative Restore

20. Non-Authoritative Restore
“Flush everything in Sysvol and ask my upstream partner for its entire Sysvol”
Stop ntfrs.exe
HKLM\System\CCS\Services\ntfrs\Parameters\Backup/Restore\Process at Startup
Modify existing REG_DWORD entry Burflags, set to hex D2
Start ntfrs.exe

21. Non-Authoritative Restores Automagically
W2k or W2k SP1– only happens “manually” – when you set BurFlags=D2
W2k SP2 – occurs automatically whenever a journal wrap is detected, no Reg hacking required
W2k SP3 – Back to manual-only but a Reg entry will make it happen automatically again
HKLM\System\CCS\Services\ntfrs\Parameters
“Enable journal wrap automatic restore” key to 1
2003/2008: MS says not to make automatic (KB )

22. FRS keeps track of things via a database…
Database lives in C:\Windows\ntfrs\jet\ntfrs.jdb
Consists of 5 tables
Connection Record table
Version vector table
File ID table (“IDtable”)
Inbound log (“inlog”)
Outbound log (“Ntfrs Outlog”)
Viewed using Ntfrsutl

23. How FRS Handles New or Modified Files
FRS classifies files as new, modified or deleted
Handles new and modified files similarly
FRS creates a Change Order (CO) in the “Inbound log” table
FRS uses the Backup API to create a compressed copy of the file in a “staging area” folder
FRS creates a Change Order (CO) in the “Outbound log” table
FRS creates a new entry in the IDTable table… but only for new files

24. NTFS CJ FRS Database ntfrs.exe IDTable
GPO created
NTFS CJ
GPO – (C) DC
FRS Database ntfrs.exe
In Log
Out Log
CO (C)GPO
CO (C)GPO
GPO
IDTable
GPO USN

25. Partner Notification Come and get it!!!!
KCC AD Connection Objects & Site Links
FRS polls AD at startup (and every 5 minutes after) to check for list of replication partners
Ntfrsutl poll /now (forces polling)
Ntfrsutl poll /now Computer (forces polling remotely)
Intra-Site (immediate replication)
Inter-Site (replication schedule)

26. GPO created DC 1 DC 2 NTFS CJ GPO - Create Change Notification
FRS Database
FRS Database
In Log
Out Log
CO New GPO
In Log
Out Log
CO New GPO
CO New GPO
CO New GPO
Change Notification
Psstt…I have changes
GPO
GPO
Entire File is Copied!!

27. FRS Issues Relocating the FRS database & logs Relocating SYSVOL
Authoritative Restore
Morphed files/folders
FRSDiag
Ultrasound
Sonar

28. Relocate FRS database & log files
Stop FRS (net stop ntfrs)
Copy ntfrs folder to new destination
Modify Working Directory to reflect new path
HKLM\SYSTEM\CCS\Services\ntfrs\Parameters
Confirm Administrator/System accounts have full control to:
ntfrs
ntfrs\Jet
ntfrs\Jet\Log
ntfrs\Jet\Sys
ntfrs\Jet\Temp
Start FRS (net start ntfrs)
Verify Inbound and Outbound replication with partners

29. Relocate Sysvol Confirm replication is working properly
Dcpromo down (demoting)
Wait for the removal of the DC from all DC’s (NTDS file system settings object deletion propagates, ADSS)
Dcpromo back up (re-promoting)
KB (manual – may the force be with you!)

30. Sysvol Authoritative Restore
When would I do this?
When many DC’s SYSVOL are inconsistent
Multiple DCs have journal_wrap errors
How do I do it?
Stop FRS on ALL DCs
Select one DC to be the reference machine (this copy of SYSVOL will be copied to all other DCs)
On the reference machine copy all folders/files that should reside in SYSVOL to a temporary folder
Restart FRS on the reference machine with the Burflag set to D4 HKLM\SYSTEM\CCS\Services\ntfrs\Parameters\Cumulative Replica Sets\GUID
Restart all other DCs with the Burflag set to D2
On the reference machine copy files/folders from the temporary location into the root of SYSVOL
Monitor that replication is consistent
Detailed steps KB

31. FRS debug logs C:\Windows\Debug ntfrs_0001.log - ntfrs_0005.log
FRS Transactions & event details
Settings range from 0-5 (5 highest – most information logged)
Default setting is 2
Log detail controlled by HKLM\System\CCS\Services\ntfrs\Parameters
Debug Log Severity
Debug Log Files
Can also be set using FRSDiag
Must stop and start ntfrs.exe

32. FRS Conflicts! Morphed Files/Folders
File/folder created on 2 different DC’s in the exact same folder with the same name prior to a replication cycle
When replication occurs – the inbound file will be renamed
Logon.vbs becomes Logon.vbs_ntfrs_0001ab39
How does this happen?
Administrators are attempting to make data consistent with manual copies
A server’s FRS was not stopped prior to an authoritative restore on another server
During an authoritative restore ONE server did not have it’s BurFlag set to D2

33. Resolve Morphed files/folders
Rename the original file/folder and the morphed file/folder to different names
Wait for this to propagate to all servers
After propagation – choose the file/folder you would like to keep and rename it back to the original name
Delete all unwanted copies of the file/folder KB

34. Tools FRSDiag Ultrasound Dump event logs Find members GUID numbers
Requires a SQL database
Installs WMI providers on each DC
Polls DCs at defined intervals on FRS status and places information in a SQL database
Configure Ultrasound to or log an event in the application log whenever an error condition is detected

35. Sonar
Sonar-d.htm
installs with Sonar & is a great document on troubleshooting FRS
Specific event ids which will help in resolving issues
Find out which servers FRS service have been disabled or are not running
SYSVOL not shared
Staging area full
Staging files being regenerated
Burflags set
NTFS change journal size
Morphed folders/files

36. Replicating Sysvol via DFS-R
All DC’s must be running Server 2008
DFS-Replication can be managed from:
XP-Pro SP2 workstation
Server 2008
Vista

37. DFS-Replication Terminology
Replication group - A set of servers, called members, that participate in replicating one or more folders.
Replicated folder - sysvol
Connection topology - Which members replicate with other members.
Schedule - When replication is available.
Upstream partner - The partner who sends the notification that it has changes for a replicating partner.
Downstream partner - The partner who received the notification from an upstream partner and initiates replication.

38. DFSR and Journal Wraps DFSR also monitors the NTFS change journal
What’s different? DFSR always heals itself
And here is how…

39. Self-Healing Journal Wraps
DFS-R stops processing changes from the NTFS change journal
Replication Stops
All entries in the DFS-R FileIDTable receive a JWED (Journal Wrapped) flag
The Directory Walker thread (DIRW) compares all Update Sequence Numbers (USNs) between the NTFS change journal and the DFS-R database

40. Upon Comparison - 3 Possible Conditions
If the USNs in the NTFS change journal and the DFS-R Database are the same:
DFS-R has the latest changes and removes the JWED flag
If the USNs are NOT the same:
The USN in the NTFS change journal is incremented
The JWED flag is cleared from the DFS-R Database
The file will be replicated normally
Once all files in the DFS-R Database have been compared to the NTFS change journal and their JWED flags are cleared
One last scan is performed to find any files still flagged JWED
If a file was deleted from the NTFS change journal, but still exists in the DFS-R database. The files are tombstoned and will eventually be deleted from the replication process.

41. Do I have all the changes?
Version Vectors (VVs) are compared
Houston
Dallas
Server VV
Houston 20
Dallas 30
Server VV
Dallas 31
Houston 20

42. RDC in Action! Dallas DC Houston DC Data (16 Bytes) MD4 Row Row Row 27
your house 42
gently dow 17
Data (16 Bytes)
MD4
Row Row Row 27
your house 42
gently dow 17

43. RDC in Action! Dallas DC Houston DC Data (16 Bytes) MD4 Row Row Row 27
your boat g 35
ently down 22
Data (16 Bytes)
MD4
Row Row Row 27
your house 42
gently dow 17

44. What if there is a Conflict?
What causes a conflict?
The same file/folder (same UID) being modified on two different servers, within one replication cycle
A file/folder is created on two different servers in the same folder with the same name (different UID), within one replication cycle

45. File and Folder Conflicts Resolved
File with the Same Name & UID
Last writer wins (based on time stamps)
File with the Same Name but different UID
First created wins (based on time stamps)
Folder with the Same Name and UID
Folder with the Same Name but different UID
The contents are consolidated

46. Performance Monitor DFS Replicated Folders DFS Replication Connections
Bandwidth Savings
Number of conflicts that have occurred
In bytes, or number of files/folders
RDC information
Staging Files
DFS Replication Connections
Number and size of files replicated
DFS Replication Service Volume
Number of entries read and accepted from the NTFS change journal (USN Journal)

47. Why Use DFSR vs FRS?
FRS silently fails if the volume SYSVOL resides on < 1GB of free space
Copies changes (RDC) not entire files
Version Vector tables
Journal Wraps are self-healing
Morphed files and folders automagically taken care of

48. Migrating Sysvol Replication to DFS-R
Pre-Migration
Migration
Demo
PDC EM - Server 2003 that’s been upgraded to 2008
Or a 2008 / 2008-R2 Server
Domain mode = Server 2008

49. Pre-Migration On 2008 Server (PDC) Pop in the Server 2008 DVD
Adprep /forestprep from Sources/Adprep folder
Raise domain functional level to Server 2003
Active Directory Domains and Trusts
From Server 2008 DVD
Adprep /domainprep
Upgrade PDC & all DCs to Server 2008
Raise domain Functional level to Server 2008
Backup SYSVOL

50. Migration Process has 2 Types of States
Stable states
Processes are complete
Can Rollback to a point
Transitional states
Processes are in a working state
Stable State T

51. Migration Process 4 5 6 7 START ( State ) PREPARED 1 RE - DIRECTED 2)
PREPARED 1 RE -
DIRECTED 2
ELIMINATED 3 4 5 6 7

52. Meet DfsrMig.exe DfsrMig.exe lives in Windows\System32 on Server 2008
Run dfsrmig from the PDC Emulator
State 0 - All DCs begin here
Replicating SYSVOL via NTFRS
State 1 “Prepared”
Dfsrmig /SetGlobalState 1
To confirm a state has been set…
Dfsrmig /GetGlobalState

53. What Happens… New Windows\SYSVOL_DFSR on all DC’s
Contents of Windows\SYSVOL copied to new SYSVOL_DFSR folder
Windows\SYSVOL_DFSR
domain & sysvol folders
Netlogon share still points to Windows\SYSVOL\sysvol\Bigfirm.Com\Scripts
SYSVOL share still points to Windows\SYSVOL\sysvol
Both SYSVOL and SYSVOL_DFSR are being replicated

54. ADUC Advanced View System DFSR-GlobalSettings is created
Domain System Volume (Replication Group)
DFSR-Replication Group
DFSR-Content – SYSVOL Share
DFSR-Topology – List DCs

55. Re-Directed State… Dfsrmig /SetGlobalState 2 Dfsrmig /GetGlobalState
UptownDC
DownTownDC
SYSVOL_DFSR
SYSVOL_DFSR
DFS-R
SYSVOL
SYSVOL
FRS
Dfsrmig /GetGlobalState
Current DFSR global state: Redirected

56. SYSVOL_DFSR & SYSVOL out of sync…
The original copy of SYSVOL to SYSVOL_DFSR was performed by Robocopy
This copy is only done once
If you need to run it again you’ll have to do it manually

57. Copying SYSVOL
ROBOCOPY c:\Windows\Sysvol\Domain c:\Windows\Sysvol_DFSR\Domain /Copyall /MIR /B /R:0 /XD “Do_Not_Remove_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_Prexisting__See_Eventlog” “ NTFRS_CMD_FILE_MOVE_ROOT” /XF “DO_NOT_REMOVE_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_PreExisting__See_Eventlog” “NTFRS_CMD_FILE_MOVE_ROOT”
Windows\Debug\SYSVOL_DFSR-Robocopy.txt

58. Re-Directed State
Redirects the SYSVOL share to the new SYSVOL_DFSR folder
HKLM\Sys\CCS\Services\Netlogon\Params
Sets SysvolReady to False
Sets Sysvol = C:\WINDOWS\SYSVOL_DFSR\sysvol
Sets SysvolReady to True

59. Migration Process X 6 4 5 7 START PREPARED RE - DIRECTED ELIMINATED (
State ) (
State 1 ) (
State 2 ) (
State 3 ) 4 5 7

60. The Eliminated State Dfsrmig /SetGlobalState 3
Deletes the NTFRS replica set from AD
Deletes the old SYSVOL folder
Leaves NTFRS on
There is no going back!

61. Health Report

62. Create a Diagnostic Report
Health Report
Propagation Test
__DFSR_DIAGNOSTICS_TEST_FOLDER__
System Volume-SYSVOL Share.xml
Propagation Report

63. Create a Diagnostic Report using
The following example will generate a health report that:
Gathers information ALL servers in the sysvol replication group.
Houston is the Reference Member.
The report will be named HealthReport.html stored in the C:\Reports folder of the local machine.
DfsrAdmin Health New /RgName: ”Domain System Volume” /RefMemName:Bigfirm\Houston /RepName:C:\Reports\HealthReport.html /FsCount:true

64. More Diagnostic Reports
The following example will generate a health report that:
Gathers information only from Dallas
Houston is the Reference Member for the sysvol replication group.
The report will be named DallasHealth.html stored in the C:\Reports folder of the local machine.
DfsrAdmin Health New /RgName:”Domain System Volume” /MemName:Bigfirm\Dallas /RefMemName:Houston /RepName:C:\Reports\DallasHealth.html /FsCount:true

65. The last Diagnostic Report… I promise
The following example will generate a health report that:
Gathers information from all Servers listed in the C:\Servers.txt file for the sysvol replication group.
Houston is the Reference Member.
The report will be named ServersHealth.html stored in the C:\Reports folder of the local machine.
DfsrAdmin Health New /RgName:”Domain System Volume” /MemberListFile:C:\Servers.txt /RefMemName:Bigfirm\Houston /RepName:C:\Reports\ServersHealth.html /FsCount:true

66. In Summary
FRS – RIP
DFS-R Rocks!
DFSRMig Works!

67. Hope you Enjoyed This Session
Please Don’t forget to fill out your evaluations
Thank You!