1. Boon and bane: Exchange ActiveSync devices and Calendar items

2. Thank you Sponsors #ThriveITconf
Silver Evening event sponsor Material Media
During the Conference closing there will be a short Presentation of Sponsors and the Prize Game.
Please do not leave too early, you might be a lucky winner.

3. About me: Ingo Gegenwarth IT Principal Consultant @SAP
MCM Exchange 2010
Office Server and Services MVP
Blog:
Twitter:
@IngoGegenwarth

4. Agenda Exchange Client Access architecture IIS logs
ActiveSyncDebugLogging & MailboxLogParser
Get-DatabaseEvent
MessageTracking
Get-calendarDiagnosticobjects
EWS for calendar items
Q&A

5. Why would you troubleshoot EAS devices?
Mobile devices are very in common and cloud services helps being productive.
But there is also a downside as these devices can cause a variety of issues, which are frustrating for end-users as well as for administrators:
Account lockouts
Calendar issues (Cancelations, date/time modification….)
Notifications sent-onbehalf of the organizer …

6. Exchange Client Access architecture

7. Exchange architecture
Client Access Server/Service is a thin, stateless front end machine that provides a unified namespace, authentication, and network security as well as proxy and redirection logic. Transport is provided by the Front End Transport service which provides mailbox locator services.
Houses the logic to proxy or redirect a specific protocol request from a client to the correct Mailbox server
Is designed to work with TCP affinity—does not require application session affinity
Provides an SMTP Front End proxy and a UM call router
Handles all inbound and outbound external SMTP traffic via Front End Transport Service and provides a
client endpoint for SMTP Traffic

8. Exchange architecture
OWA
EAS
EAC
Outlook
PowerShell
IMAP
SMTP
Telephony
SIP + RTP
Load balancer
Redirect
HTTP Proxy
IIS
POP
IMAP
SMTP UM
MBX16
HTTP
POP IMAP
SMTP
IIS
POP IMAP
Transport UM
RPS
RpcProxy
OWA, EAS, EWS, ECP, OAB, MAPI
MDB
MailQ
RPC CA

9. IIS logs

10. IIS logs
Since Exchange 2013 you will find two directories of IIS logs:
Front End (Client Access) related logs:
C:\inetpub\logs\LogFiles\W3SVC1
Back End (Mailbox) related logs:
C:\inetpub\logs\LogFiles\W3SVC2

11. IIS logs
The Client Access server/services are responsible for authentication and routing to correct mailbox server
Good tracing for:
Authentication issues
Incoming requests statistics
Contains ONLY issued EAS Cmd
Mailbox server logs all details related to Exchange ActiveSync protocol

12. Example
Field
Value
User
Ingo
DeviceID
UBDKD3QGCL3E7BHS4JJATJVFGA
DeviceType
iPhone
Cmd
Sync
Ver1
160
HH(requested host)
mail.fabrikam.com
SmtpAdrs
Ty (type)
Em ( )
Filter1 (time range)
3 (1 week)
DevOS
iOS E304 As
Allowedl
User=ingo&DeviceId=UBDKD3QGCL3E7BHS4JJATJVF GA&DeviceType=iPhone&Cmd=Sync&Log=PrxFrom: _Ver1:160_HH:mail.fabrikam.com_SmtpA drs:ingo.gegenwarth%40fabrikam.com_FldrC1:1_Fid :12_Ty:Em_Filter1:3_St:S_Sk: _Sks: _SSKb1:15_SsCmt:15_TotSvC:1_ColdSv C:1_TotLdC:1_MR:0_PSyncType1:ICS_GetChgsIter:1 _GetChgsTime:3_Pfs:1_BR:1_BPR:0_Ers:1_Fet:86_ Pk: _DevOS:iOS E304_SC1: 1_As:AllowedI_Mbx:fabex01.fabrikam.local_Cafe:fab ex02.fabrikam.local_Dc:fabdc0014.fabrikam.local_T hrottle:0…

13. Example As an example you can decode and split the log entry.
This makes it easier to read and extract data from.
PowerShell is fast and almost always available.
$URI='User=ingo&DeviceId=UBDKD3QGCL3E7BHS4JJATJVFGA&DeviceType=iPhone&Cmd=Sync&Log=PrxFrom: _Ver1:160_HH:mail.fabrikam.com_SmtpAdrs:ingo.gegenwarth%40fabrikam.com_FldrC1:1_Fid:12_Ty:Em_Filter1:3_St:S_Sk: _Sks: _SSKb1:15_SsCmt:15_TotSvC:1_ColdSvC:1_TotLdC:1_MR:0_PSyncType1:ICS_GetChgsIter:1_GetChgsTime:3_Pfs:1_BR:1_BPR:0_Ers:1_Fet:86_Pk: _DevOS:iOS E304_SC1:1_As:AllowedI_Mbx:fabex01.fabrikam.local_Cafe:fabex02.fabrikam.local_Dc:fabdc0014.fabrikam.local_Throttle:0_SBkOffD:BBkOff%3aL%2f-469%2c+ABBkOff%3aL%2f-480%2c+EffBkOff%3aL%2f-469_CmdHash1: _SyncHash1: _TmRcv:08:08: _TmSt:08:08: _TmDASt:08:08: _TmPolSt:08:08: _TmExSt:08:08: _TmExFin:08:08: _TmFin:08:08: _TmCmpl:08:08: _IcsHier:T_PersId:0_OMSt:3_Budget:(A)Owner%3aS %5FUBDKD3QGCL3E7BHS4JJATJVLQO%5FiPhone%2cConn%3a0%2cMaxConn%3a10%2cMaxBurst%3a480000%2cBalance%3a480000%2cCutoff%3a600000%2cRechargeRate%3a %2cPolicy%3aDefaultThrottlingPolicy%5F4f b-f2fcb8a9086c%2cIsServiceAccount%3aFalse%2cLiveTime%3a00%3a00%3a00%3b(D)Owner%3aS %5FUBDKD3QGCL3E7BHS4JJATJVLQO%5FiPhone%2cConn%3a0%2cMaxConn%3a10%2cMaxBurst%3a480000%2cBalance%3a %2cCutoff%3a600000%2cRechargeRate%3a %2cPolicy%3aDefaultThrottlingPolicy%5F4f b-f2fcb8a9086c%2cIsServiceAccount%3aFalse%2cLiveTime%3a00%3a00%3a _ActivityContextData:ActivityID%3d8e38d388-d89f-4e13-bbd4-e8f3e27007e2%3bI32%3aADS.C%5bfabdc0014%5d%3d1%3bF%3aADS.AL%5bfabdc0014%5d%3d1.9433%3bI32%3aADR.C%5bfabdc0014%5d%3d2%3bF%3aADR.AL%5bfabdc0014%5d%3d1.1758%3bDbl%3aMBLB.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d19891%3bI32%3aRPCSVR.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d31%3bF%3aRPCSVR.AL%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d1%3bI32%3aATE.C%5bfabdc0014.fabrikam.local%5d%3d3%3bF%3aATE.AL%5bfabdc0014.fabrikam.local%5d%3d0%3bI32%3aMB.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d31%3bF%3aMB.AL%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d %3bDbl%3aSTCPU.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d15%3bDbl%3aST.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d5%3bDbl%3aBudgUse.T%5b%5d%3d %3bI32%3aADS.C%5bfabdc0016%5d%3d1%3bF%3aADS.AL%5bfabdc0016%5d%3d1.63%3bI32%3aATE.C%5bfabdc0016.fabrikam.local%5d%3d1%3bF%3aATE.AL%5bfabdc0016.fabrikam.local%5d%3d0%3bI32%3aMAPI.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d78%3bI32%3aROP.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d %3bDbl%3aEXR.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d7%3bDbl%3aRPC.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d29%3bI32%3aRPC.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d31%3bI32%3aRPCDB.C%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d31%3bF%3aRPCDB.AL%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d1%3bDbl%3aMAPI.T%5bfabex01.9fe0117e-bbf3-426f-a2bb-3dacaa233e0d%5d%3d29%3bS%3aWLM.BT%3dEas%3bS%3aWLM.Bal%3d _'
Add-Type -AssemblyName system.web
$Decoded=[System.Web.HttpUtility]::UrlDecode($URI)
$Decoded.Split("_")

14. Field
Description
PrxTo
ProxyingTo
PrxFrom
ProxyingFrom
RdirTo
RedirectTo
PrxUser
ProxyUser
Ver
ProtocolVersion HH
Host
SmtpAdrs
UserSmtpAddress
Fid
FolderId Ty
FolderDataType
Filter
FilterType
FiltSms
SmsFilterType St
SyncType Sk
ClientSyncKey
Sks
ServerSyncKey
PSyncType
ProviderSyncType
FstSyncTime
Sync0Time
GetChgsIter
GetChangesIterations
GetChgsTime
GetChangesTime
Cli "{0}a{1}c{2}d{3}f{4}e{5}s{6}fs"
PerFolderClientAdds; PerFolderClientChanges; PerFolderClientDeletes; PerFolderClientFetches; PerFolderClientFailedToConvert; PerFolderClientSends; PerFolderClientFailedToSend
Srv "{0}a{1}c{2}d{3}s{4}e{5}r{6}A{7}sd"
PerFolderServerAdds; PerFolderServerChanges; PerFolderServerDeletes; PerFolderServerSoftDeletes; PerFolderServerFailedToConvert; PerFolderServerChangeTrackingRejected; PerFolderServerAssociatedAdds; PerFolderSkippedDeletes BR
BodyRequested
BPR
BodyPartRequested E
NumErrors Io
NumItemsOpened Hb
HeartBeatInterval
Rto
RequestTimedOut …
IIS Logs
Exchange logs a vast amount of data in the IIS logs (cs-uri-query).
This table shows only a subset, but for analysis the most important fields.

15. MailboxLogParser

16. MailboxLogParser
MailboxLogparser is a tool, which helps you analyzing Exchange ActiveSync mailbox logs. It presents the logs in a human readable format and provides capabilities to search these logs and/or export the Grid into a CSV file for further examination. Download:

17. MailboxLogParser
#Enable logging for a mailbox Set-CASMailbox < alias > -ActiveSyncDebugLogging:$true #Check if logging is enabled Get-CASMailbox < alias > -ActiveSyncDebugLogging | Format-List *debug* Note: Don’t forget the parameter ActiveSyncDebugLogging!

18. MailboxLogParser
#Retrieve MailboxLog Get-MobileDeviceStatistics -Mailbox < alias > -GetMailboxLog # Retrieve MailboxLog to an additional recipient Get-MobileDeviceStatistics -Mailbox < alias > -GetMailboxLog – Notification Addresses <additional recipient>

19. MailboxLogParser Limitations: Total number of logs 5000 MaxAge 72hours
Logsize
10MB (size >5MB compressed)

20. MailboxLogParser
Q: Why should we care about the limits? A: When you enable tracing for a mailbox to collect logs, Exchange will disable logging once a limit was hit  you might end-up with no logs when it happens. Solution: Use a scheduled task to make sure logging is enabled and to pull logs.

21. Get-DatabaseEvent

22. Get-DatabaseEvent
In Exchange 2007 a new table in each database was introduced. It’s called EventHistory. Within this table Exchange tracks modifications. The way to access and query this table was made public in 2013, June.
Prerequisites:
At least Exchange Administration Tools
PSSnapin needs to be loaded manually

23. Get-DatabaseEvent
#load PSSnapin Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support #get information from mailbox Get-Mailbox < alias > | Format-List ExchangeGUID,ServerName

24. Get-DatabaseEvent
The database event entries contains detailed information for any item

25. Get-DatabaseEvent
ClientCategory tells you, which protocol was used to modify an item.
In a delegate scenario you will see that PrincipalName and PrincipalSid have values.
Note: In Exchange 2013 there is a bug, which causes PrincipalName and PrincipalSid are not populated.

26. MessageTracking

27. Get-MessageTrackingLog Those logs contains high detailed information and are easy to parse
SourceContext
EventData
MDB
Mailbox
Event
MessageClass
CreationTime
ClientType
SubmissionAssistant
…..
MailboxDatabaseGuid
ItemEntryId
DeliveryPriority
AccountForest
FirstForestHop
E2ELatency
MsgRecipCount
…..

28. Get-MessageTrackingLog
From the SourceContext we can definitely identify, which client and what MessageClass was submitted.

29. Get-CalendarDiagnosticObjects

30. Get-CalendarDiagnosticObjects
Successor of Cmdlets
Get-CalendarDiagnosticLog
Get-CalendarDiagnosticAnalysis
Only available in Exchange-Online
Result is limited to 1000 objects (even you can specify ResultSize)
Inconsistent results (searching by MeetingId returns different result as by Subject)🙄🤷‍♂️
You can specify custom properties to gather (CustomPropertyNames)

31. Exchange Web Services (EWS)

32. Calendar Versioning
In Exchange 2010 Calendar Versioning was introduced and, in each version, evolved:
Keeps track of each change for each item
Versions are kept for 120 days
Versions are stored in folder Calendar Logging (it depends)
Single Item Recovery and Litigation Hold: Full items are stored in Recoverable Items\Deletions or Recoverable\ItemsVersions. This depends on Exchange and CU version. In Exchange 2016 CU4 even with SIR enable in Calendar Logging

33. Calendar Versioning
Single Item Recovery and Litigation Hold: Full items are stored in Recoverable Items\Deletions or Recoverable\ItemsVersions. This depends on Exchange and CU version. In Exchange 2016 CU4 even with SIR enable in Calendar Logging

34. Calendar logging Using EWS to search one or multiple mailboxes for calendar items
Search filter items
Search scope
Search filters properties
Subject
StartDateLastModified
EndDateLastModified
CleanGlobalObjectID
GlobalObjectID
AllFolders
CalendarOnly
AllItemProps

35. Calendar logging Prerequisites: Permissions: Software: FullAccess
ApplicationImpersonation
Software:
EWS managed API
PowerShell

36. Calendar logging
This is one of the most important MAPI properties as here the client, which modified the item , is logged.
Depending on Exchange version, this property is not always set.
When EAS was used, a subset of the IIS log field cs-uri-query is extracted.
In all other cases the value contains only something similar than cs(User-Agent) in IIS logs.
E.g.:
Client=MSExchangeRPC

37. In this screenshot you can see the gathered information from an example. First an appointment was created, which was than changed to a meeting request. Location and body was also changed.
Calendar logging

38. In this screenshot you can see the gathered information from an example. First an appointment was created, which was than changed to a meeting request. Location and body was also changed.
Calendar logging

39. Where to start?
It really depends on the issue you are troubleshooting.
Calendar
First choice would be EWS as gathering data is much faster than parsing several GB of IIS logs
Lockouts/No sync at all
IIS logs in W3SVC1 folder across all servers
FullResync loop
Mailbox logging might help you to identify an item, which causes a client to choke on

40. Combination
Sometimes you need to exam logs from multiple sources to find the main conclusion.
In this example the MAPI property, which represents the client was not populated.
The IIS logs shows an issues command SendMail.
As a sanity check MessageTracking was used to check the value of SourceContext.

41. Combination

42. Summary Exchange Client Access architecture IIS logs
ActiveSyncDebugLogging & MailboxLogParser
Get-DatabaseEvent
MessageTracking
Get-CalendarDiagnosticObjects
Exchange Web Services and Calendar Logging
Combination of previous mentioned technics

43. Q&A

44. THANK YOU