1. NetScaler and XenMobile Connectivity Diagnostics
May 2015
Good afternoon. Anyone who has tried to integrate different technologies knows it can be a challenge. Getting Server A to talk to Server B, routing this packet here and that packet there and trying to figure out exactly why you can’t get an ICMP response are common things every Engineer runs into at least once in their career. The NetScaler can be a powerful tool when integrating any technology, XenMobile included. Today we are going to take a look into some of the things the NetScaler can provide to a XenMobile deployment and how to troubleshoot some common issues when they arise.
Dale McCoon
Senior Technical Relationship Manager

2. A free offering to help keep your Citrix environment running well.
Over 400 plugins that detect various conditions and offer prescriptive advice. New ones added every week.
Previously known as Tools as a Service (TaaS).
Visit the Citrix Insight Services Team in the Expo Hall to learn more and receive a free gift (while supplies last)

3. Agenda What the NetScaler can Provide for XenMobile Availability
Load Balancing
SSL Bridge vs. SSL Offload
Security
Kerberos Authentication with WorxMail
XenMobile NetScaler Connector
Agenda
Lets see what we are going to cover today. We are going to focus on two main topics. Availability, in which we will cover load balancing and some of the considerations to take regarding the differences between SSL Bridge and SSL Offload. Then we will round out our discussion covering Security, in which we will talk about Kerberos Authentication with WorxMail and XenMobile NetScaler Connector

4. What the NetScaler can provide for XenMobile
Two easy to sell concepts
Availability
I don’t want my infrastructure to go down
Security
I don’t want other people to bring my infrastructure down
Two very easy to sell concepts, availability and security
Availability – I don’t want my stuff to go down
Security - I don’t want other people to bring my stuff down

5. Availability

6. Availability MDM Load Balancing SSL Bridge or SSL Offloading
Two Methods of set up
MDM Wizard (Recommended)
Manual set up
SSL Bridge or SSL Offloading
Benefits/Considerations
Configuration

7. Examples of Communication flow (SSL Bridge)
Typical set up for Load balancing MDM. SSL Bridge means the NetScaler doesn’t decrypt the traffic coming from the mobile device and its one SSL stream from the client (mobile device) all the way through to the backend XDM servers. The NetScaler is just providing monitoring and load balancing functions. There is no ability to manipulate the traffic

8. MDM Wizard
Updated XenMobile Wizard in 10.5

9. NetScaler for XenMobile Wizard Selection
Ability to select different functions within XenMobile. Includes a checklist for each function

10. HTTP/s Communication
HTTP or HTTPS communication to the XenMobile Server

11. VServer Config
Vserver Ip address

12. XenMobile Servers
Selection of backend XM servers, obviously you’d want more than one for redundancy purposes.
443 service for secure communication to MDM server
8443 service for registrations for IOS and Windows phone devices, all further communication after registration takes place over port 443

13. SSL Offload Reasons for using SSL Offload
Decreased burden on backend servers to decrypt SSL traffic
Easier to manage SSL certificates in one central location
Plain text traffic on internal network, if auditing requirements are a consideration

14. Enable SSL offload in XMS

15. Export the Device Certificate

16. Configuration Wizard

17. Configuration Wizard

18. Troubleshooting Using NSTCPDUMP to verify traffic NSTRACE to analyze
A variant of Unix based TCPDUMP
NSTRACE to analyze
A fully functional tracing tool for capturing traffic traversing the NetScaler

19. Nstcpdump.sh
Used to view live traffic on NetScaler to verify connectivity
Nstcpdump.sh is useful to view live traffic on the NetScaler. While it does have the ability to write the traffic to a file its best use is for basic network connectivity tests. In this scenario I am troubleshooting an issue where a monitor (in this case ICMP monitor) is showing down with a timeout. I went to the cli to verify connectivity
You can see here I am testing two basic filters.
Nstcpdump.sh icmp and host x.x.x.x. filters for only ICMP traffic to the listed IP address
Digging a little further and using the –e parameter shows me the source and destination MAC address for the traffic, this is useful for when you want to verify the layer 2 connectivity is correct, basically if the traffic is going out the right interface to the correct destination MAC. The –n parameter turns off name resolution to help clean up the output to make it a little more readable.

20. nstrace.sh Useful for taking traces to analyze offline
Filters available to narrow results and limit capture size
Good for troubleshooting and diagnosing complex protocol level issues

21. Security

22. Security Kerberos Authentication with WorxMail
Secure Single Sign On Solution
XenMobile NetScaler Connector
Secure Access Solution

23. Kerberos Authentication With Worxmail
Kerberos Overview
What is it, how does it work
Configuration Overview
Troubleshooting
Nskrb.debug
Communication Analysis

24. Kerberos
Kerberos can be used as a Single Sign On (SSO) mechanism on NetScaler
When challenged by a server (through a 401 Negotiate), NetScaler fetches tickets on user’s behalf
Two kinds of Kerberos SSO is possible
Kerberos SSO with constrained delegation
Kerberos SSO with impersonation
Two kinds of Kerberos SSO is possible:
Kerberos SSO with constrained delegation
Used in the cases where Netscaler does not have user password or when password is not AD password such as One-Time-Password (OTP)
This is where we create an account in AD to request a kerberos ticket on the users behalf
Kerberos SSO with impersonation
Can be used when Netscaler has user’s password
For example, when a user logs into Netscaler using his AD password
Netscaler talks to AD as user himself as it knows username/realm/password of the user

25. Kerberos in NetScaler Overview
Started supporting from: 9.3, 10.0, 10.0.e – were using likewise
Starting from X - likewise is replaced with nskrb
Starting from e - likewise is replaced with nskrb
Major Value add with nskrb:
No more likewise
domain Join is no longer required
Performance is better
Kerberos tickets are cached on NS
User Impersonation is supported.
3 options to enable NS Kerberos Constrained Delegation. (Keytab/DelegatedUserPassword/DelegatedUserCert)
PKINIT Support
PKINIT is a preauthentication mechanism for Kerberos 5 which uses X.509 certificates to authenticate the KDC to clients and vice versa

26. Keytab Config Ktpass Example:
ktpass /princ /ptype KRB5_NT_PRINCIPAL /mapuser nsi-test\svc_kcd1 /pass 1.citrix /out C:\kcd-nsi-test.keytab
/princ:
/ptype: KRB5_NT_PRINCIPAL is the general principal type 
/mapuser: maps the principal to User account
/pass: Specifies password for the principal username specified.
/out: writes the shared secret key to output file.

27. Screenshots of user account in AD

28. Delegation Settings

31. Kerberos: Troubleshooting
Common issues
DNS not configured correctly
/netscaler/nskrb kinit
Password:
kinit: krb5_get_init_creds: unable to reach any KDC in realm dale.com
Kerberos related ports are blocked by Firewall
Clock skew between Netscaler and AD too great
AD configuration incorrect
Delegation is not enabled
Setspn is done with a different account (if KCD account is added with different keytab or with password)
Certificate mapping is not done (if KCD account is added with delegatedUser’s cert pair)
CA cert is not imported to AD (if KCD account is added with delegatedUser’s cert pair)
Nskrb kinit info

32. Firewall Ports Required to be Open for KCD Communication
Protocol
Use 53
UDP/TCP
DNS 88
Kerberos
123
UDP
NTP
135
TCP
RPC Endpoint Mapper
137
NetBIOS Name Service
139
NetBIOS Session (SMB)
389
LDAP
445
SMB over TCP
464
UDP/TCP Machine password changes (typically after 30 days)
3268
 TCP
Global Catalog Search
Make this into table

33. Kerberos: Troubleshooting
How to debug a Kerberos error ?
Take nstrace and filter for ‘Kerberos’
Look at AD event logs ‘windows security log’ event id: 4768/4769/4770/4771
Check /var/krb for cached tickets
For Kerberos specific logging, enable it through windows registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\Ke rberos\Parameters Registry Value: LogLevel
Value Type: REG_DWORD
Value Data: 0x1

34. Kerberos: Troubleshooting
nskrb.debug
Insight into authentication process for Kerberos
Communication analysis
What does that 401 really mean?

35. Nskrb.debug Similar function to aaad.debug but for Kerberos
Provides debug level messaging for Kerberos authentication
Error messages are based on standard Kerberos error codes

36. Common Error Codes
0x6 - KDC_ERR_C_PRINCIPAL_UNKNOWN: Client not found in Kerberos database
1. The actual account does not exist.
2. new account is added but not yet replicated to other KDC.
3. Check if the account is expired or ‘logon restrictions’ enabled.
0x18 - KDC_ERR_PREAUTH_FAILED: Pre-authentication information was invalid
The wrong password was provided.
Verify that the time on the KDC matches the time on the client.

37. Common Error Codes
0x17 - KDC_ERR_KEY_EXPIRED: Password has expired – change password to reset
The Delegated user’s password has expired.
0x1C - KDC_ERR_PATH_NOT_ACCEPTED: KDC Policy rejects transited path
1. A trust is incorrectly set up between two domains.
Resolution: Verify that there is a two-way transitive trust set up between the user’s domain and the domain on which the user is trying to access resources.
2. Constrained delegation is being attempted across multiple domains.
Resolution: 2000/2003/2008 does not support Constrained delegation across multiple domains.

38. Communication Analysis

39. XenMobile NetScaler Connector

40. XenMobile NetScaler Connector
What it does/benefits
Access solution
Used standalone or with XenMobile Device manager
Two Methods of set up
XNC Wizard (recommended)
Manual Set Up
What: XenMobile NetScaler Connector is a solution that controls access to corporate , calendar, and contacts from Mobile Devices. Xenmobile Netscaler connector allows customers to send a list of compliant devices from XenMobile Device manager to NetScaler, which in turn controls which mobile devices are allowed to sync with the corporate Exchange server

41. XNC Communication Flow

42. XenMobile NetScaler Connector
How does it work?
2 Responder policies
2 HTTP Callouts
Exchange VServer and Services
XNC VServer and Services
Integrated Caching policies (applicable if licensed for Integrated Caching)
2 HTTP Callouts, one for a request with the device ID and one without

43. Troubleshooting Example of typical POST
What's really happening with the HTTP callout
Deconstructing the HTTP Callout and Responder policies
Expected output and response

44. Example POST
POST /Microsoft-Server- ActiveSync?User=mydomain%5CDaleM&Cmd=GetItemEstimate&DeviceId =Samsung9999&DeviceType=Samsung HTTP/1.1
User-Agent: Samsung(SRPC)/ /
Connection: keep-alive
X-MS-PolicyKey:
MS-ASProtocolVersion: 14.1
Authorization: Basic bXlkb21haW5cdmlqYXk6WE5DLUxhYg==
Content-Type: application/vnd.ms-sync.wbxml
Content-Length: 236
Host: :80
Authorizations string is Base64 encoded, can be easily decoded to check for accuracy

45. Responder Policy
add responder policy _XM_RESP_W_DEVICEID_ "HTTP.REQ.URL.QUERY.CONTAINS(\"DeviceId\") && HTTP.REQ.URL.STARTSWITH(\"/Microsoft-Server- ActiveSync\") && HTTP.REQ.METHOD.EQ(POST) && HTTP.REQ.HOSTNAME.EQ(\"callout.asfilter.internal\").NO T && SYS.HTTP_CALLOUT(_XM_W_DEVICEID_3_3_3_3).SET _TEXT_MODE(IGNORECASE).CONTAINS(\"allow\").NO T" DROP

46. Deconstructing the HTTP Callout and Responder policies
set policy httpCallout _XM_W_DEVICEID_3_3_3_3 -vServer _XM_LB_CACHE_ returnType TEXT -hostExpr "\"callout.asfilter.internal\"" -urlStemExpr "\"/services/ActiveSync/Authorize\"" -parameters user(HTTP.REQ.HEADER("authorization").AFTER_STR("Basic ").B64DECODE.BEFORE_STR(":").HTTP_URL_SAFE) agent(HTTP.REQ.HEADER("user-agent").HTTP_URL_SAFE) ip(CLIENT.IP.SRC) url((" ) resultType("json") DeviceId(HTTP.REQ.URL.QUERY.VALUE("DeviceId")) -scheme http - resultExpr "HTTP.RES.BODY(20)"

47. HTTP Callout

48. Translated by the NetScaler
Checking XNC for connectivity for HTTPCall out issues:
E.g. : Use a sample GET( this is exactly how the HTTP-Callout feature makes a request to XNC).
GET /services/ActiveSync/Authorize?user=mydomain\Dale&agent=Apple-iPhone3C2&ip= &url=aHR0cHM6Ly9uc2Nhcy50ZXN0cHJpc2UubmV0L01pY3Jvc29mdC1TZXJ2ZXItQWN0aXZlU3luYz9Vc2VyPXRlc3RwcmlzZS5uZXRca211c2VyMSZDbWQ9U3luYyZEZXZpY2VJZD1hbmRyb2lkYzEyMDQzNDM5NjEmRGV2aWNlVHlwZT1Ub3VjaERvd24=&resultType=json HTTP/1.1
Host: callout.demo.com
Result:
HTTP/ OK
Content-Length: 7
Content-Type: application/json; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 08 May :40:26 GMT
"allow"
Expected behavior is “allow” or “deny”
XNC logs are under C:\Program Files\Citrix\XenMobile NetScaler Connector\log

49. Wrap up
NetScaler provides an end to end security and availability solution for XenMobile deployments
MDM Load balancing is fairly straight forward
Kerberos deployments have a lot of moving parts but knowing each part is essential for troubleshooting

50. Resources http://support.citrix.com/article/CTX200063
xenmobile-worxmail/

51. Questions?

52. Before you leave…
Conference Surveys are available online at starting Thursday, May 14 at 9:00 a.m.
Those who provide feedback by 6pm, Friday, May 15th will receive:
$20 Amazon e-gift card
Name entered in a drawing for a free Trip to Synergy 2016 (5 chances)
Download presentations starting Monday May, 18th from the My Event Planning tool