IPSec VPN: How does it really work? Yasushi Kono (ComputerLinks Frankfurt)

Before the Agenda My intention of this presentation: My intention of this presentation: I know that many people do not have any clue what‘ s happening while establishing Security Associations prior to create VPN tunnels. This topic is quite complex. I want you to get started into this topic taking away a bit of its complexity.

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

Introductory Information on IPSec VPN Introductory Information on IPSec VPN

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 Phase 2 Phase 1 and Phase 2. In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore:

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 Phase 2 Phase 1 and Phase 2. In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacy

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 Phase 2 Phase 1 and Phase 2. In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacy 2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 Phase 2 Phase 1 and Phase 2. In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacy 2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity 3.) Diffie-Hellman Group (1, 2, 5, or 14)

Before establishing a Site-to-Site VPN connection, both gateways must agree upon parameters for encrypting communication. This negotiation process is divided into two phases: Phase 1 Phase 2 Phase 1 and Phase 2. In Phase 1 a IKE Security Association (IKE SA) will be created. Those parameters are therefore: 1.) Encryption Algorithm (3DES, AES-128, AES-256,…) to ensure privacy 2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity 3.) Diffie-Hellman Group (1, 2, 5, or 14) 4.) Method of mutual authentication (Preshared Key or Certificate)

You have the choice between two different modes in Phase 1: Main ModeMain Mode Aggressive ModeAggressive Mode Main Mode consists of a 6 packet negotiationMain Mode consists of a 6 packet negotiation In Aggressive Mode only three packets are exchanged between both VPN gateways.In Aggressive Mode only three packets are exchanged between both VPN gateways.

Before beginning to analyze the information exchanged by the gateways, let‘s have a look at the Diffie- Hellman Algorithm: Why is Diffie-Hellman necessary?

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

Usually, payloads are encrypted symmetrically by means of symmetric encryption algorithms, like 3DES or AES. The problem with symmetric encryption is exchanging the keys over the Internet while preventing them from falling into the wrong hands. One answer is asymmetric encryption.

So, when asymmetric encryption is addressing problems with symmetric encryption, why is latter technology still necessary?

The answer is: Performance!

The Diffie-Hellman Algorithm is not an Encryption Algorithm rather than a Key Agreement Protocol.

To understand the Diffie-Hellman Algorithm, you have to have basic knowledge on the Modulo operation. The Modulo operation finds the remainder of division of one number by another. Given two numbers, a and b, n = a mod b is the remainder on division of a by b.

Examples: 30 mod 16 = mod 100 = 14 8 mod 3 = 2 Isn‘t that simple?

Diffie-Hellman: Initiator takes a prime number p and an integer a with 1 < a < p and a secret integer x with: X = a x mod p X, a, p: public parameters, x: secret parameter.

Initiator sends X, a, and p to the Responder. The Responder takes a secret integer y and computes: Y = a y mod p The number Y, which is public will be sent back to the Initiator.

Initiator and Responder are computing their common encryption key: K x = Y x mod p = (a y mod p) x mod p = (a x mod p) y mod p = X y mod p = K y Hence K x = K y

Now, that we have the basic understanding of the Diffie- Hellman Algorithm, we can now fully understand IKE SA in Main Mode.

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

As all of you might already know, IKE SA in Main Mode consists of 6 packets

First of all, the Initiator sends the IKE SA Parameters to be negotiated upon to the Responder: IKE SA: 3DES or AES-128? SHA-1 or MD5? DH-Group 5 or 14? Certificate or Preshared Key?

Initiator Responder Packet Number 1:

The Responder sends back the parameters to be used in common to the Initiator: IKE SA: AES-128! SHA-1! DH-Group 14! Preshared Key!

Initiator Responder Packet Number 1: Packet Number 2:

Then, the Initiator sends the public Diffie- Hellman Parameters and a random number, which is called „Nonce“:

Initiator Responder Packet Number 3:

Then, the Responder sends back its own public Diffie-Hellman Parameters and its own random number („Nonce“):

Initiator Responder Packet Number 3: Packet Number 4:

Both parties know from each other, which parameters and encryption key to use in common. The Initiator then builds the hash of the Preshared Keybuilds the hash of the Preshared Key encrypts the Nonce of the Responder and the Hash of the Preshared Keyencrypts the Nonce of the Responder and the Hash of the Preshared Key

Initiator Responder Packet Number 5: Packet Number 6:

What happens next? Each of the Gateways are receiving their own Nonces encrypted by other parties. The next step is to decrypt the encrypted Nonces to verify the identity of the communicating gateways.

After receiving the sixth packet, you will get the following message in SmartView Tracker: IKE SA: Main Mode completion

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

Why is Quick Mode necessary? To establish the IPSec SA!

Quick Mode to establish an IPSec SA consists of 3 packets.

You have to negotiate upon: The Encryption algorithmThe Encryption algorithm The Hash AlgorithmThe Hash Algorithm The IPSec Protocol (ESP, AH)The IPSec Protocol (ESP, AH) If PFS is to be used or notIf PFS is to be used or not (if yes, the DH-Group is to be determined again)

One word regarding IPSec Protocols: Authentication Header (AH) provides: Data Integrity CheckingData Integrity Checking Replay ProtectionReplay Protection

Encapsulating Security Payload (ESP) provides: Payload EncryptionPayload Encryption Data Integrity CheckingData Integrity Checking Replay ProtectionReplay Protection

Why is it necessary to agree upon the Encryption AlgorithmEncryption Algorithm Hash AlgorithmHash Algorithm Eventually DH-GroupEventually DH-Group again? again?

Because the parameters negotiated there is for applying to the payload of the packets and not to the identity of the other gateway!

The 3 Packets in Quick Mode

The Message you will get after succesful IPSec SA Negotiation is: IKE SA: Quick Mode completion

Agenda Introductory Information on IPSec VPN Introductory Information on IPSec VPN Why Diffie-Hellman Algorithm? Why Diffie-Hellman Algorithm? IKE SA in Main Mode IKE SA in Main Mode IPSec SA in Quick Mode IPSec SA in Quick Mode Some Troubleshooting Tips Some Troubleshooting Tips

Some Troubleshooting Methods: Some Troubleshooting Methods: Mostly, you will get the right information with SmartView Tracker There, the most frequent error messages are: IKE SA: No proposal chosenIKE SA: No proposal chosen Encryption failure: No valid SAEncryption failure: No valid SA INVALID_ID_INFORMATIONINVALID_ID_INFORMATION

Some Troubleshooting Methods: Some Troubleshooting Methods: Received Notification from Peer: Malformed Payload Any solution?

One standard method for Debugging IKE/IPSec is One standard method for Debugging IKE/IPSec is vpn debug ikeon to generate the output file ike.elg.

But, without any knowledge of the theory of IPSec, is it useful to analyze the ike.elg file? But, without any knowledge of the theory of IPSec, is it useful to analyze the ike.elg file?

For more troubleshooting IKE/IPSec in a Check Point Environment, attend the next presentation: 305: Troubleshooting in the Check Point Environment – Part II By Tobias Lachmann

Any Questions?

Thanks a lot for your attention! Should you have questions: