A virtual private network or VPN leverage a private network to travel on a public network and also allow users to communicate over public or shared networks even if they are connected to the private network.



IPSEC is a standards-based security architecture for IP so this is called as IP-SEC.

IPSec is not a protocol in itself; rather, it is a set of protocols that allows authentication, confidentiality, integrity, and protection against repetitions between systems.

IPSec enables the following VPN features:

Data confidentiality: The IPSec sender can encrypt packets before transmitting them over a network.

Data integrity: The IPSec receiver authenticates packet sent by the IPSec sender to verify that the data has not been changed or should be original during transmission.

Data Source Authentication – The IPSec receiver can authenticate the source of the sent IPSec packets. This service depends on the data integrity service.

Anti-replay: The IPSec receiver can detect and reject played packets.


## IPSec is built into the Internet layer (Layer 3), provides security for almost all protocols in the TCP / IP suite, and since IPSec is transparently applied to applications, so it not really required to configure individual security for each application which uses TCP / IP.

## Anti-replay is an IPsec sub-protocol that is part of the Internet Engineering Task Force (IETF). The primary purpose of anti-replay is to prevent hackers from injecting or making changes to packets traveling from a source to a destination. The anti-replay protocol uses a one-way security association to establish a secure connection between two nodes on the network. Once a secure link is established, anti-replay protocol uses sequence numbers of packet to kill replay attacks as follows: when the source sends a message, it adds a sequence number to its packet; the sequence number starts at 0 and increases by 1 for each subsequent packet. The destination keeps a ‘sliding window’ record of the sequence numbers of the received validated packets; rejects all packets that have a sequence number that is less than the lowest value in the sliding window (i.e. too old) or already appearing in the sliding window (i.e. duplicates / repeats). Accepted packages, once validated, update the sliding window (moving the lowest sequence number out of the window if it was already full).


IPSEC – Architecture


IPSEC combines three main protocols into a Cohesive Security Framework.


The IPsec is an open standard framework uses the below protocols to perform several functions:


Authentication (AH) headers provide offline data integrity and data source authentication for IP datagrams and provide protection against replay attacks.


Encapsulation Security Loads (ESP) provide confidentiality by encrypting the data, authentication, connection-less integrity, an anti-replay service and limited confidentiality.


Security Associations (SA) provide the set of algorithms and data that provide the necessary parameters for AH and / or ESP operations.


The Internet Security Association and Key Management Protocol (ISAKMP) provides a framework to authenticate & key exchange. This includes real authenticated key material which is provided by manual configuration with pre-shared keys, Internet Key Exchange (IKE and IKEv2).


IKE is a two phase protocol


Phase 1 includes two modes – Main Mode and Aggressive Mode.

In main mode we have six messages exchange between peers and in aggressive mode we have three messages exchange.


Phase 1 Exchange- Peers negotiate a Secure, Authenticated Channel with which to communicate ‘Main Mode’ or Aggressive Mode’ accomplishes a Phase 1 exchange.


Phase 2 includes only one mode that is Quick Mode.


Phase 2 Exchange – Peers negotiate Security Associations on behalf of IPSEC services and ‘Quick Mode’ accomplishes a Phase 2 exchange.


How IPSEC Works


IPSec involves many component technologies and encryption methods. However, the IPSec operation can be divided into five main steps:

  • Interesting traffic starts the IPSec process. Traffic is considered interesting when the IPSec security policy configured on the IPSec pairs starts the IKE process.

  • IKE phase 1. IKE authenticates the IPSec peers and negotiates SA IKE during this phase, configuring a secure channel to negotiate SA IPSec in phase 2. In this phase tunnel is created.

  • IKE phase 2. IKE negotiates the IPSec SA parameters and establishes matching IPSec SAs in the pairs. In this phase tunnel creates a secure data channel over the created tunnel.

  • Data transfer. The data is transferred between IPSec pairs based on the IPSec parameters and keys stored in the SA database.

  • IPSec tunnel termination. IPSec SAs end by deletion or timeout.


IKE – Phase 1 (Main Mode) – Explanation


Initiator initiates the IKE phase 1 process by sending message 1 to the responser and responder send the response in message 2 in which IKE policy sets are created to negotiate 5 parameters.

Now the two peers have agreed on how they will establish phase 1, then they must agree on a “Shared Key”. Both ends must use the same shared key, but the shared key cannot be sent between them because the network link is not secure. To do this, they use a Diffie Hellman key exchange, this uses a mathematical process called modular exponentiation, a simple example of how it works (The mathematics involved in a real key exchange is much more complicated!).

The initiator generates a “Public Key” also called DH Public value(Xi). It also generates a Nonce (Ni)and sends them both to the responder. The responder generates a “Public Key” also called DH Public value(Xr). It also generates a Nonce(Nr) and sends them both to the initiator.

At this point, both the initiator and responder can calculate the DH shared secret key, then use the DH secret key, the “shared secret” and the Nonce of the other pair to create 3 DIGITAL KEYS, due to the nature of Diffie Hellman, each end will produce the same keys.

Key 1 = SKEYID_d Used to resolve any future IPsec keys Key 2 = SKEYID_a Used for data integrity and authentication (IKE) Key 3 = SKEYID_e Used to encrypt all additional IKE traffic.

The initiator now sends its ID to the responder (this is its IP address or a hostname). It also sends a “Hash” that authenticates the initiator to the responder since it is made from SKEYID, the pre-shared key, and other information only known to the two pairs.

Message 6 must be the mirror of message 5, the responder sends its ID (IP or hostname). Behind the initiator with its “Hash” and authenticates back to the initiator.


At this point, both pairs recalculate the hash they have received from the other partner, and they should both exit the same, if this happens, the IKE SAs are established and phase 1 is completed.


Before we look into Phase 2 (Quick Mode), we need to understand concept of Perfect Forward Secrecy (PFS).


What is PFS


PFS will make sure that the same key is not generated again, so it will force a new diffie-hellman key exchange. This would ensure that if a hacker \ criminal compromised a private key, they would only be able to access the data in transit protected by that key and not future data, as future data would not be associated with that compromised key.

Both sides of the VPN must be PFS compatible for PFS to work. When PFS is enabled, for each negotiation of a new phase 2 SA, the two gateways must generate a new set of phase 1 keys. This is an additional layer of protection that PFS adds, which ensures that if phase SAs 2 have expired, the keys used for the new Phase 2 SAs have not been generated from the current Phase 1 encryption material. Of course, if PFS is not activated, the current encoding material already set up in phase 1 will be used again to generate phase 2 SAs.


Phase 2 (Quick Mode)


Once Phase 1 has completed, the second stage of the VPN can begin. Like phase 1, this state also requires messages to be sent between peers, IPsec generally runs in “Quick Mode”, this means there are only 3 messages.


Note: If PFS is configured at one end only, it will fail at this point with an “Attribute not supported” error.


The Initiator sends another Hash to the responder, this is similar to the one used in phase 1 but also includes information within this message to ensure integrity. The responder responds with its own “hash” with the accepted proposal and its own SPI for the responder’s outgoing encrypted traffic, and finally its own key exchange payload.


– Additionally it sends SPI ( This number is the TAG for the end of the tunnel that the initiator will use for outgoing traffic)

– DH Group is only configured if PFS is enabled if there is no PFS then DH Group must not be configured.

– LAN IP Subnet of the peer side must be configured on both end and that must be mirror of each other which would be our Interesting Traffic which actually communicates over the VPN.


Once this is complete, both pairs generate new DH secret keys (If PFS is enabled) and combine them with the SKEYID_d key from phase 1 to create keys for IPsec encryption.


Message 3


The final message is sent from the Initiator to the responder, and serves to inform the responder that their previous message was received.


Once phase 2 is complete, the IPsec SAs have been established and the tunnel is operational.


Leave a Reply

Your email address will not be published. Required fields are marked *