An overview of IPSec

IPSec is a protocol suite which allows for secure communication through means of authenticating and encrypting each IP packet on a network. What it means to be a protocol suite is that IPSec is a protocol which is made up of other protocols. The protocols used in IPSec are the following:

  • Authentication Headers (AH)
  • Encapsulating Security Payloads (ESP)
  • Security Associations (SA)

At a glance, how IPSec accomplishes this

  1. The client and server must agree on a set of security protocols to use, so that information transferred can be understood
  2. An encryption algorithm needs to be selected
  3. Keys need to be exchanged

Brief History

Developed out of experimental research protocol called "swIPe" in December of 1993 at Columbia University and AT&T Bell Labs. From this, Wei Xu took over the project and was able to get a working system by the end of December 1995 when he was able to securely connect from the west coast to the east coast.

Modes of IPSec

IPSec has two different modes that it can be run in. These are Transport Mode and Tunnel mode. Each has a different purpose.

Transport Mode

Transport mode is the default mode for IPSec, and it is used for end-to-end communications (for example, for communications between a client and a server). When transport mode is used, IPSec encrypts only the IP payload. Transport mode provides the protection of an IP payload through an AH or ESP header.

Tunnel Mode

IPSec encrypts the IP header and the payload, whereas transport mode only encrypts the IP payload. Tunnel mode provides the protection of an entire IP packet by treating it as an AH or ESP payload.

As an example, given the OSI model for networking, Transport Mode will encrypt everything from the Transport layer and above. With tunnel mode, everything from the Network layer and above will be encrypted. Transport mode is useful for using IPSec on internal networks.

Security Association (SA)

The Security Association protocol is used to describe which algorithms will be used in future payloads. For instance, which encryption or hashing algorithm to use. In addition to this, Each SA consists of values such as destination address, a security parameter index (SPI), the IPSec transforms used for that session, security keys, and additional attributes such as the IPSec lifetime. After this has been decided, the machines will then attempt to share keys which will be used for future communication.

An example request may look like:

  1. Machine one sends SA ( an agreement on how to initiate communication ) to Machine two
  2. Machine two receives this message and has the option to either to accept or reject the request
  3. If Machine two accept, machine two will send the same message back to machine one

As mentioned before, when an agreement is made between two machines then they must then share keys. This is when the Internet Key Exchange (IKE) protocol is utilized. The IKE protocol uses the Diffie-Hellman Key Exchange algorithm, which is described in detail below.

First a little background on what it means to share a key. There are two methods to accomplish this:

  1. Symmetric Keys - Passing a shared secret

  2. Asymmetric Keys - Public and Private key pairing

Simply passing a symmetric is fast, but also insecure as anyone else along the path of the communication could intercept it. When using public and private key's (asymmetric), the transfer is very secure but also slower to compute. The general solution to this is to combine the two so that you get the both worlds.


Diffie-Hellman is a method to transfer cryptographic keys. It was first published in 1976 by Whitfield Diffie and Martin Hellman, although it was technically invented a few years prior. It has yet to be cracked.

How it works

  1. Bob and Alice wish to create a shared key. They have chosen Diffie-Hellman to accomplish this. They will both share or acknowledge the same n and g values where n is a prime number, n being a base.
  2. Once decided on those public values, both Bob and Alice will pick a private value "x"
  3. Each of them will now generate a public key based on the formula X = gx mod n.
  4. They will then exchange these public keys with each other
  5. Now that Bob and Alice have exchanged keys, they can then create a shared secret using the formula Xb mod p where X is either Bob or Alice's public key and b is their own private key.


  1. Alice and Bob agree to use a prime number p=23 and base g=5.
  2. Alice chooses a secret integer a=6, then sends Bob A = ga mod p
    • A = 56 mod 23
    • A = 15,625 mod 23
    • A = 8
  3. Bob chooses a secret integer b=15, then sends Alice B = gb mod p
    • B = 515 mod 23
    • B = 30,517,578,125 mod 23
    • B = 19
  4. Alice computes s = B a mod p
    • s = 196 mod 23
    • s = 47,045,881 mod 23
    • s = 2
  5. Bob computes s = A b mod p
    • s = 815 mod 23
    • s = 35,184,372,088,832 mod 23
    • s = 2
  6. Alice and Bob now share a secret: s = 2. This is because 6*15 is the same as 15*6. So somebody who had known both these private integers might also have calculated s as follows:
    • s = 56*15 mod 23
    • s = 515*6 mod 23
    • s = 590 mod 23
    • s = 807,793,566,946,316,088,741,610,050,849,573,099,185,363,389,551,639,556,884,765,625 mod 23
    • s = 2

A demonstration in javascript can be seen here

Authentication Headers (AH) and Encapsulating Security Payloads (ESP)

The Authentication Header (AH) and the Encapsulating Security Payload (ESP) which are used to authenticate (AH) and authenticate and encrypt (ESP). They are often used separately, but are sometimes used together.


The Authentication Header (AH) is used to authenticate but not to encrypt communication over a a VPN. Authentication of data is essential to security so that is ensures that data has not been tampered with, and that you are actually communicating with who you think you are.

Authentication is performed by utilizing hash functions. A few popular options are shown below

The AH consists of 5 different fields:

  1. Next Header: This identifies the protocol type of the following payload.
  2. Length: Size of the AH header in 32 bit words.
  3. SPI, Security Parameters Index: An arbitrary 32 bit value used to identify the security association of the receiving party.
  4. Sequence Number: This is a monotonically increasing 32 bit identifier that's used to assist in antireplay protection.
  5. Authentication Data: This is the Integrity Check Value calculated over the entire packet - including most of the headers - The recipient recomputes the same hash; Mismatched values mark the packet as either damaged in transit, or not having the proper secret key. These are discarded.


ESP not only can authenticate data but encrypt it as well. Unlike in the Authentication Header where it precedes the data, the ESP encapsulates the entire payload. Some popular encryption algorithms for use with ESP include: DES, Triple DES, AES, and Blowfish. These are discussed in detail below.

Authenticating data

In order to ensure that the data being sent over the VPN has not been tampered or been corrupted in some way, hashing is used as part of the IPSec protocol. Several popular options for this are SHA-1 and MD5.


Secure Hash Algorithm 1 (SHA-1) is a cryptographic hash function designed by the NSA in 1995. There are actually four different SHA algorithms (SHA-0 - SHA-3), however SHA-1 is current the most widely used of them all.

SHA-1 will produce a 160 bit message digest, and the principles behind the algorithm are actually based on what is used in MD4 and MD5


The Message-Digest Algorithm (MD5) is a cryptographic hash function which produces a 128 bit message digest. While still widely used, several flaws have been discovered which have pushed people in the direction of using other hash algorithms such as SHA-1.



Data Encryption Standard (DES) is a previously dominant encryption method which was developed in the early 1970's. It was based on a design by Horst Feistel ( no picture available ). It's popularity has been reduced due to the 56 bit key being too small. As an attempt to illustrate this, and the Electronic Frontier Foundation came together in 1999 and were able to break the key in 22 hours.


3DES leverages the original DES algorithm, but actually uses it three times using three different keys. This results in a 168 bit key. For a detailed analysis of 3DES, please refer to my other classmates research entirely on the subject here


Advanced Encryption Standard (AES) is a symmetric key encryption algorithm developed in 1998 which was invented by Joan Daemen and Vincent Rijmen. It is widely used and can encrypt into 128, 192, or 256 bit key sizes.

  1. KeyExpansion-round keys are derived from the cipher key using Rijndael's key schedule.
  2. Initial Round
    1. AddRoundKey - each byte of the state is combined with the round key using bitwise xor.
  3. Rounds
    1. SubBytes - a non-linear substitution step where each byte is replaced with another according to a lookup table.
    2. ShiftRows-a transposition step where each row of the state is shifted cyclically a certain number of steps.
    3. MixColumns-a mixing operation which operates on the columns of the state, combining the four bytes in each column.
    4. AddRoundKey
  4. Final Round (no MixColumns)
    1. SubBytes
    2. ShiftRows
    3. AddRoundKey

An interactive example of AES encryption can be seen here


Blowfish is a symmetric key encryption algorithm designed by Bruce Schneier in 1993. It is able to encrypt with key sizes between 32 and 448 bits.

An example implementation I wrote can be found here


Back to top