Transport and Tunnel Modes in IPsec

The IPsec standards define two distinct modes of IPsec operation, transport mode and tunnel mode. The modes do not affect the encoding of packets. The packets are protected by AH, ESP, or both in each mode. The modes differ in policy application when the inner packet is an IP packet, as follows:

• In transport mode, the outer header determines the IPsec policy that protects the inner IP packet.

• In tunnel mode, the inner IP packet determines the IPsec policy that protects its contents.

In transport mode, the outer header, the next header, and any ports that the next header supports, can be used to determine IPsec policy. In effect, IPsec can enforce different transport mode policies between two IP addresses to the granularity of a single port. For example, if the next header is TCP, which supports ports, then IPsec policy can be set for a TCP port of the outer IP address. Similarly, if the next header is an IP header, the outer header and the inner IP header can be used to determine IPsec policy.

Tunnel mode works only for IP-in-IP datagrams. Tunneling in tunnel mode can be useful when computer workers at home are connecting to a central computer location. In tunnel mode, IPsec policy is enforced on the contents of the inner IP datagram. Different IPsec policies can be enforced for different inner IP addresses. That is, the inner IP header, its next header, and the ports that the next header supports, can enforce a policy. Unlike transport mode, in tunnel mode the outer IP header does not dictate the policy of its inner IP datagram.

Therefore, in tunnel mode, IPsec policy can be specified for subnets of a LAN behind a router and for ports on those subnets. IPsec policy can also be specified for particular IP addresses, that is, hosts, on those subnets. The ports of those hosts can also have a specific IPsec policy. However, if a dynamic routing protocol is run over a tunnel, do not use subnet selection or address selection because the view of the network topology on the peer network could change. Changes would invalidate the static IPsec policy. For examples of tunneling procedures that include configuring static routes, see Protecting a VPN With IPsec.

In Oracle Solaris, tunnel mode can be enforced only on an IP tunneling network interface. For information about tunneling interfaces, see Chapter 6, Configuring IP Tunnels, in Configuring and Administering Oracle Solaris 11.1 Networks. The ipsecconf command provides a tunnel keyword to select an IP tunneling network interface. When the tunnel keyword is present in a rule, all selectors that are specified in that rule apply to the inner packet.

In transport mode, ESP, AH, or both, can protect the datagram.

The following figure shows an IP header with an unprotected TCP packet.

Figure 6-3 Unprotected IP Packet Carrying TCP Information

In transport mode, ESP protects the data as shown in the following figure. The shaded area shows the encrypted part of the packet.

Figure 6-4 Protected IP Packet Carrying TCP Information

In transport mode, AH protects the data as shown in the following figure.

Figure 6-5 Packet Protected by an Authentication Header

AH protection, even in transport mode, covers most of the IP header.

In tunnel mode, the entire datagram is inside the protection of an IPsec header. The datagram in Figure 6-3 is protected in tunnel mode by an outer IPsec header, and in this case ESP, as is shown in the following figure.

Figure 6-6 IPsec Packet Protected in Tunnel Mode

The ipsecconf command includes keywords to set tunnels in tunnel mode or transport mode.

• For details on per-socket policy, see the ipsec(7P) man page.

• For an example of per-socket policy, see How to Use IPsec to Protect a Web Server From Nonweb Traffic.

• For more information about tunnels, see the ipsecconf(1M) man page.

• For an example of tunnel configuration, see How to Protect a VPN With IPsec in Tunnel Mode.