HP HP-UX IPSec User's Guide

Configuring Microsoft Windows

Vista and Windows Server

2008 to Operate with HP-UX IPSec

Introduction......................................................................................................................................... 2

Related Documentation ..................................................................................................................... 2

Protocol implementation differences.................................................................................................... 2

IKE version .................................................................................................................................. 2

IKE authentication method ............................................................................................................. 2

IKE default algorithms ................................................................................................................... 2

IKE aggressive mode .................................................................................................................... 3

IPsec default transform .................................................................................................................. 3

Configuration overview ........................................................................................................................ 3

Configuring Windows IKE algorithms..................................................................................................... 3

Configuring connection security rules ..................................................................................................... 4

Using the Windows Firewall MMC to configure connection security rules ............................................... 4

Using the netsh advfirewall consec command to configure connection security rules................................. 5

Configuring inbound and outbound firewall rules .................................................................................... 6

(Optional) Configuring IPsec transforms.................................................................................................. 6

Introduction

This document contains tips for configuring IPsec on Microsoft Windows Vista and Windows Server 2008 systems to operate with HP-UX IPSec versions A.02.01 and later. For information on configuring IPsec on Microsoft Windows XP and Windows 2003 systems to operate with HP-UX IPSec, see in Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec.

The intended audience for this document is a network security administrator who is familiar with Microsoft Windows Vista or Windows Server 2008, the HP-UX IPSec product, and the IP Security protocol suite.

Related Documentation

The following documents are available from the HP Technical Documentation website at

http://docs.hp.com:

For information on configuring IPsec on Microsoft Windows XP and Windows 2003 systems to operate with HP-UX IPSec, see Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec.

For general information about configuring HP-UX IPSec, see the HP-UX IPSec A.02.01 Administrator's Guide or the HP-UX IPSec A.03.00 Administrator's Guide.

For information about configuring HP-UX IPSec to use Microsoft Windows security certificates, see

Using Microsoft Windows Certificates with HP-UX IPSec A.02.01and Using Microsoft Windows Certificates with HP-UX IPSec A.03.00.

Protocol implementation differences

HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there are features in the protocol suite that HP-UX implemented that Microsoft did not implement, and vice-versa. There are also differences in default parameter values.

IKE version

At the time this document was published, Microsoft Windows Vista and Windows Server 2008 supported only Internet Key Exchange (IKE) IKE version 1. HP-UX IPSec version A.03.00 supports IKE version 1 and IKE version 2. In this document, the term “IKE” refers to IKE version 1.

IKE authentication method

The Windows default IKE authentication method is Kerberos. RFC 2408 defines an optional Kerberos Token payload, but does not describe how to implement it. HP-UX IPSec does not support Kerberos for IKE authentication. You must configure the Microsoft connection security rule to use certificates or preshared keys for IKE authentication.

On HP-UX A.02.00 and A.02.01 systems, you specify the IKE authentication method using the – authentication option in the ipsec_config add ike command.

On HP-UX A.03.00 systems, you specify the IKE authentication method using the –local_method and –remote_method options in the ipsec_config add auth command.

IKE default algorithms

The Windows advanced firewall configures the IKE encryption and hash (integrity) algorithms as pairs in security methods. By default, the Windows configuration contains the following security methods:

 AES-128 encryption and SHA-1 integrity  AES-128 encryption and MD5 integrity

The HP-UX IPSec default IKE algorithms are 3DES encryption and MD5 integrity.

At least one Windows security method must match the HP-UX IKE parameters. To change or add IKE algorithms for Microsoft Windows, use the procedure described in “Configuring Windows IKE algorithms.” To change specify IKE algorithms for HP-UX IPSec, use the –encryption and –hash options in the ipsec_config add ike or ipsec_config add ikev1 command.

IKE aggressive mode

HP-UX supports aggressive mode (AM) to establish IKE Security Associations (SAs). IKE AM is an optional feature and is not supported on Windows.

IPsec default transform

The default Windows IPsec transform is ESP with AES-128 encryption and SHA-1 authentication. On HP-UX systems, you must explicitly specify this transform using the –action ESP_AES128_HMAC_SHA1 option in the ipsec_config add host command.

Configuration overview

To configure IPsec on Microsoft Windows Vista and Windows Server 2008 systems, you must complete the following tasks:

1. Configure IKE algorithms, if needed.

2. Configure connection security rules.

3. Configure firewall rules.

4. (Optional) Configure additional IPsec transforms.

Connection security rules specify IPsec parameters, including IKE and IPsec security algorithms, IKE authentication methods and IP address filters for the packets to be secured by IPsec. Depending on the method used to configure a rule, the rule can also specify protocols and port numbers for the filter.

Firewall rules are required to allow inbound and outbound packets when the Windows firewall is enabled. Firewall rules specify a packet filter and the action for packets that match the filter, such as requiring IPsec security to the packets.

Configuring a connection security rule is not sufficient to allow IPsec packets to pass in and out of a Windows system. If you have a connection security rule to secure packets between the local system and a remote system but do not have a firewall rule to allow packets to and from the remote system, the firewall will block the packets.

NOTE: Microsoft Windows Vista and Windows Server 2008 also support IPsec Policy Agent rules. These rules are functionally the same as the IPsec rules on Windows XP and Windows Server 2003 systems and can be configured using the IPsec Policy MMC snap-in as documented in Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec. Microsoft provides IPsec Policy Agent rules for backwards compatibility only and does not recommend the use of these rules on Windows Vista and Windows Server 2008 systems.

Configuring Windows IKE algorithms

The default HP-UX IKE algorithms do not match the default Windows IKE algorithms, so you must configure the IKE algorithms on either the Windows system or the HP-UX system to match the algorithms on the other system.

+ 5 hidden pages