HP HP-UX IPSec Setup and Install

HP-UX IPSec Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec

HP Part Number: J4256-90025 Published: June 2007 Edition: 1.0

About This Document.........................................................................................................9

Typographic Conventions......................................................................................................................9

Introduction..........................................................................................................................................11

Testing Environment.......................................................................................................................11

Known Problem with Windows 2000 SP1 and SP2...................................................................11

Protocol Implementation Differences..............................................................................................12

Windows IP Security Configuration Overview....................................................................................13

Configuring a Windows Host-to-Host Policy.......................................................................................14

Step 1: Starting the IP Security Policies Snap-in Configuration Utility...........................................15

Step 2: Creating a Policy..................................................................................................................15

Step 3: Adding a Rule......................................................................................................................16

Step 4: Creating the IP Filter List and Filters for the Rule...............................................................18

Step 5: Configuring Filter Actions for the Rule...............................................................................21

Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule..................25

Step 7: Configuring the Connection Type for the Rule...................................................................26

Step 8: Modifying IKE Parameters for the Policy............................................................................26

Step 9: Starting the IP Security Service............................................................................................29

Step 10: Assigning the IP Security Policy........................................................................................30

Step 11: Verifying the Configuration...............................................................................................31

Example...........................................................................................................................................31

Windows Configuration.............................................................................................................31

HP-UX Configuration................................................................................................................32

Additional Options...............................................................................................................32

Configuring a Windows End-to-End Tunnel Policy.............................................................................33

Outbound Tunnel Rule Requirements............................................................................................33

Inbound Tunnel Rule Requirements...............................................................................................33

Configuring a Tunnel Rule..............................................................................................................33

Example...........................................................................................................................................34

Windows Configuration.............................................................................................................34

Outbound Rule.....................................................................................................................34

Inbound Rule........................................................................................................................35

Additional Parameters..........................................................................................................36

HP-UX Configuration................................................................................................................37

Troubleshooting Tips............................................................................................................................38

Using IKE Logging on HP-UX Systems..........................................................................................38

Using IKE Logging on Windows Systems.......................................................................................38

Additional Windows Troubleshooting Tools..................................................................................39

Comparing HP-UX and Windows IPsec Configuration Parameters....................................................40

Mirrored Filters...............................................................................................................................41

Filter Selection.................................................................................................................................42

IKE Parameter Selection..................................................................................................................42

IKE SA Key (Master Key) Lifetime Values......................................................................................42

HP-UX IKE SA Lifetime Values.................................................................................................42

Windows IKE SA Lifetime Values..............................................................................................43

Maximum Quick Modes..................................................................................................................43

Perfect Forward Secrecy (PFS).........................................................................................................43

IPsec SA Key (Session Key) Lifetime Values...................................................................................43

HP-UX IPsec SA Lifetime Values...............................................................................................43

Windows IPsec SA Lifetime Values...........................................................................................44

Related Publications..............................................................................................................................45

Table of Contents 3

glossary.............................................................................................................................47

4 Table of Contents

List of Figures

1 IP Security Policy Wizard..............................................................................................................16

2 Rules Tab.......................................................................................................................................17

3 Rule Properties Dialog Box...........................................................................................................17

4 Creating an IP Filter List...............................................................................................................18

5 Address Tab for Filter Properties..................................................................................................19

6 Protocol Tab for Filter Properties..................................................................................................20

7 Selecting the Filter List for a Rule.................................................................................................21

8 Security Methods for Filter Action................................................................................................22

9 Security Method Dialog Box.........................................................................................................23

10 Custom Security Methods Settings Dialog Box............................................................................24

11 Selecting the Filter Action.............................................................................................................25

12 Configuring A Preshared Key.......................................................................................................26

13 General Policy Properties Dialog Box ..........................................................................................27

14 Key Exchange Settings Dialog Box ...............................................................................................28

15 IKE Security Algorithms Dialog Box ............................................................................................29

16 IPSEC Services Properties Dialog Box...........................................................................................30

17 Assigning the IP Security Policy...................................................................................................31

18 Outbound Rule Filter....................................................................................................................35

19 Outbound Rule Tunnel Settings....................................................................................................35

20 Inbound Rule Filter.......................................................................................................................36

21 Inbound Rule Tunnel Settings.......................................................................................................36

List of Tables

1 IPsec Parameters on Windows and HP-UX .................................................................................40

About This Document

This document describes how to configure Microsoft Windows IP Security to operate with the HP-UX IPSec product.

Typographic Conventions

This document uses the following typographical conventions:

%, $, or #

audit(5) A manpage. The manpage name is audit, and it is located in

Command Computer output

Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you

ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. [ERROR NAME] Key The name of a keyboard key. Return and Enter both refer to the

Term The defined use of an important word or phrase.

User input

Variable

[] The contents are optional in syntax. If the contents are a list

{} The contents are required in syntax. If the contents are a list

... The preceding element can be repeated an arbitrary number of

 Indicates the continuation of a code example. | Separates items in a list of choices. WARNING A warning calls attention to important information that if not

CAUTION A caution calls attention to important information that if not

IMPORTANT This alert provides essential information to explain a concept or

NOTE A note contains additional information to emphasize or

A percent sign represents the C shell system prompt. A dollar sign represents the system prompt for the Bourne, Korn, and POSIX shells. A number sign represents the superuser prompt.

Section 5. A command name or qualified command phrase. Text displayed by the computer.

must hold down the key labeled Ctrl while you press another key or mouse button.

The name of an error, usually returned in the errno variable.

same key.

Commands and other text that you type. The name of a placeholder in a command, function, or other

syntax display that you replace with an actual value.

separated by |, you must choose one of the items.

times.

understood or followed will result in personal injury or nonrecoverable system problems.

understood or followed will result in data loss, data corruption, or damage to hardware or software.

to complete a task

supplement important points of the main text.

Typographic Conventions 9

Introduction

This document contains the following sections:

• “Windows IP Security Configuration Overview” (page 13)

This section contains a brief overview of the Windows IPsec configuration parameters and the terminology used in the Windows IPsec configuration utilities.

• “Configuring a Windows Host-to-Host Policy” (page 14)

This section describes how to configure IP Security (IPsec) on a Windows client to secure IP packets sent to and received from an HP-UX system in a host-to-host topology.

• “Configuring a Windows End-to-End Tunnel Policy” (page 33)

This section describes how to configure IPsec on a Windows client to secure IP packets sent to and received from an HP-UX system in an end-to-end tunnel topology.

• “Troubleshooting Tips” (page 38)

This section contains troubleshooting tips.

• “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40)

This section compares how HP-UX and Windows systems configure and use IPsec parameters.

• “Related Publications” (page 45)

This section contains a list of related HP-UX and Microsoft publications.

The procedures and examples in this document use preshared keys for IKE authentication. For information about using certificates for IKE authentication with Microsoft Windows, see Using Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com.

The intended audience for this document is an HP-UX IPSec administrator who is familiar with the HP-UX IPSec product and with the IP Security protocol suite. If you are not familiar with the HP-UX IPSec product, see the appropriate version of the HP-UX IPSec Administrator's Guide, available at http://docs.hp.com.

NOTE: The IP Security protocol suite is often referred to as IPsec. The HP-UX product that implements the IP Security protocol suite is HP-UX IPSec.

Testing Environment

The procedures in this white paper were tested using the following environment:

Known Problem with Windows 2000 SP1 and SP2

For this white paper, HP did not test with Windows 2000 systems. However, there is a known problem with Windows 2000 base systems and Windows 2000 systems with Service Pack 1 (SP1) or Service Pack 2 (SP2). The IP Security module on these systems does not properly process IPSec ESP packets that are fragmented across IP packets and drops these packets. The symptoms vary according to how the applications handle the dropped packets.

This problem is caused by a defect in the Windows 2000 SP1/ SP2 software and is fixed in Windows 2000 Service Pack 3 (SP3).

DescriptionComponent

Versions A.02.01 and A.02.01.01HP-UX IPSec

Windows XP with Service Pack 2 (SP2)Microsoft Windows Client

Introduction 11

The above problem typically occurs with ESP-encrypted UDP or ICMP packets that are fragmented by IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You may still experience problems with ESP-encrypted TCP packets sent from an HP-UX system to a Windows 2000 system if an intermediate IP gateway fragments the ESP packet.

Protocol Implementation Differences

HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there are features in the protocol suite that HP-UX implemented which Microsoft did not implement, and vice-versa.

The following features are implemented by HP-UX IPSec version A.02.01 but not by Microsoft Windows XP:

• Advanced Encryption Standard (AES): HP-UX IPSec supports ESP encryption using the following protocols: AES, Triple Data Encryption Standard (3DES), and Data Encryption Standard (DES). Windows XP and Windows 2000 support 3DES and DES, but do not support AES.

• Aggressive Mode (AM): HP-UX supports AM exchanges to establish IKE Security Associations (SAs). AM is an optional feature and is not supported on Windows.

The following features are implemented by Microsoft Windows XP, but not by HP-UX IPSec version A.02.01:

• Kerberos: Windows supports Internet Key Exchange (IKE) authentication using Kerberos. RFC 2408 defines an optional Kerberos Token payload, but does not describe how to implement it. This feature is not supported on HP-UX.

• Perfect Forward Secrecy (PFS) for keys only: HP-UX IPSec supports PFS for keys in conjunction with PFS for all identities, but does not support PFS for keys only. Windows supports PFS for keys only (“session key PFS”) and PFS for keys in conjuctions with PFS for all identities (“master key PFS”). See “Perfect Forward Secrecy (PFS)” (page 43) for more information.

Windows IP Security Configuration Overview

On Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IP Security policy. You can create multiple IP Security policies, but only one local policy can be active on the system. If the system is a member of a Windows Active Directory domain, you can use an IP Security policy from a Group Policy defined for the domain.

A Windows IP Security policy defines the parameters used to negotiate Internet Key Exchange Security Associations (IKE SAs) and IPsec SAs. An IKE SA is a bi-directional, secure communication channel that two peers establish before negotiating IPSec SAs. One of the primary activities during the IKE SA negotiation is the authentication of each peer's identity.

After two peers establish an IKE SA, they can negotiate IPsec SAs. Each IPsec SA is a uni-directional, secure communication channel. The IPsec SA operating parameters include the IPsec protocol used (Encapsulating Security Payload, ESP, or Authentication Header, AH) and the cryptographic algorithms. IPsec SAs are negotiated in pairs (one for each direction of traffic).

Each Windows IP Security policy contains the following components:

• Rules

A policy contains one or more rules. The main purpose of a rule is to assign actions for address filters. Each rule contains the following components:

— IP Filter List

An IP Filter list contains one or more filters. Each filter contains the following components:

◦ Addressing

The source and destination IP addresses, network masks, and a flag that indicates if the filter is mirrored (bi-directional).

◦ Protocol

The upper-layer protocol, and source and destination ports, if applicable.

◦ Description

The filter name and a description.

— Filter Action

The filter action specifies the action to take for the rule, and can be one of the following actions:

◦ allow: allow the packet to pass ◦ block: discard the packet ◦ negotiate security: negotiate IPsec Authentication Header (AH) or Encapsulating

Security Payload (ESP) Security Associations (SAs)

— Authentication Methods

The authentication methods specify the type of Internet Key Exchange (IKE) authentication to use (preshared key or certificates with RSA signatures). If you are using preshared key authentication, the authentication methods also specify the value of the preshared key.

Windows IP Security Configuration Overview 13

— Tunnel Settings

The tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settings also specify the tunnel destination endpoint.

— Connection Type

The connection type specifies the connection (link) types for the rule, such as LAN.

• General

The general parameters for a policy specify IKE SA parameters, such as the IKE encryption algorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes. The parameters correspond to IKE SA proposals. You can configure multiple IKE SA proposals and specify the preference order. The proposals are used for all rules in the policy.

By comparison, a minimal HP-UX IPSec configuration consists of one or more IPsec host policies, one or more IKE policies, and one or more authentication records. The IPsec host policies specify address filters, and you can configure separate IKE policies for each peer. “Comparing HP-UX

and Windows IPsec Configuration Parameters” (page 40) lists IPsec configuration parameters

and how they are configured in the HP-UX IPSec and the Windows IP Security configuration utilities.

Configuring a Windows Host-to-Host Policy

This section describes one method for configuring host-to-host policy on a Windows XP client using the IP Security Policies snap-in utility. Windows also supports command-line utilities to configure IP Security policies: ipseccmd on Windows XP systems and netsh on Windows 2003 systems. For more information about these utilities, see the Windows documentation set.

To use this method, complete the following steps:

1. Start the IP Security Policies snap-in utility. See “Step 1: Starting the IP Security Policies

Snap-in Configuration Utility” (page 15).

2. Create an IP Security policy. See “Step 2: Creating a Policy” (page 15).

3. Add a rule to the policy. See “Step 3: Adding a Rule” (page 16).

4. Create a Filter List for the rule and configure filters. See “Step 4: Creating the IP Filter List

and Filters for the Rule” (page 18).

5. Configure filter actions for the rule. The filter actions contain IPsec transforms or other actions. See “Step 5: Configuring Filter Actions for the Rule” (page 21).

6. Configure the IKE authentication method and preshared key for the rule. See “Step 6:

Configuring the IKE Authentication Method and Preshared Key for the Rule” (page 25).

7. Specify the network link (connection) types for the rule. See“Step 7: Configuring the

Connection Type for the Rule” (page 26).

8. Modify the IKE SA parameters for the policy. By default, Windows clients will use IKE SA parameters that are compatible with the default HP-UX IPSec parameters. If these parameters are acceptable, you can skip this step. See “Step 8: Modifying IKE Parameters for the Policy”

(page 26).

9. Start the IP Security service. The IP Security service must be running before you can assign the new IP Security policy. See “Step 9: Starting the IP Security Service” (page 29).

10. Assign (activate) the new IP Security Policy. See “Step 10: Assigning the IP Security Policy”

(page 30).

11. Verify the configuration. See “Step 11: Verifying the Configuration” (page 31).

Because this is a host-to-host rule, we will use the default value for the rule tunnel setting (no tunnel). For information about configuring a tunnel rule and the tunnel setting, see “Configuring

a Windows End-to-End Tunnel Policy” (page 33).

Step 1: Starting the IP Security Policies Snap-in Configuration Utility

Use the following procedure to start the IP Security Policies configuration utility. This utility is a snap-in module for the Microsoft Management Console (MMC).

1. Start the Microsoft Management Console (MMC). From the Microsoft Start menu, click Run and type MMC. Click OK.

2. If the IP Security Policies snap-in configuration utility is not loaded, use the following procedure to add it:

a. From the MMC window, click File→Add/Remove Snap-in. b. From the Add/Remove Standalone Snap-in window, click Add. c. From the Add Standalone Snap-in window, scroll down to IP Security Policy

Management and select it. Click Add.

d. In the Select Computer or Domain window, select Local computer (in this procedure,

we are configuring IP Security for the local computer). Click Finish.

e. Close the Add Standalone Snap-in window by clicking Close. f. Close the Add/Remove Snap-in window by clicking OK.

Step 2: Creating a Policy

Use the following procedure to create an IP Security policy. An IP Security policy is a set of IPsec configuration parameters. Only one local IP Security policy can be active (assigned) on a system.

1. In the left navigation pane of the IP Security Policy Management snap-in, click IP Security Policies on Local Computer to display all IP Security Policies. Depending on your Windows

platform, there may be IP Security Policies already configured.

2. Right click IP Security Policies on Local Computer and select Create IP Security Policy.

3. The Policy Wizard starts and displays a startup message. Click Next.

4. The Policy Wizard opens the IP Security Policy Name window. Enter a name in the Name

field. This name is used only for internal identification.

Click Next.

5. The Policy Wizard opens the Requests for Secure Communication window. Clear the Activate the default response rule check box, as shown in Figure 1. (The default response rule is a

pre-configured rule that causes the Windows system to dynamically build a filter list based on the receipt of IKE requests. By default, the Windows system attempts to use IPsec only if it receives an IKE request from a remote system.)

Click Next.

Configuring a Windows Host-to-Host Policy 15

Figure 1 IP Security Policy Wizard

6. The Policy Wizard opens the Completing the IP Security policy wizard window. Select the

Edit properties check box if it is not already selected.

Click Finish.

The IP Security configuration utility opens the Policy Properties dialog box. The title of the window will be name Policy, where name is the policy name.

Step 3: Adding a Rule

The primary purpose of a rule is to assign actions to filters. A rule also specifies IKE authentication methods. Use the following procedure to add a rule to the IP Security policy:

1. Select the Rules tab in the Policy Properties dialog box.

Clear the Use Add Wizard box check box if it is selected (Figure 2).

Click Add.

Figure 2 Rules Tab

2. The IP Security configuration utility opens the Rule Properties dialog box, which has a tab for each category of rule configuration data: IP Filter List, Filter Action, Authentication Methods, Tunnel Setting, and Connection Type ( Figure 3).

Figure 3 Rule Properties Dialog Box

Configuring a Windows Host-to-Host Policy 17

TIP: After you have created a rule, you can open the Rules Properties dialog box by right clicking the rule and selecting Properties.

Step 4: Creating the IP Filter List and Filters for the Rule

An IP filter list can contain one or more filters. IPsec uses the filters to determine which rule to apply to an IP packet. The IP Security configuration utility displays the rules for a policy in reverse alphabetical order based on the name of the IP filter list for the rule.

Filter Order There is no method for specifying the search or priority order for the filters in a rule or for the order of rules in a policy. The Windows IP Security module automatically creates an internal filter list and orders the filters from most specific to least specific.

Use the following procedure to configure the IP filter list and a filter:

1. Create an IP filter list.

Select the IP Filter tab from the Rule Properties dialog box.

The IP Filter List tab shows a list of filters already defined for IP Security policies. Each rule can have only one filter list, but the filter list can specify multiple filters. In this example, we will create a new filter list that contains one filter.

Click Add at the bottom of the dialog box.

The IP Security configuration utility opens the IP Filter List dialog box. In the Name field, enter a name for the filter list. This name is used only for internal identification. Optionally, add a description. In Figure 4, the administrator enters the name foo.

Clear the Use Add Wizard box check box if it is selected.

Click Add.

Figure 4 Creating an IP Filter List

The IP Security configuration utility opens a Filter Properties dialog box.

2. Select the Addressing tab in the Filter Properties dialog box. Use the drop-down menus to specify the address types for the source and destination addresses. The selections are:

• My IP Address

• Any IP Address

• A specific DNS Name

• A specific IP Address

• A specific IP Subnet Enter the source and destination IP addresses or DNS names for the filter. If you selected A

specific IP Subnet, enter the subnet mask.

WARNING! Be careful when configuring filters that affect packets required for basic network operation, such as packets exchanged with DNS servers and ICMP packets exchanged with routers. If you configure a policy that requires IP Security for these packets and the remote node does not support IP Security, your system can lose network functionality.

Leave the Mirrored check box selected, which creates a bi-directional filter that applies to packets to and from the destination system. See “Mirrored Filters” (page 41) for more information about mirrored filters.

In Figure 5, the administrator specifies an address filter with the Windows system address (10.1.1.1) as the source address and the HP-UX system address (10.2.2.2) as the destination address. The Mirrored check box is selected, so the address filter also matches packets from the HP-UX system.

Figure 5 Address Tab for Filter Properties

3. Select the Protocol tab in the Filter Properties dialog box. By default, the filter applies to all protocol types. Select the protocol type (for example, TCP) from the drop-down box. If you select TCP or UDP, you can also specify the From (source) port and To (destination) port.

Click OK to return to the Filter Properties dialog box.

1. HP-UX did not test the My IP Address selection with multihomed Windows systems. However, the Windows

documentation states that in a multi-homed system, My IP Address matches every IP address on the system.

Configuring a Windows Host-to-Host Policy 19

In Figure 6, the administrator specifies protocol information for a Windows system that will be a telnet client. The protocol type is TCP, the source port is a wildcard (any port), and the destination port is the IANA registered TCP port number for the telnet service, 23.

Figure 6 Protocol Tab for Filter Properties

4. From the IP Filter List dialog box, you can add another filter to the filter list by clicking the Add button.

Click OK in the IP Filter List dialog box to return to the IP Filter List tab in the Rule Properties dialog box.

5. Add the filter list to the rule by selecting the option button for the filter list you just created. In Figure 7, the administrator added the filter list foo for the rule.

Figure 7 Selecting the Filter List for a Rule

Step 5: Configuring Filter Actions for the Rule

The filter action specifies the action to take for the rule, such as allow (pass), block (discard), or negotiate security (negotiate IPsec AH or ESP Security Associations). If you select negotiate security, the filter action also specifies parameters for IPsec Security Association (SA) proposals: ESP or AH transforms and IPSec SA key lifetimes. A rule can have only one filter action, but the filter action can specify multiple IPsec SA proposals. You can specify the order for the IPsec SA proposals.

The filter actions you configure in the Windows IP Security rule must be compatible with the value or values specified for the -action argument in the HP-UX ipsec_config add host or add tunnel command.

Use the following procedure to configure filter actions:

1. Select the Filter Action tab from the Rule Properties dialog box.

The Filter Action tab shows a list of filter actions already defined for IP Security. In this procedure, we will create a new filter action.

Clear the Use Add Wizard check box if it is selected and click Add.

2. The IP Security configuration utility opens the Filter Action Properties dialog box with the following tabs:

Configuring a Windows Host-to-Host Policy 21

Security Methods•

• General

Select the Security Methods tab, then select Negotiate security. Verify that the following check boxes are not selected:

• Accept unsecured communication, but always respond using IPSec.

• Allow unsecured communication with non-IPSec-aware computer.

In addition, verify that the Session key perfect forward secrecy (PFS) check box is not selected. (HP-UX does not support session key PFS, also referred to as PFS for keys only. HP-UX supports PFS for keys only in conjunction with PFS for identities. See “Perfect Forward

Secrecy (PFS)” (page 43) for more information.)

For example:

Figure 8 Security Methods for Filter Action

Click Add.

The IP Security configuration utility opens the Security Method dialog box (Figure 9):

2. HP-UX IPSec does not have options that are equivalent to these check boxes. If an HP-UX IPsec policy requires IP security, then HP-UX always requires IP security for packets that match the policy and drops any packets that match the policy but are not secured.

Figure 9 Security Method Dialog Box

The Encryption and Integrity and Integrity only methods each correspond to a set of predefined parameters for an IPsec SA proposal, including an IPsec transform type (such as ESP). The transforms and additional SA parameters defined for these methods may vary according to the Windows release installed. On Windows XP systems with SP2, these methods are defined as follows:

• Encryption and Integrity

Authenticated ESP using 3DES encryption and SHA1 authentication (this is equivalent to the HP-UX IPSec transform ESP_3DES_HMAC_SHA1). No SA lifetimes are specified, and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA

Key (Session Key) Lifetime Values” (page 43) for more information).

• Integrity only

Authenticated ESP using NULL encryption and SHA1 authentication. (this is equivalent to the HP-UX IPSec transform ESP_NULL_HMAC_SHA1) No SA lifetimes are specified, and these settings are compatible with the HP-UX default SA lifetimes (see “IPsec SA

Key (Session Key) Lifetime Values” (page 43) for more information).

To use a predefined method, select the appropriate method from the list and click OK to return to the Security Methods tab in the Filter Actions dialog box.

To create a custom method, use the following procedure: a. Select Custom in the Security Method dialog box, then click Settings to open the

Custom Security Method Settings dialog box.

b. In the Custom Security Method Settings dialog box (Figure 10), select the appropriate

transform type, algorithms, and session key lifetimes. See “IPsec SA Key (Session Key)

Lifetime Values” (page 43) for more information about IPsec session key lifetimes.

Configuring a Windows Host-to-Host Policy 23

Figure 10 Custom Security Methods Settings Dialog Box

c. Click OK to return to the Security Methods tab in the Filter Actions dialog box.

If the parameters you configured for a custom method match a predefined method, the configuration utility will display an informative message and select the matching predefined method.

3. From the Security Methods tab, you can add more methods (IPsec SA proposals) by clicking

the Add button. If you have multiple IPsec SA proposals, the configuration utility lists them in the preference order IKE will use when negotiating IPsec SAs. You can change the order by using the Move up and Move down buttons. You can also use the Delete button to delete IPsec SA proposals (such as proposals that use DES, which has been cracked—data encrypted using DES has been decrypted by unauthorized parties) .

4. (Optional) Configure the action name. When you create a new action (transform), the

configuration utility assigns it the name New Filter Action. To change the name, select the General tab from the Action Properties dialog box. Enter a new name and click OK to return to the Filter Action tab in the Rule Properties dialog box.

5. Apply the new action to the rule. In the Filter Action tab, click on the option button for the

filter action you just created to apply the action to the rule you are configuring. For example, in Figure 11, the administrator created the action my_action and selected it for the rule.

Click Apply.

Figure 11 Selecting the Filter Action

Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule

When configuring a rule to be compatible with HP-UX IPSec, the authentication method specifies the IKE authentication method (preshared key or certificates) for IPsec. The authentication method must match the value specified for the -authentication argument in the ipsec_config add ike command.

Windows also allows you to configure Kerberos (Active Directory) as an authentication method for IKE (this is the default), but HP-UX does not support this authentication method.

Use the following procedure to configure the IKE authentication method:

1. Select the Authentication Methods tab from the Rule Properties dialog box.

2. Click Add to open the Authentication Method dialog box.

3. To use IKE authentication with a preshared key, select Use this string. This is equivalent

to specifying -authentication PSK in the ipsec_config add ike command.

Enter the preshared key as ASCII text. Do not enclose the key in double quotes. The preshared key must match the preshared key on the HP-UX system, which is configured using the

-preshared argument in the ipsec_config add auth command. For example:

Configuring a Windows Host-to-Host Policy 25

Figure 12 Configuring A Preshared Key

To use IKE authentication with certificates, select Use a certificate from this certification authority (CA). Click Browse. The IP Security configuration utility opens a Select Certificate

box with a list of CA certificates stored on your system. Select the CA for the appropriate CA and click OK. (For additional information about configuring Microsoft Windows certificates, see Using Microsoft Windows Certificates with HP-UX IPSec, available at

http://docs.hp.com.

4. After you have specified the IKE authentication method, click OK to return to the

Authentication Methods tab in the Rule Properties dialog box.

5. In the Rule Properties dialog box, remove the Kerberos authentication method from the

authentication methods list by highlighting it and clicking Remove.

The configuration utility will display a confirmation message (Are you sure?). Click Yes

Step 7: Configuring the Connection Type for the Rule

The connection type specifies the types of network connection to which the rule will apply. By default, the IP Security configuration utility creates rules that apply to all network connection types. To change the connection type, use the following procedure:

1. Select the Connection Type tab from the Rule Properties dialog box.

2. The IP Security configuration utility opens the Connection Type dialog box with the following

selections:

• All network connections: the rule applies to all network connections

• Local area network (LAN): the rule applies only to LAN connections

• Remote access: the rule applies only to VPN and dial-up connections

Select the appropriate connection type and click OK. If you have configured all the required parameters for a rule, the IP Security configuration utility will return to the Policy Properties dialog box.

Step 8: Modifying IKE Parameters for the Policy

By default, HP-UX IPSec negotiates IKE SAs using a single proposal with the following parameters:

• Encryption algorithm: 3DES

• Hash algorithm: MD5

• Diffie-Hellman Group: 2

• Maximum lifetime: 28,800 seconds (8 hours)

• Maximum Quick Modes: 100

You can specify alternative values for the above parameters in the ipsec_config add ike command.

On Windows XP systems with SP2, IP Security policies are pre-configured with four IKE SA proposals. The second IKE proposal matches the default HP-UX IPSec IKE proposal3, and will be used by the two systems if no changes are made to the default configuration data. If these IKE parameters meet your security requirements, you do not need to modify the IKE parameters and can skip to “Step 10: Assigning the IP Security Policy” (page 30).

Use the following procedure to modify the Windows IKE SA parameters:

1. From the Policy Properties dialog box, select the General tag. The IP Security configuration utility opens the General dialog box (Figure 13).

Click Advanced4. (Ignore the field labeled Check for policy changes. This field is used only when the policy is stored in an Active Directory.)

Figure 13 General Policy Properties Dialog Box

2. The IP Security configuration utility opens the Key Exchange Settings dialog box (Figure 14).

3. By default, the first Windows XP proposal has the following parameters: Encryption - 3DES; Hash - SHA1;

Diffie-Hellman Group - 2. The third and fourth Windows proposals are weaker, and use DES encryption and Diffie-Hellman Group 1. Refer to the Windows documentation for more information.

4. On Windows 2003 servers, this button is labeled Settings.

Configuring a Windows Host-to-Host Policy 27

Figure 14 Key Exchange Settings Dialog Box

Configure the fields as follows:

• Master key perfect forward secrecy (PFS)

Selecting this check box sets the maximum number of IPsec or Quick Mode (QM) negotiations that IKE can perform using an IKE SA to 1. It is equivalent to specifying

-maxqm 1 in the ipsec_config add ike command. PFS is computationally expensive and HP recommends that you enable it only in hostile environments. See

“Maximum Quick Modes” (page 43) and “Perfect Forward Secrecy (PFS)” (page 43)

for more information.

• Authenticate and generate a new key after every: ____ minutes

This field specifies the maximum lifetime for an IKE SA in units of time. It is equivalent to the -life argument of the ipsec_add ike command.

TIP: Note that this value is specified in minutes on Windows systems and in seconds on HP-UX systems.

• Authenticate and generate a new key after every: ____ sessions

This field specifies the maximum QM negotiations per IKE SA. It is equivalent to the

-maxqm argument of the ipsec_add ike command. See “Maximum Quick Modes”

(page 43) and “Perfect Forward Secrecy (PFS)” (page 43) for more information.

Modify the values as appropriate. If you do not want to modify the IKE encryption, hash, or Diffie-Hellman Group parameters, click OK to return to the General dialog box and continue to the next step.

To modify IKE encryption algorithm, hash algorithm, or Diffie-Hellman Group parameters, click Methods. The IP Security configuration utility opens the Key Exchange Security Methods dialog box, which lists IKE algorithms and Diffie-Hellman Groups that correspond to IKE SA proposals in order of preference. You can change the order by using the Move up and Move down buttons. You can also delete IKE SA proposals (such as proposals that use DES, which has been cracked) using the Delete button.

To add another IKE SA proposal (method), click Add.

The IP Security configuration utility opens the IKE Security algorithms dialog box (Figure 15).

Figure 15 IKE Security Algorithms Dialog Box

Use the drop-down menus to select the appropriate integrity algorithm, encryption algorithm, and Diffie-Hellman Group (these are equivalent to the -hash, -encryption, and -group arguments of the ipsec_config add ike command).

3. Click OK to return to the Key Exchange Security Methods dialog box.

Click OK to return to the Key Exchange Settings dialog box.

Click Close to close the Policy Properties dialog box.

Step 9: Starting the IP Security Service

Use the following procedure to start the IP Security service. The IP Security service must be running before you can assign a policy.

1. From the Microsoft Start menu, select Control Panel→Administrative Tools→Services.

2. Scroll down and select IPSEC Services.

3. The Service manager opens the IPSEC Services Properties dialog box (Figure 16).

In the Startup type selection menu, select Automatic.

If the Service status is not Started, click Star t.

Click OK to close the IPSEC Services Properties dialog box.

Close the Services dialog box.

Configuring a Windows Host-to-Host Policy 29

Figure 16 IPSEC Services Properties Dialog Box

Alternatively, you can manually start the IP Security service by entering the following Windows command:

net start policyagent

You can also use the following sequence of commands to manually stop and restart the IP Security service. This also clears any existing IPsec SAs,:

net stop policyagent

net start policyagent

Step 10: Assigning the IP Security Policy

The IP Security subsystem will not use the new policy until you assign (activate) it. Only one IP Security policy can be assigned or active for the system. To assign the new IP Security policy, return to the MMC window. Right click the policy in the MMC window and select Assign, as shown in Figure 17.

Figure 17 Assigning the IP Security Policy

Step 11: Verifying the Configuration

To verify your configuration, generate traffic that matches the address filter.

On the HP-UX system, enter the following command to verify that the IKE SA and IPsec SAs are established:

ipsec_report -sa

Example

In this example, IPsec secures telnet connections from the Windows system to the HP-UX system, using authenticated ESP.

The Windows system's address is 10.1.1.1

The HP-UX system's address is 10.2.2.2.

Windows Configuration

The Windows administrator configures and assigns an IP Security policy with the following parameters:

• One rule, with the following parameters: — Filter List: One filter, with the following parameters:

◦ Addressing:

– Source address: the Windows system's address. – Destination address: the HP-UX system's address. – Mirrored: yes (the Mirrored box is selected). These parameters are shown in Figure 5 (page 19).

◦ Protocol: TCP; source port any, destination port 23 (telnet).

– Protocol: TCP – From port: any – To port: 23 (telnet server) These parameters are shown in Figure 6 (page 20).

— Filter Action: Negotiate security, using the default settings for Encryption and Integrity

(authenticated ESP using 3DES and SHA1).

— Authentication Method: IKE using the preshared key my_preshared_key, as shown

in Figure 12 (page 26). — Tunnel Settings: No tunnel (this is the default). — Connection Type: All network connections (this is the default).

• General parameters: The general parameters for the policy are set to the default values (four IKE SA proposals, including 3DES encryption, SHA1 integrity and Diffie-Hellman Group

2).

Configuring a Windows Host-to-Host Policy 31

HP-UX Configuration

On the HP-UX system, the administrator configures the following policies and records:

ipsec_config add host telnet_from_foo1 \

-source 10.2.2.2/32/TELNET -destination 10.1.1.1 \

-action ESP_3DES_HMAC_SHA1

ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK

ipsec_config add auth foo1 -remote 10.1.1.1 \

-psk my_preshared_key

If the HP-UX IPSec subsystem is not already started, the administrator starts it using the ipsec_admin -start command.

Additional Options

For information on additional options and commands, see the HP-UX IPSec Administrator's Guide and the following manpages:

ipsec_admin(1M) ipsec_config(1M) ipsec_policy(1M) ipsec_report(1M)

Configuring a Windows End-to-End Tunnel Policy

The only IPsec tunnel topology supported between an HP-UX system and a Windows system is an end-to-end tunnel.5The procedure for configuring an end-to-end tunnel policy on Windows system is the same as procedure for configuring a host policy, except that you must configure two, non-mirrored rules: one rule for outbound packets and one rule for inbound packets, as described in the sections that follow.

NOTE: Do not configure any other rules in the policy with the HP-UX system address as the destination address. This prevents the Microsoft system from applying the tunnel transform over a host-to-host (transport) transform. In end-to-end tunnel topologies, HP-UX IPSec does not support transport transforms over a tunnel transform.

Outbound Tunnel Rule Requirements

The outbound tunnel rule must have the following parameters:

• Filter List: One filter, with the following parameters: — Address:

◦ Source address: the HP-UX system's address. ◦ Destination address: this must be a specific IP address and must be the Windows

system's address.

◦ Mirrored: no (the Mirrored box is cleared).

— Protocol Type: none (wildcard). The Windows documentation states that the filters in

tunnel rules must not specify protocols or ports to ensure that IP Security can correctly process IP fragments.

• Tunnel Setting — Tunnel endpoint: the HP-UX system's address. This is the address of the tunnel endpoint

closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter.

Inbound Tunnel Rule Requirements

The inbound tunnel rule must have the following parameters:

• Filter List: One filter, with the following parameters: — Address:

◦ Source address: the Windows system's address. ◦ Destination address: this must be a specific IP address and must be the HP-UX

system's address.

◦ Mirrored: no (the Mirrored box is cleared).

— Protocol Type: none (wildcard).

• Tunnel Setting — Tunnel endpoint: the Windows system's address. This is the address of the tunnel

endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter

Configuring a Tunnel Rule

Use the following procedure to configure an outbound or inbound tunnel rule.

5. You can also configure an IPsec topology where packets exchanged between an HP-UX system and a Windows

system are tunneled through an IPsec gateway device, but neither HP-UX nor Windows systems can be configured as IPsec gateways. The only topology in which an HP-UX system can act as an IPsec gateway is when the HP-UX system is a Home Agent for Mobile IPv6 clients. The HP-UX IPSec Administrator's Guide describes how to configure a host-to-gateway IPsec topology using HP-UX and a Cisco router.

Configuring a Windows End-to-End Tunnel Policy 33

TIP: The tunnel setting is used by all packets selected using the address filters for the rule. Do not include any filters for host-to-host (non-tunneled) packets in the filter list for a rule with a tunnel.

1. Start the IP Security Policies snap-in if necessary.

2. Create an IP Security policy or modify an existing policy. To modify an existing policy, select

the policy in the right navigation pane and right click the policy. Select Properties.

3. The IP Security configuration utility opens the Policy Properties dialog box. Select the Rules tab. Click Add to create a new rule or select a rule you want to modify and click Edit.

4. Configure a new rule or modify an existing rule with the appropriate address filter for the outbound tunnel rule or inbound tunnel rule, as described in “Outbound Rule” (page 34) or “Inbound Rule” (page 35). See “Step 4: Creating the IP Filter List and Filters for the Rule”

(page 18) if you need additional information about configuring address filters.

Record the destination address; you will need it to configure the tunnel endpoint.

5. Return to the Rule Properties dialog box. Select the Tunnel Setting tab.

6. The IP Security configuration utility opens the Tunnel Setting dialog box.

Select The tunnel endpoint is specified by this IP address.

Enter the IP address of the tunnel endpoint closest to the destination. Since this is an end-to-end tunnel, it is the same as the destination address in the address filter.

7. Click Close to close the Tunnel Setting dialog box.

8. If this is a new rule, complete the configuration by configuring the appropriate filter action,

authentication methods, and connection type.

Click Close to close the Rule Properties dialog box.