HP HP-UX IPSec Setup and Install
HP-UX IPSec
Configuring Microsoft Windows IP Security
to Operate with HP-UX IPSec
HP Part Number: J4256-90025
Published: June 2007
Edition: 1.0
2
Table of Contents
About This Document.........................................................................................................9
Typographic Conventions......................................................................................................................9
Introduction..........................................................................................................................................11
Testing Environment.......................................................................................................................11
Known Problem with Windows 2000 SP1 and SP2...................................................................11
Protocol Implementation Differences..............................................................................................12
Windows IP Security Configuration Overview....................................................................................13
Configuring a Windows Host-to-Host Policy.......................................................................................14
Step 1: Starting the IP Security Policies Snap-in Configuration Utility...........................................15
Step 2: Creating a Policy..................................................................................................................15
Step 3: Adding a Rule......................................................................................................................16
Step 4: Creating the IP Filter List and Filters for the Rule...............................................................18
Step 5: Configuring Filter Actions for the Rule...............................................................................21
Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule..................25
Step 7: Configuring the Connection Type for the Rule...................................................................26
Step 8: Modifying IKE Parameters for the Policy............................................................................26
Step 9: Starting the IP Security Service............................................................................................29
Step 10: Assigning the IP Security Policy........................................................................................30
Step 11: Verifying the Configuration...............................................................................................31
Example...........................................................................................................................................31
Windows Configuration.............................................................................................................31
HP-UX Configuration................................................................................................................32
Additional Options...............................................................................................................32
Configuring a Windows End-to-End Tunnel Policy.............................................................................33
Outbound Tunnel Rule Requirements............................................................................................33
Inbound Tunnel Rule Requirements...............................................................................................33
Configuring a Tunnel Rule..............................................................................................................33
Example...........................................................................................................................................34
Windows Configuration.............................................................................................................34
Outbound Rule.....................................................................................................................34
Inbound Rule........................................................................................................................35
Additional Parameters..........................................................................................................36
HP-UX Configuration................................................................................................................37
Troubleshooting Tips............................................................................................................................38
Using IKE Logging on HP-UX Systems..........................................................................................38
Using IKE Logging on Windows Systems.......................................................................................38
Additional Windows Troubleshooting Tools..................................................................................39
Comparing HP-UX and Windows IPsec Configuration Parameters....................................................40
Mirrored Filters...............................................................................................................................41
Filter Selection.................................................................................................................................42
IKE Parameter Selection..................................................................................................................42
IKE SA Key (Master Key) Lifetime Values......................................................................................42
HP-UX IKE SA Lifetime Values.................................................................................................42
Windows IKE SA Lifetime Values..............................................................................................43
Maximum Quick Modes..................................................................................................................43
Perfect Forward Secrecy (PFS).........................................................................................................43
IPsec SA Key (Session Key) Lifetime Values...................................................................................43
HP-UX IPsec SA Lifetime Values...............................................................................................43
Windows IPsec SA Lifetime Values...........................................................................................44
Related Publications..............................................................................................................................45
Table of Contents 3
glossary.............................................................................................................................47
4 Table of Contents
List of Figures
1 IP Security Policy Wizard..............................................................................................................16
2 Rules Tab.......................................................................................................................................17
3 Rule Properties Dialog Box...........................................................................................................17
4 Creating an IP Filter List...............................................................................................................18
5 Address Tab for Filter Properties..................................................................................................19
6 Protocol Tab for Filter Properties..................................................................................................20
7 Selecting the Filter List for a Rule.................................................................................................21
8 Security Methods for Filter Action................................................................................................22
9 Security Method Dialog Box.........................................................................................................23
10 Custom Security Methods Settings Dialog Box............................................................................24
11 Selecting the Filter Action.............................................................................................................25
12 Configuring A Preshared Key.......................................................................................................26
13 General Policy Properties Dialog Box ..........................................................................................27
14 Key Exchange Settings Dialog Box ...............................................................................................28
15 IKE Security Algorithms Dialog Box ............................................................................................29
16 IPSEC Services Properties Dialog Box...........................................................................................30
17 Assigning the IP Security Policy...................................................................................................31
18 Outbound Rule Filter....................................................................................................................35
19 Outbound Rule Tunnel Settings....................................................................................................35
20 Inbound Rule Filter.......................................................................................................................36
21 Inbound Rule Tunnel Settings.......................................................................................................36
5
6
List of Tables
1 IPsec Parameters on Windows and HP-UX .................................................................................40
7
8
About This Document
This document describes how to configure Microsoft Windows IP Security to operate with the
HP-UX IPSec product.
Typographic Conventions
This document uses the following typographical conventions:
%, $, or #
audit(5) A manpage. The manpage name is audit, and it is located in
Command
Computer output
Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you
ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH.
[ERROR NAME]
Key The name of a keyboard key. Return and Enter both refer to the
Term The defined use of an important word or phrase.
User input
Variable
[] The contents are optional in syntax. If the contents are a list
{} The contents are required in syntax. If the contents are a list
... The preceding element can be repeated an arbitrary number of
Indicates the continuation of a code example.
| Separates items in a list of choices.
WARNING A warning calls attention to important information that if not
CAUTION A caution calls attention to important information that if not
IMPORTANT This alert provides essential information to explain a concept or
NOTE A note contains additional information to emphasize or
A percent sign represents the C shell system prompt. A dollar
sign represents the system prompt for the Bourne, Korn, and
POSIX shells. A number sign represents the superuser prompt.
Section 5.
A command name or qualified command phrase.
Text displayed by the computer.
must hold down the key labeled Ctrl while you press another
key or mouse button.
The name of an error, usually returned in the errno variable.
same key.
Commands and other text that you type.
The name of a placeholder in a command, function, or other
syntax display that you replace with an actual value.
separated by |, you must choose one of the items.
separated by |, you must choose one of the items.
times.
understood or followed will result in personal injury or
nonrecoverable system problems.
understood or followed will result in data loss, data corruption,
or damage to hardware or software.
to complete a task
supplement important points of the main text.
Typographic Conventions 9
10
Introduction
This document contains the following sections:
• “Windows IP Security Configuration Overview” (page 13)
This section contains a brief overview of the Windows IPsec configuration parameters and
the terminology used in the Windows IPsec configuration utilities.
• “Configuring a Windows Host-to-Host Policy” (page 14)
This section describes how to configure IP Security (IPsec) on a Windows client to secure
IP packets sent to and received from an HP-UX system in a host-to-host topology.
• “Configuring a Windows End-to-End Tunnel Policy” (page 33)
This section describes how to configure IPsec on a Windows client to secure IP packets sent
to and received from an HP-UX system in an end-to-end tunnel topology.
• “Troubleshooting Tips” (page 38)
This section contains troubleshooting tips.
• “Comparing HP-UX and Windows IPsec Configuration Parameters” (page 40)
This section compares how HP-UX and Windows systems configure and use IPsec parameters.
• “Related Publications” (page 45)
This section contains a list of related HP-UX and Microsoft publications.
The procedures and examples in this document use preshared keys for IKE authentication. For
information about using certificates for IKE authentication with Microsoft Windows, see Using
Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com.
The intended audience for this document is an HP-UX IPSec administrator who is familiar with
the HP-UX IPSec product and with the IP Security protocol suite. If you are not familiar with
the HP-UX IPSec product, see the appropriate version of the HP-UX IPSec Administrator's Guide,
available at http://docs.hp.com.
NOTE: The IP Security protocol suite is often referred to as IPsec. The HP-UX product that
implements the IP Security protocol suite is HP-UX IPSec.
Testing Environment
The procedures in this white paper were tested using the following environment:
Known Problem with Windows 2000 SP1 and SP2
For this white paper, HP did not test with Windows 2000 systems. However, there is a known
problem with Windows 2000 base systems and Windows 2000 systems with Service Pack 1 (SP1)
or Service Pack 2 (SP2). The IP Security module on these systems does not properly process IPSec
ESP packets that are fragmented across IP packets and drops these packets. The symptoms vary
according to how the applications handle the dropped packets.
This problem is caused by a defect in the Windows 2000 SP1/ SP2 software and is fixed in Windows
2000 Service Pack 3 (SP3).
DescriptionComponent
Versions A.02.01 and A.02.01.01HP-UX IPSec
Windows XP with Service Pack 2 (SP2)Microsoft Windows Client
Introduction 11
The above problem typically occurs with ESP-encrypted UDP or ICMP packets that are fragmented
by IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You may
still experience problems with ESP-encrypted TCP packets sent from an HP-UX system to a
Windows 2000 system if an intermediate IP gateway fragments the ESP packet.
Protocol Implementation Differences
HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there
are features in the protocol suite that HP-UX implemented which Microsoft did not implement,
and vice-versa.
The following features are implemented by HP-UX IPSec version A.02.01 but not by Microsoft
Windows XP:
• Advanced Encryption Standard (AES): HP-UX IPSec supports ESP encryption using the
following protocols: AES, Triple Data Encryption Standard (3DES), and Data Encryption
Standard (DES). Windows XP and Windows 2000 support 3DES and DES, but do not support
AES.
• Aggressive Mode (AM): HP-UX supports AM exchanges to establish IKE Security
Associations (SAs). AM is an optional feature and is not supported on Windows.
The following features are implemented by Microsoft Windows XP, but not by HP-UX IPSec
version A.02.01:
• Kerberos: Windows supports Internet Key Exchange (IKE) authentication using Kerberos.
RFC 2408 defines an optional Kerberos Token payload, but does not describe how to
implement it. This feature is not supported on HP-UX.
• Perfect Forward Secrecy (PFS) for keys only: HP-UX IPSec supports PFS for keys in
conjunction with PFS for all identities, but does not support PFS for keys only. Windows
supports PFS for keys only (“session key PFS”) and PFS for keys in conjuctions with PFS for
all identities (“master key PFS”). See “Perfect Forward Secrecy (PFS)” (page 43) for more
information.
12
Windows IP Security Configuration Overview
On Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IP
Security policy. You can create multiple IP Security policies, but only one local policy can be
active on the system. If the system is a member of a Windows Active Directory domain, you can
use an IP Security policy from a Group Policy defined for the domain.
A Windows IP Security policy defines the parameters used to negotiate Internet Key Exchange
Security Associations (IKE SAs) and IPsec SAs. An IKE SA is a bi-directional, secure
communication channel that two peers establish before negotiating IPSec SAs. One of the primary
activities during the IKE SA negotiation is the authentication of each peer's identity.
After two peers establish an IKE SA, they can negotiate IPsec SAs. Each IPsec SA is a
uni-directional, secure communication channel. The IPsec SA operating parameters include the
IPsec protocol used (Encapsulating Security Payload, ESP, or Authentication Header, AH) and
the cryptographic algorithms. IPsec SAs are negotiated in pairs (one for each direction of traffic).
Each Windows IP Security policy contains the following components:
• Rules
A policy contains one or more rules. The main purpose of a rule is to assign actions for
address filters. Each rule contains the following components:
— IP Filter List
An IP Filter list contains one or more filters. Each filter contains the following
components:
◦ Addressing
The source and destination IP addresses, network masks, and a flag that indicates
if the filter is mirrored (bi-directional).
◦ Protocol
The upper-layer protocol, and source and destination ports, if applicable.
◦ Description
The filter name and a description.
— Filter Action
The filter action specifies the action to take for the rule, and can be one of the following
actions:
◦ allow: allow the packet to pass
◦ block: discard the packet
◦ negotiate security: negotiate IPsec Authentication Header (AH) or Encapsulating
Security Payload (ESP) Security Associations (SAs)
— Authentication Methods
The authentication methods specify the type of Internet Key Exchange (IKE)
authentication to use (preshared key or certificates with RSA signatures). If you are
using preshared key authentication, the authentication methods also specify the value
of the preshared key.
Windows IP Security Configuration Overview 13
— Tunnel Settings
The tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settings
also specify the tunnel destination endpoint.
— Connection Type
The connection type specifies the connection (link) types for the rule, such as LAN.
• General
The general parameters for a policy specify IKE SA parameters, such as the IKE encryption
algorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes.
The parameters correspond to IKE SA proposals. You can configure multiple IKE SA
proposals and specify the preference order. The proposals are used for all rules in the policy.
By comparison, a minimal HP-UX IPSec configuration consists of one or more IPsec host policies,
one or more IKE policies, and one or more authentication records. The IPsec host policies specify
address filters, and you can configure separate IKE policies for each peer. “Comparing HP-UX
and Windows IPsec Configuration Parameters” (page 40) lists IPsec configuration parameters
and how they are configured in the HP-UX IPSec and the Windows IP Security configuration
utilities.
Configuring a Windows Host-to-Host Policy
This section describes one method for configuring host-to-host policy on a Windows XP client
using the IP Security Policies snap-in utility. Windows also supports command-line utilities to
configure IP Security policies: ipseccmd on Windows XP systems and netsh on Windows 2003
systems. For more information about these utilities, see the Windows documentation set.
To use this method, complete the following steps:
1. Start the IP Security Policies snap-in utility. See “Step 1: Starting the IP Security Policies
Snap-in Configuration Utility” (page 15).
2. Create an IP Security policy. See “Step 2: Creating a Policy” (page 15).
3. Add a rule to the policy. See “Step 3: Adding a Rule” (page 16).
4. Create a Filter List for the rule and configure filters. See “Step 4: Creating the IP Filter List
and Filters for the Rule” (page 18).
5. Configure filter actions for the rule. The filter actions contain IPsec transforms or other
actions. See “Step 5: Configuring Filter Actions for the Rule” (page 21).
6. Configure the IKE authentication method and preshared key for the rule. See “Step 6:
Configuring the IKE Authentication Method and Preshared Key for the Rule” (page 25).
7. Specify the network link (connection) types for the rule. See“Step 7: Configuring the
Connection Type for the Rule” (page 26).
8. Modify the IKE SA parameters for the policy. By default, Windows clients will use IKE SA
parameters that are compatible with the default HP-UX IPSec parameters. If these parameters
are acceptable, you can skip this step. See “Step 8: Modifying IKE Parameters for the Policy”
(page 26).
9. Start the IP Security service. The IP Security service must be running before you can assign
the new IP Security policy. See “Step 9: Starting the IP Security Service” (page 29).
10. Assign (activate) the new IP Security Policy. See “Step 10: Assigning the IP Security Policy”
(page 30).
11. Verify the configuration. See “Step 11: Verifying the Configuration” (page 31).
Because this is a host-to-host rule, we will use the default value for the rule tunnel setting (no
tunnel). For information about configuring a tunnel rule and the tunnel setting, see “Configuring
a Windows End-to-End Tunnel Policy” (page 33).
14
Step 1: Starting the IP Security Policies Snap-in Configuration Utility
Use the following procedure to start the IP Security Policies configuration utility. This utility is
a snap-in module for the Microsoft Management Console (MMC).
1. Start the Microsoft Management Console (MMC). From the Microsoft Start menu, click Run
and type MMC. Click OK.
2. If the IP Security Policies snap-in configuration utility is not loaded, use the following
procedure to add it:
a. From the MMC window, click File→Add/Remove Snap-in.
b. From the Add/Remove Standalone Snap-in window, click Add.
c. From the Add Standalone Snap-in window, scroll down to IP Security Policy
Management and select it. Click Add.
d. In the Select Computer or Domain window, select Local computer (in this procedure,
we are configuring IP Security for the local computer). Click Finish.
e. Close the Add Standalone Snap-in window by clicking Close.
f. Close the Add/Remove Snap-in window by clicking OK.
Step 2: Creating a Policy
Use the following procedure to create an IP Security policy. An IP Security policy is a set of IPsec
configuration parameters. Only one local IP Security policy can be active (assigned) on a system.
1. In the left navigation pane of the IP Security Policy Management snap-in, click IP Security
Policies on Local Computer to display all IP Security Policies. Depending on your Windows
platform, there may be IP Security Policies already configured.
2. Right click IP Security Policies on Local Computer and select Create IP Security Policy.
3. The Policy Wizard starts and displays a startup message. Click Next.
4. The Policy Wizard opens the IP Security Policy Name window. Enter a name in the Name
field. This name is used only for internal identification.
Click Next.
5. The Policy Wizard opens the Requests for Secure Communication window. Clear the Activate
the default response rule check box, as shown in Figure 1. (The default response rule is a
pre-configured rule that causes the Windows system to dynamically build a filter list based
on the receipt of IKE requests. By default, the Windows system attempts to use IPsec only
if it receives an IKE request from a remote system.)
Click Next.
Configuring a Windows Host-to-Host Policy 15
Loading...