HP HP-UX IPSec User's Guide

Configuring Microsoft Windows
Vista and Windows Server
2008 to Operate with HP-UX IPSec
Introduction......................................................................................................................................... 2
Related Documentation ..................................................................................................................... 2
Protocol implementation differences.................................................................................................... 2
IKE version .................................................................................................................................. 2
IKE default algorithms ................................................................................................................... 2
IKE aggressive mode .................................................................................................................... 3
IPsec default transform .................................................................................................................. 3
Configuration overview ........................................................................................................................ 3
Configuring Windows IKE algorithms..................................................................................................... 3
Configuring connection security rules ..................................................................................................... 4
Using the Windows Firewall MMC to configure connection security rules ............................................... 4
Using the netsh advfirewall consec command to configure connection security rules................................. 5
Configuring inbound and outbound firewall rules .................................................................................... 6
(Optional) Configuring IPsec transforms.................................................................................................. 6
Introduction
This document contains tips for configuring IPsec on Microsoft Windows Vista and Windows Server 2008 systems to operate with HP-UX IPSec versions A.02.01 and later. For information on configuring IPsec on Microsoft Windows XP and Windows 2003 systems to operate with HP-UX IPSec, see in Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec.
The intended audience for this document is a network security administrator who is familiar with Microsoft Windows Vista or Windows Server 2008, the HP-UX IPSec product, and the IP Security protocol suite.
Related Documentation
The following documents are available from the HP Technical Documentation website at
http://docs.hp.com:
For information on configuring IPsec on Microsoft Windows XP and Windows 2003 systems to operate with HP-UX IPSec, see Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec.
For general information about configuring HP-UX IPSec, see the HP-UX IPSec A.02.01 Administrator's Guide or the HP-UX IPSec A.03.00 Administrator's Guide.
For information about configuring HP-UX IPSec to use Microsoft Windows security certificates, see
Using Microsoft Windows Certificates with HP-UX IPSec A.02.01and Using Microsoft Windows Certificates with HP-UX IPSec A.03.00.
Protocol implementation differences
HP-UX and Microsoft Windows both implement the IP Security protocol suite. However, there are features in the protocol suite that HP-UX implemented that Microsoft did not implement, and vice-versa. There are also differences in default parameter values.
IKE version
At the time this document was published, Microsoft Windows Vista and Windows Server 2008 supported only Internet Key Exchange (IKE) IKE version 1. HP-UX IPSec version A.03.00 supports IKE version 1 and IKE version 2. In this document, the term “IKE” refers to IKE version 1.
IKE authentication method
The Windows default IKE authentication method is Kerberos. RFC 2408 defines an optional Kerberos Token payload, but does not describe how to implement it. HP-UX IPSec does not support Kerberos for IKE authentication. You must configure the Microsoft connection security rule to use certificates or preshared keys for IKE authentication.
On HP-UX A.02.00 and A.02.01 systems, you specify the IKE authentication method using the – authentication option in the ipsec_config add ike command.
On HP-UX A.03.00 systems, you specify the IKE authentication method using the –local_method and –remote_method options in the ipsec_config add auth command.
IKE default algorithms
The Windows advanced firewall configures the IKE encryption and hash (integrity) algorithms as pairs in security methods. By default, the Windows configuration contains the following security methods:
AES-128 encryption and SHA-1 integrityAES-128 encryption and MD5 integrity
2
The HP-UX IPSec default IKE algorithms are 3DES encryption and MD5 integrity.
At least one Windows security method must match the HP-UX IKE parameters. To change or add IKE algorithms for Microsoft Windows, use the procedure described in “Configuring Windows IKE algorithms.” To change specify IKE algorithms for HP-UX IPSec, use the –encryption and –hash options in the ipsec_config add ike or ipsec_config add ikev1 command.
IKE aggressive mode
HP-UX supports aggressive mode (AM) to establish IKE Security Associations (SAs). IKE AM is an optional feature and is not supported on Windows.
IPsec default transform
The default Windows IPsec transform is ESP with AES-128 encryption and SHA-1 authentication. On HP-UX systems, you must explicitly specify this transform using the –action ESP_AES128_HMAC_SHA1 option in the ipsec_config add host command.
Configuration overview
To configure IPsec on Microsoft Windows Vista and Windows Server 2008 systems, you must complete the following tasks:
1. Configure IKE algorithms, if needed.
2. Configure connection security rules.
3. Configure firewall rules.
4. (Optional) Configure additional IPsec transforms.
Connection security rules specify IPsec parameters, including IKE and IPsec security algorithms, IKE authentication methods and IP address filters for the packets to be secured by IPsec. Depending on the method used to configure a rule, the rule can also specify protocols and port numbers for the filter.
Firewall rules are required to allow inbound and outbound packets when the Windows firewall is enabled. Firewall rules specify a packet filter and the action for packets that match the filter, such as requiring IPsec security to the packets.
Configuring a connection security rule is not sufficient to allow IPsec packets to pass in and out of a Windows system. If you have a connection security rule to secure packets between the local system and a remote system but do not have a firewall rule to allow packets to and from the remote system, the firewall will block the packets.
NOTE: Microsoft Windows Vista and Windows Server 2008 also support IPsec Policy Agent rules. These rules are functionally the same as the IPsec rules on Windows XP and Windows Server 2003 systems and can be configured using the IPsec Policy MMC snap-in as documented in Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec. Microsoft provides IPsec Policy Agent rules for backwards compatibility only and does not recommend the use of these rules on Windows Vista and Windows Server 2008 systems.
Configuring Windows IKE algorithms
The default HP-UX IKE algorithms do not match the default Windows IKE algorithms, so you must configure the IKE algorithms on either the Windows system or the HP-UX system to match the algorithms on the other system.
3
This section describes how to configure Windows IKE algorithms using the Windows Firewall with Advanced Security MMC. You can also use the Windows netsh advfirewall set global mainmode command to configure IKE algorithms.
For information on how to configure IKE algorithms on HP-UX systems, see “IKE default algorithms.”
Use the following procedure to configure Windows IKE algorithms using the Windows Firewall with Advanced Security MMC:
1. Select Control Panel -> Administrative Tools -> Windows Firewall with Advanced Security on Local Computer to start the Windows firewall MMC.
2. Select Properties in the right menu. In the Properties dialog box, click the IPsec Settings tab.
3. In the IPsec defaults section, click Customize.
The MMC opens the Customize IPsec Settings dialog box.
4. In the Key exchange (Main Mode) section, select Advanced.
Click Customize.
The MMC opens the Customize Advanced Key Exchange Settings dialog box.
5. In the Security methods section, click Add.
The MMC opens the Security Method dialog box.
6. In the Encryption algorithm section, select an IKE encryption algorithm, such as 3DES.
In the Integrity algorithm section, select the IKE hash algorithm, such as MD5.
7. Click OK to close the Security Method dialog box.
Click OK to close the Customize Advanced Key Exchange Settings dialog box.
Click OK to close the Customize IPsec Settings dialog box.
Click OK to close the Properties dialog box.
Configuring connection security rules
This section describes two methods to configure connection security rules on Microsoft Windows Vista and Windows 2008 systems:
Using the Windows Firewall with Advanced Security Microsoft Management Console (MMC) snap-
in. The Firewall MMC provides a GUI that enables you to configure a connection security rule for all packets between IP addresses, regardless of protocols and port numbers. It does not allow you to specify protocols or port numbers for the filter.
Using the Microsoft netsh advfirewall consec command. This command enables you to
create a connection security rule that includes protocol and ports in the filter.
Using the Windows Firewall MMC to configure connection security rules
Use the following procedure to use the Windows Firewall with Advanced Security MMC to configure a connection security rule:
1. Select Control Panel -> Administrative Tools -> Windows Firewall with Advanced Security on Local Computer to start the Windows firewall MMC.
2. Select Connection Security Rules in the left menu.
4
3. Select New Rule in the right menu.
4. In the wizard Rule Type dialog box, select Custom.
Click Next.
5. In the Endpoints dialog box, enter the IP addresses.
Click Next.
6. In the Requirements dialog box, select Require inbound and outbound.
Click Next.
7. In the Authentication Method dialog box, configure the IKE authentication method.
To use certificates for IKE authentication, select Computer Certificate and enter the distinguished name for the CA in the CA Name field.
To use preshared keys, select Advanced and click the Customize button. In the Customize Advanced Authentication Methods dialog box, select Preshared Key and enter an ASCII string in the Key field.
Click Next.
8. The Profile dialog box specifies when the rule will be active according to the network status of the system. Select the profiles appropriate for your topology. HP selected all the profiles.
Click Next.
9. In the Name dialog box, enter a name for the rule.
Click Finish.
Using the netsh advfirewall consec command to configure connection security rules
To configure a connection security rule with specific ports or protocols, you must use the Microsoft netsh advfirewall consec command. The netsh command has a built-in help utility that you can use to display syntax information.
In most cases, you can use syntax similar to the following:
netsh advfirewall consec add rule name=rule_name enable=yes endpoint1=ip_addr endpoint2=ip_addr [protocol=protocol][port1=port] [port2=port] action=requireinrequireout auth1=computerpsk auth1psk=my_preshared_key
For example, the following command configures a rule that applies IPsec security for the telnet (port
514) service on the Windows 2008 server (10.0.0.208) from the HP-UX system (10.0.0.11):
netsh advfirewall consec add rule name=iop-rule enable=yes endpoint1=10.0.0.208 endpoint2=10.0.0.11 protocol=tcp port1=514 action=requireinrequireout auth1=computerpsk auth1psk=MyKey
Rules configured using the netsh advfirewall consec command are bidirectional. IPsec will be used for TCP packets from 10.0.0.11 to port 514 on 10.0.0.208 and for packets in the reverse direction.
The default IPsec transform is ESP with AES-128 encryption and SHA-1 authentication. To specify a different transform, use the qmsecmethods option.
5
Configuring inbound and outbound firewall rules
In addition to the connection security rule, you must configure inbound and outbound firewall rules to allow packets to pass through the firewall and to specify that the packets will be secured using IPsec. Use the following procedure to add an inbound or outbound firewall rule with the Windows Firewall with Advanced Security MMC:
1. Select Control Panel -> Administrative Tools -> Windows Firewall with Advanced Security on Local Computer to start the Windows firewall MMC.
2. In the left menu, select Inbound Rules or Outbound Rules.
3. In the right menu, select New Rule.
The MMC starts the New Inbound Rule Wizard or the New Outbound Rule Wizard.
4. In the Rule type dialog box, select Custom.
Click Next.
5. In the Programs dialog box, select All programs.
Click Next.
6. In the Protocol and Ports dialog box, specify the appropriate protocol and port numbers. If you select ICMP or ICMPv6 for the protocol type, you can specify the ICMP message types for the filter by selecting Customize for the ICMP settings.
Click Next.
7. In the Scope dialog box, specify the appropriate local and remote IP addresses.
Click Next.
8. In the Action dialog box, select Allow the connection if it is secure and Require the connections to be encrypted. This causes Windows to apply IPsec to the packets as defined by a connection security rule.
9. In the Profile dialog box, select the appropriate profiles for the rule. HP selected all the profiles.
Click Next.
10. In the Name dialog box, enter a name for the rule.
Click Finish
(Optional) Configuring Windows IPsec transforms
The default Windows IPsec transform is ESP with AES-128 encryption and SHA-1 authentication. Use the following procedure to configure different transforms for all Windows IPsec SAs:
1. Select Control Panel -> Administrative Tools -> Windows Firewall with Advanced Security on Local Computer to start the Windows firewall MMC.
2. In the left menu, select Windows Firewall with Advanced Security on Local Computer.
3. Select Properties in the right menu. In the Properties dialog box, select IPsec Settings. Under IPsec defaults, select Customize.`
6
The MMC opens the Customize IPsec Setting dialog box.
4. In the Data Protection (Quick Mode) section, select Advanced and click the Customize button.
The MMC opens a Customize Data Protection Settings dialog box.
5. In the Data integrity and encryption section and click Add to add additional transforms and algorithms. Specify the new transform.
Click OK.
7
©
2009
Hewlett
-
Packard Development Company, L.P
. The information contained
herein is subject to change without notice. The only warranties for HP products and
J4256
services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.
Itanium is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.
-90047, May 2009
Loading...