Nortel Networks NN46120-104 User Manual

Nortel VPN Gateway

User Guide

Release: 7.1 Document Revision: 02.01

www.nortel.com

NN46120-104

216368-G

Nortel VPN Gateway Release: 7.1 Publication: NN46120-104 Document status: Standard Document release date: 14 April 2008

The information in this document is subject to change without notice. The statements, configurations, technical data, and recommendations in this document are believed to be accurate and reliable, but are presented without express or implied warranty. Users must take full responsibility for their applications of any products specified in this document. The information in this document is proprietary to Nortel Networks.

*Nortel, Nortel Networks, the Nortel logo and the Globemark are trademarks of Nortel Networks.

Export

This product, software and related technology is subject to U.S. export control and may be subject to export or import regulations in other countries. Purchaser must strictly comply with all such laws and regulations. A license to export or reexport may be required by the U.S. Department of Commerce.

Licensing

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit

ttps://www.openssl.org/).

This product includes cryptographic software written by Eric Young (eay@cryptsoft.com).

This product includes software written by Tim Hudson (tjh@cryptsoft.com).

This product includes software developed by the Apache Software Foundation h

ttp://www.apache.org/

Preface 7

Who Should Use This Book 8 Related documentation 9 Product Names 10 How This Book Is Organized 11 Typographic Conventions 13 How to Get Help 14 Getting help from the Nortel Web site 14 Getting help over the phone from a Nortel Solutions Center 14 Getting help from a specialist by using an Express Routing Code 14 Getting help through a Nortel distributor or reseller 14

Introducing the VPN Gateway 15

SSL Acceleration 16 VPN 17 Hardware Platforms 18 Feature List 19

Introducing the ASA 310-FIPS 27

HSM Overview 28 Extended Mode vs. FIPS Mode 29 The Concept of iKey Authentication 30 Additional HSM Information 33

Initial Setup 35

Clusters 36 IP Address Types 37 Ports 38 Interfaces 39 Configuration at Boot Up 41 Installing an NVG in a New Cluster 42 Joining a VPN Gateway to an Existing Cluster 52 Installing an ASA 310-FIPS 58 Reinstalling the Software 70

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Upgrading the NVG Software 73

Performing Minor/Major Release Upgrades 74

Managing Users and Groups 79

User Rights and Group Membership 80 Adding a New User 81 Changing a Users Group Assignment 86 Changing a Users Password 88 Deleting a User 91

Certificates and Client Authentication 93

Generating and Submitting a CSR Using the CLI 94 Adding Certificates to the NVG 99 Update Existing Certificate 107 Configure a Virtual SSL Server to Require a Client Certificate 108 Generating client certificates 110 Managing Revocation of Client Certificates 116 Client certificate support 123 Signing CSRs 124 Generate Test Certificate 126 General Commands 128

Virtual Desktop 131

Running the Virtual Desktop on Client Computers 132 Licensing vdesktop 132 Launch Vdesktop from Portal 133 Virtual Desktop Operations 134

The Command Line Interface 135

Connecting to the VPN Gateway 136 Accessing the NVG Cluster 140 CLI vs. Setup 142 Command Line History and Editing 143 Idle Timeout 144

Troubleshooting the NVG 145

Cannot Connect to VPN Gateway through Telnet or SSH 146 Cannot Add an NVG to a Cluster 148 Cannot Contact the MIP 149 The NVG Stops Responding 151 A User Password is Lost 152 An ASA 310-FIPS Stops Processing Traffic 153 Resetting HSM Cards on the ASA 310-FIPS 155 AnASA 310-FIPS Cluster Must be Reconstructed onto New Devices 158 A User Fails to Connect to the VPN 163 User Unable to Connect to the VPN Gateway through the Net Direct Client 168

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Licensing 5

Cannot download the NetDirect Zipped file from client PC 171 System Diagnostics 172 Unable to download NetDirect from VPN server 175

Supported Ciphers 177

Cipher List Formats 179 Modifying a Cipher List 180 Supported Cipher Strings and Meanings 181

The SNMP Agent 183

Supported MIBs 184 Supported Traps 189

Syslog Messages 191

List of Syslog Messages 192 Syslog Messages in Alphabetical Order 209

222

License Information 223 HSM Security Policy 233 Definition of Key Codes 253

Syntax Description 254

SSH host keys 257

Methods for Protection 258 The VPN Gateway 259

Adding User Preferences Attribute to Active Directory 261

262

Using the Port Forwarder API 271

General 272 Creating a Port Forwarder 273 Demo Application 274 Creating a Port Forwarder Authenticator 276 Adding a Port Forwarder Logger 279 Connecting Through a Proxy 282 Monitoring the Port Forwarder 283

Glossary 285 Index 295

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Preface

This User’s Guide describes how to perform basic configuration and maintenance of the Nortel VPN Gateway (NVG).

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

8 Preface

Who Should Use This Book

This User’s Guide is intended for network installers and system administrators engaged in configuring and maintaining a network. It assumes that you are familiar with Ethernet concepts and IP addressing.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Product Names

The software described in this manual runs on several different hardware models. Whenever the generic terms Nortel VPN Gateway, VPN gateway or NVG are used in the documentation, the following hardware models are implied:

•

Nortel VPN Gateway 3050 (NVG 3050)

•

Nortel VPN Gateway 3070 (NVG 3070)

•

Nortel SSL VPN Module 1000 (SVM 1000)

• Nortel SSL Accelerator 310-FIPS (ASA 310-FIPS)

• The integrated SSL Accelerator (SSL processor) on the Nortel

2424-SSL switch

• Nortel VPN Gateway Universal Serial Bus

Similarly, all references to the old product name – iSD-SSL or iSD – in commands or screen outputs should be interpreted as applying to the preceding hardware models.

Note: Manufacturing of the Nortel SSL Accelerator (formerly Alteon

SSL Accelerator) has been discontinued.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

How This Book Is Organized

The chapters in this book are organized as follows:

Users Guide

“Introducing the VPN Gateway” (page 15) provides an overview of the

major features of the VPN Gateway, including its physical layout and the basic concepts of its operation.

“Introducing the ASA 310-FIPS” (page 27) provides information about

the ASA 310 equipped with HSM cards, as well as information about the available security modes and the concept of iKey authentication.

“Initial Setup” (page 35) describes how to install the NVG in a new cluster,

and how to add an NVG to an existing cluster. The chapter also provides information about the concept of NVG clusters, as well as the usage and configuration of ports and networks within a cluster. A section describing how to reinstall the software is also included.

“Upgrading the NVG Software” (page 73) describes how to upgrade the

NVG software for a minor release upgrade, and a major release upgrade, as well as upgrading from software versions earlier than 2.0.11.16 to version 3.0.7.

“Managing Users and Groups” (page 79) describes the management

of users, groups, and passwords. The chapter also explains how the Administrator user role can be fully separated from the Certificate Administrator user role.

“Certificates and Client Authentication” (page 93) describes how to

generate and prepare keys and certificates for use with the NVG.

“The Command Line Interface” (page 135) describes how to connect to the

NVG and access the information and configuration menus.

“Troubleshooting the NVG” (page 145) provides suggestions for

troubleshooting basic problems. Information about performing system diagnostics on the NVG is also included, as well as some operations related to the ASA 310-FIPS model.

Appendices

provides a list of ciphers supported in this product.

“The SNMP Agent” (page 183) provides information about the SNMP

agent on the NVG, and which MIBs (Management Information Bases) are supported.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

12 Preface

“Syslog Messages” (page 191), contains a list of all syslog messages

that can be sent to a syslog server that is added to the NVG system configuration.

“License Information” (page 223) provides licensing information for the

software used in this product.

“HSM Security Policy” (page 233) provides detailed information about the

security policy of the CryptoSwift

HSM card that comes installed in the

ASA 310-FIPS.

“Definition of Key Codes” (page 253) provides information about how

to compile a keycode definition file to be used with the Terminal applet available on the Telnet/SSH tab (located under the Portal’s Advanced tab).

“SSH host keys” (page 257) provides information about the purpose of

SSH host keys and how they are used to protect the connection between the SSH client and the VPN Gateway.

“Adding User Preferences Attribute to Active Directory” (page

261) provides step-by-step instructions on how to add the User

Preferences attribute to Active Directory. This is required to support storage of Portal bookmarks in Active Directory.

“Using the Port Forwarder API” (page 271) provides instructions on how to

perform the tasks needed when using the Port Forwarder API. The Port Forwarder API is used to provide tunnels through the Nortel VPN Gateway (NVG) without the user having to start any applets from the Portal.

“Glossary” (page 285) includes definitions of terminology used throughout

this document.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Typographic Conventions

The following table describes the typographic styles used in this book.

Table 1 Typographic Conventions

Typeface or Symbol

AaBbCc123

[ ] Command items shown inside

Meaning Example

This type is used for names of commands, files, and directories used within the text.

It also depicts on-screen computer output and prompts.

This bold type appears in command examples. It shows text that must be typed in exactly as shown.

This italicized type appears in command examples as a parameter placeholder. Replace the indicated text with the appropriate real name or value when using the command. Do not type the brackets.

This also shows book titles, special terms, or words to be emphasized.

brackets are optional and can be used or excluded as the situation demands. Do not type the brackets.

View the readme.tx t file.

Main#

Main# sys

To establish a Telnet session, enter:

host# telnet <IP address>

Read your User’s Guide thoroughly.

host# ls[-a]

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

14 Preface

How to Get Help

This section explains how to get help for Nortel products and services.

Getting help from the Nortel Web site

The best way to get technical support for Nortel products is from the Nortel Technical Support Web site: h This site provides quick access to software, documentation, bulletins, and tools to address issues with Nortel products. From this site, you can:

•

download software, documentation, and product bulletins for answers to technical issues

• sign up for automatic notification of new software and documentation

for Nortel equipment

•

open and manage technical support cases

Getting help over the phone from a Nortel Solutions Center

If you do not find the information you require on the Nortel Technical Support web site, and have a Nortel support contract, you can also get help over the phone from a Nortel Solutions Center. In North America, call 1-800-4NORTEL (1-800-466-7835). Outside North America, go to the following web site to obtain the phone number for your region:

ww.nortel.com/callus

ttps://www.nortel.com/support/

Getting help from a specialist by using an Express Routing Code

An Express Routing Code (ERC) is available for many Nortel products and services. When you use an ERC, your call is routed to a technical support person who specializes in supporting that product or service. To locate the ERC for your product or service, go to: h

ttp://www.nortel.com/erc/

Getting help through a Nortel distributor or reseller

If you purchased a service contract for your Nortel product from a distributor or authorized reseller, contact the technical support staff for that distributor or reseller.

Nortel VPN Gateway

NN46120-104 02.01 Standard

User Guide

14 April 2008

Introducing the VPN Gateway

The Nortel VPN Gateway (NVG) software includes two major functionality groups:

•

SSL Acceleration

•

VPN

These features can be used separately or be combined. This User’s Guide covers the basic tasks that need to be completed irrespective of which feature you wish to deploy.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

16 Introducing the VPN Gateway

SSL Acceleration

The VPN Gateway can function as a peripheral Secure Sockets Layer (SSL) offload platform that attaches to a Nortel Application Switch or a comparable switch from another vendor. (The VPN Gateway can also operate in standalone mode, i.e. without being connected to a switch.)

The VPN Gateway performs a TCP three-way handshake with the client through the Nortel Application Switch and performs all the SSL encryption and decryption for the session. Combined with the load balancing features of the Nortel Application Switch, the VPN Gateway offloads SSL encryption/decryption functions from back-end servers.

For examples on how to configure the VPN Gateway for SSL Acceleration, see the Application Guide for SSL Acceleration.

For more information about the basic operations of the VPN Gateway, see the "Public Key Infrastructure and SSL" chapter in the Application Guide for SSL Acceleration.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

VPN

Getting help through a Nortel distributor or reseller 17

The VPN feature supports remote access to intranet or extranet resources (applications, mail, files, intranet web pages) through a secure connection. What information should be accessible to the remote user after login is determined by access rules (ACLs).

The intranet’s resources can be accessed in clientless mode, transparent mode or both:

•

From any computer connected to the Internet (clientless mode). The remote user connects to the VPN Gateway through a secure SSL connection through the web browser. When successfully authenticated, the user can access services and resources on the intranet from a Web Portal provided by the VPN Gateway. Clientless mode also enables download of the Net Direct client, a simple and secure method for accessing intranet resources through the remote user’s native applications.

•

From a computer with the Nortel IPsec VPN client (formerly Contivity VPN client) or the Nortel SSL VPN client installed (transparent mode).

For examples on how to configure the VPN Gateway for VPN deployment, see the Application Guide for VPN.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

18 Introducing the VPN Gateway

Hardware Platforms

The VPN Gateway software is supported on the following hardware platforms:

•

Nortel VPN Gateway 3050 and 3070

•

Nortel SSL VPN Module 1000

• Nortel SSL Accelerator 310 and 410

• Nortel SSL Accelerator 310-FIPS, with FIPS-compliant Hardware

Security Module (HSM). See “Introducing the ASA 310-FIPS” (page

27).

• Nortel 2424-SSL Application Switch

For a detailed technical specification of the hardware platforms, see the "Specifications" appendix in the VPN Gateway 3050/3070 Hardware

Installation Guide and the Alteon SSL Accelerator Hardware Installation Guide respectively.

No hardware installation is required for the Nortel Application Switch 2424-SSL. The VPN Gateway software resides on the SSL Processor which is mounted inside the switch chassis.

Note: Manufacturing of the Nortel SSL Accelerator (formerly Alteon

SSL Accelerator) has been discontinued.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Feature List

Software Features

Web Portal

• Web Portal interface for remote users accessing the VPN Gateway in

clientless mode, that is, through the browser.

•

Corporate resources available to users as preconfigured group links or accessible through the Portal tabs.

• Support for native Telnet and SSH (including X11 forwarding) access

to intranet servers through terminal Java applet (available on the Portal’s Advanced tab).

• Support for handling plugins, Flash and Java applets using HTTP

proxy Java applet (available on the Portal’s Advanced tab).

•

Support for application tunneling (port forwarding) through SOCKS encapsulated in SSL (available on the Portal’s Advanced tab).

•

API provided for developing a custom application that automatically logs in the user to the desired VPN and executes a previously configured port forwarder link

Software Features 19

•

Support for customizing the Web Portal, for example, color, logo, language and company name.

•

Three user views available (novice, medium and advanced) to limit access to Portal tabs.

•

Support for automatic redirection of requests to another URL (Portal pass-through).

•

Support for Portal bookmarks.

•

Ability to specify domains for which single sign-on is allowed.

•

Net Direct client (SSL). VPN client temporarily downloaded from the Portal and removed when the user exits the session. On Windows, Net Direct is also available as an installable client (setup.exe file).

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

20 Introducing the VPN Gateway

Transparent Mode Access

Access to intranet resources in transparent mode, that is, without going through the Web Portal, is accomplished using Windows VPN clients installed on the client PCs. In this mode, remote users will experience network access as if sitting within the local area network. The following VPN clients are available:

• Nortel SSL VPN client (TDI and LSP version).

• Nortel IPsec VPN client (formerly the Contivity VPN client). Not

supported on the ASA 310, ASA 310-FIPS and ASA 410 hardware models.

• Net Direct installable client.

User Authentication

User authentication is supported using the following methods:

• RADIUS (including Challenge/Response)

•

LDAP (including Microsoft Active Directory)

• NTLM (Windows NT Domain, including Microsoft Active Directory)

•

Secure Computing SafeWord (RADIUS)

•

Netegrity SiteMinder

•

RSA SecurID (native or through RADIUS)

• RSA ClearTrust

•

ActivCard (RADIUS)

•

Novell NDS/eDirectory (LDAP)

•

Client certificate authentication

• Local database authentication

User Authorization

User authorization is controlled through the user’s group membership. Two different authorization profile types are supported:

• The base profile defines a group member’s access rights to networks,

services and paths.

• The extended profile (optional) also defines a group member’s access

rights depending on conditions related to the user’s connection, for example, source network, authentication method, access method, client certificate installed and/or Tunnel Guard checks passed.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Client Security

• Tunnel Guard. Feature for checking the security aspects of the remote

PC client, that is, installed antivirus software, DLLs, executables and so on.

• WholeSecurity support. Lets you enable a scan of the client PC before

the remote user is allowed to log in to the VPN.

• User session auto-logoff.

•

Cache and browser history automatically cleared (only for Internet Explorer).

Accounting and Auditing

•

Support for logging user session start and stop messages to a syslog or RADIUS accounting server. The messages can include VPN ID, user name, gateway address, session ID, session time and cause of termination.

• Support for logging CLI and Web User Interface operations (for

example, login, logout and executed operation) to a syslog or RADIUS accounting server.

Software Features 21

Networking

•

Supports creating multiple interfaces within a cluster, for example, to separate client traffic and management traffic. (Not supported on the Nortel Application Switch 2424-SSL).

•

Support for clustering over multiple subnets.

•

Supports assigning two physical network ports to one interface, to create a port failover (high availability) solution where one VPN Gateway is attached to two Nortel Application Switches.

Secure Service Partitioning

The NVG software provides the ability to partition a cluster of VPN Gateways into separate VPNs. This gives service providers (ISPs) the possibility to host multiple VPN end-customers on a shared Remote Access Services (RAS) platform. Requires a license.

• Supports hosting of up to 250 public termination points for

end-customer SSL and IPsec VPNs.

• Secure VPN binding. Each VPN is bound to a private IP interface.

VLAN tagging can be used when private IP address spaces overlap.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

22 Introducing the VPN Gateway

• Private network authentication. Existing authentication servers within

the customer’s private network can be used.

• Access control. Unique access rules can be specified for each user

group in the various VPNs.

• Private network name resolution. If desired, private network DNS

servers can be mapped to the VPN.

• Split administration. VPN Portal management is enabled for each

VPN customer through a web interface, without exposing global administration access.

• High availability. The Secure Service Partitioning solution is compatible

with the NVG cluster’s high availability solutions.

Branch Office Tunnels

The NVG software provides the ability to configure IPsec-based branch office tunnels. Several peer-to-peer branch office tunnels can be

configured for each virtual private network (VPN). The following number of branch office tunnels can be configured per

hardware model:

•

NVG 3070: 2500

•

NVG 3050: 1000

•

Nortel 2424-SSL Application Switch: 500

For example, a cluster of two NVG 3070s support 5000 branch office tunnels.

Portal Guard

Feature used to "convert" an existing HTTP site to generate HTTPS links, secure cookies and so on. The VPN Gateway will not only handle the SSL processing but also see to it that all existing web links are rewritten to HTTPS. This eliminates the need to rewrite each link manually. Requires a license.

SSL Acceleration

The NVG software also includes features for SSL acceleration. Note that these features in some cases require interoperation with a Nortel Application Switch.

• Supports accelerated SSL processing by offloading SSL encryption

and decryption from backend servers.

• Supports load balancing of encrypted and unencrypted traffic for up

to 256 backend servers, with health checking and persistent client connections.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

• Ability to create multiple clusters of VPN Gateways, each capable of

serving its own group of real servers.

• Supports rewriting of client requests.

•

Ability to transmit additional information to the backend servers.

• Supports end-to-end encryption.

• Compatible with all Nortel Application Switches, Nortel Web Switches

and comparable switches from other vendors.

SSL Acceleration is covered in the Application Guide for SSL Acceleration.

Scalability and Redundancy

• Support for 256 VPN Gateways per cluster

•

Support for 256 virtual SSL servers

• Provides dynamic plug and play – VPN Gateways can be added to or

removed from a cluster dynamically without disrupting network traffic

• Provides a single system image (SSI) – all VPN Gateways in a given

cluster are configured as a single system

Software Features 23

•

High level of redundancy in the master/slave cluster design; even if three master VPN Gateways in a cluster would fail, additional slave NVGs will still be operational and can accept configuration changes

Certificate and Key Management

•

Server and client authentication

•

Generation and revocation of client certificates

•

Automatic retrieval of certificate revocation lists (CRLs)

•

Validation of private keys and certificates

•

Generation of certificate signing requests (CSRs)

•

Generation of self-signed certificates

Public Key Infrastructure

• RSA pair key generation

• Server certificate enrollment

• Server key and certificate import/export

• Key and certificate renewal

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

24 Introducing the VPN Gateway

Supported Key and Certificate Formats

• PEM

• DER

•

NET

• PKCS12

• PKCS8

• KEY(MS IIS4.0)

Supported Handshake Protocols

• SSL versions 2.0, 3.0

•

TLS version 1.0

Hash Algorithms

• Message Digest 5 (MD5)

•

SHA1

Cipher Suites

All ciphers covered by SSL version 2.0, 3.0 and TLS version 1.0, except the IDEA and FORTEZZA ciphers. Also see “Supported Ciphers” (page

177).

Management

•

Web User Interface (HTTP or HTTPS).

•

Command Line Interface (CLI) access through Telnet/SSH or serial port.

•

SNMP version 1, version 2c and version 3.

• RADIUS authentication of CLI/BBI administrator users (including

console access).

Statistics

• Statistics can be viewed per access method (SSL or IPsec) for the

whole cluster as well as for specific VPN Gateways, SSL servers and VPNs.

• Support for histograms, for example, to measure transactions per

second (TPS) and throughput.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Virtual Desktop

Symantec On-Demand Agent (SODA) provides a Virtual Desktop environment to secure Web-based applications and services. Virtual Desktop is a Java application that provides protection against lost or theft of sensitive information. Files created while in the virtual desktop are encrypted as they are saved to a hard drive or removable media. Integrating Virtual Desktop with NVG will provide a secure environment for end users while accessing confidential information.

Secure Portable Office (SPO) Client

The SPO client provides VPN access from portable storage such as USB compliant flash memory and CD ROM.

The SPO client provides enhanced mobility, portability, and security compared to traditional VPN access methods. The SPO client can be deployed and managed from the NVG server thus simplifying SPO client maintenance and updates.

For more information about Secure Portal Office Client, see Configuration Secure Portable Client Guide.

Software Features 25

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

26 Introducing the VPN Gateway

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Introducing the ASA 310-FIPS

This section provides information about the ASA 310-FIPS model, which comes installed with the HSM (Hardware Security Module) card. The HSM card complies with all the security requirements specified by the Federal Information Processing Standard (FIPS) 140-1, Level 3 standards. Each ASA 310-FIPS device is equipped with two identical HSM cards.

Note: When using the ASA 310-FIPS device in a cluster, remember

that all NVG devices in the cluster must be of the ASA 310-FIPS model.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

28 Introducing the ASA 310-FIPS

HSM Overview

The HSM card found on the ASA 310-FIPS model is an SSL accelerator, just like the ordinary CryptoSwift card found on the regular ASA 410 model. In addition to cryptographic acceleration, the HSM card brings extra security to sensitive operations and is designed to withstand physical tampering.

•

The HSM card provides a secure storage area for cryptographic key information. The storage area is secured by a constantly monitored tamper detection circuit. If tampering is detected, the battery backup power to memory circuits on the card is removed. Critical security parameters, such as private keys that are in the storage area, will then be destroyed and rendered useless to the intruder.

• Any sensitive information that is transferred between two HSM cards

within the same ASA 310-FIPS, or between any number of HSM cards within a cluster of ASA 310-FIPS devices, is encrypted using a shared secret stored (also known as a wrap key) on the HSM card.

• Some user operations require a two-phase authentication, which

involves using both hardware tokens (called iKeys) and an associated password to provide an extra layer of security. For example, if the ASA 310-FIPS is power cycled (as in the case of theft), no SSL traffic is processed until the operator logs in to the HSM card using both an iKey and the correct password.

•

All cryptographic requests, such as generating private keys or performing encryption, are automatically routed to the HSM card by the NVG application and performed on the HSM card only.

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

Extended Mode vs. FIPS Mode

When installing the very first ASA 310-FIPS into a new cluster, you can choose to initialize the HSM cards in either Extended mode or FIPS mode. Extended mode is the default selection, and is appropriate whenever your security policy does 140-1, Level 3 standard (see the following for more information).

The main difference between Extended mode and FIPS mode involves how private keys are handled. For both modes, all private keys are stored encrypted in the database on the ASA 310 FIPS. When the HSM card is initialized in Extended mode, the encrypted private key needed to perform a specific operation is transferred to the HSM card over the PCI bus. The private key is then decrypted on the HSM card itself, using the wrap key that was generated during the initialization and because stored on the card. The private key is thus never exposed in plain text outside the HSM card.

When the HSM card is initialized in FIPS mode, the encrypted private key needed to perform a specific operation is read from the database into RAM, together with the wrap key from the HSM card. The private key is then decrypted in RAM, where it remains accessible for subsequent operations.

not explicitly require that you conform to the FIPS

FIPS140-1 Level 3 Security 29

Also, when the ASA 310-FIPS is initialized in FIPS mode, all private keys must be generated on the ASA 310-FIPS device itself. Importing private keys, or certificate files that contain private keys, is not allowed due to the FIPS security requirements. This means that certain CLI commands that are used for importing certificates and keys through a copy and paste operation, or through TFTP/FTP/SCP/SFTP, cannot be used when the ASA 310-FIPS is initialized in FIPS mode.

FIPS140-1 Level 3 Security

The HSM card contains all of the security requirements specified by the FIPS 140-1, Level 3 standards. FIPS 140-1 is a U.S. government standard for implementations of cryptographic modules, that is, hardware or software that encrypts and decrypts data or performs other cryptographic operations (such as creating or verifying digital signatures).

FIPS 140-1 is binding on U.S. government agencies deploying applications that use cryptography to secure sensitive but unclassified (SBU) information, unless those agencies have been specifically exempted from compliance by the relevant U.S. laws referenced in the standard.

For more information about the FIPS specification, visit http://csrc.nist.gov/ publications/fips/index.htmland scroll down to "FIPS 140-1".

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

30 Introducing the ASA 310-FIPS

The Concept of iKey Authentication

Access to sensitive data on a ASA 310-FIPS is protected by a combination of hardware tokens (called iKeys), passwords, and encryption procedures.

The iKey is a cryptographic token that is used as part of the authentication process for certain operations involving the HSM cards. Whenever you perform an operation on the ASA 310-FIPS calling for iKey authentication, you are prompted by the Command Line Interface to insert the requested iKey into the USB port on the appropriate HSM card. (When prompted for a particular iKey, a flashing LED always directs you to the correct HSM card.)

Types of iKeys

For each HSM card there are two unique iKeys used for identity-based authentication: the HSM-SO iKey, and the HSM-USER iKey. Each of these iKeys define the two user roles available: Security Officer and User. A password must be defined for each user role, and the passwords are directly associated with the corresponding iKey. The ASA 310-FIPS is equipped with two HSM cards, and you therefore need to maintain two pairs of HSM-SO and HSM-USER iKeys with their associated passwords for each single ASA 310-FIPS device.

After a HSM card has been initialized, that card will only accept the HSM-SO and HSM-USER iKeys that were used when initializing that particular card. You cannot create backup copies of the associated HSM-SO iKey and HSM-USER iKey, and a lost HSM-SO or HSM-USER password cannot be retrieved. It is therefore extremely important that you establish routines for how the iKeys are handled.

Wrap Keys for ASA 310-FIPS Clusters

In addition to the HSM-SO and HSM-USER iKeys specific for each HSM card, one pair of iKeys (the black HSM-CODE iKeys) need also be maintained for each cluster of ASA 310-FIPS units.

Note: You are strongly recommended to label two of the black

HSM-CODE iKeys "CODE-SO" and "CODE-USER" respectively; these iKeys will be referred to as such both in the documentation and in the Command Line Interface.

During the initialization of the first ASA 310-FIPS in a cluster, a wrap key is automatically generated. The wrap key is a secret shared among all ASA 310-FIPS in the cluster. It encrypts and decrypts sensitive information that is sent over the PCI bus within an ASA 310-FIPS, and over the network among the ASA 310-FIPS devices in the cluster. By inserting the CODE-SO iKey and the CODE-USER iKey in turns when requested

Nortel VPN Gateway

User Guide

NN46120-104 02.01 Standard

14 April 2008

+ 270 hidden pages

Nortel Networks NN46120-104 User Manual

Export

Licensing

Contents

Preface

Who Should Use This Book

Related documentation

Product Names

How This Book Is Organized

Users Guide

Appendices

Typographic Conventions

How to Get Help

Getting help from the Nortel Web site

Getting help over the phone from a Nortel Solutions Center

Getting help from a specialist by using an Express Routing Code

Getting help through a Nortel distributor or reseller

Introducing the VPN Gateway

SSL Acceleration

Hardware Platforms

Feature List

Software Features

Web Portal

Transparent Mode Access

User Authentication

User Authorization

Client Security

Accounting and Auditing

Networking

Secure Service Partitioning

Branch Office Tunnels

Portal Guard

SSL Acceleration

Scalability and Redundancy

Certificate and Key Management

Public Key Infrastructure

Supported Key and Certificate Formats

Supported Handshake Protocols

Hash Algorithms

Cipher Suites

Management

Statistics

Virtual Desktop

Secure Portable Office (SPO) Client

Introducing the ASA 310-FIPS

HSM Overview

Extended Mode vs. FIPS Mode

FIPS140-1 Level 3 Security

The Concept of iKey Authentication

Types of iKeys

Wrap Keys for ASA 310-FIPS Clusters