Avaya IPsec User Manual

BayRS Version 14.00 Part No. 308630-14.00 Rev 00

September 1999 4401 Great America Parkway

Santa Clara, CA 95054

Configuring IPsec Services

and recommendations in this document are believed to be accurate and reliable, but are presented without express or implied warranty. Users must t ak e full re sponsib ility fo r th eir a pplic atio ns o f a ny products specified in this document. The information in this document is proprietary to Nortel Networks NA Inc.

The software described in this document is furnished under a license agreement and may only be used in accordance with the terms of that license. A summary of the Software License is included in this document.

Trademarks

NORTEL NETWORKS is a trademark of Nortel Networks. Bay Networks, ACE, AFN, AN, BCN, BLN, BN, BNX, CN, FRE, LN, Optivity, Optivity Policy Services, and PPX

are regi ster ed t rade mark s and Adv an ced Remo te Nod e, A NH, AR N, ASN, B ayRS , Ba ySecu re, BaySt ack , BaySt rea m, BCC, BCNX, BLNX, FN, Passport, SN, SPEX, Switch Node, Sy stem 5000, and T ok enSpeed are trademarks of Nortel Networks.

Microsoft, MS, Windows, and Windows NT are registered trademarks of Microsoft Corporation. All other trademarks and registered trademarks are t he property of their respective owners.

Restricted Rights Legend

Use, duplication, or disclosure by the United States Government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Sof tware clause at DFARS 252.227-7013.

Notwithstanding any other license agreement that may pertain to, or accompany the delivery of, this computer software, the rights of the United States Government regarding its use, reproduction, and disclosure are as set forth in the Commercial Computer Software-Restricted Rights cl ause at FAR 52.227-19.

Statement of Conditions

In the interest of improvi ng internal design, operational fun ction , a n d/o r relia bi lity, No rtel Ne tworks NA Inc. re serv e s the right to make changes to the products described in this document without notice.

Nortel Networks NA Inc. does not assume any liability that may occur due to the use or application of the product(s) or circuit layout(s) described herein.

Portions of the code in this software product may be Copyright © 1988, Regents of the University of California. All rights reserved. Redistribution and use in source and binary forms of such portions are permitted, provided that the above copyright notice and this paragraph are duplicated in all such forms and that any docu mentation, advertising materials, and other materials related to such distribution and use acknowledge that su ch portions of the software were developed by the University of California, Berkeley. The name of the University may not be used to endorse or promote products derived from such portions of the software without specific prior written permission.

SUCH PORTIONS OF THE SOFTWARE ARE PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

In addition, the program and information containe d herein are licensed only pursuant to a license agreement that contains restrictions on use and disclosure (that may incorporate by reference certain limitations and notices imposed by third parties).

Nortel Networks NA Inc. Software License Agreement

NOTICE: Please carefully read this license agre ement before copying or using the accompanying software or installing the hardware unit with pre-enabled software (each of which is referred to as “Software” in this Agreement). BY COPYING OR USING THE SOFTWARE, YOU ACCEPT ALL OF THE TERMS AND CONDITIONS OF

308630-14.00 Rev 00

THIS LICENSE AGREEMENT. THE TERMS EXPRESSED IN THIS AGREEMENT ARE THE ONLY TERMS UNDER WHICH NORTEL NETWORKS WILL PERMIT YOU TO USE THE SOFTWARE. If you do not accept these terms and conditions, return the product, unused and in the original shipping container, within 30 days of purchase to obtain a credit for the full purchase price.

1. License Grant. Nortel Networks NA Inc. (“Nortel Networks”) grants the end user of the Software (“Licensee”) a personal, nonex clusive, nontransfera ble lic ense: a) to u se the Softw are eit her on a single compute r or, if applicable, on a single authorized device identified by host ID, for which it was originally acquired; b) to copy the Software solely for backup purposes in support of authorized use of t he Software; and c) to use and copy the associated user manual solely in support of authoriz ed use of th e Softwa re b y Licen see. Thi s license applies t o the So ftware o nly and d oes not extend to Nortel Networks Agent software or other Nortel Networks software products. Nortel Networks Agent software or other Nortel Networks software products are licensed for use under the terms of the applicable Nortel Networks NA Inc. Software License Agreement that accompanies such software and upon payment by the end user of the applicable license fees for such software.

2. Restrictions on use; reservation of rights. The Software and user manuals are protected und er copyright laws. Nortel Networks and/or its licensors retain all title and ownership in both the Software and user manuals, including any revisions made by Nortel Networks or it s licensors. The copyright notice must be repr oduced and included wit h any copy of any portion of the Software or user manuals. Licensee may not modify, translate, decompile, disassemble, use for any competitive analysis, reverse engineer, distribute, or create derivative works from the Software or user manuals or any copy, in whole or in part. Except as expressly provided in this Agreement, Licensee may not copy or transfer the Software or user manuals, in whole or in part. The Software and user manuals embody Nortel Networks’ and its licensors’ confidential and propriet ary in telle c tu al pro p erty. Licensee shall not sublicense, assign, or ot he rwise disclose to any third party the Software, or any information about the operation, design, performance, or implementation of the Software and user manuals that is confidential to Nortel Networks and its licensors; however, Licensee may grant permission to its consultants, subcontractors, a nd agents to use the Softw are at Licensee’s facility, provided they have agreed to use the Software only in accordance with the terms of this license.

3. Limited warranty . Nortel Networks warrants each item of Software, as delivered by Nortel Networks and properly installed and operated on Nortel Networks hardware or other equipment it is originally licensed for, to function substantially as described in its accompanying user manual during its warranty period, which begins on the date Software is first shipped to Licensee. If an y item of S oftware f ails to so function d uring its w arranty period, as the sole remedy Nortel Networks will at its discretion provide a suitable fix, patch, or workaround for the problem that may be included in a future Software release. Nortel Networks further warrants to Licensee that the media on which the Software is provided will be free from defec ts in materials and wo rkman ship under no rmal use for a peri od of 90 da ys from the date Software is first shipped to Licensee. Nortel Networks will replace defective media at no charge if it is returned to Nortel Netw orks during the warranty period along with proof of the date of ship ment. This warranty does not apply if the media has been damaged as a result of accident, misuse, or abuse. The Licensee assumes all responsibility for selection of the Software to achieve Licensee’s intended results and for the installation, use, and results obtained from the Software. Nortel Networks does not warrant a) that the functions contained in the software will meet the Licensee’s requirements, b) that the Software will operate in the hardware or software combinations that the Licensee may select, c) that the operation of the Software will be uninterrupted or error free, or d) that all defects in the operation of the Softw are will be corrected . Nortel Network s is not obligated to remedy any Software de fect that cannot be reproduced with the latest Software release. These warranties do not apply to the Software if it has been (i) altered, except by Nortel Networks or in accordance with i ts instructions; (ii) used in conjunction with another vendor’s product, resulting in the de fect; or (iii) damage d by improper environment, abuse, misuse, accident, or negligence. THE FOREGOING WARRANTIES AND LIMITATIONS ARE EXCLUSIVE REMEDIES AND ARE IN LIEU OF ALL OTHER WARRANTIES EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITA TION ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Licensee is responsible for the security of its own data and information and for maintaining adequate procedures apart from the Software to reconstruct lost or altered files, data, or programs.

4. Limitation of liability. IN NO EVENT WILL NORTEL NETWORKS OR ITS LICENSORS BE LIABLE FOR ANY COST OF SUBSTITUTE PROCUREMENT; SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES; OR ANY DAMAGES RESULTING FROM INACCURATE OR LOST DATA OR LOSS OF USE OR PROFITS ARISING OUT OF OR IN CONNECTION WITH THE PERFORMANCE OF THE SOFTWARE, EVEN

308630-14.00 Rev 00

iii

IF NORTEL NETWORKS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL THE LIABILITY OF NORTEL NETWORKS RELATING TO THE SOFTWARE OR THIS AGREEMENT EXCEED THE PRICE PAID TO NORTEL NETWORKS FOR THE SOFTWARE LICENSE.

5. Government Licensees. This provision applies to a ll Softwa re and docum entation acquired d irectly or i ndirectly by or on behalf of the United States Government. The Software and documentation are commercial products, licensed on the open market at market prices, and were developed entirely at private expense and without th e use of any U.S. Government funds. The license to the U.S. Government is granted only with restricted rights, and use, duplication, or disclosure by the U.S. Government is subject to the restrictions set forth in subparagraph (c)(1) of the Commercial Computer Software––Restricte d Rig hts cla u se o f FAR 52.227-19 and the limitations set o ut in this license for ci vilian agencies, and subparagraph (c)(1)(ii ) of the Rights in Technical Data and Computer Software clause of DFARS

252.227-7013, for agencies of t he Department of Defense or their successors, whichever is applicable.

6. Use of Software in the European Community. This provision applies to all Software acquired for use within the European Community. If Licensee uses the Software within a country in the European Community, the Software Directive enacted by the Council of European Communities Directive dated 14 May, 1991, will apply to the examination of the Software to facilitate interoperability. Licensee agrees to notify Nortel Networks of any such intended examination of the Software an d may procure support and assistance from Nortel Networks.

7. Term and termination. This license is effective until terminated; however, all of the restrictions with respect to Nortel Networks’ copyright in the Software and user manuals will cease being effective at the date of expiration of the Nortel Networks copyright; those restrictions relating to use and disclosure of Nortel Networks’ confidential information shall continue in effect. Licensee may terminate this license at any time. The license will automatically terminate if Licensee fails to comply with any of the terms and conditions of the license. Upon termination for any reason, Licensee will immediat ely destroy or return to Nortel Networks the Software, user manuals, and all copies. Nortel Networks is not liable to Licensee for damages in any form solely by reason of the termination of this license.

8. Export and Re-export. Licensee agrees not to export, directly or indirectly, the Software or related technical data or information without first obtaining any required export licenses or other governmental approvals. Without limiting the foregoing, Licensee, on behalf of itself and its subsidiaries and affiliates, agrees that it will not, without first obtaining all export licenses and approvals required by the U.S. Government: (i) export, re-export, transfer, or divert any such Software or technical data, or any direct product thereof, to any country to which such exports or re-exports are restricte d or em b argoed under Un ite d Sta t e s e xport control law s an d r egulations, or to any national or resident of such restricted or embargoed countries; or (ii) provide the Software or related technical data or information to any military end user or for any military end use, including the design, development, or production of any chemical, nuclear, or biological weapons.

9. General. If any provision of this Agreement is held to be invalid or unenforceable by a court of competent jurisdiction, the remainder of the provisions of this Agreement shall remain in full force and effect. This Agreement will be governed by the laws of the state of California.

Should you have any questions concerning this Agreement, contact Nortel Networks, 4401 Great America Par kwa y, P.O. Box 58185, Santa Clara, California 9505 4-8185.

LICENSEE ACKNOWLEDGES THAT LICENSEE HAS READ THIS AGREEMENT, UNDERSTANDS IT, AND AGREES TO BE BOUND BY ITS TERMS AND CONDITIONS. LICENSEE FURTHER AGREES THAT THIS AGREEMENT IS THE ENTIRE AND EXCLUSIVE AGREEMENT BETWEEN NORTEL NETWORKS AND LICENSEE, WHICH SUPERSEDES ALL PRIOR ORAL AND WRITTEN AGREEMENTS AND COMMUNICATIONS BETWEEN THE PARTIES PERTAINING TO THE SUBJECT MATTER OF THIS AGREEMENT. NO DIFFERENT OR ADDITIONAL TERMS WILL BE ENFORCEABLE AGAINST NORTEL NETWORKS UNLESS NORTEL NETWORKS GIVES ITS EXPRESS WRITTEN CONSENT, INCLUDING AN EXPRESS WAIVER OF THE TERMS OF THIS AGREEMENT.

DES Code Software License Agreement

Portions of the software code for BayRS IPsec were written by Eric Young, and carry the following copyright:

308630-14.00 Rev 00

This software contains a DES implementation written by Eric Young (eay@mincom.oz.au). The implementation was written so as to conform with MIT's libdes.

This library is free for commercial and non-commercial use as long as the following conditions are adhered to. The following conditions apply to all code found in this distribution.

Copyright remains Eric Young's, and as such any Copyright notices in the code are not to be removed. If this package is used in a product, Eric Young should be given attribution as the author of that the SSL library. This can be in the form of a textual message at program startup or in documentation (online or textual) provided with the package.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and /or other materials provided with the distribution.

3. All advertising materials mentioning features or use of this software must display the following acknowledgment: This product includes software developed by Eric Young (eay@mincom.oz.au)

THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAU SED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

308630-14.00 Rev 00

Preface

Before You Begin ............................................................................................................. xv

Text Conventions .............................................................................................................xvi

Acronyms ........................... .......................... .......................... ......................... ................xvii

Hard-Copy Technical Manuals ........................................................... ....... ...... ....... ...... ....x ix

How to Get Help ..............................................................................................................xix

Chapter 1 Overview of IPsec

About IPsec ............................. ....... ...... ....... ...... ....... ...... ....... ...... ...... .............................1-2

Note Regarding IPsec and NAT ......................................................................................1-2

Network Requirements for Nortel Networks Routers ......................................................1-3

Supported Routers ...................................................................................................1-3

Supported WAN Protocols .......................................................................................1-3

IPsec Services ................................................................................................................1-4

Confidentiality ....... ....... ...... ....... ...... ....... ...... ....... ...... ....... ...... ...... ....... ...... ....... ...... ...1-4

Integrity ....................................................................................................................1-4

Authentication ..........................................................................................................1-4

Additional IPsec Services ........................................................................................1-5

How IPsec Works ...........................................................................................................1-5

IPsec Protection .......................................................................................................1-5

IPsec Tunnel Mode ...................................................................................................1-6

IPsec Elements ...............................................................................................................1-7

Security Gateways ...................................................................................................1-8

Security Policies .......................................................................................................1-8

Policy Templates ................................................................................................1-9

Inbound Policies ................................................................................................1-9

Outbound Policies ............................................................................................1-10

Policy Criteria Specification .............................................................................1-10

308630-14.00 Rev 00 vii

Security Associations .............................................................................................1-11

Automated Security Associations Using Internet Key Exchange (IKE) ...........1-12

Manual Security Associations ..........................................................................1-12

Security Associations for Bidirectional Traffic ..................................................1-12

How IKE Negotiates Security Associations .....................................................1-13

Security Parameter Index (SPI) .......................................................................1-13

Summarizing Security Policies and SAs ................................................................1-14

Security Protocols .........................................................................................................1-15

Encapsulating Security Payload .............................................................................1-15

Authentication Header ............................................................................................1-16

Internet Key Exchange (IKE) Protocol ..........................................................................1-17

Perfect Forward Secrecy ........................................................................................1-17

Chapter 2 Installing IPsec

Upgrading Router Software ............................................................................................2-2

Installing the IPsec Software ................ ....... ...... ....... ...... ....... ...... ...... ....... ...... ....... .........2-2

Completing the Installation Process .........................................................................2-3

Installing Triple DES Encryption ...............................................................................2-3

Securing Your Site ..........................................................................................................2-4

Securing Your Configuration ...........................................................................................2-4

Encryption Keys .......................................................................................................2-4

Random Number Generator (RNG) .........................................................................2-5

Creating and Using Node Protection Keys (NPKs) .........................................................2-5

Generating NPKs .....................................................................................................2-5

Entering an Initial NPK and a Seed for Encryption ..................................................2-6

Changing an NPK ....................................................................................................2-8

Monitoring NPKs ......................................................................................................2-8

Chapter 3 Starting IPsec

Enabling IPsec and IKE .................. ...... ....... ............................................. ...... ....... .........3-1

Creating Policies .............................................................................................................3-2

Specifying Criteria ....................................................................................................3-2

Specifying an Action .................................................................................................3-3

Policy Considerations ...............................................................................................3-3

viii 308630-14.00 Rev 00

Creating an Outbound Policy ...................................................................................3-4

Creating an Inbound Policy ......................................................................................3-6

Creating Security Associations .......................................................................................3-8

About Automated SA Creation .................................................... ....... ...... ....... ...... ...3-8

Creating an Outbound Protect Policy With A utomated SAs (IKE) ............................3-9

About Manual SA Creation .....................................................................................3-10

Creating a Protect SA Manually .............................................................................3-11

Creating an Unprotect SA Manually .......................................................................3-12

Chapter 4 Customizing IPsec

Changing Existing Policies .............................................................................................4-1

Editing a Policy ......................... ...... ....... ...... ....... ...... ....... ...... ...... .............................4-2

Adding a Policy .........................................................................................................4-3

PPP Protocol .............. ....... ...... ....... ...... ....... ...... ....... ...... ...... ....... ......................4-3

Frame Relay Protocol ........................................................................................4-4

Reordering Policies ..................................................................................................4-6

PPP Protocol .............. ....... ...... ....... ...... ....... ...... ....... ...... ...... ....... ......................4-6

Frame Relay ......................................................................................................4-7

Changing Existing Security Associations .......................................................................4-8

Automated SA (IKE) Modifications ...........................................................................4-8

Manual SA Modifications ..........................................................................................4-9

PPP Protocol .............. ....... ...... ....... ...... ....... ...... ....... ...... ...... ....... ......................4-9

Frame Relay ....................................................................................................4-10

Disabling IPsec .................. ...... ....... ...... ....... ...... ....... ...... ....... ...... .................................4-11

Appendix A Site Manager Parameters

Node Protection Key Para meter .................................................................................... A-1

Enabling IPsec Parameters ........................................................................................... A-2

IPsec Policy Parameters ................................................................................................ A-3

Manual Security Association Parameters ...................................................................... A-4

Automated Security Association (IKE) Parameters ....................................................... A-9

308630-14.00 Rev 00 ix

Appendix B Definitions of k Commands

Appendix C Configuration Examples

Inbound and Outbound Policies ..................................................................................... C-1

Automated SA (IKE) Policy Examples ..................................................................... C-2

Manual SA Policy Examples ................................................................................... C-5

Manual Protect and Unprotect SA Configuration ........................................................... C-9

Contivity Extranet Switch Interoperability ....................................................................C-16

Supported Versions ...............................................................................................C-16

Configuring Through a Browser ............................................................................ C-16

Termin ology .............................................................. ............................................. C -17

Configuration Specifics ......................................................................................... C-18

Feature Comparison Summary ............................................................................. C-19

Features Supported by Both Platforms ........................................................... C-19

BayRS Features Not Supported by Contivity .................................................. C-19

Contivity Features Not Supported by BayRS .................................................. C-20

BayRS IPsec and NAT .................................................................................... C-20

Troubleshooting Tips ............................. ...... ............................................. ....... ...... C -2 0

BayRS Tools ...................................................................................................C-20

Contivity Tools ................................................................................................. C-21

Symptoms You May See ................................................................................. C-21

Appendix D Protocol Numbers

Assigned Internet Protocol Numbers by Name .............................................................D-2

Assigned Internet Protocol Numbers by Number ..........................................................D-6

Index

x 308630-14.00 Rev 00

Figures

Figure 1-1. IPsec Environment: Unique Security Associations (SAs)

Between Routers .....................................................................................1-6

Figure 1-2. IPsec Concepts: Security Gateways, Security Policies, and SAs ............1-7

Figure 1-3. IPsec Security Gateways and Security Policies .......................................1-8

Figure 1-4. Security Associations for Bidirectional Traffic .........................................1-13

Figure C-1. IPsec Automated Outbound Policies for RTR1, RTR2, and RTR3 ..........C-2

Figure C-2. IPsec Manual Outbound Policies for RTR1, RTR2, and RTR3 ...............C-5

Figure C-3. Single Protect/Unprotect SA Pair ............................................................ C-9

Figure C-4. Multiple Protect/Unprotect SA Pairs ...................................................... C-12

308630-14.00 Rev 00 xi

Tables

Table 1-1. Security Policy Specifications ................................................................1-14

Table 1-2. Manual Security Association (SA) Configurations .................................1-15

Table C-1. Comparison of BayRS and Contivity Terminology ................................C-17

Table D-1. Internet Protocol Numbers, Sorted by Acronym ....................................D-2

Table D-2. Internet Protocol Numbers, Sorted by Number ...................................... D-6

308630-14.00 Rev 00 xiii

This guide describes the Nortel Networks™ implementation of IP Securi ty and how to configure it on a Nortel Networks router.

Before You Begin

Before using this guide, you must complete the following procedures. For a new router:

• Install the router (see the installation guide that came with your router).

• Connect the router to the network and create a pilot configuration file (see

Quick-Starti ng Router s , Conf igur ing BaySt ac k Remote Acc ess , or Connecting ASN Routers to a Network).

Preface

Make sure that you are runni ng the lates t versio n of Nortel Netw orks BayRS Site Manager software. For information about upgrading BayRS and Site Manager, see the upgrading guide for your version of BayRS.

308630-14.00 Rev 00 xv

™

and

Configuring IPsec Services

Text Conventions

This guide uses the following text conventions:

angle brackets (< >) Indicate that you choose the text to enter based on the

description inside the brackets. Do not type the brackets when entering the command.

Example: If the command syntax is:

ping

ip_address

ping 192.32.10.12

>, you enter:

bold text

Indicates command names and options and text that you need to enter.

Example: Enter

show ip {alerts | routes

Example: Use the

dinfo

command.

braces ({}) Indicate required elements in syntax descriptions

where there is more than one option. You must choose only one of the options. Do not type the braces when entering the command.

Example: If the command syntax is:

show ip {alerts | routes show ip alerts or show ip routes

}

, you must enter either:

, but not both.

brackets ([ ]) Indicate optional elements in syntax descriptions. Do

not type the brackets when entering the command. Example: If the command syntax is:

show ip interfaces [-alerts show ip interfaces

]

, you can enter either:

show ip interfaces -alerts

italic text Indicates file and directory names, new terms, book

titles, and variables in command syntax descriptions. Where a variable is two or mor e words, the words are connected by an underscore.

Example: If the command syntax is:

show at

valid_route

is one variable and you substitute one value

for it.

xvi 308630-14.00 Rev 00

Preface

screen text Indicates system output, for example, prompts and

system messages.

Acronyms

Example:

Set Trap Monitor Filters

separator ( > ) Shows menu paths.

Example: Protocols > I P ide nti fies the I P opt ion on the Protocols menu.

vertical line (

) Separates choices for command keywords and

arguments. Enter only one of the choices. Do not type the vertical line when enteri ng the command.

Example: If the command syntax is:

show ip {alerts | routes show ip alerts

show ip routes

This guide uses the following acronyms:

3DES Triple DES AH Authentication Header CBC cipher block chaining

}

, you enter either:

, but not both.

CES Contivity Extranet Switch DES Data Encryption Standard ESP Encapsulating Security Payload HMAC Hashing Message Authentication Code IANA Internet Assigned Numbers Authority ICMP Internet Control Me ssage Protocol ICV integri ty check value IETF Internet Engineering Task Force IKE Internet Key Exchange protocol IP Internet P rotocol IPsec Internet Protocol Security

308630-14.00 Rev 00 xvii

Configuring IPsec Services

ISAKMP/Oakley Internet Security Association and Key Management

IV initialization vector MD5 Message Digest 5 MIB management information base NAT Network Address Translation NPK node protection key NVRAM nonvolatile random access memory PFS Perfect Forward Secrecy PPP Point-to-Point Protocol RFC Request for Comments RNG random number generator RSA RSA Data Security, Inc.’s public-key encryption

SA security association

Protocol (also known as IKE)

algorithm

SAD security association database SHA Secure Hash Algorithm SNMP Simple Network Management Protocol SPD security policy database SPI security parameter index VPN virtual private network WAN wide area network WEP WAN Encryption Protocol

xviii 308630-14.00 Rev 00

Hard-Copy Technical Manuals

You can print selected technical manuals and release notes free, directly from the Internet. Go to support.baynetworks.com/library/tpubs/. Find the product for which you need documentation. Then locate the specific category and model or version for your hardw are or soft ware product . Usi ng Adobe Ac robat Re ader, you can open the manuals and releas e notes, search for the sections you ne ed, and print them on most standard printers. You can download Acrobat Reader free from the Adobe Systems Web site, www.adobe.com.

You can purchase selected documentation sets, CDs, and technical publications through the collateral catalog. The catalog is located on the World Wide Web at

support.baynetworks.com/catalog.html and is divided into sections arranged

alphabetically:

• The “CD ROMs” section lists available CDs.

• The “Guides/Books” section lists books on technical topics.

• The “Technical Manuals” section lists available printed documentation sets.

Preface

How to Get Help

If you purchased a service contract for your Nortel Networks product from a distributor or authorized reseller, contact the technical support staff for that distributor or reseller for assistance.

If you purchased a Nort el Net wor ks ser vice pr ogram, c ontact one of the f ollowing Nortel Networks Technical Solutions Centers:

Technical Solutions Center Telephone Number

Billerica, MA 800-2LANWAN (800-252-6926) Santa Clara, CA 800-2LANWAN (800-252-6926) Valbonne, France 33-4-92-96-69-68 Sydney, Australia 61-2-9927-8800 Tokyo, Japan 81-3-5402-7041

308630-14.00 Rev 00 xix

Chapter 1

Overview of IPsec

This chapter descr ibes the emer ging Inte rnet Engineer ing Task Force standard s for security services over publ ic networks, commonly referred to as IP Security or IPsec. The chapter also includes information specific to the Nortel Networks implementation of IPsec and requirements for that implementation.

This chapter includes the following information:

Topic Page

™

About IPsec 1-2 Note Regarding IPsec and NAT 1-2 Network Requirements for Nortel Networks Routers 1-3 IPsec Services 1-4 How IPsec Works 1-5 IPsec Elements 1-7 Security Gateways 1-8 Security Policies 1-8 Security Associations 1-11 Summarizing Security Policies and SAs 1-14 Security Protocols 1-15 Internet Key Exchan ge (IKE) Protocol 1-17

308630-14.00 Rev 00

1-1

Configuring IPsec Services

About IPsec

IP Security (I Psec) is the Internet Enginee ring Task Force (IETF) set of emerging standards for security services for communications over public networks. The standards are documented in the IETF Requests for Comments (RFCs) 2401 through 2412. Additional RFCs may be relevant as well.

These standards were developed to ensure secure, private communications for the remote access, extranet, and intranet v irtual private networks (VPNs) used in enterprise communications. They are the security architecture for the next generation of IP, cal led IPv6, but are available for the current IPv4 Internet as well.

The Nortel Networks implementation of the IETF standards provides network (layer 3) security services for wide area network (WAN) communications on Nortel Networks routers.

Note Regarding IPsec and NAT

IPsec and Network Addr ess Translation (NA T) are not support ed to w ork t ogethe r on a BayRS platform. NAT or IPsec can process a packet, but not by both. If a packet matches the NAT source address range, NAT takes precedence over IPsec and IPsec will not see the packet.

1-2

308630-14.00 Rev 00

Overview of IPsec

Network Requirements for Nortel Networks Routers

To install the IP Security (IPsec) software, the router must be running BayRS Version 13.10 or later and Site Manager Version 7.10 or later. To use IKE and automated SAs, BayRS Version 13.20 and Site Manager Version 7.20 or later are required.

Supported Routers

Nortel Networks IP technologies are implemented on BayRS router interfaces supporting synchronous communications.

IPsec can pro vid e enc rypti on and a ut hentic atio n serv ice s to an y s erial int erf ace o n the following routers:

•BayStack

• BayStack Access Stack Node (ASN

• BayStack Advanced Remote Node

• Backbone Node (BN

• System 5000

™

Access Node (AN®)

)

™

router modules

™

)

™

(ARN™)

™

Contivity

Extranet Switch™ (CES) hardware also supports IPsec. CES does not use BayRS software, but can be configured to interoperate with it. Refer to

“Contivity Extranet Switch Interoperability” on page C-16

documentation for more inform ation.

Supported WAN Protocols

The Nortel Networks implementation of IPsec supports PPP and frame relay WAN protocols. The Nortel Networks IPsec implementation also supports dial services, which provide backup and demand services for PPP and frame relay.

308630-14.00 Rev 00

and the Contivity

1-3

Configuring IPsec Services

IPsec Ser vices

IPsec serv ices consist of confidential ity, integrity, and authentication servi ces for data packets traveling between sec urity gatew a ys.

• Confidentiality ensures the privacy of communications.

• The integrity service detects modification of data packets.

• Authentication services verify the origin of every data packet.

Confidentiality

Confidentiality is accomplished by encrypting and decrypting data packets. The Encapsulating Security Payload (ESP) protocol uses the Data Encrypt ion Standard (DES) algorithm in cipher block chaining (CBC) mode to encrypt and decrypt data packets.

You set confidentiality with the cipher algorithm and cipher key parameters. The cipher algorithm and cipher key are specified in security associations (SAs). A security association is a relationship in which two peers share the necessary information to secur ely prote ct and unpr otect data. Th e algori thm and ke y must b e identical on both ends of an IPsec SA.

Integrity

Integrity determines whether the data has been altered d uring transit. T he ESP protocol ensures that data has not been modified as it passes between the security gateways . The ESP protoco l uses the HMAC MD5 (RFC 2403) or HMAC SHA-1 (RFC 2404) transform.

You set integrity with the integrity algorithm and integrity key parameters. The integrity algorithm and integrity key must be identic al on both ends of an IPsec SA.

Authentication

Authentication ensures that data has been transmitted by the identified source.

1-4

308630-14.00 Rev 00

Additional IPsec Services

Within the IPsec framework, additional security services are provided. An access control service ensures authorized use of the network, and an auditing service tracks all actions and events.

IPsec services can be configured on an interface-by-interface basis. Up to 127 inbound and 127 outbound security policies (customized) are supported on each IPsec interface.

How IPsec Works

IPsec services are bundled as an Internet Protocol (IP) encryption packet. The packets resemble ordinary IP packets to Internet routing nodes; only the sending and receiving devices are involved in the encryption. IPsec packets are delivered over the Internet like ordinary IP packets to branch offices, corporate partners, or other remote organizations in a secure, encrypted, and private manner.

Sever al well-est ablished tech nologies pro vide enc ryption and aut henticatio n at the application laye r. IPsec adds security at the underl ying network layer, providing a higher degree of secur ity fo r all a ppl icati ons, inc luding those wit hou t an y secur ity features of their own.

Overview of IPsec

IPsec Protection

To configure a router with IPsec, you first configure the router interface as an IP interface. Then you add the IPsec software to the IP interface, creating a security gateway. A security gateway is a router between a trusted network (for example, the enterprise intranet) and an untrusted network (the Internet) that provides a security service such as IPsec.

The router interface is secured with inbound and outbound security policies that filter traffic to and from the router module. The data packets themselves are protected by IPsec protocol processing specified by security associations (SAs).

308630-14.00 Rev 00

1-5

Configuring IPsec Services

Figure 1-1 sho ws ho w IPsec can prote ct data c ommunication s within a n enterpr ise and from external hosts.

Corporate

headquarters

Server

Router A

IP security

gateway

Security

associations

(SAs A,B)

Partner

Router B Router C

Host Host

IP security

gateway

IPsec

services

Public

network

Security associations

(SAs B,C)

IPsec

services

Security

associations

(SAs C,A)

Branch office

IP security

gateway

IPsec

services

IP0088A

Figure 1-1. IPsec Environment: Unique Security Associations (SAs)

Between Routers

IPsec Tunnel Mode

When there is a security gateway at each end of a communication, the security associations between the gateways are said to be in tunnel mode. The tunnel metaphor refers to data being visible only at the beginning and end points of the communication. The IP packets protected by IPsec have regular, “visible” IP headers, but the packet contents are encrypted, and thus hidden. All BayRS IPsec communications occur in tunnel mode. Tunnel mode is especially effective for isolating and prot ecting enterp rise traf f ic tra veli ng across a publ ic data net work, as shown in Figure 1-1.

1-6

308630-14.00 Rev 00

IPsec Elements

IPsec has three important constructs:

• Security gateways

• Security policies

• Security associations (SAs) In the IPsec context, hosts communicate across an untrusted network through

security gateways (routers configured for IPsec interfaces). Security policies determine ho w the IPsec interfaces handle data packets fo r the host s on both ends of a connection. Security associations apply IPsec services to data packets traveling between the security gateways.

Overview of IPsec

Figure 1-2

associations.

Security associations

Unprotect SAs Source/Dest Addr, SPI Cipher Algo/Key, Integrity Algo/Key

Protect SAs Source/Dest Addr, SPI Cipher Algo/Key, Integrity Algo/Key

shows the logical relationship between security policies and security

IPsec gateway WAN interface

Inbound process

Inbound policies

criteria & action

(bypass, drop, log)

Outbound policies

criteria & action

(bypass, drop, log,

protect)

Outbound process

Security

policy

database

Untrusted

network

Figure 1-2. IPsec Concepts: Security Gateways, Security Policies, and SAs

308630-14.00 Rev 00

IP0087A

1-7

Configuring IPsec Services

Security Gateways

A security gateway establishes SAs between router interfaces configured with IPsec software. A Nortel Networks router becomes a security gateway when you enable IPsec on a WAN interface . In t his w a y, a Nortel Networks router opera ti ng as a security gateway provides IPsec services to its internal hosts and subnetworks.

Hosts or networks on th e e xte rnal si de of a sec urit y ga te w ay (typic ally, the overall Internet) are considered “untrusted.” Hosts or subnetworks on the internal side of a security gat e w ay (nodes on your l ocal i ntra net) are consi dered “trus te d” beca use they are controlled and securely managed by the same network administration (Figure 1-3

Trusted network

Local host

Outbound policy

Security gateway

Inbound policy (clear text only)

IPsec interface

Untrusted

network

IPsec interface

Outbound policy

Security gateway

Inbound policy (clear text only)

Figure 1-3. IPsec Security Gateways and Security Policies

When you add IPsec services to a router to create a security gateway, its internal hosts and subnetworks can communicate with external hosts that directly operate IPsec services, or with a remote security gateway that provides IPsec services for its set of hosts and subnetworks.

Security Policies

When you create an IPsec policy, you control which packets a security gateway protects, how it handles packets to or from particular addresses or in a particular protocol, and whether it logs information about these actions.

Trusted network

Remote host

IP0078A

1-8

308630-14.00 Rev 00

Overview of IPsec

There are two types of IPsec policies: inbound and outbound. An inbound policy is used for data packets arriving at a security gateway, and an outbound policy is used for data pa ck ets leaving a security gateway. Each I Psec interface c an support up to 127 inbound and 127 outbound security policies (refer to Figure 1-3

page 1-8

The criteria (“selectors”) and action specifications used in your inbound and outbound policies are stored in the security policy database (SPD).

IPsec defaults i n fa v or of more securit y rather th an less. I f an outbou nd or inbou nd packet does not match the criteria of any configured outbound or inbound policy in the SPD, the packet is dropped.

IPsec discards an y out bound clear-text da ta pack et u nle ss you explicitly conf i gure a policy to bypass or protect it.

Po licy Templates

Every IPsec polic y is ba sed on a policy template. A policy template is a pr edef ined policy definition that you can use on any IP interface. The template specifies one or more criteria and an action to apply to incoming or outgoing data packets.

A policy template and every poli cy based on it must include at least one criterion, for example, an IP source address, and one action. For example, an outbound policy might specify a pr otect ac tion. A poli cy t emplate or po lic y may inc lude tw o actions if one of the actions is logging. The criterion specification determines whether a data pack et matches a pa rticula r securit y polic y, and the action specifi es how the policy is applied to the packet.

The action specifications that you can include in inbound and outbound policies are listed in the two sections that follow.

Inbound Polici es

An inbound policy determines how a security gateway processes data packets receiv ed from a n u ntrus ted ne tw ork. Ev ery pack e t ar ri v ing at a secu rity g ateway is compared with the criteria to determine whether it matches an IPsec policy for that router. If the incoming packet matches a bypass policy, the router accepts the packet and, if the policy is so configured, logs it.

308630-14.00 Rev 00

1-9

Configuring IPsec Services

If the packet d oes n ot mat ch an y poli cy o r matches a drop poli c y, the router rejects the packet. When a packet does not match any policy, IPsec’s default action is to drop it.

For an inbound security policy, the action may be:

•Drop

• Bypass

•Log Drop and bypass are mutua lly e xc lusive. The log action may be added to ei ther, or

used alone.

Outbound Policies

An outbound policy determines ho w a se curity gat e way proces ses data pac kets f or transmission across an untrust ed netwo rk. You must assign an out bound poli cy fo r all unicast traffic leaving an IPsec interface.

For an outbound policy, the action specification may be:

• Protect

1-10

•Drop

• Bypass

•Log Any outbound policy with a protect action specification is mapped to a Protect

SA. See “

Summarizing Security Policies and SAs” on page 1-14 for detailed

information about Protect and Unprotect SAs. Drop, protect, and bypass are mutually exclusive. The log action may be added to

any of the three, or used alone.

Policy Criteria Specification

IPsec software inspects IP packet headers based on the specified criteria to determine whether a policy applies to a data packet.

You must include at least one of the following criteria, and you may specify all three criteria in an IPse c policy:

• IP source address

308630-14.00 Rev 00

+ 92 hidden pages

Avaya IPsec User Manual

Configuring IPsec Services

Contents

Figures

Tables

Before You Begin

Preface

Text Conventions

Acronyms

Hard-Copy Technical Manuals

How to Get Help

About IPsec

Note Regarding IPsec and NAT

Network Requirements for Nortel Networks Routers

Supported Routers

Supported WAN Protocols

IPsec Ser vices

Confidentiality

Integrity

Authentication

Additional IPsec Services

How IPsec Works

IPsec Protection

IPsec Tunnel Mode

IPsec Elements

Security Gateways

Security Policies

Po licy Templates

Inbound Polici es

Outbound Policies

Policy Criteria Specification