Oracle Dual Port 25 Gb User Manual

Download

Page 1

Oracle® Dual Port 25 Gb Ethernet Adapter User's Guide

Part No: E87989-01

September 2017

Page 2

Page 3

Oracle Dual Port 25 Gb Ethernet Adapter User's Guide

Part No: E87989-01

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?

ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

Page 4

Référence: E87989-01

Ce logiciel et la documentation qui l'accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des restrictions d'utilisation et de divulgation. Sauf stipulation expresse de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire, diffuser, modifier, accorder de licence, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et par quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à des fins d'interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.

Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu'elles soient exemptes d'erreurs et vous invite, le cas échéant, à lui en faire part par écrit.

Si ce logiciel, ou la documentation qui l'accompagne, est livré sous licence au Gouvernement des Etats-Unis, ou à quiconque qui aurait souscrit la licence de ce logiciel pour le compte du Gouvernement des Etats-Unis, la notice suivante s'applique :

Ce logiciel ou matériel a été développé pour un usage général dans le cadre d'applications de gestion des informations. Ce logiciel ou matériel n'est pas conçu ni n'est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer un risque de dommages corporels. Si vous utilisez ce logiciel ou ce matériel dans le cadre d'applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliés déclinent toute responsabilité quant aux dommages causés par l'utilisation de ce logiciel ou matériel pour des applications dangereuses.

Oracle et Java sont des marques déposées d'Oracle Corporation et/ou de ses affiliés. Tout autre nom mentionné peut correspondre à des marques appartenant à d'autres propriétaires qu'Oracle.

Intel et Intel Xeon sont des marques ou des marques déposées d'Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marques déposées de SPARC International, Inc. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d'Advanced Micro Devices. UNIX est une marque déposée de The Open Group.

Ce logiciel ou matériel et la documentation qui l'accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers, sauf mention contraire stipulée dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l'accès à des contenus, produits ou services tiers, ou à leur utilisation, sauf mention contraire stipulée dans un contrat entre vous et Oracle.

Accès aux services de support Oracle

Les clients Oracle qui ont souscrit un contrat de support ont accès au support électronique via My Oracle Support. Pour plus d'informations, visitez le site http://www.oracle.com/

pls/topic/lookup?ctx=acc&id=info ou le site http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs si vous êtes malentendant.

Page 5

Using This Documentation ................................... .............................................. 9

Product Documentation Library . ..................................................................... ... 9

Feedback ................................... .................................................................... 9

Understanding the Installation Process .. .......................................................... 11

Installation Task Overview (Oracle Solaris x86) ............. ..................................... 11

Installation Task Overview (Linux) ............................................................... .... 12

Installation Task Overview (Windows) ....................................................... ....... 13

Understanding the Adapter .............................................. .................... ............. 15

Shipping Kit Contents ................................... ................................................. 15

Adapter Overview .... ..................................................................... ................ 16

Front Panel Connectors and LEDs ............................... .................... ................. 17

Confirming Specifications and Requirements ................... ................................ 19

Physical Specifications ....................................................... ............................ 19

Electrical Specifications ...... ..................................................................... ....... 20

Environmental Specifications ......................................................... .................. 20

Hardware and Software Requirements ........... .................... ................................ 21

Updating Software and Firmware .................................. .................................... 23

▼ Update the OS (Oracle Solaris x86) ....................................................... ...... 23

Firmware Update Tool Overview ......................................... ............................ 24

Related Information ............... ................................................................ 24

Installing the Driver ........................................ .................... ............................... 25

▼ Verify the bnxt Driver (Oracle Solaris x86) .................................... .............. 25

Page 6

Contents

▼ Download and Install the bnxt_en Driver (Linux) ............................ .............. 26

▼ Verify the bnxt_en Driver (Linux) .. ................................................. ........... 27

▼ Download and Install the bnxt_en Driver (Windows) .......................... ............ 28

Installing the Adapter .. .................... .................................................................. 31

Adapter Description ............................................ .................... ....................... 32

Related Information ............... ................................................................ 32

Cable and Transceiver Overview ..................................................... ................. 32

Cable Cautions ............. ..................................................................... ............ 33

Related Information ............... ................................................................ 34

▼ Install the Adapter ................................... ................................................. 34

▼ Install the SFP28 Transceivers ... .................... ............................................. 35

▼ Verify the Adapter Installation (Oracle Solaris x86) ....... .................... ............. 36

▼ Verify the Adapter Installation (Linux) .......... .................... ........................... 37

▼ Verify the Adapter Installation (Windows) ...... .............................................. 37

Configuring the Network ............................................ ....................................... 39

▼ Configure the Network Interface (Oracle Solaris x86) ..................................... 40

▼ Configure the Network Interface (Windows) ................................................. 40

Boot Options ...................... .................... ...................................................... 41

▼ Boot Over the Network (PXE) ........................................................... ........ 41

▼ Boot Over a 10GbE Network (Oracle Solaris x86 and Linux) ........................... 42

Installing RoCE ............. .................... ........................................................... 44

▼ Install RoCE (Linux) ......................................... ............................... 45

▼ Install RoCE in Windows Kernel Mode ..................... ........................... 46

▼ Install RoCE in Windows User Mode ...... .................... ........................ 47

▼ Install RoCE in VMware ESXi ................ .................... ....................... 47

Verifying RoCE ......................................................... .................... ............... 48

▼ Verify RoCE in Windows .............. .................... ................................ 48

Configuring Driver Parameters ....................................... .................................. 51

▼ Set Driver Parameters (Oracle Solaris x86) .......................................... ......... 51

Driver Parameters (Oracle Solaris x86) ......................................................... .... 53

▼ Set Driver Parameters (Linux) ............... ..................................................... 53

Driver Parameters (Linux) ........ ..................................................................... . 54

6 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 7

Contents

Configuring Jumbo Frames .................................................................. ............ 55

▼ Change the MTU Permanently (Oracle Solaris x86) ....................... ................. 55

▼ Change the MTU Temporarily (Oracle Solaris x86) ........................................ 56

▼ Configure Jumbo Frames (Linux) .................................................. .............. 56

▼ Configure Jumbo Frames (Windows) ............... ............................................ 57

Configuring a Link Aggregation (Oracle Solaris x86) ........................................ 59

▼ Configure a Link Aggregation (Oracle Solaris x86) ................................ ........ 59

▼ Display Information About a Link Aggregation (Oracle Solaris x86) .................. 60

▼ Delete a Link Aggregation (Oracle Solaris x86) ............. ................................ 61

Configuring VLANs and VXLANs .............. ........................................................ 63

VLANs Overview .... ..................................................................... ................. 63

▼ Configure VLANs (Oracle Solaris x86) ..................................... ................... 64

▼ Configure VLANs (Linux) .................................... ..................................... 65

▼ Configure VLANs (Windows) ........................................................... ......... 65

▼ Configure VXLANs (Oracle Solaris x86) . .................................................... 66

Removing the Driver ............................... .......................................................... 69

▼ Remove the bnxt Driver (Oracle Solaris x86) ........................... .................... . 69

▼ Remove the bnxt_en Driver (Linux) ............................... ............................. 70

▼ Remove the bnxt_en Driver (Windows) ....... ................................................ 70

Upgrading the Adapter (Linux) ......................................... ................................ 71

▼ Install the Upgrade Package ............................... ........................................ 71

▼ Verify the Upgrade .................................................. .................... ............. 72

Troubleshooting the Adapter (Oracle Solaris x86) ................................... ......... 75

▼ Analyze Why the Device Link Is Missing (Oracle Solaris x86) ...... ................... 75

▼ Recover From a Port Hang (Oracle Solaris x86) ........................................... . 76

▼ Analyze Slow Network Performance (Oracle Solaris x86) ................... ............. 77

▼ Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle

Solaris x86) .. ..................................................................... ........................... 78

▼ Analyze Why Changing the MTU Does Not Correctly Set the Link Property

(Oracle Solaris x86) .... ..................................................................... .............. 79

Page 8

Contents

Glossary ................................... ..................................................................... .... 81

Index ................................... ..................................................................... .......... 85

8 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 9

Using This Documentation

■

Overview – Provides specifications and describes how to install and administer the Oracle Dual Port 25 Gb Ethernet Adapter.

■

Audience – Technicians, system administrators, and authorized service providers.

■

Required knowledge – Advanced experience troubleshooting and replacing hardware.

In this document, the term “adapter” refers to the Oracle Dual Port 25 Gb Ethernet Adapter, the term “x86” refers to 64-bit and 32-bit systems manufactured using processors compatible with the AMD64, Intel Xeon, or Intel Pentium product families , and the term “Oracle Solaris” refers to Oracle Solaris 11.3 SRU23 for x86.

Product Documentation Library

Documentation and resources for this product and related products are available at https://

docs.oracle.com/cd/E87266_01/.

Feedback

Provide feedback about this documentation at http://www.oracle.com/goto/docfeedback.

Using This Documentation 9

Page 10

10 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 11

Understanding the Installation Process

These topics provide an overview of the installation process for the adapter:

■

“Installation Task Overview (Oracle Solaris x86)” on page 11

■

“Installation Task Overview (Linux)” on page 12

■

“Installation Task Overview (Windows)” on page 13

Related Information

■

“Understanding the Adapter” on page 15

■

“Confirming Specifications and Requirements” on page 19

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Installation Task Overview (Oracle Solaris x86)

Step Description Links

1. Understand the adapter. “Understanding the Adapter” on page 15

2. Confirm the adapter specifications and technical requirements.

3. Determine if the driver is supported on your server and the driver is up to date.

4. If your OS is out of date, update the entire OS image, or download and apply the latest OS patch.

5. Verify the driver installation.

“Physical Specifications” on page 19

“Electrical Specifications” on page 20

“Environmental Specifications” on page 20

“Hardware and Software Requirements” on page 21

“Update the OS (Oracle Solaris x86)” on page 23

“Verify the bnxt Driver (Oracle Solaris x86)” on page 25

Understanding the Installation Process 11

Page 12

Installation Task Overview (Linux)

Step Description Links

6. Install and verify the adapter. “Install the Adapter” on page 34

“Verify the Adapter Installation (Oracle Solaris x86)” on page 36

7. Configure the network. “Configure the Network Interface (Oracle Solaris x86)” on page 40

“Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42

8. Configure the driver parameters. “Set Driver Parameters (Oracle Solaris x86)” on page 51

10. (Optional) Configure link aggregation. “Configure a Link Aggregation (Oracle Solaris x86)” on page 59

11. (Optional) Configure a VLAN or a VXLAN. “Configure VLANs (Oracle Solaris x86)” on page 64

“Configure VXLANs (Oracle Solaris x86)” on page 66

12. If desired, remove a driver.

13. Upgrade the adapter. “Install the Upgrade Package” on page 71

14. Troubleshoot adapter issues. “Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

“Remove the bnxt Driver (Oracle Solaris x86)” on page 69

Related Information

■

“Installation Task Overview (Linux)” on page 12

■

“Installation Task Overview (Windows)” on page 13

Installation Task Overview (Linux)

Step Description Links

1. Understand the adapter. “Understanding the Adapter” on page 15

2. Confirm the adapter specifications and technical requirements.

3. Determine if the driver is supported on your server and the driver is up to date.

4. If your OS is out of date, update the entire OS image, or download and apply the latest OS patch.

5. Install and verify the driver.

6. Install and verify the adapter. “Install the Adapter” on page 34

“Physical Specifications” on page 19

“Electrical Specifications” on page 20

“Environmental Specifications” on page 20

“Hardware and Software Requirements” on page 21

“Updating Software and Firmware” on page 23

“Download and Install the bnxt_en Driver (Linux)” on page 26

“Verify the bnxt_en Driver (Linux)” on page 27

“Verify the Adapter Installation (Linux)” on page 37

12 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 13

Installation Task Overview (Windows)

Step Description Links

7. Boot over the network. “Boot Over a 10GbE Network (Oracle Solaris x86 and

8. Configure the driver parameters. “Set Driver Parameters (Linux)” on page 53

9. (Optional) Configure jumbo frames. “Configure Jumbo Frames (Linux)” on page 56

10. (Optional) Configure VLANs. “Configure VLANs (Linux)” on page 65

11. (Optional) Remove a driver.

12. Upgrade the adapter. “Install the Upgrade Package” on page 71

Linux)” on page 42

“Driver Parameters (Linux)” on page 54

“Remove the bnxt_en Driver (Linux)” on page 70

Related Information

■

“Installation Task Overview (Oracle Solaris x86)” on page 11

■

“Installation Task Overview (Windows)” on page 13

Installation Task Overview (Windows)

Step Description Links

1. Understand the adapter. “Understanding the Adapter” on page 15

2. Confirm the adapter specifications and technical requirements.

3. Determine if the driver is supported on your server and the driver is up to date.

4. If your OS is out of date, update the entire OS image, or download and apply the latest OS patch.

5. Verify the driver installation.

6. Install the adapter and verify the installation. “Install the Adapter” on page 34

7. Boot over the network. “Configure the Network Interface (Windows)” on page 40

8. Configure jumbo frames. “Configure Jumbo Frames (Windows)” on page 57

9. Configure a VLAN. “Configure VLANs (Windows)” on page 65

10. If desired, remove a driver.

11. Upgrade the adapter. “Install the Upgrade Package” on page 71

“Physical Specifications” on page 19

“Electrical Specifications” on page 20

“Environmental Specifications” on page 20

“Hardware and Software Requirements” on page 21

“Updating Software and Firmware” on page 23

“Download and Install the bnxt_en Driver (Windows)” on page 28

“Verify the Adapter Installation (Windows)” on page 37

“Remove the bnxt_en Driver (Windows)” on page 70

Understanding the Installation Process 13

Page 14

Installation Task Overview (Windows)

Related Information

■

“Installation Task Overview (Oracle Solaris x86)” on page 11

■

“Installation Task Overview (Linux)” on page 12

■

“Understanding the Adapter” on page 15

14 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 15

Understanding the Adapter

These topics provide an overview of the adapter:

■

“Shipping Kit Contents” on page 15

■

“Adapter Overview” on page 16

■

“Front Panel Connectors and LEDs” on page 17

Related Information

■

“Understanding the Installation Process” on page 11

■

“Confirming Specifications and Requirements” on page 19

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Configuring Jumbo Frames” on page 55

■

“Configuring a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Configuring VLANs and VXLANs” on page 63

■

“Removing the Driver” on page 69

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Shipping Kit Contents

The carton in which the adapter was shipped contains these items:

■

Adapter with a low-profile bracket attached

■

Dual Port 25Gb Ethernet Adapter Where To Find documentation

Understanding the Adapter 15

Page 16

Adapter Overview

Related Information

■

“Adapter Overview” on page 16

■

“Front Panel Connectors and LEDs” on page 17

Adapter Overview

The Dual Port 25Gb Ethernet Adapter is a standalone card that is used for virtualized cloud deployments and network virtualization. Each adapter port uses a 25GbE SFP28 optical

transceiver with an LC/LC cable or a copper DAC that has transceivers attached at each end.

Both ports operate at the same speed. The adapter is a PCIe 3.0 x8 low-profile card.

Feature Specification

Data rate supported per port 2 x 25 GbE, where both ports operate at 25 GbpsGbps per port

Bus type PCIe V3.0, 8.0 GTps

Bus width 8-lane PCIe bus connector

Conforms to Ethernet standard 802.3

Boot ROM 8 Mb SPI Flash

EMI FCC Class A

Related Information

■

“Shipping Kit Contents” on page 15

■

“Front Panel Connectors and LEDs” on page 17

16 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 17

Front Panel Connectors and LEDs

On the front panel, two LEDs display the port speed and activity for each port. Both ports operate at the same speed. This figure and the table explain the meaning of the LEDs for ports 1 and 2. The LEDs are the same for a full panel and a half-height panel.

Note - If your adapter is installed horizontally, the ports are numbered 1 and 2, left to right. In a

vertical orientation, the ports are numbered 1-2 bottom to top.

Front Panel Connectors and LEDs

No. Description Device Mbit

1 Port 1

2 Port 2

3 Activity LED Off

Blinking Green

4 Link LED Off

Solid Green

Solid Yellow

Related Information

■

“Shipping Kit Contents” on page 15

■

“Adapter Overview” on page 16

■

“Confirming Specifications and Requirements” on page 19

No activity

Traffic flowing activity

No link

Linked at 25 Gbps

Linked at 1 Gbps or 10 Gbps

Understanding the Adapter 17

Page 18

18 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 19

Confirming Specifications and Requirements

These topics provide technical information and airflow precautions you need to understand before installing the adapter:

■

“Physical Specifications” on page 19

■

“Electrical Specifications” on page 20

■

“Environmental Specifications” on page 20

■

“Hardware and Software Requirements” on page 21

Related Information

■

“Understanding the Installation Process” on page 11

■

“Understanding the Adapter” on page 15

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Configuring Jumbo Frames” on page 55

■

“Configuring a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Configuring VLANs and VXLANs” on page 63

■

“Removing the Driver” on page 69

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Physical Specifications

Description U.S. Metric

Length 3.154 in. 169.30 mm

Confirming Specifications and Requirements 19

Page 20

Electrical Specifications

Description U.S. Metric

Height 6.665 in. 80.11 mm

Weight 0.201 lbs. 0.091 kg

Related Information

■

“Electrical Specifications” on page 20

■

“Environmental Specifications” on page 20

■

“Hardware and Software Requirements” on page 21

Electrical Specifications

Description Value

Max power consumption 9.9W at 12V

Typical active power 9.1W at 2x25GbE mode; 5% traffic

Supply voltage 12V ± 15%

Related Information

■

“Physical Specifications” on page 19

■

“Environmental Specifications” on page 20

■

“Hardware and Software Requirements” on page 21

Environmental Specifications

Specification Operation Storage

Temperature

Humidity 10% to 90% noncondensing relative humidity at

20 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

†

5°C to 35°C (-23°C to 95°F), noncondensing -40°C to 65°C (-40°F to 149°F),

27°C (80.6°F) maximum wet bulb

noncondensing

93% noncondensing relative humidity at 38°C (100.4°F) maximum wet bulb

Page 21

Hardware and Software Requirements

Specification Operation Storage

Altitude 3000 meters (9842.5) at 35°C (95°F) ambient

1,219 meters (4,000 feet) at 35°C (95°F) ambient

Vibration 0.15 G z-axis; 0.10 G in x- and y-axes (5-500 Hz

Shock 3 G, 11 msec half-sine 1 inch roll-off front to back, 20 mm step-

Airflow 400 LFM at 55°C (131°F) local ambient temperature

†

Temperature listed is for the server where the card is installed. The actual internal ambient inside the server local to the card might be higher.

sine)

12,000 meters (39,370 feet)

0.50 G z-axis; 0.25 G x- and y-axes (5500 Hz sine)

Threshold testing of castors at 0.75 m/s

Related Information

■

“Physical Specifications” on page 19

■

“Electrical Specifications” on page 20

■

“Hardware and Software Requirements” on page 21

Hardware and Software Requirements

Hardware and software support changes over time. For the latest information concerning I/O options that are supported by your server, refer to the Product Page.

For Oracle Solaris OS systems, the minimum supported version is Oracle Solaris 11.3 SRU23.

Related Information

■

“Physical Specifications” on page 19

■

“Electrical Specifications” on page 20

■

“Environmental Specifications” on page 20

■

“Updating Software and Firmware” on page 23

Confirming Specifications and Requirements 21

Page 22

22 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 23

Updating Software and Firmware

This topic provides information on updating the adapter software and firmware:

■

“Update the OS (Oracle Solaris x86)” on page 23

■

“Firmware Update Tool Overview” on page 24

Related Information

■

“Understanding the Installation Process” on page 11

■

“Understanding the Adapter” on page 15

■

“Confirming Specifications and Requirements” on page 19

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Configuring Jumbo Frames” on page 55

■

“Configuring a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Configuring VLANs and VXLANs” on page 63

■

“Removing the Driver” on page 69

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Update the OS (Oracle Solaris x86)

For the latest list of supported platforms and operating systems, see “Hardware and Software

Requirements” on page 21.

For Solaris OS systems, the minimum supported version is Oracle Solaris 11.3 SRU23.

Update the entire OS image on the client server.

Ensure that you are using at least Oracle Solaris 11.3 SRU23.

Updating Software and Firmware 23

Page 24

Firmware Update Tool Overview

For more information, refer to My Oracle Support at https://support.oracle.com or to the

Product Page.

Related Information

■

“Firmware Update Tool Overview” on page 24

Firmware Update Tool Overview

Use the firmware update tool to update the firmware. You can get the tool in these ways for the Oracle Solaris x86, Linux, and Windows environment:

■

Oracle System Assistant (OSA), a built-in tool on x86 servers or a USB thumb drive shipped with the server, which contains the firmware update tool. Refer to the text files in OSA for instructions on updating your firmware.

■

Hardware Management Pack (HMP), which includes the firmware update tool. Refer to the text files in HMP for instructions on updating your firmware.

■

The fwupdate Automatic Mode command. For more information on this command, refer to the Hardware Management Pack 2.3.x Documentation (http://docs.oracle.com/cd/

E52095_01/index.html).

Related Information

■

“Update the OS (Oracle Solaris x86)” on page 23

■

“Firmware Update Tool Overview” on page 24

■

“Installing the Driver” on page 25

24 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 25

Installing the Driver

These topics describe how to install the driver on all server types:

Description Links

Verify the driver on an Oracle Solaris x86 platform.

Download and install the driver on a Linux platform.

Download and install the driver on a Windows platform.

Related Information

■

“Understanding the Installation Process” on page 11

■

“Updating Software and Firmware” on page 23

■

“Installing the Adapter” on page 31

■

“Configuring Driver Parameters” on page 51

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

“Verify the bnxt Driver (Oracle Solaris x86)” on page 25

“Download and Install the bnxt_en Driver (Linux)” on page 26

“Verify the bnxt_en Driver (Linux)” on page 27

“Download and Install the bnxt_en Driver (Windows)” on page 28

Verify the bnxt Driver (Oracle Solaris x86)

The bnxt software package comes bundled in the Oracle Solaris software. The bnxt driver does not support SRIOV.

Note - The Oracle Solaris 11.3 SRU23 OS is the first release to support this adapter. You can

install or upgrade to this release, but the version of the driver must be the same on both the client and the server.

Check the version of the Oracle Solaris SRU.

Installing the Driver 25

Page 26

Download and Install the bnxt_en Driver (Linux)

You must have at least Oracle Solaris 11.3 SRU23 installed. For example:

$ pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.22.4.0). ... <output omitted> ... Version: 0.5.22 (Oracle Solaris 11.3.22.4.0)

See “Hardware and Software Requirements” on page 21 for more information.

Check the version of the bnxt driver.

# pkg info bnxt

Manually load the driver.

# modload /kernel/drv/arch/bnxt

Note - If the bnxt driver is not listed, the driver might not be loaded. You can use the modload

command to load the driver.

If the bnxt driver is not installed, install it. Determine the device ID for the device using the scanpci command or equivalent. For more information, refer to the Oracle Solaris scanpci(1) man page.

# add_drv -i '"pciex14e4,16d7" "pciex14e4,16ca"' bnxt

If you removed the driver and would like to reinstall the driver, install Oracle Solaris 11.3 SRU23.

Related Information

■

“Verify the bnxt Driver (Oracle Solaris x86)” on page 25

■

“Download and Install the bnxt_en Driver (Linux)” on page 26

■

“Download and Install the bnxt_en Driver (Windows)” on page 28

Download and Install the bnxt_en Driver (Linux)

If your server uses the Oracle Linux OS, you must download the bnxt_en device driver to install it. The bnxt_en driver is supported on Oracle Linux 6.8, 6.9, and 7.3.

26 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 27

Verify the bnxt_en Driver (Linux)

In a browser, log in to My Oracle Support.

Go to https://myoraclesupport.com.

Search for the Oracle Dual Port 25 Gb Ethernet Adapter.

Open the readme.txt file and follow the download and installation instructions.

Review and accept the software license agreement.

Verify that the bnxt_en driver has been installed.

# lsmod | grep bnxt_en

The output should be similar to this:

bnxt_en 118052 0

Related Information

■

“Verify the bnxt_en Driver (Linux)” on page 27

■

“Download and Install the bnxt_en Driver (Windows)” on page 28

Verify the bnxt_en Driver (Linux)

Check the version of the bnxt_en driver.

# modinfo bnxt_en filename: /lib/modules/4.1.12-94.el7uek.x86_64/updates/bnxt_en.ko version: 1.7.30 description: Broadcom BCM573xx network driver license: GPL srcversion: 7EEACBE0D0FBD47E43A386D alias: pci:v000014E4d000016E5sv*sd*bc*sc*i* alias: pci:v000014E4d000016E1sv*sd*bc*sc*i* <..output omitted..> alias: pci:v000014E4d000016C8sv*sd*bc*sc*i* alias: pci:v000014E4d000016C0sv*sd*bc*sc*i* depends: vxlan vermagic: 4.1.12-94.el7uek.x86_64 SMP mod_unload modversions

If the bnxt_en driver is not listed, the driver might not be loaded.

Installing the Driver 27

Page 28

Download and Install the bnxt_en Driver (Windows)

If the bnxt_en driver is not loaded, load it.

# modprobe bnxt_en

View additional information about the driver.

For example:

# ethtool -i ens2f0 driver: bnxt_en version: 1.7.30 firmware-version: 20.6.133/1.7.8 pkg 20.06.04.01 expansion-rom-version: bus-info: 0000:c0:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: no supports-priv-flags: no

Related Information

■

“Download and Install the bnxt_en Driver (Windows)” on page 28

■

“Download and Install the bnxt_en Driver (Linux)” on page 26

■

“Installing the Adapter” on page 31

Download and Install the bnxt_en Driver (Windows)

If the server uses Windows10, Windows8, Windows7, or Windows Vista XP, perform this procedure to download and install the device driver.

Open the Microsoft Device Manager and locate the ORACLE_SSM flash drive.

The ORACLE_SSM flash drive contains the bnxt_en driver.

Right-click the bnxt_en network driver and choose Update Driver.

Choose the Broadcom NIC driver for your OS.

Reboot the server.

Verify that the driver is installed.

28 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 29

Download and Install the bnxt_en Driver (Windows)

See “Verify the Adapter Installation (Windows)” on page 37.

Related Information

■

“Download and Install the bnxt_en Driver (Linux)” on page 26

■

“Installing the Adapter” on page 31

Installing the Driver 29

Page 30

30 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 31

Installing the Adapter

These topics describe how to install the adapter:

Description Links

Order additional hardware. “Adapter Description” on page 32

“Cable and Transceiver Overview” on page 32

Follow cable cautions. “Cable Cautions” on page 33

Understand the connectors. “Front Panel Connectors and LEDs” on page 17

Install the adapter and transceivers. “Install the Adapter” on page 34

“Install the SFP28 Transceivers” on page 35

Verify the adapter installation. “Verify the Adapter Installation (Oracle Solaris x86)” on page 36

“Verify the Adapter Installation (Linux)” on page 37

“Verify the Adapter Installation (Windows)” on page 37

Related Information

■

“Understanding the Adapter” on page 15

■

“Confirming Specifications and Requirements” on page 19

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Installing the Adapter 31

Page 32

Adapter Description

The Oracle Dual Port 25 Gb Ethernet Adapter comes with the factory default mode of 25Gb, and supports direct attach cables.

25GbE Mode Part Number

Oracle Dual Port 25 Gb Ethernet Adapter

If the bnxt_en driver is not listed, the driver might not be loaded. Use the modload command to load the driver. For instructions, refer to the Oracle Solaris modload(1M) man page.

Connector Optical/Fibre Cable Support Direct-Attach Copper Cable Support

Port 1 (A) 25 GbE 2x25 25GbE

Port 2 (B) 25 GbE

■ 7118015, factory installed

■ 7118016, Xoption

Related Information

■

“Cable Cautions” on page 33

■

“Install the Adapter” on page 34

■

“Verify the Adapter Installation (Oracle Solaris x86)” on page 36

■

“Verify the Adapter Installation (Linux)” on page 37

■

“Verify the Adapter Installation (Windows)” on page 37

Cable and Transceiver Overview

The Oracle Dual 25 Gb Ethernet adapter comes with a factory default port mode of 2x25GbE. This adapter supports two cabling option solutions: copper and optical.

Check the adapter product page for available transceivers and cables at https://www.oracle.

com/networking/index.html.

■

Copper:

■

The copper solution has the SFP28 transceiver attached.

32 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 33

■

A direct-attach SFP28 passive copper cable is available in 1, 2, 3, or 5 meters. The end of the SFP28 direct-attached copper splitter cable connects to the adapter's SFP28 port. The two pigtails connect to the SFP28 ports 1 and 2 on the adapter.

Note - When using SFP28 to SFP28 cables to connect the adapter to a 25 GbE switch,

ensure that the SFP28 ports on the switch are configured in the 25GbE mode.

■

Optical:

■

A SFP28 SR transceiver connects to the adapter's SFP28 port and the SFP28 optical cable, which is available in 5, 10, 20, 50, or 100 meters. One end of the SFP28 optical cable connects to the SFP28 transceiver on the adapter, while the other end connects to a SFP28 transceiver in a switch or other device that is configured for 25 GbE speed.

■

An SFP28 SR transceiver connects to the adapter's top SFP28 port. The SFP28 end of the splitter cable connects to the SFP28 transceiver on the adapter, while the two pigtails connect to two different 25 GbE SFP28 transceivers on the switch or other device.

Cable Cautions

To prevent data cable damage, you must follow these cautions.

Do not uncoil the cable, as a kink might occur. Hold the coil closed as you unroll the cable, pausing to allow the cable to relax as it is unrolled.

Do not pull the cable out of the shipping box, through any opening, or around any corners. Unroll the cable as you lay it down and move it through turns.

Do not twist the cable to open a kink. If it is not severe, open the kink by unlooping the cable.

Do not step on the cable or connectors. Plan cable paths away from foot traffic or rolling loads.

Do not bend the cables to a radius tighter than 85 mm (3.4 inches). Ensure that cable turns are as wide as possible.

Do not pack the cable to fit a tight space. Use an alternative cable route.

Installing the Adapter 33

Page 34

Install the Adapter

Do not straighten the cable to correct a bend that is too tight. Leave the cable bend as is.

Do not drop the cable or connectors from any height. Gently set the cable down, resting the cable connectors on a stable surface.

Do not drag the cable or its connectors over any surface. Carry the entire cable to and from the points of connection.

Related Information

■

“Adapter Description” on page 32

■

“Install the Adapter” on page 34

Do not hang the cable for a length more than 2 meters (7 feet). Minimize the hanging weight with intermediate retention points.

Do not cinch the cable with hard fasteners or cable ties. Use soft hook-and-loop fastener for bundling and securing cables.

Do not force the cable connector into the receptacle by pushing on the cable. Apply connection or disconnection forces at the connector only.

Install the Adapter

These instructions describe the basic tasks required to install the adapter. Refer to the server's installation or service manual for specific PCIe installation instructions.

Halt and power off the server.

Power off all of the peripherals connected to the server.

Open the server chassis.

Attach an antistatic wrist strap to the server chassis.

Remove the slot cover from the chassis.

34 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 35

Install the SFP28 Transceivers

Holding the adapter by the edges, align the card edge connector with the PCIe slot.

Slide the adapter face plate into the small slot at the end of the PCIe opening.

Applying even pressure at both corners of the adapter, push the adapter until it is firmly seated in the slot.

Caution - Do not use excessive force when installing the adapter into the PCIe slot. You might

damage the adapter's PCIe connector. If the adapter does not seat properly when you apply even pressure, remove the adapter, and carefully reinstall it.

Detach the wrist strap and close the server.

10.

Verify the installation.

Follow the instructions in the appropriate section:

■

“Verify the Adapter Installation (Oracle Solaris x86)” on page 36

■

“Verify the Adapter Installation (Linux)” on page 37

■

“Verify the Adapter Installation (Windows)” on page 37

Related Information

■

“Adapter Description” on page 32

■

“Cable Cautions” on page 33

■

“Install the SFP28 Transceivers” on page 35

Install the SFP28 Transceivers

When using the adapter in the 2x25 GbE mode, there are two cabling options. See “Cable and

Transceiver Overview” on page 32. The optical transceivers are available from Oracle. For

the latest information on available transceivers and cables, refer to Oracle's Product Page.

Installing the Adapter 35

Page 36

Verify the Adapter Installation (Oracle Solaris x86)

Note - There are two QSFP connector ports, port 1 (A) and port 1 (B).

Holding the optical transceiver by the edges, align the transceiver with the slot in the adapter and slide the transceiver into the opening.

Applying even pressure at both corners of the transceiver, push the transceiver until it is firmly seated in the slot.

Repeat Step 1 and Step 2 to install the second optical transceiver.

Power on the server.

Verify the adapter installation.

Follow the instructions in the appropriate section:

■

“Verify the Adapter Installation (Oracle Solaris x86)” on page 36

■

“Verify the Adapter Installation (Linux)” on page 37

■

“Verify the Adapter Installation (Windows)” on page 37

Related Information

■

“Cable and Transceiver Overview” on page 32

■

“Adapter Description” on page 32

■

“Install the Adapter” on page 34

Verify the Adapter Installation (Oracle Solaris x86)

Power on the server, and then boot the server.

Check the version of the Oracle Solaris SRU.

You must have at least Oracle Solaris 11.3 SRU23 installed. For example:

$ pkg info entire Name: entire Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.23.1.0). ... <output omitted> ... Version: 0.5.23 (Oracle Solaris 11.3.23.1.0)

36 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 37

Verify the Adapter Installation (Linux)

See “Hardware and Software Requirements” on page 21 for more information.

Verify that the adapter is properly installed and recognized by the OS.

# grep bnxt /etc/path_to_inst

If the adapter is properly installed, you should see output similar to this:

grep bnxt /etc/path_to_inst "/pci@2,0/pci8086,2f08@3/pci108e,3044@0" 0 "bnxt" "/pci@2,0/pci8086,2f08@3/pci108e,3044@0,1" 1 "bnxt"

Related Information

■

“Adapter Description” on page 32

■

“Install the Adapter” on page 34

■

“Verify the Adapter Installation (Linux)” on page 37

■

“Verify the Adapter Installation (Windows)” on page 37

Verify the Adapter Installation (Linux)

Verify the new network interface instances for the adapter.

[root@nsn169-178 Linux]# ifconfig -a | grep 00:0a ether 00:0a:f7:97:b1:4e txqueuelen 1000 (Ethernet) ether 00:0a:f7:97:b1:4f txqueuelen 1000 (Ethernet)

Related Information

■

“Adapter Description” on page 32

■

“Install the Adapter” on page 34

■

“Verify the Adapter Installation (Oracle Solaris x86)” on page 36

■

“Verify the Adapter Installation (Windows)” on page 37

Verify the Adapter Installation (Windows)

Click Control Panel.

Click Network Connection.

Installing the Adapter 37

Page 38

Verify the Adapter Installation (Windows)

If the driver is installed correctly, the adapter name Oracle Dual Port 25 Gb Ethernet Adapter is displayed in the Network Adapters folder.

In the Administration tool, click Computer Management → Device Manager → Network Adapter.

Check the driver version.

For example, the driver version for Microsoft Windows 12/16 is v20.6.123.0.

Related Information

■

“Install the Adapter” on page 34

■

“Configuring the Network” on page 39

38 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 39

Configuring the Network

These topics describe how to configure the network for the adapter:

Description Links

Configure the network interface. “Configure the Network Interface (Oracle Solaris

Boot over the network. “Boot Options” on page 41

Install and Verify RDMA on Converged Ethernet.

x86)” on page 40

“Configure the Network Interface (Windows)” on page 40

“Boot Over the Network (PXE)” on page 41

“Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42

“Install RoCE (Linux)” on page 45

“Install RoCE in Windows Kernel Mode” on page 46

“Install RoCE in Windows User Mode” on page 47

“Install RoCE in VMware ESXi” on page 47

“Verify RoCE in Windows” on page 48

Related Information

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring Driver Parameters” on page 51

■

“Configuring Jumbo Frames” on page 55

■

“Configuring a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Configuring VLANs and VXLANs” on page 63

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Configuring the Network 39

Page 40

Configure the Network Interface (Oracle Solaris x86)

Display the bnxt instances.

# dladm show-phys

For more information, refer to the Oracle Solaris dladm(1M) man page.

The output should be similar to this:

LINK MEDIA STATE SPEED DUPLEX DEVICE net9 Ethernet up 25000 full bnxt1 net8 Ethernet up 25000 full bnxt0

Set up the bnxt interfaces.

For more information, refer to the Oracle Solaris ipadm(1M) man page. Your ipadm command might look similar to this:

#ipadm create-ip net4 # ipadm create-addr -T static -a local=10.2.3.4/24 net4/v4

This command creates another address 10.2.3.5/24 on interface net1, but marks the address down until explicitly marked up:

#ipadm create-addr -T static -d -a 10.2.3.5/24 net4/v4

This command marks the address object net4/v4a up that was previously marked down.

#ipadm up-addr net4/v4a

Related Information

■

“Configure the Network Interface (Windows)” on page 40

■

“Boot Options” on page 41

■

“Installing RoCE” on page 44

Configure the Network Interface (Windows)

Use the Network and Sharing Center to configure the network settings for the bnxt_en adapter. These steps are for Microsoft Windows7.

Ensure that the adapter is installed.

40 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 41

Click Start > Control Panel > Network and Sharing Center > Change Adapter Settings.

Select Oracle Dual Port 25Gb Ethernet Adapter and click Change Settings of This Connection.

Related Information

■

“Configure the Network Interface (Oracle Solaris x86)” on page 40

■

“Boot Options” on page 41

■

“Installing RoCE” on page 44

Boot Options

The adapter supports these boot options:

■

UEFI with PXE with option ROM (Oracle x86)

■

UEFI with iSCSI with option ROM (Oracle x86)

Boot Options

Related Information

■

“Boot Over the Network (PXE)” on page 41

■

“Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42

■

Boot Over the Network (PXE)

PXE network boot is an environment for booting computers using a network interface independently of available data storage devices (such as hard disks) or installed OS. No boot media is required on the client system. With PXE, you can install an OS on an x86-based client over the network by using DHCP.

Boot over the network using PXE.

Refer to the booting with PXE instructions in the “Booting a System From the Network” in

Booting and Shutting Down Oracle Solaris 11.3 Systems.

Configuring the Network 41

Page 42

Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)

Related Information

■

“Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)” on page 42

■

“Installing RoCE” on page 44

Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)

Obtain the MAC address of the first adapter port by checking the label of the adapter.

On the adapter, the MAC address on the label is for the first port. The second port's MAC address is the MAC address from the label, plus 1.

Note -Auto negotiation is not supported when booting with Linux over the network.

Set up the PXE boot server with the MAC addresses.

Plug the Ethernet cable into the adapter's port.

Power on the server.

Press the F2 key or the Control-E keys to go to the BIOS menu.

Go to the Boot - Boot Device Priority screen and ensure that the boot order of the network devices is higher than the hard drive.

Press F10 to save the boot configuration changes and exit BIOS.

The server should reboot after saving the boot configuration.

On Oracle platforms, press F12 to install the OS from the network.

If the cable is connected to the correct port, you should see the MAC address that you assigned to your PXE server displayed by BIOS. If your platform does not support the F12 key, you might need to boot from the BIOS.

42 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 43

Boot Over a 10GbE Network (Oracle Solaris x86 and Linux)

Choose a boot option.

10.

Press F10 to save and exit.

The host will perform a PXE boot from the Oracle Dual Port 25 Gb Ethernet adapter port, DHCP, and the UEFI PXE boot server response. This screen appears.

11.

Choose the UEFI PXE boot server using TFTP protocol.

12.

Select Oracle Linux OS from the UEFI PXE boot server for the OS installation.

Configuring the Network 43

Page 44

Installing RoCE

13.

14.

Ensure that the OS installation begins on the Oracle Dual Port 25 Gb Ethernet adapter.

Ensure that the OS installer completes and begins the post-installation steps.

15.

After the OS installation completes, use the BIOS to change the boot device priority to boot from hard disk to boot up the newly installed OS.

Unless you change the boot device priority, the OS installation process repeats.

Related Information

■

“Boot Over the Network (PXE)” on page 41

■

“Installing RoCE” on page 44

Installing RoCE

The RDMA software allows computers in a network to exchange data in main memory without involving the processor, cache, or OS of either computer. RDMA can improve throughput and performance because it frees up resources, and it can also facilitate a faster data transfer rate. RDMA over Converged Ethernet (RoCE) is the network protocol that allows RDMA over an Ethernet network.

44 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 45

Install RoCE (Linux)

RoCE functionality is available in user mode and kernel mode application for Windows. RoCE

PF and SRIOV VF are available in single function mode and in multifunction mode (NIC

Partitioning mode). You can install and configure RoCE in Windows, Linux, and VMWare.

Refer to these sites for more information on supporting RoCE:

■

Windows – https://technet.microsoft.com/en-us/library/jj134210(v=ws.11).aspx

■

OEL Linux – http://www.mellanox.com/page/products_dyn?product_family=79

■

VMware – https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.

networking.doc/GUID-4A5EBD44-FB1E-4A83-BB47-BBC65181E1C2.html

Install RoCE (Linux)

Before You Begin

Locate these files before you install and configure RoCE in Oracle Linux:

■

The bnxt_en-roce file – RoCE-supported bnxt_en driver, which is part of released gzip compressed tar archive.

■

The bnxt_re file – RoCE driver.

■

The libbnxtre file – User mode RoCE library module.

The bnxt_re driver requires a special RoCE-enabled version of bnxt_en, which is included in the netxtremebnxt_en-1.7.9.tar.gz (or newer) package. The bnxt_re driver compilation depends whether IB stack is available along with the OS distribution or if an external OFED is required.

Upgrade the NIC NVRAM.

Use the RoCE supported firmware packages starting with Software Release 20.06.04.01.2.

Uncompress, build, and install the BCM5741X Linux L2 and RoCE drivers.

In a terminal window, uncompress the drivers.

# tar -xzf netxtreme-bnxt_en-1.7.9.tar.gz

Change to the directory where driver files are stored.

# cd netxtreme-bnxt_en-bnxt_re

Build and install the drivers.

# make build && make install

Configuring the Network 45

Page 46

Install RoCE in Windows Kernel Mode

Uncompress, build, and install the NetXtreme-E Linux RoCE User Library.

Uncompress the RoCE User Library file.

# tar xzf libbnxtre-0.0.18.tar.gz

Change to the directory where RoCE files are stored.

# cd libbnxtre-0.0.18

Build and install the RoCE User Library.

# configure && make && make install.

Copy the driver file to the working directory.

# cp bnxtre.driver /etc/libibverbs.d/

Display the output.

# echo "/usr/local/lib" >> /etc/ld.so.conf

Create the necessary links and cache to the shared libraries you specified.

# ldconfig -v

Refer to the bnxt_re README.txt file for more details on configurable options and recommendations.

Install RoCE in Windows Kernel Mode

RoCE functionality is available in user mode and kernel mode application. Starting with the Windows Server 2012 release, the RDMA capability in the NIC for SMB file traffic is enabled if both ends are enabled for RDMA. Broadcom NDIS miniport bnxtnd.sys starting with version 20.6.2 supports RoCEv1 and RoCEv2 with the NDKPI interface. The default setting is RoCEv1.

Upgrade the NIC NVRAM using the appropriate board packages. In CCM or in UEFI HII, enable support for RDMA.

Enable each miniport.

46 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 47

Install RoCE in Windows User Mode

You can go to the adapter's Advanced Properties page and set Network Direct Functionality to Enabled for each BCM5741X miniport.

You can also run this command in the PowerShell window:

Set-NetAdapterAdvancedProperty -RegistryKeyword *NetworkDirect -RegistryValue 1

Verify that RDMA is enabled.

These PowerShell commands return true if Network Direct is enabled.

a. Get-NetOffLoadGlobalSetting b. Get-NetAdapterRDMA

Install RoCE in Windows User Mode

RoCE functionality is available in user mode and kernel mode application. Perform this procedure before you run a user mode application that writes to NDSPI.

Copy the bxndspi.dll user mode driver file to the C:\Windows\System32 directory.

Before You Begin

Install the driver.

rundll32.exe .\bxndspi.dll,Config install|more

Install RoCE in VMware ESXi

You must have at least ESXi-6.5.0 GA build 4564106 installed before you perform this procedure.

You must also have at least ESX6.5 L2 driver version 20.6.9.0 (RoCE supported L2 driver) installed.

Ensure that the BNXTNET L2 driver is installed with the disable_roce=0 module parameter.

If the driver is not already installed with the parameter, set the module parameter:

# esxcfg-module -s "disable_roce=0" bnxtnet

Copy the bnxtroce-driver version.vib file in the /var/log/vmware directory.

Configuring the Network 47

Page 48

Verifying RoCE

$ esxcli software vib install --no-sig-check -v bnxtroce-driver version.vib

Reboot the machine.

Verify that the drivers are installed correctly.

# esxcli software vib list | grep bnxtroce

Tip - By default, ECN is disabled for RoCE traffic. To disable RoCE traffic, use the tos_ecn=0

module parameter for bnxtroce.

Verifying RoCE

Verify RoCE in Windows

Create a file share on the remote system and open that share.

Open that share with Windows Explorer or the net use command. To avoid a speed bottleneck when the hard disk is reading or writing, use a RAM disk as the network share.

Run these commands from the PowerShell.

Get-SmbMultichannelConnection | fl *RDMA* ClientRdmaCapable : True ServerRdmaCapable : True

If both Client and Server are True, any file transfers over this SMB connection use SMB.

Enable or disable SMB Multichannel support.

On the Server side:

Enable: Set-SmbServerConfiguration -EnableMultiChannel $true Disable: Set-SmbServerConfiguration -EnableMultiChannel $false

On the Client side:

Enable: Set-SmbClientConfiguration -EnableMultiChannel $true Disable: Set-SmbClientConfiguration -EnableMultiChannel $false

48 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 49

Verify RoCE in Windows

Note - By default, the driver sets up two RDMA connections for each network share per IP

address (on a unique subnet). You can increase the number of RDMA connections by adding multiple IP addresses, each with different a subnet, for the same physical port under test. Multiple network shares can be created and mapped to each link partner using the unique IP addresses that you created.

Configuring the Network 49

Page 50

50 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 51

Configuring Driver Parameters

The bnxt and bnxt_en device drivers control the adapter's interfaces. You can manually set the device driver parameters to customize each device in the server.

These topics describe how to configure driver parameters:

Description Links

Configure driver parameters in the Oracle Solaris x86 OS.

Configure driver parameters in Linux. “Set Driver Parameters (Linux)” on page 53

Related Information

■

“Understanding the Installation Process” on page 11

■

“Understanding the Adapter” on page 15

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

“Set Driver Parameters (Oracle Solaris x86)” on page 51

“Driver Parameters (Oracle Solaris x86)” on page 53

“Driver Parameters (Linux)” on page 54

Set Driver Parameters (Oracle Solaris x86)

Locate the path names and the associated instance numbers in the /etc/ path_to_inst file.

For example, on an Oracle Solaris x86 server, you should see output similar to this:

Configuring Driver Parameters 51

Page 52

Set Driver Parameters (Oracle Solaris x86)

# grep bnxt /etc/path_to_inst "/pci@0,0/pci8086,3c0a@3,2/pci108e,7b15@0" 0 "bnxt" "/pci@0,0/pci8086,3c0a@3,2/pci108e,7b15@0,1" 1 "bnxt"

In these preceding examples:

■

The first part within the double quotes specifies the hardware node name in the device tree.

■

The number not enclosed in quotes is the instance number (shown in bold for emphasis).

■

The last part in double quotes is the driver name.

Note - To unambiguously identify a PCIe device in the bnxt.conf file, use the name, parent

name, and unit address for the device. In the example, the name is pci108e,7b15, the parent is /pci@0,0/pci8086,3c0a@3,2, and the unit address is 0. Refer to the Oracle Solaris pci(4) man

page for more information about the PCIe device specification.

Set the parameters for the bnxt.conf devices in one of these ways:

Copy the bnxt.conf file to the /kernel/drv/ and edit the copied file.

Save the bnxt.conf file and reboot the system. See “Driver Parameters (Oracle Solaris

x86)” on page 53.

"/pci@2,0/pci8086,2f08@3/pci108e,3044@0" 0 "bnxt" "/pci@2,0/pci8086,2f08@3/pci108e,3044@0,1" 1 "bnxt"

Set the properties.

# dladm show-linkprop -p flowctrl net5 LINK PROPERTY PERM VALUE EFFECTIVE DEFAULT POSSIBLE net5 flowctrl rw no no no no,tx,rx,bi, pfc,auto flow_control = 3;

For bidirectional flow control:

# dladm set-linkprop -p flowctrl=bi net5

For more information, refer to the Oracle Solaris dladm(5) man page.

Related Information

■

“Driver Parameters (Oracle Solaris x86)” on page 53

■

“Set Driver Parameters (Linux)” on page 53

52 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 53

Driver Parameters (Oracle Solaris x86)

You can configure these parameters on each bnxt interface.

Type Keyword Description

Jumbo frames

Flow control

Transmit queue size

default_mtu=mtu

flow_control

tx_ring_size

Size of the default MTU (payload without the Ethernet header). Allowed values:

1500 to 9500 (default = 1500)

Ethernet flow control. Allowed values:

0 - Disable (default in Oracle Solaris 11)

1 - Receive only

2 - Transmit only

3 - Receive and transmit

Number of the transmit descriptors per transmit queue. The actual value is rounded up to the next multiple of 8. Allowed values:

Driver Parameters (Oracle Solaris x86)

Receive queue size

rx_ring_size

Related Information

■

“Set Driver Parameters (Oracle Solaris x86)” on page 51

■

“Set Driver Parameters (Linux)” on page 53

Set Driver Parameters (Linux)

Use the ethtool utility or the configtool utility to set parameters on a Linux platform.

See “Driver Parameters (Linux)” on page 54.

Related Information

■

“Driver Parameters (Linux)” on page 54

64 to 4096 (default = 1024)

Number of the transmit descriptors per receive queue. The actual value is rounded up to the next multiple of 8. Allowed values:

64 to 4096 (default = 1024)

Configuring Driver Parameters 53

Page 54

Driver Parameters (Linux)

■

“Set Driver Parameters (Oracle Solaris x86)” on page 51

Driver Parameters (Linux)

This table lists the tunable bnxt_en driver parameters for Linux OS and describes their functions.

Keyword Valid Range Default Value Description

FlowControl 0 to 3

(0=none, 1=RX only, 2=TX only, 3=RX and TX)

RxDescriptiors 64 to 4096 512

RxIntDelay 0 to 65535

(0=off)

TxDescriptors 80 to 4096 256

XsumRX 0 to 1 1

Read from the

EEPROM.

If EEPROM is not detected, default is 3.

Controls the automatic generation (TX) and response (RX) to Ethernet PAUSE frames.

Number of receive descriptors allocated by the driver. Increasing this value allows the driver to buffer more incoming packets. Each descriptor is 16 bytes. A receive buffer is also allocated for each descriptor and can be either 2048, 4056, 8192, or 16384 bytes, depending on the MTU setting. When the MTU size is 1500 or less, the receive buffer size is 2048 bytes. When the MTU is greater than 1500, the receive buffer size is either 4056, 8192, or 16384 bytes. The maximum MTU size is 16114.

Delays the generation of receive interrupts in units of

0.8192 microseconds. Receive interrupt reduction can improve CPU efficiency if properly tuned for specific network traffic. Increasing this value adds extra latency to frame reception and can end up decreasing the throughput of TCP traffic. If the system is reporting dropped receives, this value might be set too high, causing the driver to run out of available receive descriptors.

Number of transmit descriptors allocated by the driver. Increasing this value allows the driver to queue more transmits. Each descriptor is 16 bytes.

A value of 1 indicates that the driver should enable IP checksum offload for received packets (both UDP and TCP) to the Ethernet adapter hardware.

Related Information

■

“Set Driver Parameters (Linux)” on page 53

■

“Set Driver Parameters (Oracle Solaris x86)” on page 51

■

“Configuring Jumbo Frames” on page 55

54 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 55

Configuring Jumbo Frames

Jumbo frames can support up to 9500 MTU. The default value is 1500 MTU.

These topics describe how to configure jumbo frames:

■

“Change the MTU Permanently (Oracle Solaris x86)” on page 55

■

“Change the MTU Temporarily (Oracle Solaris x86)” on page 56

■

“Configure Jumbo Frames (Linux)” on page 56

■

“Configure Jumbo Frames (Windows)” on page 57

Related Information

■

“Understanding the Installation Process” on page 11

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Configuring a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Configuring VLANs and VXLANs” on page 63

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Change the MTU Permanently (Oracle Solaris x86)

Perform one of these actions to permanently change the MTU.

■

Add this line in the /etc/driver/drv/bnxt.conf file and reboot the server.

default_mtu = desired-frame-size;

where the desired-frame-size value can range from 1500 to 9500.

Configuring Jumbo Frames 55

Page 56

Change the MTU Temporarily (Oracle Solaris x86)

Note - Adding this line will make changes to all instances of bnxt. To make changes for

specific instances, see Step 2 in “Set Driver Parameters (Oracle Solaris x86)” on page 51.

■

Type:

# dladm set-linkprop -p mtu=9500 net0

Related Information

■

“Change the MTU Temporarily (Oracle Solaris x86)” on page 56

■

“Configure Jumbo Frames (Linux)” on page 56

Change the MTU Temporarily (Oracle Solaris x86)

Change the MTU temporarily with the dladm(1M) command.

For example, where the device name is xnet0, this command increases MTUs to the maximum:

# dladm set-linkprop [-t] -p mtu=9500 net0

The temporary setting lasts only until the next reboot of the server.

Related Information

■

“Change the MTU Permanently (Oracle Solaris x86)” on page 55

■

“Configure Jumbo Frames (Linux)” on page 56

■

“Configure Jumbo Frames (Windows)” on page 57

Configure Jumbo Frames (Linux)

Jumbo frames can support up to 9500 MTU. The default value is 1500 MTU.

Set the value for jumbo frames with the ifconfig(1M) command.

For example, where the IP address for eth7 is 192.1.1.200, this command increases MTUs to the maximum:

# ifconfig eth7 192.1.1.200 mtu 9500 up

For more information, refer to the Oracle Solaris ifconfig(1M) man page.

56 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 57

Related Information

■

“Configure Jumbo Frames (Windows)” on page 57

■

“Configuring VLANs and VXLANs” on page 63

Configure Jumbo Frames (Windows)

Click Control Panel.

Click Device Manager and open the Network Adapters folder.

Right-click the Oracle Dual 25G Ethernet Adapter and choose Properties.

Click Configure.

Highlight the jumbo frames item and select the desired setting.

Click Apply for each port.

Configure Jumbo Frames (Windows)

Related Information

■

“Configure Jumbo Frames (Linux)” on page 56

■

“Configuring VLANs and VXLANs” on page 63

Configuring Jumbo Frames 57

Page 58

58 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 59

Configuring a Link Aggregation (Oracle Solaris x86)

These topics describe how to configure link aggregation using the Oracle Solaris x86 OS:

■

“Configure a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Display Information About a Link Aggregation (Oracle Solaris x86)” on page 60

■

“Delete a Link Aggregation (Oracle Solaris x86)” on page 61

Related Information

■

“Understanding the Installation Process” on page 11

■

“Understanding the Adapter” on page 15

■

“Confirming Specifications and Requirements” on page 19

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Configuring Jumbo Frames” on page 55

■

“Configuring VLANs and VXLANs” on page 63

■

“Removing the Driver” on page 69

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Configure a Link Aggregation (Oracle Solaris x86)

The example in this procedure aggregates sample interfaces bnxt0, bnxt1, bnxt2, and bnxt3. Arbitrary key numbers (1 and 2) are used for each aggregation.

Note - These commands change the contents of the /etc/aggregation.conf file.

Configuring a Link Aggregation (Oracle Solaris x86) 59

Page 60

Display Information About a Link Aggregation (Oracle Solaris x86)

Configure the link aggregation containing the bnxt interfaces in the default mode.

For example:

# dladm create-aggr -l net5 -l net6 aggr1 # dladm show-aggr # ipadm create-ip aggr1 # ipadm create-addr -a 192.1.1.14/24 aggr1

For more information on link aggregation, refer to “Creating a Link Aggregation” in Managing

Network Datalinks in Oracle Solaris 11.3.

Related Information

■

“Display Information About a Link Aggregation (Oracle Solaris x86)” on page 60

■

“Delete a Link Aggregation (Oracle Solaris x86)” on page 61

Display Information About a Link Aggregation (Oracle Solaris x86)

The ipadm and dladm commands provide different details about link aggregations, as shown in these examples.

For more information, refer to “Creating a Link Aggregation” in Managing Network Datalinks

in Oracle Solaris 11.3.

For more information on the commands, refer to the Oracle Solaris ifconfig(1M) man page and the dladm(5) man page.

Use the appropriate command to obtain the desired results.

■

Use the ifconfig command to examine the details about a link aggregation.

# ifconfig aggr1 aggr1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 32 inet 192.2.2.84 netmask ffffff00 broadcast 192.2.2.255 ether 0:15:17:75:ff:81

# ifconfig aggr2aggr2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 33 inet 193.2.2.84 netmask ffffff00 broadcast 193.2.2.255 ether 0:15:17:75:ff:83

■

Use the dladm show-aggr command to show link aggregation status.

60 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 61

Delete a Link Aggregation (Oracle Solaris x86)

Related Information

■

“Configure a Link Aggregation (Oracle Solaris x86)” on page 59

■

“Delete a Link Aggregation (Oracle Solaris x86)” on page 61

Delete a Link Aggregation (Oracle Solaris x86)

Delete the IP interface that is configured over the link aggregation.

For example: type:

# ipadm delete-ip ip-aggr1

where ip-aggr1 is the IP interface over the link aggregation.

Delete each unwanted link aggregation.

For example:

# dladm delete-aggr aggr2 # dladm delete-aggr aggr2

Related Information

■

“Display Information About a Link Aggregation (Oracle Solaris x86)” on page 60

■

“Configuring VLANs and VXLANs” on page 63

Configuring a Link Aggregation (Oracle Solaris x86) 61

Page 62

62 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 63

Configuring VLANs and VXLANs

These topics explain how to configure VLANs and VXLANs:

■

“VLANs Overview” on page 63

■

“Configure VLANs (Oracle Solaris x86)” on page 64

■

“Configure VLANs (Linux)” on page 65

■

“Configure VLANs (Windows)” on page 65

■

“Configure VXLANs (Oracle Solaris x86)” on page 66

Related Information

■

“Understanding the Installation Process” on page 11

■

“Configuring the Network” on page 39

■

“Configuring Driver Parameters” on page 51

■

“Removing the Driver” on page 69

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

VLANs Overview

Virtual LANs enable you to divide the network into subnetworks without having to add to the physical network environment. The subnetworks are virtual and use the same physical network resources. VLANs facilitate network administrations because the smaller groups are easier to maintain.

You can create VLANs according to various criteria, but each VLAN must be assigned a VLAN tag or VLAN ID (VID). The VID is a 12-bit identifier between 1 and 4094 that identifies a unique VLAN.

Note - If you configure a VLAN virtual device for an Ethernet adapter, all traffic sent or

received by that Ethernet adapter must be in VLAN-tagged format.

Configuring VLANs and VXLANs 63

Page 64

Configure VLANs (Oracle Solaris x86)

Related Information

■

“Configure VLANs (Oracle Solaris x86)” on page 64

■

“Configure VLANs (Linux)” on page 65

■

“Configure VLANs (Windows)” on page 65

■

“Configure VXLANs (Oracle Solaris x86)” on page 66

Configure VLANs (Oracle Solaris x86)

Determine the types of links that are used in the server.

# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net15 Ethernet up 10000 full bnxt1 net14 Ethernet up 10000 full bnxt0

Create a VLAN link over a datalink.

# dladm create-vlan -l link -v vid vlan-link

where link specifies the link where the VLAN interface is being created, vid indicates the VLAN ID number, and vlan-link specifies the name of the VLAN, which can also be an administratively-chosen name.

Verify the VLAN configuration.

# dladm show-vlan

Create an IP interface over the VLAN.

# ipadm create-ip interface

where interface uses the VLAN name.

Related Information

■

“Configure VLANs (Linux)” on page 65

■

“Configure VLANs (Windows)” on page 65

■

“Configure VXLANs (Oracle Solaris x86)” on page 66

64 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 65

Configure VLANs (Linux)

Ensure that the bnxt_en module is loaded.

# modprobe bnxt_en

Plumb the adapter's interface.

# ifconfig eth6 ipv6addressup

where ipv6address is the IP address of the interface.

Add the VID.

For example:

# vconfig add eth6 5

where eth6 is the interface, and 5 is the VID.

Note - In Linux systems, you can use any single digit as the VID.

Configure VLANs (Linux)

Configure the bnxt_en VLAN.

For example:

# ifconfig eth6.5 ipv6addressup

where ipv6address is the IP address of the interface.

Related Information

■

“Configure VLANs (Oracle Solaris x86)” on page 64

■

“Configure VLANs (Windows)” on page 65

■

“Configure VXLANs (Oracle Solaris x86)” on page 66

Configure VLANs (Windows)

Click Control Panel.

Click Network Connection.

Click the folder icon from the sub-manual bar.

Configuring VLANs and VXLANs 65

Page 66

Configure VXLANs (Oracle Solaris x86)

Right-click the Oracle Dual Port 25 Gb Ethernet Adapter port, then select Properties.

Click Configure.

Click VLAN, then click New.

Type VLAN and the ID (for example, type VLAN10).

Click OK.

Open the Local Connections for VLAN window from the Network Connections window (Control Panel → Network Internet → Network Connections).

10.

Right-click the Properties button, and select the TCP/IPv4 port in the list.

11.

Click the Properties button, and fill in the desired IP address.

12.

Click Subnet Mask.

The value 255.255.255.0 is displayed.

13.

Click OK.

14.

Repeat Step 3 through Step 10 until all the network ports are VLAN configured.

Note - Ensure that the firewall is configured to allow VLAN traffic. Otherwise, the VLAN

might not operate properly.

Related Information

■

“Configure VLANs (Oracle Solaris x86)” on page 64

■

“Configure VLANs (Linux)” on page 65

■

“Configure VXLANs (Oracle Solaris x86)” on page 66

Configure VXLANs (Oracle Solaris x86)

VXLAN is a Layer 2 technology that enables you to create a Layer 2 network on top of a Layer 3 network, thereby providing further network isolation. VXLAN provides a virtual Layer 2 network that stretches over multiple physical Layer 2 networks. Provisioning resources in a cloud environment is not restricted to a single physical Layer 2 network. Physical servers can be a part of an VXLAN network, as long as they are connected by IPv4 or IPv6 networks.

66 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 67

Configure VXLANs (Oracle Solaris x86)

Determine the types of links that are used in the system.

# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net15 Ethernet up 10000 full bnxt1 net14 Ethernet up 10000 full bnxt0

Create an IP interface over the VXLAN.

# dladm create-vxlan -p addr=10.10.10.1,vni=100 vxlan1 # dladm create-vxlan -p addr=10.10.10.1,vni=101 vxlan2

Verify the VXLAN configuration.

# dladm show-vxlan LINK ADDR VNI MGROUP vxlan1 10.10.10.1 100 224.0.0.1 vxlan2 10.10.10.1 101 224.0.0.1

Related Information

■

“Configure VLANs (Oracle Solaris x86)” on page 64

■

“Configure VLANs (Linux)” on page 65

■

“Configure VLANs (Windows)” on page 65

■

“Removing the Driver” on page 69

Configuring VLANs and VXLANs 67

Page 68

68 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 69

Removing the Driver

It is not necessary to remove a driver when its associated device is removed from a server. However, if you want to clean up your file systems or conserve space, you can easily remove a driver.

These topics explain how to remove the bnxt and bnxt_en device drivers:

Description Links

Remove the driver on an Oracle Solaris x86 server.

Remove the driver on a Linux server.

Remove the driver on a Windows server.

Related Information

■

“Installing the Driver” on page 25

■

“Configuring Driver Parameters” on page 51

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

“Remove the bnxt Driver (Oracle Solaris x86)” on page 69

“Remove the bnxt_en Driver (Linux)” on page 70

“Remove the bnxt_en Driver (Windows)” on page 70

Remove the bnxt Driver (Oracle Solaris x86)

Remove the bnxt driver.

For example:

# pkg uninstall bnxt Packages to remove: 1 Create boot environment: Yes Create backup boot environment: No

PHASE ITEMS Removing old actions 18/18

Removing the Driver 69

Page 70

Remove the bnxt_en Driver (Linux)

Updating package state database Done Updating package cache 1/1 Updating image state Done Creating fast lookup database Done Updating package cache 1/1

Refer to the Oracle Solaris pkg(1) man page for more information.

Related Information

■

“Remove the bnxt_en Driver (Linux)” on page 70

■

“Remove the bnxt_en Driver (Windows)” on page 70

Remove the bnxt_en Driver (Linux)

Type:

#rmmod bnxt_en

Related Information

■

“Remove the bnxt_en Driver (Windows)” on page 70

■

“Remove the bnxt Driver (Oracle Solaris x86)” on page 69

■

“Remove the bnxt_en Driver (Linux)” on page 70

Remove the bnxt_en Driver (Windows)

Click Control Panel.

Click Device Manager and open the Network Adapters folder.

Right-click the Oracle Dual 25G Ethernet Adapter and choose Uninstall.

Click OK.

Related Information

■

“Remove the bnxt Driver (Oracle Solaris x86)” on page 69

■

“Remove the bnxt_en Driver (Linux)” on page 70

■

“Upgrading the Adapter (Linux)” on page 71

70 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 71

Upgrading the Adapter (Linux)

These topics explain how to upgrade the firmware on the Oracle Dual Port 25 Gb Ethernet Adapter and verify the upgrade.

The upgrade command compares the package version against the currently installed package version to determine if an upgrade should be performed. The command does not check the versions of the individual components in the NVM against those within the specified package file. Therefore, it is important that package files from the same source with the same versioning scheme or format are used for a specific controller.

These topics describe how to upgrade the adapter:

■

“Install the Upgrade Package” on page 71

■

“Verify the Upgrade” on page 72

Install the Upgrade Package

Use the bnxtnvm utility to install the NVM upgrade package file.

Determine the version and description of the package file that is currently installed.

For example:

$ bnxtnvm pkgver filename.pkg -v Package File: filename.pkg Package version: 1.0.0

Change to the NVRAM_Images directory.

Determine how you want to install the package.

Upgrading the Adapter (Linux) 71

Page 72

Verify the Upgrade

Install a package file.

$ bnxtnvm install filename.pkg

Replace the filename with the package name.

Install multiple packages for multiple controller types.

$ bnxtnvm install *.pkg

This command installs multiple packages for multiple controller types by using a wildcard in the filename argument. The correct packages are automatically installed into the supported controller devices based on PCI Device-IDs.

Install a package that is newer than the one you have.

$ bnxtnvm upgrade *.pkg

This command checks the discovered and supported devices.

For more information on the packages, refer to the readme.txt file in the bnxtnvm directory of the firmware package.

Verify the upgrade.

See “Verify the Upgrade” on page 72.

Verify the Upgrade

For more information on the packages, refer to the readme.txt file in the bnxtnvm directory of the firmware package.

Determine which version of the package is installed.

For example:

$ bnxtnvm pkgver Device: ens2f1d1 Package version: 20.06.04.01

To display more details about the device and the currently installed package, use the -v option to enable maximum verbosity.

72 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 73

Related Information

■

“Installing the Adapter” on page 31

■

“Troubleshooting the Adapter (Oracle Solaris x86)” on page 75

Verify the Upgrade

Upgrading the Adapter (Linux) 73

Page 74

74 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 75

Troubleshooting the Adapter (Oracle Solaris x86)

These topics describe how to troubleshoot the installation and operation of the Oracle Dual Port 25 Gb Ethernet Adapter on an Oracle x86 server running the Oracle Solaris 11.3 OS. These topics cover basic installation issues and are not intended to be comprehensive.

■

“Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75

■

“Recover From a Port Hang (Oracle Solaris x86)” on page 76

■

“Analyze Slow Network Performance (Oracle Solaris x86)” on page 77

■

“Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)” on page 78

■

“Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)” on page 79

Related Information

■

“Updating Software and Firmware” on page 23

■

“Installing the Driver” on page 25

■

“Installing the Adapter” on page 31

■

“Configuring Driver Parameters” on page 51

■

“Removing the Driver” on page 69

Analyze Why the Device Link Is Missing (Oracle Solaris x86)

When you use the ifconfig or ipadm commands and you see an error message similar to the one below, perform these steps.

... cannot open bnxt0; link doesn't exist ...

Troubleshooting the Adapter (Oracle Solaris x86) 75

Page 76

Recover From a Port Hang (Oracle Solaris x86)

Check the OS.

To plumb the driver, refer to the Oracle Solaris ipadm(1M) man page. For information on the network interface parameters, refer to the Oracle Solaris ifconfig(1M) man page.

Check that the adapter is seated properly in its slot, that the cables are properly attached, and that the LEDs are functioning.

Ensure that the device is installed.

Use the prtconf or the scanpci command to verify that the device is installed. For more information, refer to the Oracle Solaris prtconf(1M) man page or the scanpci(1) man page.

If the device exists, check the /etc/driver_aliases file to ensure that the file contains a bnxt entry that corresponds to the name for the device.

If the entry exists, check the /etc/path_to_inst file to ensure that the file contains a bnxt entry.

Removing a device and reseating it in another slot does not always clean up the device tree. If this occurs, you must remove the device tree and reboot the server. For more information, refer to Managing Network Virtualization and Network Resources in Oracle Solaris 11.3.

Related Information

■

“Recover From a Port Hang (Oracle Solaris x86)” on page 76

■

“Analyze Slow Network Performance (Oracle Solaris x86)” on page 77

■

“Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)” on page 78

■

“Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)” on page 79

Recover From a Port Hang (Oracle Solaris x86)

Perform one of these actions.

■

If the interface encounters a soft hang, replumb the device. For instructions, refer to the Oracle Solaris ipadm(1M) man page.

■

If the interface encounters a hard hang, reboot the server.

■

If the interface encounters an additional hard hang, capture the trace information by using the dtrace command.

For example:

76 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 77

Analyze Slow Network Performance (Oracle Solaris x86)

# dtrace -F -m 'bnxt{trace(timestamp)}'

>/tmp/dtrace.out

For more information, refer to the Oracle Solaris dtrace(1M) man page.

■

If the server is panicked, retrieve the crash dump in /var/crash.

■

If the interface encountered a hard hang or a panic, file a CR at My Oracle Support.

Attach the last page of the dtrace command output or the crash dump file to the CR.

Check the driver statistics.

# kstat bnxt:* :statistics

Use these parameters for performance tuning in the bnxt.conf file.

Parameters Description

rx_itr Interval of receive interrupts 0 to 4080, 25 (50 usec) by default.

tx_itr

rx_limit_per_intr

rx_copy_threshold

tx_copy_threshold

Interval of transmit interrupts 0 to 4080, 25 (50 usec) by default.

Maximum number of packet to receive per interrupt16 ~ 4096, 1024 by default.

Packet size to determine bcopy or not during receive0 ~ 9216, 128 by default.

Packet size to determine bcopy or not during transmit0 ~ 9216, 128 by default.

Related Information

■

“Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75

■

“Analyze Slow Network Performance (Oracle Solaris x86)” on page 77

■

“Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)” on page 78

■

“Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)” on page 79

Analyze Slow Network Performance (Oracle Solaris x86)

The adapter supports several driver parameters that affect the performance of the ports. See

“Driver Parameters (Oracle Solaris x86)” on page 53 for more information about the default

values.

View the network performance.

Troubleshooting the Adapter (Oracle Solaris x86) 77

Page 78

Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)

# truss -p PID

Look for NIS, DNS, and network routing outages.

If you find any issues, fix them before proceeding.

View the I/O statistics to ensure that there are no bottlenecks on the disk.

# iostat -xcn 5

If you discover a bottleneck, set the logging to dump to the /tmp directory. Then retest to ensure that the new configuration improved performance.

Use the vmstat and the mpstat commands to check that none of these conditions exist:

■

CPU is pegged.

■

CPU is receiving too many interrupts.

■

Memory is low.

■

Page faults are occurring.

■

Contention for resources causes too many spins on mutex (smtx).

For more information, refer to the Oracle Solaris vmstat(1M) man page or the mpstat(1M) man page.

If the performance issue points to the driver, profile the call stack for bnxt by using the DTrace script. For more information about the DTrace script, go to My Oracle Support.

Related Information

■

“Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75

■

“Recover From a Port Hang (Oracle Solaris x86)” on page 76

■

“Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)” on page 78

■

“Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)” on page 79

Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)

Ensure that the correct cable type is being used.

See “Adapter Description” on page 32.

78 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 79

Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)

Check that the switch ports have been configured to operate in the mode in which the adapter is running.

Related Information

■

“Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75

■

“Recover From a Port Hang (Oracle Solaris x86)” on page 76

■

“Analyze Slow Network Performance (Oracle Solaris x86)” on page 77

■

“Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)” on page 79

Analyze Why Changing the MTU Does Not Correctly Set the Link Property (Oracle Solaris x86)

The dladm command might display this message:

# dladm: warning: cannot set link property 'mtu' on 'net0': link busy

It is possible that some objects defined on the link need to be removed or plumbed down.

Remove and retry the command.

# dladm set-linkprop -p mtu=9500 net0

For more information, refer to the Oracle Solaris dladm(1M) man page.

Related Information

■

“Analyze Why the Device Link Is Missing (Oracle Solaris x86)” on page 75

■

“Recover From a Port Hang (Oracle Solaris x86)” on page 76

■

“Analyze Slow Network Performance (Oracle Solaris x86)” on page 77

■

“Analyze Why the Link Is Not Up After Back-To-Back Cable Connection (Oracle Solaris x86)” on page 78

Troubleshooting the Adapter (Oracle Solaris x86) 79

Page 80

80 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 81

Glossary

adapter The Oracle Dual Port 25 Gb Ethernet Adapter.

DAC Direct attached cables.

DHCP Dynamic Host Configuration Protocol. Part of the application layer in the Internet protocol

suite.

DNS Domain name system. Translates human-readable domain names into numerical identifiers.

EEPROM Electronically erasable programmable read-only memory.

EMI Electromagnetic interference. The interference caused by the magnetic fields of electronic

components.

Gb Gigabyte.

GbE Gigabit Ethernet.

Gbps Gigabits-per-second.

Glossary 81

Page 82

GT Gigabit-transfer.

GTps Gigatransfers per second.

LC/LC Little Connector fiber optic cable that connects the transceiver to the adapter. This LC

connector is on a small form-factor pluggable transceiver. LC connectors have replaced SC connectors in corporate networking environments because of their smaller size.

LFM Linear Feet per minute.

link aggregation

LOM LAN-on-motherboard. A LAN design.

Link aggregation enables several physical ports to be bundled into a single logical channel.

MAC Media access control. Enables the use of a unique address for each device on a network.

Mb Megabit.

MTU Maximum transmission unit. The MTU (payload without the Ethernet header) affects how

jumbo frames function.

NIS Network Information Service. Originally known as Yellow Pages, NIS is a protocol for

distributed system configuration data.

PCIe Peripheral Component Interconnect Express.

PF Physical function.

82 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 83

PXE Preboot execution environment. Enables clients to boot over a network interface, independent

of the OS or other devices.

RDMA Remote Direct Memory Access. Allows computers in a network to exchange data in main

memory without involving the processor, cache, or OS of either computer.

RX Response. The automatic response mechanism used by Ethernet PAUSE frames.

SPI Serial peripheral interface. A type of flash memory.

SRC Source code. The SRC RPM is used in Linux to build the driver kernel files.

TCP Transmission Control Protocol. Part of the transport layer of the Internet protocol suite.

transceiver The 25GbE SFP28 optical transceiver that both ports in the adapter use to transmit data.

TX Generation. The automatic generation mechanism used by the Ethernet PAUSE frames.

UDP User Datagram Protocol. Part of the transport layer of the Internet protocol suite.

UEFI Unified Extensible Firmware Interface. Manages the operations between hardware firmware

and the OS during the boot time.

VF Virtual function.

Glossary 83

Page 84

VID

VID VLAN identifier. A 12-bit identifier in an Ethernet header.

VLAN Virtual LAN. Splits the physical LAN into logical subparts. Multiple VLANs are supported on

a single port, enabling a server with a single adapter to have a logical presence on multiple IP subnets.

VXLAN Virtual eXtensive LAN. A tunneling mechanism for providing isolated virtual Layer 2 (L2)

segments that can span multiple physical L2 segments.

84 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 85

Index

adapter

description, 32 front panel, 17 installing, 31 LEDs, 17 overview of upgrading (Linux), 71 part numbers, 32 upgrading (Linux), 71 verifying installation (Linux), 37 verifying installation (Oracle Solaris 86), 36 verifying installation (Windows), 37

adapter upgrade

verifying (Linux), 72

auto negotiation, not supported for a Linux boot, 42

bnxt

driver parameters (Oracle Solaris x86), 53 installing the driver (Oracle Solaris x86), 25 setting driver parameters (Oracle Solaris x86), 51

bnxt_en

driver parameters (Linux), 54 installing the driver (Linux), 26 installing the driver (Windows), 28 setting driver parameters (Linux), 53

boot

options, 41 options over the network, 39

booting over a 10GbE network

Linux OS, 42

booting over the network

using PXE, 41

with DHCP, 41

bottlenecks, 77

cable

cautions, 33 copper, 32 optical, 32 overview, 32

configuring

driver parameters, 51 jumbo frames, 55 link aggregation (Oracle Solaris x86), 59 network, 39 VLANs (Linux), 65 VLANs (Oracle Solaris x86), 64 VLANs (Windows), 65

VXLANs, 66 copper cable, 32 CPU

troubleshooting interrupts, 77

troubleshooting pegging, 77 crash dump, 76

deleting

link aggregation (Oracle Solaris x86), 61 device link, missing, 75 device tree, cleaning, 76 driver

downloading and installing (Linux), 26

Page 86

Index

downloading and installing (Windows), 28 removing (Linux), 70 removing (Oracle Solaris x86), 69 removing (Windows), 70

driver parameters

setting (Linux), 53, 54 setting (Oracle Solaris x86), 51

DTrace, 76

electrical specifications, 20 environmental specifications, 20

firmware update tool, 24

upgrading adapter (Linux), 71

hard port hang, troubleshooting, 76 hardware and software requirements, 21 hardware components, 15

I/O statistics, 77 installation

tasks (Linux), 12 tasks (Oracle Solaris x86), 11 tasks (Windows), 13 verifying (Linux), 37 verifying (Oracle Solaris x86), 36 verifying (Windows), 37

installing

adapter, 31, 34 bnxt driver (Oracle Solaris x86), 25 bnxt_en driver (Linux), 26 bnxt_en driver (Windows), 28 driver, 25

transceivers, 35

interface for Windows

configuring, 40

jumbo frames

changing the MTU permanently (Oracle Solaris x86), 55 changing the MTU temporarily (Oracle Solaris x86), 56 configuring (Linux), 56 configuring (Oracle Solaris x86), 55 configuring (Windows), 57

link aggregation

configuring (Oracle Solaris x86), 59 deleting (Oracle Solaris x86), 61 displaying information (Oracle Solaris x86), 60

Linux

boot over a 10GbE network, 42 configure VLANs, 65 configuring jumbo frames, 56 downloading and installing bnxt_en driver, 26 remove the bnxt_en driver, 70 setting driver parameters, 53 task overview, 12 verifying bnxt_en driver, 27 verifying installation, 37

memory outages, 77

network

configuration, 39 configuring the interface (Windows), 40

86 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017

Page 87

Index

performance, 77 routing outages, 77 task overview, 39

network interface

configuring (Oracle Solaris x86), 40 configuring (Windows), 40

networking interfaces, 53

optical cable, 32 Oracle Quad Port 10GBase-T Adapter

description, 32

Oracle Solaris x86, 64

configure VLANs, 64 configure VXLANs, 66 configuring a link aggregation, 59, 59 downloading and installing bnxt driver, 25 remove the bnxt driver, 69 setting driver parameters, 51 setting driver parameters (Oracle Solaris x86), 53 task overview, 11 troubleshooting adapter, 75 verifying adapter installation, 36

OS, updating Oracle Solaris x86, 23

page faults, 78 panic, system, 76 performance

bottlenecks, 77

network, 77 physical specifications, 19 port hang

troubleshooting, 76

RDMA over Converged Ethernet See RoCE removing the driver

Linux, 70 Oracle Solaris x86, 69

Windows, 70 resource convention, 78 RoCE See installing (Linux) See installing on VMware ESXi See installing on Windows User Mode See installing on Windows, Kernel Mode

list of supported OS, 44

supported feature combinations, 44

setting

driver parameters (Linux), 54

driver parameters (Oracle Solaris x86), 51 SFP28 transceivers

installing, 35

overview, 32 shipping kit contents, 15 soft port hang, troubleshooting, 76 specifications

electrical, 20

environmental, 20

physical, 19 SRIOV

not suported by bnxt driver (Oracle Solaris

x86), 25 statistics, 77 system panic, 76

trace information, 76 transceivers

installing, 35

overview, 32 troubleshooting

adapter on Oracle Solaris x86, 75

CPU interrupts, 77

CPU pegging, 77

crash dump, 76

device links, 75

Page 88

Index

device tree, 76 DTrace, 76 memory outages, 78 page faults, 78 port hang, 76 resource contention, 78

update tool overview, firmware, 24 updating Oracle Solaris x86, 23 upgrading

adapter (Linux), 71

verifying

adapter upgrade (Linux), 72 bnxt_en driver (Linux), 27, 27 bnxt_en driver (Oracle Solaris x86), 25

viewing

link aggregation information (Oracle Solaris x86), 60

VLANs

configuring, 64

Linux, 65

Windows, 65 ID, 63 overview, 63

VXLANs, configuring, 66

Windows user mode See installing RoCE driver

Windows

configure VLANs, 65 configuring jumbo frames, 57 downloading and installing bnxt_en driver, 28 remove the bnxt_en driver, 70 task overview, 13 verifying adapter installation, 37

Windows kernel mode See installing RoCE driver

88 Oracle Dual Port 25 Gb Ethernet Adapter User's Guide • September 2017