Intel Omni-Path Switch 100 Series, Omni-Path Edge Switch Series, Omni-Path Director Class Switch Series Installation Manual

Intel® Omni-Path Fabric Switches Hardware
Installation Guide
Rev. 10.0
June 2019
Doc. No.: H76456, Rev.: 10.0
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.
Intel, the Intel logo, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2015–2019, Intel Corporation. All rights reserved.
®
Omni-Path Fabric Switches Hardware
Intel Installation Guide June 2019 2 Doc. No.: H76456, Rev.: 10.0
Revision History—Intel® Omni-Path Fabric

Revision History

For the latest documentation, go to: http://www.intel.com/omnipath/SwitchPublications.
Date Revision Description
June 2019 10.0 • Updated the FM Switch LED description.
April 2019 9.0 Added Appendix B Director Switch Cable Management on page 90.
February 2019 8.0 • Added an Edge switch cable port connection limitation statement to
April 2018 7.0 • Added Figure 57, "48-Port Leaf Module Port Numbering"
April 2017 6.0 • Updated Air Baffle information in "Installing the Airflow Baffles"
December 2016 5.0 Document updated for Revision 5.0
August 2016 4.0 Document updated for Revision 4.0
May 2016 3.0 Document updated for Revision 3.0
February 2016 2.0 Document updated for Revision 2.0
November 2015 1.0 Document updated for Revision 1.0
• Edge switch product naming. Updated remaining instances of "unmanaged" to "externally-managed" wherever applicable.
the section Cable Distances on page 28.
• The section Updating the Management Module IP Addresses in a
Director Class Redundant Management Configuration on page 75
was updated to use the boot command moduleip.
• Added Q7 BIOS Update on page 86
• Added module installation information for the Omni-Path Fabric Edge Hot Swap in "Installing the Edge Switch Hot-Swap Power and Fan Modules".
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 3
Intel® Omni-Path Fabric Switches Hardware
®
Omni-Path Fabric—Contents
Intel

Contents

Revision History..................................................................................................................3
Preface............................................................................................................................... 9
Intended Audience....................................................................................................... 9
Intel® Omni-Path Documentation Library.........................................................................9
How to Search the Intel® Omni-Path Documentation Set........................................ 11
Cluster Configurator for Intel® Omni-Path Fabric.............................................................12
Documentation Conventions........................................................................................ 12
Laser Safety Information.............................................................................................13
Electrostatic Discharge Sensitivity (ESDS) Precautions.................................................... 13
License Agreements....................................................................................................13
Technical Support.......................................................................................................13
1.0 Introduction................................................................................................................14
1.1 Safety Information................................................................................................14
1.1.1 Statement 1.............................................................................................15
1.1.2 Statement 2.............................................................................................17
1.1.3 Statement 3.............................................................................................19
1.1.4 Statement 4.............................................................................................20
1.1.5 Statement 5.............................................................................................22
1.1.6 Statement 6.............................................................................................24
1.1.7 Statement 7.............................................................................................25
2.0 Installation................................................................................................................. 28
2.1 Planning the Installation........................................................................................ 28
2.1.1 Cable Requirements.................................................................................. 28
2.1.2 Uninterruptible Power Supply Equipment......................................................31
2.2 Installation Tasks Checklist.....................................................................................31
2.2.1 Tools and Equipment Required.................................................................... 32
2.2.2 Check the Installation Site......................................................................... 32
2.2.3 Mark the Rack.......................................................................................... 32
2.3 Intel® Omni-Path Fabric Edge Switch Installation...................................................... 33
2.3.1 Mounting Hardware Kit Contents................................................................. 33
2.3.2 Rack Mounting..........................................................................................33
2.3.3 Installing the Airflow Baffles....................................................................... 37
2.3.4 Installing the Switch Fascia........................................................................ 40
2.3.5 Reversing the Fan Airflow...........................................................................41
2.3.6 Installing the Q7 Management Module......................................................... 43
2.4 Intel® Omni-Path Director Class Switch 100 Series Installation................................... 45
2.4.1 Mounting Hardware Kit Contents................................................................. 45
2.4.2 Unpacking the Switch................................................................................ 45
2.4.3 Installation Tasks...................................................................................... 51
2.4.4 Installing the Optional Cable Management Guide........................................... 59
2.4.5 Installing Director Class Modules.................................................................61
2.4.6 Installing the Edge Switch Hot-Swap Power and Fan Modules.......................... 65
2.5 Connect Equipment to the Ports and Power On the System.........................................68
2.5.1 Director Class Switch Port and Slot Numbering..............................................69
2.5.2 Leaf Module Port Numbering.......................................................................70
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 4 Doc. No.: H76456, Rev.: 10.0
Contents—Intel® Omni-Path Fabric
2.5.3 Edge Switch Cabling Recommendations........................................................71
2.6 Setting Up the USB Console................................................................................... 73
2.7 Bringing Up the System For the First Time................................................................73
2.7.1 Start-up Procedures.................................................................................. 73
2.7.2 Changing the Switch IP Address and Default Gateway through the CLI............. 73
2.7.3 Updating the Management Module IP Addresses in a Director Class
Redundant Management Configuration........................................................75
2.8 Component LEDs.................................................................................................. 75
2.8.1 Edge Switch LEDs..................................................................................... 76
2.8.2 Director Class LEDs................................................................................... 79
Appendix A Q7 BIOS Update............................................................................................. 86
A.1 Before Getting Started.......................................................................................... 86
A.1.1 Prior to Upgrading.....................................................................................86
A.2 Displaying the Current Version................................................................................86
A.3 SFTP Server Configuration..................................................................................... 87
A.4 Updating the Edge Switches................................................................................... 87
A.4.1 Performing the Upgrade.............................................................................87
A.5 Updating Director Class Switches............................................................................ 88
A.5.1 Performing the Upgrade.............................................................................88
Appendix B Director Switch Cable Management................................................................ 90
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 5
Intel® Omni-Path Fabric Switches Hardware
®
Omni-Path Fabric—Figures
Intel

Figures

1 Edge Switch Port Connection Limitations.....................................................................29
2 QSFP Cable Connector Orientation............................................................................. 29
3 Copper Cable Bend Radius ....................................................................................... 30
4 Fiber Optic Bend Radius ...........................................................................................31
5 Intel® Omni-Path Edge Switch Rail.............................................................................33
6 Rail Installation Pin.................................................................................................. 34
7 Installed the Rack Rails............................................................................................ 34
8 Intel® Omni-Path Fabric Edge Switch Mounted in a Four Post Standard Rack....................35
9 Switch Rail Adjustments........................................................................................... 36
10 Rail Adjustment Notches...........................................................................................36
11 Edge Baffle (Top View)............................................................................................. 38
12 Secure the Top Baffle to the Switch............................................................................39
13 Remove Rail Post.....................................................................................................39
14 Insert Clamp on Baffles............................................................................................ 39
15 Clamp over Rail Post................................................................................................ 40
16 Reattach the Rail Post.............................................................................................. 40
17 Switch Fascia.......................................................................................................... 41
18 Rail Posts for Fascia Installation................................................................................. 41
19 Switch Cover Thumbscrews (Top View).......................................................................42
20 Fan Assembly Rotation............................................................................................. 42
21 Wire Assembly Strain Relief and Airflow Direction Indicators..........................................43
22 Q7 Management Module........................................................................................... 43
23 Switch Cover Thumbscrews (Top View).......................................................................44
24 Installing the Q7 Module...........................................................................................44
25 Q7 Module Mounting Screw.......................................................................................45
26 Unpacking the Switch (1)......................................................................................... 46
27 Unpacking the Switch (2)......................................................................................... 47
28 Unpacking the Switch (3)......................................................................................... 48
29 Unpacking the Switch (4)......................................................................................... 48
30 Unpacking the Switch (5)......................................................................................... 49
31 Unpacking the Switch (6)......................................................................................... 50
32 Unpacking the Switch (7)......................................................................................... 51
33 Measure the Rack.................................................................................................... 52
34 Preset the Adjustable Rails........................................................................................52
35 Install Fixture Brackets.............................................................................................53
36 Secure Shelf to Rack................................................................................................53
37 Secure the Baffles to Top of Shelf Rails....................................................................... 54
38 Secure the Top of the Baffles to the Rack Rails.............................................................55
39 Completed Shelf Assemblies......................................................................................56
40 Slide the Chassis into the Rack..................................................................................57
41 Fasten the Chassis to the Rack.................................................................................. 58
42 Completed Switch Installations..................................................................................59
43 Installing Cable Management.................................................................................... 60
44 Module Installation 1................................................................................................62
45 Module Installation 2................................................................................................63
46 Module Installation 3................................................................................................63
47 Module Installation 4................................................................................................64
48 Module Installation 5................................................................................................64
49 Edge Power Supply Install 1......................................................................................66
50 Edge Power Supply Install 2......................................................................................67
51 Edge Fan Supply Install 1......................................................................................... 67
52 Edge Fan Supply Install 2......................................................................................... 68
53 Edge Switch Serial and Ethernet Ports........................................................................ 68
54 Director Switch Serial and Ethernet Ports ...................................................................68
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 6 Doc. No.: H76456, Rev.: 10.0
Figures—Intel® Omni-Path Fabric
55 Director Class Leaf Slot Numbering............................................................................ 69
56 Leaf Module Port Numbering..................................................................................... 70
57 48-Port Leaf Module Port Numbering ......................................................................... 70
58 Chassis Viewer GUI Leaf Port Stats Scree....................................................................71
59 48-Port Edge Switch Cabling Recommendations...........................................................72
60 Edge Switch LEDs.................................................................................................... 76
61 Ports, Fabric Manager (FM), Managed (M), and FM Switch LEDs..................................... 77
62 Activating the FM Enable Switch LED.......................................................................... 78
63 Power Supply LEDs.................................................................................................. 79
64 Airflow Direction LED............................................................................................... 79
65 Management Module LEDs........................................................................................ 80
66 Leaf Module LEDs.................................................................................................... 81
67 Spine Module LEDs.................................................................................................. 82
68 SEEB Module LEDs...................................................................................................83
69 Director Class Fan LEDs............................................................................................84
70 Director Class Power Supply LEDs..............................................................................85
71 Leaf Cabling............................................................................................................90
72 Using Velcro Ties and the Cable Management Hanger................................................... 90
73 Tying Off All Cables..................................................................................................91
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 7
Intel® Omni-Path Fabric Switches Hardware
®
Omni-Path Fabric—Tables
Intel

Tables

1 Cable Bend Radius Guidelines....................................................................................30
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 8 Doc. No.: H76456, Rev.: 10.0
Preface—Intel® Omni-Path Fabric

Preface

This manual is part of the documentation set for the Intel® Omni-Path Fabric (Intel OP Fabric), which is an end-to-end solution consisting of Intel® Omni-Path Host Fabric Interfaces (HFIs), Intel® Omni-Path switches, and fabric management and development tools.
The Intel® OP Fabric delivers the next generation, High-Performance Computing (HPC) network solution that is designed to cost-effectively meet the growth, density, and reliability requirements of large-scale HPC clusters.
Both the Intel® OP Fabric and standard InfiniBand* (IB) are able to send Internet Protocol (IP) traffic over the fabric, or IPoFabric. In this document, however, it may also be referred to as IP over IB or IPoIB. From a software point of view, IPoFabric behaves the same way as IPoIB, and in fact uses an ib_ipoib driver to send IP traffic over the ib0/ib1 ports.

Intended Audience

The intended audience for the Intel® Omni-Path (Intel® OP) document set is network administrators and other qualified personnel.

Intel® Omni-Path Documentation Library

Intel® Omni-Path publications are available at the following URLs:
Intel® Omni-Path Switches Installation, User, and Reference Guides
http://www.intel.com/omnipath/SwitchPublications
Intel® Omni-Path Software Installation, User, and Reference Guides (includes HFI documents)
http://www.intel.com/omnipath/FabricSoftwarePublications
Drivers and Software (including Release Notes)
http://www.intel.com/omnipath/Downloads
®
Use the tasks listed in this table to find the corresponding Intel® Omni-Path document.
Task
Key:
Shading indicates the URL to use for accessing the particular document.
• Intel® Omni-Path Switches Installation, User, and Reference Guides: http://www.intel.com/omnipath/SwitchPublications
• Intel® Omni-Path Software Installation, User, and Reference Guides (includes HFI documents):
http://www.intel.com/omnipath/FabricSoftwarePublications (no shading)
• Drivers, Software, and Firmware (including Release Notes): http://www.intel.com/omnipath/Downloads
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 9
Document Title Description
continued...
Intel® Omni-Path Fabric Switches Hardware
Task Document Title Description
A roadmap to Intel's comprehensive library of publications
Using the Intel® OPA documentation set
Intel® Omni-Path Fabric Quick Start Guide
describing all aspects of the product family. This document outlines the most basic steps for getting your Intel® Omni­Path Architecture (Intel® OPA) cluster installed and operational.
Provides a high level overview of the steps required to stage a customer-based installation of the Intel® Omni-Path Fabric.
Setting up an Intel OPA cluster
®
Intel® Omni-Path Fabric Setup Guide
Procedures and key reference documents, such as Intel Omni-Path user guides and installation guides, are provided to clarify the process. Additional commands and best known methods are defined to facilitate the installation process and troubleshooting.
Describes the hardware installation and initial configuration tasks for the Intel® Omni-Path Switches 100 Series. This includes: Intel® Omni-Path Edge Switches 100 Series, 24 and 48-port configurable Edge switches, and Intel® Omni-Path Director Class Switches 100 Series.
Contains instructions for installing the HFI in an Intel® OPA cluster.
Installing hardware
Intel® Omni-Path Fabric Switches Hardware Installation Guide
Intel® Omni-Path Host Fabric Interface Installation Guide
Installing host software
Installing HFI firmware
Installing switch firmware (externally-
Intel® Omni-Path Fabric Software Installation Guide
Describes using a Text-based User Interface (TUI) to guide you through the installation process. You have the option of using command line interface (CLI) commands to perform the installation or install using the Linux* distribution software.
managed switches)
Managing a switch using GUI
Installing switch firmware (managed switches)
Describes the graphical user interface (GUI) of the Intel Omni-Path Fabric GUI. This document provides task-oriented procedures for configuring and managing the Intel® Omni­Path Switch family.
Help: GUI embedded help files
Managing a switch using the CLI
Installing switch firmware (managed
Intel® Omni-Path Fabric Switches Command Line Interface Reference Guide
Describes the command line interface (CLI) task information for the Intel® Omni-Path Switch family.
Help: -help for each CLI
switches)
Provides instructions for using the set of fabric management tools designed to simplify and optimize common fabric management tasks. The management tools consist of Text-
Managing a fabric using FastFabric
Intel® Omni-Path Fabric Suite FastFabric User Guide
based User Interface (TUI) menus and command line interface (CLI) commands.
Help: -help and man pages for each CLI. Also, all host CLI commands can be accessed as console help in the Fabric Manager GUI.
The Fabric Manager uses a well defined management protocol
Intel® Omni-Path Fabric Suite Fabric Manager User Guide
to communicate with management agents in every Intel Omni-Path Host Fabric Interface (HFI) and switch. Through these interfaces the Fabric Manager is able to discover,
configure, and monitor the fabric. Managing a fabric using Fabric Manager
Intel® Omni-Path Fabric Suite Fabric Manager GUI User Guide
Provides an intuitive, scalable dashboard and set of analysis
tools for graphically monitoring fabric status and
configuration. This document is a user-friendly alternative to
traditional command-line tools for day-to-day monitoring of
fabric health.
Help: Fabric Manager GUI embedded help files
Intel® Omni-Path Fabric—Preface
®
®
®
continued...
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 10 Doc. No.: H76456, Rev.: 10.0
®
Preface—Intel
Configuring and administering Intel HFI and IPoIB driver
Running MPI applications on Intel® OPA
Writing and running middleware that uses Intel® OPA
Optimizing system performance
Designing an IP or LNet router on Intel OPA
Building Containers for Intel® OPA fabrics
Writing management applications that interface with Intel OPA
Using NVMe* over Fabrics on Intel OPA
Learning about new release features, open issues, and resolved issues for a particular release
Omni-Path Fabric
Task Document Title Description
®
®
Intel® Omni-Path Fabric Host Software User Guide
Intel® Performance Scaled Messaging 2 (PSM2) Programmer's Guide
Intel® Omni-Path Fabric Performance Tuning User Guide
Intel® Omni-Path IP and LNet Router Design Guide
®
(Old title: Intel® Omni-Path IP and
Storage Router Design Guide)
Building Containers for Intel® Omni­Path Fabrics using Docker* and Singularity* Application Note
Intel® Omni-Path Management API
®
Programmer’s Guide
Configuring Non-Volatile Memory Express* (NVMe*) over Fabrics on Intel® Omni-Path Architecture Application Note
Intel® Omni-Path Fabric Software Release Notes
Intel® Omni-Path Fabric Manager GUI Release Notes
Intel® Omni-Path Fabric Switches Release Notes (includes managed and externally-managed switches)
Intel® Omni-Path Fabric Unified Extensible Firmware Interface (UEFI) Release Notes
Intel® Omni-Path Fabric Thermal Management Microchip (TMM) Release Notes
Intel® Omni-Path Fabric Firmware Tools Release Notes
Describes how to set up and administer the Host Fabric Interface (HFI) after the software has been installed. The audience for this document includes cluster administrators and Message-Passing Interface (MPI) application programmers.
Provides a reference for programmers working with the Intel PSM2 Application Programming Interface (API). The Performance Scaled Messaging 2 API (PSM2 API) is a low­level user-level communications interface.
Describes BIOS settings and parameters that have been shown to ensure best performance, or make performance more consistent, on Intel® Omni-Path Architecture. If you are interested in benchmarking the performance of your system, these tips may help you obtain better performance.
Describes how to install, configure, and administer an IPoIB router solution (Linux* IP or LNet) for inter-operating between Intel® Omni-Path and a legacy InfiniBand* fabric.
Provides basic information for building and running Docker* and Singularity* containers on Linux*-based computer platforms that incorporate Intel® Omni-Path networking technology.
Contains a reference for programmers working with the Intel® Omni-Path Architecture Management (Intel OPAMGT) Application Programming Interface (API). The Intel OPAMGT API is a C-API permitting in-band and out-of-band queries of the FM's Subnet Administrator and Performance Administrator.
Describes how to implement a simple Intel® Omni-Path Architecture-based point-to-point configuration with one target and one host server.
®

How to Search the Intel® Omni-Path Documentation Set

Many PDF readers, such as Adobe* Reader and Foxit* Reader, allow you to search across multiple PDFs in a folder.
Follow these steps:
1. Download and unzip all the Intel® Omni-Path PDFs into a single folder.
2. Open your PDF reader and use CTRL-SHIFT-F to open the Advanced Search window.
3. Select All PDF documents in...
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 11
Intel® Omni-Path Fabric Switches Hardware
4. Select Browse for Location in the dropdown menu and navigate to the folder containing the PDFs.
5. Enter the string you are looking for and click Search.
Use advanced features to further refine your search criteria. Refer to your PDF reader Help for details.

Cluster Configurator for Intel® Omni-Path Fabric

The Cluster Configurator for Intel® Omni-Path Fabric is available at: http://
www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path­configurator.html.
This tool generates sample cluster configurations based on key cluster attributes, including a side-by-side comparison of up to four cluster configurations. The tool also generates parts lists and cluster diagrams.

Documentation Conventions

The following conventions are standard for Intel® Omni-Path documentation:
Note: provides additional information.
Caution: indicates the presence of a hazard that has the potential of causing damage to data or equipment.
Warning: indicates the presence of a hazard that has the potential of causing personal injury.
Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide. Links to websites are also shown in blue. For example:
See License Agreements on page 13 for more information.
For more information, visit www.intel.com.
Text in bold font indicates user interface elements such as menu items, buttons, check boxes, key names, key strokes, or column headings. For example:
Click the Start button, point to Programs, point to Accessories, and then click Command Prompt.
Press CTRL+P and then press the UP ARROW key.
Text in Courier font indicates a file name, directory path, or command line text. For example:
Enter the following command: sh ./install.bin
Text in italics indicates terms, emphasis, variables, or document titles. For example:
Refer to Intel® Omni-Path Fabric Software Installation Guide for details.
In this document, the term chassis refers to a managed switch.
®
Omni-Path Fabric—Preface
Intel
Procedures and information may be marked with one of the following qualifications:
(Linux) – Tasks are only applicable when Linux* is being used.
(Host) – Tasks are only applicable when Intel® Omni-Path Fabric Host Software or Intel® Omni-Path Fabric Suite is being used on the hosts.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 12 Doc. No.: H76456, Rev.: 10.0
®
Preface—Intel
Omni-Path Fabric
(Switch) – Tasks are applicable only when Intel® Omni-Path Switches or Chassis are being used.
Tasks that are generally applicable to all environments are not marked.

Laser Safety Information

This product may use Class 1 laser optical transceivers to communicate over the fiber optic conductors. The U.S. Department of Health and Human Services (DHHS) does not consider Class 1 lasers to be hazardous. The International Electrotechnical Commission (IEC) 825 Laser Safety Standard requires labeling in English, German, Finnish, and French stating that the product uses Class 1 lasers. Because it is impractical to label the transceivers, the following label is provided in this manual.

Electrostatic Discharge Sensitivity (ESDS) Precautions

The assemblies used in the switch chassis are ESD sensitive. Observe ESD handling procedures when handling any assembly used in the switch chassis.

License Agreements

This software is provided under one or more license agreements. Please refer to the license agreement(s) provided with the software for specific detail. Do not install or use the software until you have carefully read and agree to the terms and conditions of the license agreement(s). By loading or using the software, you agree to the terms of the license agreement(s). If you do not wish to so agree, do not install or use the software.

Technical Support

Technical support for Intel® Omni-Path products is available 24 hours a day, 365 days a year. Please contact Intel Customer Support or visit http://www.intel.com/omnipath/
support for additional detail.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 13
Intel® Omni-Path Fabric Switches Hardware

1.0 Introduction

This manual describes the hardware installation and initial configuration tasks for the Intel® Omni-Path Switch 100 Series. This includes:
Intel® Omni-Path Edge Switch 100 Series, 24 and 48-port configurable edge switches
Intel® Omni-Path Director Class Switch 100 Series
This manual is organized as follows:
This Introduction describes the product safety information.
Installation describes the hardware installation and initial configuration tasks.
Appendix Q7 BIOS Update provides instructions for updating the Q7 BIOS on Edge and Director Class Switches.
Intel® Omni-Path Fabric—Introduction
1.1

Safety Information

Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 14 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric

Statement 11.1.1

June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 15
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 16 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric

Statement 21.1.2

June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 17
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 18 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric

Statement 31.1.3

June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 19
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction
1.1.4
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 20 Doc. No.: H76456, Rev.: 10.0

Statement 4

Introduction—Intel® Omni-Path Fabric
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 21
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction
1.1.5

Statement 5

Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 22 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 23
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction

Statement 61.1.6

Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 24 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric

Statement 71.1.7

June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 25
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Introduction
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 26 Doc. No.: H76456, Rev.: 10.0
Introduction—Intel® Omni-Path Fabric
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 27
Intel® Omni-Path Fabric Switches Hardware

2.0 Installation

This section describes how to install and configure for first-time use:
The Intel® Omni-Path Fabric Series switches in a network environment.
Intel® Omni-Path Fabric—Installation
2.1

Planning the Installation

The Intel® Omni-Path switches are designed for installation in standard 19-inch equipment racks.
Racks should conform to conventional standards. Use the American National Standards Institute (ANSI)/Electronic Industries Association (EIA) standard ANSI/ EIA-310-D-92 and International Electrotechnical Commission (IEC) 297
Racks should meet the following mechanical recommendations:
Four-post, 19" rack to facilitate easy maintenance
Universal mounting rail hole pattern identified in IEC Standard 297
Mounting holes flush with the rails to accommodate the chassis
NOTE
Operation is subject to the following conditions:
NOTE
Use a rack grounding kit and a ground conductor that is carried back to earth or to another suitable building ground. Ground the equipment rack to earth ground.
NOTE
Provide enough room to work on the equipment. Clear the work site of any unnecessary materials. Make sure the equipment will have enough clearance for front and rear access.
2.1.1
2.1.1.1
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 28 Doc. No.: H76456, Rev.: 10.0

Cable Requirements

Cable Distances
When planning the location of the switches, consider the distance limitations for signaling, EMI, and connector compatibility. It is recommended that the user does not exceed specified transmission rate and distance limits.
NOTE
When connecting the outer ports of two (2) 48-port Edge switches (highlighted green in Figure 1 below), make certain to use active optical cables (AOC) for distances greater than 2 meters.
Installation—Intel® Omni-Path Fabric
Figure 1. Edge Switch Port Connection Limitations
2.1.1.2
Cable Guidelines
NOTE
Building and electrical codes vary depending on the location. Comply with all code specifications when planning the site and installing cable.
When running cable to the equipment, consider the following:
Do not run cables where they can be stepped on or rolled over.
Be sure cables are intact with no cuts, bends, or nicks.
If the user is making a cable, ensure that the cable is properly crimped.
Provide proper strain relief for cables.
Support cables using a cable manager mounted above connectors to avoid unnecessary weight on the cable bundles.
Bundle cable using Velcro straps to avoid injuring cables.
Keep all ports and connectors free of dust.
Untwisted Pair (UTP) cables can build up Electrostatic Discharge (ESD) charges when being placed into a new installation. Before installing category 5 UTP cables, discharge ESD from the cable by plugging it into a port on a system that is not powered on.
When required for safety and fire rating requirements, plenum-rated cable can be used. Check the local building codes to determine when it is appropriate to use plenum-rated cable, or refer to IEC standard 850.
In order to create enough space, QSFP cables connectors may need to be rotated when plugging into alternate rows of switch components.
Figure 2. QSFP Cable Connector Orientation
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 29
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
2.1.1.3
Cable Handling and Bend Radius
Intel® Omni-Path Copper Cable 100 Series:
— Recommended minimum bend radius is >69mm
Intel® Omni-Path Active Optical Cable 100 Series:
— Recommended minimum bend radius is >105 mm within 100 mm of either
cable end module, and >60 mm on the rest of the cable
Table 1. Cable Bend Radius Guidelines
Intel® Omni-Path Copper Cable 100 Series
American Wire Gauge (AWG) Size Cable Bend Radius
26 >69 mm
30 >66 mm
Intel® Omni-Path Active Optical Cable 100 Series
Gauge Bend Radius
Optical >105 mm within 100 mm of either cable end module
Figure 3. Copper Cable Bend Radius
>60 mm on the rest of the cable
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 30 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 4. Fiber Optic Bend Radius
2.1.2
2.2

Uninterruptible Power Supply Equipment

Consider the following when selecting uninterruptible power supply (UPS) equipment:
The minimum amperage requirements for a UPS:
— Calculate VA (volt-amps): Locate the voltage and amperage requirements for
each piece of equipment (usually located on a sticker on the back or bottom of the equipment). Multiply the numbers together to get VA.
— Add the VA from each piece of equipment together to find the total VA
requirement. Then add 30% to determine the minimum amperage requirements for the UPS.
Transition time (the time necessary for the UPS to transfer from utility power to full-load battery power).
The longest potential time period the UPS might be required to supply backup power.
Whether the UPS unit provides online protection.

Installation Tasks Checklist

To perform the actual switch installation, the site implementation engineer must perform the following tasks, which are detailed in this section.
CAUTION
Be sure to review the safety information before starting the installation and during the installation process (refer to Safety Information on page 14).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 31
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
1. Check the installation site to verify the installation of cabinet power feeds, rails, and grounding.
2. Unpack the equipment and inspect for any shipping damage. Any shipping damage should be reported to the shipping company.
3. Verify that the equipment shipped matches the packing list.
4. Mark the rack and install the mounting rails.
5. Physically install the switch in the rack.
2.2.1
2.2.2

Tools and Equipment Required

An ESD wrist strap
A #0 and #2 Phillips screwdriver
An M6 HEX nut wrench
M6 screws
Torque wrench
Open sockets or open wrenches
Lifting device (e.g., pallet jack) for the Intel® Omni-Path Director Class Switch 100 Series

Check the Installation Site

The switches are designed to be installed in an existing server cabinet (not a telco cabinet), where it can be mounted in a standard equipment rack.
Be sure of the following:
The cabinet has a full earth ground to provide reliable grounding.
There is enough room to work on the equipment.
The equipment will have enough clearance for front and rear access.
The cables can be accessed easily.
Water or moisture cannot enter the switch.
The ambient temperature stays between 50–113 °F (5–40 °C).
Cabinet doors do not interfere with front-to-back air flow.
The cabinet should have its own switchable power distribution. If the switch has two power supplies, it is suggested that a cabinet with dual power distribution units is used.
It is recommended that cabinet anti-tip devices are used. This is especially true if installing or removing a switch in the upper half of the cabinet when the lower half is empty.
2.2.3
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 32 Doc. No.: H76456, Rev.: 10.0

Mark the Rack

Allow enough vertical space in the rack for each specific switch installation.
1. Determine the location in the rack of the bottom of the switch.
Installation—Intel® Omni-Path Fabric
2. Mark the upper (if applicable) and lower mounting positions on the vertical rails on the front of the rack.
3. Mark the upper (if applicable) and lower mounting positions on the vertical rails on the back of the rack.
2.3
2.3.1
2.3.2
Figure 5. Intel® Omni-Path Edge Switch Rail

Intel® Omni-Path Fabric Edge Switch Installation

This section describes the rack mounting instructions for the 24- and 48-port Intel Omni-Path Fabric Edge Switches in a four-post standard-depth rack. The installation of either switch is very similar; the majority of steps apply to both switches. Where applicable, any specifics for either switch are called out.
In a standard-depth rack, the distance between the front and back mounting posts is ~28" (700 mm). Mounting rails for the switch are adjustable to accommodate racks with 26"–32" between mounting posts.

Mounting Hardware Kit Contents

The Mounting Kit contains:
ESD wrist strap
One pair of mounting rails adjustable for 26"–32" installation range
M6 hex head screws
Front 1U perforated cover

Rack Mounting

®
1. Insert the rail installation pin in to the rack hole as shown in the following figure.
This holds the rail in place so the user can install the fan side screws.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 33
Intel® Omni-Path Fabric Switches Hardware
Figure 6. Rail Installation Pin
a. Fasten the rail using two (2) M6 screws.
b. Torque screws to 30 in-lb.
c. Repeat this step for the second rail.
2. Fasten the port-side rails using four (4) M6 screws.
a. Torque screws to 30 in-lb.
Note the orientation of mounting rails in the following figure. The rails are shown from the front side of the rack.
®
Omni-Path Fabric—Installation
Intel
Figure 7. Installed the Rack Rails
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 34 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 8. Intel® Omni-Path Fabric Edge Switch Mounted in a Four Post Standard Rack
3. The rails are designed for different mounting positions, from flush mount to recessed (in 1 inch increments) as shown in the following figure.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 35
Intel® Omni-Path Fabric Switches Hardware
Figure 9. Switch Rail Adjustments
®
Omni-Path Fabric—Installation
Intel
4. To change the depth of the switch, insert your fingers into the rail adjustment notches and pull away from the switch as shown in the following figure.
This disengages the rail latches from the switch hat section.
Figure 10. Rail Adjustment Notches
5. Slide the switch to the desired depth in the rack.
The switch is adjustable in increments of 1 inch. When the switch is at the proper depth, push the rail adjustment notches back to their original position to secure in place.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 36 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.3.3

Installing the Airflow Baffles

For rack cabinet installations of high-powered servers and associated fabric equipment, it is critical for proper cooling of the equipment that air recirculation from the air outlet side of the equipment to the inlet side of the equipment be minimized.
Following are some general requirements:
All equipment in a given cabinet must be configured to the same flow pattern
All unused 'U' increments must have some type of closeout installed
Cabinet side spaces must be closed off
Any front-to-back cable pass-through paths must be configured with flexible fillers allowing for the cable to pass through-but not air. This is also true for patch panels.
Top-of-rack and bottom-of-rack transitions must be closed out
The Intel® Omni-Path Edge switch is generally shorter than the associated servers installed in the same cabinet. Therefore, the following additional considerations should be considered:
The mounting rails supplied with the edge switch are adequate for closing off the left-and-right side spaces on the cabinet from the 'U' increment the edge switch occupies. If full-depth servers are installed directly above and below the edge switch this will generally suffice to close off the space, minimizing recirculation. However, if the space above and/or below the Edge switch is empty-or occupied by racks that are not full depth-then the Air Baffle Kit that is available for the edge product needs to be installed. This kit closes out the space between the fan grill end of the switch and the front of the cabinet. The condition requiring the air baffle may also exist in top-of-rack or bottom-of-rack installations, depending on the cabinet geometry in these regions.
The Edge Air Baffle Kit mounts to the mounting rails supplied with the switch and is adjustable for various cabinet depths.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 37
Intel® Omni-Path Fabric Switches Hardware
Figure 11. Edge Baffle (Top View)
®
Omni-Path Fabric—Installation
Intel
NOTE
Installation of the baffle(s) take place after the edge switch has been rack mounted.
1. Unlock the rail system by pulling the handles out.
2. Guide the baffle onto the rails, then slide the baffle into the rack until it stops against the edge switch.
a. Baffle Extension(s): Each baffle (i.e., top and bottom) has an extension
(shown in yellow below). If not pre-configured, insert 4 screws (2 on each side) to secure the extension to the main baffle. The location of the screws depends upon the depth that the edge has been mounted.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 38 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 12. Secure the Top Baffle to the Switch
3. After baffle is fully inserted, push the handles back until the rail locks in place.
To vertically secure the baffles, insert the baffle clamp (one on each side) as shown:
1. Unscrew the end of the rail post.
Figure 13. Remove Rail Post
2. Slide the clamp into place on the top and bottom baffles.
Figure 14. Insert Clamp on Baffles
3. Push the clamp into place over the rail post.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 39
Intel® Omni-Path Fabric Switches Hardware
Figure 15. Clamp over Rail Post
4. Reattach the rail post, which locks the clamp into place.
Figure 16. Reattach the Rail Post
Intel® Omni-Path Fabric—Installation
2.3.4

Installing the Switch Fascia

The Edge switch fascia guards against ESD as well as accidental contact with the fans and power supplies.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 40 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 17. Switch Fascia
The fascia is installed by snapping it in place onto the switch rail posts.
Figure 18. Rail Posts for Fascia Installation
2.3.5

Reversing the Fan Airflow

This section details the procedure to reverse the airflow on the switch fan assembly from the default port-side exhaust position.
Tools Required
ESD wrist strap
#0 Phillips screwdriver
Procedure
NOTE
Before removing the cover from the switch, first disconnect the power cord(s).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 41
Intel® Omni-Path Fabric Switches Hardware
1. Loosen the 4 thumbscrews from the switch cover. There are 3 screws on the top, and 5 screws on the back. Lift the cover on a 45-degree angle, then pull to remove.
Figure 19. Switch Cover Thumbscrews (Top View)
®
Omni-Path Fabric—Installation
Intel
2. Holding the fan assembly on the sides lift straight up and rotate the fan assembly 180 degrees as shown in the following figure.
Figure 20. Fan Assembly Rotation
3. Lower the fan assembly straight down. The airflow direction indicators should now face towards the rear of the chassis as shown in the following figure.
NOTE
Make certain to provide strain relief for the wire assembly.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 42 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 21. Wire Assembly Strain Relief and Airflow Direction Indicators
4. Reattach the switch cover.
2.3.6

Installing the Q7 Management Module

This section details the procedure to install the Q7 Management Module into an existing externally-managed edge switch.
Figure 22. Q7 Management Module
Tools Required
ESD wrist strap
#0 Phillips screwdriver
Regular screwdriver
Procedure
NOTE
Before removing the cover from the switch, first disconnect the power cord(s).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 43
Intel® Omni-Path Fabric Switches Hardware
1. Loosen the 4 thumbscrews from the switch cover. There are 3 screws on the top, and 5 screws on the back. Lift the cover on a 45-degree angle, then pull to remove.
Figure 23. Switch Cover Thumbscrews (Top View)
2. Holding the module at approximately a 45-degree angle, slide the connector into module slot. Make certain to line up the notch on the module and the connector (see following figure).
®
Omni-Path Fabric—Installation
Intel
Figure 24. Installing the Q7 Module
3. Once the module is seated properly in the connector, secure screw to the chassis using a regular screwdriver.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 44 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 25. Q7 Module Mounting Screw
4. Reattach the switch cover.
NOTE
For information on upgrading Q7 board controller firmware (BCFW) and BIOS, refer to
Q7 BIOS Update on page 86.
2.4
2.4.1
2.4.2

Intel® Omni-Path Director Class Switch 100 Series Installation

Mounting Hardware Kit Contents

The Mounting Kit contains:
Support Shelf (1)
Baffles (2)
#10 Mounting screws
#10 Mounting nuts
M6 Mounting Screws

Unpacking the Switch

NOTE
Set aside packaging materials in the event that the switch requires return shipment.
The following images and descriptions give a high-level overview for unpacking a Director Switch (24-slot model in this example).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 45
Intel® Omni-Path Fabric Switches Hardware
Figure 26. Unpacking the Switch (1)
Intel® Omni-Path Fabric—Installation
1. Remove the banding.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 46 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 27. Unpacking the Switch (2)
2. Remove the lid.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 47
Intel® Omni-Path Fabric Switches Hardware
Figure 28. Unpacking the Switch (3)
®
Omni-Path Fabric—Installation
Intel
3. Remove the accessory kit box and white packing form. The accessory kit contains required hardware, power cables, documentation, etc.
Figure 29. Unpacking the Switch (4)
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 48 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 30. Unpacking the Switch (5)
4. Lift the box up out of the base.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 49
Intel® Omni-Path Fabric Switches Hardware
Figure 31. Unpacking the Switch (6)
®
Omni-Path Fabric—Installation
Intel
5. Remove the baffle boxes.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 50 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 32. Unpacking the Switch (7)
2.4.3
2.4.3.1
6. Remove the plastic bag, then remove the support shelf from the top of the switch.

Installation Tasks

The following section details rack mounting instructions for the 6-slot and 24-slot Intel® Omni-Path Director Class Switch 100 Series. The installations are very similar; the majority of steps apply to both switches. Where applicable, any specifics for either the 6-slot or 24-slot Director are called out.
Installing the Support Shelf
1. Measure the inside distance between the front and middle rails on the cabinet (27" in the following example).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 51
Intel® Omni-Path Fabric Switches Hardware
Figure 33. Measure the Rack
2. Before installing into the cabinet, preset the adjustable rails on the baffle base, leaving the screws loose to the dimension measured on the cabinet (27" in this example). In order to be able to fit the baffle base into the cabinet, set the rails to slightly less than the measured size (i.e., < 27"). See the following figure.
®
Omni-Path Fabric—Installation
Intel
Figure 34. Preset the Adjustable Rails
3. Install the fixture brackets to the cabinet at the desired height (see the following figure). The brackets support and help locate the installation of the baffle base. Remove and discard/recycle the brackets after baffle base has been installed.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 52 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 35. Install Fixture Brackets
4. Secure the shelf to the rack using M6 mounting screws, 3 per shelf rail (1 in the front and 2 in the back) as displayed in the following figure. Torque screws to 30 in-lb.
Figure 36. Secure Shelf to Rack
5. Attach the baffles to the top of each shelf rail using 10 mounting nuts, 5 per shelf rail as displayed in the following figure. Torque nuts to 30 in-lb.
NOTE
Make certain to tighten the adjustable rail screws (from Step 2) at this time.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 53
Intel® Omni-Path Fabric Switches Hardware
Figure 37. Secure the Baffles to Top of Shelf Rails
®
Omni-Path Fabric—Installation
Intel
6. Secure the top of the baffles to the rack rail as displayed in the following figure:
6-Slot Chassis: use 2 mounting nuts, 1 per rail.
24-Slot Chassis: use the lower of the 2 openings, 1 mounting nut per rail.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 54 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 38. Secure the Top of the Baffles to the Rack Rails
7. The completed Shelf assemblies for both the 6- and 24-slot switches are displayed in the following figure.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 55
Intel® Omni-Path Fabric Switches Hardware
Figure 39. Completed Shelf Assemblies
Intel® Omni-Path Fabric—Installation
2.4.3.2
Installing the Director Switch Chassis
WARNING
To avoid injury, do not lift a fully-loaded Director Class switch manually. Use a mechanized lift only.
1. Align the chassis with the rack.
2. Using a pallet jack, raise the unit until the bottom of the chassis is at (or slightly higher than) the surface of the support shelf installed in Steps 1 through 6. Lock the pallet jack wheels.
3. Use a group of people to push the chassis from the front to slide it off the pallet and on the support shelf. The chassis should be inserted into the rack until the rear of the chassis base is against the rear support bracket.
NOTE
If you are installing the cable management guides (see Installing the Optional
Cable Management Guide on page 59), install the cage nuts into the rack rails
before fully sliding the chassis into the rack.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 56 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 40. Slide the Chassis into the Rack
4. Fasten the chassis to the rack with M6 screws:
6-Slot Director: six (6).
24-Slot Director: twelve (12).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 57
Intel® Omni-Path Fabric Switches Hardware
Figure 41. Fasten the Chassis to the Rack
Intel® Omni-Path Fabric—Installation
5. The completed switch installations for both the 6- and 24-slot switches are displayed in the following figure.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 58 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 42. Completed Switch Installations
2.4.4

Installing the Optional Cable Management Guide

The optional Cable Management Guides are attached to the rack rails with M6 cage nuts and screws as shown in the following figure.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 59
Intel® Omni-Path Fabric Switches Hardware
Figure 43. Installing Cable Management
®
Omni-Path Fabric—Installation
Intel
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 60 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.4.5

Installing Director Class Modules

2.4.5.1
Spine, Leaf and Management Modules
NOTE
In order to maintain thermal integrity, all open Spine and Leafs slots must be populated with filler panels.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 61
Intel® Omni-Path Fabric Switches Hardware
®
Omni-Path Fabric—Installation
Intel
NOTE
When installing the Leaf Module, there is no "correct" orientation (i.e., no top or bottom).
These modules all have release arms that function in a similar way during installation. The Management Module (MM) is used here as the example.
1. Grasp and pull the release arms out and away from the module body as shown in the following figure.
Figure 44. Module Installation 1
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 62 Doc. No.: H76456, Rev.: 10.0
®
Installation—Intel
Omni-Path Fabric
Figure 45. Module Installation 2
2. Slide the module straight into an applicable open slot until it engages with the backplane connector.
CAUTION
Never push on the release arms when sliding the modules into the applicable slots. Always apply pressure to the middle of the module as shown below.
Figure 46. Module Installation 3
3. Push the release arms in until they lock in place.
NOTE
The 48-port leaf module release arms will lock in place correctly; however the module will protrude slightly from the leaf slot. This is normal.
4. Management Module only: Lower the MRL latch into the holes of the release arms. Doing this allows the MM to communicate with the SEEB module on the other side of chassis as shown in the following figure.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 63
Intel® Omni-Path Fabric Switches Hardware
Figure 47. Module Installation 4
5. Chassis management slot leaf only: If the leaf module is in one of the chassis management slots (i.e., L101 or L102) the mechanical release latch (MRL) must be moved in order to install the leaf. To move the MRL, loosen the thumbscrew attached to the Serial, Ethernet, Chassis EEPROM Board (SEEB) module, then pull down to release the MRL as shown in the following figure. This allows enough clearance for leaf module to be installed.
Figure 48. Module Installation 5
®
Omni-Path Fabric—Installation
Intel
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 64 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.4.5.2
2.4.5.3
Fans
1. Slide the fan module straight into an applicable open slot until it engages with the backplane connector.
Power Supplies
1. Loosen the thumbscrew until it disengages from the power supply unit.
2. Pull the handle down.
3. Slide the module straight into an applicable open slot until it engages with the backplane connector.
4. Push the handle back up and reconnect the thumbscrew.
2.4.6
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 65

Installing the Edge Switch Hot-Swap Power and Fan Modules

Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
NOTE
The following instructions are for the Intel® Omni-Path Edge Switch with hot swap power and fan modules (Part Numbers 100SWE48UFH and 100SWE48QFH).
2.4.6.1
Power Supply
1. Grasp the module by the handle and slide into an open slot until the unit engages with the connector.
Figure 49. Edge Power Supply Install 1
2. When fully inserted, the module sits flush with the chassis and the thumb tab is locked as shown.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 66 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 50. Edge Power Supply Install 2
2.4.6.2
Fans
1. Grasp the module by the handle and slide into an open slot until the unit engages with the connector.
Figure 51. Edge Fan Supply Install 1
2. When fully inserted, the module sits flush with the chassis and the thumb tab is locked as shown.
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 67
Intel® Omni-Path Fabric Switches Hardware
Figure 52. Edge Fan Supply Install 2
Intel® Omni-Path Fabric—Installation
2.5

Connect Equipment to the Ports and Power On the System

NOTE
Before connecting equipment, it is important to understand the locations of the serial and Ethernet ports on the Intel® Omni-Path Fabric series switches. For the edge switch, these are located on the switch port side.
Figure 53. Edge Switch Serial and Ethernet Ports
Figure 54. Director Switch Serial and Ethernet Ports
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 68 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
1. Connect a Category 5 or 6 (Cat 5/6) Ethernet cable to the RJ-45 connector(s) on the switch. Connect the other end of the Cat 5/6 to an OOB LAN workstation, another switch or a hub.
2. Connect the switch to host(s)/switch(es) using QSFP cables.
CAUTION
It is important to provide strain relief for the cable connector.
2.5.1

Director Class Switch Port and Slot Numbering

The graphic below details the leaf slot numbering schema for each Director switches.
Note:
Leaf slots 101 and 102 are referred to "Management" slots, meaning a leaf module must be present whenever a management module is installed in the corresponding slot on the front of the chassis.
Figure 55. Director Class Leaf Slot Numbering
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 69
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
2.5.2

Leaf Module Port Numbering

The physical module contains two 16-port leafs-A and B-stacked on top of each other. The physical port numbering is shown below.
Figure 56. Leaf Module Port Numbering
Figure 57. 48-Port Leaf Module Port Numbering
NOTE
When cabling the 48-port leaf module, make certain to firmly seat the cable into the port, otherwise only one of two ports may connect.
2.5.2.1
Chassis Viewer GUI Leaf Port Stats Screen
The Port Name column schema is a follows:
Lxxx = Chassis leaf module slot (i.e. L101 - L124 for the 24-slot Director)
A/B = Leaf A or B
Pxx = Physical port number (e.g., 01 - 16)
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 70 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Figure 58. Chassis Viewer GUI Leaf Port Stats Scree
2.5.3

Edge Switch Cabling Recommendations

To minimize complexity, the following figure details cabling recommendations for the 48-port edge switch:
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 71
Intel® Omni-Path Fabric Switches Hardware
Figure 59. 48-Port Edge Switch Cabling Recommendations
Intel® Omni-Path Fabric—Installation
2.5.3.1
Connecting Power
NOTE
For each power supply installed, a power cord must be installed in corresponding power inlet.
1. Provide strain relief for the power cable(s).
2. If necessary, replace the closeout over the switch fan side.
3. Connect the power cables to a power distribution unit (PDU) or a proper AC power outlet.
4. When the switch is plugged into an AC power outlet:
a. The system powers up.
b. The fans start.
c. The system performs a power-on self test.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 72 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
NOTE
For the Director switches, make certain the DC ON/OFF switch is illuminated. If it is not, press the button to supply power. The switch is located in the lower left corner on the chassis spine side.
5. The switch, power supply(s), and fan LEDs light up.
2.6
2.7
2.7.1

Setting Up the USB Console

1.
Download the driver file CDM v2.12.00 WHQL Certified.exe from http://
www.ftdichip.com/Drivers/VCP.htm.
2.
Open the FTDI Drivers Folder and Right click on the CDMv2.12 driver file and select Run As Administrator to install the necessary driver to access the USB serial console of the switch.
3. Open Putty, under Category, select Serial and enter the following parameters:
Serial Line to connect to: <Enter COM>
Speed: 115200
Data Bits: 8
Stop Bits: 1
Parity: None
Flow Control: None

Bringing Up the System For the First Time

Start-up Procedures

1. Power up the switch.
2. From its flash image on the management module, the switch begins its boot process.
3.
Verify the IP address with the CLI command showChassisIpAddr command. The system returns information similar to the following:
Chassis IP Address: 192.168.100.9 Net mask: 255.255.252.0
2.7.2

Changing the Switch IP Address and Default Gateway through the CLI

The CLI can be accessed two ways:
Using the switch serial port:
Edge switch (with optional management module): connects through the
USB serial port
Director series: connects through the USB port on the SEEB modules
Through Ethernet, using Telnet or SSH
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 73
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
NOTE
If using a serial port on a Director switch, make certain to connect to the SEEB serial port associated with the Management Module (MM) on the opposite side of the chassis.
2.7.2.1
2.7.2.2
Using the Serial Connection
1. When a serial port connection is established, the user will see a prompt. To change the chassis IP address type:
setChassisIpAddr -h <ipaddress> -m <netMask>
where -h <ipaddress> is the new IP address in dotted decimal format (that is,
xxx.xxx.xxx.xxx), and -m <netMask> is the new subnet mask in dotted
decimal format.
2. To change the switch default gateway IP address type:
setDefaultRoute -h <ipaddress>
where -h <ipaddress> is the new default gateway IP address in dotted decimal format.
3. The changes are effective immediately.
Using Ethernet
1. Access the switch with one of the following commands using the default IP address:
Telnet: telnet 192.168.100.9 SSH: ssh 192.168.100.9
2. The system prompts for a user name. In order to change the IP address and default gateway, the user must be logged in as the administrator. At the prompt type admin and press ENTER.
3.
The system prompts for a password. At the prompt type adminpass and press ENTER. The system responds with:
Welcome to the <SWITCH> CLI. Type 'list' for the list of commands.
4. To change the switch IP address type:
setChassisIpAddr -h <ipaddress> -m <netMask>
where -h <ipaddress> is the new IP address in dotted decimal format (that is,
xxx.xxx.xxx.xxx), and -m <netMask> is the new subnet mask in dotted
decimal format.
5. To change the switch default gateway IP address type:
setDefaultRoute -h <ipaddress>
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 74 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
where -h <ipaddress> is the new default gateway IP address in dotted decimal format.
6. The changes are effective immediately.
2.7.3

Updating the Management Module IP Addresses in a Director Class Redundant Management Configuration

Each management module (MM) must have a unique IP address that is different than the chassis IP address of the switch. A redundantly-managed switch will have multiple, unique IP addresses. The default IP addresses are:
Chassis: 192.168.100.9
Management Module(s): 192.168.100.10
It is necessary for each management module to have a unique IP address for the following reasons:
Unique IP addresses are used when sending syslog messages from a management module to a syslog server.
Gives the user the ability to ping each management module separately.
If the IP addresses are not unique, collisions occur, causing IP operations to fail.
NOTE
All management modules are individually tested and configured with the default IP address of 192.168.100.10. In order to avoid an address conflict, be certain to change the default IP address of each MM after inserting it into the chassis.
NOTE
Make sure when updating the MM IP, only 1 module is inserted into the chassis at a time. Repeat this procedure for the second MM.
To update the IP address on a management module, do the following:
1. Insert a MM into the chassis.
2. Connect the MM via the USB port to a serial console software (e.g., Putty or Teraterm).
3. Reset/apply power to the MM.
4.
When the system displays image1 or image2, press the SPACEBAR to interrupt the auto load sequence before the counter expires (within 5 seconds).
5. Enter moduleip <IP ADDRESS> in dotted decimal format and press ENTER.
6. To update the module netmask address enter netmask <ADDRESS> in hexadecimal format and press ENTER.
7. To update the module gateway address enter gateway <ADDRESS> in dotted decimal format and press ENTER.
8. At the boot prompt, reboot the MM.
2.8
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 75

Component LEDs

Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
2.8.1
Figure 60. Edge Switch LEDs
2.8.1.1
2.8.1.2

Edge Switch LEDs

RJ45 LEDs
The RJ45 connectors have two LEDs, Link and Speed. The Speed LED is Green when a 100 Mbps link is connected. The Link LED is Amber when an Ethernet link has been established, and blinking when the link is active.
Chassis Status LEDs
Status (STAT)
The status LED indicates one of the following conditions:
Steady Green: the module is operating normally.
Blinking Green: LED test state.
Off: module is in the removable state.
Attention (ATTN)
The Attention LED indicates one of the following conditions:
Off: the system functioning normally.
Steady Amber: the system requires some attention, which could indicate one of the following conditions:
— The switch temperature is at a warning level on the module.
— The switch silicon temperature is at a warning level (approximately 90 degrees
C).
— DC voltages on the board are slightly out of tolerance (12V Bulk, 5V, 3.3V and
1.8V are all monitored).
— The module can no longer function properly. The system will take the
appropriate actions to ensure that no damage is done to its components.
Blinking Amber (once every four seconds): LED test state.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 76 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.8.1.3
Ports, Fabric Manager (FM), Managed (M), and FM Switch LEDs
Figure 61. Ports, Fabric Manager (FM), Managed (M), and FM Switch LEDs
Ports
Each switch cable port has a Green link status LED that provide the following indications:
On: the logical link is up (port is in the Active state).
Off: the physical link is down (port is in the Down state).
Blink: traffic passing through the port
Fabric Manager (FM)
The FM LED indicates that the switch is recognized by the active Fabric Manager.
Managed (M)
The Managed (M) LED indicates that a Q7 management module is installed switch.
FM Switch
On an externally-managed switch, the FM Switch LED is used to restrict whether the end-node attached to a switch port is trusted to run a Fabric Manager, or to issue Fabric Manager-related packets and operations.
The user activates the LED using a small, straight instrument (e.g., a standard-size paper clip).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 77
Intel® Omni-Path Fabric Switches Hardware
Figure 62. Activating the FM Enable Switch LED
®
Omni-Path Fabric—Installation
Intel
The FM Enable Switch is used as follows:
NOTE
Actions take place approximately 1 second after releasing the button.
Pressing the button for 4 seconds does a "Restore Factory Defaults" and reboots the ASIC.
Pressing the button for 2 seconds (and then releasing) puts the "FM control button persistent state" in configuration mode.
— The FM enabled LED will blink "n" times, where "n" is the current state value
plus 1.
— Pressing (and releasing) the button once for less than 2 seconds advances the
FM control persistent state by 1 (round-robin), and then it returns to step 1 (i.e., blinking the FM enabled LED).
— Pressing the button for 2 seconds leaves the configuration state, and saves the
FM control button state persistently, restoring the FM enabled LED to the proper value.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 78 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.8.1.4
Figure 63. Power Supply LEDs
Power Supply LEDs
PSU1/PSU2:
Green indicates that DC power is normal.
Off indicates a DC power failure or no DC power is present.
2.8.1.5
Figure 64. Airflow Direction LED
2.8.2
Airflow Direction LED
Green indicates the fans are blowing from front to back.
Off indicates the fans are blowing from back to front.

Director Class LEDs

June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 79
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
2.8.2.1
Management Module LEDs
Figure 65. Management Module LEDs
Module Status (M_Stat)
The status LED indicates one of the following conditions:
Steady Green: the module is operating normally.
Blinking Green: LED test state.
Off: module is in the removable state.
Module Attention (M_Attn)
The Attention LED indicates one of the following conditions:
Off: the system is functioning normally.
Steady Amber: the system requires some attention, which could indicate one of the following conditions:
— The management module is at a warning level
— The microprocessor temperature is at a warning level
— DC voltages on the board are slightly out of tolerance
— The management module has taken a hot swap fault
Blinking Amber (once every four seconds): LED test state, which could indicate one of the following conditions:
— Bulk power is stable.
— The module DC-to-DC converter is enabled.
— The module is not removable.
Chassis Status and Attention (C_Stat, C_Attn)
The chassis status LED is Green when the system is functioning normally. The chassis status LED is Off when the system is not functioning normally.
Fault Tolerant (FT)
The Chassis Fault Tolerant (FT) LED indicates the following condition:
Green: Redundant Cooling, DC Power, and management all present (n+1)
Off: Redundant Cooling, DC Power, and management are not all present
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 80 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Service (SVC)
The Chassis Service (SVC) LED indicates the following condition:
Green: Chassis beacon is active
Off: Not selected
Master
Green: the module is acting as the master Management Module
Off: Management Module is not the Master
MRL Active
Green: the mechanical release latch (MRL) thumbscrew is secured to the management module, enabling communications with a mated management leaf module
Off: the mechanical release latch (MRL) thumbscrew is not engaged, disabling communications with a mated management leaf module
2.8.2.2
Leaf Module LEDs
Figure 66. Leaf Module LEDs
Port LEDs
Green: The external HFI link is in the Active Port State
Off: Link is not active
Module Status
The status LED indicates one of the following conditions:
Steady Green: the module is operating normally.
Blinking Green: LED test state.
Off: module is in the removable state.
Module Attention
The Attention LED indicates one of the following conditions:
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 81
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
Off: the system functioning normally.
Steady Amber: the system requires some attention, which could indicate one of the following conditions:
— The switch temperature is at a warning level
— The switch silicon temperature is at a warning level
— DC voltages on the board are slightly out of tolerance
2.8.2.3
Spine Module LEDs
Figure 67. Spine Module LEDs
Module Status (Stat)
The status LED indicates one of the following conditions:
Steady Green: the module is operating normally.
Blinking Green: LED test state.
Off: module is in the removable state.
Module Attention (Attn)
The Attention LED indicates one of the following conditions:
Off: the system functioning normally.
Steady Amber: the system requires some attention, which could indicate one of the following conditions:
— The switch temperature is at a warning level
— The switch silicon temperature is at a warning level
— DC voltages on the board are slightly out of tolerance
Blinking Amber (once every four seconds): LED test state.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 82 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
2.8.2.4
SEEB Module LEDs
Figure 68. SEEB Module LEDs
RJ45 LEDs
The RJ45 connectors have two LEDs, Link and Speed.The Speed LED is Green when a 100Mbps link is connected. The Link LED is Amber when an Ethernet link has been established, and blinking when the link is active.
MRL Active
Green: the mechanical release latch (MRL) thumbscrew is secured to the serial, Ethernet, chassis EEPROM board (SEEB), enabling communications with a mated management module
Off: the mechanical release latch (MRL) thumbscrew is not engaged, disabling communications with a mated management module
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 83
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Installation
2.8.2.5
Fans and Power Supplies LEDs
Fan LEDs
Figure 69. Director Class Fan LEDs
Fan LEDs indicate the following status(es):
Green indicates that the fan is functioning properly.
Amber indicates that the following warning condition exists:
— A single fan failure when the rotation speed is less than 4000 RPM or greater
than 10,950 RPM.
Green and Amber on indicates a possible problem, including:
— The fan tray is not responding to commands for configuration and
temperature-related operations.
— A fan is not responding to commands for temperature and speed related
operations.
— The fan speed has fallen below the minimum allowed RPM for a fan.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 84 Doc. No.: H76456, Rev.: 10.0
Installation—Intel® Omni-Path Fabric
Power Supply LEDs
Figure 70. Director Class Power Supply LEDs
Power LEDs indicate the following status(es):
AC: Green indicates that AC power is present.
DC: Green indicates that DC power is present.
FLT: Red indicates that DC power is off/interrupted (e.g., power cord not seated properly).
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 85
Intel® Omni-Path Fabric Switches Hardware

Appendix A Q7 BIOS Update

Intel® Omni-Path Fabric—Q7 BIOS Update
A.1
A.1.1
A.2
Before Getting Started
Ensure that the Q7 unit being updated is a QA3E. This unit can be found in either a managed Edge switch or a Director Class switch management module.
The Q7 update has two parts; board controller firmware (BCFW) and BIOS.
Two separate files are used to perform the update
The update files are bcfw: CGBCP429.q7.spkg for the firmware update and bios: QA3ER962.q7.spkg for the BIOS update
The BCFW and BIOS update requires switch firmware version 10.7 (or newer)
Prior to Upgrading
Make sure that the subnet manager (SM) is not running on the switch:
Execute the CLI command smcontrol to ensure that the embedded subnet manager (ESM) is not running on the switch. For example:
Edge-> smControl Subnet manager is not started.
Execute the CLI command who to ensure that no other CLI sessions are running during the update.
Displaying the Current Version
At the prompt, execute the commands bcfwversion and biosversion to view the current version(s). For example:
Edge-> bcfwversion Module Information -------­ Board Controller Firmware Version: CGBCP419 Edge-> biosversion Module Information -------­ System BIOS Version: QA3E915
If you have a QA31 card(s) (i.e., the BIOS version starts with QA31 instead of QA3E), replace this with a new Q7 card which contains the latest BIOS and BCFW.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 86 Doc. No.: H76456, Rev.: 10.0
Q7 BIOS Update—Intel® Omni-Path Fabric
If you have a QA3E card(s) that does not have BIOS version QA3ER962 and BCFW version CGBCP429, you will need to perform the update. Run the CLI command logShow. If BCWF and BIOS updates are required, the system returns messages similar to the following:
W|2018/01/24 14:57:18.480U: Thread "tSlot" (0x3b8a030) Osa: Module QA3E BC Version CGBCP420 is not CGBCP429; please run bcfwUpdate to fix.
W|2018/01/24 14:57:18.480U: Thread "tSlot" (0x3b8a030) Osa: Module QA3E BIOS Version QA3ER915 is not QA3ER962; please run biosUpdate to fix.
A.3
A.4
A.4.1
SFTP Server Configuration
This update requires a server with SFTP. The update files must be present on this server.
Updating the Edge Switches
Performing the Upgrade
1.
At the prompt enter sftp> put <version>.q7.spkg /firmware/bios.pkg
update and press Enter. If successful, the system displays information similar to
the following.
Uploading QA3ER962.q7.spkg to /firmware/bios.pkg QA3ER962.q7.spkg 100% 8193KB 8.0MB/s 00:01
You can check the progress of the update with the CLI command
showLastBiosRetCode -all. The system returns information similar to the
following:
Edge-> showLastBiosRetCode -all Module System BIOS Last Exit Code: 3: In Progress Edge-> showLastBiosRetCode -all Module System BIOS Last Exit Code: 0: Success
NOTE
DO NOT REBOOT AT THIS TIME. Wait until both the BCFW (via bcfwupdate) and the BIOS (via biosupdate; see Step 2) have been updated, then reboot.
Proceed to Step 2.
2.
At the prompt enter sftp> put <version>.bcfw.pkg/firmware/bcfw.pkg
update and press Enter. If successful, the system displays information similar to
the following.
sftp> put CGBCP429.q7.spkg/firmware/bcfw.pkg update Uploading CGBCP429.q7.spkg to /firmware/bcfw.pkg CGBCP429.q7.spkg 100% xxxKB X.xMB/s 00:00
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 87
Intel® Omni-Path Fabric Switches Hardware
Intel® Omni-Path Fabric—Q7 BIOS Update
You can check the progress of the update with the CLI command
showLastBcFwRetCode -all. The system returns information similar to the
following:
Edge-> showLastBcFwRetCode -all Module Board Controller Firmware Last Exit Code: 3: In Progress Edge-> showLastBcFwRetCode -all Module Board Controller Firmware Last Exit Code: 0: Success
Proceed to Step 3.
3. Reboot the switch.
4. Verify the version of the BCFW and BIOS by running the CLI commands
bcfwversion and biosversion. The system should return information similar
to the following:
Board Controller Firmware Version: CGBCP429 System BIOS Version: QA3ER962
A.5
A.5.1
Updating Director Class Switches
Update the BCFW and BIOS on Director Class switches (DCS) using the instructions in
Performing the Upgrade on page 88.
NOTE
The instructions upgrade both the Master and Slave MM.
Performing the Upgrade
1. At the master MM prompt, enter the following command to update the master and
slave: sftp> put QA3ER962.q7.spkg A:/firmware/bios.pkg
The system returns information similar to the following:
Uploading QA3ER962.q7.spkg to /firmware/A:/firmware/bios.pkg QA3ER962.q7.spkg 100% 8193KB 12.4MB/s 00:00
You can check the progress of the update with the CLI command
showLastBiosRetCode -all. The system returns information similar to the
following:
MasterM201-> showLastBiosRetCode -all M201 System BIOS Last Exit Code: 3: In Progress M202 System BIOS Last Exit Code: 0: Success MasterM201-> showLastBiosRetCode -all M201 System BIOS Last Exit Code: 0: Success M202 System BIOS Last Exit Code: 0: Success
NOTE
DO NOT REBOOT AT THIS TIME. Wait until both the BCFW (via bcfwupdate) and the BIOS (via biosupdate; see Step 2) have been updated, then reboot.
Proceed to Step 2.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 88 Doc. No.: H76456, Rev.: 10.0
Q7 BIOS Update—Intel® Omni-Path Fabric
2. At the master MM prompt, enter the following command to update the master and slave: sftp> put CGBCP429.q7.spkg A:/firmware/bcfw.pkg
The system returns information similar to the following:
Uploading CGBCP429.q7.spkg to /firmware/A:/firmware/bcfw.pkg CGBCP429.q7.spkg 100% 193KB 11.3MB/s 00:00
You can check the progress of the update with the CLI command
showLastBcFwRetCode -all. The system returns information similar to the
following:
MasterM201-> showLastBcFwRetCode -all M201 Board Controller Firmware Last Exit Code: 3: In Progress M202 Board Controller Firmware Last Exit Code: 3: In Progress MasterM201-> showLastBcFwRetCode -all M201 Board Controller Firmware Last Exit Code: 0: Success M202 Board Controller Firmware Last Exit Code: 0: Success
Proceed to Step 3.
3.
Reboot the switch with the command reboot -all.
4. Verify the version of the BCFW and BIOS by running the CLI commands bcfwversion and biosversion. The system should return information similar to the following:
Board Controller Firmware Version: CGBCP429 System BIOS Version: QA3ER962
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 89
Intel® Omni-Path Fabric Switches Hardware
®
Omni-Path Fabric—Director Switch Cable Management
Intel

Appendix B Director Switch Cable Management

This example explains cabling a Director switch in a raised floor cable management scenario.
1. Cable the Leaf modules from the top rows of the chassis down toward the floor.
2. Cable each Leaf from the bottom row up to the top row, and from the inside port (nearest the fan units) to the outside port.
Figure 71. Leaf Cabling
3. Use Velcro ties to wrap the cables for each leaf module.
4. Hang the tied cables over the cable management bracket.
Figure 72. Using Velcro Ties and the Cable Management Hanger
5. Tie off all of the cables (red band in Figure 73 below) before feeding the cables under the floor.
Intel® Omni-Path Fabric Switches Hardware Installation Guide June 2019 90 Doc. No.: H76456, Rev.: 10.0
Director Switch Cable Management—Intel
Figure 73. Tying Off All Cables
®
Omni-Path Fabric
June 2019 Installation Guide Doc. No.: H76456, Rev.: 10.0 91
Intel® Omni-Path Fabric Switches Hardware
Loading...