Oracle Exalogic EL X2-2, Exalogic EL X4-2, Exalogic EL X5-2, Exalogic EL X6-2, Exalogic EL X3-2 Multirack Cabling Manual

Download

Page 1

Oracle® Exalogic Elastic Cloud

Multirack Cabling Guide

Release EL X2-2, X3-2, X4-2, X5-2, and X6-2

E89294-01 August 2017

Page 2

Oracle Exalogic Elastic Cloud Multirack Cabling Guide, Release EL X2-2, X3-2, X4-2, X5-2, and X6-2

E89294-01

Primary Author: Salvador Esparza

Contributing Authors: Ashish Thomas, Anamika Mukherjee

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agencyspecific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.

Page 3

Preface

Audience xi

Revision History xi

Documentation Accessibility xii

List of Figures

1-1 Port Usage/Connection Groups for NM2-GW in Multirack Scenarios 1-2

1-2 Port Usage/Connection Groups for NM2-36P in Multirack Scenarios 1-3

5-1 Connectivity Between Two Exalogic Machine Full Racks in a Multirack Scenario 5-2

6-1 Connectivity Between Exalogic Machine Full Rack and Exadata Database Machine

Full Rack in a Multirack Scenario 6-2

A-1 Two Exalogic Eighth Racks A-2

A-2 Exalogic Eighth Rack and Exalogic Quarter Rack A-2

A-3 Exalogic Eighth Rack and Exalogic Half Rack A-3

A-4 Exalogic Eighth Rack and Exalogic Full Rack A-4

A-5 Two Exalogic Quarter Racks A-4

A-6 Exalogic Quarter Rack and Exalogic Half Rack A-5

A-7 Exalogic Quarter Rack and Exalogic Full Rack A-6

A-8 Two Exalogic Machine Half Racks A-6

A-9 Exalogic Half Rack and Exalogic Full Rack A-7

A-10 Two Exalogic Full Racks A-8

A-11 Eight Exalogic Full Racks A-9

B-1 Exalogic Eighth Rack and Exadata Quarter Rack B-2

B-2 Exalogic Eighth Rack and Exadata Half Rack B-2

B-3 Exalogic Eighth Rack and Exadata Full Rack B-3

B-4 Exalogic Quarter Rack and Exadata Quarter Rack B-4

B-5 Exalogic Quarter Rack and Exadata Half Rack B-5

B-6 Exalogic Quarter Rack and Exadata Full Rack B-5

B-7 Exalogic Half Rack and Exadata Quarter Rack B-6

B-8 Exalogic Half Rack and Exadata Quarter Rack B-7

B-9 Exalogic Half Rack and Exadata Full Rack B-7

B-10 Exalogic Full Rack and Exadata Quarter Rack B-8

B-11 Exalogic Full Rack and Exadata Half Rack B-9

B-12 Exalogic Full Rack and Exadata Full Rack B-9

vii

Page 8

viii

Page 9

List of Tables

5-1 Remove Factory-Installed Cross-Connects in Exalogic Machines 5-3

5-2 Connect 3-Meter Cables from Each Leaf Switch to the Spine Switch 5-4

5-3 Connect 5-Meter Cable One Rack's Leaf Switches to the Second Rack's Spine Switch 5-5

6-1 Factory-Installed Cross-Connects in Exalogic Machine to be Removed 6-3

6-2 Factory-Installed Cross-Connects in Exadata Database Machine to be Removed 6-4

6-3 Connect 3-Meter Cables from Each Leaf Switch to the Spine Switch in Exalogic

Machine Full Rack 6-5

6-4 Connect 3-Meter Cables from Each Leaf Switch in Exadata Database Machine to

Spine Switch in Exadata Database Machine 6-6

6-5 Connect 5-Meter Cables from Each Leaf Switch to the Spine Switch in Exalogic

Machine Full Rack 6-7

6-6 Connect 5-Meter Cables from Each Leaf Switch of Exadata Database Machine to the

Spine Switch of Exalogic Machine 6-8

6-7 Cross-Connections Between Exalogic Machine Quarter Rack and Exadata Database

Machine Quarter Rack 6-8

A-1 Exalogic-to-Exalogic Multirack Cabling Scenarios A-1

B-1 Exalogic-to-Exadata Multirack Cabling Scenarios B-1

Page 10

Page 11

Preface

This guide describes multirack cabling and configuration scenarios for the Oracle Exalogic machine, which is an integrated cloud machine comprising hardware and software. It includes information about hardware operations and site planning.

Note:

All hardware-related specifications in this guide are based on information for a typical deployment provided by Oracle at the time this guide was written. Oracle is not responsible for hardware problems that may result from following the average specifications in this document. For detailed information about preparing your site for multirack cabling and setup, consult your Oracle representative.

This preface contains the following sections:

• Audience

• Revision History

• Documentation Accessibility

• Related Documents

• Conventions

Audience

This guide is intended for Oracle Exalogic machine administrators and network administrators.

It is assumed that the readers of this manual have knowledge of the following:

• System administration concepts

• Hardware and networking concepts

Revision History

• E18481-08: February 2014

– Added Multirack Cabling Diagrams for Exalogic-to-Exalogic Scenarios .

– Added Multirack Cabling Diagrams for Exalogic-to-Exadata Scenarios .

Page 12

Documentation Accessibility

For information about Oracle's commitment to accessibility, visit the Oracle Accessibility Program website at

ctx=acc&id=docacc

Access to Oracle Support

Oracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visit

lookup?ctx=acc&id=info

if you are hearing impaired.

or visit

Conventions

The following text conventions are used in this document:

Convention Meaning

boldface Boldface type indicates graphical user interface elements associated

italic Italic type indicates book titles, emphasis, or placeholder variables for

monospace

with an action, or terms defined in text or the glossary.

which you supply particular values.

Monospace type indicates commands within a paragraph, URLs, code in examples, text that appears on the screen, or text that you enter.

xii

Page 13

Multirack Cabling Overview

This chapter describes the Exalogic Machine multirack cabling service. It contains the following sections:

• About the Multirack Cabling Service

• Multirack Cabling Scenarios

• Port Usage in InfiniBand Switches

• Multirack Cabling Tasks

1.1 About the Multirack Cabling Service

The multirack cabling service allows you to extend the capability of the Exalogic rack systems to beyond just a single rack. The multirack cabling service allows you to cable up to eight Exalogic machine full racks together, without requiring any external InfiniBand switches.

It is assumed that all racks are adjacent to each other, in a side-by-side configuration. If racks are not side-by-side, then you may need to purchase longer cables for the extended connections. Placing the racks in any configuration other than side-by-side may require you to purchase additional 10–meter cables. Cable lengths across computer room aisles can vary from site to site, and may increase the difficulty of determining the cable lengths that you may need to purchase.

1.2 Multirack Cabling Scenarios

You can use the multirack cabling service to connect an Exalogic machine with one or more Exalogic machines, or with one or more Exadata Database Machines. This guide discusses in detail how to perform multirack cabling tasks in the following scenarios:

• Connecting an Exalogic machine full rack to another Exalogic machine full rack, as

described in Connecting Two Exalogic Machines

• Connecting an Exalogic machine full rack to an Exadata Database Machine full

rack, as described in Connecting Exalogic and Exadata Machines

• Connecting an Exalogic machine quarter rack to an Exadata Database Machine

quarter rack, as described in Connecting Exalogic and Exadata Machines

Multirack Cabling Diagrams for Exalogic-to-Exalogic Scenarios contains network

connectivity diagrams for one Exalogic machine connected to another Exalogic machine, and for the scenario where eight Exalogic full racks are connected.

Multirack Cabling Diagrams for Exalogic-to-Exadata Scenarios contains network

connectivity diagrams for one Exalogic machine connected to one Exadata machine.

Any combination of up to eight racks of Exalogic machines or Exadata Database Machines can be connected together. If you need to connect a combination of more than eight Exalogic machine full racks or Exadata Database Machine full racks, you will require additional switches, such as an M9-648 port modular switch. A detailed

1-1

Page 14

discussion of such scenarios is beyond the scope of this guide. For more information, please contact Oracle Support.

For detailed information about the hardware components of the Exalogic machine and the rack layout, see the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

1.3 Port Usage in InfiniBand Switches

All configurations of the Exalogic machine (full, half, quarter, and eighth rack) include Sun Network QDR InfiniBand Gateway Switches. Only the Exalogic machine full rack and half rack configurations include a Sun Datacenter InfiniBand Switch 36.

This section discusses the port usage in the following InfiniBand switches in multirack configuration scenarios:

• Port Usage in NM2-GW

• Port Usage in NM2-36P

1.3.1 Port Usage in NM2-GW

Chapter 1

Port Usage in InfiniBand Switches

Figure 1-1 illustrates the port usage in each of the Sun Network QDR InfiniBand

Gateway Switches (NM2-GW).

Figure 1-1 Port Usage/Connection Groups for NM2-GW in Multirack Scenarios

1.3.2 Port Usage in NM2-36P

illustrates the port usage in the Sun Datacenter InfiniBand Switch 36 (NM2-36P), which is included in Exalogic machine full rack and half rack configurations.

1-2

Page 15

Chapter 1

Multirack Cabling Tasks

Figure 1-2 Port Usage/Connection Groups for NM2-36P in Multirack Scenarios

1.4 Multirack Cabling Tasks

Before cabling an Exalogic machine rack for multiple rack operation, you must know the system configuration and obtain all the prerequisite information for system installation. It is assumed that all Exalogic machine racks have been properly installed and verified. If you are connecting an Exalogic machine to an Exadata Database Machine, ensure that both Exalogic machine and Exadata Database Machine are installed and verified.

Procedures in this guide assume that the racks are next to each other.

Before you begin installing an Exalogic machine, perform pre–installation planning. Complete the checklists in the Oracle Exalogic Elastic Cloud Machine Owner's Guide. Verify the site space, access route, power, grounding, air flow, and network requirements that are provided in the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

Install Exalogic machine racks if the cabling service is performed as part of an initial installation. To install an Exalogic machine, follow the instructions in the Oracle Exalogic Elastic Cloud Machine Owner's Guide. Ensure that you read the entire guide before beginning installation, and follow all safety precautions.

Perform the following multirack cabling tasks, in order, after completing the installation process:

• Review safety considerations.

• Perform the preliminary tasks:

– Preparing a Multirack Cabling Plan

– Verifying Host Names and Addresses

– Performing a Cable Inventory

– Gathering the Required Tools

1-3

Page 16

Multirack Cabling Tasks

• Perform the cabling tasks:

– Shutting Down the Affected Exalogic Machines and Exadata Database

Machines

– Handling Inter-Switch Cables

– Extracting Spare InfiniBand Cables

– Unbundling the Inter-Switch Cables

– Applying Cable and Rack Labels

– Connecting the Cables

– Bundling the Cables

– Inspecting the Cables

– Verifying the Multirack Systems

Chapter 1

1-4

Page 17

Safety Considerations

This chapter describes important safety considerations that you should review before Exalogic machine cabling begins. The following topics are covered:

• Equipment Safety Precautions

• InfiniBand Cable Cautions

• Preparing the Cabling Site

• Observing the Safety Guidelines and Warnings

2.1 Equipment Safety Precautions

Observe the following precautions when cabling Exalogic machines. Otherwise, the equipment can be damaged or a malfunction can result.

• Do not block any ventilation holes.

• Do not place cables under the equipment or have cables stretched tight. Also, do

not disconnect a power cord from the equipment while its power is on.

• When disconnecting a LAN cable, you might not be able to reach the connector

lock with your fingers. If that is the case, press the connector lock with a slotted screwdriver to disconnect the cable. You could damage the system board if you force your fingers into the gap rather than using a slotted screwdriver.

• Do not place anything on top of the system or perform any work directly above it.

• Take measures to prevent static electricity from being generated at the installation

location. Note especially that static electricity is likely to be generated on carpets, and this could lead to a malfunction.

• Do not insert anything into any opening in the Exalogic machine. The Exalogic

machine contains high-voltage parts. If a metal object or another conductor is inserted into an opening in the machine, it can cause a short circuit that could cause fire, electric shock, or equipment damage.

• For details on maintenance of the servers, contact a certified service engineer.

2.2 InfiniBand Cable Cautions

Caution - Fiber optic InfiniBand cables with laser transceivers must be of a type Class I.

Caution - Do not allow any copper core InfiniBand cable to bend tighter than a 5 inch (127 mm) radius. Do not allow any optical InfiniBand cable to bend tighter than a 3.4 inch (85 mm) radius. Tight bends can damage the cable internally.

Caution - Do not use zip ties to bundle or support InfiniBand cables. The sharp edges of the ties can damage the cables internally. Use hook and loop straps.

2-1

Page 18

Caution - Do not allow any InfiniBand cable to experience extreme tension. Do not pull on an InfiniBand cable or allow it to drag. Unroll an InfiniBand cable for its length. Pulling on an InfiniBand cable can damage the cables internally.

Caution - Do not twist an InfiniBand cable more than one revolution for its entire length. Twisting an InfiniBand cable can damage the cable internally.

Caution - Do not route InfiniBand cables where they might be stepped upon or experience rolling loads. Such a crushing effect can damage the cable internally.

2.2.1 InfiniBand Cable Path Lengths

When handling a cable, consider the following:

• Bends in the cable path increase the required cable length. Bends in the cable

path are usually necessary, and each bend increases the total length. Rarely does a cable travel in a straight line from connector to connector.

• Bundling increases the required length of the cables. Bundling causes one or more

cables to follow a common path. However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many bends, one cable might experience only the inner radius of bends, while another cable might experience the outer radius of bends. In this situation, the differences of the required lengths of the cables are quite substantial.

Chapter 2

Preparing the Cabling Site

• If you are routing the InfiniBand cables under the floor, consider the height of the

raised floor.

2.2.2 InfiniBand Cable Bundling

When bundling InfiniBand cables in groups, use hook-and-loop straps to keep cables organized. If possible, use color-coordinated straps to help identify cables and their routing. The InfiniBand 4X copper conductor cables are fairly thick and heavy for their length. Consider the retention strength of the hook-and-loop straps when supporting cables. Bundle as few cables as reasonably possible.

You can bundle the cables using many hook-and-loop straps. Place the hook-and-loop straps as close together as reasonably possible, for example, every 1 foot (0.3 m). If a cable breaks free from a strap, the cable cannot fall far before it is retained by another strap. If the InfiniBand cables break free of their straps and fall free, the cables might break internally when they strike the floor or are jerked from tension.

2.2.3 Floor and Underfloor Delivery of InfiniBand Cables

The switch accepts InfiniBand cables from floor or underfloor delivery. Floor and underfloor delivery limits the tension in the InfiniBand cable to the weight of the cable for the rack height of the switch.

Note - Overhead cabling details are not included in this guide. For details on overhead cabling, contact a certified service engineer.

2.3 Preparing the Cabling Site

It is the customer's responsibility to prepare the site prior to multirack cabling of Exalogic machines. Refer to the Oracle Exalogic Elastic Cloud Machine Owner's

2-2

Page 19

Observing the Safety Guidelines and Warnings

Guide for any server or equipment you plan to install in the rack for additional site preparation requirements.

Before installing Exalogic machines, perform the following tasks:

1. Thoroughly clean and vacuum the area in preparation for the installation.

2. Note any problems or peculiarities at the site that require special equipment.

3. Verify that the installation site flooring has a high enough strength rating to

withstand the combined weight of an Exalogic machine and any installed equipment.

4. Install all necessary electrical equipment and ensure that sufficient power is

provided.

Refer to the Sun Rack II Power Distribution Units User's Guide for the power requirements of the Sun Rack II power distribution units (PDUs).

5. Ensure that the installation site provides adequate air conditioning.

6. Operate the air conditioning system for 48 hours to bring the room to the

appropriate temperature.

Chapter 2

2.4 Observing the Safety Guidelines and Warnings

Before installing the Exalogic machine, or installing any server or equipment into the rack, read the Important Safety Information for Sun Hardware Systems document included with the rack.

Observe all safety notices printed on the packaging and listed in the Sun Rack II

Safety and Compliance Guide and the Sun Rack II Power Distribution Units Users Guide. Go to http://docs.sun.com to download these guides.

Note:

Contact your service representative to confirm that your equipment is qualified for installation and use in an Exalogic machine. Oracle is not liable for any issues when you install or use non-qualified equipment.

2-3

Page 20

Chapter 2

Observing the Safety Guidelines and Warnings

2-4

Page 21

Preparing for Multirack Cabling

This chapter describes the preliminary tasks you must complete before you begin to cable multiple Exalogic machines together or a combination of Exalogic machines and Exadata Database Machines together, but after rack installation. The following topics are covered:

• Preparing a Multirack Cabling Plan

• Verifying Host Names and Addresses

• Performing a Cable Inventory

• Gathering the Required Tools

3.1 Preparing a Multirack Cabling Plan

Determine the number of Exalogic machine racks that you will connect.

See the appropriate section in this guide for a table that provides connection details for each supported multirack cabling option. Print the applicable chart, so you can have it readily available.

The system naming conventions are as follows. "R" indicates an Exalogic machine rack or an Exadata Database Machine rack. For example, R1 indicates rack 1. "U" indicates the unit height in the rack. For example, U24 indicates rack unit 24. U1 is located at the bottom of the rack, and U42 is at the top of the rack. For more information about Exalogic machine rack layout, see the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

3.2 Verifying Host Names and Addresses

Before installing the extended cabling, complete Exalogic machine network configuration to ensure that the networking is complete and the system will not have any addressing conflicts. For more information about network configuration for Exalogic machines, see the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

Verify that the installed Exalogic machine racks have been assigned unique host names and addresses. All compute nodes in all interconnected Exalogic machine racks must have unique addresses and names.

You should consider extending the IP addresses and naming conventions for interconnected racks of Exalogic machines. Extending the names to include a rack value such as name identify the source of diagnostic messages that may be displayed.

[Rack#]ComputeNode_01

allows you and Oracle repair personnel to

3-1

Page 22

3.3 Performing a Cable Inventory

Collect and inventory the cables you ordered, if any. You should inventory spare cables from the kit and the existing multirack cables. Each Exalogic machine includes a kit of spare parts.

Some spare cables are used for multirack cabling. The following table lists the spare cables available for the switch, and their length.

No additional InfiniBand cables need to be purchased when connecting an Exalogic machine full rack with another Exalogic machine full rack or with an Exadata Database Machine. The 5-meter cables provided in the spares kit with each Exalogic machine can be used for making the connections.

You can use some cables in the spare parts pool for multirack cabling.

Note:

Chapter 3

Performing a Cable Inventory

For Exalogic machine full racks, 16 QSFP passive copper cables of length 5 meter, with part number 594-6387-01, are included in the spare kit. These cables are intended for multirack cabling.

3.4 Gathering the Required Tools

Gather the following tools:

• Pen

• Notepad

• Flashlight

• Magnifying glass

• Label maker and labels (for cables, KVM console, and rack)

• Cable ties (hook-and-loop straps suggested)

3-2

Page 23

Multirack Cabling Tasks

This chapter describes multirack cabling tasks. The following topics are covered:

• Shutting Down the Affected Exalogic Machines and Exadata Database Machines

• Handling Inter-Switch Cables

• Extracting Spare InfiniBand Cables

• Unbundling the Inter-Switch Cables

• Applying Cable and Rack Labels

• Connecting the Cables

• Bundling the Cables

• Inspecting the Cables

• Verifying the Multirack Systems

4.1 Shutting Down the Affected Exalogic Machines and Exadata Database Machines

Before you can set up the multirack cabling, you must first shut down the affected Exalogic machines and Exadata Database Machines. Consider the operational impact of shutting down the systems, and take appropriate action. Refer to the Oracle Exalogic Elastic Cloud Machine Owner's Guide and the Exadata Database Machine documentation for shutdown procedures.

4.2 Handling Inter-Switch Cables

Caution - Be careful when you handle InfiniBand cables. InfiniBand cables are very fragile. The cable ends might break off if the ends are dropped, or mishandled in any way. Before you handle any InfiniBand cables, you should be aware of all handling instructions. Read and follow Important Guidelines When Handling InfiniBand Cables. This document is included in the packaging of any new cables purchased. Or, go to http://docs.sun.com to access this document.

4.3 Extracting Spare InfiniBand Cables

A spare cable bundle is provided with Exalogic machines. Extract the spare InfiniBand cables from the bundle. You need to use the InfiniBand cables only. Restore spare Ethernet cables to the original spares location. You do not use the Ethernet cables.

4-1

Page 24

4.4 Unbundling the Inter-Switch Cables

Unbundle the inter-switch cables. Remove the cable ties holding the cables together, such as hook-and-loop fasteners. Lay the cables out gently in collections of like length. Observe safety precautions. Be careful when handling the cables to avoid cable damage.

4.5 Applying Cable and Rack Labels

Re-label all inter-rack cables from leaf switches to spine switches. Re-label each cable to identify the correct locations at both cable ends, so that cables can be connected correctly during future maintenance. Label all cables using the following convention:

Rack[#]LeafSW[#]Port[#] -> Rack[#]SpineSW[#]Port[#]

When labeling racks, label all installed Exalogic machines (or a combination of Exalogic machine and Exadata Database Machine) according to the plan you prepared in the previous chapter.

Chapter 4

Unbundling the Inter-Switch Cables

4.6 Connecting the Cables

Procedures to connect cables are discussed in the following chapters:

• Connecting Two Exalogic Machines

• Connecting Exalogic and Exadata Machines

Tip:

Verify the required number of cables and cable lengths. Procedures in this guide assume that the racks are next to each other. Label the cables, on both ends. Use the labeling format described in the cabling tables located in the appendices at the end of this document.

4.7 Bundling the Cables

Bundle the cables together. Be sure and leave the last 18 inches free so that you can maneuver cables into position near the leaf switch or the spine switch. You should probably wait until each of the bundles is installed into its final position before you connect them together.

4.8 Inspecting the Cables

Perform a visual inspection to confirm that the leaf switches and spine switches in each rack (Exalogic machines and Exadata Database Machine) have the correct number of InfiniBand connections. You should also verify that the ports in the leaf switch are occupied. You should also verify that each spine switch has the correct number of cables coming into it.

4-2

Page 25

4.9 Verifying the Multirack Systems

After you have cabled and visually inspected the systems, start your Exalogic Machines and Exadata Database Machines, as appropriate. Refer to the Oracle Exalogic Elastic Cloud Machine Owner's Guide and the Exadata Database Machine documentation for start-up procedures. In addition, Oracle recommends that you verify the InfiniBand topology after connecting multiple racks. For information about verifying the InfiniBand fabric and discovering the topology, see the Oracle Exalogic Elastic Cloud Machine Owner's Guide and the Oracle Exadata Database Machine documentation.

Chapter 4

Verifying the Multirack Systems

4-3

Page 26

Chapter 4

Verifying the Multirack Systems

4-4

Page 27

Connecting Two Exalogic Machines

This chapter describes how to connect two Exalogic machines on the same InfiniBand fabric. It includes the following topics:

• Before You Begin

• Connecting Two Exalogic Machine Full Racks Together

5.1 Before You Begin

Before you start connecting multiple Exalogic machines together, ensure that you have completed all of the multirack cabling tasks described in Multirack Cabling Tasks.

5.2 Connecting Two Exalogic Machine Full Racks Together

This section discusses how to connect two Exalogic machine full racks together. It contains the following topics:

• Connectivity Diagram

• Connect Two Exalogic Machine Full Racks Together

5.2.1 Connectivity Diagram

Figure 5-1 shows how an Exalogic machine full rack is connected to another Exalogic

machine full rack.

5-1

Page 28

Connecting Two Exalogic Machine Full Racks Together

Figure 5-1 Connectivity Between Two Exalogic Machine Full Racks in a Multirack Scenario

Chapter 5

Sun Network QDR InfiniBand Gateway Switch (NM2-GW) 4x4 (four switches in each Exalogic machine full rack) cross-connects are recabled to their respective local spine switches Sun Datacenter InfiniBand Switch 36 (NM2-36P). See the red connecting lines in Figure 5-1.

In addition, each leaf switch is connected to the other rack's spine switch. For example, leaf switches in rack 1 are connected to the spine switch in rack 2. See the blue and green connecting lines in Figure 5-1.

5.2.2 Connect Two Exalogic Machine Full Racks Together

Connecting an Exalogic machine full rack to another Exalogic machine full rack involves the following steps:

1. Remove the Factory-Installed Cables

2. Remove Cables Between Gateway Switches

3. Connect Each Leaf Switch to the Spine Switch

4. Connect Each Leaf Switch in One Exalogic Rack to Spine Switches in the Other

Exalogic Rack

5-2

Page 29

5.2.2.1 Remove the Factory-Installed Cables

Table 5-1 lists the factory-installed Exalogic machine cross-connects (cables) that

should be removed. Remove these cables in each of the Exalogic machine full racks that will be connected on the same InfiniBand fabric.

Table 5-1 Remove Factory-Installed Cross-Connects in Exalogic Machines

Chapter 5

Connecting Two Exalogic Machine Full Racks Together

Cable Coming from Slot

U16 0A U26 0A 2 meter

U16 0B U26 0B 2 meter

U16 1A U26 1A 2 meter

U16 1B U26 1B 2 meter

U16 2A U24 2A 2 meter

U16 2B U24 2B 2 meter

U16 3A U24 3A 2 meter

U16 3B U24 3B 2 meter

U23 0A U24 0A 1 meter

U23 0B U24 0B 1 meter

U23 1A U24 1A 1 meter

U23 1B U24 1B 1 meter

U23 2A U26 2A 1 meter

Port Connecting to Slot Port Length

U23 2B U26 2B 1 meter

U23 3A U26 3A 1 meter

U23 3B U26 3B 1 meter

Note:

Remove the factory-installed cables as shown in Table 5-1 from both Exalogic machines.

5-3

Page 30

Connecting Two Exalogic Machine Full Racks Together

5.2.2.2 Remove Cables Between Gateway Switches

You must remove the cables connected in ports 4A, 4B, 5A, 5B, 6A, and 6B between gateway switches in both the racks.

5.2.2.3 Connect Each Leaf Switch to the Spine Switch

Table 5-2 lists the 3-meter cables that should be used to connect leaf switches (NM2-

GW) to the spine switch (NM2-36P) in the Exalogic machine rack. The 3-meter cables are bundled with the Exalogic machine. You should connect leaf switches to the spine switches in each of the Exalogic machine full racks that will be connected.

Table 5-2 Connect 3-Meter Cables from Each Leaf Switch to the Spine Switch

Chapter 5

Cable Coming from Slot

U26 0A U1 1A 3 meter

U26 0B U1 1B 3 meter

U26 1A U1 2A 3 meter

U26 1B U1 2B 3 meter

U24 0A U1 3A 3 meter

U24 0B U1 3B 3 meter

U24 1A U1 4A 3 meter

U24 1B U1 4B 3 meter

U23 0A U1 5A 3 meter

U23 0B U1 5B 3 meter

U23 1A U1 6A 3 meter

U23 1B U1 6B 3 meter

U16 0A U1 7A 3 meter

Port Connecting to Slot Port Length

U16 0B U1 7B 3 meter

U16 1A U1 8A 3 meter

U16 1B U1 8B 3 meter

Note:

Make the connections between leaf switches and spine switch as shown in

Table 5-2 for both Exalogic machines.

5-4

Page 31

Chapter 5

Connecting Two Exalogic Machine Full Racks Together

5.2.2.4 Connect Each Leaf Switch in One Exalogic Rack to Spine Switches in the Other Exalogic Rack

Table 5-3 lists the 5-meter cables (or longer) that should be used to connect leaf

switches (NM2-GW) from each Exalogic machine full rack to the spine switches (NM2-36P) in the other Exalogic machine full rack. The 5-meter cables are bundled with the Exalogic machine.

Table 5-3 Connect 5-Meter Cable One Rack's Leaf Switches to the Second Rack's Spine Switch

Cable Coming from Slot

R1-U26 2A R2-U1 9A 5 meter

R1-U26 2B R2-U1 9B 5 meter

R1-U26 3A R2-U1 10A 5 meter

R1-U26 3B R2-U1 10B 5 meter

R1-U24 2A R2-U1 11A 5 meter

R1-U24 2B R2-U1 11B 5 meter

R1-U24 3A R2-U1 12A 5 meter

R1-U24 3B R2-U1 12B 5 meter

R1-U23 2A R2-U1 13A 5 meter

R1-U23 2B R2-U1 13B 5 meter

R1-U23 3A R2-U1 14A 5 meter

R1-U23 3B R2-U1 14B 5 meter

R1-U16 2A R2-U1 15A 5 meter

Port Connecting to Slot Port Length

R1-U16 2B R2-U1 15B 5 meter

R1-U16 3A R2-U1 16A 5 meter

R1-U16 3B R2-U1 16B 5 meter

Note:

Connect leaf switches in each Exalogic machine full rack to the spine switch in the other Exalogic machine full rack, as shown in Table 5-3.

For network connectivity diagrams of other Exalogic-to-Exalogic scenarios, see

Multirack Cabling Diagrams for Exalogic-to-Exalogic Scenarios .

5-5

Page 32

Chapter 5

Connecting Two Exalogic Machine Full Racks Together

5-6

Page 33

Connecting Exalogic and Exadata Machines

This chapter describes how to connect Exalogic machine racks and Exadata Database Machine racks on the same InfiniBand fabric. It includes the following topics:

• Before You Begin

• Connecting an Exalogic Machine Full Rack and an Exadata Database Machine

Full Rack

• Connecting an Exalogic Machine Quarter Rack and an Exadata Database

Machine Quarter Rack

6.1 Before You Begin

Before you start connecting Exalogic machines and Exadata Database Machines, ensure that you have completed all of the multirack cabling tasks, as described in

Multirack Cabling Tasks.

6.2 Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

This section contains the following topics:

• Connectivity Diagram

• Connecting an Exalogic Machine Full Rack and an Exadata Database Machine

Full Rack

6.2.1 Connectivity Diagram

Figure 6-1 shows how an Exalogic machine full rack is connected to an Exadata

Database Machine full rack.

6-1

Page 34

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Figure 6-1 Connectivity Between Exalogic Machine Full Rack and Exadata Database Machine Full Rack in a Multirack Scenario

Chapter 6

Each Sun Network QDR InfiniBand Gateway Switch (NM2-GW) in the Exalogic machine full rack is connected to its local spine switch (Sun Datacenter InfiniBand Switch 36 (NM2-36P). See the red connecting lines in Figure 6-1 (EL Rack #1).

Then each leaf switch in the Exadata Database Machine full rack is connected to its local spine switch. See the red connecting lines in Figure 6-1 (ED X2-2 Rack #2).

Subsequently, each leaf switch in the Exalogic machine full rack is connected to the spine switch of the Exadata Database Machine. See the blue connecting lines in

Figure 6-1.

Finally, each leaf switch in the Exadata Database Machine is connected to the spine switch of the Exalogic machine. See the green connecting lines in Figure 6-1.

6.2.2 Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Connecting an Exalogic machine full rack to an Exadata Database Machine full rack involves the following steps:

1. Removing Factory-Installed Cables (Cross-Connects) in Exalogic Machine Full

Rack

6-2

Page 35

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

2. Removing Factory-Installed Cables (Cross-Connects) in Exadata Database

Machine Full Rack

3. Connecting Each Leaf Switch in Exalogic Machine Full Rack to Its Spine Switch

4. Connecting Each Leaf Switch in Exadata Database Machine Full Rack to Its Spine

Switch

5. Connecting Exalogic Leaf Switches to Spine Switch of Exadata Database Machine

6. Connecting Leaf Switches of Exadata Database Machine to Spine Switch of

Exalogic Machine

6.2.2.1 Removing Factory-Installed Cables (Cross-Connects) in Exalogic Machine Full Rack

Table 6-1 lists the factory-installed Exalogic machine cross-connects (cables) that

should be removed.

The system naming conventions are as follows. "R" indicates an Exalogic machine rack. For example, R1 indicates rack 1. "U" indicates the unit height in the rack. For example, U24 indicates rack unit 24. U1 is located at the bottom of the rack, and U42 is at the top of the rack. For more information about Exalogic machine rack layout, see the Oracle Exalogic Elastic Cloud Machine Owner's Guide.

Chapter 6

Table 6-1 Factory-Installed Cross-Connects in Exalogic Machine to be Removed

Cable Coming from Slot (Exalogic Machine)

R1-U16 0A R1-U26 0A 2 meter

R1-U16 0B R1-U26 0B 2 meter

R1-U16 1A R1-U26 1A 2 meter

R1-U16 1B R1-U26 1B 2 meter

R1-U16 2A R1-U24 2A 2 meter

R1-U16 2B R1-U24 2B 2 meter

R1-U16 3A R1-U24 3A 2 meter

R1-U16 3B R1-U24 3B 2 meter

R1-U23 0A R1-U24 0A 1 meter

R1-U23 0B R1-U24 0B 1 meter

R1-U23 1A R1-U24 1A 1 meter

Port Connecting to Slot

(Exalogic Machine)

Port Length

R1-U23 1B R1-U24 1B 1 meter

R1-U23 2A R1-U26 2A 1 meter

R1-U23 2B R1-U26 2B 1 meter

6-3

Page 36

Chapter 6

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Table 6-1 (Cont.) Factory-Installed Cross-Connects in Exalogic Machine to be Removed

Cable Coming from Slot (Exalogic Machine)

R1-U23 3A R1-U26 3A 1 meter

R1-U23 3B R1-U26 3B 1 meter

Port Connecting to Slot

(Exalogic Machine)

Port Length

Note:

R1 in Table 6-1 indicates an Exalogic machine full rack that is being connected to an Exadata Database Machine full rack

6.2.2.2 Removing Factory-Installed Cables (Cross-Connects) in Exadata Database Machine Full Rack

Table 6-2 lists the factory-installed Exadata Database Machine cross-connects

(cables) that should be removed.

Table 6-2 Factory-Installed Cross-Connects in Exadata Database Machine to be Removed

Cable Coming from Slot (Exadata Database Machine)

R2-U20 8A R2-U24 8A 2 meter

R2-U20 9A R2-U24 9B 2 meter

R2-U20 9B R2-U24 9A 2 meter

R2-U20 10A R2-U24 10B 2 meter

R2-U20 10B R2-U24 10A 2 meter

R2-U20 11A R2-U24 11B 2 meter

R2-U20 11B R2-U24 11A 2 meter

R2-U24 8B R2-U1 0B 3 meter

R2-U20 8B R2-U1 1B 3 meter

Port Connecting to Slot

(Exadata Database Machine)

Port Length

6-4

Page 37

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Note:

R2 in Table 6-2 indicates an Exadata Database Machine full rack that is being connected to an Exalogic machine full rack (R1).

6.2.2.3 Connecting Each Leaf Switch in Exalogic Machine Full Rack to Its Spine Switch

Table 6-3 lists the 3-meter cables that should be used to connect leaf switches (NM2-

GW) to the spine switch (NM2-36P) in the Exalogic machine rack. The 3-meter cables are bundled with the Exalogic machine. Only Exalogic machine full rack and half rack configurations include a spine switch in U1. The spine switch is not connected by default.

Table 6-3 Connect 3-Meter Cables from Each Leaf Switch to the Spine Switch in Exalogic Machine Full Rack

Chapter 6

Cable Coming from Slot (Exalogic Machine)

R1-U26 0A R1-U1 3A 3 meter

R1-U26 0B R1-U1 3B 3 meter

R1-U26 1A R1-U1 4A 3 meter

R1-U26 1B R1-U1 4B 3 meter

R1-U24 0A R1-U1 5A 3 meter

R1-U24 0B R1-U1 5B 3 meter

R1-U24 1A R1-U1 6A 3 meter

R1-U24 1B R1-U1 6B 3 meter

R1-U23 0A R1-U1 7A 3 meter

R1-U23 0B R1-U1 7B 3 meter

R1-U23 1A R1-U1 8A 3 meter

R1-U23 1B R1-U1 8B 3 meter

R1-U16 0A R1-U1 9A 3 meter

Port Connecting to Slot

(Exalogic Machine)

Port Length

R1-U16 0B R1-U1 9B 3 meter

R1-U16 1A R1-U1 10A 3 meter

R1-U16 1B R1-U1 10B 3 meter

6-5

Page 38

Chapter 6

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Note:

R1 in Table 6-3 indicates an Exalogic machine full rack that is being connected to an Exadata Database Machine full rack.

6.2.2.4 Connecting Each Leaf Switch in Exadata Database Machine Full Rack to Its Spine Switch

Table 6-4 lists the 3-meter cables that should be used to connect leaf switches to the

spine switch (NM2-36P) in the Exadata Database Machine full rack. The 3-meter cables are bundled with the Exadata Database Machine.

Table 6-4 Connect 3-Meter Cables from Each Leaf Switch in Exadata Database Machine to Spine Switch in Exadata Database Machine

Cable Coming from Slot (Exadata Database Machine)

R2-U20 8A R2-U1 11B 3 meter

R2-U20 8B R2-U1 12B 3 meter

R2-U20 9A R2-U1 13B 3 meter

R2-U20 9B R2-U1 14B 3 meter

R2-U24 8A R2-U1 11A 3 meter

R2-U24 8B R2-U1 12A 3 meter

R2-U24 9A R2-U1 13A 3 meter

R2-U24 9B R2-U1 14A 3 meter

Port Connecting to Slot

(Exadata Database Machine)

Port Length

Note:

R2 in Table 6-4 indicates an Exadata Database Machine full rack that is being connected to an Exalogic machine full rack (R1).

6.2.2.5 Connecting Exalogic Leaf Switches to Spine Switch of Exadata Database Machine

Table 6-5 lists the 5-meter cables that should be used to connect leaf switches (NM2-

GW) to the spine switch (NM2-36P) in the Exalogic machine rack. The 3-meter cables are bundled with the Exalogic machine. The spine switch is not connected by default.

6-6

Page 39

Chapter 6

Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

Table 6-5 Connect 5-Meter Cables from Each Leaf Switch to the Spine Switch in Exalogic Machine Full Rack

Cable Coming from Slot (Exalogic Machine)

R1-U26 2A R2-U1 3A 5 meter

R1-U26 2B R2-U1 3B 5 meter

R1-U26 3A R2-U1 4A 5 meter

R1-U26 3B R2-U1 4B 5 meter

R1-U24 2A R2-U1 5A 5 meter

R1-U24 2B R2-U1 5B 5 meter

R1-U24 3A R2-U1 6A 5 meter

R1-U24 3B R2-U1 6B 5 meter

R1-U23 2A R2-U1 7A 5 meter

R1-U23 2B R2-U1 7B 5 meter

R1-U23 3A R2-U1 8A 5 meter

R1-U23 3B R2-U1 8B 5 meter

R1-U16 2A R2-U1 9A 5 meter

Port Connecting to Slot

(Exadata Database Machine)

Port Length

R1-U16 2B R2-U1 9B 5 meter

R1-U16 3A R2-U1 10A 5 meter

R1-U16 3B R2-U1 10B 5 meter

Note:

R1 in Table 6-5 indicates an Exalogic machine full rack that is being connected to an Exadata Database Machine full rack (R2).

6.2.2.6 Connecting Leaf Switches of Exadata Database Machine to Spine Switch of Exalogic Machine

Table 6-6 lists the 5-meter cables that should be used to connect leaf switches of

Exadata Database Machine to the spine switch (NM2-36P) of the Exalogic machine full rack. The 5-meter cables are bundled with the Exadata Database Machine.

6-7

Page 40

Chapter 6

Connecting an Exalogic Machine Quarter Rack and an Exadata Database Machine Quarter Rack

Table 6-6 Connect 5-Meter Cables from Each Leaf Switch of Exadata Database Machine to the Spine Switch of Exalogic Machine

Cable Coming from Slot (Exadata Database Machine)

R2-U20 10A R1-U1 11B 5 meter

R2-U20 10B R1-U1 12B 5 meter

R2-U20 11A R1-U1 13B 5 meter

R2-U20 11B R1-U1 14B 5 meter

R2-U24 10A R1-U1 11A 5 meter

R2-U24 10B R1-U1 12A 5 meter

R2-U24 11A R1-U1 13A 5 meter

R2-U24 11B R1-U1 14A 5 meter

Port Connecting to Slot

(Exalogic Machine)

Port Length

Note:

R2 in Table 6-6 indicates an Exadata Database Machine full rack that is being connected to an Exalogic machine full rack (R1).

6.3 Connecting an Exalogic Machine Quarter Rack and an Exadata Database Machine Quarter Rack

In this multirack configuration scenario, all connections with both racks will remain as they are when delivered to the site. No connections will be moved or removed to make the cross connections. In addition, a spine switch in either rack is not required.

All cross-connections between the Exalogic machine quarter rack and the Exadata Database Machine quarter rack are made using the four Infiniband switch ports 1[a,b] and 2[a,b]. Table 6-7 describes the connections between the leaf switches of the two racks being connected.

Table 6-7 Cross-Connections Between Exalogic Machine Quarter Rack and Exadata Database Machine Quarter Rack

Exadata Database Machine Exalogic Machine

Leaf Switch Port Leaf Switch Port

R2-U24 1A R1-U23 1A

R2-U24 1B R1-U23 1B

6-8

Page 41

Chapter 6

Connecting an Exalogic Machine Quarter Rack and an Exadata Database Machine Quarter Rack

Table 6-7 (Cont.) Cross-Connections Between Exalogic Machine Quarter Rack and Exadata Database Machine Quarter Rack

Exadata Database Machine Exalogic Machine

Leaf Switch Port Leaf Switch Port

R2-U24 2A R1-U16 2A

R2-U24 2B R1-U16 2B

R2-U20 1A R1-U16 1A

R2-U20 1B R1-U16 1B

R2-U20 2A R1-U23 2A

R2-U20 2B R1-U23 2B

Note:

R2 in Table 6-7 indicates an Exadata Database Machine full rack that is being connected to an Exalogic machine full rack (R1).

In Table 6-7, ports 1A and 1B are horizontal connections across the two machine racks being connected (top leaf switch to top leaf switch). Ports 2A and 2B are the cross connections between the racks (top leaf switch to bottom leaf switch).

You require only 8 cables to connect an Exalogic machine quarter rack and an Exadata Database Machine. If these systems are side by side, you can use the 5meter Infiniband cables included in the spares kit to complete these connections.

For network connectivity diagrams of other Exalogic-to-Exadata scenarios, see

Multirack Cabling Diagrams for Exalogic-to-Exadata Scenarios .

6-9

Page 42

Chapter 6

Connecting an Exalogic Machine Quarter Rack and an Exadata Database Machine Quarter Rack

6-10

Page 43

Multirack Cabling Diagrams for Exalogic-toExalogic Scenarios

This appendix contains multirack cabling diagrams for the following scenarios:

• One Exalogic machine with another Exalogic machine, as listed in Table A-1.

• Eight Exalogic machines, as described in Connecting Eight Exalogic Full Racks.

Table A-1 Exalogic-to-Exalogic Multirack Cabling Scenarios

Exalogic Machine Rack Type

1/8 1/8 Connecting Two Exalogic Eighth Racks

1/8 1/4 Connecting an Exalogic Eighth Rack and Exalogic Quarter

1/8 1/2 Connecting an Exalogic Eighth Rack and Exalogic Half Rack

1/8 Full Connecting an Exalogic Eighth Rack and Exalogic Full Rack

1/4 1/4 Connecting Two Exalogic Quarter Racks

1/4 1/2 Connecting an Exalogic Quarter Rack and an Exalogic Half

1/4 Full Connecting an Exalogic Quarter Rack and an Exalogic Full

1/2 1/2 Connecting Two Exalogic Half Racks

1/2 Full Connecting an Exalogic Half Rack and an Exalogic Full Rack

Full Full Connecting Two Exalogic Full Racks

Exalogic Machine Rack Type

Section

Rack

A.1 Connecting Two Exalogic Eighth Racks

Figure A-1 illustrates how two Exalogic eighth racks are connected on the same

InfiniBand fabric.

A-1

Page 44

Connecting an Exalogic Eighth Rack and Exalogic Quarter Rack

Figure A-1 Two Exalogic Eighth Racks

Appendix A

In Figure A-1, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches).

A.2 Connecting an Exalogic Eighth Rack and Exalogic Quarter Rack

Figure A-2 illustrates how an Exalogic eighth rack is connected to an Exalogic quarter

rack on the same InfiniBand fabric.

Figure A-2 Exalogic Eighth Rack and Exalogic Quarter Rack

In Figure A-2, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches).

A-2

Page 45

Appendix A

Connecting an Exalogic Eighth Rack and Exalogic Half Rack

A.3 Connecting an Exalogic Eighth Rack and Exalogic Half Rack

Figure A-3 illustrates how an Exalogic eighth rack is connected to an Exalogic half

rack on the same InfiniBand fabric.

Figure A-3 Exalogic Eighth Rack and Exalogic Half Rack

In Figure A-3, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches)

A.4 Connecting an Exalogic Eighth Rack and Exalogic Full Rack

Figure A-4 illustrates how an Exalogic eighth rack is connected to an Exalogic full rack

on the same InfiniBand fabric.

A-3

Page 46

Connecting Two Exalogic Quarter Racks

Figure A-4 Exalogic Eighth Rack and Exalogic Full Rack

Appendix A

In Figure A-4, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches)

A.5 Connecting Two Exalogic Quarter Racks

Figure A-5 illustrates how two Exalogic quarter racks are connected on the same

InfiniBand fabric.

Figure A-5 Two Exalogic Quarter Racks

A-4

Page 47

Appendix A

Connecting an Exalogic Quarter Rack and an Exalogic Half Rack

In Figure A-5, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches).

A.6 Connecting an Exalogic Quarter Rack and an Exalogic Half Rack

Figure A-6 illustrates how an Exalogic quarter rack is connected to an Exalogic half

rack on the same InfiniBand fabric.

Figure A-6 Exalogic Quarter Rack and Exalogic Half Rack

In Figure A-6, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches).

A.7 Connecting an Exalogic Quarter Rack and an Exalogic Full Rack

Figure A-7 illustrates how an Exalogic quarter rack is connected to an Exalogic full

rack on the same InfiniBand fabric.

A-5

Page 48

Connecting Two Exalogic Half Racks

Figure A-7 Exalogic Quarter Rack and Exalogic Full Rack

Appendix A

In Figure A-7, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches)

A.8 Connecting Two Exalogic Half Racks

Figure A-8 illustrates how two Exalogic half racks are connected on the same

InfiniBand fabric.

Figure A-8 Two Exalogic Machine Half Racks

A-6

Page 49

Appendix A

Connecting an Exalogic Half Rack and an Exalogic Full Rack

In Figure A-8, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). NM2-36P represents a Sun Datacenter InfiniBand Switch 36 (spine switch).

In this configuration, four up-links from the gateway switch (leaf switch) are cabled to the Sun Datacenter InfiniBand Switch 36 (spine switches). Each leaf switch is connected to each spine switch.

A.9 Connecting an Exalogic Half Rack and an Exalogic Full Rack

Figure A-9 illustrates how an Exalogic half rack is connected to an Exalogic full rack on

the same InfiniBand fabric.

Figure A-9 Exalogic Half Rack and Exalogic Full Rack

In Figure A-9, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). NM2-36P represents a Sun Datacenter InfiniBand Switch 36 (spine switch).

In this configuration, six up-links from the gateway switch (leaf switch) are cabled to the Sun Datacenter InfiniBand Switch 36 (spine switches). Each leaf switch is connected to each spine switch.

A.10 Connecting Two Exalogic Full Racks

Figure A-10 illustrates how two Exalogic full racks are connected on the same

InfiniBand fabric.

A-7

Page 50

Figure A-10 Two Exalogic Full Racks

Appendix A

Connecting Eight Exalogic Full Racks

In Figure A-10, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). NM2-36P represents a Sun Datacenter InfiniBand Switch 36 (spine switch).

This scenario is explained in detail in Connecting Two Exalogic Machine Full Racks

Together. It is included in this appendix for the sake of completeness.

A.11 Connecting Eight Exalogic Full Racks

Figure A-11 illustrates how eight Exalogic full racks are connected on the same

InfiniBand fabric.

A-8

Page 51

Figure A-11 Eight Exalogic Full Racks

Appendix A

Connecting Eight Exalogic Full Racks

In Figure A-11, each leaf switch is connected to each spine switch (1 cable per leaf switch per spine switch).

A-9

Page 52

Appendix A

Connecting Eight Exalogic Full Racks

A-10

Page 53

Multirack Cabling Diagrams for Exalogic-toExadata Scenarios

This appendix contains multirack cabling diagrams for the Exalogic-to-Exadata scenarios listed in Table B-1.

Table B-1 Exalogic-to-Exadata Multirack Cabling Scenarios

Exalogic Machine Rack Type

1/8 1/4 Connecting an Exalogic Eighth Rack and an Exadata Quarter

1/8 1/2 Connecting an Exalogic Eighth Rack and an Exadata Half Rack

1/8 Full Connecting an Exalogic Eighth Rack and an Exadata Full Rack

1/4 1/4 Connecting an Exalogic Quarter Rack and an Exadata Quarter

1/4 1/2 Connecting an Exalogic Quarter Rack and an Exadata Half Rack

1/4 Full Connecting an Exalogic Quarter Rack and an Exadata Full Rack

1/2 1/4 Connecting an Exalogic Half Rack and an Exadata Quarter Rack

1/2 1/2 Connecting an Exalogic Half Rack and an Exadata Half Rack

1/2 Full Connecting an Exalogic Half Rack and an Exadata Full Rack

Full 1/4 Connecting an Exalogic Full Rack and an Exadata Quarter Rack

Full 1/2 Connecting an Exalogic Full Rack and an Exadata Half Rack

Exadata Database Machine Rack Type

Section

Rack

Full Full Connecting an Exalogic Full Rack and an Exadata Full Rack

B.1 Connecting an Exalogic Eighth Rack and an Exadata Quarter Rack

Figure B-1 illustrates how an Exalogic eighth rack and an Exadata quarter rack are

connected on the same InfiniBand fabric.

B-1

Page 54

Connecting an Exalogic Eighth Rack and an Exadata Half Rack

Figure B-1 Exalogic Eighth Rack and Exadata Quarter Rack

Appendix B

In Figure B-1, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.2 Connecting an Exalogic Eighth Rack and an Exadata Half Rack

Figure B-2 illustrates how an Exalogic eighth rack and an Exadata half rack are

connected on the same InfiniBand fabric.

Figure B-2 Exalogic Eighth Rack and Exadata Half Rack

In Figure B-2, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B-2

Page 55

Appendix B

Connecting an Exalogic Eighth Rack and an Exadata Full Rack

B.3 Connecting an Exalogic Eighth Rack and an Exadata Full Rack

Figure B-3 illustrates how an Exalogic eighth rack and an Exadata full rack are

connected on the same InfiniBand fabric.

Figure B-3 Exalogic Eighth Rack and Exadata Full Rack

In Figure B-3, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.4 Connecting an Exalogic Quarter Rack and an Exadata Quarter Rack

Figure B-4 illustrates how an Exalogic quarter rack and an Exadata quarter rack are

connected on the same InfiniBand fabric.

B-3

Page 56

Connecting an Exalogic Quarter Rack and an Exadata Half Rack

Figure B-4 Exalogic Quarter Rack and Exadata Quarter Rack

Appendix B

In Figure B-4, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

This scenario is explained in detail in Connecting an Exalogic Machine Quarter Rack

and an Exadata Database Machine Quarter Rack. It is included in this appendix for the

sake of completeness.

B.5 Connecting an Exalogic Quarter Rack and an Exadata Half Rack

Figure B-5 illustrates how an Exalogic quarter rack and an Exadata half rack are

connected on the same InfiniBand fabric.

B-4

Page 57

Connecting an Exalogic Quarter Rack and an Exadata Full Rack

Figure B-5 Exalogic Quarter Rack and Exadata Half Rack

Appendix B

In Figure B-5, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.6 Connecting an Exalogic Quarter Rack and an Exadata Full Rack

Figure B-6 illustrates how an Exalogic quarter rack and an Exadata full rack are

connected on the same InfiniBand fabric.

Figure B-6 Exalogic Quarter Rack and Exadata Full Rack

In Figure B-6, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B-5

Page 58

Connecting an Exalogic Half Rack and an Exadata Quarter Rack

B.7 Connecting an Exalogic Half Rack and an Exadata Quarter Rack

Figure B-7 illustrates how an Exalogic half rack and an Exadata quarter rack are

connected on the same InfiniBand fabric.

Figure B-7 Exalogic Half Rack and Exadata Quarter Rack

Appendix B

In Figure B-7, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.8 Connecting an Exalogic Half Rack and an Exadata Half Rack

Figure B-8 illustrates how an Exalogic half rack and an Exadata half rack are

connected on the same InfiniBand fabric.

B-6

Page 59

Connecting an Exalogic Half Rack and an Exadata Full Rack

Figure B-8 Exalogic Half Rack and Exadata Quarter Rack

Appendix B

In Figure B-8, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.9 Connecting an Exalogic Half Rack and an Exadata Full Rack

Figure B-9 illustrates how an Exalogic half rack and an Exadata full rack are

connected on the same InfiniBand fabric.

Figure B-9 Exalogic Half Rack and Exadata Full Rack

In Figure B-9, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B-7

Page 60

Connecting an Exalogic Full Rack and an Exadata Quarter Rack

B.10 Connecting an Exalogic Full Rack and an Exadata Quarter Rack

Figure B-10 illustrates how an Exalogic full rack and an Exadata quarter rack are

connected on the same InfiniBand fabric.

Figure B-10 Exalogic Full Rack and Exadata Quarter Rack

Appendix B

Figure B-10, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch.

NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.11 Connecting an Exalogic Full Rack and an Exadata Half Rack

Figure B-11 illustrates how an Exalogic full rack and an Exadata half rack are

connected on the same InfiniBand fabric.

B-8

Page 61

Connecting an Exalogic Full Rack and an Exadata Full Rack

Figure B-11 Exalogic Full Rack and Exadata Half Rack

Appendix B

In Figure B-11, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch. NM2-36P represents a Sun Datacenter InfiniBand Switch 36.

B.12 Connecting an Exalogic Full Rack and an Exadata Full Rack

Figure B-12 illustrates how an Exalogic full Rack and an Exadata full rack are

connected on the same InfiniBand fabric.

Figure B-12 Exalogic Full Rack and Exadata Full Rack

In Figure B-12, NM2-GW represents a Sun Network QDR InfiniBand Gateway Switch (leaf switches). NM2-36P represents a Sun Datacenter InfiniBand Switch 36 (spine switch).

B-9

Page 62

Appendix B

Connecting an Exalogic Full Rack and an Exadata Full Rack

In this configuration, the cabling within the Exadata machine is preserved. Only the traffic on the Exalogic machine and the Exalogic-to-Exadata traffic use the spine switches. The Exadata machine's local traffic is restricted to the leaf switches of the Exadata machine. InfiniBand subnets can be created to separate traffic without any recabling.

This scenario is explained in detail in Connecting an Exalogic Machine Quarter Rack

and an Exadata Database Machine Quarter Rack. It is included in this appendix for the

sake of completeness.

B-10

Page 63

Index

Multi-rack cabling

Multi-rack cabling (continued)

overview, 1-1

Index-1

Page 64

Index

Index-2

Oracle Exalogic EL X2-2, Exalogic EL X4-2, Exalogic EL X5-2, Exalogic EL X6-2, Exalogic EL X3-2 Multirack Cabling Manual

Contents

List of Figures

List of Tables

Preface

Audience

Revision History

Documentation Accessibility

Related Documents

Conventions

Multirack Cabling Overview

1.1 About the Multirack Cabling Service

1.2 Multirack Cabling Scenarios

1.3 Port Usage in InfiniBand Switches

1.3.1 Port Usage in NM2-GW

1.3.2 Port Usage in NM2-36P

1.4 Multirack Cabling Tasks

Safety Considerations

2.1 Equipment Safety Precautions

2.2 InfiniBand Cable Cautions

2.2.1 InfiniBand Cable Path Lengths

2.2.2 InfiniBand Cable Bundling

2.2.3 Floor and Underfloor Delivery of InfiniBand Cables

2.3 Preparing the Cabling Site

2.4 Observing the Safety Guidelines and Warnings

Preparing for Multirack Cabling

3.1 Preparing a Multirack Cabling Plan

3.2 Verifying Host Names and Addresses

3.3 Performing a Cable Inventory

3.4 Gathering the Required Tools

Multirack Cabling Tasks

4.1 Shutting Down the Affected Exalogic Machines and Exadata Database Machines

4.2 Handling Inter-Switch Cables

4.3 Extracting Spare InfiniBand Cables

4.4 Unbundling the Inter-Switch Cables

4.5 Applying Cable and Rack Labels

4.6 Connecting the Cables

4.7 Bundling the Cables

4.8 Inspecting the Cables

4.9 Verifying the Multirack Systems

Connecting Two Exalogic Machines

5.1 Before You Begin

5.2 Connecting Two Exalogic Machine Full Racks Together

5.2.1 Connectivity Diagram

5.2.2 Connect Two Exalogic Machine Full Racks Together

5.2.2.1 Remove the Factory-Installed Cables

5.2.2.2 Remove Cables Between Gateway Switches

5.2.2.3 Connect Each Leaf Switch to the Spine Switch

5.2.2.4 Connect Each Leaf Switch in One Exalogic Rack to Spine Switches in the Other Exalogic Rack

Connecting Exalogic and Exadata Machines

6.1 Before You Begin

6.2 Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

6.2.1 Connectivity Diagram

6.2.2 Connecting an Exalogic Machine Full Rack and an Exadata Database Machine Full Rack

6.2.2.1 Removing Factory-Installed Cables (Cross-Connects) in Exalogic Machine Full Rack

6.2.2.2 Removing Factory-Installed Cables (Cross-Connects) in Exadata Database Machine Full Rack

6.2.2.3 Connecting Each Leaf Switch in Exalogic Machine Full Rack to Its Spine Switch

6.2.2.4 Connecting Each Leaf Switch in Exadata Database Machine Full Rack to Its Spine Switch

6.2.2.5 Connecting Exalogic Leaf Switches to Spine Switch of Exadata Database Machine

6.2.2.6 Connecting Leaf Switches of Exadata Database Machine to Spine Switch of Exalogic Machine

6.3 Connecting an Exalogic Machine Quarter Rack and an Exadata Database Machine Quarter Rack

A.1 Connecting Two Exalogic Eighth Racks

A.2 Connecting an Exalogic Eighth Rack and Exalogic Quarter Rack

A.3 Connecting an Exalogic Eighth Rack and Exalogic Half Rack

A.4 Connecting an Exalogic Eighth Rack and Exalogic Full Rack

A.5 Connecting Two Exalogic Quarter Racks

A.6 Connecting an Exalogic Quarter Rack and an Exalogic Half Rack

A.7 Connecting an Exalogic Quarter Rack and an Exalogic Full Rack

A.8 Connecting Two Exalogic Half Racks

A.9 Connecting an Exalogic Half Rack and an Exalogic Full Rack

A.10 Connecting Two Exalogic Full Racks

A.11 Connecting Eight Exalogic Full Racks

B.1 Connecting an Exalogic Eighth Rack and an Exadata Quarter Rack

B.2 Connecting an Exalogic Eighth Rack and an Exadata Half Rack

B.3 Connecting an Exalogic Eighth Rack and an Exadata Full Rack

B.4 Connecting an Exalogic Quarter Rack and an Exadata Quarter Rack

B.5 Connecting an Exalogic Quarter Rack and an Exadata Half Rack

B.6 Connecting an Exalogic Quarter Rack and an Exadata Full Rack

B.7 Connecting an Exalogic Half Rack and an Exadata Quarter Rack

B.8 Connecting an Exalogic Half Rack and an Exadata Half Rack