TECHNICAL
WHITE PAPER
CONTENTS & NAVIGATION
2
Introduction
2
Software Considerations
2-3
Hardware Considerations
3-4
Operating System Conguration
4
Monitoring NVIDIA® NVLink™ Status
USING HP WORKSTATIONS WITH
NVIDIA® NVLINK™ FOR NVIDIA®
PRO GRAPHICS CARDS
1.
TECHNICAL WHITE PAPER
INTRODUCTION
CONTENTS &
NAVIGATION
2
Introduction
2
Software Considerations
2-3
Hardware Considerations
3-4
Operating System Conguration
4
Monitoring NVIDIA® NVLink™ Status
HP Workstations are fully equipped to allow users to take advantage of a new NVIDIA® technology – NVIDIA®
NVLink™. In a nutshell, NVIDIA® NVLink™ allows two matching graphics cards to communicate directly via
a high-speed interconnection eectively allowing the available memory of both GPUs to be combined.
This allows for applications and workloads designed with NVIDIA® NVLink™ in mind to benet from signicant
performance increases.
For more information, please review NVIDIA's documentation and examples:
• https://www.nvidia.com/en-us/design-visualization/nvlink-bridges/
SOFTWARE CONSIDERATIONS
To take advantage of NVIDIA® NVLink™ software must be explicitly designed to do so. Unlike some other
multi-GPU methods, NVIDIA® NVLink™ is dierent in that simply connecting NVIDIA® NVLink™ bridges does not
automatically benet a workload or use case; software must be designed with NVIDIA® NVLink™ in mind and
written in a way to take full advantage of it.
NVIDIA provides free SDKs and documentation which enable a software developer to include NVIDIA® NVLink™
support in their applications. Specically, NVIDIA® CUDA® SDK and NCCL are minimum requirements for
NVIDIA® NVLink™ enablement in software. These resources can be found here:
• https://developer.nvidia.com/cuda-toolkit
• https://docs.nvidia.com/cuda/
• https://developer.nvidia.com/nccl
2.
In addition to having compatible software, NVIDIA® NVLink™-capable hardware must be correctly installed
and congured to take advantage of NVIDIA® NVLink™.
HARDWARE CONSIDERATIONS
NVIDIA® NVLink™ is supported with HP Z8 G4, HP Z4 G4, and HP Z840 Workstations that have been properly
congured.
To prepare your system for use with NVIDIA® NVLink™ the following hardware must be installed in a
supported system:
• 2x NVIDIA® NVLink™-capable NVIDIA® cards, such as NVIDIA® RTX™ A6000 or NVIDIA® Quadro® GV100
• Please note, GPU generation mixing is not supported; the cards must be exact matches
(ie. 2x NVIDIA® Quadro® GV100)
•
NVIDIA® NVLink™ only works between exactly 2 cards (ie. 3-way NVIDIA® NVLink™
is not possible)
TECHNICAL WHITE PAPER
CONTENTS &
NAVIGATION
2
Introduction
2
Software Considerations
• NVIDIA® NVLink™ bridges that match the NVIDIA® cards installed
• Please note, with NVIDIA® Pascal & Volta GPU generations, 2 bridges are required
• Please note, bridge generation mixing is not supported; the bridges must be exact matches (ie. 2x NVIDIA®
NVLink™ bridges for NVIDIA® Quadro® GV100 connected to 2x NVIDIA® Quadro® GV100 cards)
• With the NVIDIA® RTX™ family of cards, only a single bridge is required & supported.
• NVIDIA® NVLink™ bridges are sold separately from HP systems and NVIDIA® cards, and are available
as a kit from HP
• 1x NVIDIA® card of the same GPU generation for display connectivity
• Please note, GPU generation mixing is not supported; all GPUs in the system must be of the same
generation
‒ The only exception is with NVIDIA® Quadro® GV100 (Volta), which is supported with NVIDIA® Quadro®
Pxxxx (Pascal) cards. However, NVIDIA® Quadro® GP100 (Pascal) is only supported with other Pascal
cards (NVIDIA® Quadro® Pxxxx)
• Since the NVIDIA® NVLink™-capable cards will be dedicated to compute purposes (more information
in Section 4), a 3rd card is needed to connect a display
Note: in some Linux® congurations, this 3rd card is optional
OPERATING SYSTEM CONFIGURATION
2-3
Hardware Considerations
3-4
Operating System Conguration
4
Monitoring NVIDIA® NVLink™ Status
WINDOWS
In Windows, both NVIDIA® NVLink™-capable NVIDIA® cards must be manually set to TCC mode to take full
advantage of NVIDIA® NVLink™. By default, all graphics cards are congured in WDDM mode. Please note that
all display outputs are disabled when a card is in TCC mode and will remain disabled until the card is manually
switched to WDDM mode. For more information on the dierence between these modes and why TCC mode
is required please review the following documentation:
• https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#unied-virtual-addressing
• https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#driver-model
To congure the NVIDIA® NVLink™-capable NVIDIA® cards in TCC mode:
1. Install the latest HP-provided NVIDIA graphics driver
• The latest drivers for your platform can be downloaded here: https://support.hp.com/us-en/drivers
2. Restart the computer and pres s F10 to enter F10 BIOS Setup, then navigate to the Advanced tab, then
to the Graphics Conguration link, and press Enter. From the drop-down box, select the slot in which the
non- NVIDIA® NVLink™-capable NVIDIA® card is located (ie. the card to be used for display purposes).
Press F10 again, then select the option to save changes and restart
3. Connect a display to the card in the slot selected in F10 Setup during the previous step
4. Using a command line interface, such as Command Prompt or PowerShell, locate the NVSMI tool
• The NVSMI tool can be used to toggle TCC mode and view other status information. For more information
on NVSMI, run nvidia-smi.exe -h or the following documentation: https://developer.nvidia.com/nvidia-
system-management-interface
• By default, the utility is located in C:\Program Files\NVIDIA Corporation\NVSMI
Note: with some newer driver packages, NVSMI can be invoked from anywhere without navigating
to a specic directory.
5. Run nvidia-smi.exe -l and take note of the ID numbers for both of the NVIDIA® NVLink™-capable
NVIDIA® cards
• At this point, you may also notice that both NVIDIA® NVLink™-capable NVIDIA® cards and the displaydedicated 3rd card should all be identied as being in WDDM mode
6. Run nvidia-smi.exe -i # -dm TCC for each NVIDIA® NVLink™-capable NVIDIA® card, replacing # with
the correct ID numbers (for example, 0, 1, etc.) obtained in the previous step, one at a time
Note: Ensure that nothing is connected to any of the display connectors on the NVIDIA® NVLink™-capable
cards or TCC mode will not enable correctly
Warning: Do attempt to set the display-dedicated card (3rd card) to TCC mode as you may lose the ability
to output a display on that card
7. Restart the computer
3.
8. Run nvidia-smi.exe -l and verify that both NVIDIA® NVLink™-capable NVIDIA® cards are identied as being
in TCC mode
9.
At this point, your system is fully congured for NVIDIA® NVLink™ and is ready for NVIDIA® NVLink™-capable software!
TECHNICAL WHITE PAPER
CONTENTS &
NAVIGATION
2
Introduction
2
Software Considerations
2-3
Hardware Considerations
3-4
Operating System Conguration
4
Monitoring NVIDIA® NVLink™ Status
LINUX®
In Linux®, generally no special conguration is required to enable the OS to be ready for NVIDIA® NVLink™;
it simply depends on the graphics driver and the target application. In any case, the latest HP-provided NVIDIA
graphics driver is required for NVIDIA® NVLink™ to function properly. The latest drivers for your platform can
be downloaded here: https://support.hp.com/us-en/drivers.
NVIDIA® cards properly enabled by NVIDIA® NVLink™ hardware and drivers will reect their supported status
by oering an additional SLI Mosaic mode through the nvidia-settings GUI application. Most Linux® desktop
managers will benet immediately from the multiple-card layouts (rows and columns) possible through
X server conguration. See NVIDIA® graphic driver documentation and manual pages for nvidia-settings
for more details.
MONITORING NVIDIA®
NVLINK™ STATUS
Included with the NVIDIA graphics driver for both Windows and Linux®, the NVSMI tool (introduced in Section
4, above) can be used to monitor NVIDIA® NVLink™ status at any time using the following commands:
• nvidia-smi.exe NVIDIA NVLink -s
• This command displays the current state of all possible NVIDIA® NVLink™ links
• Each bridge provides each card with 2 NVIDIA® NVLink™ links, so a total of 4 links per card will be listed
along with its status (either a speed rating or <inactive>)
‒ If <inactive> is listed, then either the bridge is missing/defective or the system is not correctly congured
• nvidia-smi.exe NVIDIA NVLink -c
• This command displays the current status of each link capability for each NVIDIA® NVLink™-capable
NVIDIA® card
• nvidia-smi.exe NVIDIA NVLink -h
• This command provides additional information on other options for controlling/monitoring NVIDIA®
NVLink™ using the NVSMI tool
4.
LET US HELP YOU CREATE AMAZING BUSINESS
SOLUTIONS TODAY
CONTACT US
© Copyr ight 2021 HP De velopme nt Compan y, L.P. The infor mation con tained he rein is subj ect to cha nge witho ut notice. T he only war ranties fo r HP produc ts and ser vices ar e set
forth i n the expr ess warra nty state ments acco mpanyin g such prod ucts and s ervice s. Nothin g herein sho uld be cons trued as con stituti ng an addi tional wa rranty. HP s hall not be
liabl e for techni cal or edi torial err ors or omis sions cont ained he rein.
NVID IA, Quad ro and NVLi nk are trad emarks a nd/or regi stered tra demark s of NVIDIA C orpora tion in the U. S. and othe r countri es.
4AA7-3070E NW, Februa ry 2021
Loading...