Mellanox BlueField SmartNIC operation manual

www.mellanox.com

Mellanox Technologies Confidential

Mellanox BlueField Software User Manual

Rev 1.1

Version 1.0.0.10521

Mellanox Technologies Confidential

Mellan ox Technolog ies 350 Oakmead Parkway Suite 100 Sunnyvale

, CA 94085

U.S

.A. www.mellanox.com Tel: (408) 970-3400 Fax: (408) 970-3403

Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®, CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®, GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellanox CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®

, Me llanox ScalableHPC®

, Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, NPS®, Open Ethernet logo, PhyX®, PlatformX®

, PSIPHY®, SiPhy®, StoreX®

, SwitchX®,

Tilera®

, Tilera logo

, TestX®

, TuneX®

, The Generation of Open Ethernet logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are regist er ed trademark s of Mellano x Technologi es, Ltd.

All ot her t rademarks are property of their respective owners. For the most u pdate d lis t of Me llanox trademark s, visit http://www.mellanox.com/page

/trademarks

NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVI DED BY MELL ANOX TECH NOLOGIES

“AS

-IS” WITH AL L FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS

. THE CUSTOMER

'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRO DUCT(S) AND/OR THE SYST EM U SING IT. THEREFORE, MEL LANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE O R WARRANT THAT THE PRODUCTS WI LL OPERATE WITH THE HIGH EST Q UALIT Y

. AN Y EXPRESS OR IMPLIED WARRANTIES

, INCLUDING

, B UT NOT L IMIT ED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED

. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND

(INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES

; LOSS OF USE, DATA, OR P R OFI TS ;

OR BUSINESS INTERRUPTION

)

HOWEVER CAUSED AND ON ANY T HEORY OF LIABILITY, WHETHER IN CONTRACT, STR ICT LIA BILI TY

, OR T ORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Doc #: MLNX-15-53886

Mellanox Technologies Confidential

Rev 1.1

Document Revision History .................................................................................................................. 6

About this Manual ................................................................................................................................. 7

1 BlueField Software Overview ....................................................................................................... 10

1.1 Debug Tools ......................................................................................................................... 10

1.2 BlueField Adapter/SmartNIC ................................................................................................ 11

1.3 BlueField-based Storage Appliance ..................................................................................... 11

1.4 BlueField Architecture .......................................................................................................... 11

2 Installation and Initialization ........................................................................................................ 13

2.1 Unpacking BlueField Software Distribution .......................................................................... 13

2.2 Upgrading Boot Software ..................................................................................................... 14

2.2.1 BFB File Overview .................................................................................................. 14

2.2.2 BlueField Boot Process .......................................................................................... 16

2.2.3 The mlxbf-bootctl Program ..................................................................................... 16

2.2.4 Upgrading the Bootloader ....................................................................................... 17

2.2.5 Updating the Boot Partition ..................................................................................... 18

2.2.6 Safely Updating with a BMC ................................................................................... 18

2.2.7 Safely Updating Boot Software from the Arm Cores .............................................. 18

2.2.8 Changing the Linux Kernel or Root File System..................................................... 19

2.3 Building Arm Trusted Firmware ............................................................................................ 19

2.3.1 Building ATF Images .............................................................................................. 20

2.3.2 Trusted Board Boot ................................................................................................. 21

2.4 Building UEFI (EDK2) ........................................................................................................... 21

2.4.1 Customizable Build Options.................................................................................... 22

2.4.2 Exporting Variables ................................................................................................. 22

2.5 Building Poky Initramfs ......................................................................................................... 23

2.5.1 Basic Quick Start to Build Poky Initramfs ............................................................... 23

2.5.2 Variables ................................................................................................................. 24

2.5.3 Downloading Upstream Yocto and Building SDK ................................................... 25

2.6 Using Yocto as a Cross-compilation SDK and Root F i lesy st em Generator ......................... 25

2.7 RShim Host Driver ................................................................................................................ 26

2.7.1 Building and Installing RShim Host Driver .............................................................. 26

2.7.2 Loading Modules .................................................................................................... 26

2.7.3 Device Files ............................................................................................................ 26

2.7.4 FAQ – What if USB and PCIe Access are Enabled? ............................................. 27

2.7.5 Multiple Board Support ........................................................................................... 27

2.7.6 Permanently Changing the MAC Address of the A rm Side .................................... 27

Mellanox Technologies Confidential

Rev 1.1

2.8 OpenOCD on BlueField ........................................................................................................ 28

3 Programming ................................................................................................................................. 29

4 UEFI Boot Option Management ................................................................................................... 30

4.1 Boot Option ........................................................................................................................... 30

4.2 List UEFI Boot Options ......................................................................................................... 30

4.3 Creating, Deleting, and Modifying UEFI Boot Opti on ........................................................... 32

5 Installing Popular Linux Distributions on BlueField ................................................................. 34

5.1 Installing CentOS 7.x Distribution ......................................................................................... 34

5.1.1 Requirements ......................................................................................................... 34

5.1.2 Host Machine Setup ............................................................................................... 34

5.1.3 Basic Yocto Installation ........................................................................................... 35

5.1.4 PXE Boot ................................................................................................................ 35

5.1.5 CentOS Installation ................................................................................................. 36

5.1.6 Post-installation ...................................................................................................... 36

5.1.7 Building a New bluefield_dd ISO Image ................................................................. 37

5.1.8 PXE Boot Flow ........................................................................................................ 38

5.1.9 Non-PXE Boot Flow ................................................................................................ 38

5.1.10 Installation Troubleshooting and FAQ .................................................................... 39

5.2 Running RedHat on BlueField .............................................................................................. 39

5.2.1 Provisioning ConnectX Firmware ........................................................................... 40

5.2.2 Managing the Driver Disk ....................................................................................... 41

5.3 Installing the Reference Yocto Distribution .......................................................................... 41

6 Troubleshooting and FAQ ............................................................................................................ 43

Mellanox Technologies Confidential

Rev 1.1

List of Figures

Figure 1 - Interfaces on BlueField .......................................................................................................... 11

Figure 2 - BlueField Bootstream ............................................................................................................ 14

Figure 3 - Basic BlueField Boot Flow ...................................................................................................... 16

Figure 4 - BlueField High Level Hardware View ..................................................................................... 20

Mellanox Technologies Confidential

Rev 1.1

Document Revision History

Rev 1.1 – September 04, 2018

Added section “2.5 Building Poky Initramfs” and moved other subsections underneath it

Rev 1.0 – August 10, 2018

First release

Mellanox Technologies Confidential

Rev 1.1

About this Manual

Welcome to the BlueField™ SW User Manual. This document provides information that explains the BlueField Software Distribution (BSD) and how to develop and/or customize applications, system software, and file system images for the BlueField platform.

Audience

This document is intended for software devel opers and DevOps engineers interested in creating and/or customizing software applications and system software for the Mellanox

BlueField SoC platform.

Document Conventions

The following lists conventions used in this document.

NOTE: Identifies important information that contains helpful suggestions.

CAUTION: Alerts you to the risk of personal injury, system damage, or loss of data.

WARNING: Warns you that failure to take or avoid a specific action might result in

personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment.

WARNING: Warns you that failure to take or avoid a specific action might result in personal injury or a malfunction of the hardware or software. Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment.

Common Abbreviations and Acronyms

Abbreviation / Acronym Whole Word / Description

ATF Arm Trusted Firmware BFB BlueField™ bootstream BSD BlueField Software Distribution eMMC Embedded Multi-Media Card ESP EFI system partition FS File system

Mellanox Technologies Confidential

Rev 1.1

Abbreviation / Acronym Whole Word / Description

FW Firmware GDB GNU Debugger GPT GUID partition HW Hardware IB InfiniBand KGDB Kernel debugger KGDBOC Kernel debugger over con s ol e NIC Network interface card OCD On-chip debugger OVS Open vSwitch PCIe PCI Express or Peripheral Component Interconnect Express SoC System on chip SW Software UEFI Unified Extensible Firmware Interface UPVS UEFI Persistent Variable Store VPI Virtual Protocol Interconnect

1 BlueField Software Overview

It is recommended to upgrade your BlueField product to the latest software and firmware versions available in order to enjoy the latest features and bug fixes.

Mellanox® provides software which enables users to fully utilize the BlueField™ SoC and enjoy the rich feature-set it provides. Usi ng the BlueField software packages, users are able to:

• Quickly and easily boot an initial Linux image on your development board

• Port existing applications to and develop new applications for BlueField

• Patch, configure, rebuild, update or otherwise customize your image

• Debug, profile, and tune their development system using open source development tools

taking advantage of the diverse and vibrant Arm ecosystem.

The BlueField family of SoC devices combines an array of 64-bit Armv8 A72 cores coupled

with the ConnectX

interconnect. Standard Linux distributions run on the Arm cores allowing common open source development tools to be used. Developers should find the programming environment familiar and intuitive which in turn allows them to quickly and efficiently design, implement and verify their control-plane and data-plane applications.

BlueField SW ships with the Mellanox BlueField Reference Platform. Bluefield SW is a reference Linux distribution based on the Yocto Poky distribution and extended to include the Mellanox OFED stack for Arm and a Linux kernel which supports NVMe-oF. This SW distribution is capable of running all customer-based Linux applications seamlessly. Yocto also provides an SDK which contains an extremely flexible cross-build environment allowing software targeted for the BlueField SoC to build on virtually any x86 server running any Linux distribution.

The following are other software elemen ts delivered with BlueField SoC:

• Arm Trusted Firmware (ATF) for BlueField

• UEFI for BlueField

• OpenBMC for BMC (ASPEED 2500) found on development board

• Hardware Diagnostics

• Mellanox OFED stack

• Mellanox MFT

1.1 Debug Tools

BlueField SoC includes hardware support for the Arm DS5 suite as well as CoreSight™ debug & trace. As such, a wide range of commercial off-the-shel f Arm debug tools should work seamlessly with BlueField.

The BlueField SoC also supports the ubiquitous GDB.

Mellanox Technologies Confidential

Rev 1.1

Linux

Cryp todev

(TRN G + pub/

private.)

Misc Platform

Drivers

(SMBus , GPIO ,

Mgmt net)

PCIe Root Complex

NVDIMM

Driver Stack

Arm v8

NVMe Driver

ConnectX Driver

(mlx 5)

IB/RD MA Core

NVMe

-oF T arge t

iSCSI

Kernel

User

ASAP^2OFE D Ve rbsCryptodev

OVSiSER

OpenS SL

SP DK DPDK

NVMe-oF

Tar get

NVDIMM

Mgmt

NVML

Lib raries

Arm v8

UEFI

BSP

ARM Trusted Firmware (ATF)

BSP

1.2 BlueField Adapter/SmartNIC

The BlueField SmartNIC is shipped wit h the BlueField Software Distribution (BSD) pre-installed. The BlueField adapter Arm execution environment has the capability of being fully isolated from the x86 host and uses a dedicated network management interface (separate from the x86 host’s management interface). The Arm cores can run the Open vSwitch Database (OVSDB) or other virtual switches to create a secure solution for bare metal provisioning.

The software package also includes support for DPDK as well as applications for encryption.

1.3 BlueField-based Storage Appliance

Mellanox® BlueField™ Software provides the foundation for building a JBOF (Just a Bunch of Flash) storage system including NVMe-oF target software, PCIe switch support, NVDIMM-N support, and NVMe disk hot-swap support.

BlueField SW allows enabling Mellanox ConnectX signature offload, erasure coding of fload, iSER, Storage Spaces Direct, and more.

offload such as RDMA/RoCE, T10 DIF

1.4 BlueField Architecture

The BlueField architecture is a combination of two preexisting standard off-the-shelf components, Arm AArch64 processors, and Mellanox ConnectX-5 network controller, each with its own rich software ecosystem. As such, almost any of the programmer-visible software interfaces in BlueField come from existing standard interfaces for the respective components.

The Arm related interfaces (including tho se r elated to the boot process, PCIe connectivity, and cryptographic operation acceleration) are standard Linux on Arm interfaces. These interfaces are enabled by drivers and low level code provided by Mellanox as part of the BlueField software delivered and upstreamed t o respective open source projects, such as Linux.

The ConnectX-5 network controller-related interfaces (including those for Ethernet and InfiniBand connectivity, RDMA and RoCE, and storage and network operation acceleration)

Figure 1 - Interfaces on BlueField

Mellanox Technologies Confidential

Rev 1.1

are identical to the interfaces that support ConnectX-5 standalone network controller cards. These interfaces take advantage of the Mellanox OFED software stack and InfiniBand verbsbased interfaces to support software.

Mellanox Technologies Confidential

Rev 1.1

2 Installation and Initialization

Disclaimer: This section is preliminary and subject to change. Please consult the README files in the BlueField™ Software Distribution (BSD) for the most updated content.

The BSD consists of the following images:

• BlueField-<bluefield_version>_install-bluewhale.bfb – installation BFB file for the

BlueWhale development board

• BlueField-<bluefield_version>_install-smartnic_MBF1M332A.bfb – installation BFB

file for the 8-core version of the BlueField SmartNIC

• BlueField-<bluefield_version>_install-smartnic_MBF1L332A.bfb – installation BFB file

for the 4-core version of the BlueField SmartNIC

• BlueField-<bluefield_version>.tar.xz – base BSD tarball which contains all the BlueField

specific source code as well as sample binary images

• core-image-full-BlueField-<bluefield_version>.<yocto_version>.tar.xz – base reference

BlueField full root filesystem tar archive

• core-image-initramfs-BlueField-<bluefield_version>.<yocto_version>.cpio.xz – base ref-

erence BlueField initramfs cpio image

• poky-glibc-x86_64-core-image-full-sdk-aarch64-toolchain-BlueField-<bluefield_

version>.<yocto_version>.sh – Yocto-produced SDK in a self-installing script. This contains all cross-build tools and utilities to allow building an image targeted for the BlueField platform on an x86 server running Linux.

2.1 Unpacking BlueField Software Distribution

To unpack the BSD, run:

$ tar xvf BlueField-<bluefield_version>.tar.xz

This unpacks to a BlueField-<bluefield_version>/ subdirectory containing the following toplevel hierarchy:

• bin – contains tools to manage the process of installing runtime software. For example,

the mlx-mkbfb tool to generate the BlueField boot stream files used to provide initial boot images.

• boot – contains the boot loader binaries built from the provided sources for each of Mel-

lanox

’s BlueField devices. The “bl*” files in a device’s dedicated folder are taken from the Arm Trusted Firmware (ATF) and each represents a different boot phase; note that the file bl1.bin corresponds to the boot ROM burned into the SoC itself. The *.fd file is the Unified Extensible Firmware Interface (UEFI) boot image. The *.bfb file is a generated BlueField boot stream file which includes all the above boot loader components.

• distro – contains information pertinent to different Linux distributions. For example, the

distro/yocto directory contains the “meta-bluefield” layer used to build a BlueField-targeted version of the standard Yocto/Poky meta-distribution.

Mellanox Technologies Confidential

Rev 1.1

BL1

R es e t

BL2

BL31 BL33

ACPI Table

Name

Boot Image

Path

Kernel

Arguments

• sample – contains sample images which can be used to boot up a BlueField

™

chip to a Linux bash prompt, to either validate that hardware is working correctly, or for experimentation. See the README and README.install files in that directory for more information.

• src – contains patches for various components (e.g. ATF, UEFI, and Linux), as well as

complete sources for Linux drivers and user-space components authored by Mellanox

and not yet upstreamed.

The BSD contains numerous README files in t he aforementioned directories which provide more information. These README files must be consulted particularly when upgrading your BSD release as they contain impor tant release-specific information.

The following README files in particular are important to consult for possible release specific information:

• sample/README.install

• sample/README

• distro/yocto/README-bluefield

• src/atf/README

• src/atf/README-bfb

• distro/rhel/pxeboot/README

2.2 Upgrading Boot Software

This section describes how to use the BlueFi eld alternate boot partition support feature to safely upgrade the boot software. We give the requirements that motivate the feature and explain the software interfaces that are used to configure it.

2.2.1 BFB File Overview

Figure 2 - BlueField Bootstream

Mellanox Technologies Confidential

Rev 1.1

The default BlueField bootstream (BFB) shown above is a standard boot BFB that is stored on the embedded Multi-Media Card (eMMC) as can be seen by the boot path that points to a GUID partition (GPT) on the eMMC device. That path is a normal UEFI boot path and it will be stored in the UPVS (UEFI Persistent Var iable Store) EEPROM as a side effect of booting with this BFB. That is, if you use the mlxbf-bootctl utility to write this BFB to the eMMC boot partition, the SoC chip will read it via the boot FIFO on the RShim device by default on the next reboot.

™

BFB files can be useful for many things such as in stalling new software on a BlueField SoC. For example, the installation BFB for BlueField platforms normally contains an initramfs file in the BFB chain. Using the initramfs (and Linux kernel Image also found in the BFB) you can do things like set the boot partition on the eMMC using mlx-bootctl or flash new HCA firmware using MFT utilities. You can also install a full root file system on the eMMC while running out of the initramfs.

The types of files possible in a BFB are listed below.

Table 1 - Types of BlueField Bootstream Files

Filename Description ID Read By

bl2-cert Trusted Boot Firmware BL2 certificate 6 BL1 bl2 Trusted Boot Firmware BL2 1 BL1 trusted-key-cert Trusted key certificate 7 BL2 bl31-key-cert EL3 Runtime Firmware BL3-1 key certificate 9 BL2 bl31-cert EL3 Runtime Firmware BL3-1 certificate 13 BL2 bl31 EL3 Runtime Firmware BL3-1 3 BL2 bl32-key-cert Secure Payload BL3-2 (Trusted OS) key certificate 10 BL2 bl32-cert Secure Payload BL3-2 (Trusted OS) certificate 14 BL2 bl32 Secure Payload BL3-2 (Trusted OS) 4 BL2 bl33-key-cert Non-Trusted Firmware BL3-3 key certificate 11 BL2 bl33-cert Non-Trusted Firmware BL3-3 certificate 15 BL2 bl33 Non-Trusted Firmware BL3-3 5 BL2 boot-acpi Name of the ACPI table 55 UEFI boot-dtb Name of the DTB file 56 UEFI boot-desc Default boot menu item description 57 UEFI boot-path Boot image path 58 UEFI boot-args Arguments for boot image 59 UEFI boot-timeout Boot menu timeout 60 UEFI uefi-tests Specify what UEFI tests to run 61 UEFI image Boot image 62 UEFI initramfs In-memory filesystem 63 UEFI

Mellanox Technologies Confidential

Rev 1.1

BL1 Boot ROM

BL2 BL31

UEFI

(BL33)

Linux kernel

ini tra mf s

ARM Trusted Firmware

(ATF)

reset

2.2.2 BlueField Boot Process

Before explaining the implementation of the solution, the BlueField boot process needs to be expanded upon.

Figure 3 - Basic BlueField Boot Flow

™

The BlueField

boot flow is comprised of 4 main phases:

• Hardware loads Arm Trusted Firmware (ATF)

• ATF loads UEFI—together ATF and UEFI make up the booter software

• UEFI loads the operating system, such as the Linux kernel

• The operating system loads applications and user data

When booting from eMMC, these stages make use o f two different types of storage within the eMMC part:

• ATF and UEFI are loaded from a special area known as an eMMC boot partition. Data

from a boot partition is automatically strea med from the eMMC device to the eMMC controller under hardware control during the initial boot-up. Each eMMC device has two boot partitions, and the partition which is used to stream the boot data is chosen by a nonvolatile configuration register in the eMMC.

• The operating system, applications, and user data co me from the remainder of the chip,

known as the user area. This area is accessed via block-size reads and writes, done by a device driver or similar software routin e.

2.2.3 The mlxbf-bootctl Program

Access to all the boot partition management is done via a program packaged with the BlueField software called “bootctl”. The binary is shipped as part of the Yocto image (under /sbin) and the sources are shipped in the “src” directory in the BlueField Runtime Distribution. A simple “make” command builds the utility.

The syntax of bootctl is as follows:

Mellanox Technologies Confidential

Rev 1.1

Flag functionality breakdown:

• --device – use a device other than the default /dev /mmcblk0

• --bootstream – write the specified bootstream to the alternate partition of the device. This

queries the base device (e.g. /dev/mmcblk0) for the alternate partition, and uses that information to open the appropriate boot partition device (e.g. /dev/mmcblk0boot0).

• --overwrite-current (used with “--bootstream”) – overwrite the current boot partition in-

stead of the alternate one. (Not recommended!)

• --output (used with “--bootstream”) – specify a file to which to write the boot partition

data (creating it if necessary), rather than using an existing master device and deriving the boot partition device.

• --watchdog-swap – arrange to start the Arm watchdog timer with a countdown of the

specified number of seconds until it triggers; also, set the boot software so that it swaps the primary and alternate partitions at the next reset.

• --nowatchdog-swap – ensure that after the next reset, no watchdog is started, and no

swapping of boot partitions occurs.

2.2.4 Upgrading the Bootloader

In most deployments, the Arm cores of BlueField™ are expected to obtain their software stack from an on-board eMMC device. Even in environments where the final OS kernel is not kept on eMMC—for instance, systems which boot over a network—the initial booter code still comes from the eMMC.

Most software stacks need to be modified or upgraded in their lifetime. Ideally, the user is able to install the new software version on their BlueField system, test it, and then fall back to a previous version if the new one does not work. In some environments, it is important that this fallback operation happen aut omati cally since there may be no physical access to the system. In others, there may be an external agent, such as a service processor, which could manage the process.

In order to satisfy the requests listed above, the following must be performed:

1. Provision two software partitions on the eMMC, 0 and 1. At any given time, one area

must be designated the primary partition, and the other the backup partition. The primary partition is the one booted on the next reboot or reset.

2. Allow software running on the Arm cores to declare that the primary partit ion is now the

backup partition, and vice versa. (For the remainder of this section, this operation is referred to as “swapping the partitions” even though only the pointer is modified, and the data on the partitions does not move.)

3. Allow an external agen t, such as a service processor, to swap the primary and backup par-

titions.

4. Allow software running on the Arm cores to reboot the system, while activating an up-

grade watchdog timer. If the upgrade watchdog expires (due to the new image being broken, invalid, or corrupt), the system automatically reboots after swapping the primary and backup partitions.

Mellanox Technologies Confidential

Rev 1.1

2.2.5 Updating the Boot Partition

To update the boot partition on the Arm cores, let us a ssume to have a new bootstream file called “bootstream.new” which we would like to install and validate. To update to the bootstream, run:

# mlxbf-bootctl --bootstream bootstream.new --swap # reboot

This writes the new bootstream to the alternate boot partition, swaps alternate and primary so that the new bootstream is used on the next reboot, and then reboots to use it. (You may also use “--overwrite-current” instead of “--swap”, which just overwrites the current boot partition. But this is not recommended as there is no easy way to recover if the new booter code does not bring the system up.)

2.2.6 Safely Updating with a BMC

The Arm cores notify the BMC prior to the reboot that an upgrade is about to happen. Software running on the BMC can then be implemented to watch the Arm cores after reboot. If after some time the BMC does not detect the Arm cores come up properly, it can use its USB debug connection to the Arm cores to properly reset the Arm cores. It first sets a suitable mode bit that the Arm booter responds to by switching the primary and alternating boot partitions as part of resetting into its original state.

2.2.7 Safely Updating Boot Software from the Arm Cores

Without a BMC, the Arm watchdog may be used to achieve similar results. If something goes wrong on the next reboot and the system does not come up properly, it will reboot and return to the original configuration. In this case, the user may run:

# mlxbf-bootctl --bootstream bootstream.new --swap --watchdog-swap 60 # reboot

With these commands, the user reboots the system, and, if it hangs for 60 seconds or more, the watchdog fires and resets the chip, t he booter swaps the partitions back again to the way they were before, and the system reboots back with the original boot partition data. Similarly, if the system comes up but panics and r eset s, the booter will again swap the boot partition back to the way it was before.

The user must ensure that Linux after the reboot is configured to boot up with the “sbsa_gwdt” driver enabled. This is the Server Base System Architecture (SBSA) Generic WatchDog Timer. As soon as the driver is loaded, it begins refreshing the watchdog and preventing it from firing, which allows the system to finish booting up safely. In the example above, 60 seconds are allowed from system reset until the Linux watchdog kernel d r iver is loaded. At that point, the user’s application may open /dev/watchdog explicitly, and the application would then become responsible for refreshing the watchdog frequently enough to keep the system from rebooting.

For documentation on the Linux watchdog subsystem, see the Linux watchdog documentation (e.g. https://www.kernel.org/doc/Documentation/watchdog/watchdog-api.txt

To disable the watchdog completely, for example, run:

# echo V > /dev/watchdog

The user may select to incorporate other featur es of the Arm generic watchdog into their application code using the programming API as well.

Mellanox Technologies Confidential

Rev 1.1

Once the system has booted up, in addition to disabling or reconfiguring the watchdog itself if the user desires, they must also clear the “swap on next reset” functionality from the booter by running:

# mlxbf-bootctl --nowatchdog-swap

Otherwise, next time the system is reset (via reboot, external reset, etc.) it assumes a failure or watchdog reset occurred and swaps the eMMC boot partition automatically.

The aforementioned steps can be done manually, or can be done automatically by software running in the newly-booted system.

2.2.8 Changing the Linux Kernel or Root File System

The solutions above simply update the boot partition to hold new boot loader software (ATF and UEFI). If the user wants to also provide a new kernel image and/or modify the root file system, the user should partition their eMMC into multiple partitions appropriately.

For example, the user may have a single FAT partition from which UEFI can read the kernel image file, but the new bootstream contains a UEFI bootpath pointing to an updated kernel image. Similarly, the user may have two Linux partitions, and their upgrade procedure would write a new filesystem into the “idle” Linux partition, then reboot with the bootstream holding kernel boot arguments which direct it to boot from the previously idle partition.

The details on how exactly to do this depend on the specifics of how and what needs to be upgraded for the specific application, but in principle any component of the system can be safely upgraded using this type of approach.

For more information, please refer to EDK2 user documentation on Github at:

https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-User-Documentation.

2.3 Building Arm Trusted Firmware

NOTE: While descriptions of Arm Trusted Firmware (ATF) are provided related to the BlueField™ platform; for general knowledge of what ATF is and how it works, please refer to ATF documents from Arm. The Arm Trusted Firmware User Guide located at “docs/user-guide.rst” in the ATF sources is a good place to start.

ATF is used in Armv8 systems for booting the chip and then providing secure interfaces. It implements various Arm interface standar ds like PSCI (Power State Coordination Interface), SMC (Secure Monitor Call) and TBBR (Trusted Board Boot Requirements). ATF is used as the primary bootloader to load UEFI (Unified Extensible Firmware Interface) on the BlueField platform.

Mellanox Technologies Confidential

Rev 1.1

AR M A72 Cores

Bo ot ROM

Bo ot FIFO

Secure Boot Registers

BOO T SRAM

boot parti tion

eMMC

RSHIM

• Lif ecy cle Fu ses

• Ke y Stor age

• MDK

BlueField SoC

EEP ROM (UPVS

)

SPI Fla sh

Figure 4 - BlueField High Level Hardware View

ATF has various bootloader stages when loading:

• BL1 – BL1 is stored in the on-chip boot ROM; it is executed when the primary core is

reset. Its main functionality is to do some initial architectural and platform initialization to the point where it can load the BL2 image, then it loads BL2 and switches execution to it.

• BL2 – BL2 is loaded and then executed on the on-chip boot SRAM. Its main functional-

ity is to perform the rest of the low-level architectural and platform initialization (e.g. initializing DRAM, setting up the System Address Mapping and calculating the Physical Memory Regions). It then loads the rest of the boot images (BL31, BL33). After loading the images, it traps itself back to BL1 via an SMC, which in turn switches execution to BL31.

• BL31 – BL31 is known as the EL3 Runtime Software. It is loaded to the boot RAM. Its

main functionality is to provide low-level runtime service support. After it finishes all its runtime software initialization, it passes control to BL33.

• BL33 – BL33 is known as the Non-trusted Firmware. For this case we are using EDK2

(Tianocore) UEFI. It is in charge of loading and passing control to the OS. For more detail on this, please see the EDK2 source.

NOTE: Some users may wish to use the GRUB2 bootloader for various reasons. In that case, UEFI would be configured to load GRUB2 instead of the Linux kernel.

2.3.1 Building ATF Images

To get the source code, directly execute the atf-56036e.patch file found in the directory /src/atf. It downloads the ATF sources from GitHub and patches it with BlueField specific code.

™

platform

Mellanox Technologies Confidential

Rev 1.1

Since BL1 is permanently burned into the BlueField on-chip boot ROM, the only real boot loader images which might need to be built are BL2 and BL31 (refer to the EDK2 documentation of how to build EDK2 to use as BL33). Thus we are building the “bl2” and “bl31” targets.

Before doing any build, the environment variable CROSS_COMPILE should point to the Arm cross-compiler which is being used. For example:

export CROSS_COMPILE=/path/to/cross/compiler/aarch64-poky-linux-

To build for the BlueField platform, we need PLAT set to “bluefield” when invoking “make”. You also need to set the TARGET_SYSTEM according to the specific system for which you are building (e.g. “bluewhale” if building for the BlueWhale reference platform). Every supported system has its own subdirectory under $SOURCE/plat/mellanox/bluefield/system/. If you are using your own system, you can use the “generic” platform as a starting point, create your own system’s subdirectory under the system directory, copy the files over from the generic subdirectory, and modify them to suit your particular machine. You can also pass the BUILD_BASE variable to specify where you want the files to be built. So to perform a basic build:

export CROSS_COMPILE=/path/to/cross/compiler/aarch64-poky-linuxmake BUILD_BASE=/path/to/build PLAT=bluefield TARGET_SYSTEM=bluewhale bl2 bl31

NOTE: If ATF is being built in an environment where the Yocto/Poky SDK script has been run (environment-setup-aarch64-poky-linux), th e user needs to set the LDFLAGS to NULL (export LDFLAGS="").

After the build finishes the needed bl2.bin and bl31.bin may be found under $BUILD_BASE/bluefield/<target_system>/release/.

2.3.2 Trusted Board Boot

The other two files in the directory (mbedtls-2.2.1.patch and gen_fuse_info.py) are related to building ATF with trusted board boot enabled.

For more information of how to perform trusted board boot, please refer to the Secure Boot document.

2.4 Building UEFI (EDK2)

After running the “edk2-*.patch” command in the directory “\src\edk2\” to set up a source tree for UEFI, cd into it and run “make -f /path/to/this/file”.

Customizations you may need or want to make are expanded on further below. Note that EDK2 requires building in the source tree. Also, the EDK2 build system fails with

parallel build, so you must build with -j1. The image built is BLUEFIELD_EFI.fd and/or BLUEFIELD_EFI_SEC.fd in the

Build/BlueField/RELEASE_GCC49/FV directory.

Mellanox Technologies Confidential

Rev 1.1

2.4.1 Customizable Build Options

The following are the customizable UEFI build options:

• The mode in which to build EDK2: DEBUG or RELEASE.

EDK2_MODE = RELEASE

• Any particular “defines” to use when building EDK2.

EDK2_DEFINES = \

-DSECURE_BOOT_ENABLE=TRUE \

-DFIRMWARE_VER=0.99 \

• Path to OpenSSL tarball: Set it to an already-downloaded locati on for the tarball, or else

this makefile will download it into the source tree.

OPENSSL_TARBALL = CryptoPkg/Library/OpensslLib/openssl-1.0.2d.tar.gz

• The compiler toolchain prefix to use for the tools: This can include the full path and pre-

fix if the tools are not in $(PATH).

GCC49_AARCH64_PREFIX = aarch64-none-elf-

NOTE: Note that GCC 4.9 or later will work. Here Yocto Poky gcc 6.3 or la t e r is used.

• If “iasl” is not in your path, specify its directory here. We use https://github.com/ac-

pica/acpica.git at commit ed0389cb or later.

IASL_PREFIX =

• If “dtc” is not in your path, specify its directory here. You can typically find it in a Linux

build tree in scripts/dtc. If you are using a Yocto SDK you can use the DTC contained within it.

DTC_PREFIX=/opt/poky/2.3.1/sysroots/x86_64-pokysdk-linux/usr/bin/

• Make sure the ARCH environment variable is NULL (un set ).

DTC_PREFIX =

• Device tree source files:

DTS_FILES = bf-full.dts DTS_DIR = ../dts

It is important to note that when you actually build EDK2 here make sure you are NOT in an environment/bash shell where environment-setup-aarch64-poky-linux was run. Simply point GCC49_AARCH64_PREFIX to:

/opt/poky/2.3.1/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-pokylinux/aarch64-poky-linux-

Running the SDK environment-setup-aarch64-poky-linux script confuses the UEFI make environment. It just needs a pointer to the tools.

2.4.2 Exporting Variables

# Export variables that need to be set in the environment. export IASL_PREFIX GCC49_AARCH64_PREFIX

FD_FILE = Build/BlueField/$(EDK2_MODE)_GCC49/FV/BLUEFIELD_EFI.fd

Mellanox Technologies Confidential

Rev 1.1

all: $(FD_FILE)

DTB_FILES = $(addprefix dtb/, $(DTS_FILES:.dts=.dtb))

$(FD_FILE): FORCE CryptoPkg/Library/OpensslLib/openssl-1.0.2d $(DTB_FILES) $(MAKE) -C BaseTools/Source/C set --; . ./edksetup.sh; \ build -n 6 -t GCC49 -a AARCH64 -p MlxPlatformPkg/BlueField.dsc \

-b $(EDK2_MODE) -DDTB_DIR=$(PWD)/dtb $(EDK2_DEFINES)

FORCE:

# Build device tree blobs for specified device tree source(s)

$(DTB_FILES): dtb/%.dtb: $(DTS_DIR)/%.dts mkdir -p $(@D) cpp -P -x assembler-with-cpp -o- $< | \ $(DTC_PREFIX)dtc -b 0 -O dtb -o $@.tmp -I dts mv -f $@.tmp $@

# These steps are documented in CryptoPkg/Library/OpensslLib/Patch-HOWTO.txt

CryptoPkg/Library/OpensslLib/openssl-1.0.2d: $(OPENSSL_TARBALL) tar -C $(@D) -xf $< cd $@ && patch -p0 < ../EDKII_openssl-1.0.2d.patch cd $(@D) && ./Install.sh

$(OPENSSL_TARBALL): curl https://www.openssl.org/source/openssl-1.0.2d.tar.gz > $@.tmp mv -f $@.tmp $@

2.5 Building Poky Initramfs

2.5.1 Basic Quick Start to Build Poky Initramfs

Run the script scripts-bluefield/yocto_dependencies from the /distro/yocto directory. Then run:

cd poky source oe-init-build-env cd conf

Customize your bblayers.conf file. To reproduce the shipped initramfs you can start with:

BBLAYERS ?= " \ <path>/distro/yocto/poky/meta \ <path>/distro/yocto/poky/meta-poky \ <path>/distro/yocto/poky/meta-yocto-bsp \ <path>/distro/yocto/meta-openembedded/meta-oe \ <path>/distro/yocto/meta-openembedded/meta-python \ <path>/distro/yocto/meta-openembedded/meta-networking \ <path>/distro/yocto/meta-openembedded/meta-filesystems \ <path>/distro/yocto/meta-openembedded/meta-perl \ <path>/distro/yocto/meta-virtualization \ <path>/distro/yocto/meta-linaro/meta-linaro \ <path>/distro/yocto/meta-linaro/meta-linaro-integration \ <path>/distro/yocto/meta-secure-core/meta-efi-secure-boot \ <path>/distro/yocto/meta-secure-core/meta-signing-key \ <path>/distro/yocto/meta-cloud-services/meta-openstack \ <path>/distro/yocto/meta-bluefield \ "

Replace “<path>” with your own path.

Mellanox Technologies Confidential

Rev 1.1

Customize your local.conf file. For example, change MACHINE to “bluefield”, include bluefield.conf, and set MLNX_OFED_PAT H.

MACHINE ??= "bluefield" include conf/bluefield.conf MLNX_OFED_PATH="<path>/distro/mlnx_ofed"

Then run:

cd .. bitbake core-image-initramfs

Note that the file system you just created is lo cated in “poky/build/tmp/deploy/images/ bluefield” while your kernel image will be in “tmp/deploy/images/bluefield”.

Common BlueField

• bitbake core-image-initramfs

• bitbake core-image-full

™

bitbake targets are:

• bitbake core-image-full-sdk -c populate_sdk

You can build Yocto/Poky on most major Linux distributions. Mellanox using CentOS 7.4, however, other distributions such as Ubuntu would also work but may require small modifications to the Yocto config files and/or recipes.

If you are not using CentOS and having difficulties, you may want to try running CentOS in a container or on a VM first in order to get a successful build with which to compare results.

For more information, please refer to the following URL:

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html.

2.5.2 Variables

Certain OFED recipes require that the source RPM or tarball already be downloaded to the build systems. These files are included in the BlueField Runtime Distribution. You should place these files anywhere you like and then set the appropriate variable in local.conf. For example, set:

MLNX_OFED_PATH=<your_local_path>/distro/mlnx_ofed

There are various variables that can be set in local.conf which add files created outside of Yocto to be copied into the root file systems:

• MLNX_OFED_PATH – location of local OFED pack ages. Look in distro/mlnx_ofed for

specific directories.

currently runs tests

• MLNX_OFED_VERSION – version of MLNX OFED (e.g. “4.2-1.4.13.”). This is used

with MLNX_OFED_PATH to find files.

• MLNX_OFED_BASE_OS – base OS version of MLNX OFED (e.g. “rhel7.3”). This is

used with MLNX_OFED_PATH to find files.

• MLNX_BLUEFIELD_VERSION_PATH – if there is a “bluefield_version” file in this

location it gets copied to /etc in the root file systems created by Yocto. See the “update_rootfs_bluefield” function in the meta-bluefield image recipes.

• MLNX_BLUEFIELD_FW_PATH – if this directory exists, the image recipes in meta-

bluefield copy the contents of this directory into /lib/firmware/mellanox on the generated root file systems

Mellanox Technologies Confidential

Rev 1.1

• MLNX_BLUEFIELD_BFB_PATH – if there are any bfb files (*.bfb) located at this lo ca-

tion, they are copied into the root file system (/lib/firmware/Mellanox). See the “update_rootfs_bluefield” function in the meta-bluefield image recipes.

• MLNX_BLUEFIELD_EXTRA_DEV_PATH – any files in the directory specified by thi s

variable are copied into /opt/mlnx/extra on the full root dev file system.

2.5.3 Downloading Upstream Yocto and Building SDK

The meta-bluefield layer supports the Mellanox BlueField SoC. To use it, edit your conf/bblayers.conf file to include this directory on the list of directories

in BBLAYERS. Similarly, you should also add the layers meta-oe, meta-python, and metanetworking from meta-openembedded, since packages in those layers are used in some of the images included in meta-bluefield/recipes-bsp/images.

You should edit your conf/local.conf file to set MACHINE to “bluefield”. To be able to build the same distro configurations used in the Mellanox

images (including using the same

kernel version shipped by Mellanox), you should also add:

include conf/bluefield.conf

NOTE: Mellanox is using Yocto Rocko 2.4 for this release.

2.6 Using Yocto as a Cross-compilation SDK and Root Filesystem Genera-

tor

You may download the Yocto/Poky SDK file from the same source from which you acquired the BlueField

poky-glibc-x86_64-core-image-full-sdk-aarch64-toolchain-2.4.1.sh

Unpacking this file into an SDK directory allows cross-compiling files which are going to run on the BlueField SoC. This directory may be located anywhere you want.

Alternatively, you may download the upstream Yocto and build your own SDK; for more information, see “2.5.3 Downloading Upstream Yocto and Building SDK

™

Runtime Distribution. Typically:

”.

To use the SDK cross-compilation tools, you should “source” the top-level “environmentsetup-aarch64-poky-linux” script to set various environment variables, including $PATH, $CC, $CROSS_COMPILE, etc. The cross-compilation tools (compiler, assembler, linker, etc.) are located in sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux; m a ny othe r useful tools are in the directories usr/bin, usr/sbin, bin, and sbin beneath sysroots/x86_64pokysdk-linux. The sysroots/aarch64-poky-linux hierarchy contains a copy of a root filesystem for Arm64 so the cross-compilation tools can find headers and libraries in it.

To compile your code you should use aarch64-poky-linux-gcc, and, if necessary, the other standard aarch64-poky-linux- tools. In general, you should take advantage of the various environment variables in your makefiles rather than relying on any specific name for the tools. Several of the tools (notably gcc) require a “--sysroot” argument which specifies the aarch64-poky-linux path—the $(CC) variable handles this for you.

+ 56 hidden pages

Mellanox BlueField SmartNIC operation manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Table of Contents

List of Figures

Document Revision History

About this Manual

Audience

Document Conventions

Common Abbreviations and Acronyms

Related Documentation

1 BlueField Software Overview

1.1 Debug Tools

1.2 BlueField Adapter/SmartNIC

1.3 BlueField-based Storage Appliance

1.4 BlueField Architecture

2 Installation and Initialization

2.1 Unpacking BlueField Software Distribution

2.2 Upgrading Boot Software

2.2.1 BFB File Overview

2.2.2 BlueField Boot Process

2.2.3 The mlxbf-bootctl Program

2.2.4 Upgrading the Bootloader

2.2.5 Updating the Boot Partition

2.2.6 Safely Updating with a BMC

2.2.7 Safely Updating Boot Software from the Arm Cores

2.2.8 Changing the Linux Kernel or Root File System

2.3 Building Arm Trusted Firmware

2.3.1 Building ATF Images

2.3.2 Trusted Board Boot

2.4 Building UEFI (EDK2)

2.4.1 Customizable Build Options

2.4.2 Exporting Variables

2.5 Building Poky Initramfs

2.5.1 Basic Quick Start to Build Poky Initramfs

2.5.2 Variables

Alternatively, you may download the upstream Yocto and build your own SDK; for more information, see “2.5.3 Downloading Upstream Yocto and Building SDK