Huawei ES3000 V3 User Manual

Download

Page 1

ES3000 V3 NVMe PCIe SSD

User Guide

Issue 20

Date 2019-02-27

HUAWEI TECHNOLOGIES CO., LTD.

Page 2

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base

Bantian, Longgang Shenzhen 518129 People's Republic of China

Website: http://www.huawei.com

Email: [email protected]

Page 3

DANGER

WARNING

CAUTION

ES3000 V3 NVMe PCIe SSD User Guide

Purpose

This document provides the product information about the Huawei ES3000 V3 PCIe SSD (ES3000 V3 for short) and describes how to install, configure, operate, and maintain the ES3000 V3.

NOTE

The document does not contain the commands for production, equipment, and return detection. These commands are commonly used during engineering implementation and fault locating. Incorrect use of the commands results in device faults or service interruption. To obtain the commands, contact Huawei technical support.

About This Document

Intended Audience

This document is intended for:

l Server installation engineers

l Server maintenance engineers

Symbol Conventions

The symbols that may be found in this document are defined as follows.

Symbol

Description

Indicates a hazard with a high level or medium level of risk which, if not avoided, could result in death or serious injury.

Indicates a hazard with a low level of risk which, if not avoided, could result in minor or moderate injury.

Indicates a potentially hazardous situation that, if not avoided, could result in equipment damage, data loss, performance deterioration, or unanticipated results.

Page 4

TIP

NOTE

ES3000 V3 NVMe PCIe SSD User Guide

Symbol Description

Change History

Changes between document issues are cumulative. The latest document issue contains all the changes made in earlier issues.

Issue 20 (2019-02-27)

Modified how to obtain the compatibility list.

About This Document

Provides a tip that may help you solve a problem or save time.

Provides additional information to emphasize or supplement important points in the main text.

Modified how to install hioadm.

Issue 19 (2018-08-07)

Modified the following:

l 5.7.3 One-Click Log Collection.

l 5.5.1 Upgrading the Controller Firmware in Linux.

Issue 18 (2018-06-27)

Modified some code to resolve the display issue.

Issue 17 (2018-04-11)

Added the 5.12 Removing an ES3000 V3 section.

Issue 16 (2018-03-02)

Added the 6.16 Restoring Default Attributes section.

Issue 15 (2017-12-04)

Updated the document name.

Issue 14 (2017-11-17)

Deleted atom write description.

Issue 13 (2017-09-28)

Added content related to the ES3620C V3 and ES3620P V3.

Page 5

ES3000 V3 NVMe PCIe SSD User Guide

Issue 12 (2017-05-05)

Optimized descriptions about some operation steps.

Issue 11 (2017-03-21)

Updated the 2.4 Specifications section.

Issue 10 (2017-02-14)

Updated the 6.4 Formatting section.

Issue 09 (2017-01-20)

Updated the 3.2 Installing ES3000 V3 section.

Issue 08 (2016-12-15)

About This Document

Added the atom write description.

Issue 07 (2016-12-05)

Updated the 2.4.2 ES3600P V3 Specifications section.

Issue 06 (2016-10-29)

Updated the 5.2.3 Upgrading the Driver in ESXi section.

Issue 05 (2016-10-08)

Updated the 3.3 Driver Information section.

Issue 04 (2016-07-20)

Added content related to the ES3600C V3.

Issue 03 (2016-05-18)

Updated the 4.2.1 Orderly Hot Plug of a Disk in Linux OS section.

Issue 02 (2016-05-01)

Updated the 3.3 Driver Information section.

Issue 01 (2015-11-11)

The issue is the first official release.

Page 6

ES3000 V3 NVMe PCIe SSD User Guide Contents

About This Document.....................................................................................................................ii

1 Safety................................................................................................................................................1

1.1 General Safety Notice.....................................................................................................................................................1

1.2 Labels on Devices...........................................................................................................................................................2

1.3 Electrical Safety..............................................................................................................................................................3

2 Getting to Know the ES3000 V3..................................................................................................6

2.1 Security...........................................................................................................................................................................6

2.2 Overview........................................................................................................................................................................ 7

2.3 Features...........................................................................................................................................................................9

2.4 Specifications................................................................................................................................................................10

2.4.1 ES3500P V3 Specifications.......................................................................................................................................10

2.4.2 ES3600P V3 Specifications.......................................................................................................................................12

2.4.3 ES3620P V3 Specifications.......................................................................................................................................14

2.4.4 ES3600C V3 Specifications...................................................................................................................................... 16

2.4.5 ES3620C V3 Specifications...................................................................................................................................... 18

2.5 System Requirements................................................................................................................................................... 20

3 Installation and Configuration................................................................................................. 22

3.1 Installation and Configuration Process.........................................................................................................................23

3.2 Installing ES3000 V3....................................................................................................................................................23

3.3 Driver Information........................................................................................................................................................29

3.4 Verifying the Upgrade Package Integrity..................................................................................................................... 30

3.5 Transferring Files by Using WinSCP........................................................................................................................... 30

3.6 Installing the Driver......................................................................................................................................................32

3.6.1 Installing the ES3000 V3 Driver in Linux.................................................................................................................32

3.6.2 Installing the ES3000 V3 Driver in Windows...........................................................................................................35

3.6.3 Installing the ES3000 V3 Driver in ESXi................................................................................................................. 42

3.7 Installing the Tool Package...........................................................................................................................................43

3.7.1 Installing the Tool Package (Linux).......................................................................................................................... 43

3.7.2 Installing the Tool Package (Windows).....................................................................................................................44

3.7.3 Installing the Tool (ESXi)..........................................................................................................................................48

4 Hot Plug.........................................................................................................................................49

4.1 OSs Supporting Hot Plug............................................................................................................................................. 50

Page 7

ES3000 V3 NVMe PCIe SSD User Guide Contents

4.2 Orderly Hot Plug.......................................................................................................................................................... 50

4.2.1 Orderly Hot Plug of a Disk in Linux OS................................................................................................................... 50

4.2.2 Orderly Hot Plug of a Disk in Windows....................................................................................................................60

5 Maintenance and Upgrade.........................................................................................................62

5.1 Uninstalling the Driver................................................................................................................................................. 62

5.1.1 Uninstalling the Driver in Linux............................................................................................................................... 63

5.1.2 Uninstalling the Driver in Windows..........................................................................................................................64

5.1.3 Uninstalling the Driver in ESXi................................................................................................................................ 65

5.2 Upgrading the Driver....................................................................................................................................................65

5.2.1 Upgrading the Driver in Linux.................................................................................................................................. 66

5.2.2 Upgrading the ES3000 V3 Driver in Windows.........................................................................................................68

5.2.3 Upgrading the Driver in ESXi................................................................................................................................... 68

5.3 Uninstalling, and Upgrading the Tool Package............................................................................................................ 69

5.3.1 Uninstalling the Tool Package (Linux)...................................................................................................................... 69

5.3.2 Uninstalling the Tool Package (Windows)................................................................................................................ 69

5.3.3 Uninstalling the Tool (ESXi)..................................................................................................................................... 70

5.3.4 Upgrading the Tool Package (Linux)........................................................................................................................ 70

5.3.5 Upgrading the Tool Package (Windows)...................................................................................................................71

5.3.6 Upgrading the Tool (ESXi)........................................................................................................................................72

5.4 Installing and Uninstalling Citrix System Drivers and Tools....................................................................................... 72

5.4.1 Installing Citrix Drivers and Tools............................................................................................................................ 72

5.4.2 Uninstalling Citrix Drivers and Tools........................................................................................................................73

5.5 Upgrading the Controller Firmware............................................................................................................................. 73

5.5.1 Upgrading the Controller Firmware in Linux............................................................................................................73

5.5.2 Upgrading the Controller Firmware in Windows......................................................................................................75

5.5.3 Upgrading the Controller Firmware in ESXi............................................................................................................ 76

5.6 Querying Health Status.................................................................................................................................................77

5.7 Information Collection................................................................................................................................................. 77

5.7.1 Collecting SSD Information...................................................................................................................................... 77

5.7.2 Collecting SSD Logs................................................................................................................................................. 78

5.7.3 One-Click Log Collection......................................................................................................................................... 79

5.8 Securely Erasing SSD Data.......................................................................................................................................... 80

5.9 Querying SSD Lifespans.............................................................................................................................................. 81

5.10 Common Faults...........................................................................................................................................................82

5.10.1 Hardware Connection Faults or Other Faults.......................................................................................................... 82

5.10.2 Disk Identification Faults or Other Faults............................................................................................................... 82

5.10.3 Disk Internal Faults or Other Faults........................................................................................................................ 82

5.11 Indicator Status Descriptions......................................................................................................................................85

5.11.1 ES3000 SSD Disk Indicator State Descriptions...................................................................................................... 85

5.11.2 ES3000 SSD Card Indicator State Descriptions......................................................................................................86

5.12 Removing an ES3000 V3........................................................................................................................................... 86

5.12.1 Removing a Non-Hot-Swappable ES3000 V3 NVMe PCIe SSD Disk.................................................................. 87

Page 8

ES3000 V3 NVMe PCIe SSD User Guide Contents

5.12.2 Removing a Hot Swappable ES3000 V3 NVMe PCIe SSD Disk...........................................................................88

5.12.3 Removing an ES3000 V3 NVMe PCIe SSD Card on a Riser Card........................................................................ 89

5.12.4 Removing an ES3000 V3 NVMe PCIe SSD Card from the Mainboard.................................................................90

6 Command Description............................................................................................................... 92

6.1 Querying Information................................................................................................................................................... 94

6.1.1 Querying the SSD List...............................................................................................................................................94

6.1.2 Querying Basic Information About a Device............................................................................................................ 95

6.1.3 Querying the SMART Information About a Device..................................................................................................96

6.1.4 Querying Advanced Information of a Device........................................................................................................... 99

6.2 Querying, Upgrading, and Activating Firmware........................................................................................................101

6.2.1 Querying the Firmware Version of an SSD............................................................................................................. 101

6.2.2 Upgrading the Firmware of an SSD........................................................................................................................ 102

6.2.3 Activating the Firmware of an SSD........................................................................................................................ 103

6.3 Querying and Setting the Capacity.............................................................................................................................104

6.3.1 Querying the Capacity of an SSD............................................................................................................................104

6.3.2 Setting the Capacity of an SSD............................................................................................................................... 105

6.4 Formatting.................................................................................................................................................................. 107

6.4.1 Formatting a Logical Disk.......................................................................................................................................107

6.4.2 Formatting an SSD.................................................................................................................................................. 108

6.5 Safely Erasing User Data of an SSD.......................................................................................................................... 110

6.6 Querying and Setting the Power Consumption Level................................................................................................ 112

6.6.1 Querying the Power Consumption Level of an SSD............................................................................................... 112

6.6.2 Setting the Power Consumption Level of an SSD...................................................................................................112

6.7 Obtaining Logs........................................................................................................................................................... 113

6.7.1 Obtaining Logs of a Device..................................................................................................................................... 113

6.7.2 Obtaining All Logs of a Device At a Time..............................................................................................................114

6.8 Querying and Setting Temperature Thresholds.......................................................................................................... 115

6.8.1 Querying the Temperature Information About a Device......................................................................................... 115

6.8.2 Setting the Temperature Thresholds of a Device..................................................................................................... 117

6.9 Querying and Setting the Write Mode........................................................................................................................119

6.9.1 Querying the Write Mode of an SSD.......................................................................................................................119

6.9.2 Setting the Write Mode of an SSD.......................................................................................................................... 119

6.10 Querying the Diagnosis Result of a Specified SSD................................................................................................. 120

6.11 Querying the Electronic Label Information About an SSD......................................................................................122

6.12 Recovering Data for an SSD.................................................................................................................................... 122

6.13 Querying and Configuring Namespaces...................................................................................................................123

6.13.1 Querying the Namespace Distribution of an SSD................................................................................................. 123

6.13.2 Creating a Namespace for an SSD........................................................................................................................ 124

6.13.3 Attaching a Namespace to an NVMe Controller...................................................................................................125

6.13.4 Detaching a Namespace from an NVMe Controller..............................................................................................126

6.13.5 Deleting a Namespace of an SSD..........................................................................................................................128

6.13.6 Setting the Number of Namespaces for an SSD....................................................................................................129

Page 9

ES3000 V3 NVMe PCIe SSD User Guide Contents

6.14 Querying or Setting the Atomic Write Status...........................................................................................................130

6.14.1 Querying the Atomic Write Status of a Specified SSD.........................................................................................130

6.14.2 Setting the Atomic Write Status for a Specified SSD........................................................................................... 131

6.15 Querying Historical Amount of Written Data.......................................................................................................... 131

6.16 Restoring Default Attributes.....................................................................................................................................132

A Supported NVMe Commands............................................................................................... 134

A.1 Supported NVMe Management Commands..............................................................................................................134

A.2 Supported NVMe I/O Commands............................................................................................................................. 135

A.3 Supported Get Log Page Commands.........................................................................................................................135

A.4 SMART Properties.....................................................................................................................................................135

A.5 Supported Set Feature Commands.............................................................................................................................146

A.6 Supported Customized Management Commands......................................................................................................148

A.6.1 Query Commands................................................................................................................................................... 148

A.6.1.1 Obtaining the Enablement Status of User Functions...........................................................................................149

A.6.1.2 Obtaining Logs.................................................................................................................................................... 150

A.6.1.3 Obtaining SSD ID Parameters............................................................................................................................. 152

A.6.1.4 Obtaining the Enablement Status of Functions................................................................................................... 154

A.6.1.5 Querying the Maximum Available Capacity of the SSD.....................................................................................155

A.6.1.6 Obtaining the Vendor Log Length....................................................................................................................... 156

A.6.1.7 Querying the PCB Version...................................................................................................................................156

A.6.1.8 Obtaining Self-Diagnosis Result......................................................................................................................... 156

A.6.1.9 Obtaining the DIEID of a Chip............................................................................................................................157

A.6.1.10 Obtaining Virtualization Information................................................................................................................ 158

A.6.1.11 Obtaining the Number of SRIOV VFs...............................................................................................................161

A.6.1.12 Querying the Average DWPD........................................................................................................................... 161

A.6.1.13 Querying UniqueID Information....................................................................................................................... 161

A.6.2 Configuration Commands.......................................................................................................................................162

A.6.2.1 Configuring the Enablement Status of User Functions....................................................................................... 163

A.6.2.2 Deleting Vendor Logs and Monitoring Data....................................................................................................... 164

A.6.2.3 Configuring the SSD Capacity............................................................................................................................ 164

A.6.2.4 Recovering Data.................................................................................................................................................. 165

A.6.2.5 Configuring the NN Value................................................................................................................................... 165

A.6.2.6 Virtualization Management Commands.............................................................................................................. 165

A.6.2.7 Configuring the Number of SRIOV VFs.............................................................................................................167

B Out-of-Band Management...................................................................................................... 168

B.1 NVMe Basic Management Command....................................................................................................................... 168

B.2 VPD............................................................................................................................................................................171

B.3 Temperature Register................................................................................................................................................. 173

C SNMP Function.........................................................................................................................175

C.1 Overview....................................................................................................................................................................175

C.2 Configuring the SNMP Function in Windows...........................................................................................................176

Page 10

ES3000 V3 NVMe PCIe SSD User Guide Contents

C.3 Configuring the SNMP Function in Linux................................................................................................................ 176

C.4 Querying SNMP Information.................................................................................................................................... 177

C.5 SNMP OID.................................................................................................................................................................179

D Obtaining Device Logs by Using an Open-Source Tool.................................................. 181

E Introduction to Windows GUI............................................................................................... 183

F Installing an OS on the ES3000 V3........................................................................................ 190

G How to Get Help.......................................................................................................................196

G.1 Preparations Before Contacting Huawei................................................................................................................... 196

G.1.1 Collecting Fault Information.................................................................................................................................. 196

G.1.2 Making Debugging Preparations............................................................................................................................ 197

G.2 Getting Help from Huawei Support Website............................................................................................................. 197

G.3 Contacting Huawei Technical Support...................................................................................................................... 197

H Acronym or Abbreviation.......................................................................................................199

Page 11

ES3000 V3 NVMe PCIe SSD User Guide

About This Chapter

This topic describes the safety precautions to be observed when you install and maintain devices.

1 Safety

1.1 General Safety Notice

This topic describes the safety precautions you must take before installing or maintaining Huawei equipment.

1.2 Labels on Devices

This topic describes the warning labels, grounding labels, and electrostatic discharge (ESD) labels on devices.

1.3 Electrical Safety

This topic describes safety notice about high voltage, thunderstorm, high electrical leakage, power cable, fuse, electrostatic discharge.

1.1 General Safety Notice

This topic describes the safety precautions you must take before installing or maintaining Huawei equipment.

Overview

l To ensure safety of humans and the equipment, pay attention to the safety symbols on the

equipment and all the safety instructions in this document.

l The "NOTE", "CAUTION", and "WARNING" marks in other documents do not

represent all the safety instructions. They are only supplements to the safety instructions.

Local Safety Regulations

이 기기는 업무용 환경에서 사용할 목적으로 적합성평가를 받은 기기로서 가정용 환경 에서 사용하는 경우 전파간섭의 우려가 있습니다.

When operating Huawei equipment, you must follow the local laws and regulations. The safety instructions in this document are only supplements to the local laws and regulations.

Page 12

ES3000 V3 NVMe PCIe SSD User Guide

Basic Requirements

l Installation and maintenance personnel must understand basic safety precautions to avoid

hazards.

l When operating Huawei equipment, in addition to following the general precautions in

this document, follow the specific safety instructions given by Huawei.

l Only trained and qualified personnel are allowed to install, operate, and maintain Huawei

equipment.

Ground

l Do not damage the ground conductor or operate the device in the absence of a properly

installed ground conductor. Conduct the electrical inspection carefully.

l The device must be connected permanently to the protection ground before an operation.

Human Safety

l Do not operate the device or cables during lightning strikes.

l Move or lift the chassis by holding its lower edge. Do not hold the handles on certain

modules such as power supply, fans, and boards because they cannot support the weight of the device.

l At least two persons are required to lift the chassis. When lifting it, keep your back

straight and move stably.

l Do not look into the optical port without eye protection.

l Do not wear jewelry or watches when you operate the device.

1 Safety

Installation

l The device (or system) must be installed in an access-controlled location.

l The device must be fixed securely on the floor or to other immovable objects such as

walls and mounting racks before operation.

l When installing the unit, always make the ground connection first and disconnect it at

the end.

l Do not block the ventilation while the device is operating. Keep a minimum distance of 5

cm between the device and the wall or other objects that may block the ventilation.

l Tighten the thumbscrews by using a tool after initial installation and subsequent access

to the panel.

1.2 Labels on Devices

This topic describes the warning labels, grounding labels, and electrostatic discharge (ESD) labels on devices.

Table 1-1 describes the safety labels on devices.

Page 13

ES3000 V3 NVMe PCIe SSD User Guide

Table 1-1 Safety labels

Label Meaning Description

1 Safety

Warning This label indicates that

wrong operations may cause device damage or human injury.

External grounding This label indicates

grounding of external devices. The terminals of the grounding cable are connected to different devices. This ensures normal running of the devices and the safety of the operator.

Internal grounding This label indicates

grounding of internal devices. The terminals of the grounding cable are connected to different components of the same device. This ensures normal running of the devices and the safety of the operator.

1.3 Electrical Safety

This topic describes safety notice about high voltage, thunderstorm, high electrical leakage, power cable, fuse, electrostatic discharge.

ESD-Preventive Label This label indicates an static

sensitive area. Do not touch the device with your hands. When operating the device within this area, take electrostatic discharge (ESD)-preventive measures. For example, wear an ESDpreventive wrist strap.

Page 14

ES3000 V3 NVMe PCIe SSD User Guide

High Voltage

l The high voltage power supply provides power for the device operation. Direct or indirect

contact (through damp objects) with high voltage and AC mains supply may result in fatal danger.

l Non-standard and improper high voltage operations may result in fire and electric shock.

Thunderstorm

Do not perform any operation, including high voltage and AC operations, on a steel tower or mast during a thunderstorm.

1 Safety

High Electrical Leakage

Ground the device before powering it on. Otherwise, personal injury or device damage may be caused by high leakage current.

If a "high electrical leakage" tag is present on the power terminal of the device, you must ground the device before powering it on.

Power Cable

Do not install or remove power cables when the device is on. Transient contact between the core of the power cable and the conductor may generate electric arcs or sparks, which may cause fire or hurt human eyes.

l Before installing or removing the power cable, turn off the power switch.

l Before connecting a power cable, check that the label on the power cable is correct.

Fuse

If a fuse is to be replaced, the new fuse must be of the same type and specifications.

Page 15

ES3000 V3 NVMe PCIe SSD User Guide

Electrostatic Discharge

The static electricity generated by human bodies may damage the electrostatic-sensitive components on boards, for example, the large-scale integrated (LSI) circuits.

Human body movement, friction between human bodies and clothes, friction between shoes and floors, or handling of plastic articles causes static electromagnetic fields on human bodies. These static electromagnetic fields cannot be eliminated until the static is discharged.

To prevent electrostatic-sensitive components from being damaged by the static on human bodies, you must wear a well-grounded ESD wrist strap when touching the device or handling boards or application-specific integrated circuits (ASICs).

Figure 1-1 shows how to wear an ESD wrist strap.

Figure 1-1 Wearing an ESD wrist strap

1 Safety

Page 16

ES3000 V3 NVMe PCIe SSD User Guide

2 Getting to Know the ES3000 V3

About This Chapter

This section describes the ES3000 V3 features.

2 Getting to Know the ES3000 V3

The ES3000 V3 introduced in this document includes two types of products: ES3000 V3 NVMe PCIe SSDs and ES3000 V3 NVMe PCIe SSD cards. V100R003C10 includes ES3000 V3 NVMe PCIe SSDs. ES3500P V3 and ES3600P V3 are available. V100R003C20 includes ES3000 V3 NVMe PCIe SSD cards. The ES3600C V3 series is available. V100R003C50 includes ES3000 V3 NVMe PCIe SSDs and NVMePCIe SSD cards. The ES3620P V3 series is available for NVMe PCIe SSDs and the ES3620C V3 series is available for NVMe PCIe SSD cards.

2.1 Security

This section describes the information security features of the ES3000 V3.

2.2 Overview

This section describes the ES3000 V3 appearance and standards.

2.3 Features

This section describes the features and technical specifications of the ES3000 V3.

2.4 Specifications

This section describes the specifications of the ES3000 V3.

2.5 System Requirements

This section describes the ES3000 V3 requirements for server software and hardware.

2.1 Security

This section describes the information security features of the ES3000 V3.

Huawei is committed to ensuring stability and security of customer networks and services.

As a storage device within a server or storage host, the ES3000 V3 provides the following security features:

l Software permission control:

Page 17

ES3000 V3 NVMe PCIe SSD User Guide

– In Linux, only administrators have the permission to install and uninstall the driver

– In Windows and ESXi, only administrators or administrator group members have

l Operation logs: The settings performed on the ES3000 V3 are recorded.

– In Linux, only administrators have the permission to perform operations on the

– In Windows and ESXi, only administrators and administrator group members have

l Security maintenance: The ES3000 V3 provides a data clearance tool that helps you

delete all user data from the devices to be repaired.

l Firmware upgrade: The upgrade tool verifies the integrity of the ES3000 V3 firmware

upgrade file to ensure that the upgrade is successful.

NOTE

To use the management tool, obtain the administrator rights. For example, the Windows administrator group must run as administrator.

2 Getting to Know the ES3000 V3

and tool packages, and use management tools.

the permission to install and uninstall the driver and tool packages, and use management tools.

ES3000 V3 and view operation logs.

the permission to perform operations on the ES3000 V3 and view operation logs.

2.2 Overview

This section describes the ES3000 V3 appearance and standards.

Two products are included: ES3000 V3 NVMe PCIe SSD disk and ES3000 V3 NVMe PCIe SSD card. The disk consists of the ES3500P V3, ES3600P V3, and ES3620P V3 series. The card consists of the ES3600C V3 and ES3620C V3 series. The ES3000 V3 meets the requirements for high input/output operations per second (IOPS), high reliability, high bandwidth, and large data storage capacity. It applies to services with frequent random read and write operations, such as search, cache, and databases.

l The ES3000 V3 NVMe PCIe SSD disk is a 2.5-inch PCIe 3.0 x 4 SSD disk that uses the

HiSilicon Hi1812 chip and complies with the SFF-8639 standard. It can be installed in universal servers, such as rack servers.

Figure 2-1 shows the appearance of the ES3000 V3 NVMe PCIe SSD disk.

Figure 2-1 ES3000 V3 NVMe PCIe SSD disk

Page 18

ES3000 V3 NVMe PCIe SSD User Guide

Label

The label on an ES3000 V3 NVMe PCIe SSD provides the World Wide Name (WWN) and serial number (S/N).

A WWN uniquely identifies an ES3000 V3 NVMe PCIe SSD and is a string generated from the MAC address. A WWN is required when you apply for technical support from Huawei.

An example S/N is 031YSW10G3000001. Figure 2-2 describes the S/N format.

Figure 2-2 Example S/N

2 Getting to Know the ES3000 V3

No.

Description

1 Indicates the material identification code

(6 characters).

2 Indicates the vendor code (two

characters).

3 Indicates the year and month (two

characters).

4 Indicates the serial number (six digits).

l The ES3000 V3 NVMe PCIe SSD card is a standard high-height, half-length PCIe 3.0 x

4 card. It can be installed in universal servers, such as rack servers.

Figure 2-3 shows the appearance of the ES3000 V3 NVMe PCIe SSD card.

Page 19

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-3 ES3000 V3 NVMe PCIe SSD card

2.3 Features

2 Getting to Know the ES3000 V3

This section describes the features and technical specifications of the ES3000 V3.

The ES3000 V3 NVMe PCIe SSD provides the following features:

l Uses the PCIe 3.0 x4 interface, complies with PCIe 3.0 electrical specifications, and is

backward compatible with PCIe 2.0 and 1.0.

l Provides various single-disk capacities to meet different application requirements.

– ES3500P V3 series single-disk capacities: 800 GB, 1200 GB, 2000 GB, and 3200

– ES3600P V3 series single-disk capacities: 800 GB, 1200 GB, 1600 GB, 2000 GB,

and 3200 GB

– ES3620P V3 series single-disk capacities: 6400 GB

– ES3600C V3 series single-disk capacities: 800 GB, 1200 GB, 1600 GB and 3200

– ES3620C V3 series single-disk capacities: 6400 GB

l Supports standard storage device operations.

– Block device operations in Linux

– Disk device operations in Windows

– Disk device operations in ESXi

l Supports device diagnosis and monitoring, and SMART information reporting.

– Checks the power supply, temperature, voltage, capacitance, and number of bad

blocks.

– Checks whether the flash component is normal.

– Monitors, queries, and warns about the SSD service life.

– Checks the SSD power consumption.

– Performs I/O statistics.

l Supports orderly hot plug (ES3500P V3, ES3600P V3, and ES3620P V3 series).

Page 20

ES3000 V3 NVMe PCIe SSD User Guide

l Ensures data integrity by using multiple technologies.

– Enhanced Low-Density Parity-Check (LDPC) algorithm: provides higher error

– Embedded RAID-like algorithm: implements channel-based error correcting. Data

– Proprietary flexible RAID algorithm: delivers RAID data protection and recovery

– Intelligent wear leveling algorithm: intelligently levels the flash chip wear pressure

– Advanced flash access technology: combines the read retry and adaptive read

– Data inspection technology: periodically inspects data and prevents errors.

– Power-off protection: prevents disk data loss when a power failure occurs on the

l Provides multiple maintenance and management tools with comprehensive functions and

high performance.

– Provides in-band online upgrades to facilitate routine maintenance.

– Provides device information in a centralized manner, including the types, capacities,

– Supports log query for querying device health status.

– Provides the manufacture dates and serial numbers to facilitate asset management.

2 Getting to Know the ES3000 V3

correction capability than that required by flash chips and ensures device reliability.

can be restored if an error occurs in a channel.

from channel failures.

and prolongs the device service life.

technologies of flash chips and ensures data validity.

server.

versions, remaining service life, and health status.

2.4 Specifications

This section describes the specifications of the ES3000 V3.

The ES3000 V3 comes in two form factors: disk and card. The disk consists of the ES3500P V3, ES3600P V3 and ES3620P V3 series. The card consists of the ES3600C V3 and ES3620C V3 series.

The ES3000 V3 driver version needs to have an I/O timeout period of 30 seconds or more.

The default power consumption level 0 ensures optimal performance. A non-default power consumption level is not recommended.

2.4.1 ES3500P V3 Specifications

Figure 2-4 shows the nameplates for the ES3500P V3 series.

Page 21

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-4 ES3500P V3 series nameplate

Table 2-1 lists the power consumption specifications for the ES3500P V3 series.

2 Getting to Know the ES3000 V3

Table 2-1 Power consumption specifications for the ES3500P V3 series

Parameter

Power consumption

Specifications

l 800 GB: 13.5 W maximum power consumptiona and 7 W standby

power consumption.

l 1200 GB: 17 W maximum power consumption and 7 W standby power

consumption.

l 2000 GB: 18.5 W maximum power consumption and 7 W standby

power consumption.

l 3200 GB: 21 W maximum power consumption and 7 W standby power

consumption.

l 4000 GB: 22 W maximum power consumption and 7 W standby power

consumption.

Disk

Standard 2.5-inch hard disk

Dimensions

a: The maximum power consumption is the test result with the maximum sequential write bandwidth.

Table 2-2 lists the reliability specifications for the ES3500P V3 series.

Page 22

ES3000 V3 NVMe PCIe SSD User Guide

Table 2-2 Reliability specifications for the ES3500P V3 series

Parameter Specifications

2 Getting to Know the ES3000 V3

Mean time

2 x 106 hours

between failures (MTBF)

Annual

≤ 0.44% failure rate (AFR)

Bit error rate

-17

(BER)

Data

≥ 3 months (40°C, powered off) protection

Table 2-3 lists the environmental specifications for the ES3500P V3 series.

Table 2-3 Environmental specifications for the ES3500P V3 series

Parameter

Temperature

Specifications

l Storage temperature: -40°C to +70°C (-40°F to +158°F)

l Drive case temperature: 0°C to 70°C (32°F to 158°F)

Altitude

l Operating: –305 m to 3048 m (–1000 ft to 10,000 ft)

l Non-operating: –305 m to 12,192 m (–1000 ft to 40,000 ft)

The highest operating temperature decreases by 1°C (33.8°F) for every

increase of 220 m (721.8 ft) in altitude.

Air speed > 450 LFM at a room temperature of 25°C

> 650 LFM at a room temperature of 35°C

Humidity

l Storage humidity: 5% RH to 95% RH

l Operating humidity: 5% RH to 95% RH

Vibration

l Operating vibration: 2.17 G

l Non-operating vibration: 3.13 G

Shock

l Operating shock: 1,000 G/0.5 ms

l Non-operating shock: 1,000 G/0.5 ms

2.4.2 ES3600P V3 Specifications

Figure 2-5 shows the nameplates for the ES3600P V3 series.

(5 Hz to 700 Hz)

RMS

(5 Hz to 800 Hz)

RMS

Page 23

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-5 ES3600P V3 series nameplate

Table 2-4 lists the power consumption specifications for the ES3600P V3 series.

2 Getting to Know the ES3000 V3

Table 2-4 Power consumption specifications for the ES3600P V3 series

Parameter

Power consumption

Specifications

l 800 GB: 14 W maximum power consumptiona and 7 W standby power

consumption.

l 1200 GB: 17.5 W maximum power consumption and 7 W standby

power consumption.

l 1600 GB: 18.5 W maximum power consumption and 7 W standby

power consumption.

l 2000 GB: 18 W maximum power consumption and 7 W standby power

consumption.

l 3200 GB: 22 W maximum power consumption and 7 W standby power

consumption.

Dimensions Standard 2.5-inch disk

a: The maximum power consumption is the test result with the maximum sequential write bandwidth.

Table 2-5 lists the reliability specifications for the ES3600P V3 series.

Page 24

ES3000 V3 NVMe PCIe SSD User Guide

Table 2-5 Reliability specifications for the ES3600P V3 series

Parameter Specifications

2 Getting to Know the ES3000 V3

Mean time

2 x 106 hours between

failures (MTBF)

Annual

≤ 0.44% failure rate (AFR)

Bit error rate

-17

(BER)

Data

≥ 3 months (40°C, powered off) protection

Table 2-6 lists the environmental specifications for the ES3600P V3 series.

Table 2-6 Environmental specifications for the ES3600P V3 series

Parameter

Temperature

Specifications

l Storage temperature: -40°C to +70°C (-40°F to +158°F)

l Drive case temperature: 0°C to 70°C (32°F to 158°F)

Altitude

l Operating: –305 m to 3048 m

l Non-operating: –305 m to 12,192 m

The highest operating temperature decreases by 1°C for every increase of

220 m in altitude.

Air speed > 450 LFM at a room temperature of 25°C

> 650 LFM at a room temperature of 35°C

Humidity

l Storage humidity: 5% RH to 95% RH

l Operating humidity: 5% RH to 95% RH

Vibration

l Operating vibration: 2.17 G

l Non-operating vibration: 3.13 G

Shock

l Operating shock: 1,000 G/0.5 ms

l Non-operating shock: 1,000 G/0.5 ms

2.4.3 ES3620P V3 Specifications

Figure 2-6 shows the nameplates for the ES3620P V3 series.

(5 Hz to 700 Hz)

RMS

(5 Hz to 800 Hz)

RMS

Page 25

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-6 ES3620P V3 series nameplates

2 Getting to Know the ES3000 V3

Table 2-7 lists the power consumption specifications for the ES3620P V3 series.

Table 2-7 Power consumption specifications for the ES3620P V3 series

Parameter

Power consumption

Specifications

6400 GB: 25 W maximum power consumptiona and 7 W standby power

consumption.

Dimensions Standard 2.5-inch disk

a: The maximum power consumption is the test result with the maximum sequential write bandwidth.

Table 2-8 lists the reliability specifications for the ES3620P V3 series.

Table 2-8 Reliability specifications for the ES3620P V3 series

Parameter

Mean time

Specifications

2 x 106 hours between

failures (MTBF)

Annual

≤ 0.44% failure rate (AFR)

Bit error rate

-17

(BER)

Page 26

ES3000 V3 NVMe PCIe SSD User Guide

Parameter Specifications

2 Getting to Know the ES3000 V3

Data

≥ 3 months (40°C, powered off) protection

NOTE

6400 GB disks of the ES3620P V3 supports the 4096B LBA format only.

Table 2-9 lists the environmental specifications for the ES3620P V3 series.

Table 2-9 Environmental specifications for the ES3620P V3 series

Parameter

Temperature

Specifications

l Storage temperature: -40°C to +70°C (-40°F to +158°F)

l Drive case temperature: 0°C to 70°C (32°F to 158°F)

Altitude

l Operating: –305 m to 3048 m

l Non-operating: –305 m to 12,192 m

The highest operating temperature decreases by 1°C for every increase of

220 m in altitude.

Air speed > 450 LFM at a room temperature of 25°C

> 650 LFM at a room temperature of 35°C

Humidity

l Storage humidity: 5% RH to 95% RH

l Operating humidity: 5% RH to 95% RH

Vibration

l Operating vibration: 2.17 G

l Non-operating vibration: 3.13 G

Shock

l Operating shock: 1,000 G/0.5 ms

l Non-operating shock: 1,000 G/0.5 ms

2.4.4 ES3600C V3 Specifications

Figure 2-7 shows the nameplates for the ES3600C V3 series.

(5 Hz to 700 Hz)

RMS

(5 Hz to 800 Hz)

RMS

Page 27

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-7 ES3600C V3 series nameplate

Table 2-10 lists the power consumption specifications for the ES3600C V3 series.

2 Getting to Know the ES3000 V3

Table 2-10 Power consumption specifications for the ES3600C V3 series

Parameter

Power consumption

Specifications

l 800 GB: 14 W maximum power consumptiona and 7 W standby power

consumption.

l 1200 GB: 17.5 W maximum power consumption and 7 W standby

power consumption.

l 1600 GB: 18.5 W maximum power consumption and 7 W standby

power consumption.

l 3200 GB: 22 W maximum power consumption and 7 W standby power

consumption.

Dimensions Standard half-height, half-length PCIe card. Dimensions: 167.5 mm × 68.9

mm × 18.6 mm (6.6 in. x 2.7 in. x 0.7 in.)

Table 2-11 lists the reliability specifications for the ES3600C V3 series.

Table 2-11 Reliability specifications for the ES3600C V3 series

Parameter

Description

Mean time

2 x 106 hours between

failures (MTBF)

Annual

≤ 0.44% failure rate (AFR)

Page 28

ES3000 V3 NVMe PCIe SSD User Guide

Parameter Description

2 Getting to Know the ES3000 V3

Bit error rate

-17

(BER)

Data

≥ 3 months (40°C, powered off) protection

Table 2-12 lists the environmental specifications for the ES3600C V3 series.

Table 2-12 Environmental specifications for the ES3600C V3 series

Parameter

Temperature

Specifications

l Storage temperature: -40°C to +70°C (-40°F to +158°F)

l Operating temperature: 0°C to 55°C (32°F to 131°F) environmental

temperature

Altitude

l Operating: –305 m to 3048 m (–1000 ft to 10,000 ft)

l Non-operating: –305 m to 12,192 m (–1000 ft to 40,000 ft)

The highest operating temperature decreases by 1°C (33.8°F) for every

increase of 220 m (721.8 ft) in altitude.

Air speed Room temperature 55 °C: air speed 300 LFM

Humidity

l Storage humidity: 5% RH to 95% RH

l Operating humidity: 5% RH to 95% RH

Vibration

l Operating shock: 0.27 G

each axis

l Non-operating shock: 1.49 G

for each axis

Shock

l Operating vibration: 31 G/2 ms. You only need to hit on the device

function pane once.

l Non-operating vibration: 70 G/2 ms. You only need to hit on the device

function pane once.

2.4.5 ES3620C V3 Specifications

Figure 2-8 shows the nameplates for the ES3620C V3 series.

(5 Hz to 500 Hz). 3 axes, 10 minutes for

RMS

(10 Hz to 500 Hz). 3 axes, 15 minutes

RMS

Page 29

ES3000 V3 NVMe PCIe SSD User Guide

Figure 2-8 ES3620C V3 series nameplates

Table 2-13 lists the power consumption specifications for the ES3620C V3 series.

2 Getting to Know the ES3000 V3

Table 2-13 Power consumption specifications for the ES3620C V3 series

Parameter

Power consumption

Specifications

6400 GB: 22 W maximum power consumptiona and 7 W standby power

consumption.

Dimensions Standard half-height, half-length PCIe card. Dimensions: 167.5 mm × 68.9

mm × 18.6 mm (6.6 in. x 2.7 in. x 0.7 in.)

a: The maximum power consumption is the test result with the maximum sequential write bandwidth.

Table 2-14 lists the reliability specifications for the ES3620C V3 series.

Table 2-14 Reliability specifications for the ES3620C V3 series

Parameter

Mean time

Description

2 x 106 hours

between failures (MTBF)

Annual failure

≤ 0.44%

rate (AFR)

Bit error rate

-17

(BER)

Data

≥ 3 months (40°C, powered off)

protection

Page 30

ES3000 V3 NVMe PCIe SSD User Guide

NOTE

6400 GB disks of the ES3620C V3 supports the 4096B LBA format only.

Table 2-15 lists the environmental specifications for the ES3620C V3 series.

Table 2-15 Environmental specifications for the ES3620C V3 series

Parameter Specifications

2 Getting to Know the ES3000 V3

Temperature

l Storage temperature: -40°C to +70°C (-40°F to +158°F)

l Operating temperature: 0°C to 55°C (32°F to 131°F) environmental

temperature

Altitude

l Operating: –305 m to 3048 m (–1000 ft to 10,000 ft)

l Non-operating: –305 m to 12,192 m (–1000 ft to 40,000 ft)

The highest operating temperature decreases by 1°C (33.8°F) for every

increase of 220 m (721.8 ft) in altitude.

Air speed Room temperature 55 °C: air speed 450 LFM

Humidity

l Storage humidity: 5% RH to 95% RH

l Operating humidity: 5% RH to 95% RH

Vibration

l Operating shock: 0.27 G

(5 Hz to 500 Hz). 3 axes, 10 minutes for

RMS

each axis

l Non-operating shock: 1.49 G

(10 Hz to 500 Hz). 3 axes, 15 minutes

RMS

for each axis

Shock

l Operating vibration: 31 G/2 ms. You only need to hit on the device

function pane once.

l Non-operating vibration: 70 G/2 ms. You only need to hit on the device

function pane once.

2.5 System Requirements

This section describes the ES3000 V3 requirements for server software and hardware.

Hardware Requirements

The minimum requirements for server hardware are as follows:

l 2 GB memory

l To use the ES3000 V3 NVMe PCIe SSD disk, the server must provide at least one 2.5-

inch SSD disk slot that supports the PCIe SFF-8639 connector.

l To use the ES3000 V3 NVMe PCIe SSD card, the server must provide at least one half-

height, half-length (167.5 mm x 70 mm x 18.6 mm) PCIe 3.0 x4 card slot.

Page 31

ES3000 V3 NVMe PCIe SSD User Guide

Software Requirements

l For details about the OS types and versions supported by the ES3000 V3, visit Server

Compatibility Checker, find the Compatibility List of the corresponding server.

l OSs listed in this section support common I/O services. To obtain the list of OSs that

support hot plug, see 4.1 OSs Supporting Hot Plug. To obtain the list of OSs that can be installed on the ES3000 V3, see F Installing an OS on the ES3000 V3.

2 Getting to Know the ES3000 V3

Page 32

ES3000 V3 NVMe PCIe SSD User Guide

3 Installation and Configuration

About This Chapter

This section describes the process for installing and configuring the ES3000 V3.

3 Installation and Configuration

If the ES3000 V3 exceeds the overtemperature threshold, handle the problem promptly; otherwise, management operations on the SSD may fail. For details about the command for querying temperature information, see 6.8.1 Querying the Temperature Information About

a Device.

3.1 Installation and Configuration Process

The installation and configuration process includes the overall procedures for installing and configuring the ES3000 V3. You can learn about ES3000 V3 installation and configuration logic.

3.2 Installing ES3000 V3

This topic describes how to install the ES3000 V3.

3.3 Driver Information

This section describes how to obtain the ES3000 V3 software package.

3.4 Verifying the Upgrade Package Integrity

Verify that the obtained upgrade package is consistent with the original one at the website http://support.huawei.com/enterprise.

3.5 Transferring Files by Using WinSCP

This section describes how to transfer files by using WinSCP. You can also transfer files by using other software.

3.6 Installing the Driver

This section describes how to install the ES3000 V3 driver in different operating systems (OSs). The ES3500P V3, ES3600P V3, ES3620P V3, ES3600C V3, and ES3620C V3 series use the same drivers and firmware.

3.7 Installing the Tool Package

This section describes how to install the tool package in Linux, Windows, and ESXi operating systems (OSs).

Page 33

ES3000 V3 NVMe PCIe SSD User Guide

3.1 Installation and Configuration Process

The installation and configuration process includes the overall procedures for installing and configuring the ES3000 V3. You can learn about ES3000 V3 installation and configuration logic.

the flowchart for installing and configuring ES3000 V3 is shown in Figure 3-1.

Figure 3-1 Installation and configuration flowchart

Start

Installing ES3000 V3

Drive Information

3 Installation and Configuration

Verifying the Upgrade

Package Integrity

Installing the Driver

Installing the Tool

Package

End

3.2 Installing ES3000 V3

This topic describes how to install the ES3000 V3.

Operation Scenario

l The method of installing the ES3000 V3 NVMe PCIe SSD disk is the same as the

method of installing a common hard disk. For details, see Installing the ES3000 V3

NVMe PCIe SSD Disk.

l If the PCIe card slots are on a riser card, the ES3000 V3 NVMe PCIe SSD card needs to

be installed on the riser card. For details, see Installing the ES3000 V3 NVMe PCIe

SSD card on a Riser Card.

l If the PCIe card slots are on the mainboard, the ES3000 V3 NVMe PCIe SSD card needs

to be installed on the mainboard. For details, see Installing the ES3000 V3 NVMe PCIe

SSD card on the Mainboard.

Page 34

ES3000 V3 NVMe PCIe SSD User Guide

Preparations

l Stop all services on the server and back up data.

l Unpack and inspect the devices.

Check that the items shipped are complete and intact based on the following list:

– ES3000 V3

– Quick Start Guide

– Warranty card

l Tools

– ESD gloves

Pay attention to the following in Linux:

3 Installation and Configuration

Used to prevent ESD damage when you insert, remove, or hold the ES3000 V3 or a precision device.

Procedure

l Do not perform direct read or write operations on a raw SSD to avoid processing the

wrong SSD due to a changed drive letter. Restarting the server, installing or uninstalling the SSD driver, inserting and removing the SSD may change the SSD drive letter.

l If you have to perform direct read or write operations on a raw SSD, use the hioadm info

command to check the SSD information and confirm the drive letter through the SSD SN.

l It is recommended that you use the mount command to mount the SSD partitions to a

specified path before performing read and write operations. To ensure normal release of system resources, use the unmount command to cancel the mounted SSD partitions before installing or uninstalling the driver, and inserting or removing the SSD.

l You can also add information to the /etc/fstab file so that SSD partitions are automatically

mounted during system startup, driver installation and uninstallation, and SSD insertion and removal. For details, run the man fstab command to view help information. After enabling the automatic mounting of SSD partitions, you do not need to run the umount command before installing or uninstalling the driver, and inserting or removing the SSD.

l Install the ES3000 V3 NVMe PCIe SSD.

a. Wear ESD gloves.

b. Take the ES3000 V3 out of the ESD bag. Check whether a 2.5-inch or 3.5-inch hard

disk tray is used.

n For a 2.5-inch hard disk tray, go to c.

n For a 3.5-inch hard disk tray, go to e.

c. Place the ES3000 V3 into the hard disk tray. See step (1) in Figure 3-2.

d. Use four screws to secure the ES3000 V3 to the hard disk tray. See step (2) in

Figure 3-2. Then go to h.

Page 35

ES3000 V3 NVMe PCIe SSD User Guide

3 Installation and Configuration

Figure 3-2 Placing the ES3000 V3 into the hard disk tray

e. Place the ES3000 V3 to the converter. See step (1) in Figure 3-3.

f. Use two screws to secure the ES3000 V3 to the converter. See step (2) in Figure

3-3.

Page 36

ES3000 V3 NVMe PCIe SSD User Guide

g. Use four screws to secure the ES3000 V3 to the hard disk tray. See step (3) in

3 Installation and Configuration

Figure 3-3 Securing the ES3000 V3 to the converter

Figure 3-4 and (4) in Figure 3-5.

Figure 3-4 Securing the ES3000 V3 to the hard disk tray (1)

Page 37

ES3000 V3 NVMe PCIe SSD User Guide

3 Installation and Configuration

Figure 3-5 Securing the ES3000 V3 to the hard disk tray (2)

h. Open the ejector lever and push the ES3000 V3 into the server until it does not

move. See step (1) in Figure 3-6.

i. Close the ejector lever to push the ES3000 V3 into place. See step (2) in Figure

3-6.

Page 38

ES3000 V3 NVMe PCIe SSD User Guide

l Install the ES3000 V3 NVMe PCIe SSD card on a riser card.

3 Installation and Configuration

Figure 3-6 Installing the ES3000 V3

a. Wear ESD gloves.

b. Turn off the power to the server and remove the chassis cover.

c. Take the ES3000 V3 out of the ESD bag.

d. Hold the upper edge of the ES3000 V3, align its connecting part with a PCIe slot on

the riser card, and insert the ES3000 V3 slowly into the PCIe slot. See Figure 3-7.

Figure 3-7 Installing the ES3000 V3 on a riser card

e. Install the riser card in the server. For details, see related server manuals.

f. Turn on the power to the server.

l Install the ES3000 V3 NVMe PCIe SSD card on the mainboard.

Page 39

ES3000 V3 NVMe PCIe SSD User Guide

a. Wear ESD gloves.

b. Turn off the power to the server and remove the chassis cover.

c. Hold the upper edge of the ES3000 V3, align its connecting part with the PCIe slot,

3 Installation and Configuration

and vertically insert the ES3000 V3 downwards into the PCIe slot. See Figure 3-2.

Figure 3-8 Installing the ES3000 V3 on the mainboard

d. Turn on the power to the server.

----End

Follow-up Procedure

After the ES3000 V3 is connected to power, the startup process takes about 30 seconds.

If the active indicator associated with the ES3000 V3 NVMe PCIe SSD is on and the fault indicator is off, the SSD hardware is installed successfully.

If the green indicator (active indicator) on the ES3000 V3 NVMe PCIe SSD is on and the orange indicator (fault indicator) is off, the SSD hardware is installed successfully.

3.3 Driver Information

This section describes how to obtain the ES3000 V3 software package.

Confirm the scenario before downloading the driver.

l If the server uses only ES3000 V3 NVMe PCIe SSD disks and does not use SSDs

provided by other vendors. You can use either the NVMe driver integrated in the OS or that provided by Huawei. For details about how to install Huawei's NVMe driver, see 3.6

Installing the Driver.

Page 40

ES3000 V3 NVMe PCIe SSD User Guide

l If the server running Linux uses both ES3000 V3 NVMe PCIe SSD disks and SSDs

provided by other vendors, use the NVMe driver integrated in the OS and you do not need to install Huawei's NVMe driver.

l If Huawei's NVMe driver has been installed in Linux and you need to use SSDs provided

by other vendors, uninstall Huawei's NVMe driver.

l For Windows Server 2008 R2 or Windows Server 2012, if the KB2990941 patch has

been installed, the OS is integrated with the NVMe driver. To install Huawei's NVMe driver, uninstall the KB2990941 patch first. For details, see 3.6.2 Installing the ES3000

V3 Driver in Windows.

l The following OSs are also integrated with the NVMe driver:

– Windows Server 2012 R2

– ESXi 6.0

– RHEL 6.5 or later

– CentOS 6.5 or later

– OEL 6.5 or later

– Ubuntu 14.04 or later

– SLES 11 SP4 or later

3 Installation and Configuration

You can download drivers as follows: log in to Huawei Enterprise support website http://

support.huawei.com/enterprise, choose Select a Product to Get Support > IT > Server >

TaiShan > ES3000 V3, click Downloads, and download the required software package. The

software package contains host tools (including hioadm) and Huawei's NVMe driver.

3.4 Verifying the Upgrade Package Integrity

Verify that the obtained upgrade package is consistent with the original one at the website http://support.huawei.com/enterprise.

Click and , load the digital certificate and software package respectively.

For details about the verification tool and method, visit the following link:

Software digital signature (OpenPGP) validation tool

3.5 Transferring Files by Using WinSCP

This section describes how to transfer files by using WinSCP. You can also transfer files by using other software.

Scenarios

Use WinSCP to transfer files from a PC to a remote host.

Prerequisites

Conditions

The File Transfer Protocol (FTP) service has been enabled on the remote host.

Data

Page 41

ES3000 V3 NVMe PCIe SSD User Guide

You have obtained the following data:

l IP address of the remote host to be connected

l User name and password for logging in to the remote host to be connected

Software

WinSCP.exe: a free software available on the Internet.

Procedure

Step 1 Open the WinSCP folder, and double-click WinSCP.exe.

The WinSCP Login dialog box is displayed, as shown in Figure 3-9.

NOTE

To change the UI language, click Languages.

Figure 3-9 WinSCP login

3 Installation and Configuration

Step 2 Set the login parameters.

The parameters are described as follows:

l Host name: Enter the IP address of the remote host to be connected. For example,

191.100.34.32.

l Port number: The default value is 22.

l User name: Enter the user name. For example, admin.

l Password: Enter the password.

Step 3 Click Login.

The WinSCP file transfer window is displayed.

Page 42

ES3000 V3 NVMe PCIe SSD User Guide

NOTE

l If a key file was not selected during the first login, a warning Continue connecting and add host

l By default, the C:\Documents and Settings\Administrator\My Documents directory of the local

Step 4 In the right pane, choose a directory of the remote host for storing files, for example, /root/

firmware.

Step 5 In the left pane, select a directory of the local PC for storing files, for example E:\Software,

and then select the file to be transferred.

Step 6 Choose File > Copy.

The Copy dialog box is displayed.

Step 7 Confirm the file to be copied and the directory of the remote host, and then click Copy.

The system starts to copy the file from the local PC to the remote host.

----End

3 Installation and Configuration

key to cache is displayed. Click Yes. The WinSCP file transfer page is displayed.

PC is opened in the left pane, and the /root directory of the remote host is opened in the right pane.

3.6 Installing the Driver

3.6.1 Installing the ES3000 V3 Driver in Linux

Operation Scenario

If you decide to use Huawei's NVMe driver after reading 3.3 Driver Information, install it by referring to this section before using the ES3000 V3 for the first time.

If you install multiple ES3000 V3s on a server, you need to install the driver only once.

NOTE

The Linux driver package naming rules are as follows:

l RHEL, CentOS, or OEL driver package name: kmod-hiodriver-Driver version-Supported kernel

version.System distributor.Hardware platform.rpm, such as kmod-

hiodriver-3.3.5.2-3.10.0_123.el7.centos.x86_64.rpm, kmodhiodriver-3.3.5.2-3.10.0_123.el7.redhat.x86_64.rpm, or kmodhiodriver-3.3.5.2-3.10.0_123.el7.oracle.x86_64.rpm.

l SLES driver package name: hiodriver-kmp-Kernel feature-Driver version-Supported kernel

version.System distributor.Hardware platform.rpm, such as hiodriver-kmp-

default-3.3.5.2-3.12.28_4.suse12.x86_64.rpm.

l Ubuntu or Debian driver package name: hiodriver -Driver version-Supported kernel

version.System distributor.Hardware platform.deb, such as

hiodriver-3.3.5.3-3.19.0_15_generic.ubuntu.amd64.deb.

l UVP driver package name: hiodriver-kmp-xen-Driver version-Supported kernel version.System

distributor.Hardware platform.rpm.

Page 43

ES3000 V3 NVMe PCIe SSD User Guide

Preparations

Before installing the ES3000 V3 driver, check that:

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the ES3000 V3 driver package to the server, and verified the

package integrity.

l You have uploaded the driver package to a directory (for example, /home) on the server

by using WinSCP.

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Go to the directory that stores the ES3000 V3 driver package, for example, /home.

Step 3 Run the installation command based on the OS type.

l For RHEL, CentOS, OEL, SLES, or UVP, run rpm -ivh driver package name.

An example package name for CentOS is rpm -ivh kmod- hiodriver-3.3.5.3-2.6.32_431.el6.centos.x86_64.rpm.

l For Ubuntu or Debian, run dpkg -i driver package name.

An example package name for Ubuntu is dpkg -i hiodriver-3.3.5.3-3.19.0_15_generic.ubuntu.amd64.deb.

3 Installation and Configuration

NOTE

– If the CPU hardware capabilities are insufficient for processing high-speed I/O services of NVMe

devices during interruption, it is recommended that you configure the NVMe driver to work in threaded_irq mode before installing it. The configuration method is as follows:

1. Run the vim /etc/modprobe.d/nvme.conf command to create or open the configuration file for editing.

2. Enter i.

3. In the configuration file, change the value of options nvme use_threaded_interrupts to 1:

options nvme use_threaded_interrupts=1

4. Press Esc, enter :wq, and press Enter to save the edits.

5. Run the chmod +x /etc/modprobe.d/nvme.conf command to change the permission on nvme.conf.

– If the NVMe driver works in threaded_irq mode, the I/O latency increases: Both the 4 KB read

latency and write latency increase by 5 μs.

Step 4 Check whether the following message indicating an incorrect installation package is

displayed.

Error: The installing package does not match the current OS. Please execute command: rpm -e kmod-hiodriver to uninstall the installing rpm.

l If yes, go to Step 5.

l If no, go to Step 6.

Step 5 Uninstall the installed package as prompted. Upload the correct driver package for the OS to

the server and perform Step 2 again to install the driver.

Step 6 Check whether the following message is displayed. The message indicates that the original

NVMe driver fails to be uninstalled during the installation.

Warning: fail to uninstall the old nvme kernel module. Please uninstall the old nvme kernel module manually or reboot the system.

Page 44

ES3000 V3 NVMe PCIe SSD User Guide

l If yes, go to Step 9.

l If no, go to Step 7.

Step 7 Check whether the following message is displayed. The message indicates that the new

NVMe driver fails to be installed during the installation.

Warning: fail to install the new nvme kernel module. Please install the new nvme kernel module manually or reboot the system.

l If yes, go to Step 8.

l If no, go to Step 12.

Step 8 Check whether restarting the OS will affect services.

l If yes, restart the OS when no service is running and go to Step 12.

l If no, restart the OS and go to Step 12.

Step 9 Check whether restarting the OS will affect services.

l If yes, go to Step 10.

l If no, restart the OS and go to Step 12.

Step 10 Run the modprobe -r nvme command to uninstall the original driver. During the

uninstallation, check whether a message is displayed indicating that the driver is in use. If yes, the uninstallation fails.

3 Installation and Configuration

l If yes, restart the OS when no service is running and go to Step 12.

l If no, go to Step 11.

Step 11 Run the modprobe nvme command to install the new driver. Check whether a warning or

error is displayed during the installation.

l If yes, restart the OS and go to Step 12.

l If no, go to Step 12.

Step 12 Run the modinfo nvme command to check the driver author information to determine

whether the driver is successfully installed.

If the author in the command output is as follows, the driver is successfully installed.

author: Huawei Technologies Co., Ltd.

l If yes, go to Step 13.

l If no, go to Step 15.

Step 13 If a PCIe SSD device has been installed on the server, check whether the driver has initialized

the device.

Check whether the NVMe device exists in the /dev directory to determine whether the device initialization is complete.

l If yes, go to Step 14.

l If no, go to Step 15.

Step 14 if the OS uses the irqbalance service to balance CPU interrupts, restart the service by using

one of the following commands after re-loading the driver module:

l For RHEL 6 or CentOs 6, run service irqbalance restart.

l For SLES 11 or UVP, run service irq_balancer restart.

l For SLES 12 or RHEL 7, run systemctl restart irqbalance.service.

Page 45

ES3000 V3 NVMe PCIe SSD User Guide

NOTE

l For RHEL 6, CentOs 6, or SLES 12, you can add IRQBALANCE_ARGS="--hintpolicy=exact" to the

end of the /etc/sysconfig/irqbalance file content to better balance interrupts.

l To configure the irqbalance service in other OSs, refer to documents related to the OSs.

If the OS cannot find the service, it indicates that the OS does not use the irqbalance service for balance interrupts. If the command can be run, check whether errors are reported after running the command for restarting the irqbalance service.

l If yes, go to Step 15.

l If no, no further action is required.

Step 15 Contact Huawei technical support for help.

NOTE

If the ES3000 V3 hiodriver is used, certain operating systems such as SUSE Linux will record the module verification failed error in dmesg logs. This error message is a notification indicating that an operating system detects a third-party driver with an unmatched signature. It does not affect services. For detailed reasons and solutions, see Why the "module verification failed " Message Is Displayed

When ES3000 V3 PCIe SSDs Use Huawei-developed SUSE NVMe Drivers.

3 Installation and Configuration

----End

3.6.2 Installing the ES3000 V3 Driver in Windows

Operation Scenario

If you decide to use Huawei's NVMe driver after reading 3.3 Driver Information, install it by referring to this section before using the ES3000 V3 for the first time.

NOTE

l If you install multiple ES3000 V3s on a server, you need to install the driver only once.

l The ES3000 V3 tool is incompatible with the NVMe driver built in the Windows OS. Ensure that

you have installed Huawei's NVMe driver before using the tool.

l To install Windows on the ES3000 V3, see F Installing an OS on the ES3000 V3.

Preparations

Before installing the ES3000 V3 driver, check that:

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the driver package and verified its integrity. For details, see 3.4

Verifying the Upgrade Package Integrity.

l You have uploaded the driver package to the server by using WinSCP or other software.

Procedure

Select an operation based on the OS type.

l For Windows Server 2012 R2, go to 8.

l For Windows Server 2008 R2 or Windows Server 2012, go to 1.

Check whether the SSD driver is provided by the OS.

1 Log in to the OS as an administrator.

Page 46

ES3000 V3 NVMe PCIe SSD User Guide

2 Right-click Computer and choose Properties from the shortcut menu. The System

window is displayed, as shown in Figure 3-10.

Figure 3-10 System

3 Installation and Configuration

3 Click Device Manager. The Device Manager is displayed, as shown in Figure 3-11.

Page 47

ES3000 V3 NVMe PCIe SSD User Guide

Figure 3-11 Device Manager

3 Installation and Configuration

The following are three possible cases:

– If the device name is Standard NVM Express Controller, the device uses the

NVMe driver provided by the OS.

– If the device name is Huawei ES3000V3 storage controller, the device uses

Huawei's SSD driver.

– The device does not have a driver.

4 Determine the next step based on the driver type.

– If the driver is provided by the OS, go to 5.

– If the driver is provided by Huawei, uninstall the driver by referring to 5.1.2

Uninstalling the Driver in Windows and go to 8.

– If the device does not have a driver, go to 8.

5 Choose Start > Control Panel > Programs and Features.

6 Click View installed updates. The Installed Updates window is displayed, as shown in

Figure 3-12.

Page 48

ES3000 V3 NVMe PCIe SSD User Guide

Figure 3-12 Installed Updates

7 Right-click the KB2990941 patch in the Microsoft Windows column and choose

Uninstall from the shortcut menu.

Install the driver.

8 Double-click the driver installation file hiodriver_v3.3.5.2_2008R2_x84_64.msi.

There are three driver installation file types for three Windows versions. *.*.*.* indicates the version number. This procedure uses hiodriver_v3.3.5.2_2008R2_x84_64.msi as an example.

– hiodriver_v*.*.*.*_2008R2_x84_64.msi

– hiodriver_v*.*.*.*_2012_x84_64.msi

– hiodriver_v*.*.*.*_2012R2_x84_64.msi

The installation interface is displayed, as shown in Figure 3-13.

3 Installation and Configuration

NOTE

Figure 3-13 Installation interface

Page 49

ES3000 V3 NVMe PCIe SSD User Guide

9 Click Next.

The End-User License Agreement window is displayed, as shown in Figure 3-14.

Figure 3-14 End-User License Agreement window

3 Installation and Configuration

10 Select I accept the terms in the License Agreement and click Next.

The Choose Setup Type window is displayed, as shown in Figure 3-15.

Page 50

ES3000 V3 NVMe PCIe SSD User Guide

Figure 3-15 Choose Setup Type window

3 Installation and Configuration

11 Click Complete.

The Ready to install window is displayed, as shown in Figure 3-16.

Figure 3-16 Ready to install window

Page 51

ES3000 V3 NVMe PCIe SSD User Guide

12 Click Install.

When the installation is complete, the Completing Setup Wizard window is displayed, as shown in Figure 3-17.

Figure 3-17 Completing Setup Wizard window

3 Installation and Configuration

13 Click Finish.

Verify the Installation.

14 Open Device Manager and expand Storage controllers.

15 Expand Disk drives.

Check whether Huawei ES3000V3 storage controller exists under Storage controllers. See Figure 3-18.

– If yes, go to 15.

– If no, contact Huawei technical support.

Figure 3-18 Expanding Storage controllers

Check whether NVMe HWE32P43032M000N SCSI Disk Device exists under Disk drives. See Figure 3-19.

Page 52

ES3000 V3 NVMe PCIe SSD User Guide

– If yes, the driver has been successfully installed.

– If no, contact Huawei technical support.

Figure 3-19 Expanding Disk drives

----End

3.6.3 Installing the ES3000 V3 Driver in ESXi

Operation Scenario

3 Installation and Configuration

Preparations

Procedure

Step 1 Log in to the OS as an administrator or a member of the administrator group.

Step 2 Install the ES3000 V3 driver.

You need to install the ES3000 V3 driver on the server before using the ES3000 V3 for the first time.

NOTE

l If you install multiple ES3000 V3s on a server, you need to install the driver only once.

l To install ESXi on the ES3000 V3, see F Installing an OS on the ES3000 V3.

Before installing the ES3000 V3 driver, check that:

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the ES3000 V3 driver package to the server, and verified the

package integrity.

l You have uploaded the driver package to the server by using WinSCP or other software.

The following procedure assumes that the driver package is in the /tmp directory, and the driver package name is hiodriver-3.3.5.2-1OEM.550.0.0.2768847.x86_64.vib.

Run the following command to install the ES3000 V3 driver package:

esxcli software vib install -v driver package in a directory

The following is an example command:

esxcli software vib install -v /tmp/hiodriver-3.3.5.2-1OEM.550.0.0.2768847.x86_64.vib

Page 53

ES3000 V3 NVMe PCIe SSD User Guide

NOTE

l In the driver file name, 3.3.5.2 is the driver version. The actual driver version may differ.

l Enter a complete directory after -v.

l If the "Could not find a trusted signer" error occurs indicating that the driver version is not certified by

ESXi IOVP, add --no-sig-check at the end of the command to skip signature check.

Step 3 After the driver is installed, run the reboot command to restart the OS.

Step 4 After the restart, run the esxcli software vib list | grep hiodriver command. If the following

command output is displayed, the driver is installed properly.

[root@esxi113:~] esxcli software vib list | grep hiodriverhiodriver 3.3.5.2-1OEM.

550.0.0.1391871 Huawei ESXiCertified 2016-01-04

Step 5 If the SSD device has been inserted into the server, run the lspci | grep "Huawei ES3000

V3" command. If the device is displayed, it is detected by the OS.

[root@esxi113:~] lspci | grep "Huawei ES3000 V3" 0000:06:00.0 Mass storage controller: Huawei Technologies Co Ltd. Huawei ES3000 V3 storage Controller [vmhba3]

Step 6 Run the vmkload_mod -l | grep hiodriver command to check whether the driver is loaded.

If the command output is similar to the following, the device is found, which indicates that the driver is loaded.

[root@esxi113:~] vmkload_mod -l | grep hiodriver hiodriver 2 112

l If yes, no further action is required.

l If no, go to Step 7.

3 Installation and Configuration

Step 7 Contact Huawei technical support for help.

----End

3.7 Installing the Tool Package

This section describes how to install the tool package in Linux, Windows, and ESXi operating systems (OSs).

3.7.1 Installing the Tool Package (Linux)

Operation Scenario

Install the tool package for managing the ES3000 V3.

The tool package naming rules are as follows:

l For RHEL, CentOS, OEL, or SLES, the package name is hioadm-version number-

release number.hardware architecture.rpm, for example, hioadm-3.3.5.2-1.x86_64.rpm.

l For Ubuntu or Debian, the package name is hioadm-version number-release

number.hardware architecture.deb, for example, hioadm-3.3.5.2-1.amd64.deb.

Preparations

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the tool package and verified its integrity.

l You have used WinSCP or other software to upload the tool package to a directory (for

example, /home) on the server.

Page 54

ES3000 V3 NVMe PCIe SSD User Guide

Procedure

Step 1 Log in to the operating system (OS) as an administrator.

Step 2 Go to the directory that stores the tool package, for example, /home.

Step 3 Run the installation command based on the OS type.

l For RHEL, CentOS, OEL, or SLES, run rpm -ivh tool package name.

Example: rpm -ivh hioadm-3.3.5.2-1.x86_64.rpm

l For Ubuntu or Debian, run dpkg -i tool package name.

Example: dpkg -i hioadm-3.3.5.2-1.amd64.deb

Step 4 Run the following command to obtain help information about the tool:

hioadm -h or hioadm --help

Step 5 Copy linux/tool/ host_info_collect.sh to /opt/hio.

----End

3.7.2 Installing the Tool Package (Windows)

3 Installation and Configuration

Operation Scenario

Install the tool package for managing the ES3000 V3. The ES3000 V3 tool is incompatible with the NVMe driver built in the Windows OS. Ensure that you have installed Huawei's NVMe driver before using the tool.

Preparations

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the tool package and verified its integrity.

l You have uploaded the tool package to the server by using Windows Secure Copy

(WinSCP) or other software.

Procedure

Step 1 Log in to the operating system (OS) as an administrator or administrator group member.

Step 2 Double-click the tool package file hioadm_v3.3.5.4_x86_64.msi.

NOTE

v3.3.5.4 indicates a driver version number. This procedure uses hioadm_v3.3.5.4_x86_64.msi as an example.

The installation interface is displayed, as shown in Figure 3-20.

Page 55

ES3000 V3 NVMe PCIe SSD User Guide

Figure 3-20 Installation interface

3 Installation and Configuration

Step 3 Click Next.

The End-User License Agreement window is displayed, as shown in Figure 3-21.

Figure 3-21 End-User License Agreement window

Page 56

ES3000 V3 NVMe PCIe SSD User Guide

Step 4 Select I accept the terms in the License Agreement and click Next.

The Destination Folder window is displayed, as shown in Figure 3-22.

Figure 3-22 Destination Folder window

3 Installation and Configuration

Step 5 Select a folder for installing the tool and click Next.

The Ready to install window is displayed, as shown in Figure 3-23.

Page 57

ES3000 V3 NVMe PCIe SSD User Guide

Figure 3-23 Ready to install window

3 Installation and Configuration

Step 6 Click Install to start installation.

When the installation is complete, the Completing Setup Wizard window is displayed, as shown in Figure 3-24.

Figure 3-24 Completing Setup Wizard window

Page 58

ES3000 V3 NVMe PCIe SSD User Guide

Step 7 Click Finish.

Step 8 Copy windows\Windows_Tool\diagnostic_win.bat to C:\Program Files (x86)\hioadm.

----End

3.7.3 Installing the Tool (ESXi)

Operation Scenario

Install the tool for managing the ES3000 V3.

Procedure

Method 1

Step 1 Integrated into the driver package, the tool will be automatically installed during the driver

package installation. For details about how to install the driver package, see 3.6.3 Installing

the ES3000 V3 Driver in ESXi.

3 Installation and Configuration

Step 2 Run the following command to obtain help information about the tool:

hioadm -h

Step 3 Copy vmware/host_okc.sh to /scratch/hio/.

----End

Method 2

Step 1 Copy binary tool file hioadm in the driver package to the system.

Step 2 Run the following command to grant binary tool file hioadm the execute permission.

chmod +x hioadm

Step 3 Copy file hioadm at Step 2 to the /bin directory so that the tool can be accessed directly.

----End

Page 59

ES3000 V3 NVMe PCIe SSD User Guide

About This Chapter

This section describes how to hot-plug a PCIe SSD.

4 Hot Plug

Hot plug includes the following two ways:

l Orderly hot plug: Users can directly insert a PCIe SSD when the OS is running, but

needs to notify the OS before removing a PCIe SSD.

l Surprise hot plug: Users can directly insert or remove a PCIe SSD.

The ES3000 V3 NVMe PCIe SSD card does not support orderly or surprise hot plug. The ES3000 V3 NVMe PCIe SSD disk supports orderly hot plug.

l Swap only one ES3000 V3 disk at a time. After one ES3000 V3 disk is removed or

inserted, wait for at least 3 seconds before swapping or inserting another disk.

l SSD hot plug may affect services running on the SSD. Before performing a hot plug,

notify system maintenance personnel to ensure service security.

l If you insert the ES3000 V3 disk when it is not installed on a hard disk tray, it may fail to

be detected.

l After the ES3000 V3 disk is removed, wait for at least 3 seconds before inserting it again.

If you insert the ES3000 V3 disk again when it is halfway removed, it may fail to be detected.

l Do not pause when inserting the ES3000 V3 disk, otherwise, it may fail to be detected and

leads to system exceptions. If the disk cannot be detected or system exceptions occur, power cycle the server after the disk is properly installed.

l If the ES3000 V3 fails to be detected due to improper insertion, remove and insert it again.

4.1 OSs Supporting Hot Plug

4.2 Orderly Hot Plug

Page 60

ES3000 V3 NVMe PCIe SSD User Guide

4.1 OSs Supporting Hot Plug

Ensure that the server and BIOS supports hot plug.

Only SSDs for data storage can be hot-plugged. Do not hot-plug SSDs running OSs.

The following OSs support hot plug of the ES3000 V3:

l Mainstream Linux OSs:

– Red Hat: 6.6, 6.7, 7.0, 7.1, 7.2

– CentOS: 6.6, 6.7, 6.8, 7.0, 7.1, 7.2

– SLES: 11SP4, 12, 12 SP1

– Oracle: OEL 7.0, OEL 7.1, OEL 7.2

– Ubuntu: 14.04 LTS, 15.04

– SUSE: 12 SP1

l Mainstream Windows OSs:

– Windows Server 2008 R2 64-Bit

– Windows Server 2012 64-Bit

– Windows Server 2012 R2 64-Bit

4 Hot Plug

4.2 Orderly Hot Plug

You can insert the ES3000 V3 disks to running servers. For Linux and Windows, notify the OS by referring to this section before hot removing the ES3000 V3 disk. Currently ESXi does not support orderly hot removal of the ES3000 V3.

4.2.1 Orderly Hot Plug of a Disk in Linux OS

Operation Scenario

In Linux, perform the operations described in this topic before performing an orderly hot plug of an ES3000 V3 disk.

Preparations

You have correctly installed the ES3000 V3 disk on the server.

Configuring Linux kernel

In Linux (such as Red Hat7.0), hot plug of the ES3000 V3 NVMe PCIe SSD disk will cause unmatched maximum payload size, I/O write errors, and other issues. This is because Linux does not fully support hot plug. To avoid this issue, make changes to the Linux kernel file. The changes vary with the Linux OS type.

Perform an operation based on the OS type:

l Red Hat (7.0, 7.1, 7.2) and CentOS (7.0, 7.1, 7.2) => Go to Red Hat (7.0, 7.1, 7.2) and

CentOS (7.0, 7.1, 7.2).

Page 61

ES3000 V3 NVMe PCIe SSD User Guide

l Red Hat (6.6, 6.7) and CentOS (6.6, 6.7) => Go to Red Hat (6.6, 6.7) and CentOS (6.6,

6.7).

l Oracle Enterprise Linux (7.0, 7.1, 7.2) => Go to Oracle Enterprise Linux (7.0, 7.1,

7.2).

l SLES 12 => Go to SLES 12.

l SLES 11 SP4 => Go to SLES 11 SP4.

l Ubuntu (14.04/15.04) => Go to Ubuntu (14.04/15.04).

Operations for different OSs are as follows:

l Red Hat (7.0, 7.1, 7.2) and CentOS (7.0, 7.1, 7.2)

The following procedure uses CentOS 7.0 as an example.

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /boot/grub2/grub.cfg command to open the boot configuration file.

c. Enter i.

d. In the boot configuration file, add pciehp.pciehp_force=1 pci=pcie_bus_perf to

4 Hot Plug

the boldface part shown in the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) ### BEGIN /etc/grub.d/10_linux ### menuentry 'CentOS Linux, with Linux 3.10.0-123.el7.x86_64' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-123.el7.x86_64-advanced-b054ccd0c3d3-47eb-a449-f897f63f1b76' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod xfs set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1' b64fedb9-b1e9-465d-994f-9cee287c1d57 else search --no-floppy --fs-uuid --set=root b64fedb9b1e9-465d-994f-9cee287c1d57 fi linux16 /vmlinuz-3.10.0-123.el7.x86_64 root=UUID=b054ccd0c3d3-47eb-a449-f897f63f1b76 ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet LANG=en_US.UTF-8 pciehp.pciehp_force=1

pci=pcie_bus_perf

initrd16 /initramfs-3.10.0-123.el7.x86_64.img } menuentry 'CentOS Linux, with Linux 0rescue-17e12da386e9435aa7f2b6b718079678' --class centos --class gnulinux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-17e12da386e9435aa7f2b6b718079678-advanced-b054ccd0c3d3-47eb-a449-f897f63f1b76' { load_video insmod gzio insmod part_msdos insmod xfs set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1' b64fedb9-b1e9-465d-994f-9cee287c1d57 else

Page 62

ES3000 V3 NVMe PCIe SSD User Guide

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Restart the OS for the settings to take effect.

g. Go to Notifying the OS.

l Red Hat (6.6, 6.7) and CentOS (6.6, 6.7)

The following procedure uses Red Hat 6.7 as an example:

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /boot/grub/grub.conf command to open the boot configuration file.

c. Enter i.

d. In the boot configuration file, add pciehp.pciehp_force=1 pci=pcie_bus_perf to

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Restart the OS for the settings to take effect.

g. Go to Notifying the OS.

l Oracle Enterprise Linux (7.0, 7.1, 7.2)

The following procedure uses Oracle Enterprise Linux 7.0 as an example:

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /boot/grub2/grub.cfg command to open the boot configuration file.

c. Enter i.

d. In the boot configuration file, add pciehp.pciehp_force=1 pci=pcie_bus_perf to

4 Hot Plug

search --no-floppy --fs-uuid --set=root b64fedb9b1e9-465d-994f-9cee287c1d57 fi linux16 /vmlinuz-0-rescue-17e12da386e9435aa7f2b6b718079678 root=UUID=b054ccd0-c3d3-47eb-a449-f897f63f1b76 ro rd.lvm.lv=centos/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet pciehp.pciehp_force=1 pci=pcie_bus_perf initrd16 /initramfs-0-rescue-17e12da386e9435aa7f2b6b718079678.img } if [ "x$default" = 'CentOS Linux, with Linux 3.10.0-123.el7.x86_64' ]; then default='Advanced options for CentOS Linux>CentOS Linux, with Linux

3.10.0-123.el7.x86_64'; fi; ### END /etc/grub.d/10_linux ### …(Some code omitted.)

the boldface part shown in the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) title Red Hat Enterprise Linux 6 (2.6.32-573.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-573.el6.x86_64 ro root=/dev/mapper/ VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/ lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=VolGroup/ lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

pciehp.pciehp_force=1 pci=pcie_bus_perf

initrd /initramfs-2.6.32-573.el6.x86_64.img …(Some code omitted.)

the boldface part shown in the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) ### BEGIN /etc/grub.d/10_linux ### menuentry 'Oracle Linux Server, with Linux 3.10.0-123.el7.x86_64' -class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-123.el7.x86_64-advancedeba3dd96-8be3-46ce-a8d9-21699e3fbedb' { load_video

Page 63

ES3000 V3 NVMe PCIe SSD User Guide

4 Hot Plug

set gfxpayload=keep insmod gzio insmod part_msdos insmod xfs set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

pci=pcie_bus_perf

initrd16 /initramfs-3.10.0-123.el7.x86_64.img } menuentry 'Oracle Linux Server, with Unbreakable Enterprise Kernel

3.8.13-35.3.1.el7uek.x86_64' --class oracle --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.8.13-35.3.1.el7uek.x86_64-advanced-eba3dd96-8be3-46cea8d9-21699e3fbedb' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod xfs set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1' fcf9f115-8f51-496e-a7a8-905f03350b4e else search --no-floppy --fs-uuid --set=root fcf9f115-8f51-496ea7a8-905f03350b4e fi linux16 /vmlinuz-3.8.13-35.3.1.el7uek.x86_64 root=UUID=eba3dd96-8be3-46ce-a8d9-21699e3fbedb ro crashkernel=auto vconsole.font=latarcyrheb-sun16 rd.lvm.lv=ol/swap rd.lvm.lv=ol/root vconsole.keymap=us rhgb quiet pciehp.pciehp_force=1 pci=pcie_bus_perf initrd16 /initramfs-3.8.13-35.3.1.el7uek.x86_64.img } menuentry 'Oracle Linux Server, with Linux 0-rescuea6e5b98906f643a09908dd24f3b775f7' --class oracle --class gnu-linux -class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0rescue-a6e5b98906f643a09908dd24f3b775f7-advanced-eba3dd96-8be3-46cea8d9-21699e3fbedb' { load_video insmod gzio insmod part_msdos insmod xfs set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

Page 64

ES3000 V3 NVMe PCIe SSD User Guide

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Restart the OS for the settings to take effect.

g. Go to Notifying the OS.

l SLES 12

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /boot/grub2/grub.cfg command to open the boot configuration file.

c. Enter i.

d. In the boot configuration file, add pciehp.pciehp_force=1 pci=pcie_bus_perf to

4 Hot Plug

if [ "x$default" = 'Oracle Linux Server, with Unbreakable Enterprise Kernel 3.8.13-35.3.1.el7uek.x86_64' ]; then default='Advanced options for Oracle Linux Server>Oracle Linux Server, with Unbreakable Enterprise Kernel 3.8.13-35.3.1.el7uek.x86_64'; fi; ### END /etc/grub.d/10_linux ### …(Some code omitted.)

the boldface part shown in the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) ### BEGIN /etc/grub.d/10_linux ### menuentry 'SLES12' --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-dcbfdecb-bcd9-4003-ba51a95c7e31c56e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos2

--hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 --hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecb-bcd9-4003ba51-a95c7e31c56e fi echo 'Loading Linux 3.12.28-4-default ...' linux /boot/vmlinuz-3.12.28-4-default root=UUID=dcbfdecbbcd9-4003-ba51-a95c7e31c56e ${extra_cmdline} resume=/dev/disk/by-uuid/ 9a586421-38ef-43bb-9209-afe5eede22e3 splash=silent quiet crashkernel=216M-:108M showopts pciehp.pciehp_force=1 pci=pcie_bus_perf echo 'Loading initial ramdisk ...' initrd /boot/initrd-3.12.28-4-default } submenu 'Advanced options for SLES12' --hotkey=1 $menuentry_id_option 'gnulinux-advanced-dcbfdecb-bcd9-4003-ba51-a95c7e31c56e' { menuentry 'SLES12, with Linux 3.12.28-4-default' --hotkey=2 -class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-advanced-dcbfdecbbcd9-4003-ba51-a95c7e31c56e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hintbios=hd0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 -hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecbbcd9-4003-ba51-a95c7e31c56e

Page 65

ES3000 V3 NVMe PCIe SSD User Guide

4 Hot Plug

fi echo 'Loading Linux 3.12.28-4-default ...' linux /boot/vmlinuz-3.12.28-4-default root=UUID=dcbfdecb-bcd9-4003-ba51-a95c7e31c56e ${extra_cmdline} resume=/dev/disk/by-uuid/9a586421-38ef-43bb-9209-afe5eede22e3 splash=silent quiet crashkernel=216M-:108M showopts

pciehp.pciehp_force=1 pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' initrd /boot/initrd-3.12.28-4-default } menuentry 'SLES12, with Linux 3.12.28-4-default (recovery mode)'

--hotkey=3 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-recovery-dcbfdecbbcd9-4003-ba51-a95c7e31c56e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hintbios=hd0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 -hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecbbcd9-4003-ba51-a95c7e31c56e fi echo 'Loading Linux 3.12.28-4-default ...' linux /boot/vmlinuz-3.12.28-4-default root=UUID=dcbfdecb-bcd9-4003-ba51-a95c7e31c56e ${extra_cmdline} showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe crashkernel=216M-:108M

pciehp.pciehp_force=1 pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' initrd /boot/initrd-3.12.28-4-default } }

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ### menuentry 'SLES12, with Xen hypervisor' --class sles12 --class gnu-linux

--class gnu --class os --class xen $menuentry_id_option 'xen-gnulinuxsimple-dcbfdecb-bcd9-4003-ba51-a95c7e31c56e' { insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos2

--hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 --hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecb-bcd9-4003ba51-a95c7e31c56e fi echo 'Loading Xen 4.4.1_06-2.2 ...' if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then xen_rm_opts= else xen_rm_opts="no-real-mode edd=off" fi multiboot /boot/xen-4.4.1_06-2.2.gz placeholder crashkernel=216M-:108M ${xen_rm_opts} echo 'Loading Linux 3.12.28-4-xen ...' module /boot/vmlinuz-3.12.28-4-xen placeholder root=UUID=dcbfdecb-bcd9-4003-ba51-a95c7e31c56e ${extra_cmdline} resume=/dev/disk/by-uuid/9a586421-38ef-43bb-9209-afe5eede22e3 splash=silent quiet crashkernel=216M-:108M showopts

Page 66

ES3000 V3 NVMe PCIe SSD User Guide

4 Hot Plug

pciehp.pciehp_force=1 pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' module --nounzip /boot/initrd-3.12.28-4-xen } submenu 'Advanced options for SLES12 (with Xen hypervisor)' $menuentry_id_option 'gnulinux-advanced-dcbfdecb-bcd9-4003-ba51a95c7e31c56e' { submenu 'Xen hypervisor, version 4.4.1_06-2.2' $menuentry_id_option 'xen-hypervisor-4.4.1_06-2.2-dcbfdecb-bcd9-4003ba51-a95c7e31c56e' { menuentry 'SLES12, with Xen 4.4.1_06-2.2 and Linux

3.12.28-4-xen' --class sles12 --class gnu-linux --class gnu --class os -class xen $menuentry_id_option 'xen-gnulinux-3.12.28-4-xen-advanceddcbfdecb-bcd9-4003-ba51-a95c7e31c56e' { insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hintbios=hd0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 -hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecb-bcd9-4003-ba51-a95c7e31c56e fi echo 'Loading Xen 4.4.1_06-2.2 ...' if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then xen_rm_opts= else xen_rm_opts="no-real-mode edd=off" fi multiboot /boot/xen-4.4.1_06-2.2.gz placeholder crashkernel=216M-:108M ${xen_rm_opts} echo 'Loading Linux 3.12.28-4-xen ...' module /boot/vmlinuz-3.12.28-4-xen placeholder root=UUID=dcbfdecb-bcd9-4003-ba51-a95c7e31c56e ${extra_cmdline} resume=/dev/disk/by-uuid/9a586421-38ef-43bb-9209-afe5eede22e3 splash=silent quiet crashkernel=216M-:108M showopts

pciehp.pciehp_force=1 pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' module --nounzip /boot/initrd-3.12.28-4-xen } menuentry 'SLES12, with Xen 4.4.1_06-2.2 and Linux

3.12.28-4-xen (recovery mode)' --class sles12 --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-3.12.28-4xen-recovery-dcbfdecb-bcd9-4003-ba51-a95c7e31c56e' { insmod part_msdos insmod btrfs set root='hd0,msdos2' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hintbios=hd0,msdos2 --hint-efi=hd0,msdos2 --hint-baremetal=ahci0,msdos2 -hint='hd0,msdos2' dcbfdecb-bcd9-4003-ba51-a95c7e31c56e else search --no-floppy --fs-uuid --set=root dcbfdecb-bcd9-4003-ba51-a95c7e31c56e fi echo 'Loading Xen 4.4.1_06-2.2 ...' if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then xen_rm_opts= else xen_rm_opts="no-real-mode edd=off" fi multiboot /boot/xen-4.4.1_06-2.2.gz placeholder ${xen_rm_opts} echo 'Loading Linux 3.12.28-4-xen ...' module /boot/vmlinuz-3.12.28-4-xen placeholder

Page 67

ES3000 V3 NVMe PCIe SSD User Guide

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Restart the OS for the settings to take effect.

g. Go to Notifying the OS.

l SLES 11SP4

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /etc/init.d/boot.local command to open the boot configuration file.

c. Enter i.

4 Hot Plug

root=UUID=dcbfdecb-bcd9-4003-ba51-a95c7e31c56e single ${extra_cmdline}

pciehp.pciehp_force=1 pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' module --nounzip /boot/initrd-3.12.28-4-xen } } } ### END /etc/grub.d/20_linux_xen ### …(Some code omitted.)

NOTE

Whether the file content includes "### BEGIN /etc/grub.d/20_linux_xen ###" depends on whether Xen Hypervisor is selected during OS installation.

d. Add the following commands to the end of the boot configuration file content.

modprobe -r pciehp > /dev/null 2>&1

modprobe pciehp pciehp_force=1

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Run the vim /boot/grub/menu.lst command to open the boot configuration file.

g. Enter i.

h. In the boot configuration file, add pci=pcie_bus_perf to the boldface part shown in

the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) ###Don't change this comment - YaST2 identifier: Original name: xen### title Xen -- SUSE Linux Enterprise Server 11 SP4 - 3.0.101-63 root (hd0,1) kernel /boot/xen.gz vga=mode-0x314 module /boot/vmlinuz-3.0.101-63-xen root=/dev/sda2 resume=/dev/sda1 splash=silent showopts vga=0x314 pci=pcie_bus_perf module /boot/initrd-3.0.101-63-xen

###Don't change this comment - YaST2 identifier: Original name: linux### title SUSE Linux Enterprise Server 11 SP4 - 3.0.101-63 root (hd0,1) kernel /boot/vmlinuz-3.0.101-63-default root=/dev/sda2 resume=/dev/ sda1 splash=silent crashkernel=256M-:128M showopts vga=0x314

pci=pcie_bus_perf

initrd /boot/initrd-3.0.101-63-default

###Don't change this comment - YaST2 identifier: Original name: failsafe### title Failsafe -- SUSE Linux Enterprise Server 11 SP4 - 3.0.101-63 root (hd0,1) kernel /boot/vmlinuz-3.0.101-63-default root=/dev/sda2 showopts ide=nodma apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe vga=0x314 pci=pcie_bus_perf initrd /boot/initrd-3.0.101-63-default …(Some code omitted.)

Page 68

ES3000 V3 NVMe PCIe SSD User Guide

i. Press Esc, enter :wq, and press Enter to save the edits.

j. Restart the OS for the settings to take effect.

k. Go to Notifying the OS.

l Ubuntu (14.04 or 15.04)

The following procedure uses Ubuntu 14.04 as an example:

a. Log in to the operating system (OS) as an administrator.

b. Run the vim /boot/grub/grub.cfg command to open the boot configuration file.

c. Enter i.

d. In the boot configuration file, add pciehp.pciehp_force=1 pci=pcie_bus_perf to

4 Hot Plug

NOTE

Whether the file content includes "title Xen" depends on whether Xen Hypervisor is selected during OS installation.

the boldface part shown in the following:

Between the added content and its preceding content, a space is required and no line feed is allowed.

…(Some code omitted.) ### BEGIN /etc/grub.d/10_linux ### function gfxmode { set gfxpayload="${1}" if [ "${1}" = "keep" ]; then set vt_handoff=vt.handoff=7 else set vt_handoff= fi } if [ "${recordfail}" != 1 ]; then if [ -e ${prefix}/gfxblacklist.txt ]; then if hwmatch ${prefix}/gfxblacklist.txt 3; then if [ ${match} = 0 ]; then set linux_gfx_mode=keep else set linux_gfx_mode=text fi else set linux_gfx_mode=text fi else set linux_gfx_mode=keep fi else set linux_gfx_mode=text fi export linux_gfx_mode menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-b7889e8e-dbc4-4fe9-bf5cdab9c20a93f8' { recordfail load_video gfxmode $linux_gfx_mode insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 -hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 b7889e8e-dbc4-4fe9bf5c-dab9c20a93f8 else search --no-floppy --fs-uuid --set=root b7889e8e-dbc4-4fe9-bf5cdab9c20a93f8 fi

Page 69

ES3000 V3 NVMe PCIe SSD User Guide

e. Press Esc, enter :wq, and press Enter to save the edits.

f. Restart the OS for the settings to take effect.

g. Go to Notifying the OS.

4 Hot Plug

linux /boot/vmlinuz-3.13.0-24-generic root=UUID=b7889e8edbc4-4fe9-bf5c-dab9c20a93f8 ro pciehp.pciehp_force=1 pci=pcie_bus_perf initrd /boot/initrd.img-3.13.0-24-generic } submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinuxadvanced-b7889e8e-dbc4-4fe9-bf5c-dab9c20a93f8' { menuentry 'Ubuntu, with Linux 3.13.0-24-generic' --class ubuntu -class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.13.0-24-generic-advanced-b7889e8e-dbc4-4fe9-bf5cdab9c20a93f8' { recordfail load_video gfxmode $linux_gfx_mode insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 b7889e8e-dbc4-4fe9bf5c-dab9c20a93f8 else search --no-floppy --fs-uuid --set=root b7889e8e-dbc4-4fe9bf5c-dab9c20a93f8 fi echo 'Loading Linux 3.13.0-24-generic ...' linux /boot/vmlinuz-3.13.0-24-generic root=UUID=b7889e8edbc4-4fe9-bf5c-dab9c20a93f8 ro pciehp.pciehp_force=1 pci=pcie_bus_perf echo 'Loading initial ramdisk ...' initrd /boot/initrd.img-3.13.0-24-generic } menuentry 'Ubuntu, with Linux 3.13.0-24-generic (recovery mode)' -class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.13.0-24-generic-recovery-b7889e8edbc4-4fe9-bf5c-dab9c20a93f8' { recordfail load_video insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1

pci=pcie_bus_perf

echo 'Loading initial ramdisk ...' initrd /boot/initrd.img-3.13.0-24-generic } } ### END /etc/grub.d/10_linux ### …(Some code omitted.)

----End

Page 70

ES3000 V3 NVMe PCIe SSD User Guide

Notifying the OS

Step 1 Log in to the OS as the root user.

Step 2 Stop all services accessing the ES3000 V3 disk.

Step 3 If you have used the mount command to attach the disk partitions to certain paths, use the

umount command to detach the partitions.

Step 4 Run the following command to notify the OS to take power off the device.

$slot in the commands indicates the actual number of the slot where the device is attached.

echo 0 > /sys/bus/pci/slots/$slot/power

Step 5 if the OS uses the irqbalance service to balance CPU interrupts, restart the service by using

one of the following commands after performing a hot plug:

l For Red Hat 6 or CentOs 6, run service irqbalance restart.

l For SLES 11, run service irq_balancer restart.

l For SLES 12 or Red Hat 7, run systemctl restart irqbalance.service.

4 Hot Plug

NOTE

l For Red Hat 6, CentOs 6, or SLES 12, you can add IRQBALANCE_ARGS="--hintpolicy=exact" to the

end of the /etc/sysconfig/irqbalance file content to better balance interrupts.

l To configure the irqbalance service in other OSs, refer to documents related to the OSs.

----End

4.2.2 Orderly Hot Plug of a Disk in Windows

Operation Scenario

Notify the Windows OS before hot removing the ES3000 V3 disk.

Preparations

l You have correctly installed the ES3000 V3 disk on the server.

l You have installed the Windows tool package. For details, see 3.7.2 Installing the Tool

Package (Windows).

Procedure

Step 1 Log in to the OS as an administrator or administrator group member.

Step 2 Stop all services accessing the ES3000 V3 disk.

Step 3 Double-click the HioadmGUI_DotNet_x.0.exe file in the tool installation directory to

display the Windows GUI window, as shown in Figure 4-1.

l If the .NET Framework version is 2.0 or 3.5, double-click the

HioadmGUI_DotNet_2.0.exe file in the tool installation directory.

l If the .NET Framework version is 4.0 or 4.5, double-click the

HioadmGUI_DotNet_4.0.exe file in the tool installation directory.

Page 71

ES3000 V3 NVMe PCIe SSD User Guide

Figure 4-1 Basic information

4 Hot Plug

Step 4 Click eject in Figure 4-1 and then click eject succeed OK.

If a success message is displayed, you can hot remove the ES3000 V3 disk. If a failure message indicating the OS is accessing the ES3000 V3 disk, stop related services before clicking eject again. You can hot remove the ES3000 V3 disk only when a success message is displayed.

----End

Page 72

ES3000 V3 NVMe PCIe SSD User Guide

About This Chapter

This topic describes how to perform hardware maintenance, software upgrades, and troubleshooting for the ES3000 V3.

5 Maintenance and Upgrade

5.1 Uninstalling the Driver

This topic describes how to uninstall the ES3000 V3 driver on various operating systems (OSs).

5.2 Upgrading the Driver

This section describes how to upgrade the ES3000 V3 driver in different operating systems (OSs).

5.3 Uninstalling, and Upgrading the Tool Package

5.4 Installing and Uninstalling Citrix System Drivers and Tools

5.5 Upgrading the Controller Firmware

This section describes how to upgrade the ES3000 V3 controller firmware in different operating systems (OSs).

5.6 Querying Health Status

5.7 Information Collection

5.8 Securely Erasing SSD Data

5.9 Querying SSD Lifespans

5.10 Common Faults

5.11 Indicator Status Descriptions

5.12 Removing an ES3000 V3

5.1 Uninstalling the Driver

This topic describes how to uninstall the ES3000 V3 driver on various operating systems (OSs).

Page 73

ES3000 V3 NVMe PCIe SSD User Guide

5.1.1 Uninstalling the Driver in Linux

Operation Scenario

If you want to stop using or replace the ES3000 V3, uninstall the driver first.

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Stop or exit the programs or files that are accessing the ES3000 V3.

Step 3 Run the uninstallation command based on the OS type.

l For RHEL, CentOS, or OEL, run the rpm -e kmod-hiodriver command.

l For SLES or UVP, run the rpm -e hiodriver-kmp-kernel feature command. Example:

rpm -e hiodriver-kmp-default

l For Ubuntu or Debian, run the dpkg -r hiodriver command.

Step 4 Check whether the following message is displayed. The message indicates that the installed

NVMe driver fails to be uninstalled.

Warning: fail to uninstall the installed nvme kernel module. Please uninstall the installed nvme kernel module manually or reboot the system.

l If yes, go to Step 7.

l If no, go to Step 5.

5 Maintenance and Upgrade

Step 5 Check whether the following message is displayed. The message indicates that the NVMe

driver built in the OS fails to be loaded.

Warning: fail to install the in-kernel nvme kernel module. Please install the inkernel nvme kernel module manually or reboot the system.

l If yes, go to Step 6.

l If no, go to Step 10.

Step 6 Check whether restarting the OS will affect services.

l If yes, go to Step 9.

l If no, restart the OS and go to Step 10.

Step 7 Check whether restarting the OS will affect services.

l If yes, go to Step 8.

l If no, restart the OS and go to Step 10.

Step 8 Run the modprobe -r nvme command to uninstall the driver. During the uninstallation, check

whether a message is displayed indicating that the driver is in use. If yes, the uninstallation fails.

l If yes, restart the OS and go to Step 10.

l If no, go to Step 9.

Step 9 Run the modprobe nvme command to install the open-source driver. Check whether a

warning or error is displayed during the installation.

l If yes, restart the OS and go to Step 10.

l If no, go to Step 10.

Page 74

ES3000 V3 NVMe PCIe SSD User Guide

Step 10 Run the modinfo nvme command to check the driver author information to determine

whether the driver is successfully installed.

The following command output indicates that the driver is successfully uninstalled.

modinfo: could not find module nvme

The following author in the command output also indicates that the driver is successfully uninstalled.

author: Matthew Wilcox <[email protected]>

l If yes, no further action is required.

l If no, go to Step 11.

Step 11 Contact Huawei technical support for help.

----End

5.1.2 Uninstalling the Driver in Windows

Operation Scenario

If you want to stop using or replace the ES3000 V3, uninstall its driver first.

5 Maintenance and Upgrade

Procedure

Uninstall the ES3000 V3 driver by following the procedure described in this topic.

Do not uninstall the ES3000 V3 driver by using Device Manager. If you have uninstalled the ES3000 V3 driver by using Device Manager, you still need to uninstall the driver by following the procedure described in this topic before installing the driver again. To install the driver, see 3.6.2 Installing the ES3000 V3 Driver in Windows.

l Uninstall the driver by using Control Panel.

a. Log in to the OS as an administrator or a member of the administrator group.

b. Stop the programs that are accessing the ES3000 V3.

c. Choose Start > Control Panel.

The Control Panel window is displayed.

d. Click Uninstall a program.

The Uninstall or change a program window is displayed, as shown in Figure 5-1.

Figure 5-1 Uninstall or change a program

e. Right-click the ES3000 V3 driver, and choose Uninstall/Change from the shortcut

menu.

Page 75

ES3000 V3 NVMe PCIe SSD User Guide

f. Click Yes to uninstall the ES3000 V3 driver and its components.

l Uninstall the driver by using the installation package.

a. Log in to the OS as an administrator or a member of the administrator group.

b. Stop the programs that are accessing the ES3000 V3.

c. Double-click the driver installation file in .msi format and select the uninstallation

5 Maintenance and Upgrade

A confirmation dialog box is displayed, as shown in Figure 5-2.

Figure 5-2 Confirming the uninstallation

option.

Uninstall the driver by following on-screen instructions.

----End

5.1.3 Uninstalling the Driver in ESXi

Operation Scenario

If you want to stop using or replace the ES3000 V3, uninstall the driver first.

Procedure

Step 1 Log in to the OS as an administrator or a member of the administrator groupo ,hadeo.

Step 2 Stop or exit the programs or files that are accessing the ES3000 V3.

Step 3 Run the following command to uninstall the ES3000 V3 driver package:

esxcli software vib remove -n hiodriver

Step 4 After the driver is uninstalled, run the reboot command to restart the OS.

Step 5 After the restart, run the vmkload_mod -l | grep hiodriver command. The following

command output indicates that the driver has been uninstalled:

[root@esxi113:~] vmkload_mod -l | grep hiodriver [root@esxi113:~]

----End

5.2 Upgrading the Driver

This section describes how to upgrade the ES3000 V3 driver in different operating systems (OSs).

Page 76

ES3000 V3 NVMe PCIe SSD User Guide

5.2.1 Upgrading the Driver in Linux

Operation Scenario

You can upgrade the driver after a new version is released.

The Linux driver package naming rules are as follows:

l RHEL, CentOS, or OEL driver package name: kmod-hiodriver-Driver version-

Supported kernel version.System distributor.Hardware platform.rpm, such as kmod-

hiodriver-3.3.5.2-3.10.0_123.el7.centos.x86_64.rpm, kmodhiodriver-3.3.5.2-3.10.0_123.el7.redhat.x86_64.rpm, or kmodhiodriver-3.3.5.2-3.10.0_123.el7.oracle.x86_64.rpm.

l SLES driver package name: hiodriver-kmp-Kernel feature-Driver version-Supported

kernel version.System distributor.Hardware platform.rpm, such as hiodriver-kmp-

default-3.3.5.2-3.12.28_4.suse12.x86_64.rpm.

l Ubuntu or Debian driver package name: hiodriver -Driver version-Supported kernel

version.System distributor.Hardware platform.deb, such as

hiodriver-3.3.5.3-3.19.0_15_generic.ubuntu.amd64.deb.

l UVP driver package name: hiodriver-kmp-xen-Driver version-Supported kernel

version.System distributor.Hardware platform.rpm.

5 Maintenance and Upgrade

Preparations

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Go to the directory that stores the ES3000 V3 driver package, for example, /home.

Step 3 Run the upgrade command based on the OS type.

Step 4 Check whether the following message indicating an incorrect upgrade package is displayed.

Before upgrading the driver, check that:

l You have downloaded the new driver package to the server and verified the package

integrity.

l You have checked the new driver version.

l For RHEL, CentOS, OEL, or SLES, run rpm -Uvh driver package name.

An example package name for CentOS is rpm -Uvh kmodhiodriver-3.30.60.1-2.6.32_431.el6.centos.x86_64.rpm.

l For Ubuntu or Debian, run dpkg -i driver package name.

An example package name for Ubuntu is dpkg -i hiodriver-3.3.5.3-3.19.0_15_generic.ubuntu.amd64.deb.

Error: The installing package does not match the current OS. Please execute command: rpm -e kmod-hiodriver to uninstall the installing rpm.

l If yes, go to Step 5.

l If no, go to Step 6.

Step 5 Uninstall the installed package as prompted. Upload the correct upgrade package for the OS

to the server and perform Step 2 again to install the package.

Page 77

ES3000 V3 NVMe PCIe SSD User Guide

Step 6 Check whether the following message is displayed. The message indicates that the original

NVMe driver fails to be uninstalled during the upgrade.

Warning: fail to uninstall the old nvme kernel module. Please uninstall the old nvme kernel module manually or reboot the system.

l If yes, go to Step 9.

l If no, go to Step 7.

Step 7 Check whether the following message is displayed. The message indicates that the new

NVMe driver fails to be installed during the upgrade.

Warning: fail to install the new nvme kernel module. Please install the new nvme kernel module manually or reboot the system.

l If yes, go to Step 8.

l If no, go to Step 12.

Step 8 Check whether restarting the OS will affect services.

l If yes, go to Step 11.

l If no, restart the OS and go to Step 12.

Step 9 Check whether restarting the OS will affect services.

l If yes, go to Step 10.

l If no, restart the OS and go to Step 12.

5 Maintenance and Upgrade

Step 10 Run the modprobe -r nvme command to uninstall the original driver. During the

uninstallation, check whether a message is displayed indicating that the driver is in use. If yes, the uninstallation fails.

l If yes, restart the OS and go to Step 12.

l If no, go to Step 11.

Step 11 Run the modprobe nvme command to install the new driver. Check whether a warning or

error is displayed during the installation.

l If yes, restart the OS and go to Step 12.

l If no, go to Step 12.

Step 12 Run the modinfo nvme command to check the version information to determine whether the

driver is successfully upgraded.

If the version number in the command output is the target version number, the upgrade is successful.

The following is an example command:

version: 3.3.5.3

l If yes, go to Step 13.

l If no, go to Step 15.

Step 13 If an SSD device has been installed on the server, check whether the driver has initialized the

NVMe device.

Check whether the NVMe device exists in the /dev directory.

l If yes, go to Step 14.

l If no, go to Step 15.

Step 14 if the OS uses the irqbalance service to balance CPU interrupts, restart the service by using

one of the following commands after re-loading the driver module:

Page 78

ES3000 V3 NVMe PCIe SSD User Guide

l For RHEL 6 or CentOs 6, run service irqbalance restart.

l For SLES 11, run service irq_balancer restart.

l For SLES 12 or RHEL 7, run systemctl restart irqbalance.service.

NOTE

l For RHEL 6, CentOs 6, or SLES 12, you can add IRQBALANCE_ARGS="--hintpolicy=exact" to the

end of the /etc/sysconfig/irqbalance file content to better balance interrupts.

l To configure the irqbalance service in other OSs, refer to documents related to the OSs.

Check whether errors are reported after running the command for restarting the irqbalance service.

l If yes, go to Step 15.

l If no, no further action is required.

Step 15 Contact Huawei technical support for help.

----End

5.2.2 Upgrading the ES3000 V3 Driver in Windows

5 Maintenance and Upgrade

Operation Scenario

You can upgrade the ES3000 V3 driver after a later version is released.

Preparations

Before upgrading the driver, check that:

l You have downloaded the later driver software to the server, and verified the package

integrity.

l You have checked the driver version.

Procedure

Step 1 Uninstall the existing driver. For details, see 5.1.2 Uninstalling the Driver in Windows.

Step 2 Install the new driver. For details, see 3.6.2 Installing the ES3000 V3 Driver in Windows.

----End

5.2.3 Upgrading the Driver in ESXi

Operation Scenario

You can upgrade the driver after a new version is released.

Preparations

Before upgrading the driver, download the new driver package to the server and verify the package integrity.

Page 79

ES3000 V3 NVMe PCIe SSD User Guide

5 Maintenance and Upgrade

Procedure

Step 1 Run the hioadm -v command to check whether the tool version is later than 3.3.7.5.

l If yes, go to Step 3.

l If no, go to Step 2.

Step 2 Run the following command to uninstall the hioadm tool:

esxcli software vib remove -n hioadm

Step 3 Install the new driver. For details, see 3.6.3 Installing the ES3000 V3 Driver in ESXi.

NOTE

Install the new driver package directly. The system will automatically replace the old driver.

----End

5.3 Uninstalling, and Upgrading the Tool Package

You can download the software package as follows: log in to Huawei Enterprise support website http://support.huawei.com/enterprise, choose Select a Product to Get Support > IT > Server > TaiShan > ES3000 V3, click Downloads, and download the required software package. The software package contains host tools (including hioadm) and Huawei's NVMe driver.

5.3.1 Uninstalling the Tool Package (Linux)

Operation Scenario

Uninstall the tool package to free up space when ES3000 V3 management is not needed.

Procedure

Step 1 Log in to the operating system (OS) as an administrator.

Step 2 Run the uninstallation command based on the OS type.

l For RHEL, CentOS, OEL, or SLES, run the rpm -e hioadm command.

l For Ubuntu or Debian, run the dpkg -r hioadm command.

----End

5.3.2 Uninstalling the Tool Package (Windows)

Operation Scenario

Uninstall the tool package to free up space when ES3000 V3 management is not needed.

Procedure

Step 1 Log in to the operating system (OS) as an administrator or administrator group member.

Step 2 Choose Start > Control Panel.

The Control Panel window is displayed.

Page 80

ES3000 V3 NVMe PCIe SSD User Guide

Step 3 Click Uninstall a program.

The Uninstall or change a program window is displayed, as shown in Figure 5-3.

Figure 5-3 Uninstall or change a program

Step 4 Find and right-click the tool package, and choose Uninstall.

A confirmation dialog box is displayed, as shown in Figure 5-4.

Figure 5-4 Confirming the uninstallation

5 Maintenance and Upgrade

Step 5 Click Yes.

----End

5.3.3 Uninstalling the Tool (ESXi)

Procedure

Step 1 The tool is integrated into the driver package. Uninstalling the driver package also uninstalls

the tool. You can uninstall the driver package by referring to 5.1.3 Uninstalling the Driver in

ESXi.

----End

5.3.4 Upgrading the Tool Package (Linux)

Operation Scenario

Upgrade the tool package to better manage the ES3000 V3 when a new version is available.

The tool package naming rules are as follows:

l For RHEL, CentOS, OEL, or SLES, the package name is hioadm-version number-

release number.hardware architecture.rpm, for example, hioadm-3.3.5.2-1.x86_64.rpm.

l For Ubuntu or Debian, the package name is hioadm-version number-release

number.hardware architecture.deb, for example, hioadm-3.3.5.2-1.amd64.deb.

Page 81

ES3000 V3 NVMe PCIe SSD User Guide

Preparations

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the new tool package version and verified its integrity.

l You have used WinSCP or other software to upload the tool package to the server, such

as the /home directory.

Procedure

Step 1 Log in to the operating system (OS) as an administrator.

Step 2 Go to the directory that stores the tool package, for example, /home.

Step 3 Run the upgrade command based on the OS type.

l For RHEL, CentOS, OEL, or SLES, run rpm -Uvh tool package name.

Example: rpm -Uvh hioadm-3.3.6.1-1.x86_64.rpm

l For Ubuntu or Debian, run dpkg -i tool package name.

Example: dpkg -i hioadm-3.30.60.1-1.amd64.deb

5 Maintenance and Upgrade

Step 4 Run the following command to obtain help information about the tool:

hioadm -h or hioadm --help

----End

5.3.5 Upgrading the Tool Package (Windows)

Operation Scenario

Upgrade the tool package to better manage the ES3000 V3 when a new version is available.

Two upgrade methods are available:

l Uninstall the old version and install the new version.

l Upgrade the old version directly.

Preparations

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the new tool package version and verified its integrity.

l You have uploaded the tool package to the server by using Windows Secure Copy

(WinSCP) or other software.

Procedure

Step 1 Uninstall the old tool package. For details, see 5.3.2 Uninstalling the Tool Package

(Windows).

Step 2 Install the new tool package. For details, see 3.7.2 Installing the Tool Package (Windows).

----End

Page 82

ES3000 V3 NVMe PCIe SSD User Guide

5 Maintenance and Upgrade

5.3.6 Upgrading the Tool (ESXi)

Operation Scenario

Upgrade the tool to better manage the ES3000 V3 when a new version is available.

Procedure

Step 1 The tool is integrated into the driver package. You can install the driver package by referring

to 5.2.3 Upgrading the Driver in ESXi.

----End

5.4 Installing and Uninstalling Citrix System Drivers and Tools

5.4.1 Installing Citrix Drivers and Tools

Scenario

Preparations

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Go to the directory that stores the .iso package, for example, /home.

Step 3 Mount the .iso package to a directory, for example, /tmp/citrix.

Step 4 Go to the directory to which the .iso package is mounted, for example, /tmp/citrix.

On Citrix, drivers and tools are provided in the .iso format.

The tool package name format is as follows:

hiodriver-<tool version>-<OS version>.iso, for example, hiodriver-3.3.9.1-6u2.iso.

l You have correctly installed the ES3000 V3 on the server.

l You have downloaded the .iso package and verified its integrity.

l You have used WinSCP or other software to upload the .iso package to the server, such

as the /home directory.

mount -o loop hiodriver-3.3.9.1-6u2.iso /tmp/citrix

cd /tmp/citrix

Step 5 Run install.sh in /tmp/citrix to install the drivers and tools.

./install.sh

Step 6 Run the following commands to obtain the tool help information or query the installed drivers

and tools:

hioadm –h

Page 83

ES3000 V3 NVMe PCIe SSD User Guide

hioadm –help

rpm –qa | grep hiodriver

----End

5.4.2 Uninstalling Citrix Drivers and Tools

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Run the following command to query the hiodriver packages installed in the OS:

rpm –qa | grep hiodriver

The following provides an example:

[root@xenserver-citrix6u2sp1 ~]# rpm -qa | grep hiodriver hiodriver-modules-xen-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.0-20170114051700 hiodriver-data-1.0-20170114051700 hiodriver-modules-kdump-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.0-20170114051700

Step 3 Run the following command to uninstall the hiodriver packages one by one:

rpm –e xxxxxx

5 Maintenance and Upgrade

The following provides an example:

[root@xenserver-citrix6u2sp1 ~]# rpm -e hiodriver-modulesxen-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.0-20170114051700

[root@xenserver-citrix6u2sp1 ~]# rpm -e hiodriver-data-1.0-20170114051700 NOTE: There are some user files under the directory /opt/hio. If you don't need them, you can delete them manually. Uninstall ES3000 V3 hiodriver-data-1.0 successfully. [root@xenserver-citrix6u2sp1 ~]# rpm -e hiodriver-modules-

kdump-2.6.32.43-0.4.1.xs1.8.0.835.170778-1.0-20170114051700

Step 4 Run the following command again to check that all the hiodriver packages are uninstalled:

rpm –qa | grep hiodriver

----End

5.5 Upgrading the Controller Firmware

This section describes how to upgrade the ES3000 V3 controller firmware in different operating systems (OSs).

NOTE

The ES3500, ES3600P, and ES3600C series use the same drivers and firmware.

5.5.1 Upgrading the Controller Firmware in Linux

Operation Scenario

You can upgrade the controller firmware when a new version is released.

Preparations

Before upgrading the firmware, check that:

Page 84

ES3000 V3 NVMe PCIe SSD User Guide

l Stop or exit the ES3000 V3 SSD applications or services, and umount the file system

that uses the SSD device.

l The firmware upgrade does not damage data on SSDs. However, you are advised to back

up important data on SSDs.

l You have downloaded the firmware package to the server, and verified the package

integrity.

l You have uploaded the controller firmware file ES3000V3_FW_2.43.bin to the server.

Procedure

Step 1 Log in to the OS as an administrator.

Step 2 Run the following command to query the SSD controller (such asnvme0) firmware version:

hioadm updatefw -d nvme0

The command output contains the following controller firmware version:

slot version activation slot1 2.09 slot2 2.10 current slot3 2.11 next

5 Maintenance and Upgrade

NOTE

The version number to the left of next indicates the version that takes effect after the next reset. The version number to the left of current indicates the version in use.

Step 3 Access the folder where the controller firmware is stored, for example, /home.

NOTE

You can use an absolute or relative path when upgrading the driver. If you do not perform this step, you must enter an absolute path in the command in the next step.

Step 4 Run the following command to upgrade the controller firmware.

l Method 1: hioadm updatefw -d nvme0 -f ES3000V3_FW_2.43.bin, then go to step 5,

6, 7.

l Method 2: hioadm updatefw -d nvme0 -f ES3000V3_FW_2.43.bin -a 1, go to step 5,

then go to #es3000_user_050/step7 directly.

Step 5 Enter y and press Enter.

Step 6 After the upgrade, run the reboot command to restart the OS.

Step 7 Run the following command to check whether the SSD device controller firmware is

upgraded:

hioadm updatefw -d nvme0

l If yes, no further action is required.

l If no, go to Step 8.

Step 8 Contact Huawei technical support for help.

----End

Page 85

ES3000 V3 NVMe PCIe SSD User Guide

5 Maintenance and Upgrade

5.5.2 Upgrading the Controller Firmware in Windows

Operation Scenario

You are advised to upgrade the controller firmware immediately after a new version is released.

Preparations

Before upgrading the firmware, check that:

l You have uploaded the ES3000 V3 software package to a directory on the server, for

example, the D:\ directory.

l Stop or exit the ES3000 V3 SSD applications or services, and uninstall the file system

that uses the SSD device (make the to-be-upgraded SSD to be offline in the disk manager).

l The firmware upgrade does not damage data on SSDs. However, you are advised to back

up important data on SSDs.

Procedure

Step 1 Log in to the OS as an administrator or a member of the administrator group.

Step 2 Access the software package directory, such as the D:\ directory.

Step 3 Install the Windows driver and tool package. For details, see 3.6.2 Installing the ES3000 V3

Step 4 Choose Start > Run.

Step 5 Run the cd command to go to the directory where the hioadm.exe file is stored.

Step 6 Run the following command to query the SSD controller (for example, nvme0) firmware

Step 7 Upgrade the controller firmware.

l You have downloaded the latest firmware file to a specified folder on the server and

verified the file integrity.

l You have upgraded the firmware driver to a compatible version.

Driver in Windows and 3.7.2 Installing the Tool Package (Windows).

The hioadm.exe file will be generated in the installation directory.

The Windows CLI is displayed.

version:

hioadm updatefw -d nvme0

hioadm updatefw -d nvme0 -f D:\ES3000V3_FW_2.43.bin

Step 8 Restart the OS after the upgrade.

Step 9 Re-log in to the OS, run the hioadm.exe file again, and run the following command to check

whether the SSD controller firmware has been upgraded:

hioadm updatefw -d nvme0

l If yes, no further action is required.

l If no, go to Step 10.

Page 86

ES3000 V3 NVMe PCIe SSD User Guide

Step 10 Contact Huawei technical support for help.

----End

5.5.3 Upgrading the Controller Firmware in ESXi

Operation Scenario

You can upgrade the controller firmware when a new version is released.

Preparations

Before upgrading the firmware, check that:

l Stop or exit the ES3000 V3 SSD applications or services.

l The firmware upgrade does not damage data on SSDs. However, you are advised to back

up important data on SSDs.

l You have downloaded the firmware package to the server, and verified the package

integrity.

l You have uploaded the controller firmware file ES3000V3_FW_2.43.bin to the server.

5 Maintenance and Upgrade

Procedure

Step 1 Log in to the OS as an administrator or a member of the administrator group.

Step 2 Run the following command to query the SSD controller (such asnvme0) firmware version:

Step 3 Navigate to the folder where the controller firmware is stored, for example, /tmp.

Step 4 Upgrade the controller firmware.

hioadm updatefw -d nvme0

The command output contains the following controller firmware version:

slot version activation 1 1.20 2 1.20 next 3 1.20 current

NOTE

The version number to the left of next indicates the version that takes effect after the next reset. The version number to the left of current indicates the current version in use.

NOTE

You can use an absolute path or a relative path when upgrading the driver. If you do not perform this step, you must enter an absolute path in the command in the next step.

hioadm updatefw -d nvme0 -f ES3000V3_FW_2.43.bin

Step 5 Enter y and press Enter.

Step 6 After the upgrade, run the reboot command to restart the OS.

Step 7 Run the following command to check whether the SSD device controller firmware is

upgraded:

hioadm updatefw -d nvme0

Page 87

ES3000 V3 NVMe PCIe SSD User Guide

l If yes, no further action is required.

l If no, go to Step 8.

Step 8 Contact Huawei technical support for help.

----End

5.6 Querying Health Status

Operation Scenario

Check the SSD health status.

Procedure

Step 1 Log in to the OS as an administrator or a member in the administrator group.

Step 2 Run the following command to query the health status of the SSD controller (such as nvme0):

hioadm info -d nvme0

5 Maintenance and Upgrade

device status in the command output indicates the SSD controller health status.

l healty indicates that the SSD is healthy.

l warning indicates that the SSD is faulty.

[root@localhost tool]# hioadm info -d nvme0

Namespace<1> size: 1.6TB, 1600321314816Byte formatted LBA size: 512 Byte formatted metadata size: 0 Byte

maximum capacity : 1.6TB current capacity : 1.6TB volatile write cache : Enable serial number : 0503023HDCN107C80013 model number : HWE32P430016M00N firmware version : 2.14 NVMe version : 1.2 device status : healthy

----End

5.7 Information Collection

5.7.1 Collecting SSD Information

Operation Scenario

Collect SSD information when maintaining SSDs.

Procedure

Step 1 Log in to the OS as an administrator or a member in the administrator group.

Page 88

ES3000 V3 NVMe PCIe SSD User Guide

Step 2 Run the following command to query basic information about the SSD controller (such as

nvme0):

hioadm info -d nvme0

The command output is similar to the following:

[root@localhost tool]# hioadm info -d nvme0

Namespace<1> size: 1.6TB, 1600321314816Byte formatted LBA size: 512 Byte formatted metadata size: 0 Byte

Step 3 Run the following command to query the SMART information of the SSD controller (such as

nvme0):

hioadm info -d nvme0 –s

5 Maintenance and Upgrade

The command output is similar to the following:

[root@localhost tool]# hioadm info -d nvme0 -s critical warning : no warning composite temperature : 308 degrees Kelvin (35 degrees Celsius) available spare : 100% available spare threshold : 10% percentage used : 0% data units read : 68.8 MB data units written : 0.0 MB host read commands : 17748 host write commands : 0 controller busy time : 0 mins power cycles : 89 times power on hours : 1164 h unsafe shutdowns : 35 times media and data integrity errors: 0 number of error information log entries: 0 warning composite temperature time: 0 min critical composite temperature time: 0 min data status : OK

----End

5.7.2 Collecting SSD Logs

Operation Scenario

Collect SSD information when maintaining SSDs.

Procedure

Step 1 Log in to the OS as an administrator or a member in the administrator group.

Step 2 Run the following command to collect the SSD controller (such as nvme0) log information:

hioadm log -d nvme0 -a

The command output shows the log collecting result and file path.

Page 89

ES3000 V3 NVMe PCIe SSD User Guide

[root@localhost tool]# hioadm log -d nvme0 -a

firmware key log saving..........OK

save firmware key log file successed(/opt/hio/nvme0_key.log)

firmware all log saving..........OK

save firmware all log file successed(/opt/hio/nvme0_fw.log)

firmware core dump log saving..........OK

save firmware core dump log file successed(/opt/hio/nvme0_coredump.log)

other log saving..........OK

save other logs file successed(/opt/hio/nvme0_other.log) logs has been saved.

----End

5.7.3 One-Click Log Collection

Scenario

Collect SSD and system operation logs when maintaining SSD devices.

Prerequisite

You have downloaded the software package from the Software Download tab at http:// support.huawei.com/enterprise (Product Support > Enterprise Data Center > Servers > Acceleration Component > ES3000 V3.

5 Maintenance and Upgrade

Linux

Step 1 Copy host_info_collect.sh from /linux/tool in the package to /opt/hio.

Step 2 Log in to the OS as an administrator or a member in the administrator group.

Step 3 Run the bash /opt/hio/host_info_collect.sh command.

Step 4 Obtain the log file hostInfoCollect.zip from the /opt/hio directory.

Step 5 (Optional) Delete the log file after use to free disk space.

Windows

Step 1 Copy diagnostic_win.bat from /windows/Windows_Tool in the package to C:\Program Files

Step 2 Log in to the OS as an administrator or a member in the administrator group.

Install the hioadm tool before performing one-click log collection. For details about how to install the tool, see section 3.7 Installing the Tool Package.

----End

(x86)\hioadm.

Step 3 Double-click diagnostic_win.ba in C:\Program Files (x86)\hioadm\.

Step 4 Obtain all log files from the C:\Program Files (x86)\hioadm\ directory.

Step 5 (Optional) Delete the log file after use to free disk space.

----End

Page 90

ES3000 V3 NVMe PCIe SSD User Guide

ESXi

Step 1 Copy host_okc.sh from /vmware in the package to /scratch/hio/.

Step 2 Log in to the OS as an administrator or a member in the administrator group.

Step 3 Run the sh /scratch/hio/host_okc.sh command.

Step 4 Obtain all log files from the /scratch/hio/okc directory.

Step 5 (Optional) Delete the log file after use to free disk space.

----End

5.8 Securely Erasing SSD Data

Operation Scenario

Collect SSD logs when maintaining SSD devices.

5 Maintenance and Upgrade

Preparations

Before performing operations, stop all I/O services of the SSD and perform the following:

l For Linux OSs, unmount all logical disks.

l For ESXi OSs, delete all partitions of all logical disks.

l For Windows OSs, make all logical disks offline, as shown in Figure 5-5.

Figure 5-5 Making all logical disks offline

Procedure

Step 1 Log in to the OS as an administrator or a member in the administrator group.

Page 91

ES3000 V3 NVMe PCIe SSD User Guide

Step 2 Run the following command to securely erase data from the SSD (such as nvme0):

hioadm secure -d nvme0

The warning of the risks of data erasure is displayed.

[root@localhost tool]# hioadm secure -d nvme0

WARNING! The secure operation will clear all data. WARNING! You have selected the format operation. Proceed with the secure? (Y|N):

Step 3 After confirming the risks, enter y to start the secure erasure.

The operation result is displayed.

[root@localhost tool]# hioadm secure -d nvme0

WARNING! The secure operation will clear all data. WARNING! You have selected the format operation. Proceed with the secure? (Y|N): y

Formatting.... OK

secure operate successed. Please remove and then install the NVMe driver or reboot the OS.

----End

5 Maintenance and Upgrade

5.9 Querying SSD Lifespans

Scenario

Query the lifespan of an SSD.

Procedure

Step 1 Log in to the OS as an administrator or a member in the administrator group.

Step 2 Run the following command to query the SMART information of the SSD controller (such as

nvme0):

hioadm info -d nvme0 –s

The command output is similar to the following:

Page 92

ES3000 V3 NVMe PCIe SSD User Guide

Step 3 The percentage used parameter displays the used lifespan percentage of the SSD.

If the value of percentage used exceeds 100%, the SSD lifespan has expired, and Huawei's warranty service is unavailable. You need to back up data on the SSD as soon as possible and replace the SSD.

----End

5.10 Common Faults

5.10.1 Hardware Connection Faults or Other Faults

Hardware connection faults are mostly identified by indicators. If the yellow indicator of a device is on and the green indicator is off 30 seconds after a new disk is installed or during proper operating, hardware faults occur. See 5.11 Indicator Status Descriptions to identify the faults.

5 Maintenance and Upgrade

Typical fault causes are as follows:

l The disk is not securely installed.

l The disk is installed in a non-NVMe slot.

l A disk hardware port fault occurs.

5.10.2 Disk Identification Faults or Other Faults

Disk identification faults indicate scenarios where the host OS disk manager (for example, fdisk-l in Linux and the device manager in Windows) cannot properly identify ES3000 SSDs when hardware connections are correct. If disk identification faults occur, collect all logs. For details, see 5.7.3 One-Click Log Collection.

Typical fault causes are as follows:

l The NVMe driver is not installed on the OS.

l OS driver loading fails.

5.10.3 Disk Internal Faults or Other Faults

Disk internal faults indicate scenarios where disk internal alarms occur while the OS can properly identify disks. If such faults occur, the disks do not function properly or cannot be used. This topic uses an example to describe how to identify such faults.

Use Instance

Step 1 Determine the disk health status. For details, see 6.1.2 Querying Basic Information About a

Device.

# Query the basic information of nvme0.

[root@localhost tool]# hioadm info -d nvme0

Namespace<1> size: 1.6TB, 1600321314816Byte formatted LBA size: 512 Byte

Page 93

ES3000 V3 NVMe PCIe SSD User Guide

formatted metadata size: 0 Byte

device status in the command output indicates the SSD controller health status.

The value healthy indicates that the device is healthy.

The value warning indicates that an exception occurs on the device. For details, see Step 2.

Step 2 Determine the SMART status of the disk. For details, see 6.1.3 Querying the SMART

Information About a Device.

# Query the SMART information of nvme0.

5 Maintenance and Upgrade

Critical Warning: Critical exceptions occur on the device, and emergency handling is required.

Page 94

ES3000 V3 NVMe PCIe SSD User Guide

Table 5-1 Critical Warning parameters

Parameter Description Bit

5 Maintenance and Upgrade

critical warning Critical warning, such as

overtemperature and insufficient redundant space.

l 0: No warning.

l 1: The available spare

space has fallen below the threshold.

l 2: The temperature is

above an over temperature threshold or below an under temperature threshold.

l 3: The NVM subsystem

reliability has been degraded due to significant media related errors or any internal error that degrades NVM subsystem reliability.

l 4: The media has been

placed in read only mode.

l 5: The volatile memory

backup device has failed.

Typical fault causes are as follows:

l Available spare space below the threshold: Causes include excessive number of damaged

blocks.

l Temperature above the overtemperature threshold or below the undertemperature

threshold: Causes include disk temperature below 0°C or above 78°C.

l NVM subsystem reliability degraded due to internal errors: Causes include excessive

number of failed disk granules or internal subsystem operating exceptions.

l Media placed in read only mode: Causes include capacitor failures.

l Volatile memory backup device failure: Causes include capacitor voltage below 28 V or

above 35 V.

----End

Fault Diagnosis Instance

The available space is smaller than the threshold.

Step 1 Check whether the value of available spare in the SMART information exceeds 10%, if the

value is greater than or equal to 10%, stop using the disk and back up data immediately. Otherwise, go to Step 2.

Page 95

ES3000 V3 NVMe PCIe SSD User Guide

Step 2 Obtain disk logs by following instructions in 5.7.3 One-Click Log Collection and contact

Huawei technical support.

----End

The service life exceeds the threshold.

Step 1 Check whether the value of percentage used in the SMART information exceeds 100%, if the

value is greater than or equal to 100%, stop using the disk and back up data immediately. Otherwise, go to Step 2.

Step 2 Collect disk log information. For details, see 5.7.3 One-Click Log Collection. Contact

Huawei technical support.

----End

The temperature exceeds the threshold.

Step 1 Check that the server where the disk resides provides proper cooling. If the server reports no

temperature alarm and the fan modules are operating properly, go to Step 2.

Step 2 Check that the server where the disk resides provides proper cooling. If the server reports no

temperature alarm and the fan modules are operating properly, go to Step 2.

5 Maintenance and Upgrade

Step 3 Obtain disk logs by following instructions in 5.7.3 One-Click Log Collection and contact

Huawei technical support.

----End

l An internal error downgrades the NVM subsystem reliability.

l The medium is read-only.

l Device backup failed.

Step 1 Obtain disk logs by following instructions in 5.7.3 One-Click Log Collection and contact

Huawei technical support.

----End

5.11 Indicator Status Descriptions

5.11.1 ES3000 SSD Disk Indicator State Descriptions

Table 5-2 ES3000 SSD disk indicator state descriptions

Green Indicator

Yellow Indicator State Description

Off Off The NVMe SSD disk cannot

be detected.

On Off The NVMe SSD disk is

detected and working properly.

2 Hz Off Data access to the NVMe

SSD disk is in progress.

Page 96

ES3000 V3 NVMe PCIe SSD User Guide

Green Indicator Yellow Indicator State Description

Off 2 Hz The NVMe SSD disk is in

Off 0.5 Hz The NVMe SSD completes

On/Off On The NVMe SSD disk is

5 Maintenance and Upgrade

the hot swap process.

the hot swap process and is removable.

faulty.

5.11.2 ES3000 SSD Card Indicator State Descriptions

Table 5-3 ES3000 SSD card indicator state descriptions

Green Indicator

Off Off The NVMe SSD card is not

On Off The NVMe SSD card is

2 Hz Off Data access to the NVMe

Off On Loading the NVMe SSD

Off 0.5 Hz The NVMe SSD card

On Blinking Reserved

On On The NVMe SSD card is

Yellow Indicator State Description

5.12 Removing an ES3000 V3

powered on.

working.

SSD card is in progress.

card firmware fails.

firmware is being initialized.

faulty.

Operation Scenario

l Removing an ES3000 V3 NVMe PCIe SSD disk

– If the system does not support hot swap, the method for removing an NVMe PCIe

SSD disk is the same as that for removing a common hard disk. For details, see

5.12.1 Removing a Non-Hot-Swappable ES3000 V3 NVMe PCIe SSD Disk.

– If the system supports hot swap, hot-swap an NVMe PCIe SSD disk. For details,

see 5.12.2 Removing a Hot Swappable ES3000 V3 NVMe PCIe SSD Disk.

Page 97

ES3000 V3 NVMe PCIe SSD User Guide

Hot swap: Users are allowed to take out and replace faulty hard disks, PSUs, or cards when the system is not stopping or the power is not switched off. Surprised hot swap and orderly host swap are supported. For details, see 4 Hot Plug.

l Removing an ES3000 V3 NVMe PCIe SSD card

– If PCIe card slots are on a riser card, the SSD card needs to be removed from the

– If PCIe card slots are on the mainboard, the SSD card needs to be removed from the

Preparations

Before removing an ES3000 V3 NVMe PCIe SSD, make the following preparations:

l Wear ESD gloves.

Used to prevent ESD damage when you insert, remove, or hold the ES3000 V3 or a precision device.

l Back up data.

– Stop the access to a hot swappable SSD disk.

– Stop the services on the server and power off the server if you need to remove an

5 Maintenance and Upgrade

NOTE

riser card. For details, see 5.12.3 Removing an ES3000 V3 NVMe PCIe SSD

Card on a Riser Card.

mainboard. For details, see 5.12.4 Removing an ES3000 V3 NVMe PCIe SSD

Card from the Mainboard.

SSD card or a non-hot-swappable SSD disk.

5.12.1 Removing a Non-Hot-Swappable ES3000 V3 NVMe PCIe SSD Disk

Prerequisites

Wear ESD gloves.

Procedure

Step 1 Determine the position of the disk in the server and power off the server.

Step 2 Press the ejector release button. See (1) in Figure 5-6.

Page 98

ES3000 V3 NVMe PCIe SSD User Guide

Figure 5-6 Removing a disk

Step 3 Holding onto the disk ejector lever, pull out the disk. See (2) and (3) in Figure 5-6.

Step 4 Place the removed hard disk into an ESD bag.

5 Maintenance and Upgrade

----End

5.12.2 Removing a Hot Swappable ES3000 V3 NVMe PCIe SSD Disk

Prerequisites

Check that the current environment supports 4.1 OSs Supporting Hot Plug.

Wear ESD gloves.

Procedure

Step 1 Determine the position of the disk in the server and stop the services on the ES3000 V3

NVMe PCIe SSD disk to be removed.

Step 2 Perform operations described in 4.2 Orderly Hot Plug.

Step 3 Press the ejector release button. See (1) in Figure 5-7.

Page 99

ES3000 V3 NVMe PCIe SSD User Guide

Figure 5-7 Removing a disk

Step 4 Holding onto the disk ejector lever, pull out the disk. See (2) and (3) in Figure 5-7.

Step 5 Place the removed hard disk into an ESD bag.

5 Maintenance and Upgrade

----End

5.12.3 Removing an ES3000 V3 NVMe PCIe SSD Card on a Riser Card

Prerequisites

Wear ESD gloves.

Procedure

Step 1 Turn off the power to the server and remove the chassis cover.

Step 2 Remove all the cables from the riser card.

Step 3 Hold the upper part of an ES3000 V3 NVMe PCIe SSD card and remove it slowly. See

Figure 5-8.

Page 100

ES3000 V3 NVMe PCIe SSD User Guide

Figure 5-8 Removing an ES3000 V3 NVMe PCIe SSD card

5 Maintenance and Upgrade

Step 4 Place the removed PCIe card in an ESD bag.

----End

5.12.4 Removing an ES3000 V3 NVMe PCIe SSD Card from the Mainboard

Prerequisites

Wear ESD gloves.

Procedure

Step 1 Turn off the power to the server and remove the chassis cover.

Step 2 Open the latch on the PCIe card.

Step 3 Hold the upper part of an ES3000 V3 NVMe PCIe SSD card and remove it upwards slowly.

See Figure 5-9.