Sun Microsystems 3900, 6900 User Manual

Sun StorEdge™3900 and 6900

Series 2.0 Troubleshooting Guide

Sun Microsystems, Inc.
4150 Network Circle
Santa Clara, CA 95054 U.S.A.
650-960-1300

Part No. 816-5255-12
March 2003, Revision A

Send comments about this document to: [e-mail]

Copyright 2003Sun Microsystems, Inc.,4150 NetworkCircle, SantaClara, California95054, U.S.A.All rightsreserved.
Sun Microsystems, Inc.has intellectualproperty rightsrelating to technology embodied in the productthat isdescribed inthis document.In

particular,and without limitation, these intellectual property rightsmay includeone ormore ofthe U.S.patents listedat
http://www.sun.com/patents and one or moreadditional patentsor pendingpatent applicationsin theU.S. andin othercountries.

This document and the productto whichit pertainsare distributedunder licensesrestricting their use, copying, distribution, and
decompilation. No part of the product orof thisdocument maybe reproducedin any form by any means without prior written authorization of
Sun and its licensors, if any.

Third-partysoftware, includingfont technology,is copyrighted and licensed fromSun suppliers.
Parts of the product maybe derivedfrom BerkeleyBSD systems,licensed fromthe University of California. UNIX is a registered trademarkin

the U.S. and in other countries, exclusively licensed throughX/Open Company,Ltd.
Sun, Sun Microsystems,the Sunlogo, AnswerBook2,Sun StorEdge,StorTools,docs.sun.com, SunEnterprise, SunFire, SunOS, Netra, SunSolve

and Solaris aretrademarks, registeredtrademarks, or service marks of Sun Microsystems, Inc. inthe U.S.and othercountries. AllSPARC
trademarks are usedunder licenseand aretrademarks orregisteredtrademarks ofSPARCInternational, Inc.in theU.S. and other countries.
Productsbearing SPARCtrademarks arebased uponan architecturedevelopedby SunMicrosystems, Inc.

All SPARCtrademarks areused underlicense andare trademarks or registered trademarksof SPARCInternational, Inc.in theU.S. andin other
countries. Products bearingSPARCtrademarks arebased upon an architecture developedby SunMicrosystems, Inc.

The OPEN LOOK and Sun™ Graphical User Interface was developed bySun Microsystems,Inc. forits usersand licensees. Sun acknowledges
the pioneering effortsof Xeroxin researchingand developing the concept of visual orgraphical userinterfaces forthe computerindustry.Sun
holds a non-exclusive license fromXerox tothe XeroxGraphical User Interface, which license also covers Sun’s licensees who implement OPEN
LOOK GUIs and otherwise comply with Sun’s written license agreements.

Netscape Navigator is a trademark or registeredtrademark ofNetscape CommunicationsCorporation inthe UnitedStates andother countries.
U.S. Government Rights—Commercialuse. Governmentusers aresubject tothe SunMicrosystems, Inc. standardlicense agreementand

applicable provisions ofthe FAR and its supplements.
DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES,

INCLUDING ANY IMPLIED WARRANTYOF MERCHANTABILITY,FITNESS FORA PARTICULARPURPOSE ORNON-INFRINGEMENT,
ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Copyright 2003 Sun Microsystems, Inc.,4150 NetworkCircle, SantaClara, California95054, Etats-Unis.Tousdroitsréservés.
Sun Microsystems, Inc.a lesdroits depropriété intellectuels relatantsà latechnologie incorporéedans leproduit quiest décritdans ce

document. En particulier,et sans la limitation, ces droits depropriété intellectuelspeuvent inclureun ou plus des brevetsaméricains énumérés
à http://www.sun.com/patentset unou lesbrevets plussupplémentaires ou les applications de brevet enattente dansles Etats-Uniset dans
les autres pays.

Ce produit oudocument estprotégé parun copyrightet distribuéavec deslicences quien restreignentl’utilisation,la copie,la distribution,et la
décompilation. Aucune partie de ce produit oudocument nepeut êtrereproduitesous aucuneforme, parquelquemoyen quece soit,sans
l’autorisation préalable et écrite de Sun et de ses bailleurs delicence, s’ily ena.

Le logiciel détenu par des tiers, et qui comprendla technologierelative auxpolices decaractères, est protégépar uncopyright etlicencié pardes
fournisseurs de Sun.

Des parties de ce produitpourront êtredérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque
déposée aux Etats-Unis et dans d’autres payset licenciéeexclusivement parX/Open Company,Ltd.

Sun, Sun Microsystems,le logoSun, AnswerBook2,Sun StorEdge,StorTools,docs.sun.com, SunEnterprise, SunFire, SunOS, Netra, SunSolve,
et Solaris sont des marquesde fabriqueou desmarques déposées,ou marquesde service, de Sun Microsystems,Inc. auxEtats-Unis etdans
d’autrespays. Toutesles marquesSPARCsont utilisées sous licence et sont des marques defabrique oudes marques déposées de SPARC
International, Inc. aux Etats-Unis et dans d’autres pays.Les produitsportant lesmarques SPARCsont baséssur unearchitecturedéveloppée
par Sun Microsystems,Inc.

Toutes les marquesSPARCsont utiliséessous licenceet sontdes marquesde fabrique ou des marquesdéposées deSPARCInternational, Inc.
aux Etats-Unis et dans d’autrespays. Lesproduits protantles marques SPARCsont baséssur unearchitecture développéepar Sun
Microsystems,Inc.

L’interfaced’utilisation graphique OPEN LOOK et Sun™ a été développéepar SunMicrosystems, Inc.pour sesutilisateurs etlicenciés. Sun
reconnaîtles effortsde pionniersde Xeroxpour la rechercheet ledéveloppment duconcept desinterfaces d’utilisationvisuelle ougraphique
pour l’industrie de l’informatique. Sun détient une license non exclusive do Xerox surl’interface d’utilisationgraphique Xerox,cette licence
couvrant également les licenciées de Sun qui mettent en place l’interfaced ’utilisationgraphique OPENLOOK etqui enoutre seconforment
aux licences écrites de Sun.

Netscape Navigator est une marque de Netscape Communications Corporation aux Etats-Unis et dans d’autrespays.
LA DOCUMENTATION EST FOURNIE "EN L’ETAT" ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES

OU TACITES SONT FORMELLEMENTEXCLUES, DANSLA MESUREAUTORISEE PARLA LOIAPPLICABLE, YCOMPRIS NOTAMMENT
TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A
L’ABSENCE DE CONTREFAÇON.

Please

Recycle

Contents

Preface XV

How This Book Is Organized XV
Using UNIX Commands XVI
Typographic Conventions XVII
Shell Prompts XVII
Related Documentation XVIII
Accessing Sun Documentation Online XX
Sun Welcomes Your Comments XX

1. Introduction 1

Predictive Failure Analysis (PFA) Capabilities 2

2. General Troubleshooting Procedures 3

High-Level Troubleshooting Tasks 3
Host-Side Troubleshooting 6
Storage Service Processor-Side Troubleshooting 6
Verifying the Configuration Settings 7

▼ To Verify Configuration Settings 7

Clearing the Lock File 10

▼ To Clear the Lock File 10

Sun Proprietary/Confidential: Internal Use Only

Contents III

Sun StorEdge 6900 Series Multipathing Example 11
Multipathing Options in the Sun StorEdge 6900 Series 16

Manually Halting the I/O 17

▼ To Quiesce the I/O 17
▼ To Unconfigure the c2 Path 17

Suspending the I/O 18

▼ To Put the c2 Path Back into Production 19

▼ To View the Dynamic Multi-Pathing (DMP) Properties 20

▼ To Put the DMP-Enabled Paths Back into Production 22

3. Troubleshooting Tools 23

Storage Automated Diagnostic Environment 2.2 23

Example Topology 24

Generating Component-Specific Event Grids 25

▼ To Customize an Event Report 25

Microsoft Windows 2000 System Errors 26
Command Line Test Examples 27

qlctest(1M) 27
switchtest(1M) 28

Monitoring Sun StorEdge T3 and T3+ Arrays Using the Explorer Data Collection

Utility 29

▼ To Install the Explorer Data Collection Utility on the Storage Service

Processor 29

Monitoring Host Bus Adapters (HBAs) Using QLogic SANblade Manager 32

4. Troubleshooting Ethernet Hubs 35

5. Troubleshooting the Fibre Channel (FC) Links 37

FC Links 38

FC Link Diagrams 39

Contents IV

Sun Proprietary/Confidential: Internal Use Only

Troubleshooting the A1 or B1 FC Link 42

Verifying the Data Host 45
FRU Tests Available for the A1 or B1 FC Link Segment 46

▼ To Isolate the A1 or B1 FC Link 48

Troubleshooting the A2 or B2 FC Link 49

Verifying the Data Host 51
Verifying the A2 or B2 FC Link 52
FRU Tests Available for the A2 or B2 FC Link Segment 52

▼ To Isolate the A2 or B2 FC Link 52

Troubleshooting the A3 or B3 FC Link 54

Verifying the Data Host 56
Verifying the Storage Service Processor-Side 57
FRU Tests Available for the A3 or B3 FC Link Segment 57

▼ To Isolate the A3 or B3 FC Link 58

Quiescing the I/O on the A3 or B3 Link 59
Suspending the I/O on the A3 to B3 Link 59

Troubleshooting the A4 or B4 FC Link 60

Verifying the Data Host 62

Sun StorEdge 3900 Series 62
Sun StorEdge 6900 Series 62

FRU Tests Available for the A4 or B4 FC Link Segment 64

▼ To Isolate the A4 or B4 FC Link 64

6. Troubleshooting Host Devices 67

Using the Host Event Grid 67

▼ To Access the Host Event Grid 67

Replacing the Master, Alternate Master, and Slave Monitoring Host 71

▼ To Replace the Master Host 71

Sun Proprietary/Confidential: Internal Use Only

Contents V

▼ To Replace the Alternate Master or Slave Monitoring Host 72

7. Troubleshooting Switches 73

About the Switches 73

Zone Modifications 74
Switchless Configurations 75

▼ Diagnosing and Troubleshooting Switch Hardware Problems 75

Using the Switch Event Grid 77

▼ To Use the Switch Event Grid 77

setupswitch Exit Values 85

8. Troubleshooting the Sun StorEdge T3+ Array Devices 87

Troubleshooting the T1 or T2 Data Path 88

Notification Events 89

▼ To Verify the Storage Service Processor 92

FRU Tests Available for the T1 or T2 Data Path FRU 93

▼ To Isolate the T1 or T2 Data Path 94

Sun StorEdge T3+ Array Event Grid 95

▼ To Use the Sun StorEdge T3+ Array Event Grid 95

9. Troubleshooting Virtualization Engine Devices 107

About the Virtualization Engine 107
Virtualization Engine Diagnostics 108

Service Request Numbers (SRNs) 108
Service and Diagnostic Codes 108
Retrieving Service Information 108

CLI Interface 108

Error Log Analysis Commands 109

▼ To Display the Log Files and Retrieve SRNs 109

VI Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

▼ To Clear the Log 110

Virtualization Engine LEDs 110

Power LED Codes 111
Interpreting LED Service and Diagnostic Codes 111
Back Panel Features 112
Ethernet Port LEDs 112

FC Link Error Status Report 113

▼ To Check the FC Link Error Status Manually 113

Translating Host-Device Names 115
Displaying the VLUN Serial Number 116

▼ To Display Devices That are Not Sun StorEdge Traffic Manager (MPxIO)-

Enabled 116

▼ To Display Sun StorEdge Traffic Manager (MPxIO)-Enabled Devices 117

Viewing the Virtualization Engine Map 118

▼ To Failback the Virtualization Engine 120

Manually Clearing and Restoring the SAN Database 123

▼ To Reset the SAN Database on Both Virtualization Engines 124

▼ To Reset the SAN Database on a Single Virtualization Engine 125

Restarting the slicd Daemon 126

▼ To Restart the slicd Daemon 126

Diagnosing a creatediskpools(1M) Failure 129
Virtualization Engine Event Grid 132

▼ To Use the Virtualization Engine Event Grid 132

10. Troubleshooting Using Microsoft Windows 2000 137

General Notes 137
Troubleshooting Tasks Using Microsoft Windows 2000 138

Launching the Sun StorEdge T3+ Array Failover Driver GUI 138
Checking the Version of the Sun StorEdge T3+ Array Failover Driver 139

Sun Proprietary/Confidential: Internal Use Only

Contents VII

▼ To Use the Sun StorEdge T3+ Array Failover Driver GUI 140
▼ To Use the Sun StorEdge T3+ Array Failover Driver Command Line

Interface (CLI) 142

11. Example of Fault Isolation 147

A. Virtualization Engine References 155

SRN Reference 155
SRN/SNMP Single Point-of-Failure Descriptions 159
Port Communication Numbers 160
Virtualization Engine Service Codes 160

B. Configuration Utility Error Messages 163

Virtualization Engine Error Messages 164
Switch Error Messages 168
Sun StorEdge T3+ Array Partner Group Error Messages 171
Other Error Messages 175

VIII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

List of Figures

FIGURE 2-1 Sun StorEdge 6900 Series Logical View 11
FIGURE 2-2 Primary Data Paths to the Alternate Master 12
FIGURE 2-3 Primary Data Paths to the Master Sun StorEdge T3+ Array 13
FIGURE 2-4 Path Failure—Before the Second Tier of Switches 14
FIGURE 2-5 Path Failure—I/O Routed Through Both HBAs 15
FIGURE 3-1 Storage Automated Diagnostic Environment Example Topology 24
FIGURE 3-2 Microsoft Windows 2000 Event Properties System Log 26
FIGURE 3-3 Qlogic SANblade Manager HBA Driver and Firmware Versions 33
FIGURE 3-4 QLogic SANblade Manager Diagnostics 34
FIGURE 5-1 Sun StorEdge 3900 Series FC Link Diagram 39
FIGURE 5-2 Sun StorEdge 6900 Series FC Link Diagram 41
FIGURE 5-3 Data Host Notification of Intermittent Problems 43
FIGURE 5-4 Data Host Notification of Severe Link Error 43
FIGURE 5-5 Storage Service Processor Notification 44
FIGURE 5-6 A2 or B2 FC Link Host-Side Event 49
FIGURE 5-7 A2 or B2 FC Link Storage Service Processor-Side Event 50
FIGURE 5-8 A3 or B3 FC Link Host-Side Event 54
FIGURE 5-9 A3 or B3 FC Link Storage Service Processor-Side Event 55
FIGURE 5-10 A3 or B3 FC Link Storage Service Processor-Side Event 55

Sun Proprietary/Confidential: Internal Use Only

List of Figures IX

FIGURE 5-11 A4 or B4 FC Link Data-Host Notification 60
FIGURE 5-12 Storage Service Processor-Side Notification 61
FIGURE 6-1 Sample Host Event Grid 68
FIGURE 7-1 Switch Event Grid 77
FIGURE 8-1 Storage Service Processor Event 89
FIGURE 8-2 Virtualization Engine Alert 90
FIGURE 8-3 Manage Configuration Files Menu 92
FIGURE 8-4 Example Link Test Text Output from the Storage Automated Diagnostic Environment 93
FIGURE 8-5 Sun StorEdge T3+ Array Event Grid 95
FIGURE 9-1 Virtualization Engine Front Panel LEDs 111
FIGURE 9-2 Virtualization Engine Back Panel 112
FIGURE 9-3 Virtualization Engine Event Grid 132
FIGURE 10-1 Launching the Sun StorEdge T3+ Array Failover Driver 138
FIGURE 10-2 Sun StorEdge T3+ Array Failover Driver Versions 2.0.0.123 and 2.1.0.104 139
FIGURE 10-3 Healthy Sun StorEdge 3900 series system, shown using Multipath Configurator 140
FIGURE 10-4 Sun StorEdge 3900 series system with a LUN failover, shown using Multipath

Configurator 141

FIGURE 10-5 Multipath Configurator Array Properties 141
FIGURE 10-6 Multipath Configurator LUN Properties Detail 142
FIGURE 10-7 Sun StorEdge T3+ Array Failover Driver CLI Output for the Sun StorEdge 3900 Series 143
FIGURE 10-8 Sun StorEdge T3+ Array Failover Driver CLI Example Output for the Sun StorEdge 6900

Series 144

FIGURE 11-1 Alerts Display Using the Storage Automated Diagnostic Environment 147
FIGURE 11-2 Drilling Down for Sun StorEdge T3+ Array Failover Driver Fault Detail 148
FIGURE 11-3 Fault Confirmation Using QLogic SunBlade 149
FIGURE 11-4 Diagnostics Using QLogic SunBlade 150
FIGURE 11-5 Storage Automated Diagnostic Environment Test from Topology 151
FIGURE 11-6 Storage Automated Diagnostic Environment Test from Topology Pull-Down Menu 152
FIGURE 11-7 Storage Automated Diagnostic Environment Test from Topology Test Detail 152

List of Figures X

Sun Proprietary/Confidential: Internal Use Only

FIGURE 11-8 Successful Switch Test Results 153
FIGURE 11-9 Multipath Recovery using the Sun StorEdge T3+ Array Multipath Configurator 154
FIGURE 11-10 Recovered Paths 154

Sun Proprietary/Confidential: Internal Use Only

List of Figures XI

XII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

List of Tables

TABLE 1-1 Sun StorEdge 3900 and 6900 Series Configurations 1
TABLE 3-1 Event Grid Sorting Criteria 25
TABLE 5-1 FC Links 38
TABLE 5-2 Ax to Bx FC Links. 40
TABLE 6-1 Storage Automated Diagnostic Environment Event Grid for the Host 69
TABLE 7-1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches 78
TABLE 7-2 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches 82
TABLE 0-1 setupswitch Exit Values 85

TABLE 8-1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3+ Array 96
TABLE 9-1 Virtualization Engine LEDs 110
TABLE 9-2 LED Diagnostic Codes 111
TABLE 9-3 Speed, Activity, and Validity of the Link 112
TABLE 9-4 Virtualization Engine Statistical Data 113
TABLE 9-5 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine 133
TABLE 10-1 Tips for Interpreting Sun StorEdge 6910 Series CLI Output 145
TABLE A-1 SRN Reference 156
TABLE A-2 SRN/SNMP Single Point-of-Failure Table 159
TABLE A-3 Port CommunicationNumbers 160
TABLE A-4 Virtualization Engine Service Codes —0 -399 Host-Side Interface Driver Errors 160

List of Tables XIII

Sun Proprietary/Confidential: Internal Use Only

TABLE A-5 Virtualization Engine Service Codes —400-599 Device-Side Interface Driver Errors 162
TABLE B-1 Virtualization Engine Error Messages 164
TABLE B-2 Sun StorEdge Network FC Switch Error Messages 168
TABLE B-3 Sun StorEdge T3+ Array Error Messages 171
TABLE B-4 Other SUNWsecfg Error Messages 175

XIV Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

Preface

The Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide provides guidelines
for isolating problems in supported configurations of the Sun StorEdge
6900 series. For detailed configuration information, refer to the Sun StorEdge 3900
and 6900 Series Reference Manual.

The scope of this troubleshooting guide is limited to information pertaining to the
components of the Sun StorEdge 3900 and 6900 series, including the Storage Service
Processor, Sun StorEdge 1 Gbit and 2 Gbit switches, Sun StorEdge T3+ arrays, and
the virtualization engines in the Sun StorEdge 6900 series. This guide is written for

TM

personnel who have been fully trained on all the components in the

Sun
configuration.

TM

3900 and

How This Book Is Organized

This book contains the following topics:

Chapter 1 introduces the Sun StorEdge 3900 and 6900 series storage subsystems.
Chapter 2 offers general troubleshooting guidelines, such as manually halting the

I/O and returning paths to production.
Chapter 3 presents information about tools used to troubleshoot. Tools include the

Storage Automated Diagnostic Environment, component-specific event grids,
command line examples, and QLogic’s SANblade Manager.

Chapter 4 discusses Ethernet hub troubleshooting. Information associated with the
3Com Ethernet hubs is limited in this guide, however, because 3Com does not allow
duplication of its information.

Chapter 5 provides Fibre Channel (FC) link diagrams and troubleshooting
procedures.

Sun Proprietary/Confidential: Internal Use Only

XV

Chapter 6 provides information on host device troubleshooting.
Chapter 7 provides information on troubleshooting a Sun StorEdge Network FC

switch-8 and switch-16 switch device.
Chapter 8 describes how to troubleshoot the Sun StorEdge T3+ array devices. Also

included in this chapter is information about the Explorer Data Collection Utility.
Chapter 9 provides detailed information for troubleshooting the virtualization

engines.
Chapter 10 describes how to troubleshoot using Microsoft Windows 2000. It also

explains how to launch the Sun StorEdge T3+ Array Failover Driver GUI and
interpret the multipath configurator.

Chapter 11 provides an example of fault isolation. It begins with how to discover an
error and shows the user steps that are necessary for resolution.

Appendix A provides virtualization engine references, including Service Request
Numbers (SRNs) and Simple Network Management Protocol (SNMP) Reference, an
SRN/SNMP single point-of-failure table, and port communication and service code
tables.

Appendix B provides a list of SUNWsecfg(1M) error messages and
recommendations for corrective action.

Using UNIX Commands

This document may not contain information on basic UNIX®commands and
procedures such as shutting down the system, booting the system, and configuring
devices.

See one or more of the following documents for this information:

■ Solaris Handbook for Sun Peripherals

■ AnswerBook2™ online documentation for the Solaris™ operating environment

■ Other software documentation that you received with your system

XVI Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

Typographic Conventions

Typeface Meaning Examples

AaBbCc123 The names of commands, files,

and directories; on-screen
computer output

AaBbCc123

AaBbCc123 Book titles, new words or terms,

What you type, when
contrasted with on-screen
computer output

words to be emphasized

Command-line variable; replace
with a real name or value

Edit your.login file.
Use ls -a to list all files.

% You have mail.
%

su

Password:

Read Chapter 6 in the User’s Guide.
These are called class options.
You must be superuser to do this.

To delete a file, type rm filename.

Shell Prompts

Shell Prompt

C shell machine-name%
C shell superuser machine-name#
Bourne shell and Korn shell $
Bourne shell and Korn shell superuser #

Sun Proprietary/Confidential: Internal Use Only

Preface XVII

Related Documentation

Product Title Part Number

Late-breaking News • Sun StorEdge 3900 and 6900 Series 2.0 Release Notes 816-5254
Sun StorEdge 3900 and

6900 series information

Sun StorEdge T3 and
T3+ array

Diagnostics • Storage Automated Diagnostics Environment User’s Guide 816-3142
Sun StorEdge SAN 4.0

(1 Gb switches)

Sun StorEdge SAN 4.1
(2 Gb switches)

3Com Ethernet hubs • SuperStack 3 Baseline Hub 12-Port TP User Guide

• Sun StorEdge 3900 and 6900 Series 2.0 Installation Guide

• Sun StorEdge 3900 and 6900 Series 2.0 Reference and Service Guide

• Sun StorEdge 3900 and 6900 Series 2.0 Regulatory and Safety
Compliance Manual

• Sun StorEdge 3900 and 6900 Series 2.0 Site Prep Guide

• Sun StorEdge T3+ Array Release Notes

• Sun StorEdge T3+ Array Start Here

• Sun StorEdge T3 and T3+ Array Regulatory and Safety Compliance
Manual

• Sun StorEdge T3+ Array Installation and Configuration Manual

• Sun StorEdge T3+ Array Administrator’s Guide

• Sun StorEdge T3 Array Cabinet Installation Guide

• Sun StorEdge SAN 4.0 Release Guide to Documentation

• Sun StorEdge SAN 4.0 Release Installation Guide

• Sun StorEdge SAN 4.0 Release Configuration Guide

• Sun StorEdge Network 2 Gb FC Switch-16 FRU Installation

• Sun StorEdge SAN 4.0 Release Notes

• Sun StorEdge SAN 4.1 Release Guide to Documentation

• Sun StorEdge SAN 4.1 Release Installation Guide

• Sun StorEdge SAN 4.1 Release Configuration Guide

• Sun StorEdge SAN 4.1 2 Gb Brocade Silkworm Fabric Switch Guide to
Documentation

• Sun StorEdge SAN 4.1 2 Gb McData Intrepid Director Switch Guide to
Documentation

• Sun StorEdge SAN 4.1 Release Notes

• SuperStack 3 Baseline Hub 24-Port TP User Guide

816-5252
816-5253

816-5257
816-5256

816-4771
816-4768
816-0774

816-4769
816-4770
806-7979

816-4470
816-4469
806-5513
816-5285
816-4472

817-0061
817-0056
817-0057
817-0062

817-0063

817-0071
3C16440A

3C16441A

XVIII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

Product Title Part Number

SANbox-8/16
Segmented Loop FC
Switch

Expansion cabinet • Sun StorEdge Expansion Cabinet Installation and Service Manual 805-3067
Storage Server Processor • Sun V100 Server User ’s Guide

• SANbox-8/16 Segmented Loop Fibre Channel Switch Management
User’s Manual

• SANbox-8 Segmented Loop Fibre Channel Switch Installer’s/User’s
Manual

• SANbox-16 Segmented Loop Fibre Channel Switch Installer’s/User’s
Manual

• Netra X1 Server User’s Guide

• Netra X1 Server Hard Disk Drive Installation Guide

875-3060

875-1881

875-3059

806-5980
806-5980
806-7670

Sun Proprietary/Confidential: Internal Use Only

Preface XIX

Accessing Sun Documentation Online

You can view, print, or purchase a broad selection of Sun documentation, including
localized versions, at:

http://www.sun.com/documentation

Sun Welcomes Your Comments

Sun is interested in improving its documentation and welcomes your comments and
suggestions. You can email your comments to Sun at:

[e-mail]

Please include the part number (816-5255) of your document in the subject line of
your email.

XX Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

CHAPTER

Series System

1

Sun StorEdge
3900 series

Introduction

The Sun StorEdge 3900 and 6900 series storage subsystems are complete
preconfigured storage solutions. The configurations for each of the storage
subsystems are shown in

TABLE1-1 Sun StorEdge 3900 and 6900 Series Configurations

Sun StorEdge
3910 system

Two 8-port
switches

TABLE 1-1.

Sun StorEdge
Fibre Channel

Switches

Supported

1

Sun StorEdg e

T3+ Array

Partner
Groups

Supported

One to four

Additional

Array Partner

Groups

Supported

with Optional

Additional
Expansion

Cabinet

N/A

Virtualization

Engine

N/A

2

3900SL

Sun StorEdge
6900 series

3

6910SL

3

6960SL

1

1 Gbit or 2 Gbit switches

2

3900SL—No switches

3

6910SL and 6960SL—No front-end switches; two back-end switches

Sun StorEdge
3960 system

Sun StorEdge
6910 system

Sun StorEdge
6960 system

Two 16-port
switches

Four 8-port
switches

Four 16-port
switches

One to four

One to three

One to three

Sun Proprietary/Confidential: Internal Use Only

One to five

One to four

One to four

One virtualization
engine pair

Two virtualization
engine pairs

1

Predictive Failure Analysis (PFA) Capabilities

The Storage Automated Diagnostic Environment software provides the health and
monitoring functions for the Sun StorEdge 3900 and 6900 series systems. This
software provides the following predictive failure analysis (PFA) capabilities:

■ FC links—Fibre Channel (FC) links are monitored at all end points using the

Fibre Channel-Extended Link Service (FC-ELS) link counters. When link errors
surpass the threshold values, an alert is sent. This enables Sun-trained personnel
to replace components that are experiencing high transient fault levels before a
hard fault occurs.

■ Enclosure status—Many devices, like the Sun StorEdge FC switch-8 and switch-

16 switch and the Sun StorEdge T3+ array, cause the Storage Automated
Diagnostic Environment alerts to be sent if the temperature thresholds are
exceeded. This enables Sun-trained personnel to address the problem before the
component and enclosure fails.

■ Single Point-of-Failure (SPOF) notification—Storage Automated Diagnostic

Environment notification for path failures and failovers (that is, Sun StorEdge
Traffic Manager software failover) can be considered a PFA method, since Sun­trained personnel are notified and can repair the primary path. This eliminates
the time of exposure to SPOF and helps to preserve customer availability during
the repair process.

PFA is not always effective in detecting or isolating failures. The remainder of this
document provides guidelines that you can use to troubleshoot problems that occur
in supported components of the Sun StorEdge 3900 and 6900 series.

2 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

CHAPTER

2

General Troubleshooting Procedures

This chapter contains the following sections:

■ “High-Level Troubleshooting Tasks” on page 3

■ “Host-Side Troubleshooting” on page 6

■ “Storage Service Processor-Side Troubleshooting” on page 6

■ “Verifying the Configuration Settings” on page 7

■ “Sun StorEdge 6900 Series Multipathing Example” on page 11

■ “Multipathing Options in the Sun StorEdge 6900 Series” on page 16

High-Level Troubleshooting Tasks

This section lists the high-level steps you can take to isolate and troubleshoot
problems in the Sun StorEdge 3900 and 6900 series. It offers a methodical approach,
and lists the tools and resources available at each step.

Note – A single problem can cause various errors throughout the storage area

network (SAN). A good practice is to begin by investigating the devices that have
experienced “Loss of Communication” events in the Storage Automated Diagnostic
Environment. These errors usually indicate more serious problems.

A “Loss of Communication” error on a switch, for example, could cause multiple
ports and host bus adapters (HBAs) to go offline. Concentrating on the switch and
fixing that failure can help bring the ports and HBAs back online.

Sun Proprietary/Confidential: Internal Use Only

3

1. Discover the error by checking one or more of the following messages or files:

■ Storage Automated Diagnostic Environment alerts or email messages

■ /var/adm/messages

■ Sun StorEdge T3+ array syslog file

■ Storage Service Processor messages

■ /var/adm/messages.t3 messages

■ /var/adm/log/SEcfglog file

2. Determine the extent of the problem by using one or more of the following methods:

■ Review the Storage Automated Diagnostic Environment topology view.

■ Using the Storage Automated Diagnostic Environment revision checking

functionality, determine whether the package or patch is installed.

■ Verify the functionality using one of the following tools:

■ checkdefaultconfig(1M)

■ cfgadm -al output

■ luxadm(1M) output

■ Review the multipathing status using the Sun StorEdge Traffic Manager (MPxIO)

software or vxdmp(1M) command.

3. Check the status of a Sun StorEdge T3+ array by using one or more of the following methods:

■ Review the Storage Automated Diagnostic Environment device monitoring

reports.

■ Run the checkt3config(1M) and showt3(1M) commands, which check and

display the Sun StorEdge T3+ array configuration.

■ Manually open a Telnet session to the Sun StorEdge T3+ array.

■ Review the luxadm(1M) display output.

■ Review the LED status on the Sun StorEdge T3+ array.

■ Review the Explorer Data Collection Utility output, which is located on the

Storage Service Processor.

4 Sun StorEdge 3900 and 6900 2.0 Series Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

4. Check the status of the Sun StorEdge network FC switch-8 and switch-16 switches
using the following tools:

■ Review the Storage Automated Diagnostic Environment device monitoring

reports.

■ Run the checkswitch(1M) and showswitch(1M) commands, which check and

display the Sun StorEdge FC switch configurations.

■ Review the online and offline LED status codes and POST error codes, which can

be found in the Sun StorEdge SAN 4.0 and SAN 4.1 Release Installation Guide.

■ Review the Explorer Data Collection Utility output, which is located on the

Storage Service Processor.

■ Refer to the SANsurfer GUI, which supports the Sun StorEdge 4.0 Release, or the

SANbox Manager, which supports the Sun StorEdge 4.1 Release.

Note – To run the SANsurfer GUI or SANbox Manager from the Storage Service

Processor, you must export X-Display.

5. Check the status of the virtualization engine using one or more of the following methods:

■ Review the Storage Automated Diagnostic Environment device monitoring

reports.

■ Run the checkve(1M), checkvemap(1M) and showvemap(1M) commands, which

check and display the virtualization host and LUN configurations.

■ Refer to the LED status blink codes “Virtualization Engine LEDs” on page 110.

6. Quiesce the I/O along the path to be tested using one of the following methods:

■ For installations using VERITAS Dynamic Multi-Pathing (DMP), disable

vxdmpadm(1M).

■ For installations using the Sun StorEdge Traffic Manager (MPxIO) software,

unconfigure the Fabric device.

■ Refer to “To Quiesce the I/O” on page 17.

■ Halt the application.

7. Test and isolate field-replaceable units (FRUs) using the following tools:

■ Storage Automated Diagnostic Environment diagnostic tests (this might require a

loopback cable for isolation)

■ Sun StorEdge T3+ array tests, including t3test(1M), t3ofdg(1M), and

t3volverify(1M), which can be found in the Storage Automated Diagnostic

Environment User’s Guide

Chapter 2 General Troubleshooting Procedures 5

Sun Proprietary/Confidential: Internal Use Only

Note – These tests isolate the problem to a FRU that must be replaced. Follow the

instructions in the Sun StorEdge 3900 and 6900 Series 2.0 Reference and Service Guide
and the Sun StorEdge 3900 and 6900 Series 2.0 Installation Guide for proper FRU
replacement procedures.

8. Verify the fix using the following tools:

■ Storage Automated Diagnostic Environment GUI Topology View and Diagnostic

Tests

■ /var/adm/messages on the data host

9. Return the path to service with one of the following methods:

■ Use the multipathing software

■ Restart the application

Host-Side Troubleshooting

Host-side troubleshooting refers to the messages and errors that the data host detects.
Usually these messages appear in the /var/adm/messages file.

Storage Service Processor-Side Troubleshooting

Storage Service Processor-side troubleshooting refers to messages, alerts, and errors
that the Storage Automated Diagnostic Environment detects while running on the
Storage Service Processor. You can find these messages by monitoring the following
Sun StorEdge 3900 series and Sun StorEdge 6900 series components:

■ Sun StorEdge network FC switch-8 and switch-16 switches

■ Virtualization engine

■ Sun StorEdge T3+ array

Combining the host-side messages and errors and the Storage Service Processor-side
messages, alerts, and errors into a meaningful context is essential for proper
troubleshooting.

6 Sun StorEdge 3900 and 6900 2.0 Series Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

Verifying the Configuration Settings

During the course of troubleshooting, you might need to verify configuration
settings on the various components in the Sun StorEdge 3900 or 6900 series.

▼ To Verify Configuration Settings

1. Run one of the following scripts:

■ Run the runsecfg(1M) script and select the various Verify menu selections for

the Sun StorEdge T3+ arrays, the Sun StorEdge network FC switch-8 and switch­16 switches, and the virtualization engine components.

■ Run the checkdefaultconfig(1M) script to check all accessible components.

The output is shown in

■ Run the checkswitch(1M) | checkt3config(1M) | checkve(1M) |

checkvemap(1M) scripts from /opt/SUNWsecfg/bin to check the settings on

the Sun StorEdge network FC switch-8 and switch-16 switches, the Sun StorEdge
T3+ array, and the virtualization engine.

The scripts check the default configuration files in the /opt/SUNWsecfg/etc
directory and compare the current, live settings to those of the defaults. Any
differences are marked with a FAIL.

CODE EXAMPLE 2-1.

Note – For cluster configurations and systems that are attached to Microsoft

Windows NT, the default configurations may not match the current installed
configuration. Be aware of this when running the verification scripts. Certain items
may be flagged as FAIL in these special circumstances.

Chapter 2 General Troubleshooting Procedures 7

Sun Proprietary/Confidential: Internal Use Only

CODE EXAMPLE 2-1 checkdefaultconfig(1M) Output

# /opt/SUNWsecfg/checkdefaultconfig

Checking all accessible components.....

Checking switch: sw1a
Switch sw1a - PASSED
Checking switch: sw1b
Switch sw1b - PASSED
Checking switch: sw2a
Switch sw2a - PASSED
Checking switch: sw2b
Switch sw2b - PASSED
Please enter the Sun StorEdge T3+ array password :

Checking T3+: t3b0

Checking : t3b0 Configuration.......

Checking command ver : PASS
Checking command vol stat : PASS
Checking command port list : PASS
Checking command port listmap : PASS

Checking command sys list : FAIL <-- Failure Noted

Checking T3+: t3b2

Checking : t3b2 Configuration.......

Checking command ver : PASS
Checking command vol stat : PASS
Checking command port list : PASS
Checking command port listmap : PASS
Checking command sys list : PASS
<snip>

Checking Virtualization Engine Pair Parameters: v1a
v1a configuration check passed

Checking Virtualization Engine Pair Parameters: v1b
v1b configuration check passed

Checking Virtualization Engine Pair Configuration: v1
checkvemap: virtualization engine map v1 verification complete: PASS.

8 Sun StorEdge 3900 and 6900 2.0 Series Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only

2. If anything is marked FAIL, check the /var/adm/log/SEcfglog file for the details of the failure.

Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : ----------

-SAVED CONFIGURATION--------------.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : blocksize : 16k.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : cache : auto.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : mirror : auto.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : mp_support : rw.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : rd_ahead : off.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : recon_rate : med.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : sys memsize : 32
MBytes.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : cache memsize :
256 MBytes.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : .
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : ----------

-CURRENT CONFIGURATION------------.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : blocksize : 16k.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : cache : auto.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : mirror : off.

Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : mp_support : rw.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : rd_ahead : off.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : recon_rate : med.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : sys memsize : 32
MBytes.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : cache memsize :
256 MBytes.
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : .
Mon Jan 7 18:07:51 PST 2002 checkt3config: t3b0 INFO : ----------

In this example, the mirror setting in the Sun StorEdge T3+ array system settings is
“off.” The saved configuration setting for this parameter, which is the default
setting, should be “auto.”

3. Fix the FAIL condition, and then verify the settings again.

# /opt/SUNWsecfg/bin/checkt3config -n t3b0

Checking : t3b0 Configuration.......

Checking command ver : PASS
Checking command vol stat : PASS
Checking command port list : PASS
Checking command port listmap : PASS
Checking command sys list : PASS

Chapter 2 General Troubleshooting Procedures 9

Sun Proprietary/Confidential: Internal Use Only

Clearing the Lock File

If you interrupt any of the Configuration Utility scripts (by typing Control-C, for
example), a lock file might remain in the /opt/SUNWsecfg/etc directory, causing
subsequent commands to fail. Use the following procedure to clear the lock file.

▼ To Clear the Lock File

1. Type the following command:

# /opt/SUNWsecfg/bin/removelocks

usage : removelocks [-t|-s|-v]

where:

-t - remove all T3+ related lock files.

-s - remove all switch related lock files.

-v - remove all virtualization engine related lock files.

# /opt/SUNWsecfg/bin/removelocks -v

Note – After making any change to the virtualization engine configuration, the

script saves a new copy of the virtualization engine map. This may take a minimum
of two minutes, during which time no additional virtualization engine changes are
accepted.

If a process such as savevemap(1M) is running, you cannot remove the lock file
using the removelocks(1M) command. This process causes a component to be
unavailable.

2. Monitor the /var/adm/log/SEcfglog file to see when the savevemap(1M) process successfully exits.

CODE EXAMPLE 2-2 savevemap(1M) Output

Tue Jan 29 16:12:34 MST 2002 savevemap: v1 ENTER.
Tue Jan 29 16:12:34 MST 2002 checkslicd: v1 ENTER.
Tue Jan 29 16:12:42 MST 2002 checkslicd: v1 EXIT.

Tue Jan 29 16:14:01 MST 2002 savevemap: v1 EXIT.

When savevemap: ve-pair EXIT is displayed, the savevemap(1M) process has
successfully exited.

10 Sun StorEdge 3900 and 6900 2.0 Series Troubleshooting Guide • March 2003

Sun Proprietary/Confidential: Internal Use Only