Sun Microsystems 3900, 6900 User Manual
Sun StorEdge™3900 and 6900
Series Troubleshooting Guide
Sun Microsystems, Inc.
4150 Network Circle
Santa Clara, CA 95054 U.S.A.
650-960-1300
Part No. 816-4290-11
March 2002, Revision A
Send comments about this document to: docfeedback@sun.com
Copyright 2002Sun Microsystems, Inc.,4150 NetworkCircle, SantaClara, CA95054 U.S.A.All rightsreserved.
This product ordocument isdistributed underlicenses restrictingits use,copying, distribution,and decompilation.No partof thisproduct or
document may be reproduced inany formby anymeans withoutprior writtenauthorization ofSun andits licensors,if any. Third-party
software,including fonttechnology,is copyrighted and licensed fromSun suppliers.
Parts of the product maybe derivedfrom BerkeleyBSD systems,licensed fromthe University of California. UNIX is a registered trademarkin
the U.S. and other countries, exclusively licensed through X/OpenCompany,Ltd.
Sun, Sun Microsystems,the Sunlogo, AnswerBook2,Sun StorEdge,StorTools,docs.sun.com, SunEnterprise, SunFire, SunOS, Netra, and
Solaris are trademarks,registered trademarks, or service marks of Sun Microsystems, Inc.in theU.S. andother countries.All SPARC
trademarks are usedunder licenseand aretrademarks orregisteredtrademarks ofSPARCInternational, Inc.in theU.S. and other countries.
Productsbearing SPARC trademarksare basedupon anarchitecturedeveloped bySun Microsystems,Inc.
The OPEN LOOK and Sun™ Graphical User Interface was developed bySun Microsystems,Inc. forits usersand licensees.Sun acknowledges
the pioneering effortsof Xeroxin researchingand developing the concept of visual orgraphical userinterfaces forthe computerindustry.Sun
holds a non-exclusive license fromXerox tothe XeroxGraphical User Interface, which license also covers Sun’s licensees who implement OPEN
LOOK GUIs and otherwise comply with Sun’s written license agreements.
Federal Acquisitions: CommercialSoftware—Government UsersSubject toStandard License TermsandConditions.
DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES,
INCLUDING ANY IMPLIED WARRANTY OFMERCHANTABILITY, FITNESS FOR APARTICULARPURPOSE OR NON-INFRINGEMENT,
ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
Copyright 2002 Sun Microsystems, Inc.,4150 NetworkCircle, SantaClara, CA95054 Etats-Unis.Tousdroitsréservés.
Ce produit oudocument estdistribué avecdes licencesqui enrestreignent l’utilisation, la copie, la distribution, et la décompilation. Aucune
partie de ce produit oudocument nepeut êtrereproduitesous aucuneforme, parquelque moyenque cesoit, sansl’autorisation préalableet
écrite de Sun et de ses bailleurs de licence, s’il y en a.Le logicieldétenu pardes tiers,et quicomprend latechnologie relativeaux policesde
caractères,est protégépar uncopyright etlicencié pardes fournisseursde Sun.
Des parties de ce produitpourront êtredérivées des systèmes Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque
déposée aux Etats-Unis et dans d’autres payset licenciéeexclusivement parX/Open Company, Ltd.
Sun, Sun Microsystems,le logoSun, AnswerBook2,Sun StorEdge,StorTools,docs.sun.com, SunEnterprise, SunFire, SunOS, Netra, et Solaris
sont des marquesde fabriqueou desmarques déposées,ou marquesde service, de Sun Microsystems,Inc. auxEtats-Unis etdans d’autrespays.
Toutes lesmarques SPARC sontutilisées souslicence etsont desmarques de fabrique ou des marques déposées de SPARCInternational, Inc.
aux Etats-Unis et dans d’autres pays.Les produitsportant lesmarques SPARC sont basés sur unearchitecture développéepar Sun
Microsystems,Inc.
L’interfaced’utilisation graphique OPEN LOOK et Sun™ a été développéepar SunMicrosystems, Inc.pour sesutilisateurs etlicenciés. Sun
reconnaîtles effortsde pionniersde Xeroxpour la rechercheet ledéveloppement duconcept desinterfaces d’utilisationvisuelle ougraphique
pour l’industrie de l’informatique. Sun détient une licence non exclusive deXerox surl’interface d’utilisationgraphique Xerox,cette licence
couvrant également les licenciés de Sun qui mettent en place l’interfaced’utilisation graphiqueOPEN LOOKet quien outrese conformentaux
licences écrites de Sun.
LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES
OU TACITES SONT FORMELLEMENTEXCLUES, DANSLA MESUREAUTORISEE PAR LALOI APPLICABLE,Y COMPRIS NOTAMMENT
TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A
L’ABSENCE DE CONTREFAÇON.
Please
Recycle
Contents
1. Introduction 1
Predictive Failure Analysis Capabilities 2
2. General Troubleshooting Procedures 3
Troubleshooting Overview Tasks 3
Multipathing Options in the Sun StorEdge 6900 Series 7
Alternatives to Sun StorEdge Traffic Manager 8
▼ To Quiesce the I/O 8
▼ To Unconfigure the c2 Path 8
▼ To Suspend the I/O 10
▼ To Return the Path to Production 10
▼ To View the VxDisk Properties 11
▼ To Quiesce the I/O on the A3/B3 Link 13
▼ To Suspend the I/O on the A3/B3 Link 13
▼ To Return the Path to Production 14
Fibre Channel Links 15
Fibre Channel Link Diagrams 16
Host Side Troubleshooting 18
Storage Service Processor Side Troubleshooting 18
For Internal Use Only
Contents iii
Command Line Test Examples 19
qlctest(1M) 19
switchtest(1M) 20
Storage Automated Diagnostic Environment Event Grid 21
▼ To Customize an Event Report 21
3. Troubleshooting the Fibre Channel Links 23
A1/B1 Fibre Channel (FC) Link 23
▼ To Verify the Data Host 25
FRU Tests Available for A1/B1 FC Link Segment 26
▼ To Isolate the A1/B1 FC Link 28
A2/B2 Fibre Channel (FC) Link 29
▼ To Verify the Host Side 31
▼ To Verify the A2/B2 FC Link 33
▼ To Isolate the A2/B2 FC Link 33
A3/B3 Fibre Channel (FC) Link 35
▼ To Verify the Host Side 37
▼ To Verify the Storage Service Processor 38
FRU Tests Available for the A3/B3 FC Link Segment 38
▼ To Isolate the A3/B3 FC Link 39
A4/B4 Fibre Channel (FC) Link 40
▼ To Verify the Data Host 42
Sun StorEdge 3900 Series 42
Sun StorEdge 6900 Series 42
FRU tests available for the A4/B4 FC Link Segment 44
▼ To Isolate the A4/B4 FC Link 44
4. Configuration Settings 47
Verifying Configuration Settings 47
For Internal Use Only
Contents iv
▼ To Verify Configuration Settings 47
▼ To Clear the Lock File 50
5. Troubleshooting Host Devices 53
Host Event Grid 53
▼ Using the Host Event Grid 53
Replacing the Master, Alternate Master, and Slave Monitoring Host 57
▼ To Replace the Master Host 57
▼ To Replace the Alternate Master or Slave Monitoring Host 58
Conclusion 59
6. Troubleshooting Sun StorEdge FC Switch-8 and Switch-16 Devices 61
Sun StorEdge Network FC Switch-8 and Switch-16 Switch Description 61
▼ To Diagnose and Troubleshoot Switch Hardware 62
Switch Event Grid 62
▼ Using the Switch Event Grid 62
Replacing the Master Midplane 68
▼ To Replace the Master Midplane 68
Conclusion 68
7. Troubleshooting Virtualization Engine Devices 69
Virtualization Engine Description 69
Virtualization Engine Diagnostics 70
Service Request Numbers 70
Service and Diagnostic Codes 70
▼ To Retrieve Service Information 70
CLI Interface 70
▼ To Display Log Files and Retrieve SRNs 71
▼ To Clear the Log 72
For Internal Use Only
Contents v
Virtualization Engine LEDs 72
Power LED Codes 73
Interpreting LED Service and Diagnostic Codes 73
Back Panel Features 74
Ethernet Port LEDs 74
Fibre Channel Link Error Status Report 75
▼ To Check Fibre Channel Link Error Status Manually 76
Translating Host Device Names 78
▼ To Display the VLUN Serial Number 79
Devices That Are Not Sun StorEdge Traffic Manager-Enabled 79
Sun StorEdge Traffic Manager-Enabled Devices 80
▼ To View the Virtualization Engine Map 81
▼ To Failback the Virtualization Engine 83
▼ To Replace a Failed Virtualization Engine 84
▼ To Manually Clear the SAN Database 86
▼ To Reset the SAN Database on Both Virtualization Engines 86
▼ To Reset the SAN Database on a Single Virtualization Engine 86
Stopping and Restarting the SLIC Daemon 87
▼ To Restart the SLIC Daemon 87
Sun StorEdge 6900 Series Multipathing Example 89
One Sun StorEdge T3+ array partner pair with 1 500GB RAID 5 LUN per
brick (2 LUNs total) 89
Virtualization Engine Event Grid 95
▼ Using the Virtualization Engine Event Grid 95
8. Troubleshooting the Sun StorEdge T3+ Array Devices 99
Explorer Data Collection Utility 99
▼ To Install Explorer Data Collection Utility on the Storage Service
Processor 99
vi Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
Troubleshooting the T1/T2 Data Path 102
Notes 102
T1/T2 Notification Events 103
Sun StorEdge T3+ Array Storage Service Processor Verification 106
T1/T2 FRU Tests Available 107
Notes 108
T1/T2 Isolation Procedures 108
Sun StorEdge T3+ Array Event Grid 109
▼ Using the Sun StorEdge T3+ Array Event Grid 109
Replacing the Master Midplane 122
▼ To Replace the Master Midplane 122
Conclusion 122
9. Troubleshooting Ethernet Hubs 123
setupswitch Exit Values 141
For Internal Use Only
Contents vii
viii Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
List of Figures
FIGURE 2-1 Sun StorEdge 3900 Series Fibre Channel Link Diagram 16
FIGURE 2-2 Sun StorEdge 6900 Series Fibre Channel Link Diagram 17
FIGURE 3-1 Data Host Notification of Intermittent Problems 23
FIGURE 3-2 Data Host Notification of Severe Link Error 24
FIGURE 3-3 Storage Service Processor Notification 24
FIGURE 3-4 A2/B2 FC Link Host Side Event 29
FIGURE 3-5 A2/B2 FC Link Storage Service Processor Side Event 30
FIGURE 3-6 A3/B3 FC Link Host-Side Event 35
FIGURE 3-7 A3/B3 FC Link Storage Service Processor-Side Event 36
FIGURE 3-8 A3/B3 FC Link Storage Service Processor-Side Event 36
FIGURE 3-9 A4/B4 FC Link Data Host Notification 40
FIGURE 3-10 Storage Service Processor Notification 41
FIGURE 5-1 Host Event Grid 54
FIGURE 6-1 Switch Event Grid 63
FIGURE 7-1 Virtualization Engine Front Panel LEDs 73
FIGURE 7-2 Sun StorEdge 6900 Series Logical View 90
FIGURE 7-3 Primary Data Paths to the Alternate Master 91
FIGURE 7-4 Primary Data Paths to the Master Sun StorEdge T3+ Array 92
FIGURE 7-5 Path Failure—Before the Second Tier of Switches 93
List of Figures ix
FIGURE 7-6 Path Failure —I/O Routed through Both HBAs 94
FIGURE 7-7 Virtualization Engine Event Grid 95
FIGURE 8-1 Storage Service Processor Event 103
FIGURE 8-2 Virtualization Engine Alert 105
FIGURE 8-3 Manage Configuration Files Menu 106
FIGURE 8-4 Example Link Test Text Output from the Storage Automated Diagnostic Environment 107
FIGURE 8-5 Sun StorEdge T3+ array Event Grid 109
List of Figures x
Preface
The Sun StorEdge 3900 and 6900 Series Troubleshooting Guide provides guidelines
for isolating problems in supported configurations of the Sun StorEdge
6900 series. For detailed configuration information, refer to the Sun StorEdge 3900
and 6900 Series Reference Manual.
The scope of this troubleshooting guide is limited to information pertaining to the
components of the Sun StorEdge 3900 and 6900 series, including the Storage Service
Processor and the virtualization engines in the Sun StorEdge 6900 series. This guide
is written for Sun personnel who have been fully trained on all the components in
the configuration.
TM
3900 and
How This Book Is Organized
This book contains the following topics:
Chapter 1 introduces the Sun StorEdge 3900 and 6900 series storage subsystems.
Chapter 2 offers general troubleshooting guidelines, such as quiescing the I/O, and
tools you can use to isolate and troubleshoot problems.
Chapter 3 provides Fibre Channel link troubleshooting procedures.
Chapter 4 presents information about configuration settings, specific to the Sun
StorEdge 3900 and 6900 series. It also provides a procedure for how to clear the lock
file.
Chapter 5 provides information on host device troubleshooting.
Chapter 6 provides information on Sun StorEdge network FC switch-8 and switch-
16 switch device troubleshooting.
xi
Chapter 7 provides detailed information for troubleshooting the virtualization
engines.
Chapter 8 describes how to troubleshoot the Sun StorEdge T3+ array devices. Also
included in this chapter is information about the Explorer Data Collection Utility.
Chapter 9 discusses ethernet hub troubleshooting. Information associated with the
3COM Ethernet hubs is limited in this guide, however, as this is third-party
information.
Appendix A provides virtualization engine references, including SRN and SNMP
Reference, an SRN/SNMP single point of failure table, and port communication and
service code tables.
Appendix B provides a list of SUNWsecfg Error Messages and recommendations for
corrective action.
Using UNIX Commands
This document may not contain information on basic UNIX®commands and
procedures such as shutting down the system, booting the system, and configuring
devices.
See one or more of the following for this information:
■ Solaris Handbook for Sun Peripherals
■ AnswerBook2™ online documentation for the Solaris™ operating environment
■ Other software documentation that you received with your system
xii Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
Typographic Conventions
Typeface Meaning Examples
AaBbCc123 The names of commands, files,
and directories; on-screen
computer output
AaBbCc123
AaBbCc123 Book titles, new words or terms,
What you type, when
contrasted with on-screen
computer output
words to be emphasized
Command-line variable; replace
with a real name or value
Edit your.login file.
Use ls -a to list all files.
% You have mail.
%
su
Password:
Read Chapter 6 in the User’s Guide.
These are called class options.
You must be superuser to do this.
To delete a file, type rm filename.
Shell Prompts
Shell Prompt
C shell machine_name%
C shell superuser machine_name#
Bourne shell and Korn shell $
Bourne shell and Korn shell superuser #
Preface xiii
Related Documentation
Product Title Part Number
Late-breaking News • Sun StorEdge 3900 and 6900 Series Release Notes 816-3247
Sun StorEdge 3900 and 6900
series hardware information
Sun StorEdge T3 and T3+
array
Diagnostics • Storage Automated Diagnostics Environment User’s Guide 816-3142
Sun StorEdge network FC
switch-8 and switch-16
SANbox switch management
using SANsurfer
Expansion cabinet • Sun StorEdge Expansion Cabinet Installation and Service
Storage server processor • Netra X1 Server User’s Guide
• Sun StorEdge 3900 and 6900 Series Site Preparation Guide
• Sun StorEdge 3900 and 6900 Series Regulatory and Safety
Compliance Manual
• Sun StorEdge 3900 and 6900 Series Hardware Installation and
Service Manual
• Sun StorEdge T3 and T3+ Array Start Here
• Sun StorEdge T3 and T3+ Array Installation, Operation, and
Service Manual
• Sun StorEdge T3 and T3+ Array Administrator’s Guide
• Sun StorEdge T3 and T3+ Array Configuration Guide
• Sun StorEdge T3 and T3+ Array Site Preparation Guide
• Sun StorEdge T3 and T3+ Field Service Manual
• Sun StorEdge T3 and T3+ Array Release Notes
• Sun StorEdge Network FC Switch-8 and Switch-16 Release Notes
• Sun StorEdge Network FC Switch-8 and Switch-16 Installation
and Configuration Guide
• Sun StorEdge Network FC Switch-8 and Switch-16 Best
Practices Manual
• Sun StorEdge Network FC Switch-8 and Switch-16 Operations
Guide
• Sun StorEdge Network FC Switch-8 and Switch-16 Field
Troubleshooting Guide
• SANbox 8/16 Segmented Loop Switch Management User ’s
Manual
• SANbox-8 Segmented Loop Fibre Channel Switch Installer’s/
User’s Manual
• SANbox-16 Segmented Loop Fibre Channel Switch Installer’s/
User’s Manual
Manual
• Netra X1 Server Hard Disk Drive Installation Guide
816-3242
816-3243
816-3244
816-0772
816-0773
816-0776
816-0777
816-0778
816-0779
816-0781
816-0842
816-0830
816-2688
816-1986
816-1701
875-3060
875-1881
875-3059
805-3067
806-5980
806-7670
xiv Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
Accessing Sun Documentation Online
A broad selection of Sun system documentation is located at:
http://www.sun.com/products-n-solutions/hardware/docs
A complete set of Solaris documentation and many other titles are located at:
http://docs.sun.com
Sun Welcomes Your Comments
Sun is interested in improving its documentation and welcomes your comments and
suggestions. You can email your comments to Sun at:
docfeedback@sun.com
Please include the part number (816-4290-10) of your document in the subject line of
your email.
Preface xv
xvi Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
CHAPTER
1
Introduction
The Sun StorEdge 3900 and 6900 series storage subsystems are complete
preconfigured storage solutions. The configurations for each of the storage
subsystems are shown in
TABLE1-1
Series System
Sun StorEdge
3900 series
Sun StorEdge
3910 system
TABLE 1-1.
Sun StorEdge
Fibre Channel
Switch Supported
Two 8-port
switches
Sun StorEdge T3+
Array Partner Groups
Supported
1to4
Additional Array
Partner Groups
Supported with
Optional Additional
Expansion Cabinet
Not applicable
Sun StorEdge
6900 series
Sun StorEdge
3960 system
Sun StorEdge
6910 system
Sun StorEdge
6960 system
Two 16-port
switches
Two 8-port
switches
Two 16-port
switches
1to4
1to3
1to3
1to5
1to4
1
Predictive Failure Analysis Capabilities
The Storage Automated Diagnostic Environment software provides the health and
monitoring functions for the Sun StorEdge 3900 and 6900 series systems. This
software provides the following predictive failure analysis (PFA) capabilities.
■ FC links—Fibre Channel links are monitored at all end points using the link FC-
ELS link counters. When link errors surpass the threshold values, an alert is sent.
This enables Sun personnel to replace components that are experiencing high
transient fault levels before a hard fault occurs.
■ Enclosure status—Many devices, like the Sun StorEdge network FC switch-8 and
switch-16 switch and the Sun StorEdge T3+ array, will cause the Storage
Automated Diagnostic Environment alerts to be sent if the temperature
thresholds are exceeded. This enables Sun-trained personnel to address the
problem before the component and enclosure fails.
■ SPOF notification—Storage Automated Diagnostic Environment notification for
path failures and failovers (that is, Sun StorEdge Traffic Manager software
failover) can be considered PFA, since Sun-trained personnel are notified and can
repair the primary path. This eliminates the time of exposure to single points of
failure and helps to preserve customer availability during the repair process.
PFA is not always effective in detecting or isolating failures. The remainder of this
document provides guidelines that can be used to troubleshoot problems that occur
in supported components of the Sun StorEdge 3900 and 6900 series.
2 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
CHAPTER
2
General Troubleshooting Procedures
This chapter contains the following sections:
■ “Troubleshooting Overview Tasks” on page 3
■ “Multipathing Options in the Sun StorEdge 6900 Series” on page 7
■ “Fibre Channel Links” on page 15
■ “Storage Automated Diagnostic Environment Event Grid” on page 21
Troubleshooting Overview Tasks
This section lists the high-level steps to isolate and troubleshoot problems in the Sun
StorEdge 3900 and 6900 series. It offers a methodical approach and lists the tools and
resources available at each step.
Note – A single problem can cause various errors throughout the SAN. A good
practice is to begin by investigating the devices that have experienced “Loss of
Communication” events in the Storage Automated Diagnostic Environment. These
errors usually indicate more serious problems.
A “Loss of Communication” error on a switch, for example, could cause multiple
ports and HBAs to go offline. Concentrating on the switch and fixing that failure can
help bring the ports and HBAs back online.
3
1. Discover the error by checking one or more of the following messages or files:
■ Storage Automated Diagnostic Environment alerts or email messages
■ /var/adm/messages
■ Sun StorEdge T3+ array syslog file
■ Storage Service Processor messages
■ /var/adm/messages.t3 messages
■ /var/adm/log/SEcfglog file
2. Determine the extent of the problem by using one or more of the following methods:
■ Storage Automated Diagnostic Environment Topology view
■ Storage Automated Diagnostic Environment Revision Checking (manual patch or
package, to check whether the package or patch is installed)
■ Verify the functionality using one of the following:
■ checkdefaultconfig(1M)
■ checkt3config(1M)
■ cfgadm -al output
■ luxadm(1M) output
■ Check the multipathing status using the Sun StorEdge Traffic Manager software
or VxDMP.
3. Check the status of a Sun StorEdge T3+ array by using one or more of the following methods:
■ Storage Automated Diagnostic Environment device monitoring reports
■ Run the SEcfg script, which displays and shows the Sun StorEdge T3+ array
configuration
■ Manually open a telnet session to the Sun StorEdge T3+ array
■ luxadm(1M) display output
■ LED status on the Sun StorEdge T3+ array
■ Explorer Data Collection Utility output (located on the Storage Service Processor)
4 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
4. Check the status of the Sun StorEdge FC network switch-8 and switch-16 switches
using the following tools:
■ Storage Automated Diagnostic Environment device monitoring reports
■ Run the SEcfg script, which displays and shows the Sun StorEdge T3+ array
configuration
■ LED Status (online/offline, POST error codes found in the Sun StorEdge network
FC switch-8 and switch-16 switch Installation and Configuration Guide)
■ Explorer Data Collection Utility output (located on the Storage Service Processor)
■ SANsurfer GUI
Note – To run the SANsurfer GUI from the Storage Service Processor, you must
export X-Display.)
5. Check the status of the virtualization engine using one or more of the following methods:
■ Storage Automated Diagnostic Environment device monitoring reports
■ Run the SEcfg script, which displays and shows the virtualization engine
■ Refer to the LED status blink codes in Chapter 7.
6. Quiesce the I/O along the path to be tested as follows:
■ For installations using VERITAS VxDMP, disable vxdmpadm
■ For installations using the Sun StorEdge Traffic Manager software, unconfigure
the Fabric device.
■ Refer to “To Quiesce the I/O” on page 8
■ Halt the application.
7. Test and isolate the FRUs using the following tools:
■ Storage Automated Diagnostic Environment diagnostic tests (this might require
the use of a loopback cable for isolation)
■ Sun StorEdge T3+ array tests, including t3test(1M), t3ofdg(1M), and
t3volverify(1M), which can be found in the Storage Automated Diagnostic
Environment User’s Guide.
Note – These tests isolate the problem to a FRU that must be replaced. Follow the
instructions in the Sun StorEdge 3900 and 6900 Series Reference Manual and the Sun
StorEdge 3900 and 6900 Installation and Service Manual for proper FRU replacement
procedures.
Chapter 2 General Troubleshooting Procedures 5
For Internal Use Only
8. Verify the fix using the following tools:
■ Storage Automated Diagnostic Environment GUI Topology View and Diagnostic
Tests
■ /var/adm/messages on the data host
9. Return the path to service by using one of the following methods:
■ Multipathing software
■ Restarting the application
6 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
Multipathing Options in the Sun StorEdge 6900 Series
Using the virtualization engines presents several challenges in how multipathing is
handled in the Sun StorEdge 6900 series.
Unlike Sun StorEdge T3+ array and Sun StorEdge network FC switch-8 and switch16 switch installations, which present primary and secondary pathing options, the
virtualization engines present only primary pathing options to the data host. The
virtualization engines handle all failover and failback operations and mask those
operations from the multipathing software on the data host.
The following example illustrates a Sun StorEdge Traffic Manager problem on a Sun
StorEdge 6900 series system.
# luxadm display
/dev/rdsk/c6t29000060220041F96257354230303052d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/
c6t29000060220041F96257354230303052d0s2
Status(Port A): O.K.
Status(Port B): O.K.
Vendor: SUN
Product ID: SESS01
WWN(Node): 2a000060220041f4
WWN(Port A): 2b000060220041f4
WWN(Port B): 2b000060220041f9
Revision: 080C
Serial Num: Unsupported
Unformatted capacity: 102400.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
/dev/rdsk/c6t29000060220041F96257354230303052d0s2
/devices/scsi_vhci/ssd@g29000060220041f96257354230303052:c,raw
Controller /devices/pci@6,4000/SUNW,qlc@2/fp@0,0
Device Address 2b000060220041f4,0
Class primary
State ONLINE
Controller /devices/pci@6,4000/SUNW,qlc@3/fp@0,0
Device Address 2b000060220041f9,0
Class primary
State ONLINE
For Internal Use Only
Chapter 2 General Troubleshooting Procedures 7
Note that in the Class and State fields, the virtualization engines are presented as
two primary/ONLINE devices. The current Sun StorEdge Traffic Manager design
does not enable you to manually halt the I/O (that is, you cannot perform a failover
to the secondary path) when only primary devices are present.
Alternatives to Sun StorEdge Traffic Manager
As an alternative to using Sun StorEdge Traffic Manager, you can manually halt the
I/O using one of two methods: quiesce I/O and unconfigure the c2 path. These
methods are explained below.
▼ To Quiesce the I/O
1. Determine the path you want to disable.
2. Type:
# cfgadm -c unconfigure device
▼ To Unconfigure the c2 Path
1. Type:
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 disk connected configured unknown
c0::dsk/c0t1d0 disk connected configured unknown
c1 scsi-bus connected configured unknown
c1::dsk/c1t6d0 CD-ROM connected configured unknown
c2 fc-fabric connected configured unknown
c2::210100e08b23fa25 unknown connected unconfigured unknown
c2::2b000060220041f4 disk connected configured unknown
c3 fc-fabric connected configured unknown
c3::210100e08b230926 unknown connected unconfigured unknown
c3::2b000060220041f9 disk connected configured unknown
c4 fc-private connected unconfigured unknown
c5 fc connected unconfigured unknown
8 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
2. Using Storage Automated Diagnostic Environment Topology GUI, determine
which virtualization engine is in the path you need to disable.
3. Use the world wide name (WWN) of the virtualization engine that is in the
unconfigure command, as follows:
# cfgadm -c unconfigure c2::2b000060220041f4
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 disk connected configured unknown
c0::dsk/c0t1d0 disk connected configured unknown
c1 scsi-bus connected configured unknown
c1::dsk/c1t6d0 CD-ROM connected configured unknown
c2 fc-fabric connected unconfigured unknown
c2::210100e08b23fa25 unknown connected unconfigured unknown
c2::2b000060220041f4 disk connected unconfigured unknown
c3 fc-fabric connected configured unknown
c3::210100e08b230926 unknown connected unconfigured unknown
c3::2b000060220041f9 disk connected configured unknown
c4 fc-private connected unconfigured unknown
c5 fc connected unconfigured unknown
4. Verify that I/O has halted.
This halts the I/O only up to the A3/B3 link (see
FIGURE 2-2). I/O continues to move
over the T1 and T2 paths, as well as the A4/B4 links to the Sun StorEdge T3+ array.
For Internal Use Only
Chapter 2 General Troubleshooting Procedures 9
▼ To Suspend the I/O
Use one of the following methods to suspend the I/O while the failover occurs:
1. Stop all customer applications that are accessing the Sun StorEdge T3+ array.
2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait
for a Sun StorEdge T3+ array LUN failover.
■ After the failover occurs, replace the cable and proceed with testing and FRU
isolation.
■ After testing and any FRU replacement is finished, return the Controller state
back to the default by using virtualization engine failback. Refer to
“Virtualization Engine Failback” on page 81.
Note – To confirm that a failover is occurring, open a telnet session to the Sun
StorEdge T3+ array and check the output of port listmap.
Another, but slower, method is to run the runsecfg script and verify the
virtualization engine maps by polling them against a live system.
Caution – During the failover, SCSI errors will occur on the data host and a brief
suspension of I/O will occur.
▼ To Return the Path to Production
1. Type cfgadm -c configure device.
# cfgadm -c configure c2::2b000060220041f4
2. Verify that I/O has resumed on all paths.
10 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
▼
To View the VxDisk Properties
1. Type the following:
# vxdisk list Disk_1
Device: Disk_1
devicetag: Disk_1
type: sliced
hostid: diag.xxxxx.xxx.COM
disk: name=t3dg02 id=1010283311.1163.diag.xxxxx.xxx.com
group: name=t3dg id=1010283312.1166.diag.xxxxx.xxx.com
flags: online ready private autoconfig nohotuse autoimport imported
pubpaths: block=/dev/vx/dmp/Disk_1s4 char=/dev/vx/rdmp/Disk_1s4
privpaths: block=/dev/vx/dmp/Disk_1s3 char=/dev/vx/rdmp/Disk_1s3
version: 2.2
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=209698816
private: slice=3 offset=1 len=4095
update: time=1010434311 seqno=0.6
headers: 0 248
configs: count=1 len=3004
logs: count=1 len=455
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-003021[002773]: copy=01 offset=000231 enabled
log priv 003022-003476[000455]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 2
c20t2B000060220041F4d0s2 state=enabled
c23t2B000060220041F9d0s2 state=enabled
# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c0 OTHER_DISKS ENABLED OTHER_DISKS
c2 SENA ENABLED SENA0
c3 SENA ENABLED SENA0
c20 Disk ENABLED Disk
c23 Disk ENABLED Disk
From the VxDisk output, notice that there are two physical paths to the LUN:
■ c20t2B000060220041F4d0s2
■ c23t2B000060220041F9d0s2
Both of these paths are currently enabled with VxDMP.
Chapter 2 General Troubleshooting Procedures 11
For Internal Use Only
2. Use the luxadm(1M) command to display further information about the underlying LUN.
# luxadm display /dev/rdsk/c20t2B000060220041F4d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c20t2B000060220041F4d0s2
Status(Port A): O.K.
Vendor: SUN
Product ID: SESS01
WWN(Node): 2a000060220041f4
WWN(Port A): 2b000060220041f4
Revision: 080C
Serial Num: Unsupported
Unformatted capacity: 102400.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
/dev/rdsk/c20t2B000060220041F4d0s2
/devices/pci@a,2000/pci@2/SUNW,qlc@4/fp@0,0
ssd@w2b000060220041f4,0:c,raw
# luxadm display /dev/rdsk/c23t2B000060220041F9d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c23t2B000060220041F9d0s2
Status(Port A): O.K.
Vendor: SUN
Product ID: SESS01
WWN(Node): 2a000060220041f9
WWN(Port A): 2b000060220041f9
Revision: 080C
Serial Num: Unsupported
Unformatted capacity: 102400.000 MBytes
Write Cache: Enabled
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
/dev/rdsk/c23t2B000060220041F9d0s2
/devices/pci@e,2000/pci@2/SUNW,qlc@4/fp@0,0/
ssd@w2b000060220041f9,0:c,raw
12 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
▼
To Quiesce the I/O on the A3/B3 Link
1. Determine the path you want to disable.
2. Disable the path by typing the following:
# vxdmpadm disable ctlr=<c#>
3. Verify that the path is disabled:
# vxdmpadm listctlr all
Steps 1 and 2 halt I/O only up to the A3/B3 link. I/O will continue to move over the
T1 & T2 paths, as well as the A4/B4 links to the Sun StorEdge T3+ array.
▼ To Suspend the I/O on the A3/B3 Link
Use one of the following methods to suspend I/O while the failover occurs:
1. Stop all customer applications that are accessing the Sun StorEdge T3+ array.
2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait
for a Sun StorEdge T3+ array LUN failover.
a. After the failover occurs, replace the cable and proceed with testing and FRU
isolation.
b. After testing is complete and any FRU replacement is finished, return the
controller state back to the default by using the virtualization engine failback
command.
Caution – This action will cause SCSI errors on the data host and a brief suspension
of I/O while the failover occurs.
Chapter 2 General Troubleshooting Procedures 13
For Internal Use Only
▼ To Return the Path to Production
1. Type:
# vxdmpadm enable ctlr=<c#>
2. Verify that the path has been re-enabled by typing:
# vxdmpadm listctlr all
14 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002
Loading...