IBM System Storage DS4000 EXP420 Installation, User And Maintenance Instructions

IBM System Storage DS4000 EXP420 Storage Expansion Enclosure

Installation, User’ s an d Maintenance Guid e
GC27-2050-02
IBM System Storage DS4000 EXP420 Storage Expansion Enclosure

Installation, User’ s an d Maintenance Guid e
GC27-2050-02
Note:
Before using this information and the product it supports, be sure to read the general information in “Notices” on page 137.
Third Edition (December 2007)
US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix
Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi
About this document . . . . . . . . . . . . . . . . . . . . . xvii
Who should read this document . . . . . . . . . . . . . . . . . . xvii
How this document is organized . . . . . . . . . . . . . . . . . . xvii
DS4000 Storage Subsystem installation tasks - General overview . . . . . . xviii
Getting information, help, and service . . . . . . . . . . . . . . . . xxii
Before you call . . . . . . . . . . . . . . . . . . . . . . . . xxii
Using the documentation . . . . . . . . . . . . . . . . . . . . xxii
Finding DS4000 readme files . . . . . . . . . . . . . . . . . . xxiii
Web sites . . . . . . . . . . . . . . . . . . . . . . . . . xxiii
Software service and support . . . . . . . . . . . . . . . . . . xxiv
Hardware service and support . . . . . . . . . . . . . . . . . . xxiv
Fire suppression systems . . . . . . . . . . . . . . . . . . . xxv
How to send your comments . . . . . . . . . . . . . . . . . . xxv
Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . .1
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . .1
Operating system support . . . . . . . . . . . . . . . . . . . .2
Fibre channel defined . . . . . . . . . . . . . . . . . . . . . .2
SATA defined . . . . . . . . . . . . . . . . . . . . . . . . .2
Inventory checklist . . . . . . . . . . . . . . . . . . . . . . . .2
EXP420 components . . . . . . . . . . . . . . . . . . . . . . .3
Enhanced Value Disk Drive Modules (EV-DDMs) . . . . . . . . . . . .3
Environmental Service Modules (ESMs) . . . . . . . . . . . . . . .4
Connectors, switch, and enclosure ID . . . . . . . . . . . . . . .5
Rear drive channel ports . . . . . . . . . . . . . . . . . . . .6
Power supply and fan units . . . . . . . . . . . . . . . . . . . .6
Small Form-Factor Pluggable (SFP) modules . . . . . . . . . . . . .8
Software and hardware compatibility and upgrades . . . . . . . . . . . .9
Software and firmware support code upgrades . . . . . . . . . . . . .9
EXP420 hardware and software compatibility . . . . . . . . . . . . .10
Determining firmware levels . . . . . . . . . . . . . . . . . . .10
Product updates and support notifications . . . . . . . . . . . . . .12
Best practices guidelines . . . . . . . . . . . . . . . . . . . .12
Specifications . . . . . . . . . . . . . . . . . . . . . . . . .13
Area requirements . . . . . . . . . . . . . . . . . . . . . .13
Dimensions . . . . . . . . . . . . . . . . . . . . . . . .13
Weight . . . . . . . . . . . . . . . . . . . . . . . . . .14
Shipping dimensions . . . . . . . . . . . . . . . . . . . . .14
Environmental requirements and specifications . . . . . . . . . . . .14
Temperature and humidity . . . . . . . . . . . . . . . . . . .15
Altitude . . . . . . . . . . . . . . . . . . . . . . . . . .15
Airflow and heat dissipation . . . . . . . . . . . . . . . . . .15
Acoustic noise . . . . . . . . . . . . . . . . . . . . . . .16
Electrical requirements . . . . . . . . . . . . . . . . . . . . .16
Site wiring and power . . . . . . . . . . . . . . . . . . . .17
Power recovery after ac power failure . . . . . . . . . . . . . .17
Power cords and receptacles . . . . . . . . . . . . . . . . . .17
© Copyright IBM Corp. 2006, 2007 iii
Heat output, airflow, and cooling . . . . . . . . . . . . . . . . .17
Chapter 2. Installing the EXP420 . . . . . . . . . . . . . . . . . .19
Installation overview . . . . . . . . . . . . . . . . . . . . . . .19
Handling static-sensitive devices . . . . . . . . . . . . . . . . . .20
Preparing for installation . . . . . . . . . . . . . . . . . . . . .21
Tools and hardware required . . . . . . . . . . . . . . . . . . .21
Preparing the site . . . . . . . . . . . . . . . . . . . . . . .22
Preparing the rack . . . . . . . . . . . . . . . . . . . . . .22
Installing the support rails . . . . . . . . . . . . . . . . . . . . .23
Installing the EXP420 . . . . . . . . . . . . . . . . . . . . . .26
Removing the CRUs . . . . . . . . . . . . . . . . . . . . . .26
Removing an ESM . . . . . . . . . . . . . . . . . . . . .27
Removing a power supply and fan unit . . . . . . . . . . . . . .28
Removing an EV-DDM . . . . . . . . . . . . . . . . . . . .29
Installing the EXP420 into a rack . . . . . . . . . . . . . . . . .30
Replacing the EXP420 CRUs . . . . . . . . . . . . . . . . . .32
Replacing an ESM . . . . . . . . . . . . . . . . . . . . .32
Replacing a power supply and fan unit . . . . . . . . . . . . . .33
Replacing an EV-DDM . . . . . . . . . . . . . . . . . . . .34
Chapter 3. Cabling the EXP420 . . . . . . . . . . . . . . . . . .37
Enclosure ID settings . . . . . . . . . . . . . . . . . . . . . .37
Fibre-channel loop and ID settings . . . . . . . . . . . . . . . . . .38
Cabling the EXP420 . . . . . . . . . . . . . . . . . . . . . . .38
Installing SFP modules . . . . . . . . . . . . . . . . . . . . . .39
Removing SFP modules . . . . . . . . . . . . . . . . . . . . .41
Handling fibre-optic cables . . . . . . . . . . . . . . . . . . . .42
Using LC-LC fibre-channel cables . . . . . . . . . . . . . . . . . .43
Connecting an LC-LC cable to an SFP module . . . . . . . . . . . .44
Removing an LC-LC fibre-channel cable . . . . . . . . . . . . . .45
Cabling the EXP420 to a storage subsystem . . . . . . . . . . . . . .46
EXP420 loop redundancy . . . . . . . . . . . . . . . . . . . . .48
Example of a correct redundant drive loop . . . . . . . . . . . . . .48
Example of an incorrect redundant drive loop . . . . . . . . . . . . .50
Cabling the power supply . . . . . . . . . . . . . . . . . . . . .52
Chapter 4. Operating the EXP420 . . . . . . . . . . . . . . . . .53
Performing the DS4000 Health Check process . . . . . . . . . . . . .53
Web pages . . . . . . . . . . . . . . . . . . . . . . . . .54
Hardware responsibilities . . . . . . . . . . . . . . . . . . . .54
Powering on the EXP420 . . . . . . . . . . . . . . . . . . . . .55
Turning on the EXP420 . . . . . . . . . . . . . . . . . . . .55
Firmware updates . . . . . . . . . . . . . . . . . . . . . . . .57
Finding controller, storage expansion enclosure, and drive information . . . .58
Troubleshooting the storage expansion enclosure . . . . . . . . . . . .59
Checking the LEDs . . . . . . . . . . . . . . . . . . . . . . .59
Power supply and fan unit LEDs . . . . . . . . . . . . . . . . .60
Front LEDs . . . . . . . . . . . . . . . . . . . . . . . . .60
Rear LEDs . . . . . . . . . . . . . . . . . . . . . . . . .61
Seven-segment numeric display . . . . . . . . . . . . . . . . .62
Powering off the storage expansion enclosure . . . . . . . . . . . . .64
Turning off the EXP420 . . . . . . . . . . . . . . . . . . . .64
Performing an emergency shutdown . . . . . . . . . . . . . . . .67
Restoring power after an unexpected shutdown . . . . . . . . . . . .67
Recovering from an overheated power supply and fan unit . . . . . . . . .68
iv IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Chapter 5. Installing and replacing devices . . . . . . . . . . . . .73
Handling static-sensitive devices . . . . . . . . . . . . . . . . . .73
Service Action Allowed Status LED . . . . . . . . . . . . . . . . .73
Adding the EXP420 to an existing and configured redundant drive channel/loop
pair . . . . . . . . . . . . . . . . . . . . . . . . . . . .74
Working with hot-swap EV-DDMs . . . . . . . . . . . . . . . . . .76
Installing hot-swap EV-DDMs . . . . . . . . . . . . . . . . . . .78
Replacing hot-swap EV-DDMs . . . . . . . . . . . . . . . . . .80
Replacing multiple EV-DDMs . . . . . . . . . . . . . . . . . . .81
Replacing all EV-DDMs at the same time . . . . . . . . . . . . .82
Replacing the EV-DDMs one at a time . . . . . . . . . . . . . .84
Verifying the Link Rate setting . . . . . . . . . . . . . . . . . . .86
Replacing a power supply and fan unit . . . . . . . . . . . . . . . .89
Replacing an Environmental Service Module (ESM) . . . . . . . . . . .94
Replacing an SFP module . . . . . . . . . . . . . . . . . . . .96
Replacing a midplane . . . . . . . . . . . . . . . . . . . . . .98
Chapter 6. Hardware maintenance . . . . . . . . . . . . . . . . 101
General checkout . . . . . . . . . . . . . . . . . . . . . . . 101
Solving problems . . . . . . . . . . . . . . . . . . . . . . . 101
Parts listing . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Appendix A. Records . . . . . . . . . . . . . . . . . . . . . 109
Identification numbers . . . . . . . . . . . . . . . . . . . . . . 109
Storage subsystem and controller information record . . . . . . . . . .110
Sample information record . . . . . . . . . . . . . . . . . . . 111
Installed device records . . . . . . . . . . . . . . . . . . . . .112
Appendix B. Rack mounting template . . . . . . . . . . . . . . .113
Appendix C. Power cords . . . . . . . . . . . . . . . . . . . .117
Appendix D. Additional DS4000 documentation . . . . . . . . . . . 121
DS4000 Storage Manager Version 10 library . . . . . . . . . . . . . 121
DS4800 Storage Subsystem library . . . . . . . . . . . . . . . . . 122
DS4700 Storage Subsystem library . . . . . . . . . . . . . . . . . 123
DS4500 Storage Subsystem library . . . . . . . . . . . . . . . . . 124
DS4400 Storage Subsystem library . . . . . . . . . . . . . . . . . 125
DS4300 Storage Subsystem library . . . . . . . . . . . . . . . . . 126
DS4200 Express Storage Subsystem library . . . . . . . . . . . . . 127
DS4100 Storage Subsystem library . . . . . . . . . . . . . . . . . 128
DS4000 Storage Expansion Enclosure documents . . . . . . . . . . . 129
Other DS4000 and DS4000-related documents . . . . . . . . . . . . 130
Appendix E. Accessibility . . . . . . . . . . . . . . . . . . . . 131
Appendix F. Product Recycling and Disposal . . . . . . . . . . . . 133
Appendix G. Battery Return Program . . . . . . . . . . . . . . . 135
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Important notes . . . . . . . . . . . . . . . . . . . . . . . . 138
Product recycling and disposal . . . . . . . . . . . . . . . . . . 138
Battery return program . . . . . . . . . . . . . . . . . . . . . 139
Contact Information . . . . . . . . . . . . . . . . . . . . . . . 140
Contents v
Electronic emission notices . . . . . . . . . . . . . . . . . . . . 140
Federal Communications Commission (FCC) statement . . . . . . . . 140
Chinese class A compliance statement . . . . . . . . . . . . . . . 140
Industry Canada Class A emission compliance statement . . . . . . . . 141
Australia and New Zealand Class A statement . . . . . . . . . . . . 141
United Kingdom telecommunications safety requirement . . . . . . . . 141
European Union EMC Directive conformance statement . . . . . . . . 141
Taiwan Class A warning statement . . . . . . . . . . . . . . . . 141
Japanese Voluntary Control Council for Interference (VCCI) statement 142
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
vi IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

Figures

1. EXP420 hot-swap drive bays . . . . . . . . . . . . . . . . . . . . . . . . . . .3
2. Back view; power switch, enclosure ID, and connectors . . . . . . . . . . . . . . . . .5
3. Rear view of drive channel ports . . . . . . . . . . . . . . . . . . . . . . . . .6
4. Power supply and fan unit components . . . . . . . . . . . . . . . . . . . . . . .7
5. Power supply and fan unit and airflow . . . . . . . . . . . . . . . . . . . . . . .8
6. SFP module and fibre optic cable . . . . . . . . . . . . . . . . . . . . . . . . .9
7. EXP420 dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
8. EXP420 airflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
9. Example of cold aisle/hot aisle rack configuration . . . . . . . . . . . . . . . . . . .18
10. Example of EXP420 serial number, machine type, and model number location . . . . . . . .20
11. Front rack mounting template . . . . . . . . . . . . . . . . . . . . . . . . . .23
12. Rear rack mounting template . . . . . . . . . . . . . . . . . . . . . . . . . .24
13. Installing the support rails . . . . . . . . . . . . . . . . . . . . . . . . . . .26
14. Removing and replacing an Environmental Service Module (ESM) . . . . . . . . . . . .28
15. Removing a power supply and fan unit . . . . . . . . . . . . . . . . . . . . . . .29
16. Removing a EV-DDM CRU . . . . . . . . . . . . . . . . . . . . . . . . . . .30
17. Installing the EXP420 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31
18. Securing the storage expansion enclosure to the rack . . . . . . . . . . . . . . . . .32
19. Removing and replacing an Environmental Service Module (ESM) . . . . . . . . . . . .33
20. Replacing a power supply and fan unit . . . . . . . . . . . . . . . . . . . . . . .34
21. Replacing an EV-DDM . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35
22. Storage expansion enclosure seven-segment enclosure ID . . . . . . . . . . . . . . .38
23. SFP module and protective cap . . . . . . . . . . . . . . . . . . . . . . . . .40
24. Installing an SFP module into the host port . . . . . . . . . . . . . . . . . . . . .41
25. Unlocking the SFP module latch - plastic variety . . . . . . . . . . . . . . . . . . .41
26. Unlocking the SFP module latch - wire variety . . . . . . . . . . . . . . . . . . . .42
27. Recommended bending and looping specifications for fibre-optic cables . . . . . . . . . .43
28. LC-LC fibre-channel cable . . . . . . . . . . . . . . . . . . . . . . . . . . .43
29. Removing fibre-optic cable protective caps . . . . . . . . . . . . . . . . . . . . .44
30. Inserting an LC-LC fibre-channel cable into an SFP module . . . . . . . . . . . . . . .45
31. LC-LC fibre-channel cable lever and latches . . . . . . . . . . . . . . . . . . . . .45
32. Removing the LC-LC fibre-channel cable . . . . . . . . . . . . . . . . . . . . . .46
33. Installing an SFP module and LC-LC cable in a supported DS4000 storage subsystem . . . . .47
34. Installing an SFP module and connecting an LC-LC cable to the EXP420 . . . . . . . . . .48
35. Correct EXP420 redundant drive loop configuration . . . . . . . . . . . . . . . . . .49
36. Incorrect EXP420 redundant drive loop configuration . . . . . . . . . . . . . . . . .51
37. Power supply switches and connectors . . . . . . . . . . . . . . . . . . . . . . .56
38. Power supply and fan unit LEDs . . . . . . . . . . . . . . . . . . . . . . . . .60
39. Front LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
40. Rear LEDs, controls, and connectors . . . . . . . . . . . . . . . . . . . . . . .62
41. Numeric display LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63
42. Power supply switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70
43. Cabling the EXP420 to an existing and configured drive loop . . . . . . . . . . . . . .75
44. Hot-swap EV-DDM LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
45. EV-DDM CRU handle . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80
46. Link rate LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88
47. Link rate switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .88
48. Replacing a power supply and fan unit . . . . . . . . . . . . . . . . . . . . . . .93
49. Removing and replacing an Environmental Service Module (ESM) . . . . . . . . . . . .95
50. Replacing an SFP module . . . . . . . . . . . . . . . . . . . . . . . . . . .97
51. Replacing a midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99
52. EXP420 storage expansion enclosure parts list . . . . . . . . . . . . . . . . . . . 108
53. Front rack mounting template . . . . . . . . . . . . . . . . . . . . . . . . . .114
© Copyright IBM Corp. 2006, 2007 vii
54. Rear rack mounting template . . . . . . . . . . . . . . . . . . . . . . . . . .115
viii IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

Tables

1. Where to find DS4000 installation and configuration procedures . . . . . . . . . . . . . xviii
2. Description of Figure 2 on page 5 . . . . . . . . . . . . . . . . . . . . . . . . .5
3. Description of Figure 3 on page 6 . . . . . . . . . . . . . . . . . . . . . . . . .6
4. Description of Figure 4 on page 7 . . . . . . . . . . . . . . . . . . . . . . . . .7
5. Software and firmware levels for the EXP420 . . . . . . . . . . . . . . . . . . . . .9
6. EXP420 hardware and software compatibility . . . . . . . . . . . . . . . . . . . .10
7. EXP420 weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
8. EXP420 component weights . . . . . . . . . . . . . . . . . . . . . . . . . .14
9. EXP420 shipping carton dimensions . . . . . . . . . . . . . . . . . . . . . . . .14
10. Temperature and humidity requirements . . . . . . . . . . . . . . . . . . . . . .15
11. EXP420 altitude ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
12. EXP420 power and heat dissipation . . . . . . . . . . . . . . . . . . . . . . . .16
13. EXP420 sound levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16
14. EXP420 ac power requirements . . . . . . . . . . . . . . . . . . . . . . . . .17
15. Description of Figure 37 on page 56 . . . . . . . . . . . . . . . . . . . . . . . .57
16. Power supply and fan unit LEDs . . . . . . . . . . . . . . . . . . . . . . . . .60
17. Front LEDs and controls . . . . . . . . . . . . . . . . . . . . . . . . . . . .61
18. Rear LEDs, controls, and connectors . . . . . . . . . . . . . . . . . . . . . . .62
19. Numeric display diagnostic codes . . . . . . . . . . . . . . . . . . . . . . . . .64
20. Description of Figure 42 on page 70 . . . . . . . . . . . . . . . . . . . . . . . .70
21. Drive LED activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78
22. Data transfer rates for drive modules . . . . . . . . . . . . . . . . . . . . . . .87
23. Symptom-to-FRU index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
24. Parts listing (EXP420 storage expansion enclosure) . . . . . . . . . . . . . . . . . 108
25. Storage subsystem and controller information record . . . . . . . . . . . . . . . . .110
26. Sample information record . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
27. Hard disk drive record . . . . . . . . . . . . . . . . . . . . . . . . . . . .112
28. IBM power cords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117
29. DS4000 Storage Manager Version 10 titles by user tasks . . . . . . . . . . . . . . . 121
30. DS4800 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 122
31. DS4700 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 123
32. DS4500 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 124
33. DS4400 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 125
34. DS4300 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 126
35. DS4200 Express Storage Subsystem document titles by user tasks . . . . . . . . . . . 127
36. DS4100 Storage Subsystem document titles by user tasks . . . . . . . . . . . . . . . 128
37. DS4000 Storage Expansion Enclosure document titles by user tasks . . . . . . . . . . . 129
38. DS4000 and DS4000–related document titles by user tasks . . . . . . . . . . . . . . 130
39. DS4000 Storage Manager alternate keyboard operations . . . . . . . . . . . . . . . 131
© Copyright IBM Corp. 2006, 2007 ix
x IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

Safety

The caution and danger statements that this document contains can be referenced in the multilingual IBM® Safety Information document that is provided with your IBM System Storage™ EXP420 storage expansion enclosure. Each caution and danger statement is numbered for easy reference to the corresponding statements in the translated document.
v Danger: These statements indicate situations that can be potentially lethal or
extremely hazardous to you. A danger statement is placed just before the description of a potentially lethal or extremely hazardous procedure, step, or situation.
v Caution: These statements indicate situations that can be potentially hazardous
to you. A caution statement is placed just before the description of a potentially hazardous procedure step or situation.
v Attention: These notices indicate possible damage to programs, devices, or
data. An attention notice is placed just before the instruction or situation in which damage could occur.
Before installing this product, read the following danger and caution notices.
© Copyright IBM Corp. 2006, 2007 xi
Statement 1:
DANGER
Electrical
current from power, telephone, and communication cables is
hazardous.
To avoid a shock hazard: v Do not connect or disconnect any cables or perform installation,
maintenance, or reconfiguration of this product during an electrical storm.
v Connect all power cords to a properly wired and grounded electrical
outlet.
v Connect to properly wired outlets any equipment that will be attached to
this product.
v When possible, use one hand only to connect or disconnect signal
cables.
v Never turn on any equipment when there is evidence of fire, water, or
structural damage.
v Disconnect the attached power cords, telecommunications systems,
networks, and modems before you open the device covers, unless instructed otherwise in the installation and configuration procedures.
v Connect and disconnect cables as described in the following table when
installing, moving, or opening covers on this product or attached devices.
To Connect: To Disconnect:
1. Turn everything OFF.
2. First, attach all cables to devices.
3. Attach signal cables to connectors.
4. Attach power cords to outlet.
1. Turn everything OFF.
2. First, remove power cords from outlet.
3. Remove signal cables from connectors.
4. Remove all cables from devices.
5. Turn device ON.
xii IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Statement 3:
CAUTION: When laser products (such as CD-ROMs, DVD drives, fibre optic devices, or transmitters) are installed, note the following:
v Do not remove the covers. Removing the covers of the laser product could
result in exposure to hazardous laser radiation. There are no serviceable parts inside the device.
v Use of controls or adjustments or performance of procedures other than
those specified herein might result in hazardous radiation exposure.
DANGER
laser products contain an embedded Class 3A or Class 3B laser
Some diode. Note the following.
Laser radiation when open. Do not stare into the beam, do not view directly with optical instruments, and avoid direct exposure to the beam.
Class 1 Laser statement
Class 1 Laser Product Laser Klasse 1 Laser Klass 1 Luokan 1 Laserlaite Apparell Laser de Calsse 1À
IEC 825-11993 CENELEC EN 60 825
Safety xiii
Statement 4:
18 kg (39.7 lb) 32 kg (70.5 lb) 55 kg (121.2 lb)
CAUTION: Use safe practices when lifting.
Statement 5:
CAUTION: The power control button on the device and the power switch on the power supply do not turn off the electrical current supplied to the device. The device also might have more than one power cord. To remove all electrical current from the device, ensure that all power cords are disconnected from the power source.
2
1
xiv IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Statement 8:
CAUTION: Never remove the cover on a power supply or any part that has the following label attached.
Hazardous voltage, current, and energy levels are present inside any component that has this label attached. There are no serviceable parts inside these components. If you suspect a problem with one of these parts, contact a service technician.
Cable Warning:
WARNING: Handling the cord on this product or cords associated with accessories
sold with this product, will expose you to lead, a chemical known to the State of California to cause cancer, and birth defects or other reproductive harm. Wash
hands after handling.
Safety xv
xvi IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

About this document

This document provides instructions for installing and customizing the configuration of your IBM® System Storage DS4000 EXP420 storage expansion enclosure. It also provides maintenance procedures and troubleshooting information.

Who should read this document

This document is intended for system operators and service technicians who have extensive knowledge of Fibre Channel and network technology.

How this document is organized

Chapter 1, “Introduction,” on page 1 describes the IBM System Storage EXP420 storage expansion enclosure. This chapter includes an inventory checklist and an overview of the storage expansion enclosure features, operating specifications, and components.
Chapter 2, “Installing the EXP420,” on page 19 contains information on how to install the EXP420.
Chapter 3, “Cabling the EXP420,” on page 37 contains information on how to cable the EXP420.
Chapter 4, “Operating the EXP420,” on page 53 contains information on how to power on and off the storage expansion enclosure, recover from an overheated power supply and fan unit, troubleshoot the storage expansion enclosure, and interpret LEDs.
Chapter 5, “Installing and replacing devices,” on page 73 contains step-by-step instructions about how to install or remove customer replaceable units (CRUs), such as Enhanced Value Disk Drive Modules (EV-DDMs), power supply and fan units, environmental service modules (ESMs), midplane, and SFP modules.
Chapter 6, “Hardware maintenance,” on page 101 describes problems and symptoms that are specific to your storage expansion enclosure. It also provides a parts list for the EXP420.
Appendix A, “Records,” on page 109 provides a table that you can use to record and update important information about your EXP420, including serial number and device records. Whenever you add options to your EXP420, be sure to update the information in this table.
Appendix B, “Rack mounting template,” on page 113 provides the rack mounting templates for installation of the EXP420. If you want to tear out the templates from the document for use during installation, use these copies of the templates.
Appendix C, “Power cords,” on page 117 lists power cord information for the EXP420.
Appendix D, “Additional DS4000 documentation,” on page 121 lists additional DS4000 documents.
Appendix E, “Accessibility,” on page 131 details accessibility information.
© Copyright IBM Corp. 2006, 2007 xvii
Appendix F, “Product Recycling and Disposal,” on page 133 details the European directive about product recycling.

DS4000 Storage Subsystem installation tasks - General overview

Table 1 provides a sequential list of many installation and configuration tasks that are common to most DS4000 configurations. When you install and configure your DS4000 storage subsystem, refer to this table to find the documentation that explains how to complete each task.
Table 1. Where to find DS4000 installation and configuration procedures
Installation task Where to find information or procedures
1 Plan the installation
v DS4000 Storage Manager Concepts Guide
v DS4000 Storage Manager Installation and Host Support
Guide for AIX, HP-UX, Solaris, and Linux on POWER
v DS4000 Storage Manager Installation and Host Support
Guide for Windows 2000/Server 2003/2008, NetWare, VMWare ESX Server, and Linux
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User's, and
Maintenance Guide
v DS4400 Fibre Channel Storage Server Installation and
Support Guide
v DS4500 Storage Subsystem Installation, User's, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
2 Mount the DS4000
storage subsystem in the rack
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v DS4400 and DS4500 Rack Mounting Instructions
v DS4300 Rack Mounting Instructions
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4100 Storage Subsystem Installation, User’s and
Maintenance Guide
xviii IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Table 1. Where to find DS4000 installation and configuration procedures (continued)
Installation task Where to find information or procedures
3 Mount the DS4000
EXP storage expansion unit in the rack
v DS4000 EXP100 Storage Expansion Unit Installation, User’s
and Maintenance Guide
v DS4000 EXP420 Storage Expansion Enclosures Installation,
User’s, and Maintenance Guide
v DS4000 EXP700 and EXP710 Storage Expansion Enclosures
Installation, User’s, and Maintenance Guide
v DS4000 EXP810 Storage Expansion Enclosures Installation,
User’s, and Maintenance Guide
v FAStT EXP500 Installation and User’s Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
4 Route the storage
expansion unit Fibre Channel cables
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4400 Fibre Channel Cabling Instructions
v DS4500 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
5 Route the host
server Fibre Channel cables
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4400 Fibre Channel Cabling Instructions
v DS4500 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
About this document xix
Table 1. Where to find DS4000 installation and configuration procedures (continued)
Installation task Where to find information or procedures
6 Power up the
subsystem
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4400 Fibre Channel Storage Server Installation and
Support Guide
v DS4500 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
7 Configure DS4000
network settings
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4400 Fibre Channel Storage Server Installation and
Support Guide
v DS4500 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
8 Zone the fabric
switch (SAN-attached only)
v DS4000 Storage Manager Installation and Host Support
Guide for AIX, HP-UX, Solaris, and Linux on POWER
v DS4000 Storage Manager Installation and Host Guide for
Windows 2000/Server 2003/2008, NetWare, VMWare ESX Server, and Linux
v DS4000 Storage Manager Copy Services Guide (describes
switch zoning for the Remote Mirror Option)
v See also the documentation provided by the switch
manufacturer.
xx IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Table 1. Where to find DS4000 installation and configuration procedures (continued)
Installation task Where to find information or procedures
9 Install DS4000
Storage Manager software on the management station
10 Install host software
(failover drivers) on host server
11 Start DS4000
Storage Manager
12 Set the DS4000
Storage Manager clock
v DS4000 Storage Manager Installation and Host Support
Guide for AIX, HP-UX, Solaris, and Linux on POWER
v DS4000 Storage Manager Installation and Host Support
Guide for Windows 2000/Server 2003/2008, NetWare, VMWare ESX Server, and Linux
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
v DS4000 Storage Manager online help (for post-installation
tasks)
13 Set the DS4000
Storage Manager host default type
14 Verify DS4000
subsystem health
v DS4100 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4200 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4300 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4400 Fibre Channel Storage Server Installation and
Support Guide
v DS4500 Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4700 Express Storage Subsystem Installation, User’s, and
Maintenance Guide
v DS4800 Storage Subsystem Installation, User’s, and
Maintenance Guide
15 Enable DS4000
Storage Manager premium feature keys
Copy Services premium features
DS4000 Storage Manager Copy Services Guide
FC/SATA Intermix premium feature
DS4000 Fibre Channel and Serial ATA Intermix Premium Feature Installation Overview
Storage Partitioning (and general premium features information)
v DS4000 Storage Manager Concepts Guide
v DS4000 Storage Manager Installation and Host
Support Guide for AIX, HP-UX, Solaris, and Linux on POWER
v DS4000 Storage Manager Installation and Host
Support Guide for Windows 2000/Server 2003/2008, NetWare, VMWare ESX Server, and Linux
About this document xxi
Table 1. Where to find DS4000 installation and configuration procedures (continued)
Installation task Where to find information or procedures
16 Configure arrays and
logical drives
17 Configure host
partitions
18 Verify host access to
DS4000 storage
v DS4000 Storage Manager Installation and Host Support
Guide for AIX, HP-UX, Solaris, and Linux on POWER
v DS4000 Storage Manager Installation and Host Support
Guide for Windows 2000/Server 2003/2008, NetWare, VMWare ESX Server, and Linux
v IBM System Storage Quick Start Guide, Quick reference for
the DS4700 and DS4200, Sections 2, 3, and 4 also for installing the EXP810 and EXP420
v IBM System Storage Quick Start Guide, Quick reference for
the DS4800
v DS4000 Storage Manager online help

Getting information, help, and service

If you need help, service, or technical assistance or just want more information about IBM products, you will find a wide variety of sources available from IBM to assist you. This section contains information about where to go for additional information about IBM and IBM products, what to do if you experience a problem with your system, and whom to call for service, if it is necessary.

Before you call

Before you call, take these steps to try to solve the problem yourself:
v Check all cables to make sure that they are connected.
v Check the power switches to make sure that the system is turned on.
v Use the troubleshooting information in your system documentation, and use the
diagnostic tools that come with your system.
v Check for technical information, hints, tips, and new device drivers at the IBM
support Web site pages that are listed in this section.
v Use an IBM discussion forum on the IBM Web site to ask questions.
can solve many problems without outside assistance by following the
You troubleshooting procedures that IBM provides in the DS4000 Storage Manager online help or in the documents that are provided with your system and software. The information that comes with your system also describes the diagnostic tests that you can perform. Most subsystems, operating systems, and programs come with information that contains troubleshooting procedures and explanations of error messages and error codes. If you suspect a software problem, see the information for the operating system or program.

Using the documentation

Information about your IBM system and preinstalled software, if any, is available in the documents that come with your system. This includes printed books, online documents, readme files, and help files. See the troubleshooting information in your system documentation for instructions for using the diagnostic programs. The troubleshooting information or the diagnostic programs might tell you that you need additional or updated device drivers or other software.
xxii IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

Finding DS4000 readme files

1. Go to the following Web site:
www.ibm.com/systems/support/storage/disk
2. Click the link for your Storage Subsystem (for example, DS4800).
3. When the subsystem support page opens, click the Download tab.
4. Under the Download tab, click Storage Manager, firmware, HBA, tools,
support & pubs (including readmes).
5. Now, click the appropriate tab for the type of readme file that you are looking
for:
v Firmware v Storage Mgr v HBA v Tools
table displays as you click each tab.
A
6. In the table, click on the appropriate link in the Current version and readmes
column.
7. Click the link for the readme file.

Web sites

The most up-to-date information about DS4000 storage subsystems and DS4000 Storage Manager, including documentation and the most recent software, firmware, and NVSRAM downloads, can be found at the following Web sites.
DS4000 Midrange Disk Systems
Find the latest information about IBM System Storage disk storage systems, including all of the DS4000 storage subsystems:
www.ibm.com/servers/storage/disk/ds4000
IBM System Storage products
Find information about all IBM System Storage products:
www.storage.ibm.com
Support for IBM System Storage disk storage systems
Find links to support pages for all IBM System Storage disk storage systems, including DS4000 storage subsystems and expansion units:
www.ibm.com/systems/support/storage/disk
System Storage DS4000 interoperability matrix
Find the latest information about operating system and HBA support, clustering support, storage area network (SAN) fabric support, and DS4000 Storage Manager feature support:
www.ibm.com/servers/storage/disk/ds4000/interop-matrix.html
Storage Area Network (SAN) support
Find information about using SAN switches, including links to user guides and other documents:
www.ibm.com/systems/support/storage/san
DS4000 technical support
Find downloads, hints and tips, documentation, parts information, HBA and Fibre Channel support:
About this document xxiii
http://www.ibm.com/systems/support/storage/disk
Select your Storage Subsystem (for example, DS4800).
Premium feature activation
Activate a DS4000™ premium feature by using the online tool:
www-912.ibm.com/PremiumFeatures/jsp/keyInput.jsp
IBM publications center
Find IBM publications:
www.ibm.com/shop/publications/order/
Support for System p™ servers
Find the latest information supporting System p AIX® and Linux® servers:
www.ibm.com/systems/support/supportsite.wss/
brandmain?brandind=5000025
Support for System x™ servers
Find the latest information supporting System x Intel
servers:
www.ibm.com/systems/support/supportsite.wss/
brandmain?brandind=5000008
Fix delivery center for AIX and Linux on POWER
Find the latest AIX and Linux on POWER information and downloads:
www-912.ibm.com/eserver/support/fixes/fcgui.jsp
In the Product family drop-down menu, select UNIX® servers. Then select
your product and fix type from the subsequent drop-down menus.
Eserver System p and AIX information center
Find everything you need to know about using AIX with System p and
POWER servers:
publib.boulder.ibm.com/infocenter/pseries/index.jsp?

Software service and support

Through IBM Support Line, for a fee you can get telephone assistance with usage, configuration, and software problems. For information about which products are supported by Support Line in your country or region, go to the following Web site:
www.ibm.com/services/sl/products
®
- and AMD-based
For more information about the IBM Support Line and other IBM services, go to the following Web sites:
v www.ibm.com/services
v www.ibm.com/planetwide

Hardware service and support

You can receive hardware service through IBM Integrated Technology Services or through your IBM reseller, if your reseller is authorized by IBM to provide warranty service. Go to the following Web site for support telephone numbers:
www.ibm.com/planetwide
xxiv IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
In the U.S. and Canada, hardware service and support is available 24 hours a day, 7 days a week. In the U.K., these services are available Monday through Friday, from 9 a.m. to 6 p.m.

Fire suppression systems

A fire suppression system is the responsibility of the customer. The customer’s own insurance underwriter, local fire marshal, or a local building inspector, or both, should be consulted in selecting a fire suppression system that provides the correct level of coverage and protection. IBM designs and manufactures equipment to internal and external standards that require certain environments for reliable operation. Because IBM does not test any equipment for compatibility with fire suppression systems, IBM does not make compatibility claims of any kind nor does IBM provide recommendations on fire suppression systems.

How to send your comments

Your feedback is important to help us provide the highest quality information. If you have any comments about this document, you can submit them in one of the following ways:
E-mail
Submit your comments by e-mail to:
Mail
starpubs@us.ibm.com
Be sure to include the name and order number of the document and, if applicable, the specific location of the text that you are commenting on, such as a page number or table number.
Fill out the Readers’ Comments form (RCF) at the back of this document and return it by mail or give it to an IBM representative.
If the RCF has been removed, send your comments to:
International
Business Machines Corporation
Information Development Department GZW 9000 South Rita Road Tucson, Arizona USA 85744-0001
Be sure to include the name and order number of the document and, if
applicable, the specific location of the text that you are commenting on, such as a page number or table number.
About this document xxv
xxvi IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide

Chapter 1. Introduction

This chapter describes the operating specifications, features, and components for the IBM System Storage EXP420 storage expansion enclosure (hereafter referred to as EXP420 or storage expansion enclosure). This chapter also includes a list of hardware that comes with the storage expansion enclosure.

Overview

The IBM System Storage DS4000 EXP420 storage expansion enclosure (Machine Type 1812, Models 8VA and 8VH) provides high-capacity, SATA disk interface storage. The storage expansion enclosure delivers fast, high-volume data transfer, retrieval, and storage functions for multiple drives to multiple hosts. The storage expansion enclosure provides continuous, reliable service, using hot-swap technology for easy replacement without shutting down the system and support redundant, dual-loop configurations. External cables and Small Form-Factor Pluggable (SFP) modules connect the DS4000 storage subsystem to the storage expansion enclosure.
The EXP420 is a rack-mountable enclosure that supports up to 16 500 GB S ATA Enhanced Value Disk Drive Modules (EV-DDMs), offering up to 8.0 TB when using 500 GB SATA EV-DDMs. Coupled with a DS4000 storage subsystem such as the DS4200 Express, you can configure RAID-protected storage solutions of up to 56 TB, providing economical and scalable storage for your rapidly growing application needs for limited access, data reference storage capacity.
The EXP420 uses the latest fibre channel and SATA technology and is designed with redundant 2 or 4 Gbps fibre channel connections to the DS4000 storage subsystem offering reliability and performance.
In this initial release, you cannot intermix the EXP420 with any other storage
Note:
expansion enclosures in the same DS4000 storage subsystem configuration. In addition, the only DS4000 storage subsystems that the EXP420 can be attached to are the DS4200 Express models 7VA and 7VH. Contact your IBM representative or reseller for information about intermix functionality or additional DS4000 storage subsystem support in the future.
The EXP420 supports redundant, dual drive channel/loop configurations with the DS4200 Express Storage Subsystems and other EXP420 storage expansion enclosures. External cables and small form-factor pluggable (SFP) modules connect the controller to the storage expansion enclosure. EXP420s attached to supported DS4000 storage subsystems can be used with storage area networks to satisfy the needs of various fixed content, data reference applications that require large amounts of storage capacity but do not have the high utilization and access characteristics satisfied by Fibre Channel disk drive storage. The storage expansion enclosure provides continuous, reliable service and uses hot-swap technology for easy replacement without the need to shut down the system. Up to six EXP420 enclosures can be connected together in a redundant fibre-channel drive channel/loop pair in a DS4200 Express Storage Subsystem configuration, providing connections to a maximum of 112 hard drives.
© Copyright IBM Corp. 2006, 2007 1

Operating system support

For supported operating systems, see the latest DS4000 Storage Manager host software readme file and the IBM DS4000 series products interoperability matrix at the following Web site for additional host operating system support:
www-1.ibm.com/servers/storage/disk/ds4000/interop-matrix.html
See “Finding DS4000 readme files” on page xxiii to learn how to access the DS4000 readme files on the Web.

Fibre channel defined

Fibre channel technology is outlined in the SCSI-3 Fibre Channel Protocol (SCSI-FCP) standard. Fibre channel is a high-speed data transport technology that is used for mass storage and networking.
Using a fibre-channel arbitrated loop (FC-AL), more than 100 fibre-channel devices can be supported, compared to 15 small computer system interface (SCSI) devices. The connection from the DS4000 storage subsystem to the EXP420 is either a 2 Gbps or 4 Gbps fibre-channel interface speed that supports data transfer rates up to 400 MBps half-duplex and 800 MBps full-duplex on optical interfaces.

SATA defined

The Serial Advanced Technology Attachment (SATA) interface offers increased data rate performance over Parallel Advanced Technology Attachment (ATA), while maintaining the benefits of ATA. S ATA is designed to overcome the performance barriers that have been forecasted for current parallel technologies while maintaining the cost-efficiency of Parallel ATA . SATA specifications allow for thinner, more flexible cables, and lower pin counts. It also enables easier, more flexible cable routing management and the use of smaller connectors than is possible with the existing Parallel ATA technology.
The Serial ATA Working Group introduced the first SATA specification, Serial ATA
1.0, in 2001. The Web site is:
www.serialata.org.

Inventory checklist

The EXP420 ships with the following items:
v Power cables (2 jumper line cords)
v Rack-mounting hardware kit (1), including:
Rails (2) (right and left assembly)
M5 black hex-head slotted screws (12)
Washers (8)
Blank trays (16) (Your storage expansion enclosure might come with up to 16
v
SATA EV-DDMs.)
v 4 Gbps SFPs (2) (already installed in the EXP420 ESM ports)
v Box ID labels (used to label the enclosure IDs on the front of the EXP420)
A rack mounting template and instructions for installing the support rails and the storage expansion enclosure are provided in “Installing the support rails” on page
23.
2 IBM System Storage DS4000 EXP420 Storage Expansion Enclosure: Installation, User’s and Maintenance Guide
Loading...
+ 160 hidden pages