IBM X3850, X3950 User Manual

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

Product Guide

September 2008

IBM System x3850 M2 / x3950 M2

Architecture Overview

CONTENTS

Architecture Overview

Key Highlights

Key Features

Key Options

x3850/x3950 M2 Images

x3850/x3950 M2 Specs

The Bottom Line

For More Information

Legal Information

Outstanding performance, superior mainframe-like reliability, and fault-tolerant memory characteristics

IBM® has been designing and implementing chipsets under the X-Architecture™ name since

2001. eX4 technology represents the fourth generation of products based on the same design principle IBM began with in 1997: to offer systems that are expandable, offer “big iron” reliability, availability, and serviceability (RAS) features, with extremely competitive price/performance on an Intel® Xeon® processor-based system.

The eX4 technology is primarily designed around three major workloads. These workloads are database servers, server consolidation using virtualization services, and Enterprise Resource Planning (application and database) servers.

If you are adopting industry-standard Xeon EM64T servers for running business critical applications, the systems that run these applications need the type of technology designed into IBM’s eX4 technology systems. The eX4 chipset represents a $100M+ investment in designing a flagship offering that can harness the power of 4-socket-and-up 64-bit x86 (x64) Xeon processors. The eX4 design includes two 4U 4-socket rack-optimized chassis (x3850 M2 and x3950 M2) with the ability to scale up to 16 sockets and up to 96 cores. In fact, you can start with a single-node x3850 M2 chassis, and by adding a ScaleXpander Option Kit, you can turn it into an x3950 M2, ready to expand into a multi-node configuration providing more flexibility in deploying your scalable solution than ever before.

The x3850 M2 now offers selected models integrated with VMware ESXi 3.5 preloaded on an IBM 4GB USB 2.0 Flash Key. It operates in a diskless configuration, offers a smaller memory footprint, extremely high performance, and stronger security, making getting a system up and running in a virtualized environment faster and easier than ever before.

IBM X-Architecture pioneered XpandOnDemand™ (“pay as you grow”) scalability, which allows chassis to be simply cabled together to form larger scale-up systems. This unique capability allows IBM to sell a large SMP (symmetric multiprocessing) system at entry price points. Other Intel OEMs can’t match this capability. If they offer a 16-socket product at all, they require you to start out with a 16-socket chassis, whether you’ll ever grow into it or not. With XpandOnDemand, you can start small, with a 4- or 8-socket configuration, and later expand as your needs change, without requiring you to buy more than you need up-front or throw away parts later as you expand.

IBM’s eX4 technology-based systems are the ideal solution for scale-up database-serving applications on Microsoft® Windows® with Microsoft SQL Server® or IBM DB2®, as well as Linux® with Oracle 9i RAC and 10g, or IBM DB2. Database hosting demands ultimate server reliability features, and once installed, they grow and grow. The eX4 design offers two strong selling arguments against the competition: lower initial purchase price, and potentially much lower longterm TCO.

Another strong application area for the eX4-based systems is enterprise server consolidation activities. Larger servers have more processor, memory and I/O resources, which make maximum use of any applicable virtual machine software licensing fees and deliver superior system utilization levels. The name of the game in consolidation activities is to deploy the fewest new servers possible and help IT staff manage more images with the same or fewer overall people.

One of the fastest growing application areas for eX4-based systems is ERP workloads, including SAP and Oracle. eX4-based systems can offer considerable savings over UNIX deployments, using our certified solution stacks on either Windows or Linux.

The 4-socket eX4 servers are designed to protect your data with high performance, high reliability, and high availability. They support the latest six-, quad- and dual-core Intel® Xeon™ processors, designed with high-performance quad 1066MHz front-side buses (FSB), 64-bit

extensions (EM64T), and either 8MB (dual-core); 4MB, 6MB, or 8MB (quad-core); or 9MB of L2 cache; 8MB, 12MB or 16MB (six-core) of L3 cache, and up to 256MB of L4 cache per

chassis (all models), to help provide you with the computing power you need to match your

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

business needs and growth. In addition, the x3850 M2 and x3950 M2 use industry-standard DDR II memory with Chipkill™ ECC (Error Checking and Correcting) protection—for high performance and reliability and lower energy consumption than fully buffered memory. For even higher levels of availability, the eX4 servers also offer Memory ProteXion™, memory

scrubbing, and selectable memory mirroring. A dual-port integrated high-speed Gigabit Ethernet controller with TOE (TCP Offload Engine) is standard, as are seven high-performance PCIe x8 adapter slots (two of them hot-plug).

The x3850 M2 and x3950 M2 offer industry-leading scalability, including quad-processor support (upgradeable to 16 processors/96 cores), up to 256GB of memory per chassis (upgradeable to 1TB overall), 256MB of L4 cache per chassis (upgradeable to 1GB), and up to four 2.5-inch internal high-performance Serial-Attach SCSI (SAS) hot-swap hard disk drives with an internal storage capacity of 587.2GB per chassis (up to 16 drives and 2.3TB overall). Hardware-based RAID-0/1 support is standard. An optional IBM ServeRAID-MR10k SAS RAID controller adds 256MB of battery-backed cache to the onboard controller to provide five additional RAID levels, including RAID-10/5/50/6/60. The 4U size of the chassis helps you maximize your rack investments. Up to 10 of these chassis can be installed in a single 42U rack, for a total of up to 40 processors, 70 PCIe slots (20 of them hot-plug), and 40 HDDs, offering an ideal balance of performance, storage and I/O slots per rack. Optional Advanced Connectivity Technology (ACT) interconnect cabling helps reduce cable clutter and cost and minimizes installation time when interconnecting many rack-mounted servers.

Standard in both the x3850 M2 and the x3950 M2 is the Remote Supervisor Adapter II that communicates with the Baseboard Management Controller (BMC) to enable the user to manage and control the server easily—both locally and remotely. This high level of manageability is designed to help keep management costs down and the system up. The dropdown light path diagnostics panel enables quick servicing of the system if a problem develops. These advanced features help maximize network availability by increasing uptime, as do hot-

swap/redundant HDDs, power and fans; Active Memory™; temperature-controlled fans with Calibrated Vectored Cooling™; IPMI 2.0 support, including highly secure remote power control and Serial over LAN; as well as text- and graphics-console redirect over LAN.

With the inclusion of unique IBM service and support features such as IBM Systems Director, IBM Systems Director Active Energy Manager™ for x86 (formerly known as

PowerExecutive), and IBM ServerGuide™, the x3850/x950 M2 is designed for maximum uptime.

If you need a balance of high-performance four-socket processing, scalability beyond eight sockets, and large I/O capacity in a rack-dense environment, these are the ideal systems.

Key Highlights

eX4 Design Features

eX4 design offers numerous features per chassis to boost performance and reduce product and operating costs:

• Support for up to four Intel Xeon processors with four 1066MHz front side buses and 4MBto-16MB of integrated Level 2 cache (model dependant), 8MB, 12MB, or 16MB of external L3 cache (model dependant), Intel VT technology, and Intel EM64T technology

• Supports Xeon six-core, quad-core or dual-core processors (scalable to 16 processors and 32-to-96 cores, using the ScaleXpander Option Kit)

• A built-in ‘snoop’ filter to lower the amount of inter-processor communication needed in an SMP system

• Fast PC2-5300 DDR II ECC memory with Chipkill error correction, Memory ProteXion™ (redundant bit steering), memory scrubbing, selectable memory mirroring, and hot- add/hot-swap memory protection provides speed and high availability

• Seven 64-bit high-speed (4Gbps) PCIe x8 adapter slots offer investment protection by supporting high-performance adapters, such as 10Gb Ethernet, Fibre Channel and InfiniBand cards; the two Active™ PCIe hot-plug slots help increase system availability

• Up to 587.2GB of internal SAS storage using four 2.5-inch HDDs per chassis (scalable to

2.3TB using the ScaleXpander Option Kit)

• Baseboard Management Controller (BMC) and Remote Supervisor Adapter II standard

• Support for an optional ServeRAID-MR10k RAID controller that supports RAID-

0/1/10/5/50/6, and 60

IBM System x3850 M2-specific

• A 4U rack-optimized model (expandable to 16U using a ScaleXpander Option Kit and three additional chassis)

• Installing the ScaleXpander Option Kit turns the x3850 M2 into an x3950 M2 (including a new bezel, cable management arm, and scalability “key”)

• 4GB USB 2.0 Flash Key with VMware ESXi preinstalled (in selected models)

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

IBM System x3950 M2-specific

• ScaleXpander Option Kit standard

• Offers physical partitioning at four-socket granularity allowing you to run multiple

operating systems on the same integrated hardware

• Support for up to two ServeRAID-MR10k options if a multi-chassis configuration is used

Key Features

High-Performance Xeon Processors

The x3850 M2 and x3950 M2 ship with two high-performance Intel Xeon processors, and support up to four, allowing you to upgrade as business needs require. The eX4 servers also go a step further by allowing you to add additional nodes (chassis) to increase the number of sockets from 4 to as many as 16 (using four total chassis). (The ScaleXpander Option Kit converts a 4-socket x3850 M2 into an ultra-expandable x3950 M2.) With multiple chassis, you can use physical partitioning to segment the one physical server into multiple logical servers for virtualization.

Processors can be added or replaced through the top cover of the system. The system offers a choice of processor clock rates, FSB speeds and power draw:

• 130W six-core Xeon processor model X7460 at 2.67GHz, with 64-bit extensions, low power draw per core (21.67W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB; 1.5MB

per core), and 16MB of shared L3 cache

• 130W quad-core Xeon processor model X7350 at 2.93GHz, with 64-bit extensions, 32.5W per core of power draw, quad 1066MHz FSBs, and 8MB of L2 cache (2 x 4MB; 2MB per core)

• 90W six-core Xeon processor model E7450 at 2.4GHz, with 64-bit extensions, extremely low power draw per core (15W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB;

1.5MB per core), and 12MB of shared L3 cache

• 90W quad-core Xeon processor model E7420 at 2.13GHz, with 64-bit extensions, low power draw per core (22.5W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB; 1.5MB

per core), and 8MB of shared L3 cache

• 80W quad-core Xeon processor model E7330 at 2.4GHz, with 64-bit extensions, low power draw per core (20W), quad 1066MHz FSBs, and 6MB of L2 cache (2 x 3MB; 1.5MB per core)

• 80W quad-core Xeon processor models E7320 at 2.13GHz, with 64-bit extensions, low power draw per core (20W), quad 1066MHz FSBs, and 4MB of L2 cache (2 x 2MB; 1MB per core)

• 80W dual-core Xeon processor model E7210 at 2.4GHz, with 64-bit extensions, 40W per core of power draw, quad 1066MHz FSBs, and 8MB of L2 cache (2 x 4MB; 4MB per core)

• 50W quad-core Xeon processor model L7445 at 2.13GHz, with 64-bit extensions, extremely

low power draw per core (12.5W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB; 1.5MB per core), and 12MB of L3 cache

Also available via special bid/configure-to-order:

• 90W quad-core Xeon processor model E7440 at 2.4GHz, with 64-bit extensions, extremely

low power

draw per core (15W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x

3MB; 1.5MB per core), and 16MB of shared L3 cache

• 90W quad-core Xeon processor model E7430 at 2.13GHz, with 64-bit extensions, extremely low power draw per core (12.5W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB; 1.5MB per core), and 12MB of shared L3 cache

• 65W six-core Xeon processor model L7455 at 2.13GHz, with 64-bit extensions, extremely low power draw per core (10.83W), quad 1066MHz FSBs, 9MB of L2 processor cache (3 x 3MB; 1.5MB per core), and 12MB of shared L3 cache

The dual-core Xeon processors contain two complete processor cores. Each processor contains two 4MB L2 caches, 4MB per core (8MB total per processor). The two cores appear to software as separate physical processors. The dual-core processors can offer more than double the performance of a same-speed single-core Xeon processor (depending on workload). Similarly, quad-core Xeon processors contain four processor cores. The cores share dual 2MB, 3MB, or 4MB L2 caches (4MB, 6MB, or 8MB total per processor). That is, two cores per L2 cache. They can offer more than double the performance of dual-core Xeon processors (again, depending on workload). The six-core processors contain six processor cores. The cores share dual 4MB, 6MB, or 8MB L2 caches (8MB, 12MB, or 16MB total per processor).

The servers provide four separate 1066MHz FSBs (front-side buses), one per processor. Each of the four FSBs (which connect memory and I/O to the processor) boasts a peak rate of

8.53GBps (34.1GBps aggregate).This contrasts to the two 667MHz FSBs (with up to two

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

twice as many FSBs as before and each is 60% faster than before.

Intel Extended Memory 64 Technology (EM64T) 64-bit extensions allow the Xeon processor to use large memory addressing when running with a 64-bit operating system. This in turn lets individual software processes directly access more than 4GB of RAM, which was the limit of 32bit addressing. This can result in much higher performance for certain kinds of programs, such as database management and CAD. Additional registers and instructions (SSE3) can further boost performance for applications written to use them. Customers should contact their software provider to determine their software support for EM64T.

Intel’s Virtualization Technology (VT) integrates hardware-level virtualization hooks that allow operating system vendors to better utilize the hardware for virtualization workloads, a key workload for the eX4 platform.

eX4 Chipset ‘Snoop’ Filtering

One of the core features of the eX4 chipset that gives the platform a performance advantage is the snoop filter. The Xeon Coherency Protocol or “snoop” is an operation that occurs whenever a processor in an SMP system needs to update a memory address during normal operation. The snoop occurs when the processor getting ready to operate on a piece of data asks the other processors in the SMP complex to verify they have not modified the same piece of data without writing back from their cache. This operation increases traffic on the front side bus.

The eX4 chipset contains 324MB of EDRAM within the Northbridge chip. This copies all data as it is written to the processor cache, allowing the chipset to respond directly to the snoop requests. This reduces the overall traffic across the FSB and helps to improve system performance over other architectures. (Intel now offers a first-generation snoop filter in their 7300 chipset; however it contains only 64MB of RAM which is 80% smaller than the eX4 fourth- generation solution.)

Advanced Buffer eXecution

The IBM Advanced Buffer eXecution (ABX) chips (2 per chassis) provide the x3850 M2 / 3950 M2’s DDR2 memory with up to 60% more bandwidth than other vendors can manage using more expensive (and more energy-hungry) Fully Buffered DIMMs (FB-DIMMs). The ABX chip uses two buffers per memory card to re-drive the signals from the eX4 memory controller to the DIMMs, bypassing the latency-adding buffers used on each FB-DIMM. The use of ABX reduces latency by 20% vs. FB-DIMMs.

This feature is a unique IBM enhancement, not offered by other x86 server architectures (using either Intel or AMD processors).

XceL4v Dynamic Server Cache

Another performance feature of the eX4 chipset is the XceL4v L4 cache. When using a single node (chassis), the cache works with the snoop filter to help reduce FSB traffic. When more than one node is used, 256MB of virtual cache per node (taken from main memory) is used for interprocessor communications between chassis, to keep data in synch. In a 4chassis configuration, this amounts to as much as 1GB of L4 cache. This not only compensates for any performance hit that might otherwise result from sending data across the distances between processors in multiple chassis, it actually results in a performance improvement versus a single chassis. (IBM X3 and eX4 servers have achieved well over 100 #1 results on industry-standard benchmarks, such as TPC-C, TPC-E, TPC-H, SAP SD, vConsolidate, Vmark, and more.)

This feature is a unique IBM enhancement, not offered by other x86 server architectures (using either Intel or AMD processors).

Scalable Virtual L 4

cache

IB: 2 x12

533 MT/s

1066 MT/s

2.5 GT/s

SP: 3 x8

2.1 GT/s

2.1

GT/s

South

Bridge

I/F

Memory

DIMMS

Flash

I/F

eX4 chipset

NNNNNNN

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

High-Speed Scalability Ports

When more than one chassis is used, they are simply cabled together via one or more scalability cables. Up to three cables connected to low-latency, 10.24Gbps full-duplex ports (5.12Gbps in each direction), supported by up to 1GB of L4 cache, provide exceptional interchassis throughput (30.72Gbps aggregate). Because of this design, an x3950 M2 server has the incredible flexibility of starting out as a 4-socket server with up to 256GB of memory, 256MB of L4 cache, seven PCIe adapter slots, and four HDD bays, and then doubling, tripling, or even quadrupling those numbers while maintaining first-in-class performance.

This feature is another unique IBM enhancement, not offered by other x86 server architectures.

DDR II ECC Memory with Active Memory Protection

The eX4 servers support up to 256GB of memory per node (chassis) in 32 DIMM sockets (up to 1TB for a 4-node/16-processor configuration). It uses PC5300 double data rate II (DDR II)

memory (operating at 533MHz) for faster access, and provides IBM Active Memory features, including advanced IBM Chipkill memory protection, for up to 16X better error correction than standard ECC memory, IBM Memory ProteXion, hot-add/hot-swap memory support, and optional memory mirroring. (Notes: Hot-add capability requires operating system support. Hotadd memory and mirroring are mutually exclusive. Conversely, hot-swap requires mirroring.)

The standard configuration includes two memory cards, which support up to eight DIMMs apiece. The system is upgradeable to four memory cards with eight DIMMs each. With configurations of 8-to-16 DIMMs, using two memory cards saves cost, but using four cards increases performance. (With more than 16 DIMMs, four memory cards are required.)

The memory architecture of the eX4 chipset provides up to 60% more aggregate memory bandwidth (up to 34.1GBps when using four memory cards vs. a maximum of 21.3GBps bandwidth in the previous generation), for exceptional memory performance, and quadruple the system memory capacity of the predecessor x3850 server. By performing reads and writes simultaneously, it eliminates the previous memory read-to-write blocking latency. In addition, it also offers innovative data reliability and security features to help improve data integrity, including enhanced CRC protection, data retry on error detect and buffer registers for improved fault isolation. Even running at 533Mhz, with twice the number of channels and twice the DIMM capacity supported, the x3850 M2 and x3950 M2 deliver a memory subsystem with greater throughput than the competition. In addition, the DDR2 DIMMs use up to 37% less energy than the competition’s FB-DIMMs.

Memory scrubbing is an automatic daily test of all system memory. It detects and reports memory errors that might be developing before they cause a server outage. Memory scrubbing and Memory ProteXion work together. When a bit error is detected, memory scrubbing determines whether the error is recoverable. If the error is recoverable Memory ProteXion technology will write the data to new location; if it is not recoverable, scrubbing sends an alert to light path diagnostics, which then notifies IBM Systems Director.

Memory ProteXion technology (also called redundant bit steering) provides multichip error

protection and works in conjunction with Chipkill technology—which provides multibit protection per chip—and standard ECC protection, to provide three-level memory correction. For increased availability, the eX4 design offers an additional level of Active Memory protection: memory mirroring.

Memory mirroring works much like disk mirroring. The total memory is divided into two channels. Data is written concurrently to both channels. If a DIMM fails in one of the DIMMs in the primary channel, it is instantly disabled and the mirrored (backup) memory in the other channel becomes active (primary) until the failing DIMM is replaced. Note: Mirroring requires DIMMs to be installed in multiples of four (one pair per memory channel).

A secondary benefit of memory mirroring is that the failed DIMM and its memory card can be hot-swapped out to replace the failed DIMM. This is a hardware feature that is operating system independent.

DIMMs must be installed in matching pairs. Memory is available in kits consisting of two 1GB,

2GB, 4GB or 8GB DIMMs.

Flexible Internal Storage Capacity

The eX4 servers offer a choice of internal storage, supporting up to four (2.5-inch) hot-swap high-performance Serial-Attach SCSI (SAS) drives per chassis. In addition, selected models of the x3850 M2 include an integrated 4GB USB 2.0 Flash Key with VMware preloaded as an embedded virtualization hypervisor.

Combining high-performance architecture with high-performance Xeon processors

Please see the Legal Information section for important notices and information.

2.5-inch SAS

• 10,000 RPMs — 73.4 or 146.8GB (587.2GB maximum per chassis; 2.349TB in a 4-chassis configuration)

• 15,000 RPMs — 73.4GB (293.6GB per chassis; 1.174TB per 4 chassis)

2-5-inch drives not only require less space than 3.5-inch drives, they weigh less, consume half the power, produce less noise, seek faster, and offer increased reliability.

The hot-swap SAS drives use the Converged Tray for interchangeability with other IBM System x™ systems. If you need more storage space, terabyte capacities are possible with external direct-attach NAS and SAN solutions.

Integrated Virtualization

Selected models of the x3850 M2 ship with an IBM 4GB USB 2.0 Flash Key installed preloaded with VMware ESXi 3.5. This is an embedded version of VMware ESX 3.5, fully contained on the flash drive and requiring no disk space—not an “ESX Lite.” Rather than management through a Service Console based on a Linux operating system, ESXi 3.5 relies on aggregate management tools, including VirtualCenter, the Remote Command Line interface and the introduction of CIM for standards-based and agentless hardware monitoring.

VMware ESXi 3.5 includes all the performance, scalability and compatibility features of ESX 3.5, including full VMFS support across FC SAN, iSCSI SAN, and NAS, and 4-way VSMP. Because it runs from flash memory, it’s extremely fast and ideal for diskless configurations. It also offers enhanced security, because it runs without an operating system-based console and is updated/patched much like firmware. Licensing works the same as for “standard” ESX 3.5.

If you prefer Microsoft Hyper-V™ to VMware, the intent is for the x3850 M2 and x3950 M2 to fully support Microsoft’s hypervisor software installed on the system HDD at a future date.

Disk/Tape Controllers

All models include an integrated eight-port LSI 1078 Serial-Attach SCSI (SAS) controller. This controller supports up to four internal SAS LVD (low-voltage differential) drives.

The integrated SAS controller offers hardware RAID-0/1 support for those SAS drives. The ServeRAID-MR10k option adds additional RAID levels, including RAID-10, 5, 50, 6 and 60, along with 256MB of battery-backed cache memory for higher performance and data integrity, without consuming a valuable PCIe adapter slot.

The SAS controller provides data transfer speeds of up to 300MB per second1 in each direction (full-duplex) across the SAS bus, for an aggregate speed of 600MBps, nearly double that of Ultra320 SCSI’s 320MBps (half-duplex) bandwidth. The serial design of the SAS bus allows maximum performance to be maintained as additional drives are added.

Additional external SAS/SATA disk storage, as well as tape backup, is available using the external SAS port on the system unit, or via one of several supported iSCSI or FC SAN controllers.

Drive Bays

The eX4 servers contain five drive bays (per chassis) in all—including four 2.5-inch HDD bays that support hot-swap SAS drives totaling up to 587.6GB.

A 24X/10X/24X/8X2 speed (ultraslim, 0.5”) CD-RW/DVD-ROM combo drive with an IDE interface ships standard in all x3850 M2 and x3950 M2 servers. No floppy drive is supplied with any model; an external USB floppy drive may be used, if needed.

For still more storage, a direct-attach, iSCSI, or FC SAN external expansion option can be added, using an optional controller.

High-Performance Adapter Slots

There are seven physical x8 (“by 8”) PCIe (PCI Express) adapter slots standard (per chassis). Each is capable of supporting x4 or x8 adapters. All slots are half-length/full-height.

Electrically they are x8 slots as well (meaning that they operate at full x8 4GBps speeds). Slots 6 and 7 are Active PCIe hot-add/hot-swap slots, offering the capability to add or replace

adapters without having to shut down the system.

PCI Express is a high-performance, low-latency, next-generation serial I/O bus that is rapidly replacing the older parallel PCI and PCI-X buses. A x8 PCIe adapter offers approximately four

Data transfer rates depend on many factors and are often less than the maximum possible.

+ 13 hidden pages