1
Cisco UCS S3260 M5 Server with
Cloudian HyperStore Object Storage
Design Guide
Last Updated: January 24, 2019
2
About the Cisco Validated Design Program
The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented
to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:
http://www.cisco.com/go/designzone.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY,
"DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS
DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING,
USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT,
SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS
OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF
CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR
APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER
PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN
TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON
FACTORS NOT TESTED BY CISCO.
CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx,
the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live,
Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting
To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork
Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco
Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers,
Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified
Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime
Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without
Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive,
HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, LightStream, Linksys, MediaTone, MeetingPlace,
MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX,
PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way
to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco
Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use
of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)
© 2019 Cisco Systems, Inc. All rights reserved.
3
Table of Contents
Executive Summary .............................................................................................................................................................................. 5
Solution Overview ................................................................................................................................................................................. 6
Introduction....................................................................................................................................................................................... 6
Audience .......................................................................................................................................................................................... 6
Purpose of this Document ................................................................................................................................................................ 6
Solution Summary ............................................................................................................................................................................ 6
Technology Overview ........................................................................................................................................................................... 8
Cisco Unified Computing System ..................................................................................................................................................... 8
Cisco UCS Manager .................................................................................................................................................................... 8
Cisco UCS 6300 Fabric Interconnects ........................................................................................................................................ 9
Cisco UCS 9332 Nexus Switches ............................................................................................................................................. 10
Cisco UCS S3260 M5 Storage Server ...................................................................................................................................... 11
Cisco UCS C220 M5 Rack-Mount Server ................................................................................................................................. 12
Cisco UCS Virtual Interface Card 1387 ..................................................................................................................................... 13
Red Hat Enterprise Linux 7.5 .......................................................................................................................................................... 13
Cloudian HyperStore ........................................................................................................................................................................... 15
Cloudian Object Storage ................................................................................................................................................................ 15
Cloudian HyperStore Design .......................................................................................................................................................... 16
Cloudian HyperStore Architecture .................................................................................................................................................. 16
Cloudian Management Console ..................................................................................................................................................... 17
S3 Compatible ................................................................................................................................................................................ 18
Integrated Billing, Management, and Monitoring ............................................................................................................................ 18
Cloudian Features .......................................................................................................................................................................... 19
Cisco and Cloudian ............................................................................................................................................................................. 20
Infinitely Scalable on Demand ........................................................................................................................................................ 20
Security .......................................................................................................................................................................................... 20
Data Protection ............................................................................................................................................................................... 20
Effortless Data Movement .............................................................................................................................................................. 20
Solution Design ................................................................................................................................................................................... 21
Deployment Architecture ................................................................................................................................................................ 21
System Hardware and Software Specifications ............................................................................................................................. 22
Software Versions ...................................................................................................................................................................... 22
Hardware Requirements and Bill of Materials ............................................................................................................................. 22
Physical Topology and Configuration ............................................................................................................................................. 23
Network Topology .......................................................................................................................................................................... 24
High Availability ............................................................................................................................................................................... 24
4
Design Criteria .................................................................................................................................................................................... 26
Requirements ................................................................................................................................................................................. 26
Physical Infrastructure Considerations ............................................................................................................................................ 27
Single Node versus Dual Node Cisco UCS S3260 .................................................................................................................... 27
Replication versus Erasure Coding ................................................................................................................................................. 27
Replication .................................................................................................................................................................................. 27
Erasure Coding .......................................................................................................................................................................... 27
Supported Erasure Coding Configurations ................................................................................................................................. 28
Flash Storage ............................................................................................................................................................................. 29
JBOD versus RAID0 Disks .......................................................................................................................................................... 30
Memory Sizing ........................................................................................................................................................................... 30
Network Considerations ............................................................................................................................................................. 30
Uplinks ....................................................................................................................................................................................... 30
Multi-Site Deployments.............................................................................................................................................................. 30
Expansion of the Cluster ................................................................................................................................................................ 31
Deployment Guide .............................................................................................................................................................................. 33
Summary ............................................................................................................................................................................................ 34
About the Authors ............................................................................................................................................................................... 35
Acknowledgements ........................................................................................................................................................................ 35
5
Executive Summary
Cisco Validated Designs consist of systems and solutions that are designed, tested, and documented to facilitate
and improve customer deployments. These designs incorporate a wide range of technologies and products into a
portfolio of solutions that have been developed to address the business needs of our customers.
The purpose of this document is to describe the design of Cloudian HyperStore on Red Hat Enterprise Linux on
the latest generation of Cisco UCS S3260 M5 Servers. This validated design provides the framework of designing
and deploying Cloudian SDS software on Cisco UCS S3260 storage servers. The Cisco Unified Computing
System provides the compute, network, and storage access components for the Cloudian HyperStore, deployed
as a single cohesive system.
This design guide describes how the Cisco Unified Computing System (Cisco UCS) can be used in conjunction
with Cloudian 7.1.2. With the continuous evolution of data, there has been lack of enterprise manageability for
large capacity storage servers, Cisco UCS with Cloudian provide the solution to solve the challenges, as it
increased demand to have Cloudian HyperStore validated on Cisco UCS servers. The Cisco UCS S3260 Storage
Server, originally designed for the data center, together with Cloudian HyperStore is optimized for object storage
solutions, making it an excellent fit for unstructured data workloads such as backup, archive, file celebration,
storage as a service, IOT, video surveillance and cloud storage. The S3260 delivers a complete infrastructure with
exceptional scalability for computing and storage resources together with 40 Gigabit Ethernet networking.
Cisco and Cloudian are collaborating to offer customers a scalable object storage solution for unstructured data
that integrates UCS with Cloudian HyperStore. With the power of the Cisco UCS management framework, the
solution is cost effective to deploy and manage and will enable the next-generation cloud deployments that drive
business agility, lower operational costs and avoid vendor lock-in.
Solution Overview
6
Solution Overview
Introduction
Object storage is a highly scalable system for organizing and storing data objects. Object storage does not use a
file system structure, instead it ingests data as objects with unique keys into a flat directory structure and the
metadata is stored with the objects instead of hierarchical journal or tree. Search and retrieval is performed using
RESTful API’s, which uses HTTP verbs such as GETs and PUTs. Most of the newly generated data, about 60 to
80 percent, is unstructured today and new approaches using x86 servers are proving to be more cost effective,
providing storage that can be expanded as easily as your data grows. Scale-out Object storage is the newest
cost effective approach for handling large amounts of data in the Petabyte and Exabyte range.
The Cloudian HyperStore is a Software-Defined Storage software that is designed to create unbounded scale-out
storage systems that accommodates Petabyte scale data from multiple applications and use-cases, including
both object and file based applications
Together with Cisco UCS, Cloudian Hyperstore can deliver a fully enterprise-ready solution that can manage different workloads and still remain flexible. The Cisco UCS S3260 Storage Server is an excellent platform to use with
the main types of Object and File workloads, such as capacity-optimized and performance-optimized workloads. It
is best suited for sequential access, as opposed to random access to unstructured data, and to whatever data size.
It is essentially designed for Applications, not direct end-users.
This document describes the architecture and design procedures of Cloudian Hyperstore software on Cisco UCS
S3260 M5 servers along with Cisco UCS C220 M5 Rack-Mount servers for load balancing.
Audience
The audience for this document includes, but is not limited to, sales engineers, field consultants, professional
services, IT managers, partner engineers, IT architects, and customers who want to take advantage of an
infrastructure that is built to deliver IT efficiency and enable IT innovation. The reader of this document is expected
to have the necessary training and background to install and configure Red Hat Enterprise Linux, Cisco Unified
Computing System, Cisco Nexus and Cisco UCS Manager, as well as a high-level understanding of Cloudian
Hyperstore Software and its components. External references are provided where applicable and it is
recommended that the reader be familiar with these documents.
Readers are also expected to be familiar with the infrastructure, network and security policies of the customer
installation.
Purpose of this Document
This document describes the steps required to design Cloudian HyperStore 7.1.2 on Cisco UCS platform. It
discusses design choices and best practices using this shared infrastructure platform.
Solution Summary
This solution is focused on Cloudian HyperStore Cluster on Red Hat Linux 7 on Cisco Unified Computing System.
The advantages of Cisco UCS and Cloudian HyperStore combine to deliver an object storage solution that is
simple to install, scalable and high performance. The configuration uses the following components for the
deployment:
Solution Overview
7
Cisco Unified Computing System
Cisco UCS 6332 Series Fabric Interconnects
Cisco UCS S3260 M5 storage servers
Cisco S3260 system IO controller with VIC 1380
Cisco C220M5 servers with VIC 1387
Cisco Nexus C9332PQ Series Switches
Cloudian HyperStore 7.1.2
Red Hat Enterprise Linux 7.5
Technology Overview
8
Technology Overview
Cisco Unified Computing System
Cisco Unified Computing System is a state-of-the-art data center platform that unites computing, network,
storage access, and virtualization into a single cohesive system.
The main components of Cisco Unified Computing System are:
Computing - The system is based on an entirely new class of computing system that incorporates rack-
mount and blade servers based on Intel Xeon Processor scalable family. The Cisco UCS servers offer the
patented Cisco Extended Memory Technology to support applications with large datasets and allow more
virtual machines per server.
Network - The system is integrated onto a low-latency, lossless, 40-Gbps unified network fabric. This
network foundation consolidates LANs, SANs, and high-performance computing networks which are
separate networks today. The unified fabric lowers costs by reducing the number of network adapters,
switches, and cables, and by decreasing the power and cooling requirements.
Virtualization - The system unleashes the full potential of virtualization by enhancing the scalability,
performance, and operational control of virtual environments. Cisco security, policy enforcement, and
diagnostic features are now extended into virtualized environments to better support changing business
and IT requirements.
Storage access - The system provides consolidated access to both SAN storage and Network Attached
Storage (NAS) over the unified fabric. By unifying the storage access, Cisco Unified Computing System can
access storage over Ethernet (NFS or iSCSI), Fibre Channel, and Fibre Channel over Ethernet (FCoE). This
provides customers with choice for storage access and investment protection. In addition, the server
administrators can pre-assign storage-access policies for system connectivity to storage resources,
simplifying storage connectivity, and management for increased productivity.
Cisco Unified Computing System is designed to deliver:
A reduced Total Cost of Ownership (TCO) and increased business agility.
Increased IT staff productivity through just-in-time provisioning and mobility support.
A cohesive, integrated system, which unifies the technology in the data center.
Industry standards supported by a partner ecosystem of industry leaders.
Cisco UCS Manager
Cisco UCS Manager (UCSM) provides a unified, embedded management of all software and hardware
components of the Cisco Unified Computing System across multiple chassis, rack servers, and thousands of
virtual machines. It supports all Cisco UCS product models, including Cisco UCS B-Series Blade Servers, Cisco
UCS C-Series Rack-Mount Servers, and Cisco UCS Mini, as well as the associated storage resources and
networks. Cisco UCS Manager is embedded on a pair of Cisco UCS 6300 or 6200 Series Fabric Interconnects
using a clustered, active-standby configuration for high availability. The manager participates in server
provisioning, device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and
statistics collection.
Technology Overview
9
An instance of Cisco UCS Manager with all Cisco UCS components managed by it forms a Cisco UCS domain,
which can include up to 160 servers. In addition to provisioning Cisco UCS resources, this infrastructure
management software provides a model-based foundation for streamlining the day-to-day processes of
updating, monitoring, and managing computing resources, local storage, storage connections, and network
connections. By enabling better automation of processes, Cisco UCS Manager allows IT organizations to achieve
greater agility and scale in their infrastructure operations while reducing complexity and risk. The manager
provides flexible role and policy-based management using service profiles and templates.
Cisco UCS Manager manages Cisco UCS systems through an intuitive HTML 5 or Java user interface and a CLI. It
can register with Cisco UCS Central Software in a multi-domain Cisco UCS environment, enabling centralized
management of distributed systems scaling to thousands of servers. Cisco UCS Manager can be integrated with
Cisco UCS Director to facilitate orchestration and to provide support for converged infrastructure and
Infrastructure as a Service (IaaS).
The Cisco UCS XML API provides comprehensive access to all Cisco UCS Manager functions. The API provides
Cisco UCS system visibility to higher-level systems management tools from independent software vendors (ISVs)
such as VMware, Microsoft, and Splunk as well as tools from BMC, CA, HP, IBM, and others. ISVs and in-house
developers can use the XML API to enhance the value of the Cisco UCS platform according to their unique
requirements. Cisco UCS PowerTool for Cisco UCS Manager and the Python Software Development Kit (SDK)
help automate and manage configurations within Cisco UCS Manager.
Cisco UCS 6300 Fabric Interconnects
The Cisco UCS 6300 Series Fabric Interconnects are a core part of Cisco UCS, providing both network
connectivity and management capabilities for the system. The Cisco UCS 6300 Series offers line-rate, lowlatency, lossless 10 and 40 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), and Fibre Channel functions.
The Cisco UCS 6300 Series provides the management and communication backbone for the Cisco UCS BSeries Blade Servers, 5100 Series Blade Server Chassis, and C-Series Rack Servers managed by Cisco UCS. All
servers attached to the fabric interconnects become part of a single, highly available management domain. In
addition, by supporting unified fabric, the Cisco UCS 6300 Series provides both LAN and SAN connectivity for all
servers within its domain.
Technology Overview
10
From a networking perspective, the Cisco UCS 6300 Series uses a cut-through architecture, supporting
deterministic, low-latency, line-rate 10 and 40 Gigabit Ethernet ports, switching capacity of 2.56 terabits per
second (Tbps), and 320 Gbps of bandwidth per chassis, independent of packet size and enabled services. The
product family supports Cisco® low-latency, lossless 10 and 40 Gigabit Ethernet unified network fabric
capabilities, which increase the reliability, efficiency, and scalability of Ethernet networks. The fabric interconnect
supports multiple traffic classes over a lossless Ethernet fabric from the server through the fabric interconnect.
Significant TCO savings can be achieved with an FCoE optimized server design in which network interface cards
(NICs), host bus adapters (HBAs), cables, and switches can be consolidated.
The Cisco UCS 6332 32-Port Fabric Interconnect is a 1-rack-unit (1RU) Gigabit Ethernet, and FCoE switch
offering up to 2.56 Tbps throughput and up to 32 ports. The switch has 32 fixed 40-Gbps Ethernet and FCoE
ports.
Both the Cisco UCS 6332UP 32-Port Fabric Interconnect and the Cisco UCS 6332 16-UP 40-Port Fabric
Interconnect have ports that can be configured for the breakout feature that supports connectivity between 40
Gigabit Ethernet ports and 10 Gigabit Ethernet ports. This feature provides backward compatibility to existing
hardware that supports 10 Gigabit Ethernet. A 40 Gigabit Ethernet port can be used as four 10 Gigabit Ethernet
ports. Using a 40 Gigabit Ethernet SFP, these ports on a Cisco UCS 6300 Series Fabric Interconnect can connect
to another fabric interconnect that has four 10 Gigabit Ethernet SFPs. The breakout feature can be configured on
ports 1 to 12 and ports 15 to 26 on the Cisco UCS 6332UP fabric interconnect. Ports 17 to 34 on the Cisco
UCS 6332 16-UP fabric interconnect support the breakout feature.
Cisco UCS 9332 Nexus Switches
The Cisco Nexus 9000 Series Switches include both modular and fixed-port switches that are designed to
overcome these challenges with a flexible, agile, low-cost, application-centric infrastructure.
The Cisco Nexus 9300 platform consists of fixed-port switches designed for top-of-rack (ToR) and middle-ofrow (MoR) deployment in data centers that support enterprise applications, service provider hosting, and cloud
computing environments. They are Layer 2 and 3 nonblocking 10 and 40 Gigabit Ethernet switches with up to
2.56 terabits per second (Tbps) of internal bandwidth.
The Cisco Nexus 9332PQ Switch is a 1-rack-unit (1RU) switch that supports 2.56 Tbps of bandwidth and over
720 million packets per second (mpps) across thirty-two 40-Gbps Enhanced QSFP+ ports
All the Cisco Nexus 9300 platform switches use dual- core 2.5-GHz x86 CPUs with 64-GB solid-state disk
(SSD) drives and 16 GB of memory for enhanced network performance.
With the Cisco Nexus 9000 Series, organizations can quickly and easily upgrade existing data centers to carry 40
Gigabit Ethernet to the aggregation layer or to the spine (in a leaf-and-spine configuration) through advanced and
cost-effective optics that enable the use of existing 10 Gigabit Ethernet fiber (a pair of multimode fiber strands).
Cisco provides two modes of operation for the Cisco Nexus 9000 Series. Organizations can use Cisco NX-OS
Software to deploy the Cisco Nexus 9000 Series in standard Cisco Nexus switch environments. Organizations
also can use a hardware infrastructure that is ready to support Cisco Application Centric Infrastructure (Cisco ACI)
to take full advantage of an automated, policy-based, systems management approach.
Technology Overview
11
Cisco UCS S3260 M5 Storage Server
The Cisco UCS S3260 Storage Server is a modular, high-density, high availability, dual-node rack server, well
suited for service providers, enterprises, and industry-specific environments. It addresses the need for dense
cost effective storage for the ever-growing data needs. Designed for a new class of cloud-scale applications, it is
simple to deploy and excellent for big data applications, software-defined storage environments, and other
unstructured data repositories, media streaming, and content distribution.
Extending the capability of the Cisco UCS C3000 portfolio, the Cisco UCS S3260 helps you achieve the highest
levels of data availability. With dual-node capability that is based on the Intel Xeon scalable processors, it features
up to 600 TB of local storage in a compact 4-rack-unit (4RU) form factor. All hard-disk drives can be
asymmetrically split between the dual-nodes and are individually hot-swappable. The drives can be built-in in an
enterprise-class Redundant Array of Independent Disks (RAID) redundancy or be in a pass-through mode.
This high-density rack server comfortably fits in a standard 32-inch depth rack, such as the Cisco R42610 RackServer.
The Cisco UCS S3260 is deployed as a standalone server in both bare-metal or virtualized environments. Its
modular architecture reduces TCO by allowing you to upgrade individual components over time and as use cases
evolve, without having to replace the entire system.
The Cisco UCS S3260 uses a modular server architecture that, using Cisco’s blade technology expertise, allows
you to upgrade the computing or network nodes in the system without the need to migrate data migration from
one system to another. It delivers the following:
Dual server nodes
Up to 44 computing cores per server node
Up to 60 drives mixing a large form factor (LFF) with up to 28 solid-state disk (SSD) drives plus 2 SSD
SATA boot drives per server node
Up to 1.5 TB of memory per server node (3 TB Total ) with 128GB DIMMs
Support for 12-Gbps serial-attached SCSI (SAS) drives
A system I/O Controller either with HBA Passthrough or RAID controller, with DUAL LSI 3316 Chip
Cisco VIC 1300 Series Embedded Chip supporting Dual-port 40Gbps
Loading...