Cisco HyperFlex Hyperconverged System

Design and Deployment of Cisco HyperFlex for Virtual

Server Infrastructures
Last Updated: July 8, 2016
About Cisco Validated Designs
The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reli-
able, and more predictable customer deployments. For more information visit

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS")

IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES,
INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT
SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES,
INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INA-
BILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION
OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO,
ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING
THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco
logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn
and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst,
CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cis-
co Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation,
EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet
Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace,
MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels,
ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your In-
ternet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its
affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of
the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2016 Cisco Systems, Inc. All rights reserved.

Table of Contents
Executive Summary ................................................................................................................................................................. 7

Solution Overview .................................................................................................................................................................... 8

Introduction ......................................................................................................................................................................... 8

Audience ............................................................................................................................................................................. 9

Purpose of this Document .................................................................................................................................................... 9

Solution Summary ................................................................................................................................................................ 9

Technology Overview ............................................................................................................................................................ 12

Cisco Unified Computing System ....................................................................................................................................... 12

Cisco UCS Fabric Interconnect .......................................................................................................................................... 13

Cisco UCS 6248UP Fabric Interconnect ......................................................................................................................... 13

Cisco HyperFlex HX-Series Nodes .................................................................................................................................... 14

Cisco HyperFlex HX220c M4S Node.............................................................................................................................. 14

Cisco HyperFlex HX240c M4SX Node ........................................................................................................................... 15

Cisco VIC 1227 MLOM Interface Card ........................................................................................................................... 16

Cisco UCS B200 M4 Node............................................................................................................................................. 17

Cisco UCS 5108 Blade Chassis ..................................................................................................................................... 18

Cisco UCS 2204XP Fabric Extender .............................................................................................................................. 19

Cisco HyperFlex Converged Data Platform Software ......................................................................................................... 20

Cisco HyperFlex HX Data Platform Administration Plug-in ............................................................................................. 20

Cisco HyperFlex HX Data Platform Controller................................................................................................................. 21

Data Operations and Distribution ................................................................................................................................... 21

Solution Design ...................................................................................................................................................................... 24

Requirements .................................................................................................................................................................... 24

Physical Components .................................................................................................................................................... 24

Cisco Solution IDs ......................................................................................................................................................... 26

Software Components ................................................................................................................................................... 27

Considerations................................................................................................................................................................... 28

Version Control .............................................................................................................................................................. 28

vCenter Server .............................................................................................................................................................. 28

Scale ............................................................................................................................................................................. 28

Capacity ........................................................................................................................................................................ 29

Physical Topology .............................................................................................................................................................. 30

Topology Overview ........................................................................................................................................................ 30

Fabric Interconnects ...................................................................................................................................................... 32

HX-Series Rack Mount Servers...................................................................................................................................... 33

B-Series Blade Servers ................................................................................................................................................. 34

Logical Topology ............................................................................................................................................................... 35

Logical Network Design ................................................................................................................................................. 35

Design Elements .................................................................................................................................................................... 38

Network Design ................................................................................................................................................................. 38

Cisco UCS Uplink Connectivity ...................................................................................................................................... 38

VLANs and Subnets ....................................................................................................................................................... 42

Jumbo Frames ............................................................................................................................................................... 43

Multicast ........................................................................................................................................................................ 43

Cisco UCS Design ............................................................................................................................................................. 43

Cisco UCS Organization................................................................................................................................................. 43

Cisco UCS LAN Policies ................................................................................................................................................ 44

Cisco UCS Server Policies ............................................................................................................................................. 57

Cisco UCS Service Profile Templates ............................................................................................................................ 67

ESXi Host Design ............................................................................................................................................................... 70

Virtual Networking Design .............................................................................................................................................. 70

VMDirectPath I/O Pass-through ..................................................................................................................................... 72

Storage Platform Controller VMs.................................................................................................................................... 72

Installation ............................................................................................................................................................................. 77

Prerequisites...................................................................................................................................................................... 77

IP Addressing ................................................................................................................................................................ 77

DHCP ............................................................................................................................................................................ 80

DNS ............................................................................................................................................................................... 81

NTP ............................................................................................................................................................................... 82

VLANs ........................................................................................................................................................................... 83

Network Uplinks ............................................................................................................................................................ 83

Usernames and Passwords............................................................................................................................................ 85

Physical Installation........................................................................................................................................................ 86

Cabling .......................................................................................................................................................................... 86

Cisco Nexus Switch Configuration ..................................................................................................................................... 89

Cisco UCS Installation ........................................................................................................................................................ 89

Cisco UCS 6248UP Fabric Interconnect A ..................................................................................................................... 89

Cisco UCS 6248UP Fabric Interconnect B ..................................................................................................................... 91

Cisco UCS Manager ...................................................................................................................................................... 92

Cisco UCS Configuration ................................................................................................................................................... 93

Cisco UCS Firmware...................................................................................................................................................... 93

NTP ............................................................................................................................................................................... 93

Uplink Ports ................................................................................................................................................................... 95

Uplink Port Channels ..................................................................................................................................................... 96

Chassis Discovery Policy ............................................................................................................................................... 97

Server Ports................................................................................................................................................................... 98

Server Discovery ........................................................................................................................................................... 99

HyperFlex Installer Deployment........................................................................................................................................ 100

Installer Connectivity.................................................................................................................................................... 100

Deploy Installer OVA .................................................................................................................................................... 101

HyperFlex Installer Web Page ...................................................................................................................................... 104

HyperFlex Cisco UCS Configuration ................................................................................................................................. 106

Verify Cisco UCS Configuration ................................................................................................................................... 115

Tagged VLANs Alternate Configuration........................................................................................................................ 117

ESXi Hypervisor Installation .............................................................................................................................................. 118

ESXi Kickstart ISO Creation .......................................................................................................................................... 119

Cisco UCS vMedia and Boot Policies ........................................................................................................................... 122

Install ESXi ................................................................................................................................................................... 126

Undo vMedia and Boot Policy Changes ....................................................................................................................... 129

Configure ESXi Management ....................................................................................................................................... 129

ESXi Post-Installation Configuration ............................................................................................................................. 134

HyperFlex Installation ....................................................................................................................................................... 141

HyperFlex Cluster Expansion ........................................................................................................................................... 150

Standard Cluster Expansion ......................................................................................................................................... 151

Hybrid Cluster Expansion ............................................................................................................................................. 154

HyperFlex Post Installation Configuration ......................................................................................................................... 159

Create HyperFlex Datastores ....................................................................................................................................... 159

Configure HyperFlex Auto Support .............................................................................................................................. 163

Enable vCenter Features .............................................................................................................................................. 164

Configure Additional VM Guest VLANs ........................................................................................................................ 166

Configure VM Networking ............................................................................................................................................ 168

Configure VMotion Networking .................................................................................................................................... 172

Configure ESXi Syslog ................................................................................................................................................. 178

Management ........................................................................................................................................................................ 180

vCenter Web Client Plugin ............................................................................................................................................... 180

Summary ..................................................................................................................................................................... 181

Monitor ........................................................................................................................................................................ 181

Manage ....................................................................................................................................................................... 183

Management Best Practices ............................................................................................................................................ 185

ReadyClones ............................................................................................................................................................... 185

Snapshots ................................................................................................................................................................... 186

Storage vMotion .......................................................................................................................................................... 189

Virtual Disk Placement ................................................................................................................................................. 189

HX Maintenance Mode................................................................................................................................................. 190

Failure Scenarios ............................................................................................................................................................. 191

Network Failures .......................................................................................................................................................... 192

Disk Failures ................................................................................................................................................................ 193

Maintenance Mode ...................................................................................................................................................... 194

Node Failure ................................................................................................................................................................ 195

Validation ............................................................................................................................................................................. 196

Post Install Checklist ........................................................................................................................................................ 196

Verify Redundancy ........................................................................................................................................................... 196

Appendix ............................................................................................................................................................................. 198

A: Cluster Capacity Calculations ...................................................................................................................................... 198

B: Example Scripts ........................................................................................................................................................... 198

Create DHCP Reservations .......................................................................................................................................... 198

Configure ESXi Hosts ................................................................................................................................................... 199

Configure ESXi Post Install ........................................................................................................................................... 201

C: Example Cisco Nexus 9372 Switch Configurations ...................................................................................................... 202

Switch A ...................................................................................................................................................................... 202

Switch B ...................................................................................................................................................................... 205

About Authors ..................................................................................................................................................................... 209

Acknowledgements ......................................................................................................................................................... 209

Resources............................................................................................................................................................................ 210

Glossary .......................................................................................................................................................................... 210

References .......................................................................................................................................................................... 214

Executive Summary
The past decade has witnessed major shifts in the data center, and the most significant one being the
widespread adoption of virtualization of servers as the primary computing platform for most businesses. The
flexibility, speed of deployment, ease of management, portability, and improved resource utilization has led
ually unless
circumstances made it impossible. While the benefits of virtualization are clear, the proliferation of virtual
environments has brought other technology stacks into the spotlight, highlighting where they do not offer the
same levels of simplicity, flexibility, and rapid deployment as the virtualized compute platforms do.
Networking and storage systems in particular have come under increasing scrutiny to be as agile as
hypervisors and virtual servers. Cisco offers powerful solutions for rapid deployment and easy management
of virtualized computing platforms, including integrated networking capabilities, with the Cisco Unified
Computing System (Cisco UCS) product line. Now with the introduction of Cisco HyperFlex, we bring similar
enhancements to the virtualized servers and Hyperconverged storage market.

Cisco HyperFlex systems have been developed using the Cisco UCS platform, which combines Cisco HX-
Series x86 servers and integrated networking technologies with Cisco UCS Fabric Interconnects, into a
single management domain, along with industry leading virtualization hypervisor software from VMware, and
new software defined storage technology. The combination creates a virtualization platform that also
provides the network connectivity for the guest virtual machine (VM) connections, and the distributed
storage to house the VMs using Cisco UCS x86 servers instead of specialized components. The unique
storage features of the newly developed log based filesystem enable rapid cloning of VMs, snapshots
without the traditional performance penalties, data deduplication and compression, without having to
purchase all-flash based storage systems. All configuration, deployment, management, and monitoring tasks
of the solution can be done with the existing tools for Cisco UCS and VMware, such as Cisco UCS Manager
and VMware vCenter. This powerful linking of advanced technology stacks into a single, simple, rapidly
deployable solution makes Cisco HyperFlex a true second generation hyperconverged platform for the
modern data center.
Solution Overview

Introduction
The Cisco HyperFlex System provides an all-purpose virtualized server platform, with hypervisor hosts,
network connectivity, and virtual server storage across a set of Cisco UCS HX-Series x86 rack-mount
servers. Legacy data center deployments relied on a disparate set of technologies, each performing a
distinct and specialized function, such as network switches connecting endpoints and transferring Ethernet
network traffic, and Fibre Channel (FC) storage arrays providing block based storage devices via a unique
storage array network (SAN). Each of these systems had unique requirements for hardware, connectivity,
management tools, operational knowledge, monitoring, and ongoing support. A legacy virtual server
environment operated in silos, within which only a single technology operated, along with their correlated
software tools and support staff. Silos were often divided between the x86 computing hardware, the
networking connectivity of those x86 servers, SAN connectivity and storage device presentation, the
hypervisors, virtual platform management, and the guest VMs themselves along with their Operating Systems
and applications. This model proves to be inflexible, difficult to navigate, and is susceptible to numerous
operational inefficiencies.

To cater for the needs of the modern and agile data center, a new model called converged architecture
gained wide acceptance. A converged architecture attempts to collapse the traditional siloed architecture by
combining various technologies into a single environment, which has been designed to operate together in
pre-defined, tested, and validated designs. A key component of the converged architecture was the
revolutionary combination of x86 rack and blade servers, along with converged Ethernet and Fibre Channel
networking offered by the Cisco UCS platform. Converged architectures leverage Cisco UCS, plus new
deployment tools, management software suites, automation processes, and orchestration tools to overcome
the difficulties deploying traditional environments, and do so in a much more rapid fashion. These new tools
place the ongoing management and operation of the system into the hands of fewer staff, with faster
deployment of workloads based on business needs, while still remaining at the forefront in providing
flexibility to adapt to changing workload needs, and offering the highest possible performance. Cisco has
proved to be incredibly successful in these areas with our partners, developing leading solutions such as
Cisco FlexPod, SmartStack, VersaStack, and vBlock architectures. Despite the advancements, since these
converged architectures incorporate legacy technology stacks, particularly in the storage subsystems, there
often remained a division of responsibility amongst multiple teams of administrators. Alongside the
tremendous advantages of converged infrastructure approach, there is also a downside wherein these
architectures use a complex combination of components, where a simpler system would suffice to serve the
required workloads.

Significant changes in the storage marketplace have given rise to the software defined storage (SDS)
system. Legacy FC storage arrays continued to utilize a specialized subset of hardware, such as Fibre
Channel Arbitrated Loop (FC-AL) based controllers and disk shelves along with optimized Application
Specific Integrated Circuits (ASIC), read/write data caching modules and cards, plus highly customized
software to operate the arrays. With the rise in the Serial Attached SCSI (SAS) bus technology and its
inherent benefits, storage array vendors began to transition their internal architectures to SAS, and with
dramatic increases in processing power in the recent x86 processor architectures, fewer or no custom
ASICs are used. With the shrink in the disk physical sizes, servers began to have the same density of storage
per rack unit (RU) as the arrays themselves, and with the proliferation of NAND based flash memory solid
state disks (SSD), they also now had access to input/output (IO) devices whose speed rivaled that of
dedicated caching devices. As servers now contained storage devices and technology to rival many
dedicated arrays in the market, the remaining major differentiator between them was the software providing
allocation, presentation and management of the storage, plus the advanced features many vendors offered.
This led to the increased adoption of software defined storage, where the x86 servers with the storage
devices ran software to effectively turn one or more of them into a storage array much the same as the
traditional arrays were. In a somewhat unexpected turn of events, some of the major storage array vendors
themselves were pioneers in this field, recognizing the shift in the market and attempting to profit from their
unique software features, versus specialized hardware as they had done in the past.

Some early uses of SDS systems simply replaced the traditional storage array in the converged architectures
as described earlier. This infrastructure approach still used a separate storage system from the virtual server
hypervisor platform, and depending on the solution provider, also still used separate network devices. If the
server that hosted the virtual servers, also provided the SDS environment in the same model of servers,
could they not simply do both things at once and collapse the two functions into one? This idea and
combination of resources is what the industry has given the moniker of a hyperconverged infrastructure.
Hyperconverged infrastructures combine the computing, memory, hypervisor, and storage devices of
servers into a single monolithic platform for virtual servers. There is no longer a separate storage system, as
the servers running the hypervisors also provide the software defined storage resources to store the virtual
servers, effectively storing the virtual machines on themselves. A hyperconverged infrastructure is far more
self-contained, simpler to use, faster to deploy, easier to consume, yet flexible and with high performance.
By combining the convergence of compute and network resources provided by Cisco UCS, along with the
new hyperconverged storage software, the Cisco HyperFlex system uniquely provides the compute
resources, network connectivity, storage, and hypervisor platform to run an entire virtual environment, all
contained in a single system.

Audience
The intended audience for this document includes, but is not limited to, sales engineers, field consultants,
professional services, IT managers, partner engineering, and customers deploying the Cisco HyperFlex
System. External references are provided wherever applicable, but readers are expected to be familiar with
VMware specific technologies, infrastructure concepts, networking connectivity, and security policies of the
customer installation.

Purpose of this Document

This document describes the steps required to deploy, configure, and manage a Cisco HyperFlex system.
The document is based on all known best practices using the software, hardware and firmware revisions
specified in the document. As such, recommendations and best practices can be amended with later
versions. This document showcases the installation and configuration of a single 8 node Cisco HyperFlex
cluster, and a 4+4 hybrid Cisco HyperFlex cluster, in a typical customer data center environment. While
readers of this document are expected to have sufficient knowledge to install and configure the products
used, configuration details that are important to the deployment of this solution are provided in this CVD.

Solution Summary
The Cisco HyperFlex system provides a fully contained virtual server platform, with compute and memory
resources, integrated networking connectivity, a distributed high performance log-based filesystem for VM
storage, and the hypervisor software for running the virtualized servers, all within a single Cisco UCS
management domain.
Figure 1 HyperFlex System Overview

The following are components of a Cisco HyperFlex system:

Cisco UCS 6248UP Fabric Interconnects

Cisco HyperFlex HX220c M4S or HX240c M4SX rack-mount servers

Cisco HX Data Platform Software

VMware vSphere ESXi Hypervisor

VMware vCenter Server (end-user supplied)

Optional components for additional compute only resources are:

Cisco UCS 5108 Chassis

Cisco UCS 2204XP Fabric Extender

Cisco UCS B200 M4 blade servers

Figure 2 Cisco HyperFlex System
Technology Overview

Cisco Unified Computing System

The Cisco Unified Computing System is a next-generation data center platform that unites compute,
network, and storage access. The platform, optimized for virtual environments, is designed using open
industry-standard technologies and aims to reduce total cost of ownership (TCO) and increase business
agility. The system integrates a low-latency, lossless 10 Gigabit Ethernet unified network fabric with
enterprise-class, x86-architecture servers. It is an integrated, scalable, multi chassis platform in which all
resources participate in a unified management domain.

The main components of Cisco Unified Computing System are:

Computing The system is based on an entirely new class of computing system that incorporates
rack-mount and blade servers based on Intel Xeon Processors.

Network The system is integrated onto a low-latency, lossless, 10-Gbps unified network fabric.
This network foundation consolidates LANs, SANs, and high-performance computing networks
which are separate networks today. The unified fabric lowers costs by reducing the number of
network adapters, switches, and cables, and by decreasing the power and cooling requirements.

Virtualization The system unleashes the full potential of virtualization by enhancing the scalability,
performance, and operational control of virtual environments. Cisco security, policy enforcement, and
diagnostic features are now extended into virtualized environments to better support changing
business and IT requirements.

Storage access The system provides consolidated access to both SAN storage and Network
Attached Storage (NAS) over the unified fabric. By unifying the storage access the Cisco Unified
Computing System can access storage over Ethernet, Fibre Channel, Fibre Channel over Ethernet
(FCoE), and iSCSI. This provides customers with choice for storage access and investment
protection. In addition, the server administrators can pre-assign storage-access policies for system
connectivity to storage resources, simplifying storage connectivity, and management for increased
productivity.

Management The system uniquely integrates all system components which enable the entire
solution to be managed as a single entity by the Cisco UCS Manager (UCSM). The Cisco UCS
Manager has an intuitive graphical user interface (GUI), a command-line interface (CLI), and a robust
application programming interface (API) to manage all system configuration and operations.

The Cisco Unified Computing System is designed to deliver:

A reduced Total Cost of Ownership and increased business agility.

Increased IT staff productivity through just-in-time provisioning and mobility support.

A cohesive, integrated system which unifies the technology in the data center. The system is
managed, serviced and tested as a whole.

Scalability through a design for hundreds of discrete servers and thousands of virtual machines and
the capability to scale I/O bandwidth to match demand.

Industry standards supported by a partner ecosystem of industry leaders.

Cisco UCS Fabric Interconnect
The Cisco UCS 6200 Series Fabric Interconnect is a core part of the Cisco Unified Computing System,
providing both network connectivity and management capabilities for the system. The Cisco UCS 6200
Series offers line-rate, low-latency, lossless 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE) and
Fibre Channel functions.

The Cisco UCS 6200 Series provides the management and communication backbone for the Cisco UCS C-
Series and HX-Series rack-mount servers, Cisco UCS B-Series Blade Servers and Cisco UCS 5100 Series
Blade Server Chassis. All servers and chassis, and therefore all blades, attached to the Cisco UCS 6200
Series Fabric Interconnects become part of a single, highly available management domain. In addition, by
supporting unified fabric, the Cisco UCS 6200 Series provides both the LAN and SAN connectivity for all
blades within its domain.

From a networking perspective, the Cisco UCS 6200 Series uses a cut-through architecture, supporting
deterministic, low-latency, line-rate 10 Gigabit Ethernet on all ports, 1Tb switching capacity, 160 Gbps
bandwidth per chassis, and independent of packet size and enabled services. The product family supports
Cisco low-latency, lossless 10 Gigabit Ethernet unified network fabric capabilities, which increase the
reliability, efficiency, and scalability of Ethernet networks. The Fabric Interconnect supports multiple traffic
classes over a lossless Ethernet fabric from a server through an interconnect. Significant TCO savings come
from an FCoE-optimized server design in which network interface cards (NICs), host bus adapters (HBAs),
cables, and switches can be consolidated.

Cisco UCS 6248UP Fabric Interconnect

The Cisco UCS 6248UP 48-Port Fabric Interconnect is a one rack unit (1 RU) 10 Gigabit Ethernet, FCoE and
Fiber Channel switch offering up to 960-Gbps throughput and up to 48 ports. The switch has 32 1/10-Gbps
fixed Ethernet, FCoE and FC ports and one expansion slot.
Figure 3 Cisco UCS 6248UP Fabric Interconnect

Cisco HyperFlex HX-Series Nodes

A HyperFlex cluster requires a minimum of three HX-Series nodes. Data is replicated across at least two of
these nodes, and a third node is required for continuous operation in the event of a single-node failure. The
HX-Series nodes combine the CPU and RAM resources for hosting guest virtual machines, with the physical
storage resources used by the HyperFlex software. Each HX-Series node is equipped with one high-
performance SSD drive for data caching and rapid acknowledgment of write requests and is also is

Cisco HyperFlex HX220c M4S Node

The Cisco HyperFlex HX220c M4S rackmount server is one rack unit (1 RU) high and can mount in an
industry-standard 19-inch rack. This small footprint configuration contains a minimum of three nodes with
six 1.2 terabyte (TB) SAS drives that contribute to cluster storage capacity, a 120 GB SSD housekeeping
drive, a 480 GB SSD caching drive, and two Cisco Flexible Flash (FlexFlash) Secure Digital (SD) cards that
act as mirrored boot drives.
Figure 4 HX220c M4S Node

Cisco HyperFlex HX240c M4SX Node

The Cisco HyperFlex HX240c M4S rackmount server is two rack unit (2 RU) high and can mount in an
industry-standard 19-inch rack. This capacity optimized configuration contains a minimum of three nodes, a
minimum of six and up to twenty-three 1.2 TB SAS drives that contribute to cluster storage, a single 120 GB
SSD housekeeping drive, a single 1.6 TB SSD caching drive, and two FlexFlash SD cards that act as mirrored
boot drives.
Figure 5 HX240c M4SX Node

Cisco VIC 1227 MLOM Interface Card

The Cisco UCS Virtual Interface Card (VIC) 1227 is a dual-port Enhanced Small Form-Factor Pluggable
(SFP+) 10-Gbps Ethernet and Fibre Channel over Ethernet (FCoE)-capable PCI Express (PCIe) modular LAN-
on-motherboard (mLOM) adapter installed in the Cisco UCS HX-Series Rack Servers (Figure 6). The mLOM
slot can be used to install a Cisco VIC without consuming a PCIe slot, which provides greater I/O
expandability. It incorporates next-generation converged network adapter (CNA) technology from Cisco,
enabling a policy-based, stateless, agile server infrastructure that can present up to 256 PCIe standards-
compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs)
or host bus adapters (HBAs). The personality of the card is determined dynamically at boot time using the
service profile associated with the server. The number, type (NIC or HBA), identity (MAC address and World
Wide Name [WWN]), failover policy, and quality-of-service (QoS) policies of the PCIe interfaces are all
determined using the service profile.
Figure 6 Cisco VIC 1227 mLOM Card

Cisco UCS B200 M4 Node

For workloads that require additional computing and memory resources, but not additional storage capacity
or IOPs, a compute-intensive hybrid cluster configuration is allowed. This configuration contains a minimum
of three HX240c M4SX Nodes with up to four Cisco UCS B200 M4 Blade Servers for additional computing
capacity. The HX240c M4SX Nodes are configured as described previously, and the Cisco UCS B200 M4
servers are equipped with FlexFlash SD cards as boot drives, or can be configured to boot from an external
SAN. Use of the B200 M4 compute nodes also requires the Cisco UCS 5108 blade server chassis, and a pair
of Cisco UCS 2204XP Fabric Extenders.
Figure 7 B200 M4 Node

Cisco UCS 5108 Blade Chassis

The Cisco UCS 5100 Series Blade Server Chassis is a crucial building block of the Cisco Unified Computing
System, delivering a scalable and flexible blade server chassis. The Cisco UCS 5108 Blade Server Chassis,
is six rack units (6 RU) high and can mount in an industry-standard 19-inch rack. A single chassis can house
up to eight half-width Cisco UCS B-Series Blade Servers and can accommodate both half-width and full-
width blade form factors.

Four single-phase, hot-swappable power supplies are accessible from the front of the chassis. These power
supplies are 92 percent efficient and can be configured to support non-redundant, N+1 redundant, and grid
redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors
(one per power supply), and two I/O bays for Cisco UCS Fabric Extenders. A passive mid-plane provides up
to 40 Gbps of I/O bandwidth per server slot from each Fabric Extender. The chassis is capable of supporting
40 Gigabit Ethernet standards.
Figure 8 Cisco UCS 5108 Blade Chassis Front and Rear Views

Cisco UCS 2204XP Fabric Extender

The Cisco UCS 2200 Series Fabric Extenders multiplex and forward all traffic from blade servers in a chassis
to a parent Cisco UCS fabric interconnect over from 10-Gbps unified fabric links. All traffic, even traffic
between blades on the same chassis or virtual machines on the same blade, is forwarded to the parent
interconnect, where network profiles are managed efficiently and effectively by the fabric interconnect. At
the core of the Cisco UCS fabric extender are application-specific integrated circuit (ASIC) processors
developed by Cisco that multiplex all traffic.

The Cisco UCS 2204XP Fabric Extender has four 10 Gigabit Ethernet, FCoE-capable, SFP+ ports that
connect the blade chassis to the fabric interconnect. Each Cisco UCS 2204XP has sixteen 10 Gigabit
Ethernet ports connected through the midplane to each half-width slot in the chassis. Typically configured in
pairs for redundancy, two fabric extenders provide up to 80 Gbps of I/O to the chassis.
Figure 9 Cisco UCS 2204XP Fabric Extender

Cisco HyperFlex Converged Data Platform Software

The Cisco HyperFlex HX Data Platform is a purpose-built, high-performance, distributed file system with a
wide array of enterprise-class data managemen
distributed storage technology, exceeding the boundaries of first-generation hyperconverged
infrastructures. The data platform has all the features that you would expect of an enterprise shared storage
system, eliminating the need to configure and maintain complex Fibre Channel storage networks and
devices. The platform simplifies operations and helps ensure data availability. Enterprise-class storage
features include the following:

Replication replicates data across the cluster so that data availability is not affected if single or
multiple components fail (depending on the replication factor configured).

Deduplication is always on, helping reduce storage requirements in virtualization clusters in which
multiple operating system instances in client virtual machines result in large amounts of replicated
data.

Compression further reduces storage requirements, lowering costs, and the log-structured file
system is designed to store variable-sized blocks, minimizing internal fragmentation.

Thin provisioning allows large volumes to be created without requiring storage to support them until

proposition.

Fast, space-efficient clones, called HyperFlex ReadyClones, rapidly replicate storage volumes so
that virtual machines can be replicated simply through a few small metadata operations, with actual
data copied only for new write operations.

Snapshots help facilitate backup and remote-replication operations: needed in enterprises that
require always-on data availability.

Cisco HyperFlex HX Data Platform Administration Plug-in

The Cisco HyperFlex HX Data Platform is administered through a VMware vSphere web client plug-in.
Through this centralized point of control for the cluster, administrators can create volumes, monitor the data
platform health, and manage resource use. Administrators can also use this data to predict when the cluster
will need to be scaled.
Figure 10 vCenter HyperFlex Web Client Plugin

Cisco HyperFlex HX Data Platform Controller

A Cisco HyperFlex HX Data Platform controller resides on each node and implements the Cisco HyperFlex
HX Distributed Filesystem. The controller runs in user space within a virtual machine and intercepts and
handles all the I/O from guest virtual machines. The platform controller VM uses the VMDirectPath I/O
feature to provide PCI pass- method gives
the controller VM full control of the physical disk resources, utilizing the SSD drives as a read/write caching
layer, and the HDDs as a capacity layer for the distributed storage. The controller integrates the data
platform into VMware software through the use of two preinstalled VMware ESXi vSphere Installation
Bundles (VIBs):

IO Visor: This VIB provides a network file system (NFS) mount point so that the ESXi hypervisor can

perspective, it is simply attached to a network file system.

VMware API for Array Integration (VAAI): This storage offload API allows vSphere to request
advanced file system operations such as snapshots and cloning. The controller implements these
operations through manipulation of metadata rather than actual data copying, providing rapid
response, and thus rapid deployment of new environments.

Data Operations and Distribution

The Cisco HyperFlex HX Data Platform controllers handle all read and write operation requests from the
guest VMs to their virtual disks (VMDK) stored in the distributed datastores in the cluster. The data platform
distributes the data across multiple nodes of the cluster, and also across multiple capacity disks of each
node, according to the replication level policy selected during the cluster setup. This method avoids storage
hotspots on specific nodes, and on specific disks of the nodes, and thereby also avoids networking hotspots
or congestion from accessing more data on some nodes versus others.

Replication Factor
The policy for the number of duplicate copies of each storage block is chosen during cluster setup, and is
referred to as the replication factor (RF). The default setting for the Cisco HyperFlex HX Data Platform is
replication factor 3 (RF=3).

Replication Factor 3: For every I/O write committed to the storage layer, 2 additional copies of the
blocks written will be created and stored in separate locations, for a total of 3 copies of the blocks.
Blocks are distributed in such a way as to ensure multiple copies of the blocks are not stored on the
same disks, nor on the same nodes of the cluster. This setting can tolerate simultaneous failures of 2
disks, or 2 entire nodes without losing data and without resorting to restore from backup or other
recovery processes.

Replication Factor 2: For every I/O write committed to the storage layer, 1 additional copy of the
blocks written will be created and stored in separate locations, for a total of 2 copies of the blocks.
Blocks are distributed in such a way as to ensure multiple copies of the blocks are not stored on the
same disks, nor on the same nodes of the cluster. This setting can tolerate a failure of 1 disk, or 1
entire node without losing data and without resorting to restore from backup or other recovery
processes.

Data Write Operations

For each write operation, data is written to the local caching SSD on the node where the write originated,
and replica copies of that write are written to the caching SSD of the remote nodes in the cluster, according
to the replication factor setting. For example, at RF=3 a write will be written locally where the VM originated
the write, and two additional writes will be committed in parallel on two other nodes. The write operation will
not be acknowledged until all three copies are written to the caching layer SSDs. Written data is also cached
in a write log area resident in memory in the controller VM, along with the write log on the caching SSDs.
This process speeds up read requests when reads are requested of data that has recently been written.

Data Destaging, Deduplication, and Compression

The Cisco HyperFlex HX Data Platform constructs multiple write caching segments on the caching SSDs of
each node in the distributed cluster. As write cache segments become full, and based on policies
accounting for I/O load and access patterns, those write cache segments are locked and new writes roll
over to a new write cache segment. When the number of locked cache segments reaches a particular
threshold, they are destaged to the HDD capacity layer of the filesystem. During the destaging process, data
is deduplicated and compressed before being written to the HDD capacity layer. The resulting data after
deduplication and compression can now be written in a single sequential operation to the HDDs of the
server, avoiding disk head seek thrashing and accomplishing the task in the minimal amount of time (Figure
11). Since the data is already deduplicated and compressed before being written, the platform avoids
additional I/O overhead often seen on competing systems, which must later do a
read/dedupe/compress/write cycle. Deduplication, compression and destaging take place with no delays or
I/O penalties to the guest VMs making requests to read or write data.
Figure 11 HyperFlex HX Data Platform Data Movement

Data Read Operations

For data read operations, data may be read from multiple locations. For data that was very recently written,
the data is likely to still exist in the write log of the local platform controller memory, or the write log of the
local caching SSD. If local write logs do not contain the data, the distributed filesystem metadata will be
queried to see if the data is cached elsewhere, either in write logs of remote nodes, or in the dedicated read
cache area of the local and remote SSDs. Finally, if the data has not been accessed in a significant amount
of time, the filesystem will retrieve the data requested from the HDD capacity layer. As requests for reads
are made to the distributed filesystem and the data is retrieved from the HDD capacity layer, the caching
SSDs populate their dedicated read cache area to speed up subsequent requests for the same data. This
multi-tiered distributed system with several layers of caching techniques ensures that data is served at the
highest possible speed, leveraging the caching SSDs of the nodes fully and equally.
Solution Design

Requirements
The following sections detail the physical hardware, software revisions, and firmware versions required to
install the Cisco HyperFlex system. The components described are for a single 8 node Cisco HX cluster, or
for a si

Physical Components
Table 1 HyperFlex System Components
Component Hardware Required

Fabric Interconnect Two Cisco UCS 6248UP Fabric Interconnects

(optional) plus 16 port UP expansion module

Servers Eight Cisco HX-series HX220c M4S servers

Or

Eight Cisco HX-series HX240c M4SX servers

Or

Four Cisco HX-Series HX240c M4SX servers plus four Cisco UCS B200 M4
blade servers

Chassis Cisco UCS 5108 Blade Chassis (only if using the B200 M4 servers)

Fabric Extenders Cisco UCS 2204XP Fabric Extenders (required for the 5108 blade chassis
and B200 M4 blades)

The following table lists the hardware component options for the HX220c M4S server model:

Table 2 HX220c M4S Server Options

HX220c M4S options Hardware Required

Processors Chose a matching pair of any model of Intel E5-26xx v3 or v4

processors.

Memory 128 GB to 1536 GB of total memory using 16 GB, 32 GB or 64 GB

DDR4 2133-MHz RDIMM/PC4-17000 modules

Or

128 GB to 1536 GB of total memory using 16 GB, 32 GB or 64 GB

DDR4 2400-MHz RDIMM/PC4-19200 modules

Disk Controller Cisco 12Gbps Modular SAS HBA

SSD One 120 GB 2.5 Inch Enterprise Value 6G SATA SSD

And

One 480 GB 2.5 Inch Enterprise Performance 6G SATA SSD (3X

endurance)

HDD Six 1.2 TB SAS 12Gbps 10K RPM SFF HDD

Network Cisco UCS VIC1227 VIC MLOM Dual 10 GbE port

Boot Devices Two 64GB SD Cards for Cisco UCS Servers

For full specifications for the HX220c M4S, download the following document:
http://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-infrastructure/hyperflex-hx-
series/datasheet-c78-736817.pdf

The following table lists the hardware component options for the HX240c-M4SX server model:

Table 3 HX240c M4SX Server Options

HX240c M4SX options Hardware Required

Processors Chose a matching pair of any model of Intel E5-26xx v3 or v4

processors.

Memory 128 GB to 1536 GB of total memory using 16 GB, 32 GB or 64 GB

DDR4 2133-MHz RDIMM/PC4-17000 modules

Or

128 GB to 1536 GB of total memory using 16 GB, 32 GB or 64 GB

DDR4 2400-MHz RDIMM/PC4-19200 modules

Disk Controller Cisco 12Gbps Modular SAS HBA

SSD One 120 GB 2.5 Inch Enterprise Value 6G SATA SSD (in the rear
internal device bay)

And

One 1.6 TB 2.5 Inch Enterprise Performance 6G SATA SSD (3X

endurance)

HDD Minimum of six, up to twenty-three 1.2 TB SAS 12Gbps 10K RPM

SFF HDD

Network Cisco UCS VIC1227 VIC MLOM Dual 10 GbE port

Boot Devices Two 64GB SD Cards for Cisco UCS Servers

For full specifications for the HX240c M4SX, download the following document:
http://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-infrastructure/hyperflex-hx-
series/datasheet-c78-736818.pdf
The following table lists the hardware component options for the B200 M4 server model:

Table 4 B200 M4 Server Options

B200 M4 options Hardware Required

Processors Chose any Intel E5-26xx v3 or v4 processor model. Use of all v3 or all v4
processors within the same HyperFlex cluster is recommended.

Memory Any supported amount of total memory using 16 GB, 32 GB or 64 GB

DDR4 2133-MHz RDIMM/PC4-17000 or 2400-MHz RDIMM/PC4-19200
modules

Disk Controller none

SSD none

HDD none

Network Cisco UCS VIC1340 VIC MLOM

Boot Devices Two 64GB SD Cards for Cisco UCS Servers

Or

Configured to boot from an external SAN LUN via FC or iSCSI

Cisco Solution IDs

The following table lists the Cisco solution IDs that can be referenced by your Cisco Account Team, or your
Cisco Partner sales team to accelerate the order process using pre-defined HyperFlex solution groupings.
Additional Configure to Order (CTO) and SmartPlay bundles are also available, which give complete flexibility
of all the individual components which can be ordered.

Table 5 Cisco HyperFlex Solution IDs

Solution ID Description

HX-VSI-HX220M4-C1SKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX220c M4S servers, with E5-2680 v3 CPUs and 384 GB RAM

Includes vSphere 6.0 Enterprise Licenses (8CPU)

HX-VSI-HX220M4-C1VSKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX220c M4S servers, with E5-2680 v3 CPUs and 384 GB RAM

HX-VSI-HX220M4-C1XKA1 1x HX220c M4S server, with E5-2680 v3 CPUs and 384 GB RAM to expand
an existing cluster

Includes vSphere 6.0 Enterprise Licenses (2CPU)

HX-VSI-HX220M4-C1VXKA1 1x HX220c M4S server, with E5-2680 v3 CPUs and 384 GB RAM to expand
an existing cluster
HX-VSI-HX240M4-C2SKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX240c M4S servers, with E5-2680 v3 CPUs, 384 GB RAM and 15 1.2TB
capacity disks

Includes vSphere 6.0 Enterprise Licenses (8CPU)

HX-VSI-HX240M4-C2VSKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX240c M4S servers, with E5-2680 v3 CPUs, 384 GB RAM and 15 1.2TB
capacity disks

HX-VSI-HX240M4-C2XKA1 1x HX240c M4S server, with E5-2680 v3 CPUs, 384 GB RAM and 15 1.2TB
capacity disks to expand an existing cluster

Includes vSphere 6.0 Enterprise Licenses (2CPU)

HX-VSI-HX240M4-C2VXKA1 1x HX240c M4S server, with E5-2680 v3 CPUs, 384 GB RAM and 15 1.2TB
capacity disks to expand an existing cluster

HX-VSI-HX240M4-C3SKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX240c M4S servers, with E5-2680 v3 CPUs, 384 GB RAM and 23 1.2TB
capacity disks

Includes vSphere 6.0 Enterprise Licenses (8CPU)

HX-VSI-HX240M4-C3VSKA1 2x Cisco UCS 6248UP Fabric Interconnects

4x HX240c M4S servers, with E5-2680 v3 CPUs, 384 GB RAM and 23 1.2TB
capacity disks

HX-VSI-HX240M4-C3VXKA1 1x HX240c M4S server, with E5-2680 v3 CPUs, 384 GB RAM and 23 1.2TB
capacity disks to expand an existing cluster

Software Components
The following table lists the software components and the versions required for the Cisco HyperFlex system:

Table 6 Software Components

Component Software Required

Hypervisor VMware ESXi 5.5 update 3b, patch 1 (Or) VMware ESXi 6.0 update 1b or update 2

Use of a published Cisco custom ESXi ISO installer file is required.

VMware vSphere Enterprise or Enterprise Plus licensing is recommended.

VMware vSphere Standard, Essentials Plus, and ROBO editions are also
supported but only for vSphere 6.0 versions. Currently for these editions,
upgrades of HX Data Platform software would need to occur in an offline
maintenance window.
Management Server VMware vCenter Server for Windows or vCenter Server Appliance 5.5 update 3b or later.

Refer to http://www.vmware.com/resources/compatibility/sim/interop_matrix.php for

interoperability of your ESXi version and vCenter Server.

Cisco HyperFlex HX Cisco HyperFlex HX Data Platform Software 1.7.1

Data Platform

Cisco UCS Firmware Cisco UCS Infrastructure software, B-Series and C-Series bundles, revision 2.2(6f)

To support Intel E5-26xx v4 processors, the Cisco UCS Infrastructure software, B-Series
and C-Series bundles must be upgraded to revision 2.2(7c)

Considerations

Version Control
Note that the software revisions listed in Table 6 are the only valid and supported configuration at the time
of the publishing of this validated design. Special care must be taken not to alter the revision of the
hypervisor, vCenter server, Cisco HX platform software, or the Cisco UCS firmware without first consulting
the appropriate release notes and compatibility matrixes to ensure that the system is not being modified into
an unsupported configuration.

vCenter Server
This document does not cover the installation and configuration of VMware vCenter Server for Windows, or
the vCenter Server Appliance. The vCenter Server must be installed and operational prior to the installation
of the Cisco HyperFlex HX Data Platform software. The following best practice guidance applies to
installations of HyperFlex 1.7.1:

Do not modify the default TCP port settings of the vCenter installation. Using non-standard ports can
lead to failures during the installation.

Building the vCenter server as a virtual machine inside the HyperFlex cluster environment on a HX-
Series node is not allowed. There are no valid locations within the HX-Series servers that will contain
enough usable space in a VMFS datastore to house a vCenter server, as once the software is
installed, all of the available disks and space will be fully used by the HyperFlex cluster. Building any
virtual machines on the HyperFlex servers prior to installing the HyperFlex HX Data Platform software
is therefore not possible, as it will lead to installation failures. Please build the vCenter server on a
physical server or in a virtual environment outside of the HyperFlex cluster.

Scale
Cisco HyperFlex clusters currently scale up from a minimum of 3 to a maximum of 8 converged nodes per
cluster, i.e. 8 nodes providing storage resources to the HX Distributed Filesystem. For the compute intensive
iguration with 3-8 Cisco HX240c M4SX model servers can be combined with up to 4
Cisco B200 M4 blades, called compute-only nodes. The number of compute-only nodes cannot exceed the
number of converged nodes.

additional servers to the Cisco UCS domain, installing an additional HyperFlex cluster on them, and
controlling them via the same vCenter server. A maximum of 4 HyperFlex clusters can be managed by a
single vCenter server, therefore the maximum size of a single HyperFlex environment is 32 converged
nodes, plus up to 16 additional compute-only blades.

Capacity
Overall usable cluster capacity is based on a number of factors. The number of nodes in the cluster must be
considered, plus the number and size of the capacity layer disks. Caching disk sizes are not calculated as
part of the cluster capacity. The replication factor of the HyperFlex HX Data Platform also affects the cluster
capacity as it defines the number of copies of each block of data written.

Disk drive manufacturers have adopted a size reporting methodology using calculation by powers of 10, also
known as decimal prefix. As an example, a 120 GB disk is listed with a minimum of 120 x 10^9 bytes of
usable addressable capacity, or 120 billion bytes. However, many operating systems and filesystems report
their space based on standard computer binary exponentiation, or calculation by powers of 2, also called
binary prefix. In this example, 2^10 or 1024 bytes make up a kilobyte, 2^10 kilobytes make up a megabyte,
2^10 megabytes make up a gigabyte, and 2^10 gigabytes make up a terabyte. As the values increase, the
disparity between the two systems of measurement and notation get worse, at the terabyte level, the
deviation between a decimal prefix value and a binary prefix value is nearly 10%.

The International System of Units (SI) defines values and decimal prefix by powers of 10 as follows:

Table 7 SI Unit Values (Decimal Prefix)

Value Symbol Name

1000 bytes kB kilobyte

1000 kB MB megabyte

1000 MB GB gigabyte

1000 GB TB terabyte

The International Organization for Standardization (ISO) and the International Electrotechnical Commission
(IEC) defines values and binary prefix by powers of 2 in ISO/IEC 80000-13:2008 Clause 4 as follows:

Table 8 IEC Unit Values (Binary Prefix)

Value Symbol Name

1024 bytes KiB kibibyte

1024 KiB MiB mebibyte

1024 MiB GiB gibibyte

1024 GiB TiB tebibyte

For the purpose of this document, the decimal prefix numbers are used only for raw disk capacity as listed
by the respective manufacturers. For all calculations where raw or usable capacities are shown from the
perspective of the HyperFlex software, filesystems or operating systems, the binary prefix numbers are
used. This is done primarily to show a consistent set of values as seen by the end user from within the
HyperFlex vCenter Web Plugin when viewing cluster capacity, allocation and consumption, and also within
most operating systems.

The following table lists a set of HyperFlex HX Data Platform cluster usable capacity values, using binary
prefix, for an array of cluster configurations. These values are useful for determining the appropriate size of
HX cluster to initially purchase, and how much capacity can be gained by adding capacity disks. The
calculations for these values are listed in appendix A.

Table 9 Cluster Usable Capacities

HX-Series Node Capacity Capacity Cluster Cluster
Server Quantity Disk Size Disk Usable Usable
Model (each) Quantity Capacity at Capacity at
(per node) RF=2 RF=3

HX220c 8 1.2 TB 6 24.1 TiB 16.07 TiB

M4S

6 24.1 TiB 16.07 TiB

HX240c 8 1.2 TB 15 60.24 TiB 40.16 TiB

M4SX
23 92.38 TiB 61.58 TiB

Physical Topology

Topology Overview
The Cisco HyperFlex system is composed of a pair of Cisco UCS 6248UP Fabric Interconnects, along with
up to 8 HX-Series rack mount servers per cluster, with the option of adding Cisco UCS 5108 Blade chassis
to house Cisco UCS B200 M4 blade servers for additional compute resources in a hybrid cluster design. Up
to 4 separate HX clusters can be installed under a single pair of Fabric Interconnects. The Fabric
Interconnects both connect to every HX-Series rack mount server, and both connect to every Cisco UCS

are made from the Fabric Interconnects to the customer data center network at the time of installation.
Figure 12 HyperFlex Standard Topology
Figure 13 HyperFlex Hybrid Topology

Fabric Interconnects
Fabric Interconnects (FI) are deployed in pairs, wherein the two units operate as a management cluster,
while forming two separate network fabrics, referred to as the A side and B side fabrics. Therefore, many
design elements will refer to FI A or FI B, alternatively called fabric A or fabric B. Both Fabric Interconnects
are active at all times, passing data on both network fabrics for a redundant and highly available
configuration. Management services, including Cisco UCS Manager, are also provided by the two FIs but in a
clustered manner, where one FI is the primary, and one is secondary, with a roaming clustered IP address.
This primary/secondary relationship is only for the management cluster, and has no effect on data
transmission.

Fabric Interconnects have the following ports, which must be connected for proper management of the
Cisco UCS domain:

Mgmt: A 10/100/1000 Mbps port for managing the Fabric Interconnect and the Cisco UCS domain
via GUI and CLI tools. Also used by remote KVM, IPMI and SoL sessions to the managed servers
within the domain. This is typically connected to the customer management network.

L1: A cross connect port for forming the Cisco UCS management cluster. This is connected directly
to the L1 port of the paired Fabric Interconnect using a standard CAT5 or CAT6 Ethernet cable with
RJ45 plugs. It is not necessary to connect this to a switch or hub.

L2: A cross connect port for forming the Cisco UCS management cluster. This is connected directly
to the L2 port of the paired Fabric Interconnect using a standard CAT5 or CAT6 Ethernet cable with
RJ45 plugs. It is not necessary to connect this to a switch or hub.
Console: An RJ45 serial port for direct console access to the Fabric Interconnect. Typically used
during the initial FI setup process with the included serial to RJ45 adapter cable. This can also be
plugged into a terminal aggregator or remote console server device.

HX-Series Rack Mount Servers

The HX-Series converged servers are connected directly to the Cisco UCS Fabric Interconnects in Direct
Connect mode. This option enables Cisco UCS Manager to manage the HX-Series rack-mount servers using
a single cable for both management traffic and data traffic. Both the HX220c M4S and HX240c M4SX
servers are configured with the Cisco VIC 1227 network interface card (NIC) installed in a modular LAN on
motherboard (MLOM) slot, which has dual 10 Gigabit Ethernet (GbE) ports. The standard and redundant
connection practice is to connect port 1 of the VIC 1227 to a port on FI A, and port 2 of the VIC 1227 to a
port on FI B (Figure 14). Failure to follow this cabling practice can lead to errors, discovery failures, and loss
of redundant connectivity.
Figure 14 HX-Series Server Connectivity

B-Series Blade Servers

Hybrid HyperFlex clusters also incorporate 1-4 Cisco UCS B200 M4 blade servers for additional compute
capacity. Like all other Cisco UCS B-series blade servers, the B200 M4 must be installed within a Cisco UCS
5108 blade chassis. The blade chassis comes populated with 1-4 power supplies, and 8 modular cooling
fans. In the rear of the chassis are two bays for installation of Cisco Fabric Extenders. The Fabric Extenders
(also commonly called IO Modules, or IOMs) connect the chassis to the Fabric Interconnects. Internally, the
Fabric Extenders connect to the Cisco VIC 1340 card installed in each blade server across the chassis
backplane. The standard connection practice is to connect 1-4 10 GbE links from the left side IOM, or IOM
1, to FI A, and to connect the same number of 10 GbE links from the right side IOM, or IOM 2, to FI B (Figure
15). All other cabling configurations are invalid, and can lead to errors, discovery failures, and loss of
redundant connectivity.
Figure 15 Cisco UCS 5108 Chassis Connectivity

Logical Topology

Logical Network Design

The Cisco HyperFlex system has communication pathways that fall into four defined zones (Figure 16):

Management Zone: This zone comprises the connections needed to manage the physical hardware,
the hypervisor hosts, and the storage platform controller virtual machines (SCVM). These interfaces
and IP addresses need to be available to all staff who will administer the HX system, throughout the
LAN/WAN. This zone must provide access to Domain Name System (DNS) and Network Time
Protocol (NTP) services, and allow Secure Shell (SSH) communication. The VLAN used for
management traffic must be able to traverse the network uplinks from the Cisco UCS domain,
reaching both FI A and FI B. In this zone are multiple physical and virtual components:

— Fabric Interconnect management ports.

— Cisco UCS external management interfaces used by the servers and blades, which answer via the FI
management ports.

— ESXi host management interfaces.

— Storage Controller VM management interfaces.

— A roaming HX cluster management interface.

VM Zone: This zone comprises the connections needed to service network IO to the guest VMs that
will run inside the HyperFlex hyperconverged system. This zone typically contains multiple VLANs,
that are trunked to the Cisco UCS Fabric Interconnects via the network uplinks, and tagged with
802.1Q VLAN IDs. These interfaces and IP addresses need to be available to all staff and other
computer endpoints which need to communicate with the guest VMs in the HX system, throughout
the LAN/WAN.

Storage Zone: This zone comprises the connections used by the Cisco HX Data Platform software,
ESXi hosts, and the storage controller VMs to service the HX Distributed Filesystem. These interfaces
and IP addresses need to be able to communicate with each other at all times for proper operation.
During normal operation, this traffic all occurs within the Cisco UCS domain, however there are
hardware failure scenarios where this traffic would need to traverse the network northbound of the
Cisco UCS domain. For that reason, the VLAN used for HX storage traffic must be able to traverse
the network uplinks from the Cisco UCS domain, reaching FI A from FI B, and vice-versa. This zone
is primarily jumbo frame traffic therefore jumbo frames must be enabled on the Cisco UCS uplinks. In
this zone are multiple components:

— A vmkernel interface on each ESXi host in the HX cluster, used for storage traffic.

— Storage Controller VM storage interfaces.

— A roaming HX cluster storage interface.

VMotion Zone: This zone comprises the connections used by the ESXi hosts to enable vMotion of
the guest VMs from host to host. During normal operation, this traffic all occurs within the Cisco UCS
domain, however there are hardware failure scenarios where this traffic would need to traverse the
network northbound of the Cisco UCS domain. For that reason, the VLAN used for HX storage traffic
must be able to traverse the network uplinks from the Cisco UCS domain, reaching FI A from FI B,
and vice-versa.

Refer to the following figure for an illustration of the logical network design:
Figure 16 Logical Network Design
Design Elements
Installation of the HyperFlex system is primarily done via a deployable HyperFlex installer virtual machine,
available for download at cisco.com as an OVA file. The installer VM does most of the Cisco UCS
configuration work, it can be leveraged to simplify the installation of ESXi on the HyperFlex hosts, and also
performs significant portions of the ESXi configuration. Finally, the installer VM is used to install the
HyperFlex HX Data Platform software and create the HyperFlex cluster. Because this simplified installation
method has been developed by Cisco, this CVD will not give detailed manual steps for the configuration of
all the elements that are handled by the installer. Instead, the elements configured will be described and
documented in this section, and the subsequent sections will guide you through the manual steps needed for
installation, and how to utilize the HyperFlex Installer for the remaining configuration steps.

Network Design

Cisco UCS Uplink Connectivity

Cisco UCS network uplinks connect Cisco UCS Fabric Interconnects to the
LAN in the customer data center. All Cisco UCS uplinks operate as trunks, carrying multiple 802.1Q VLAN
IDs across the uplinks. The default Cisco UCS behavior is to assume that all VLAN IDs defined in the Cisco
UCS configuration are eligible to be trunked across all available uplinks.

Cisco UCS Fabric Interconnects appear on the network as a collection of endpoints versus another network
switch. Internally, the Fabric Interconnects do not participate in spanning-tree protocol (STP) domains, and
the Fabric Interconnects cannot form a network loop, as they are not connected to each other with a layer 2
Ethernet link. All link up/down decisions via STP will be made by the upstream root bridges.

Uplinks need to be connected and active from both of the Fabric Interconnects. For redundancy, multiple
uplinks can be used on each FI, either as 802.3ad Link Aggregation Control Protocol (LACP) port-channels,
or using individual links. For the best level of performance and redundancy, uplinks can be made as LACP
port-channels to a pair of upstream Cisco switches using the virtual port channel (vPC) feature. Using vPC
uplinks allows all uplinks to be active passing data, plus protects against any individual link failure, and the
failure of an upstream switch. Other uplink configurations can be redundant, but spanning-tree protocol loop
avoidance may disable links if vPC is not available.

All uplink connectivity methods must allow for traffic to pass from one Fabric Interconnect to the other, or
from fabric A to fabric B. There are scenarios where cable, port or link failures would require traffic that
normally does not leave the Cisco UCS domain, to now be forced over the Cisco UCS uplinks. Additionally,
this traffic flow pattern can be seen briefly during maintenance procedures, such as updating firmware on
the Fabric Interconnects, which requires them to be rebooted. The following sections and figures detail
several uplink connectivity options.

Single Uplinks to Single Switch

This connection design is susceptible to failures at several points; single uplink failures on either Fabric
Interconnect can lead to connectivity losses or functional failures, and the failure of the single uplink switch
will cause a complete connectivity outage.
Figure 17 Connectivity with Single Uplink to Single Switch

Port Channels to Single Switch

This connection design is now redundant against the loss of a single link, but remains susceptible to the
failure of the single switch.
Figure 18 Connectivity with Port-Channels to Single Switch

Single Uplinks or Port Channels to Multiple Switches

This connection design is redundant against the failure of an upstream switch, and redundant against a
single link failure. In normal operation, STP is likely to block half of the links to avoid a loop across the two
upstream switches. The side effect of this is to reduce bandwidth between the Cisco UCS domain and the
LAN. If any of the active links were to fail, STP would bring the previously blocked link online to provide
access to that Fabric Interconnect via the other switch. It is not recommended to connect both links from a
single FI to a single switch, as that configuration is susceptible to a single switch failure breaking connectivity
from fabric A to fabric B. For enhanced redundancy, the single links in the figure below could also be port-
channels.
Figure 19 Connectivity with Multiple Uplink Switches

vPC to Multiple Switches

This recommended connection design relies on using Cisco switches that have the virtual port channel
feature, such as Catalyst 6000 series switches running VSS, and Cisco Nexus 5000, 7000 and 9000 series
switches. Logically the two vPC enabled switches appear as one, and therefore spanning-tree protocol will
not block any links. This configuration allows for all links to be active, achieving maximum bandwidth
potential, and multiple redundancies at each level.
Figure 20 Connectivity with vPC

Example configurations for Cisco Nexus 9372 switches using vPC are provided in appendix C: Example Nex-
us 9372 Switch Configurations.

VLANs and Subnets

For the base HyperFlex system configuration, multiple VLANs need to be carried to the Cisco UCS domain
from the upstream LAN, and these VLANs are also defined in the Cisco UCS configuration. The following
table lists the VLANs created by the HyperFlex installer in Cisco UCS, and their functions:

Table 10 VLANs
VLAN Name VLAN ID Purpose

hx-inband-mgmt Customer supplied ESXi host management

interfaces

HX Storage Controller VM
management interfaces

HX Storage Cluster roaming

management interface

hx-storage-data Customer supplied ESXi host storage vmkernel

interfaces

HX Storage Controller storage

network interfaces

HX Storage Cluster roaming

storage interface

vm-network Customer supplied Guest VM network interfaces

hx-vmotion Customer supplied ESXi host vMotion vmkernel

interfaces

A dedicated network or subnet for physical device management is often used in data centers. In this scenario,
the mgmt0 interfaces of the Fabric Interconnects would be connected to that dedicated network or subnet.
This is a valid configuration for HyperFlex installations with the following caveat; wherever the HyperFlex in-
staller is deployed it must have IP connectivity to the subnet of the mgmt0 interfaces of the Fabric Intercon-
nects, and also have IP connectivity to the subnets used by the hx-inband-mgmt and hx-storage-data VLANs
listed above.

Jumbo Frames
All HyperFlex storage traffic traversing the hx-storage-data VLAN and subnet is configured to use jumbo
frames, or to be precise all communication is configured to send IP packets with a Maximum Transmission
Unit (MTU) size of 9000 bytes. Using a larger MTU value means that each IP packet sent carries a larger
payload, therefore transmitting more data per packet, and consequently sending and receiving data faster.
This requirement also means that the Cisco UCS uplinks must be configured to pass jumbo frames. Failure to
configure the Cisco UCS uplink switches to allow jumbo frames can lead to service interruptions during
some failure scenarios, particularly when cable or port failures would cause storage traffic to traverse the
northbound Cisco UCS uplink switches.

Multicast
The HyperFlex Storage Platform Controller VMs configure a roaming management virtual IP address using
UCARP, which relies on IPv4 multicast to function properly. The controller VMs advertise to one another
using the standard VRRP IPv4 multicast address of 224.0.0.18, which is a link-local address, and is therefore
not forwarded by routers or inspected by IGMP snooping. This requirement means the Cisco UCS uplinks
and uplink switches must allow IPv4 multicast traffic for the HyperFlex management VLAN. Failure to
configure the Cisco UCS uplink switches to allow IPv4 multicast traffic can lead to service interruptions
during some failure scenarios, particularly when cable or port failures would cause multicast management
traffic to traverse the northbound Cisco UCS uplink switches.

Cisco Nexus switches allow multicast traffic and enable IGMP snooping by default.

Cisco UCS Design

This section on Cisco UCS design will describe the elements within Cisco UCS Manager that are configured
by the Cisco HyperFlex installer. Many of the configuration elements are fixed in nature, meanwhile the
HyperFlex installer does allow for some items to be specified at the time of creation, for example, VLAN
names and IDs, IP pools and more. Where the elements can be manually set during the installation, those
items will be noted in <<>> brackets.

Cisco UCS Organization

During the HyperFlex Installation a Cisco UCS Sub-Organization is created n - -
organization is created underneath the root level of the Cisco UCS hierarchy, and is used to contain all
policies, pools, templates and service profiles used by HyperFlex. This arrangement allows for organizational
control using Role-Based Access Control (RBAC) and administrative locales at a later time if desired. In this
way, control can be granted to administrators of only the HyperFlex specific elements of the Cisco UCS
domain, separate from control of root level elements or elements in other sub-organizations.
Figure 21 Cisco UCS HyperFlex Sub-Organization

Cisco UCS LAN Policies

QoS System Classes

Specific Cisco UCS Quality of Service (QoS) system classes are defined for a Cisco HyperFlex system.
These classes define Class of Service (CoS) values that can be used by the uplink switches north of the
Cisco UCS domain, plus which classes are active, along with whether packet drop is allowed, the relative
weight of the different classes when there is contention, the maximum transmission unit (MTU) size, and if
there is multicast optimization applied. QoS system classes are defined for the entire Cisco UCS domain, the
classes that are enabled can later be used in QoS policies, which are then assigned to Cisco UCS vNICs.
The following table and figure details the QoS System Class settings configured for HyperFlex:

Table 11 QoS System Classes

Priority Enabled CoS Packet Weight MTU Multicast
Drop Optimized

Platinum Yes 5 No 4 9216 No

Gold Yes 4 Yes 4 Normal No

Silver Yes 2 Yes Best- Normal Yes

effort

Bronze Yes 1 Yes Best- 9216 No

effort

Best Yes Any Yes Best- Normal No

Effort effort

Fibre Yes 3 No 5 FC N/A

Channel
Figure 22 QoS System Classes

The Silver QoS system class has the multicast optimized setting enabled because the Silver class is used by
the Silver QoS policy, which is in turn used by the management network vNICs. They transmit IPv4 multicast
packets to determine the master server for the roaming cluster management IP address, configured by
UCARP.

QoS Policies
In order to apply the settings defined in the Cisco UCS QoS System Classes, specific QoS Policies must be
created, and then assigned to the vNICs, or vNIC templates used in Cisco UCS Service Profiles. The
following table details the QoS Policies configured for HyperFlex, and their default assignment to the vNIC
templates created:

Table 12 QoS Policies

Policy Priority Burst Rate Host Used by vNIC
Control Template:

Platinum Platinum 10240 Line- None storage-data-a

rate
storage-data-b

Gold Gold 10240 Line- None vm-network-a

rate
vm-network-b

Silver Silver 10240 Line- None hv-mgmt-a

rate
hv-mgmt-b

Bronze Bronze 10240 Line- None hv-vmotion-a

rate
hv-vmotion-b

Best Best Effort 10240 Line- None N/A

Effort rate

Platinum Platinum 10240 Line- None storage-data-a

rate
storage-data-b

Gold Gold 10240 Line- None vm-network-a

rate
vm-network-b
Multicast Policy
A UCS Multicast Policy is configured by the HyperFlex installer, which is referenced by the VLANs that are
created. The policy allows for future flexibility if a specific multicast policy needs to be created and applied
to other VLANs, which may be used by non-HyperFlex workloads in the Cisco UCS domain. The following
table and figure details the Multicast Policy configured for HyperFlex:

Table 13 Multicast Policy

Name IGMP Snooping State IGMP Snooping Querier
State

HyperFlex Enabled Disabled

Figure 23 Multicast Policy

VLANs
VLANs are created by the HyperFlex installer to support a base HyperFlex system, with a single VLAN
defined for guest VM traffic, and a VLAN for vMotion. Names and IDs for the VLANs are defined in the Cisco
UCS configuration page of the HyperFlex installer web interface. The VLANs listed in Cisco UCS must
already be present on the upstream network, and the Cisco UCS FIs do not participate in VLAN Trunk
Protocol (VTP). The following table and figure details the VLANs configured for HyperFlex:

Table 14 Cisco UCS VLANs

Name ID Type Transport Native VLAN Sharing

<<hx-inband- <user_defined> LAN Ether No None

mgmt>>

<<hx-storage- <user_defined> LAN Ether No None

data>>

<<vm- <user_defined> LAN Ether No None

network>>

<<hx- <user_defined> LAN Ether No None

vmotion>>
Figure 24 Cisco UCS VLANs

Management IP Address Pool

A Cisco UCS Management IP Address Pool must be populated with a block of IP addresses. These IP
addresses are assigned to the Cisco Integrated Management Controller (CIMC) interface of the rack-mount
and blade servers that are managed in the Cisco UCS domain. The IP addresses are the communication
endpoints for various functions, such as remote KVM, virtual media, Serial over LAN (SoL), and Intelligent
Platform Management Interface (IPMI) for each rack-mount or blade server. Therefore, a minimum of one IP
-
of-
which answer ARP requests for the management addresses. Because of this arrangement, the IP addresses

- ed with a block of IP addresses, a subnet mask, and a

default gateway by the HyperFlex installer.
Figure 25 Management IP Address Pool
Figure 26 IP Address Block

MAC Address Pools

One of the core benefits of the Cisco UCS and Virtual Interface Card (VIC) technology is the assignment of
the personality of the card via Cisco UCS Service Profiles. The number of virtual NIC (vNIC) interfaces, their
VLAN association, MAC addresses, QoS policies and more are all applied dynamically as part of the
association process. Media Access Control (MAC) addresses use 6 bytes of data as a unique address to
identify the interface on the layer 2 network. All devices are assigned a unique MAC address, which is
ultimately used for all data transmission and reception. Cisco UCS Manager picks a MAC address from a
pool of addresses for each vNIC defined in the service profile, when that service profile is created. The MAC
addresses are then configured on the Cisco VIC cards when the service profile is assigned to a physical
server.

Best practices mandate that MAC addresses used for Cisco UCS domains use 00:25:B5 as the first three
bytes, which is one of the Organizationally Unique Identifiers (OUI) registered to Cisco Systems, Inc. The
fourth byte (for example, 00:25:B5:xx) is specified during the HyperFlex installation. The fifth byte is set
automatically by the HyperFlex installer, to correlate to the Cisco UCS fabric and the vNIC placement order.
Finally, the last byte is incremented according to the number of MAC addresses created in the pool. To avoid
overlaps, you must ensure that the first four bytes of the MAC address pools are unique for each HyperFlex
system installed in the same layer 2 network, and also different from other Cisco UCS domains which may
exist.

The following table details the MAC Address Pools configured for HyperFlex, and their default assignment to
the vNIC templates created:

Table 15 MAC Address Pools

Name Block Start Size Assignment Used by vNIC
Order Template:

hv-mgmt-a 00:25:B5:<xx>:A1:01 100 Sequential hv-mgmt-a

hv-mgmt-b 00:25:B5:<xx>:B2:01 100 Sequential hv-mgmt-b

hv-vmotion-a 00:25:B5:<xx>:A7:01 100 Sequential hv-vmotion-a

hv-vmotion-b 00:25:B5:<xx>:B8:01 100 Sequential hv-vmotion-b

storage-data-a 00:25:B5:<xx>:A3:01 100 Sequential storage-data-a

storage-data-b 00:25:B5:<xx>:B4:01 100 Sequential storage-data-b

vm-network-a 00:25:B5:<xx>:A5:01 100 Sequential vm-network-a

vm-network-b 00:25:B5:<xx>:B6:01 100 Sequential vm-network-b

Figure 27 MAC Address Pools

Network Control Policies

Cisco UCS Network Control Policies control various aspects of the behavior of vNICs defined in the Cisco
UCS Service Profiles. Settings controlled include enablement of Cisco Discovery Protocol (CDP), MAC
address registration, MAC address forging, and the action taken on the vNIC status if the Cisco UCS network
uplinks are failed. Two policies are configured by the HyperFlex Installer, HyperFlex-infra is applied to the
-vm, which is only applied to the
vNIC interfaces carrying guest VM traffic. This allows for more flexibility, even though the policies are
currently configured with the same settings. The following table details the Network Control Policies
configured for HyperFlex, and their default assignment to the vNIC templates created:

Table 16 Network Control Policy

Name CDP MAC Action on MAC Used by vNIC
Register Uplink Fail Security Template:
Mode

HyperFlex-infra Enabled Only Link-down Forged: hv-mgmt-a

Native Allow
VLAN hv-mgmt-b

hv-vmotion-a

hv-vmotion-b

storage-data-a

storage-data-b
HyperFlex-vm Enabled Only Link-down Forged: vm-network-a
Native Allow
VLAN vm-network-b
Figure 28 Network Control Policy

vNIC Templates
Cisco UCS Manager has a feature to configure vNIC templates, which can be used to simplify and speed up
configuration efforts. VNIC templates are referenced in service profiles and LAN connectivity policies, versus
configuring the same vNICs individually in each service profile, or service profile template. VNIC templates
contain all the configuration elements that make up a vNIC, including VLAN assignment, MAC address pool
selection, fabric A or B assignment, fabric failover, MTU, QoS policy, Network Control Policy, and more.
Templates are created as either initial templates, or updating templates. Updating templates retain a link
between the parent template and the child object, therefore when changes are made to the template, the
changes are propagated to all remaining linked child objects. The following tables detail the settings in each
of the vNIC templates created by the HyperFlex installer:

Table 17 vNIC Template hv-mgmt-a

vNIC Template Name: hv-mgmt-a

Setting Value

Fabric ID A

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 1500

MAC Pool hv-mgmt-a

QoS Policy silver

Network Control Policy HyperFlex-infra

VLANs <<hx-inband-mgmt>> Native: yes

Table 18 vNIC Template hv-mgmt-b
vNIC Template Name: hv-mgmt-b

Setting Value

Fabric ID B

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 1500

MAC Pool hv-mgmt-b

QoS Policy silver

Network Control Policy HyperFlex-infra

VLANs <<hx-inband-mgmt>> Native: yes

Table 19 vNIC Template hv-vmotion-a

vNIC Template Name: hv-vmotion-a

Setting Value

Fabric ID A

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 9000

MAC Pool hv-vmotion-a

QoS Policy broze

Network Control Policy HyperFlex-infra

VLANs <<hx-vmotion>> Native: yes

Table 20 vNIC Template hx-vmotion-b

vNIC Template Name: hv-vmotion-b

Setting Value

Fabric ID B

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 9000

MAC Pool hv-vmotion-b

QoS Policy broze

Network Control Policy HyperFlex-infra

VLANs <<hx-vmotion>> Native: yes

Table 21 vNIC Template storage-data-a

vNIC Template Name: storage-data-a

Setting Value

Fabric ID A

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 9000

MAC Pool storage-data-a

QoS Policy platinum

Network Control Policy HyperFlex-infra

VLANs <<hx-storage-data>> Native: yes

Table 22 vNIC Template storage-data-b

vNIC Template Name: storage-data-b

Setting Value

Fabric ID B

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 9000

MAC Pool storage-data-b

QoS Policy platinum

Network Control Policy HyperFlex-infra

VLANs <<hx-storage-data>> Native: yes

Table 23 vNIC Template vm-network-a
vNIC Template Name: vm-network-a

Setting Value

Fabric ID A

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 1500

MAC Pool vm-network-a

QoS Policy gold

Network Control Policy HyperFlex-vm

VLANs <<vm-network>> Native: no

Table 24 vNIC Template vm-network-b

vNIC Template Name: vm-network-b

Setting Value

Fabric ID B

Fabric Failover Disabled

Target Adapter

Type Updating Template

MTU 1500

MAC Pool vm-network-b

QoS Policy gold

Network Control Policy HyperFlex-vm

VLANs <<vm-network>> Native: no

LAN Connectivity Policies

Cisco UCS Manager has a feature for LAN Connectivity Policies, which aggregates all of the vNICs or vNIC
templates desired for a service profile configuration into a single policy definition. This simplifies
configuration efforts by defining a collection of vNICs or vNIC templates once, and using that policy in the
service profiles or service profile templates. The HyperFlex installer configures a LAN Connectivity Policy
named HyperFlex, which contains all of the vNIC templates defined in the previous section, along with an
Adapter Policy named HyperFlex, also configured by the HyperFlex installer. The following table details the
LAN Connectivity Policy configured for HyperFlex:
Table 25 LAN Connectivity Policy
Policy Name Use vNIC vNIC Name vNIC Template Adapter Policy
Template Used:

HyperFlex Yes hv-mgmt-a hv-mgmt-a HyperFlex

hv-mgmt-b hv-mgmt-b

hv-vmotion-a hv-vmotion-a

hv-vmotion-b hv-vmotion-b

storage-data-a storage-data-a

storage-data-b storage-data-b

vm-network-a vm-network-a

vm-network-b vm-network-b

Cisco UCS Server Policies

Adapter Policies
Cisco UCS Adapter Policies are used to configure various settings of the Converged Network Adapter (CNA)
installed in the Cisco UCS blade or rack-mount servers. Various advanced hardware features can be enabled
or disabled depending on the software or operating system being used. The following figures detail the LAN
Connectivity Policy configured for HyperFlex:
Figure 29 Cisco UCS Adapter Policy Resources
Figure 30 Cisco UCS Adapter Policy Options

BIOS Policies
Cisco HX-Series servers have a set of pre-defined BIOS setting defaults defined in Cisco UCS Manager.
These settings have been optimized for the Cisco HX-Series servers running HyperFlex. The HyperFlex
h all settings set to the defaults, except for enabling
the Serial Port A for Serial over LAN (SoL) functionality. This policy allows for future flexibility in case
situations arise where the settings need to be modified from the default configuration.

Boot Policies
Cisco UCS Boot Policies define the boot devices used by blade and rack-mount servers, and the order that
they are attempted to boot from. Cisco HX-Series rack-mount servers and compute-only B200 M4 blade
servers have their VMware ESXi hypervisors installed to an internal pair of mirrored Cisco FlexFlash SD
cards, therefore they require a boot policy defining that the servers should boot from that location. The
m SD card. The
following figure details the HyperFlex Boot Policy configured to boot from SD card:
Figure 31 Cisco UCS Boot Policy

A compute-only B200 M4 blade server can optionally be configured to boot from an FC based SAN LUN. A
custom boot policy must be made to support this configuration, and used in the service profiles and templates
assigned to the B200 M4 blades.

Host Firmware Packages

Cisco UCS Host Firmware Packages represent one of the most powerful features of the Cisco UCS platform;
the ability to control the firmware revision of all the managed blades and rack-mount servers via a policy
specified in the service profile. Host Firmware Packages are defined and referenced in the service profiles
and once a service profile is associated to a server, the firmware of all the components defined in the Host
Firmware Package are automatically upgraded or downgraded to match the package. The HyperFlex installer

applying firmware revisions to all components that match a specific Cisco UCS firmware bundle, versus
defining the firmware revisions part by part. The following figure details the Host Firmware Package
configured by the HyperFlex installer:
Figure 32 Cisco UCS Host Firmware Package

Local Disk Configuration Policies

Cisco UCS Local Disk Configuration Policies are used to define the configuration of disks installed locally
within each blade or rack-mount server, most often to configure Redundant Array of
Independent/Inexpensive Disks (RAID) levels when multiple disks are present for data protection. Since HX-
Series converged nodes providing storage resources do not require RAID, the HyperFlex installer creates a
allows any local disk configuration. The policy also
enables the embedded FlexFlash SD cards used to boot the VMware ESXi hypervisor. The following figure
details the Local Disk Configuration Policy configured by the HyperFlex installer:
Figure 33 Cisco UCS Local Disk Configuration Policy

Maintenance Policies
Cisco UCS Maintenance Policies define the behavior of the attached blades and rack-mount servers when
changes are made to the associated service profiles. The default Cisco UCS Maintenance Policy setting is

result in an immediate reboot of that server. The Cisco best practice is to use a Maintenance Policy set to
- knowledgement by a user with the appropriate rights within Cisco
UCS, before the server is rebooted to apply the changes. The HyperFlex installer creates a Maintenance
- tails the
Maintenance Policy configured by the HyperFlex installer:
Figure 34 Cisco UCS Maintenance Policy

Power Control Policies

Cisco UCS Power Control Policies allow administrators to set priority values for power application to servers
in environments where power supply may be limited, during times when the servers demand more power

capping disabled, and fans allowed to run at full speed when necessary. The following figure details the
Power Control Policy configured by the HyperFlex installer:
Figure 35 Cisco UCS Power Control Policy

Scrub Policies
Cisco UCS Scrub Policies are used to scrub, or erase data from local disks, BIOS settings and FlexFlash SD
cards. If the policy settings are enabled, the information is wiped when the service profile using the policy is

all settings disabled, therefore all data on local disks, SD cards and BIOS settings will be preserved if a
service profile is disassociated. The following figure details the Scrub Policy configured by the HyperFlex
installer:
Figure 36 Cisco UCS Scrub Policy

Serial over LAN Policies

Cisco UCS Serial over LAN (SoL) Policies enable console output which is sent to the serial port of the server,
to be accessible via the LAN. For many Linux based operating systems, such as VMware ESXi, the local
serial port can be configured as a local console, where users can watch the system boot, and communicate
with the system command prompt interactively. Since many blade servers do not have physical serial ports,
and often administrators are working remotely, the ability to send and receive that traffic via the LAN is very
helpful. Connections to a SoL session can be initiated from Cisco UCS Manager. The HyperFlex installer

configured by the HyperFlex installer:

Figure 37 Cisco UCS Serial over LAN Policy

vMedia Policies
Cisco UCS Virtual Media (vMedia) Policies automate the connection of virtual media files to the remote KVM
session of the Cisco UCS blades and rack-mount servers. Using a vMedia policy can speed up installation
time by automatically attaching an installation ISO file to the server, without having to manually launch the
remote KVM console and connect them one-by-one. The HyperFlex installer creates a vMedia Policy named
ined. The following figure details the vMedia Policy
configured by the HyperFlex installer:
Figure 38 Cisco UCS vMedia Policy

Cisco UCS Service Profile Templates

Cisco UCS Manager has a feature to configure service profile templates, which can be used to simplify and
speed up configuration efforts when the same configuration needs to be applied to multiple servers. Service
profile templates are used to spawn multiple service profile copies to associate with the servers, versus
configuring the same service profile manually each time it is needed. Service profile templates contain all the
configuration elements that make up a service profile, including vNICs, vHBAs, local disk configurations, boot
policies, host firmware packages, BIOS policies and more. Templates are created as either initial templates,
or updating templates. Updating templates retain a link between the parent template and the child object,
therefore when changes are made to the template, the changes are propagated to all remaining linked child
object - -

only nodes needs to differ from the configuration of the HyperFlex converged storage nodes. The following
table details the service profile templates configured by the HyperFlex installer:

Table 26 HyperFlex Service Profile Templates

Service Profile Template Name: hx-nodes compute-nodes

Setting Value Value

UUID Pool Hardware Default Hardware Default

Associated Server Pool None None

Maintenance Policy HyperFlex HyperFlex

Management IP Address Policy None None

Local Disk Configuration Policy HyperFlex HyperFlex

LAN Connectivity Policy HyperFlex HyperFlex

Boot Policy HyperFlex HyperFlex

BIOS Policy HyperFlex HyperFlex

Firmware Policy HyperFlex HyperFlex

Power Control Policy HyperFlex HyperFlex

Scrub Policy HyperFlex HyperFlex

Serial over LAN Policy HyperFlex HyperFlex

vNIC/vHBA Placement
In order to control the order of detection of the vNICs and vHBAs defined in service profiles, Cisco UCS
allows for the definition of the placement of the vNICs and vHBAs across the cards in a blade or rack-mount
server, and the order they are seen. Since HX-series servers are configured with a single Cisco UCS VIC
1227 mLOM card, the only valid placement is on card number 1. In certain hardware configurations, the
physical mapping of cards and port extenders to their logical order is not linear, therefore each card is
referred to as a virtual connection, or vCon. Because of this, the interface where the placement and order is
defined does not refer to physical cards, but instead refers to vCons. Therefore, all the vNICs defined in the
service profile templates for HX-series servers places them on vCon 1, then their order is defined.

Through the combination of the vNIC templates created (vNIC Templates), the LAN Connectivity Policy (LAN
Connectivity Policies), and the vNIC placement, every VMware ESXi server will detect the network interfaces
in the same order, and they will always be connected to the same VLANs via the same network fabrics. The
following table outlines the vNICs, their placement, their order, the fabric they are connected to, their default
VLAN, and how they are enumerated by the ESXi hypervisor:

Table 27 vNIC Placement

vNIC Placement Ord Fabric VLAN ESXi interface
er enumeration

hv-mgmt-a 1 1 A <<hx-inband- vmnic0

mgmt>>

hv-mgmt-b 1 2 B <<hx-inband- vmnic1

mgmt>>

storage-data-a 1 3 A <<hx-storage- vmnic2

data>>

storage-data-b 1 4 B <<hx-storage- vmnic3

data>>

vm-network-a 1 5 A <<vm- vmnic4

network>>

vm-network-b 1 6 B <<vm- vmnic5

network>>
hv-vmotion-a 1 7 A <<hx-vmotion>> vmnic6

hv-vmotion-b 1 8 B <<hx-vmotion>> vmnic7

Figure 39 vNIC Placement

ESXi VMDirectPath relies on a fixed PCI address for the pass-through devices. If the vNIC configuration is
changed (add/remove vNICs), then the order of the devices seen in the PCI tree will change. The administra-
tor will have to reconfigure the ESXi VMDirectPath configuration to select the 12 Gbps SAS HBA card, and re-
configure the storage controller settings of the controller VM.

ESXi Host Design

The following sections detail the design of the elements within the VMware ESXi hypervisors, system
requirements, virtual networking and the configuration of ESXi for the Cisco HyperFlex HX Distributed Data
Platform.

Virtual Networking Design

The Cisco HyperFlex system has a pre-defined virtual network design at the ESXi hypervisor level. Four
different virtual switches are created by the HyperFlex installer, each using two uplinks, which are each
serviced by a vNIC defined in the Cisco UCS service profile. The vSwitches created are:

vswitch-hx-inband-mgmt: This is the default vSwitch0 which is renamed by the ESXi kickstart file
as part of the automated installation. The default vmkernel port, vmk0, is configured in the standard
Management Network port group. The switch has two uplinks, active on fabric A and standby on
fabric B, without jumbo frames. A second port group is created for the Storage Platform Controller
VMs to connect to with their individual management interfaces. By default, all traffic is untagged.

vswitch-hx-storage-data: This vSwitch is created as part of the automated installation. A vmkernel

port, vmk1, is configured in the Storage Hypervisor Data Network port group, which is the interface
used for connectivity to the HX Datastores via NFS. The switch has two uplinks, active on fabric B
and standby on fabric A, with jumbo frames required. A second port group is created for the Storage
Platform Controller VMs to connect to with their individual storage interfaces. By default, all traffic is
untagged.

vswitch-hx-vm-network: This vSwitch is created as part of the automated installation. The switch
has two uplinks, active on both fabrics A and B, and without jumbo frames. The HyperFlex installer
does not configure any port groups by default, as it is expected to be done as part of a post-install
step. By default, all traffic to this vSwitch is tagged.

vmotion: This vSwitch is created as part of the automated installation. A vmkernel port, used for
vMotion is not created, as it is expected to be done as part of a post-install step. The switch has two
uplinks, active on fabric A and standby on fabric B, with jumbo frames required. By default, all traffic
is untagged.

The following table and figures help give more details into the ESXi virtual networking design as built by the
HyperFlex installer:

Table 28 Virtual Switches

Virtual Port Groups Active Passive VLAN IDs Jumbo
Switch vmnic(s) vmnic(s)

vswitch-hx- Management vmnic0 vmnic1 untagged no

inband-mgmt Network

Storage
Controller
Management
Network

vswitch-hx- Storage vmnic3 vmnic2 untagged yes

storage-data Controller
Data
Network

Storage
Hypervisor
Data
Network

vswitch-hx- none vmnic4,vmnic5 none <vm- no

vm-network network>

vmotion none vmnic6 vmnic7 untagged yes

Figure 40 ESXi Network Design

VMDirectPath I/O Pass-through

VMDirectPath I/O allows a guest VM to directly access PCI and PCIe devices in an ESXi host as though they
were physical devices belonging to the VM itself, also referred to as PCI pass-through. With the appropriate
driver for the hardware device, the guest VM sends all I/O requests directly to the physical device, bypassing
the hypervisor. In the Cisco HyperFlex system, the Storage Platform Controller VMs use this feature to gain
full control of the Cisco 12Gbps SAS HBA cards in the Cisco HX-series rack-mount servers. This gives the
controller VMs direct hardware level access to the physical disks installed in the servers, which they
consume to construct the Cisco HX Distributed Filesystem. Only the disks connected directly to the Cisco
SAS HBA or to a SAS extender, in turn connected to the SAS HBA are controlled by the controller VMs.
Other disks, connected to different controllers, such as the SD cards, remain under the control of the ESXi
hypervisor. The configuration of the VMDirectPath I/O feature is done by the Cisco HyperFlex installer, and
requires no manual steps.

Storage Platform Controller VMs

A key component of the Cisco HyperFlex system is the Storage Platform Controller Virtual Machine running
on each of the nodes in the HyperFlex cluster. The controller VMs cooperate to form and coordinate the
Cisco HX Distributed Filesystem, and service all the guest VM IO requests. The controller VMs are deployed
as a vSphere ESXi agent, which is similar in concept to that of a Linux or Windows service. ESXi agents are
tied to a specific host, they start and stop along with the ESXi hypervisor, and the system is not considered
to be online and ready until both the hypervisor and the agents have started. Each ESXi hypervisor host has a
single ESXi agent deployed, which is the controller VM for that node, and it cannot be moved or migrated to
another host. The collective ESXi agents are managed via an ESXi agency in the vSphere cluster.
The storage controller VM runs custom software and services that manage and maintain the Cisco HX
Distributed Filesystem. The services and processes that run within the controller VMs are not exposed as
part of the ESXi agents to the agency, therefore the ESXi hypervisors nor vCenter server have any direct
knowledge of the storage services provided by the controller VMs. Management and visibility into the
function of the controller VMs, and the Cisco HX Distributed Filesystem is done via a plugin installed to the
vCenter server or appliance managing the vSphere cluster. The plugin communicates directly with the
controller VMs to display the information requested, or make the configuration changes directed, all while
operating within the same web-based interface of the vSphere Web Client. The deployment of the controller
VMs, agents, agency, and vCenter plugin are all done by the Cisco HyperFlex installer, and requires no
manual steps.

Controller VM Locations
The physical storage location of the controller VMs differs between the Cisco HX220c-M4S and HX240c-
M4SX model servers, due to differences with the physical disk location and connections on the two models
of servers. The storage controller VM is operationally no different from any other typical virtual machines in
an ESXi environment. The VM must have a virtual disk with the bootable root filesystem available in a location
separate from the SAS HBA that the VM is controlling via VMDirectPath I/O. The configuration details of the
models are as follows:

HX220c
placed on a 3.5 GB VMFS datastore, and that datastore is provisioned from the internal mirrored SD
cards. The controller VM has full control of all the front facing hot-swappable disks via PCI pass-
through control of the SAS HBA. The controller VM operating system sees the 120 GB SSD, also
and logs on
this disk. The remaining disks seen by the controller VM OS are used by the HX Distributed
filesystem for caching and capacity layers.

HX240c: The HX240c-M4SX server has a built-in SATA controller provided by the Intel Wellsburg
Platform Controller Hub (PCH) chip, and the 120 GB housekeeping disk is connected to it, placed in
an internal drive carrier. Since this model does not connect the 120 GB housekeeping disk to the
SAS HBA, the ESXi hypervisor remains in control of this disk, and a VMFS datastore is provisioned
there, using the entire disk. On this VMFS datastore, a 2.2 GB virtual disk is created and used by the
controller VM as /dev/sda for the root filesystem, and an 87 GB virtual disk is created and used by
the controller VM as /dev/sdb, placing the HyperFlex binaries and logs on this disk. The front-facing
hot swappable disks, seen by the controller VM OS via PCI pass-through control of the SAS HBA,
are used by the HX Distributed filesystem for caching and capacity layers.

The following figures detail the Storage Platform Controller VM placement on the ESXi hypervisor hosts:
Figure 41 HX220c Controller VM Placement

B200 M4 compute-only blades also place a lightweight storage controller VM on a 3.5 GB VMFS datastore,
provisioned from the boot drive.
Figure 42 HX240c Controller VM Placement

CPU Resource Reservations

Since the storage controller VMs provide critical functionality of the Cisco HX Distributed Data Platform, the
HyperFlex installer will configure CPU resource reservations for the controller VMs. This reservation
guarantees that the controller VMs will have CPU resources at a minimum level, in situations where the
physical CPU resources of the ESXi hypervisor host are being heavily consumed by the guest VMs. The
following table details the CPU resource reservation of the storage controller VMs:

Table 29 Controller VM CPU Reservations

Number of vCPU Shares Reservation Limit

8 Low 10800 MHz unlimited

Memory Resource Reservations

Since the storage controller VMs provide critical functionality of the Cisco HX Distributed Data Platform, the
HyperFlex installer will configure memory resource reservations for the controller VMs. This reservation
guarantees that the controller VMs will have memory resources at a minimum level, in situations where the
physical memory resources of the ESXi hypervisor host are being heavily consumed by the guest VMs. The
following table details the memory resource reservation of the storage controller VMs:
Table 30 Controller VM Memory Reservations
Server Model Amount of Guest Memory Reserve All Guest Memory

HX220c-M4S 48 GB Yes

HX240c-M4SX 72 GB Yes

B200 M4 compute-only blades have a lightweight storage controller VM, it is configured with only 1 vCPU and
512 MB of memory reservation.
Installation
Cisco HyperFlex systems are normally ordered with a factory pre-installation process having been done prior
to the hardware delivery. This factory integration work will deliver the HyperFlex servers with the proper
firmware revisions pre-set, a copy of the VMware ESXi hypervisor software pre-installed, and some
components of the Cisco HyperFlex software already installed. Once on site, the final steps to be performed

to the previous factory work. For the purpose of this document, the entire setup process is described as
though no factory pre-installation work was done, yet still leveraging the tools and processes developed by
Cisco to simplify the installation and dramatically reduce the deployment time.

Installation of the Cisco HyperFlex system is primarily done via a deployable HyperFlex installer virtual
machine, available for download at cisco.com as an OVA file. The installer VM performs most of the Cisco
UCS configuration work, and it is used to simplify the installation of ESXi on the HyperFlex hosts. The
HyperFlex installer VM also installs the HyperFlex HX Data Platform software and creates the HyperFlex
cluster, while concurrently performing many of the ESXi configuration tasks automatically. Because this
simplified installation method has been developed by Cisco, this CVD will not give detailed manual steps for
the configuration of all the elements that are handled by the installer. The following sections will guide you
through the prerequisites and manual steps needed prior to using the HyperFlex installer, how to utilize the
HyperFlex Installer, and finally how to perform the remaining post-installation tasks.

Prerequisites
Prior to beginning the installation activities, it is important to gather the following information:

IP Addressing
IP addresses for the Cisco HyperFlex system need to be allocated from the appropriate subnets and VLANs
to be used. IP addresses that are used by the system fall into the following groups:

Cisco UCS Manager: These addresses are used and assigned by Cisco UCS manager. Three IP
addresses are used by Cisco UCS Manager, one address is assigned to each Cisco UCS Fabric
Interconnect, and the third IP address is a roaming address for managing the active FI of the Cisco
UCS cluster. In addition, at least one IP address per Cisco UCS blade or HX-series rack-mount
server is required for the default ext-mgmt IP address pool, which is assigned to the CIMC interface
of the physical servers. Since these management addresses are assigned from a pool, they need to
be provided in a contiguous block of addresses. These addresses must all be in the same subnet.

HyperFlex and ESXi Management: These addresses are used to manage the ESXi hypervisor hosts,
and the HyperFlex Storage Platform Controller VMs. Two IP addresses per node in the HyperFlex
cluster are required, and a single additional IP address is needed as the roaming HyperFlex cluster
management interface from the same subnet. These addresses can be assigned from the same
subnet at the Cisco UCS Manager addresses, or they may be separate.

HyperFlex Storage: These addresses are used by the HyperFlex Storage Platform Controller VMs, as
vmkernel interfaces on the ESXi hypervisor hosts, for sending and receiving data to/from the HX
Distributed Data Platform Filesystem. Two IP addresses per node in the HyperFlex cluster are
required, and a single additional IP address is needed as the roaming HyperFlex cluster storage
interface from the same subnet. It is recommended to provision a subnet that is not used in the
network for other purposes, and it is also possible to use non-routable IP address ranges for these
interfaces. Finally, if the Cisco UCS domain is going to contain multiple HyperFlex clusters, it is
possible to use a different subnet and VLAN ID for the HyperFlex storage traffic for each cluster. This
is a safer method, guaranteeing that storage traffic from multiple clusters cannot intermix.

VMotion: These IP addresses are used by the ESXi hypervisor hosts as vmkernel interfaces to enable
vMotion capabilities. One or more IP addresses per node in the HyperFlex cluster are required from
the same subnet. Multiple addresses and vmkernel interfaces can be used if you wish to enable
multi-nic vMotion.

The following tables will assist with gathering the required IP addresses for the installation of an 8 node
standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the addresses required, and an example
configuration that will be used during the setup steps in this CVD:

Table 31 HyperFlex Cluster IP Addressing

UCS Manage-
Address Group: ment HyperFlex and ESXi Management HyperFlex Storage VMotion

VLAN ID:

Subnet:

Subnet Mask:

Gateway:

Storage Con- ESXi Hypervi-

UCS Manage- troller Man- sor Storage Storage Con- VMotion
ment Address- ESXi Manage- agement Inter- vmkernel In- troller Storage vmkernel Inter-
Device es ment Interface face terface Interface face

Fabric Interconnect
A

Fabric Interconnect
B

UCS Manager

HyperFlex Cluster

HyperFlex Node #1

HyperFlex Node #2

HyperFlex Node #3

HyperFlex Node #4

HyperFlex Node #5

HyperFlex Node #6

HyperFlex Node #7

HyperFlex Node #8
Table 32 HyperFlex Hybrid Cluster IP Addressing
UCS Manage-
Address Group: ment HyperFlex and ESXi Management HyperFlex Storage VMotion

VLAN ID:

Subnet:

Subnet Mask:

Gateway:

Storage Con- ESXi Hypervi-

UCS Manage- troller Man- sor Storage Storage Con- VMotion
ment Address- ESXi Manage- agement Inter- vmkernel In- troller Storage vmkernel Inter-
Device es ment Interface face terface Interface face

Fabric Interconnect
A

Fabric Interconnect
B

UCS Manager

HyperFlex Cluster

HyperFlex Node #1

HyperFlex Node #2

HyperFlex Node #3

HyperFlex Node #4

Blade #1

Blade #2

Blade #3

Blade #4

Table 33 HyperFlex Cluster Example IP Addressing

UCS Manage-
Address Group: ment HyperFlex and ESXi Management HyperFlex Storage VMotion

VLAN ID: 133 10 51 200

Subnet: 10.29.133.0 192.168.10.0 192.168.51.0 192.168.200.0

Subnet Mask: 255.255.255.0 255.255.255.0 255.255.255.0 255.255.255.0

Gateway: 10.29.133.1 192.168.10.1 192.168.51.1

Storage Con- ESXi Hypervi-
UCS Manage- troller Man- sor Storage Storage Con- VMotion
ment Address- ESXi Manage- agement Inter- vmkernel In- troller Storage vmkernel Inter-
Device es ment Interface face terface Interface face

Fabric Interconnect
A 10.29.133.104

Fabric Interconnect
B 10.29.133.105

UCS Manager 10.29.133.106

HyperFlex Cluster 192.168.10.50 192.168.51.20

HyperFlex Node #1 10.29.133.111 192.168.10.11 192.168.10.51 192.168.51.11 192.168.51.21 192.168.200.11

HyperFlex Node #2 10.29.133.112 192.168.10.12 192.168.10.52 192.168.51.12 192.168.51.22 192.168.200.12

HyperFlex Node #3 10.29.133.113 192.168.10.13 192.168.10.53 192.168.51.13 192.168.51.23 192.168.200.13

HyperFlex Node #4 10.29.133.114 192.168.10.14 192.168.10.54 192.168.51.14 192.168.51.24 192.168.200.14

HyperFlex Node #5 10.29.133.115 192.168.10.15 192.168.10.55 192.168.51.15 192.168.51.25 192.168.200.15

HyperFlex Node #6 10.29.133.116 192.168.10.16 192.168.10.56 192.168.51.16 192.168.51.26 192.168.200.16

HyperFlex Node #7 10.29.133.117 192.168.10.17 192.168.10.57 192.168.51.17 192.168.51.27 192.168.200.17

HyperFlex Node #8 10.29.133.118 192.168.10.18 192.168.10.58 192.168.51.18 192.168.51.28 192.168.200.18

Table cells shaded in black do not require an IP address.

The Cisco UCS Management, and HyperFlex and ESXi Management IP addresses can come from the same
subnet, or be separate, as long as the HyperFlex installer can reach them both.

DHCP
By default, the ESXi hypervisor installation is configured to use Dynamic Host Configuration Protocol (DHCP)
for automatic IP address assignment to the default ESXi management interfaces. In many environments,
these management interfaces are configured manually, and this CVD will document that procedure.
However, if it is preferred to use DHCP, you may do so following these guidelines:

Configure a DHCP server with the appropriate scope for the subnet containing the ESXi hypervisor
management interfaces.

Configure a listening interface of the DHCP server in the same subnet as the ESXi hypervisor
management interfaces, or configure a DHCP relay agent on the
requests to the remote server.

Configure a DHCP reservation for each ESXi hypervisor host, so that each host will always be offered
the same IP address lease. The MAC addresses for the ESXi hypervisor management vNICs can be
found in Cisco UCS Manager, and exported to a file to ease the task of creating the reservations.
Figure 43 Export MAC Addresses

An example PowerShell script for creating the DHCP reservations from the Cisco UCS Manager CSV export
file on a Windows DHCP server is included in the appendix: Create DHCP Reservations

DNS
DNS servers must be available to query in the HyperFlex and ESXi Management group. DNS records need to
be created prior to beginning the installation. At a minimum, it is highly recommended to create records for
agement interfaces. Additional records can be created for the Storage
Controller Management interfaces, ESXi Hypervisor Storage interfaces, and the Storage Controller Storage
interfaces if desired.

The following tables will assist with gathering the required DNS information for the installation of an 8 node
standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the information required, and an example
configuration that will be used during the setup steps in this CVD:

Table 34 DNS Server Information

Item Value

DNS Server #1

DNS Server #2

DNS Domain

vCenter Server Name

SMTP Server Name

Cisco UCS Domain Name

HX Server #1 Name

HX Server #2 Name

HX Server #3 Name

HX Server #4 Name

HX Server #5 Name

HX Server #6 Name
HX Server #7 Name

HX Server #8 Name

Table 35 DNS Server Example Information

Item Value

DNS Server #1 192.168.0.2

DNS Server #2

DNS Domain hx.lab.cisco.com

vCenter Server Name vcenter1.hx.lab.cisco.com

SMTP Server Name 192.168.0.4

Cisco UCS Domain Name SJC02-151-AAK29

HX Server #1 Name hx220-01.hx.lab.cisco.com

HX Server #2 Name hx220-02.hx.lab.cisco.com

HX Server #3 Name hx220-03.hx.lab.cisco.com

HX Server #4 Name hx220-04.hx.lab.cisco.com

HX Server #5 Name hx220-05.hx.lab.cisco.com

HX Server #6 Name hx220-06.hx.lab.cisco.com

HX Server #7 Name hx220-07.hx.lab.cisco.com

HX Server #8 Name hx220-08.hx.lab.cisco.com

NTP
Consistent time is required across the components of the HyperFlex system, provided by reliable NTP
servers, accessible for synchronization in the Cisco UCS Management network group, and the HyperFlex and
ESXi Management group. NTP is used by Cisco UCS Manager, the ESXi hypervisor hosts, and the HyperFlex
Storage Platform Controller VMs.

The following tables will assist with gathering the required NTP information for the installation of an 8 node
standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the information required, and an example
configuration that will be used during the setup steps in this CVD:

Table 36 NTP Server Information

Item Value

NTP Server #1

NTP Server #2

Timezone
Table 37 NTP Server Example Information
Item Value

NTP Server #1 192.168.0.4

NTP Server #2

Timezone (UTC-8:00) Pacific Time

VLANs
Prior to the installation, the required VLAN IDs need to be documented, and created in the upstream network
if necessary. At a minimum there are 4 VLANs that need to be trunked to the Cisco UCS Fabric Interconnects
that comprise the HyperFlex system; a VLAN for the HyperFlex and ESXi Management group, a VLAN for the
HyperFlex Storage group, a VLAN for the VMotion group, and at least one VLAN for the guest VM traffic. The
VLAN IDs must be supplied during the HyperFlex Cisco UCS configuration step, and the VLAN names can
optionally be customized.

The following tables will assist with gathering the required VLAN information for the installation of an 8 node
standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the information required, and an example
configuration that will be used during the setup steps in this CVD:

Table 38 VLAN Information

Name ID

<<hx-inband-mgmt>>

<<hx-storage-data>>

<<vm-network>>

<<hx-vmotion>>

Table 39 VLAN Example Information

Name ID

hx-inband-mgmt 10

hx-storage-data 51

vm-prod-100 100

hx-vmotion 200

Network Uplinks
The Cisco UCS uplink connectivity design needs to be finalized prior to beginning the installation. One of the
early manual tasks to be completed is to configure the Cisco UCS network uplinks and verify their operation,
prior to beginning the HyperFlex installation steps. Refer to the network uplink design possibilities in the
Network Design section.

The following tables will assist with gathering the required network uplink information for the installation of
an 8 node standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the information required, and an
example configuration that will be used during the setup steps in this CVD:
Table 40 Network Uplink Configuration
Table 41 Network Uplink Example Configuration

Usernames and Passwords

Several usernames and passwords need to be defined or known as part of the HyperFlex installation
process. The following tables will assist with gathering the required username and password information for
the installation of an 8 node standard HyperFlex cluster, or a 4+4 hybrid cluster, by listing the information
required, and an example configuration that will be used during the setup steps in this CVD:

Table 42 Usernames and Passwords

Account Username Password

Cisco UCS Administrator admin <<ucs_admin_pw>>

ESXi Administrator root <<esxi_root_pw>>

HyperFlex Administrator admin <<hx_admin_pw>>

VCenter Administrator <<vcenter_administrator>> <<vcenter_admin_pw>>

Table 43 Example Usernames and Passwords

Account Username Password

Cisco UCS Administrator admin 1q2w3e4r

ESXi Administrator root Cisco123

HyperFlex Administrator admin Cisco123

VCenter Administrator [email protected] !Q2w3e4r

Physical Installation
Install the Fabric Interconnects, the HX-Series rack-mount servers, the Cisco UCS 5108 chassis, the Cisco
UCS 2204XP Fabric Extenders, and the Cisco UCS B200 M4 blades according to their corresponding
hardware installation guides:

Cisco UCS 6248UP Fabric Interconnect:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/6200-install-guide/6200_HIG.pdf

HX220c-M4S Server:
http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX220c_M4/HX220c.html

HX240c-M4SX Server:
http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX240c_M4/HX240c.html

Cisco UCS 5108 Chassis, Servers and Fabric Extenders:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-
guide/ucs5108_install.pdf

Cabling
The physical layout of the HyperFlex system was previously described in section Physical Topology. The
Fabric Interconnects, HX-series rack-mount servers, Cisco UCS chassis and blades need to be cabled
properly before beginning the installation activities.

The following table provides a cabling example for installation of a Cisco HyperFlex system, with eight
HX240c-M4SX servers, and one Cisco UCS 5108 chassis.

Table 44 Example Cabling

Device Port Connected To Port Type Length Note

Cisco UCS6248-A L1 Cisco UCS6248-B L1 CAT5 1FT

Cisco UCS6248-A L2 Cisco UCS6248-B L2 CAT5 1FT

Cisco UCS6248-A mgmt0 Customer Network

Cisco UCS6248-A 1/1 HX240c-M4SX #1 mLOM port 1 Twinax 3M Server 1

Cisco UCS6248-A 1/2 HX240c-M4SX #2 mLOM port 1 Twinax 3M Server 2

Cisco UCS6248-A 1/3 HX240c-M4SX #3 mLOM port 1 Twinax 3M Server 3

Cisco UCS6248-A 1/4 HX240c-M4SX #4 mLOM port 1 Twinax 3M Server 4

Cisco UCS6248-A 1/5 HX240c-M4SX #5 mLOM port 1 Twinax 3M Server 5

Cisco UCS6248-A 1/6 HX240c-M4SX #6 mLOM port 1 Twinax 3M Server 6

Cisco UCS6248-A 1/7 HX240c-M4SX #7 mLOM port 1 Twinax 3M Server 7

Cisco UCS6248-A 1/8 HX240c-M4SX #8 mLOM port 1 Twinax 3M Server 8

Cisco UCS6248-A 1/9 2204XP #1 IOM1 port 1 Twinax 3M Chassis 1

Cisco UCS6248-A 1/10 2204XP #1 IOM1 port 2 Twinax 3M Chassis 1

Cisco UCS6248-A 1/11 2204XP #1 IOM1 port 3 Twinax 3M Chassis 1

Cisco UCS6248-A 1/12 2204XP #1 IOM1 port 4 Twinax 3M Chassis 1

Cisco UCS6248-A 1/13

Cisco UCS6248-A 1/14

Cisco UCS6248-A 1/15

Cisco UCS6248-A 1/16

Cisco UCS6248-A 1/17

Cisco UCS6248-A 1/18

Cisco UCS6248-A 1/19

Cisco UCS6248-A 1/20

Cisco UCS6248-A 1/21

Cisco UCS6248-A 1/22

Cisco UCS6248-A 1/23

Cisco UCS6248-A 1/24

Cisco UCS6248-A 1/25

Cisco UCS6248-A 1/26

Cisco UCS6248-A 1/27

Cisco UCS6248-A 1/28

Cisco UCS6248-A 1/29

Cisco UCS6248-A 1/30 LAN uplink

Cisco UCS6248-A 1/31

Cisco UCS6248-A 1/32 LAN uplink

Device Port Connected To Port Type Length Note

Cisco UCS6248-B L1 Cisco UCS6248-A L1 CAT5 1FT

Cisco UCS6248-B L2 Cisco UCS6248-A L2 CAT5 1FT

Cisco UCS6248-B mgmt0 Customer Network

Cisco UCS6248-B 1/1 HX240c-M4SX #1 mLOM port 2 Twinax 3M Server 1

Cisco UCS6248-B 1/2 HX240c-M4SX #2 mLOM port 2 Twinax 3M Server 2

Cisco UCS6248-B 1/3 HX240c-M4SX #3 mLOM port 2 Twinax 3M Server 3

Cisco UCS6248-B 1/4 HX240c-M4SX #4 mLOM port 2 Twinax 3M Server 4

Cisco UCS6248-B 1/5 HX240c-M4SX #5 mLOM port 2 Twinax 3M Server 5

Cisco UCS6248-B 1/6 HX240c-M4SX #6 mLOM port 2 Twinax 3M Server 6

Cisco UCS6248-B 1/7 HX240c-M4SX #7 mLOM port 2 Twinax 3M Server 7

Cisco UCS6248-B 1/8 HX240c-M4SX #8 mLOM port 2 Twinax 3M Server 8

Cisco UCS6248-B 1/9 2204XP #1 IOM2 port 1 Twinax 3M Chassis 1

Cisco UCS6248-B 1/10 2204XP #1 IOM2 port 2 Twinax 3M Chassis 1

Cisco UCS6248-B 1/11 2204XP #1 IOM2 port 3 Twinax 3M Chassis 1

Cisco UCS6248-B 1/12 2204XP #1 IOM2 port 4 Twinax 3M Chassis 1

Cisco UCS6248-B 1/13

Cisco UCS6248-B 1/14

Cisco UCS6248-B 1/15

Cisco UCS6248-B 1/16

Cisco UCS6248-B 1/17

Cisco UCS6248-B 1/18

Cisco UCS6248-B 1/19

Cisco UCS6248-B 1/20

Cisco UCS6248-B 1/21

Cisco UCS6248-B 1/22

Cisco UCS6248-B 1/23

Cisco UCS6248-B 1/24

Cisco UCS6248-B 1/25

Cisco UCS6248-B 1/26

Cisco UCS6248-B 1/27

Cisco UCS6248-B 1/28

Cisco UCS6248-B 1/29

Cisco UCS6248-B 1/30 LAN uplink

Cisco UCS6248-B 1/31

Cisco UCS6248-B 1/32 LAN uplink

Cisco Nexus Switch Configuration
A Cisco HyperFlex system can be installed in any data center network connected to upstream 10 Gigabit
Ethernet switches. As such, this document does not provide a detailed configuration for the uplink switches,
however example configurations for Cisco Nexus 9372 switches using vPC are provided in appendix C:
Example Nexus 9372 Switch Configurations.

Cisco UCS Installation

This section describes the steps to initialize and configure the Cisco UCS Fabric Interconnects, to prepare
them for the HyperFlex installation.

Cisco UCS 6248UP Fabric Interconnect A

To configure Fabric Interconnect A, complete the following steps:

1. Ensure the Fabric Interconnect cabling is properly connected, including the L1 and L2 cluster links,
and power the Fabric Interconnects on by inserting the power cords.

2. Connect to the console port on the first Fabric Interconnect, which will be designated as the A fabric
device. Use the supplied Cisco console cable (CAB-CONSOLE-RJ45=), and connect it to a built-in
DB9 serial port, or use a USB to DB9 serial port adapter.

3. Start your terminal emulator software.

4.
the terminal emulation to VT100, and the settings to 9600 baud, 8 data bits, no parity, and 1 stop bit.

5. Open the connection just created. You may have to press ENTER to see the first prompt.

6. Configure the first Fabric Interconnect, using the following example as a guideline:

---- Basic System Configuration Dialog ----

This setup utility will guide you through the basic configuration of

the system. Only minimal configuration including IP connectivity to

the Fabric interconnect and its clustering mode is performed through these
steps.

Type Ctrl-C at any time to abort configuration and reboot system.

To back track or make modifications to already entered values,

complete input till end of section and answer no when prompted

to apply configuration.

Enter the configuration method. (console/gui) ? console

Enter the setup mode; setup newly or restore from backup. (setup/restore) ?
setup
You have chosen to setup a new Fabric interconnect. Continue? (y/n): y

Enforce strong password? (y/n) [y]: y

Enter the password for "admin":

Confirm the password for "admin":

Is this Fabric interconnect part of a cluster(select 'no' for standalone)?

(yes/no) [n]: yes

Enter the switch fabric (A/B) []: A

Enter the system name: SJC02-151-AAK29

Physical Switch Mgmt0 IP address : 10.29.133.104

Physical Switch Mgmt0 IPv4 netmask : 255.255.255.0

IPv4 address of the default gateway : 10.29.133.1

Cluster IPv4 address : 10.29.133.106

Configure the DNS Server IP address? (yes/no) [n]: yes

DNS IP address : 171.70.168.183

Configure the default domain name? (yes/no) [n]: yes

Default domain name : hx.lab.cisco.com

Join centralized management environment (UCS Central)? (yes/no) [n]: no

Following configurations will be applied:

Switch Fabric=A

System Name=SJC02-151-AAK29

Enforced Strong Password=no

Physical Switch Mgmt0 IP Address=10.29.133.104

Physical Switch Mgmt0 IP Netmask=255.255.255.0

Default Gateway=10.29.133.1

Ipv6 value=0

DNS Server=171.70.168.183

Domain Name=hx.lab.cisco.com

Cluster Enabled=yes

Cluster IP Address=10.29.133.106

NOTE: Cluster IP will be configured only after both Fabric Interconnects are
initialized
Apply and save the configuration (select 'no' if you want to re-enter)?
(yes/no): yes

Applying configuration. Please wait.

Configuration file - Ok

Cisco UCS 6248UP Fabric Interconnect B

To configure Fabric Interconnect B, complete the following steps:

1. Connect to the console port on the first Fabric Interconnect, which will be designated as the B fabric
device. Use the supplied Cisco console cable (CAB-CONSOLE-RJ45=), and connect it to a built-in
DB9 serial port, or use a USB to DB9 serial port adapter.

2. Start your terminal emulator software.

3.
the terminal emulation to VT100, and the settings to 9600 baud, 8 data bits, no parity, and 1 stop bit.

4. Open the connection just created. You may have to press ENTER to see the first prompt.

5. Configure the second Fabric Interconnect, using the following example as a guideline:

---- Basic System Configuration Dialog ----

This setup utility will guide you through the basic configuration of

the system. Only minimal configuration including IP connectivity to

the Fabric interconnect and its clustering mode is performed through these
steps.

Type Ctrl-C at any time to abort configuration and reboot system.

To back track or make modifications to already entered values,

complete input till end of section and answer no when prompted

to apply configuration.

Enter the configuration method. (console/gui) ? console

Installer has detected the presence of a peer Fabric interconnect. This Fabric
interconnect will be added to the cluster. Continue (y/n) ? y

Enter the admin password of the peer Fabric interconnect:

Connecting to peer Fabric interconnect... done

Retrieving config from peer Fabric interconnect... done

Peer Fabric interconnect Mgmt0 IPv4 Address: 10.29.133.104

Peer Fabric interconnect Mgmt0 IPv4 Netmask: 255.255.255.0

Cluster IPv4 address : 10.29.133.106

Peer FI is IPv4 Cluster enabled. Please Provide Local Fabric Interconnect
Mgmt0 IPv4 Address

Physical Switch Mgmt0 IP address : 10.29.133.105

Apply and save the configuration (select 'no' if you want to re-enter)?
(yes/no): yes

Applying configuration. Please wait.

Configuration file – Ok

Cisco UCS Manager

Log in to the Cisco UCS Manager environment by completing the following steps:

1. Open a web browser and navigate to the Cisco UCS Manager Cluster IP address, For example,
https://10.29.133.106

2. Click the Launch UCS Manager button to download and run the Cisco UCS Manager Java web ap-
plet.

3. If any security prompts appear, click Continue to download the Cisco UCS Manager JNLP file.

4. If prompted to run the downloaded java application, click Run.

5. If prompted to accept any SSL security certificates, click Accept.

6. At th
entered during the initial console configuration.

7. Click No when prompted to enable Cisco Smart Call Home, this feature can be enabled at a later
time.
Figure 44 Cisco UCS Manager

Cisco UCS Configuration

Configure the following ports, settings and policies in the Cisco UCS Manager interface prior to beginning
the HyperFlex installation.

Cisco UCS Firmware

As documented in the Software Components section, this document is based on Cisco UCS infrastructure,
B-series bundle, and C-Series bundle software versions 2.2(6f). If the firmware version of the Fabric
Interconnects differs from this, the version must be upgraded or downgraded to match the requirements
prior to completing any further steps. To upgrade the Cisco UCS Manager version, the Fabric Interconnect
firmware, and the server bundles to version 2.2(6f), refer to the instructions at:
http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/firmware-mgmt/gui/2-
2/b_GUI_Firmware_Management_22.pdf

To support Intel E5-26xx v4 processors, the Cisco UCS Infrastructure software, B-Series and C-Series bun-
dles must be upgraded to revision 2.2(7c).

NTP
To synchronize the Cisco UCS environment time to the NTP server, complete the following steps:

1. In Cisco UCS Manager, click the Admin tab in the navigation pane. 


2. In the navigation pane, select All > Timezone Management. 


3. In the Properties pane, select the appropriate time zone in the Timezone menu. 



4. Click Add NTP Server. 


5. Enter the NTP server IP address and click OK. 


6. Click OK.
7. Click Save Changes, and then click OK.
Figure 45 NTP Settings

Uplink Ports
The Ethernet ports of a Cisco UCS 6248UP Fabric Interconnect are all capable of performing several
functions, such as network uplinks or server ports, and more. By default, all ports are unconfigured, and their
function must be defined by the administrator. To define the specified ports to be used as network uplinks to
the upstream network, complete the following steps:

1. In Cisco UCS Manager, click the Equipment tab in the navigation pane.

2. Select Fabric Interconnects > Fabric Interconnect A > Fixed Module or Expansion Module 2 as
appropriate > Ethernet Ports

3. Select the ports that are to be uplink ports, right click them, and click Configure as Uplink Port.

4. Click Yes to confirm the configuration, and click OK.

5. Select Fabric Interconnects > Fabric Interconnect B > Fixed Module or Expansion Module 2 as
appropriate > Ethernet Ports

6. Select the ports that are to be uplink ports, right click them, and click Configure as Uplink Port.

7. Click Yes to confirm the configuration, and click OK.

8. Verify all the necessary ports are now configured as uplink ports, their role will be listed as network.
Figure 46 Cisco UCS Uplink Ports

Uplink Port Channels

If the Cisco UCS uplinks from one Fabric Interconnect are to be combined into a port channel or vPC, you
must separately configure the port channels using the previously configured uplink ports. To configure the
necessary port channels in the Cisco UCS environment, complete the following steps:

1. In Cisco UCS Manager, click the LAN tab in the navigation pane. 



2. Under LAN > LAN Cloud, click to expand the Fabric A tree. 


3. Right-click Port Channels underneath Fabric A and click Create Port Channel. 


4. Enter the port channel ID number as the unique ID of the port channel. 


5. Enter the name of the port channel. 


6. Click Next.

7. Click on each port from Fabric Interconnect A that will participate in the port channel, and click the
>> button to add them to the port channel.

8. Click Finish.

9. Click OK.

10. Under LAN > LAN Cloud, click to expand the Fabric B tree. 


11. Right-click Port Channels underneath Fabric B and click Create Port Channel. 


12. Enter the port channel ID number as the unique ID of the port channel. 


13. Enter the name of the port channel. 


14. Click Next.

15. Click on each port from Fabric Interconnect B that will participate in the port channel, and click the
>> button to add them to the port channel.

16. Click Finish.

17. Click OK.


18. Verify the necessary port channels have been created. It can take a few minutes for the newly
formed port channels to converge and come online.
Figure 47 Cisco UCS Uplink Port Channels

Chassis Discovery Policy

If the Cisco HyperFlex system will use B200 M4 blades as compute-only nodes in a hybrid design, additional
settings must be configured for connecting the Cisco UCS 5108 blade chassis. The Chassis Discovery policy
defines the number of links between the Fabric Interconnect and the Cisco UCS 2204XP Fabric Extenders
which must be connected and active, before the chassis will be discovered. This also effectively defines how
many of those connected links will be used for communication. The Link Grouping Preference setting
specifies if the links will operate independently, or if Cisco UCS Manager will automatically combine them
into port-channels. Cisco best practices recommends using link grouping, and 4 links per side per Cisco
UCS 5108 chassis. To configure the necessary policy and setting, complete the following steps:

1. In Cisco UCS Manager, click the Equipment tab in the navigation pane, and click on Equipment in
the top of the navigation tree on the left. 


2. In the properties pane, click Policies tab. 


3. Click the Global Policies sub-tab, set the Chassis/FEX Discovery Policy to match the number of up-
link ports that
are cabled per side, between the chassis and the Fabric Interconnects. 


4. Set the Link Grouping Preference option to Port Channel. 


5. Click Save Changes. 


6. Click OK.
Figure 48 Cisco UCS Chassis Discovery Policy

Server Ports
The Ethernet ports of a Cisco UCS Fabric Interconnect connected to the rack-mount servers, or to the blade
chassis must be defined as server ports. Once a server port is activated, the connected server or chassis will
begin the discovery process shortly afterwards. Rack-mount servers and blade chassis are automatically
numbered in the order which they are first discovered. For this reason, it is important to configure the server
ports sequentially in the order you wish the physical servers and/or chassis to appear within Cisco UCS
manager. For example, if you installed your servers in a cabinet or rack with server #1 on the bottom,
counting up as you go higher in the cabinet or rack, then you need to enable the server ports to the bottom-
most server first, and enable them one-by-one as you move upward. You must wait until the server appears
in the Equipment tab of Cisco UCS Manager before configuring the ports for the next server. The same
numbering procedure applies to blade server chassis.

To define the specified ports to be used as server ports, complete the following steps:

1. In Cisco UCS Manager, click the Equipment tab in the navigation pane.

2. Select Fabric Interconnects > Fabric Interconnect A > Fixed Module or Expansion Module 2 as
appropriate > Ethernet Ports.

3. Select the first port that is to be a server port, right click it, and click Configure as Server Port.

4. Click Yes to confirm the configuration, and click OK.

5. Select Fabric Interconnects > Fabric Interconnect B > Fixed Module or Expansion Module 2 as
appropriate > Ethernet Ports.

6. Select the matching port as chosen for Fabric Interconnect A that is to be a server port, right click it,
and click Configure as Server Port.

7. Click Yes to confirm the configuration, and click OK.

8. Wait for a brief period, until the rack-mount server appears in the Equipment tab underneath Equip-
ment > Rack Mounts > Servers, or the chassis appears underneath Equipment > Chassis.

9. Repeat Steps 2-8 for each server port, until all rack-mount servers and chassis appear in the order
desired in the Equipment tab.
Figure 49 Cisco UCS Server Ports

Server Discovery
As previously described, once the server ports of the Fabric Interconnects are configured and active, the
servers connected to those ports will begin a discovery process. During
hardware inventories are collected, along with their current firmware revisions. Before continuing with the
HyperFlex installation processes, which will create the service profiles and associate them with the servers,
wait for all of the servers to finish their discovery process and to show as unassociated servers that are

1. In Cisco UCS Manager, click the Equipment tab in the navigation pane, and click on Equipment in
the top of the navigation tree on the left. 


2. In the properties pane, click Servers tab. 


3. Click the Blade Servers or Rack-Mount Servers sub- a-

tus in the Overall Status column.
Figure 50 Cisco UCS Server Discovery

HyperFlex Installer Deployment

The Cisco HyperFlex software is distributed as a deployable virtual machine, contained in an Open Virtual
Appliance (OVA) file format. The HyperFlex OVA file is available for download at cisco.com:
https://software.cisco.com/portal/pub/download/portal/select.html?&mdfid=286305544&flowid=79522&soft
wareid=286305994

This document is based on the Cisco HyperFlex 1.7.1 release filename: Cisco-HX-Data-Platform-
Installer-v1.7.1-14786.ova

The HyperFlex installer OVA file can be deployed as a virtual machine in an existing VMware vSphere
environment, VMware Workstation, VMware Fusion, or other virtualization environment which supports the
import of OVA format files. For the purpose of this document, the process describes uses an existing ESXi
server to run the HyperFlex installer OVA, and deploying it via the VMware vSphere (thick) client.

Installer Connectivity
The Cisco HyperFlex Installer VM must be deployed in a location that has connectivity to the following
network locations and services:

Connectivity to the vCenter Server or vCenter Appliance which will manage the HyperFlex cluster(s)
to be installed.

Connectivity to the management interfaces of the Fabric Interconnects that contain the HyperFlex
cluster(s) to be installed.

Connectivity to the management interface of the ESXi hypervisor hosts which will host the HyperFlex
cluster(s) to be installed.

Connectivity to the DNS server(s) which will resolve host names used by the HyperFlex cluster(s) to
be installed.

Connectivity to the NTP server(s) which will synchronize time for the HyperFlex cluster(s) to be
installed.
Connectivity from the staff operating the installer to the webpage hosted by the installer, and to log
in to the installer via SSH.

If the network where the HyperFlex installer VM is deployed has DHCP services available to assign the
proper IP address, subnet mask, default gateway, and DNS servers, the HyperFlex installer can be deployed
using DHCP. If a static address must be defined, use the following table to document the settings to be used
for the HyperFlex installer VM:

Table 45 HyperFlex Installer Settings

Setting Value

Hostname

IP Address

Subnet Mask

Default Gateway

DNS Server #1

DNS Server #2

Deploy Installer OVA

To deploy the HyperFlex installer OVA, complete the following steps:

1. Open the vSphere (thick) client, connect and log in to an ESXi host or vCenter server where the in-
staller OVA will be deployed.

2. Click File > Deploy OVF Template.

3. Click Browse and locate the Cisco-HX-Data-Platform-Installer-v1.7.1-14786.ova file, click on

the file and click Open.

4. Click Next twice.

5. Modify the name of the virtual machine to be created if desired, and click on a folder location to
place the virtual machine, and then click Next.

6. Click on a specific host or cluster to locate the virtual machine and click Next.

7. Click on the Thin Provision option and click Next.

8. Modify the network port group selection from the drop down list in the Destination Networks column,
choosing the network the installer VM will communicate on, and click Next.

9. If DHCP is to be used for the installer VM, leave the fields blank and click Next. If static address set-
tings are to be used, fill in the fields for the installer VM hostname, default gateway, DNS server, IP
address, and subnet mask, then click Next. (Figure 51)

10. Check the box to power on after deployment, and click Finish. (Figure 52)
The installer VM will take a few minutes to deploy, once it has deployed and the virtual machine has started,
proceed to the next step.
Figure 51 HyperFlex OVA Deployment Settings
Figure 52 HyperFlex OVA Deployment Summary

HyperFlex Installer Web Page

The HyperFlex installer is accessed via a webpage using your local computer and a web browser. If the
HyperFlex installer was deployed with a static IP address, then the IP address of the website is already
known. If DHCP was used, open the local console of the installer VM. In the console you will see an interface
similar to the example below, showing the IP address that was leased:

Data Platform Installer.

*******************************************

You can start the installation by visiting

the following URL:

http://192.168.10.210

*******************************************

Cisco-HX-Data-Platform-Installer login:

To access the HyperFlex installer webpage, complete the following steps:

1. Open a web browser on the local computer and navigate to the IP address of the installer VM. For
example, open http://192.168.10.210

2. Click Accept or Continue to bypass any SSL certificate errors.

3. At the security prompt, enter the username: admin

4. At the security prompt, enter the password: Cisco123

5. Verify the version of the installer in the lower right-hand corner of the Welcome page is the correct
version, and click Continue.

6. Read and accept the End User Licensing Agreement, and click Continue.
Figure 53 HyperFlex Installer Welcome Page

HyperFlex Cisco UCS Configuration

The first step of the HyperFlex installation is to use the HyperFlex installer webpage to perform the remaining
Cisco UCS configuration tasks. The HyperFlex installer will perform the following tasks within Cisco UCS:

- sub-organization.

Configure QoS system classes.

Create the VLANs.

Create the out-of-band management IP address block.

Set the HX-

Create the required pools and policies used by HyperFlex.

Create the service profile templates and service profiles for the HX-Series servers.

Associate the service profiles with the HX-Series servers.

To configure the Cisco UCS settings, policies, and templates for HyperFlex, complete the following steps:

1. Open the HyperFlex installer webpage (HyperFlex Installer Web Page).

2. Click Configure UCS Manager. (Figure 54)

3. Enter the Cisco UCS manager DNS hostname or IP address, the admin username, and the password,
then click Continue. (Figure 55)
4. On the following page, click the boxes next to the discovered and unassociated HX-Series servers
which will be used to build the HyperFlex cluster(s). (Figure 56)

5. Enter a custom 4th byte for the MAC address pools. MAC addresses can only be entered in hexadec-
imal format (00 to FF). (Figure 57)

6. Click to expand the LAN Configuration section. Enter the desired VLAN names and VLAN IDs for the
four required VLANs. (Figure 58)

7. -
management IP address pool. (Figure 59)

8. Click to expand the Advanced Configuration section. Customize the HyperFlex Cluster Name value
as desired. This text is entered as the User Label in Cisco UCS manager for the service profiles.
(Figure 60)

9. Click Configure.

10. On the following page, monitor the configuration process until it completes. Click Show Details to
see detailed information on the steps being performed. The configuration will take a few minutes.
(Figure 61)
Figure 54 HyperFlex Installer Main Page
Figure 55 HyperFlex Cisco UCS Manager Configuration
Figure 56 Cisco UCS Server Selection
Figure 57 Cisco UCS MAC Address Pool Configuration
Figure 58 Cisco UCS VLAN Configuration
Figure 59 Cisco UCS Management IP Address Configuration
Figure 60 Cisco UCS Advanced Configuration
Figure 61 HyperFlex Cisco UCS Configuration Completed

Verify Cisco UCS Configuration

To verify the HyperFlex installer has properly configured Cisco UCS Manager, complete the following steps:

1. Click Launch UCS Manager from the Cisco UCS Manager Configuration summary page of the Hy-
perFlex installer webpage, or open Cisco UCS manager as documented in Cisco UCS Manager.

2. In Cisco UCS Manager, click Servers tab in the navigation pane.

3. Expand Servers > Service Profiles > root > Sub-Organizations > hx-cluster and verify the service
profiles were configured for the servers you selected in the configuration page during the previous
task. (Figure 62)

4. In Cisco UCS Manager, click the LAN tab in the navigation pane.

5. Click VLANs in the navigation tree, and verify in the configuration pane the VLANs have been created
with the names and IDs specified in the previous task. (Figure 63)
Figure 62 Cisco UCS Service Profiles
Figure 63 Cisco UCS VLANs

Tagged VLANs Alternate Configuration

The default Cisco UCS configuration uses native, or untagged traffic for the HyperFlex and ESXi
management group, the HyperFlex storage group, and the VMotion network group. In some cases, the
standards set at the customer data center may require that all traffic be tagged with the appropriate 802.1Q
VLAN ID. In order to accommodate that requirement, changes must first be made in the Cisco UCS
configuration, before the VLAN IDs can be properly assigned and used in later configuration steps. To modify
the Cisco UCS settings for tagged VLANs, complete the following steps:

1. In Cisco UCS Manager, click the LAN tab in the navigation pane.

2. Expand LAN > Policies > root > Sub-Organizations > hx-cluster > vNIC-Templates.

3. Click on the vNIC -mgmt-

4. In the configuration pane, click Modify VLANs.

5. In the Modify VLANs window, click the radio button in the Native VLAN column which is already se-
lected, to clear or unselect that button, and click OK. (Figure 64)

6. Click OK.

7. Repeat steps 3-6 for each vNIC Template that is required to carry tagged traffic. For example, to
have all vNICs carry tagged traffic, the changes must be made to the following vNIC templates:

a. hv-mgmt-a
b. hv-mgmt-b
c. hv-vmotion-a
d. hv-vmotion-b
e. storage-data-a
f. storage-data-b
Figure 64 Modify Cisco UCS Native VLANs

ESXi Hypervisor Installation

The HyperFlex installer VM can be used to build a custom ESXi installation ISO using a kickstart file, which
automates the installation process of ESXi to the HyperFlex nodes. The HyperFlex system requires a Cisco
custom ESXi ISO file to be used, which has Cisco hardware specific drivers pre-installed to ease the
installation process, as detailed in section Software Components. The Cisco custom ESXi ISO file is available
for download at cisco.com:
https://software.cisco.com/portal/pub/download/portal/select.html?&mdfid=286305544&flowid=79522&soft
wareid=286305994

This document is based on the Cisco custom ESXi 6.0 Update 1B ISO release filename: Vmware-ESXi-
6.0.0-3380124-Custom-Cisco-6.0.1.2.iso

The kickstart process will automatically perform the following tasks with no user interaction required:

Accept the End User License Agreement.

Configure the root password to: Cisco123

Install ESXi to the internal mirrored Cisco FlexFlash SD cards.

Set the default management network to use vmnic0, and obtain an IP address via DHCP.

Enable SSH access to the ESXi host.

Enable the ESXi shell.

Enable serial port com1 console access to facilitate Serial over LAN access to the host.
Configure the ESXi configuration to always use the current hardware MAC address of the network
interfaces, even if they change.

Rename the default vSwitch0 to vswitch-hx-inband-mgmt.

ESXi Kickstart ISO Creation

To create the custom kickstart ESXi installation ISO file, complete the following steps:

1. Copy the base ISO file, Vmware-ESXi-6.0.0-3380124-Custom-Cisco-6.0.1.2.iso to the Hyper-

Flex installer VM using SCP, SFTP or any available method. Place the file in the
/var/hyperflex/source-images folder.

2. Login to the HyperFlex installer VM via SSH, using Putty or a similar tool. Username is: root Pass-
word is: Cisco123

3. Enter the following commands to integrate the kickstart file with the base ISO file, and create a new
ISO file from them:

# cd /opt/springpath/install-esxi

# ./makeiso.sh /opt/springpath/install-esxi/cisco-hx-esxi-6.0.ks.cfg \

/var/hyperflex/source-images/Vmware-ESXi-6.0.0-3380124-Custom-Cisco-6.0.1.2.iso \

/var/www/localhost/images/HX-Vmware-ESXi-6.0.0-3380124-Custom-Cisco-6.0.1.2.iso

4. You will see output from the command as below:

mount: block device /var/hyperflex/source-images/Vmware-ESXi-6.0.0-3380124-

Custom-Cisco-6.0.1.2.iso is write-protected, mounting read-only

Warning: creating filesystem that does not conform to ISO-9660.

I: -input-charset not specified, using utf-8 (detected in locale settings)

Size of boot image is 4 sectors -> No emulation

2.78% done, estimate finish Fri May 27 22:32:00 2016

5.54% done, estimate finish Fri May 27 22:32:00 2016

8.31% done, estimate finish Fri May 27 22:32:00 2016

11.08% done, estimate finish Fri May 27 22:32:00 2016

13.85% done, estimate finish Fri May 27 22:32:00 2016

16.62% done, estimate finish Fri May 27 22:32:00 2016

19.39% done, estimate finish Fri May 27 22:32:00 2016

22.16% done, estimate finish Fri May 27 22:32:00 2016

24.93% done, estimate finish Fri May 27 22:32:00 2016

27.70% done, estimate finish Fri May 27 22:32:00 2016

30.47% done, estimate finish Fri May 27 22:32:00 2016

33.24% done, estimate finish Fri May 27 22:32:00 2016

36.02% done, estimate finish Fri May 27 22:32:00 2016

38.78% done, estimate finish Fri May 27 22:32:00 2016

41.56% done, estimate finish Fri May 27 22:32:00 2016

44.32% done, estimate finish Fri May 27 22:32:00 2016

47.09% done, estimate finish Fri May 27 22:32:00 2016

49.86% done, estimate finish Fri May 27 22:32:00 2016

52.63% done, estimate finish Fri May 27 22:32:00 2016

55.40% done, estimate finish Fri May 27 22:32:00 2016

58.17% done, estimate finish Fri May 27 22:32:00 2016

60.94% done, estimate finish Fri May 27 22:32:00 2016

63.71% done, estimate finish Fri May 27 22:32:00 2016

66.48% done, estimate finish Fri May 27 22:32:00 2016

69.25% done, estimate finish Fri May 27 22:32:00 2016

72.02% done, estimate finish Fri May 27 22:32:00 2016

74.79% done, estimate finish Fri May 27 22:32:00 2016

77.56% done, estimate finish Fri May 27 22:32:00 2016

80.33% done, estimate finish Fri May 27 22:32:00 2016

83.10% done, estimate finish Fri May 27 22:32:00 2016

85.87% done, estimate finish Fri May 27 22:32:00 2016

88.64% done, estimate finish Fri May 27 22:32:00 2016

91.41% done, estimate finish Fri May 27 22:32:00 2016

94.18% done, estimate finish Fri May 27 22:32:00 2016

96.95% done, estimate finish Fri May 27 22:32:00 2016

99.72% done, estimate finish Fri May 27 22:32:00 2016

Total translation table size: 2048

Total rockridge attributes bytes: 10943

Total directory bytes: 6144

Path table size(bytes): 50

Max brk space used 22000

180507 extents written (352 MB)

Successfully created ESXi ISO Image /var/www/localhost/images/HX-Vmware-ESXi-

6.0.0-3380124-Custom-Cisco-6.0.1.2.iso
5. Verify the newly created ISO file exists in the proper webpage location, by opening a web browser
and navigating to the IP address of the HyperFlex installer VM, followed by /images. For example,
http://192.168.10.210/images

6. The new file, named HX-Vmware-ESXi-6.0.0-3380124-Custom-Cisco-6.0.1.2.iso will be listed

in the webpage file listing.
Figure 65 Custom ESXi ISO Web Listing

Cisco UCS vMedia and Boot Policies

Using a Cisco UCS vMedia policy, the mounting of the newly created custom kickstart ESXi installation ISO
mount this file,
and the boot policy must be modified temporarily to boot from the remotely mounted vMedia file. Once
these two tasks are completed, the servers can be rebooted, and they will automatically boot from the
remotely mounted vMedia file, installing and configuring ESXi on the servers as described above in ESXi
Hypervisor Installation.

To configure the Cisco UCS vMedia and Boot Policies, complete the following steps:

1. In Cisco UCS Manager, click the Servers tab in the navigation pane.

2. Expand Servers > Policies > root > Sub-Organizations > hx-cluster > vMedia Policies.

3. Click on the vMedia Policy HyperFlex .

4. In the configuration pane, click Create vMedia Mount.

5. Enter a name for the mount. For example, ESX.

6. Select the CDD option.

7. Select HTTP as the protocol.

8. Enter the IP address of the HyperFlex installer VM. For example, 192.168.10.210

9. Select None as the Image Variable Name.

10. Enter HX-Vmware-ESXi-6.0.0-3380124-Custom-Cisco-6.0.1.2.iso as the Remote File.

11. Enter /images/ as the Remote Path. (Figure 66)

12. Click Save Changes, and click OK.

13. Select Servers > Service Profile Templates > root > Sub-Organizations > hx-cluster.

14. Click on the Service Profile Template hx-nodes .

15. In the configuration pane, click the vMedia Policy tab.

16. Click Modify vMedia Policy.

17. Chose the HyperFlex vMedia Policy from the drop down selection, and click OK twice. (Figure 67)

18. Select Servers > Policies > root > Sub-Organizations > hx-cluster > Boot Policies.

19. Boot Policy HyperFlex .

20. In the navigation pane, click to expand the section titled CIMC Mounted vMedia.

21. Click the entry labeled Add CIMC Mounted CD/DVD.

22. Select the CIMC Mounted CD/DVD entry in the Boot Order list, and click the Move Up button until
the CIMC Mounted CD/DVD entry is listed first. (Figure 68)

23. Click Save Changes, and click OK.

Figure 66 Add vMedia Policy Mount
Figure 67 Modify Service Profile Template vMedia Policy
Figure 68 Modify Boot Policy

Install ESXi
To begin the installation after modifying the vMedia policy, Boot policy and service profile template, the
servers need to be rebooted. To monitor the progress of one or more servers, it is advisable to open a
remote KVM console session to watch the installation. To open the KVM console and reboot the servers,
complete the following steps:

1. In Cisco UCS Manager, click the Servers tab in the navigation pane.

2. Expand Servers > Service Profiles > root > Sub-Organizations > hx-cluster.

3. Click on the first se rack-unit-1 .

4. In the configuration pane, click KVM Console.

5. Click Continue to any security alerts that appear. The remote KVM Console window will appear

6. Repeat Steps 2-4 for any additional servers whose console you wish to monitor during the installa-
tion.

7. In Cisco UCS Manager, click the Equipment tab in the navigation pane.

8. Expand Equipment > Rack Mounts > Servers.

9. In the configuration pane, click the first server to be rebooted, then shift+click the last server to be
rebooted, selecting them all.

10. Right-click the mouse, and click Reset. (Figure 69)

11. Click OK.

12. Select Power Cycle and click OK.

13. Click OK.

The servers you are monitoring in the KVM console windows will now immediately reboot, boot from the
remote vMedia mount and install ESXi. There may be error messages seen on screen, but they can be safely
ignored.
Figure 69 Reset Servers
Figure 70 ESXi Boot from Installation ISO

Undo vMedia and Boot Policy Changes

Once all the servers have booted from the remote vMedia file and begun their installation process, the
changes to the boot policy need to be quickly undone, to prevent the servers from going into a boot loop,
constantly booting from the installation ISO file. To revert the boot policy settings, complete the following
steps:

1. Select Servers > Policies > root > Sub-Organizations > hx-cluster > Boot Policies.

2. Boot Policy HyperFlex .

3. Select the CIMC Mounted CD/DVD entry in the Boot Order list, and click Delete.

4. Click Save Changes, and click OK.

The changes made to the vMedia policy and service profile template may also be undone once the ESXi
installations have all completed fully, or they may be left in place for future installation work.

Configure ESXi Management

After the kickstart ESXi installations are completed, the ESXi management network configuration needs to be
set on each server. If the environment has been configured with a DHCP server to provide the IP addresses,
then these steps do not need to be completed. To configure the ESXi management networking settings,
complete the following steps:

1. In Cisco UCS Manager, open the server KVM console of the first ESXi host.
2. Press F2 to login to the ESXi Direct Console User Interface (DCUI), with username: root and pass-
word: Cisco123. (Figure 71)

3. Press the down arrow to select Configure Management Network and press ENTER.

4. If the management interfaces have been configured to carry VLAN tagged traffic, as documented in
Tagged VLANs Alternate Configuration, press the down arrow to select VLAN (optional) and press
ENTER, then continue to step 5. If traffic is untagged, continue to step 6.

5. Enter the VLAN ID number in the field and press ENTER.

6. Press the down arrow to select IPv4 Configuration and press ENTER.

7. Press the down arrow to highlight Set Static IPv4 Address and Network Configuration, and then
press space bar to select it.

8. Press the down arrow, and enter the IP address, Subnet Mask and Default Gateway information in
the fields, and then press OK. (Figure 72)

9. Press the down arrow to select DNS Configuration and press ENTER.

10. Enter the primary DNS server address, secondary DNS server address (if applicable), and the ESXi
elds, then press ENTER. (Figure 73)

11. Press the down arrow to select Custom DNS Suffixes and press ENTER.

12. Enter the preferred DNS suffix, and then press ENTER.

13. Press ESC to exit the network configuration settings, and press Y to accept the changes and restart
the management network.

14. Press the down arrow to select Test Management Network and press ENTER.

15. Press ENTER to test using the values you supplied earlier, all addresses should ping and the DNS
server should resolve the hostname of the ESXi host. If any tests do not work properly, re-check your
values and correct as necessary.

16. Press ENTER to exit the test.

17. Press ESC to log out of the DCUI.

18. Repeat steps 1-17 for each remaining server that will be a part of the HyperFlex cluster.

A sample script exists in the HyperFlex installer VM to configure the ESXi hosts’ management interfaces via
SoL. To use the script, log in to the installer VM via SSH and run /bin/setstaticip_ucs.py
Figure 71 ESXi DCUI
Figure 72 ESXi IP Address Configuration
Figure 73 ESXi DNS Configuration
Figure 74 ESXi Network Test

ESXi Post-Installation Configuration

After the ESXi management network has been configured on each server, there are a few remaining
configuration steps necessary before beginning the HyperFlex installation. These final steps prepare the ESXi
hypervisor hosts with the remaining prerequisite settings, using the vCenter Web Client.

Create vCenter Datacenter and Cluster

In the vCenter server Web Client, create the Datacenter object and vSphere cluster where the HyperFlex
hosts will be placed. To configure the vCenter Datacenter and cluster, complete the following steps:

1. Open the vCenter Web Client webpage and login, For example,
https://vcenter1.hx.lab.cisco.com/vsphere-client/

2. In the home pane, click Hosts and Clusters (Figure 75)

3. In the Navigator pane, right-click on the vCenter server name, and click New Datacenter.

4. Enter the datacenter name and click OK. (Figure 76)

5. In the navigator pane, right-click the datacenter that was just created, and click New Cluster.

6. Enter the cluster name, leave all other settings disabled or turned off, and click OK. (Figure 77)
Figure 75 vCenter Web Client Home Page
Figure 76 vCenter New Datacenter
Figure 77 vCenter New Cluster

Add ESXi Hosts to vCenter

The ESXi hosts must be added to the vCenter server prior to beginning the HyperFlex installation. To
configure the vCenter data center and cluster, complete the following steps:

1. In the vSphere Web Client, right-click the cluster object where the HyperFlex nodes will reside and
click Add Host.

2. Enter the name of the host to be added and click Next. (Figure 78)

3. Enter the root username and password, and click Next. (Figure 79)

4. Click Yes to accept the security certificate.

5. Click Next after reviewing the host summary.

6. Click on a license key to apply to the server, if applicable, and click Next.

7. Leave the lockdown mode option set to disabled, which is the default, and click Next.

8. On the summary page, click Finish.

9. Repeat steps 1-8 for each additional host in the HyperFlex cluster.
Figure 78 vCenter Add Host
Figure 79 vCenter Host Password

Configure NTP
NTP is required for proper operation of the HyperFlex environment. To configure NTP settings on the ESXi
hosts, complete the following steps:

1. In the vSphere Web Client, right-click the ESXi host to be configured, and click Settings.

2. Under the System section, click Time Configuration.

3. Select the Use Network Time Protocol option. (Figure 80)

4. Change the NTP Service Startup Policy to Start and Stop with the Host.

5. Enter the NTP server address(es) in the NTP Servers field.

6. Click on the Start button

7. Click OK.

8. Repeat steps 1-7 for each additional host in the HyperFlex cluster.
Figure 80 vCenter NTP Settings

Maintenance Mode
The ESXi hosts must be placed into maintenance mode prior to beginning the HyperFlex installation. To
place the ESXi hosts into maintenance mode, complete the following steps:

1. In the vSphere Web Client, right-click the ESXi host to be configured, click Maintenance Mode, and
click Enter Maintenance Mode.

2. Click OK.

3. Repeat steps 1-2 for each additional host in the HyperFlex cluster.
Figure 81 vCenter Enter Maintenance Mode

An example script to configure NTP, place the ESXi hosts into maintenance mode, and add them to vCenter is
provided in the appendix: Configure ESXi Hosts

HyperFlex Installation
The installation of the HyperFlex platform software is done via the HyperFlex installer webpage, installed as
documented in section HyperFlex Installer Deployment. To deploy a HyperFlex cluster, complete the
following steps:

1. Open the HyperFlex installer webpage (HyperFlex Installer Web Page).

2. Click the Configure Cluster button.

3. Enter the IP address values or DNS names in the first section. (Figure 82)

a. Click the Add button if more than 4 hosts are being installed, until there are enough lines to ac-
commodate the number of nodes being installed.

b. If IP addresses are entered into the first line, which has a grey background, the lines underneath
it will auto-populate with incremented IP addresses.

The first line with the grey background is not a node that is being configured.

4. Enter the subnet mask and gateway address for the Hypervisor and ESXi management group, and
the HyperFlex storage group.

It is necessary to enter the gateway addresses for both fields, even though they are marked as option-
al.

5. Enter the values in the Cisco HX Cluster section. (Figure 83)

6. Enter the values in the Hypervisor Credentials section.

7. Enter the desired password in the Controller VM section.

8. Enter the values in the vCenter Configuration section.

9. Enter the values in the System Services section.

10. Enter the values in the Auto Support section. (Figure 85)

11. Click the arrow to expand the Advanced Configuration section.

12. Check the box to Clean-Up Disk Partitions.

13. (Optional) If the management and/or storage interfaces have been configured to carry VLAN tagged
traffic, as documented in Tagged VLANs Alternate Configuration, check the box to override the de-
fault network settings, then continue to step 14. If traffic is untagged, continue to step 16.

14. Enter the VLAN IDs for the management and storage networks if you are using tagged traffic, and
optionally change the names of the vSwitches that are created.

15. Click Validate.

16. If the validation checks return any errors, re-check the values entered and correct as necessary, oth-
erwise click Deploy. (Figure 86)

The cluster configuration values entered can be saved in a JSON file format for re-use at a later time by click-
ing the link Save Configuration File.

The HyperFlex installer combines many steps, configuring the ESXi hosts, rebooting them, deploying storage
controller VMs, installing software and creating the cluster. The installation can take over 30 minutes to com-
plete.
Figure 82 HyperFlex Installer IP Addresses
Figure 83 HyperFlex Installer IP Addressing Hints
Figure 84 HyperFlex Installer Configuration
Figure 85 HyperFlex Installer Advanced Configuration
Figure 86 HyperFlex Installer Validation
Figure 87 HyperFlex Installer Deployment Progress
Figure 88 HyperFlex Installer Creating Cluster
Figure 89 HyperFlex Installation Complete

When the HyperFlex installation has completed, continue with the post-installation tasks as documented in
HyperFlex Post Installation Configuration. If a hybrid cluster is being built, continue with the tasks
documented in Hybrid Cluster Expansion.

HyperFlex Cluster Expansion

The process to expand a HyperFlex cluster can be used to grow an existing HyperFlex cluster with additional
converged storage nodes, or to expand an existing cluster with additional compute-only nodes to create a
hybrid cluster. Prior to beginning any expansion, the following requirements must be met:

Verify the storage cluster state is healthy in the HyperFlex Web Client Plugin Summary.


The new node is a server that meets the compute node system requirements listed in Requirements,
including network and disk requirements.

The software and firmware versions on the node match the Cisco HX Data Platform software version
and the vSphere software version Requirements, and match the existing servers.

The new node must be physically installed and discovered as described in Server Discovery.

The new node uses the same network configuration as the other nodes in the storage cluster,
including the VLAN IDs and VLAN tagging. This is accomplished via the use of Cisco UCS service
profiles.

To prevent errors during cluster expansions, it is recommended to disable vSphere High Availability
(HA) during the expansion, and then re-enable the feature after the expansion completes.
Standard Cluster Expansion
Prior to expanding the cluster, the new nodes must be prepared and configured similar to a cluster
installation. There are some specific differences in the processes which will be outlined below. To expand an
existing standard HyperFlex cluster, complete the following steps:

Configure Cisco UCS

1. Open the HyperFlex installer webpage (HyperFlex Installer Web Page).

2. Click Configure UCS Manager.

3. Enter the Cisco UCS manager DNS hostname or IP address, the admin username, and the password,
and click Continue.

4. On the following page, click the boxes next to the discovered and unassociated HX-Series servers
which will be used to expand the HyperFlex cluster(s). (Figure 90)

5. Click Configure.

The appropriate service profile(s) will be created and associated with the new server that will expand the
cluster.
Figure 90 HyperFlex Installer Select Expansion Server

Install ESXi
Install and configure the ESXi hypervisor as documented in the section ESXi Hypervisor Installation. The node
must meet the following requirements before continuing with the HyperFlex cluster expansion:

The new node must be in maintenance mode while configuring the storage cluster.

The new node must be a member in the same vCenter cluster and vCenter datacenter as the existing
nodes.

The new node has SSH enabled.

The new node has at least one valid DNS and NTP server configured.

Expand the Cluster

1. Open the HyperFlex installer webpage.

2. Click Expand Cluster.

3. Enter the IP address values or DNS names in the first section.

a. Click the Add button if more than 1 new host is being added, until there are enough lines to ac-
commodate the number of nodes being added.

b. If IP addresses are entered into the first line, which has a grey background, the lines underneath
it will auto-populate with incremented IP addresses.

The first line with the grey background is not a node that is being configured.

4. Enter the existing HyperFlex cluster management IP address.

5. Enter the values in the Hypervisor Credentials section.

6. Enter the password of the existing cluster Controller VMs.

7. Enter the values in the vCenter Configuration section for the existing vCenter server.

8. Click the arrow to expand the Advanced Configuration section.

9. Check the box to Clean-up Disk Partitions.

10. Click Validate.

11. If the validation checks return any errors, re-check the values entered and correct as necessary, oth-
erwise click Deploy.
Figure 91 HyperFlex Installer Expand Cluster

After the cluster expansion has completed, continue with the HyperFlex Post Installation Configuration steps.

Hybrid Cluster Expansion

Hybrid HyperFlex clusters are built by first installing a standard cluster, then expanding the cluster with
compute-only nodes. Prior to expanding the cluster, the new nodes must be prepared and configured similar
to a cluster installation. There are some specific differences in the processes which will be outlined below.
To expand an existing HyperFlex cluster, creating a hybrid cluster, complete the following steps:

Configure Cisco UCS

1. In Cisco UCS Manager, click the Servers tab in the navigation pane.

2. Expand Servers > Service Profile Templates > root > Sub-Organizations > hx-cluster.

3. Right-click the Service Profile Template compute-nodes and click Create Service Profiles
from Template.

4. Enter the naming prefix of the service profiles that will be created from the template.

5. Enter the starting number of the service profiles being created and the number of service profiles to
be created. (Figure 92)

6. Click OK twice.
7. Expand Servers > Service Profiles > root > Sub-Organizations > hx-cluster.

8. Click on the first service profile created for the additional B200 M4 compute-only nodes, and right
click.

9. Click Change Service Profile Association.

10. In the Server Assignment dropdown list, change the selection to select Existing Server.

11. Select Available Servers.

12. In the list of available servers, chose the B200 M4 blade server to assign the service profile to and
click OK. (Figure 93)

13. Click Yes to accept that the server will reboot to apply the service profile.

14. Click OK.

15. Repeat steps 7-14 for each additional compute-only B200 M4 blade that will be added to the clus-
ter.
Figure 92 Cisco UCS Manager Create Service Profiles from Template
Figure 93 Cisco UCS Manager Associate Service Profile

Install ESXi
Install and configure the ESXi hypervisor as documented in the section ESXi Hypervisor Installation. The node
must meet the following requirements before continuing with the HyperFlex cluster expansion:

The new node must be in maintenance mode while configuring the storage cluster.

The new node must be a member in the same vCenter cluster and vCenter datacenter as the existing
nodes.

The new node has SSH enabled.

The new node has at least one valid DNS and NTP server configured.

Expand the Cluster

1. Open the HyperFlex installer webpage.

2. Click Expand Cluster.

3. Click the black X next to the first line of IP addresses to delete it.
4. Click the downward arrow on the side of the Add button, and then click Add Compute Node to add
a line for a compute-only node to be added to the cluster.

a. Click the downward arrow on the side of the Add button, and then click Add Compute Node if
more than 1 new host is being added, until there are enough lines to accommodate the number
of nodes being added.

5. Enter the IP address values or DNS names for the new node(s).

6. Enter the existing HyperFlex cluster management IP address.

7. Enter the values in the Hypervisor Credentials section.

8. Enter the password of the existing cluster Controller VMs.

9. Enter the values in the vCenter Configuration section for the existing vCenter server.

10. Click Validate.

11. If the validation checks return any errors, re-check the values entered and correct as necessary, oth-
erwise click Deploy.
Figure 94 HyperFlex Installer Expand Hybrid Cluster

After the cluster expansion has completed, continue with the HyperFlex Post Installation Configuration steps.

HyperFlex Post Installation Configuration

After the HyperFlex cluster has been installed, there are some post-installation configuration steps that need
to be completed prior to installing or migrating any virtual machines to the cluster.

Create HyperFlex Datastores

The new HyperFlex cluster has no default datastores configured for virtual machine storage; therefore the
datastores must be created using the VMware vCenter Web Client plugin. A minimum of two datastores is
recommended to satisfy vSphere High Availability datastore heartbeat requirements, although one of the two
datastores can be very small. It is important to recognize that all HyperFlex datastores are thinly provisioned,
meaning that their configured size can far exceed the actual space available in the HyperFlex cluster. Alerts
will be raised by the HyperFlex system in the vCenter plugin when actual space consumption results in low
amounts of free space, and alerts will be sent via auto support email alerts. Overall space consumption in the
HyperFlex clustered filesystem is optimized by the default deduplication and compression features.
Figure 95 HyperFlex Datastore Thin Provisioning

To create HyperFlex datastores, complete the following steps:

1. After the HyperFlex installation is complete, close the vCenter Web Client, and re-open it from the
start.

2. In the home pane, click Hosts and Clusters.

3. In the Navigator pane, click on the vSphere cluster name.

4. In the center pane, a new section titled Cisco HyperFlex Systems now exists. Click on the link with
the HyperFlex cluster name. (Figure 96)

a. Alternatively, from the home screen click on VCenter Inventory Lists.

b. In the Navigator pane, click on Cisco HX Data Platform. (Figure 97)

c. In the Navigator pane, click the name of the HyperFlex cluster.

5. In the center pane, click Manage.

6. Click Datastores.

7. Click the button to Create Datastore. (Figure 98)

8. Enter the datastore name and size desired, and click OK.

9. Repeat steps 7-8 for each datastore required.

Figure 96 vSphere Cluster
Figure 97 Cisco HX Data Platform
Figure 98 Create Datastore

Configure HyperFlex Auto Support

Auto support email settings for the email server address and sender email address are configured during the
HyperFlex installation, however the recipient address needs to be configured for auto support emails to be
sent and received successfully. To configure the auto support email settings, complete the following steps:

1. Log in to the management IP address of the first storage platform controller VM via SSH, or open the
interactive CLI web interface of each controller VM.

2. Enter the commands as shown in the example below to set the recipient address, and the confirm
the configuration:

# stcli services asup recipients set --recipients [email protected]

# stcli services asup show

recipientList:

[email protected]

enabled: True

# stcli services smtp show

smtpServer: 192.168.0.4

fromAddress: [email protected]

# sendasup –t

3. Repeat steps 1-2 for each remaining storage platform controller VM in the HyperFlex cluster.
Enable vCenter Features
After the HyperFlex cluster has been installed, additional features available in vCenter should be enabled for
the cluster, specifically vCenter High Availability (HA) and Distributed Resource Scheduler (DRS). To
configure the HA and DRS features, complete the following steps:

1. Open the vCenter Web Client webpage and login.

2. In the home pane, click on Hosts and Clusters.

3. Right-click on the cluster containing the HyperFlex nodes and click Settings.

4. In the center pane, under Services, click on vSphere DRS.

5. In the Edit Cluster Settings window, check the box to Turn on vSphere DRS, and leave the remaining
settings at their defaults. (Figure 99)

6. In the Edit Cluster Settings window, click on vSphere HA on the left.

7. Check the box to turn on vSphere HA.

8. Click on the section Datastore for Heartbeating to expand the section and view the options.

9. Click the option Automatically select datastores accessible from the host. (Figure 100)

10. Click OK.

Figure 99 vSphere Enable DRS
Figure 100 vSphere Enable HA

Configure Additional VM Guest VLANs

In many cases the configuration of the HyperFlex system requires configuration of multiple VLAN IDs for the
guest virtual machines to communicate on. If additional VLANs are required, the VLANs must be defined in
UCS Manager before they can be configured as part of the ESXi host networking port groups. To configure
additional VLANs for guest VM traffic, complete the following steps:

1. In Cisco UCS Manager, click the LAN tab in the navigation pane.

2. Select LAN > LAN Cloud > VLANs.

3. Right-click VLANs and select Create VLANs.

4. Enter the name of the VLAN.

5. Select the Multicast Policy HyperFlex .

6. Leave the setting for Common/Global.

7. Enter the VLAN ID number.

8. Leave the Sharing Type set to none.

9. Click OK twice.
10. Repeat steps 3-9 for each additional VLAN required.
Figure 101 Cisco UCS Create VLAN

Configure VM Networking
The HyperFlex installer does not configure any ESXi port groups to connect the guest virtual machines. The
configuration of virtual machine port groups used by the guest VMs must be completed after the HyperFlex
installation, and configured according to the guest VM requirements. To configure guest VM networking,
complete the following steps:

1. Open the vCenter Web Client webpage and login.

2. In the home pane, click on Hosts and Clusters.

3. Right-click on the first ESXi host and click Settings.

4. In the center pane, click Networking.

5. Click the name of the virtual switch for guest VM traffic; by default this is named vswitch-hx-vm-
network.

6. Click on the Add Host Networking button. (Figure 102)

7. In the Add Networking window, select Virtual Machine Port Group for a Standard Switch, and click
Next. (Figure 103)

8. Verify the correct virtual switch is selected and click Next.

9. Enter the desired name for the port group, and enter the VLAN ID number that matches the VLAN ID
for the guest VM traffic, then click Next. (Figure 104)

10. Click Finish.

11. Repeat steps 3-10 for each additional host in the HyperFlex cluster.

Repeat the steps in this section for each additional port group required for guest VM traffic, each of them par-
ticipating in a different VLAN as defined in Cisco UCS Manager.
Figure 102 ESXi Host Networking
Figure 103 ESXi Add Networking
Figure 104 ESXi Port Group Settings

Configure VMotion Networking

The HyperFlex installer does not configure any ESXi port groups to enable vMotion of the guest virtual
machines. The configuration of a vMotion port group and vmkernel port must be completed after the
HyperFlex installation. To configure vMotion networking, complete the following steps:

1. Open the vCenter Web Client webpage and login.

2. In the home pane, click on Hosts and Clusters.

3. Right-click on the first ESXi host and click Settings.

4. In the center pane, click Networking.

5. Click the name of the virtual switch for vMotion traffic; by default this is named vMotion.

6. Click on the Add Host Networking button.

7. In the Add Networking window, select VMkernel Networking Adapter, and click Next. (Figure 105)

8. Verify the correct virtual switch is selected and click Next.

9. Enter the desired name of the port group. (Figure 106)

10. If the vMotion interfaces have been configured to carry VLAN tagged traffic, as documented in
Tagged VLANs Alternate Configuration, then modify the VLAN ID to match the value of the vMotion
VLAN defined in Cisco UCS Manager.
11. Check the box for vMotion traffic, and click Next.

12. Select Use Static IPv4 Settings, enter the desired IP address and subnet mask, and click Next.
(Figure 107)

13. Click Finish.

14. In the center pane, click to expand the VMkernel port that was just created, and click on the port.
(Figure 108)

15. Click the Edit Settings button.

16. In the Edit Settings window, click NIC Settings.

17. Change the MTU value to 9000, and click OK. (Figure 109)

18. Repeat steps 3-17 for each additional host in the HyperFlex cluster.
Figure 105 ESXi Add VMkernel Interface
Figure 106 ESXi VMkernel Settings
Figure 107 ESXi VMkernel IP Settings
Figure 108 ESXi Edit VMkernel Settings
Figure 109 ESXi VMkernel Jumbo Frames

Configure ESXi Syslog

By default, ESXi will store system and debug logs locally to the Cisco HX-Series rack mount servers. On the
HX220c-M4S model, ESXi will generate an alert regarding this configuration due to the logs being stored on
the SD cards, whose controller being a USB device, is considered to be a removable device. To clear this
fault, it is necessary to configure sending logs via Syslog to the vCenter server. For the HX240c-M4SX this
fault will be cleared after a reboot of ESXi, since the logs will be written to the 120 GB housekeeping disk,
nevertheless it is recommended to store the logs remotely. To configure syslog, complete the following
steps:

1. Open the vCenter Web Client webpage and login.

2. In the home pane, click on Hosts and Clusters.

3. Right-click on the first ESXi host and click Settings.

4. In the center pane, under the System section, click Advanced System Settings.

5. From the list on the right, select Syslog.global.logHost, then click Edit Settings button.

6. Enter the IP address of the vCenter server or vCenter appliance which will receive the syslog mes-
sages, followed by the TCP port, which is 514 by default. For example, 192.168.10.101:514

7. Click OK.

8. Repeat steps 3-7 for each additional host in the HyperFlex cluster.
Figure 110 ESXi Configure Syslog

An example script to configure ESXi guest VM networking, vMotion networking, and syslog settings is included
in the appendix: Configure ESXi Post Install
Management

vCenter Web Client Plugin

The Cisco HyperFlex vCenter Web Client Plugin is installed by the HyperFlex installer to the specified
vCenter server or vCenter appliance. The plugin is accessed as part of the vCenter Web Client interface, and
is the primary tool used to monitor and configure the HyperFlex cluster. To manage HyperFlex cluster using
the plugin, complete the following steps:

1. Open the vCenter Web Client, and login.

2. In the home pane, click on Hosts and Clusters.

3. In the Navigator pane, click on the vSphere cluster name.

4. In the center pane, a section titled Cisco HyperFlex Systems exists. Click on the link with the Hyper-
Flex cluster name.

a. Alternatively, from the home screen click on vCenter Inventory Lists.

b. In the Navigator pane, click on Cisco HX Data Platform.

c. In the Navigator pane, click the name of the HyperFlex cluster.

Figure 111 HyperFlex Web Client Plugin Summary

Summary
From the Web Client Plugin Summary screen, several elements are presented:

Overall cluster usable capacity, used capacity, free capacity, datastore capacity provisioned, and the
amount of datastore capacity provisioned beyond the actual cluster capacity.

Deduplication and compression savings percentages calculated against the data stored in the
cluster.

The cluster health state, and the number of node failures that can occur before the cluster goes into
read-only or offline mode.

A snapshot of performance over the previous hour, showing IOPS, throughput, and latencies.

Monitor
From the Web Client Plugin Monitor tab, several elements are presented:

Clicking on the Performance button gives a larger view of the performance charts. If a full webpage
screen view is desired, click the Preview Interactive Performance charts hyperlink.

Clicking on the Events button displays a HyperFlex event log, which can be used to diagnose errors
and view system activity events.
Figure 112 HyperFlex Plugin Performance Monitor
Figure 113 HyperFlex Plugin Events

Manage
From the Web Client Plugin Manage tab, several elements are presented:

Clicking on the Cluster button gives an inventory of the HyperFlex cluster and the physical assets of
the cluster hardware.

Clicking on the Datastores button allows datastores to be created, edited, deleted, mounted and
unmounted, along with space summaries and performance snapshots of that datastore.
Figure 114 HyperFlex Plugin Manage Cluster
Figure 115 HyperFlex Plugin Manage Datastores

Management Best Practices

In this section, various best practices and guidelines are given for management and ongoing use of the
Cisco HyperFlex system. These guidelines and recommendations apply only to the software versions upon
which this document is based, listed in Software Components, and may be amended in future releases.

ReadyClones
For the best possible performance and functionality of the virtual machines that will be created using the
HyperFlex ReadyClone feature, the following guidelines for preparation of the base VMs to be cloned should
be followed:

Base VMs must be stored in a HyperFlex datastore.

All virtual disks of the base VM must be stored in the same HyperFlex datastore.

Base VMs can only have HyperFlex native snapshots, no VMware redo-log based snapshots can be
present.
Create a copy of the base VM to be cloned on each of the nodes in the HyperFlex cluster. To follow
this recommendation, temporarily disable the vSphere DRS feature, afterwards the base VM can be
copied into the HyperFlex system multiple times, targeting each node in turn. Alternatively, the base
VM can be created in the HyperFlex cluster on one node, and then cloned via the standard vSphere
clone feature to the other nodes.

Create ReadyClones from the multiple base VMs evenly. For example, on a 4 node cluster where 80
clones are required, create one base VM located on each of the 4 nodes, then create 20
ReadyClones of each base VM.

Base VMs being cloned must be powered off.

Snapshots
HyperFlex native snapshots are high performance snapshots that are space-efficient, crash-consistent, and
application consistent, taken by the HyperFlex Distributed Filesystem, rather than using VMware native redo-
log based snapshots. For the best possible performance and functionality of HyperFlex native snapshots, the
following guidelines should be followed:

Ensure that the first snapshot taken of a guest VM is a HyperFlex native snapshot, by using the
apshot Now or
Schedule Snapshot. Failure to do so reverts to VMware redo-log based snapshots. (Figure 116)

Take the initial HyperFlex native snapshot with the virtual machine powered off. This creates what is
called the Sentinel snapshot. The Sentinel snapshot becomes a base snapshot that all future
snapshots are added to, and prevents the VM from reverting to VMware redo-log based snapshots.
Failure to do so can cause performance degradation when taking snapshots later, while the VM is
performing large amounts of storage IO.

client snapshot menu. As long as the initial snapshot was a HyperFlex native snapshot, each
additional snapshot is also considered to be a HyperFlex native snapshot.

Do not delete the Sentinel snapshot unless you are deleting all the snapshots entirely.

Do not revert the VM to the Sentinel snapshot. (Figure 117)

If large numbers of scheduled snapshots need to be taken, distribute the time of the snapshots taken
by placing the VMs into multiple folders or resource pools. For example, schedule two resource
groups, each with several VMs, to take snapshots separated by 15 minute intervals in the scheduler
window. Snapshots will be processed in batches of 8 at a time, until the scheduled task is
completed. (Figure 118)
Figure 116 vSphere Web Client HX Snapshot Now
Figure 117 HyperFlex Sentinel Snapshot
Figure 118 vSphere Client HX Schedule Snapshots

Storage vMotion
The Cisco HyperFlex Distributed Filesystem can create multiple datastores for storage of virtual machines.
While there can be multiple datastores for logical separation, all of the files are located within a single
distributed filesystem. As such, performing storage vMotions of virtual machine disk files has little value in
the HyperFlex system. Furthermore, storage vMotions create additional filesystem consumption and
generate additional unnecessary metadata within the filesystem, which must later be cleaned up via the
nal cleaner process.

It is recommended to not perform storage vMotions of the guest VMs between datastores within the same Hy-
perFlex cluster. Storage vMotions between different HyperFlex clusters, or between HyperFlex and non-
HyperFlex datastores are permitted.

Virtual Disk Placement

HyperFlex clusters can create multiple datastores for logical separation of virtual machine storage, yet the
files are all stored in the same underlying distributed filesystem. The only difference between one datastore
and another are their names and their configured sizes. Due to this, there is no compelling reason for a
virtual machine to be stored on a particular datastore versus another.
All of the virtual disks that make up a single virtual machine must be placed in the same datastore. Spreading
the virtual disks across multiple datastores provides no benefit, and can cause ReadyClone and snapshot er-
rors.

HX Maintenance Mode
has been installed by the
HyperFlex plugin. This option directs the storage platform controller on the node to shutdown gracefully,
redistributing storage IO to the other nodes with minimal impact. Using the standard Maintenance Mode
menu in the vSphere Web Client, or the vSphere (thick) Client can be used, but graceful failover of storage
IO and shutdown of the controller VM is not guaranteed.

In order to minimize the performance impact of placing a HyperFlex converged storage node into maintenance
mode, it is recommended to use the HX Maintenance Mode menu selection to enter or exit maintenance mode
whenever possible.
Figure 119 vSphere Web Client HX Maintenance Mode

Failure Scenarios
This section provides a description of several failure scenarios within a HyperFlex system, the response
behavior by the HyperFlex software, ESXi and the VMs, and recovery behavior for each scenario. The
following scenarios apply to the default HyperFlex cluster settings of replication factor set to 3, and access
policy set to strict:
Network Failures

HX-Series Server Link Failure

One network link between an HX-Series server and the Fabric Interconnect fails, either due to a physical port
failure, cable failure, or an incorrect configuration change within Cisco UCS Manager.

Response

Within ESXi, all vmnic interfaces connected to the failed link will go offline.

Traffic that was previously transmitted and received on that link, based on the explicit vSwitch uplink
failover order, will now traverse the surviving link.

All VMs remain online and fully functional, no migrations occur and no storage interruptions are seen.

Recovery

Once the link failure is repaired, all VMnic interfaces will return to an online state, and all network
traffic will resume their normal pathways.

Cisco UCS Uplink Failure

All Cisco UCS network uplinks from a single Fabric Interconnect fail, either due to physical port failures,
cable failures, an incorrect configuration change in Cisco UCS Manager, or a failure or misconfiguration of
the upstream switch.

Response

will show link down.

Traffic that was previously transmitted and received on that fabric, based on the explicit vSwitch
uplink failover order, will now traverse the opposite fabric for all nodes.

All VMs remain online and fully functional, no migrations occur and no storage interruptions are seen.

Recovery

Once the uplink failure is repaired, all vNICs will return to an online state, and all network traffic will
resume their normal pathways.

Additional east-west traffic going from fabric A to fabric B, and vice-versa, across the Cisco UCS
network uplinks will occur briefly, as the vNICs are not always brought online at the exact same
times.

Fabric Interconnect Failure

A Fabric Interconnect fails, or goes offline due to planned or unplanned maintenance activity.

Response

All of the vNICs serviced by the failed FI fabric go offline and the corresponding ESXi vmnic
interfaces will show link down.
Traffic that was previously transmitted and received on that fabric, based on the explicit vSwitch
uplink failover order, will now traverse the opposite fabric for all nodes.

All VMs remain online and fully functional, no migrations occur and no storage interruptions are seen.

Recovery

Once the FI is returned to service, after the network uplinks automatically reconnect and come
online, all vNICs will return to an online state, and all network traffic will resume their normal
pathways.

Additional east-west traffic going from fabric A to fabric B, and vice-versa, across the Cisco UCS
network uplinks will occur briefly, as the vNICs are not always brought online at the exact same
times.

Disk Failures

Capacity HDD Failure

A capacity HDD fails in an HX-Series converged node, or is removed.

Response

The HyperFlex cluster state changes to unhealthy. Alerts are raised in vCenter for the disk failure and
cluster health.

An immediate rebalance job is triggered to return the system to the specified replication factor,
replicating the missing data on the disk from the remaining copies.

When the rebalance job finishes, the cluster state returns to healthy.

All VMs remain online and fully functional, no migrations occur and no storage interruptions are seen.

Total cluster storage capacity will be reduced accordingly.

Recovery

Once the failed disk is replaced, the HX software will automatically begin using the replaced disk for
new data storage.

A default nightly automatic rebalance job will evenly distribute data across the cluster, including the
new disk, ensuring that there are no hotspots or cold spots where space consumption is
unnecessarily high or low.

Total cluster storage capacity will return to previous levels.

Caching SSD Failure

A caching SSD fails in an HX-Series converged node, or is removed.

Response

The HyperFlex cluster state changes to unhealthy. Alerts are raised in vCenter for the disk failure and
cluster health.

An immediate rebalance job is triggered.

When the rebalance job finishes, the cluster state returns to healthy.

All VMs remain online and fully functional, no migrations occur and no storage interruptions are seen.

Recovery

Once the failed disk is replaced, the HX software will not automatically begin using the replaced disk
for storage, a manual rebalance job must be started from the command line of one of the controller
VMs:

# stcli rebalance start --force

Housekeeping SSD Failure

A housekeeping SSD fails in an HX-Series converged node, or is removed.

Response

HX220c-M4S server: the storage controller VM remains online, although services within the VM will
fail. All guest VMs remain online and fully functional, no migrations occur and no storage interruptions
are seen.

HX240c-M4SX server: the response is functionally identical to a Node Failure.

Recovery

Once the failed disk is replaced, specific recovery steps must be taken to return the node to full
service. Please contact Cisco TAC for assistance.

Maintenance Mode
An HX-Series converged node must be taken offline for planned maintenance.

Response

With VMware DRS enabled: placing the node into HX Maintenance Mode will evacuate the guest VMs
via vMotion, the controller VM will shut down, and the node is ready to shut down or reboot.

Without VMware DRS enabled: placing the node into HX Maintenance Mode will not succeed until the
guest VMs are manually moved via vMotion or powered off, then the controller VM will shut down,
and the node is ready to shut down or reboot.

The cluster state changes to unhealthy. Alerts are raised in vCenter for cluster health and node
removal.

All VMs remain online and fully functional, and no storage interruptions are seen.

If the node remains offline for more than 2 hours, a rebalance job will be triggered to redistribute
data across the cluster and bring the number of data copies back in line with the configured
replication factor. The cluster state would then return to healthy.

Total cluster storage capacity will be reduced accordingly.

Recovery

Once the node is back online, exiting from maintenance mode will trigger the storage controller VM
to start.
After a brief period for services on the controller VM to start, the cluster state will return to healthy.

The node is available to host VMs, by manually moving them or allowing DRS to automatically
balance the load.

If the node was offline for more than 2 hours and the auto rebalance job ran, the node is considered
to be empty when it re-enters the cluster; even if the disks still have their old data, the data will be
discarded. To redistribute the data across the node, either wait for the default automatic rebalance
job to run, or manually run a rebalance from the command line of one of the controller VMs:

# stcli rebalance start –force

Total cluster storage capacity will return to previous levels.

Node Failure
An HX-Series converged node goes offline due to an unexpected failure.

Response

With VMware HA enabled: the guest VMs will be restarted on the remaining nodes of the cluster after
a brief period of time.

Without VMware HA enabled: the guest VMs will not be restarted on the remaining nodes of the
cluster and will remain offline without manual intervention.

The cluster state changes to unhealthy. Alerts are raised in vCenter for cluster health and node
removal.

storage interruptions are

seen.

If the node remains offline for more than 2 hours, a rebalance job will be triggered to redistribute
data across the cluster and bring the number of data copies back in line with the configured
replication factor. The cluster state would then return to healthy.

Total cluster storage capacity will be reduced accordingly.

Recovery

Once the node is back online the storage controller VM will start.

After a brief period for services on the controller VM to start, the cluster state will return to healthy.

The node is available to host VMs, by manually moving them or allowing DRS to automatically
balance the load.

If the node was offline for more than 2 hours and the auto rebalance job ran, the node is considered
to be empty when it re-enters the cluster; even if the disks still have their old data, the data will be
discarded. To redistribute the data across the node, either wait for the default automatic rebalance
job to run, or manually run a rebalance from the command line of one of the controller VMs:

# stcli rebalance start –force

Total cluster storage capacity will return to previous levels.

Validation
This section provides a list of items that should be reviewed after the HyperFlex system has been deployed
and configured. The goal of this section is to verify the configuration and functionality of the solution, and
ensure that the configuration supports core availability requirements.

Post Install Checklist

The following tests are critical to functionality of the solution, and should be verified before deploying for
production:

Verify the expected number of converged storage nodes and compute-only nodes are members of
the HyperFlex cluster in the vSphere Web Client plugin manage cluster screen.

Verify the expected cluster capacity is seen in the vSphere Web Client plugin summary screen.

Create a test virtual machine that accesses the HyperFlex datastore and is able to perform read/write
operations.

Perform the virtual machine migration (vMotion) of the test virtual machine to a different host in the
cluster.

During the vMotion of the virtual machine, make sure the test virtual machine can perform a
continuous ping to default gateway and to check if the network connectivity is maintained during and
after the migration.

Verify Redundancy
The following redundancy checks can be performed to verify the robustness of the system. Network traffic,
such as a continuous ping from VM to VM, or from vCenter to the ESXi hosts should not show significant
failures (one or two ping drops might be observed at times). Also, all of the HyperFlex datastores must
remain mounted and accessible from all the hosts at all times.

Administratively disable one of the server ports on Fabric Interconnect A which is connected to one
of the HyperFlex converged storage hosts. The ESXi virtual switch uplinks for fabric A should now
show as failed, and the standby uplinks on fabric B will be in use for the management and vMotion
virtual switches. Upon administratively re-enabling the port, the uplinks in use should return to
normal.

Administratively disable one of the server ports on Fabric Interconnect B which is connected to one
of the HyperFlex converged storage hosts. The ESXi virtual switch uplinks for fabric B should now
show as failed, and the standby uplinks on fabric A will be in use for the storage virtual switch. Upon
administratively re-enabling the port, the uplinks in use should return to normal.

Place a representative load of guest virtual machines on the system. Put one of the ESXi hosts in
maintenance mode, using the HyperFlex HX maintenance mode option. All the VMs running on that
host should be migrated via vMotion to other active hosts via vSphere DRS, except for the storage
platform controller VM, which will be powered off. No guest VMs should lose any network or storage
accessibility during or after the migration. This test assumes that enough RAM is available on the
remaining ESXi hosts to accommodate VMs from the host put in maintenance mode. The HyperFlex
cluster will show in an unhealthy state.
Reboot the host that is in maintenance mode, and exit it from maintenance mode after the reboot.
The storage platform controller will automatically start when the host exits maintenance mode. The
HyperFlex cluster will show as healthy after a brief time to restart the services on that node. VSphere
DRS should rebalance the VM distribution across the cluster over time.

vCenter alerts do not automatically clear, even when the fault has been resolved. Once the cluster
health is verified, the alerts must be manually cleared.

Reboot one of the Cisco UCS Fabric Interconnects while traffic is being sent and received on the
storage datastores and the network. The reboot should not affect the proper operation of storage
access and network traffic generated by the VMs. Numerous faults and errors will be noted in Cisco
UCS Manager, but all will be cleared after the FI comes back online.
Appendix

A: Cluster Capacity Calculations

A HyperFlex HX Data Platform cluster capacity is calculated as follows:

(((<capacity disk size in GB> X 10^9) / 1024^3) X <number of capacity disks per node> X <number of
HyperFlex nodes> X 0.92) / replication factor

Divide the result by 1024 to get a value in TiB

The replication factor value is 3 if the HX cluster is set to RF=3, and the value is 2 if the HX cluster is set to
RF=2.

The 0.92 multiplier accounts for an 8% reservation set aside on each disk by the HX Data Platform software
for various internal filesystem functions.

Calculation example:

<capacity disk size in GB> = 1200 for 1.2 TB disks

<number of capacity disks per node> = 15 for an HX240c M4SX model server

<number of HyperFlex nodes> = 8

replication factor = 3

Result: (((1200*10^9)/1024^3)*15*8*0.92)/3 = 41127.2049

41127.2049 / 1024 = 40.16 TiB

B: Example Scripts
Example PowerShell scripts are provided to assist with the configuration of the HyperFlex system. The
scripts must be modified for their proper use according to the settings required by the system being
installed. Additional PowerShell integrations may be required to be installed on the system where the scripts
will be run, such as vSphere PowerCLI or Cisco UCS PowerTool Suite.

Create DHCP Reservations

# Create_DHCP_Reservations.ps1

# Description: Imports a UCS MAC address pool CSV export file and creates DHCP
reservations

# Usage: Modify the $csv variable to point to the UCS CSV export file. Modify the
$ip

# variable to set the starting IP address of the last octet. Modify the $address
variable to

# contain the first three octets of the IP addresses. Modify the -ScopeId value
to the name
# of the DHCP scope where the reservations are being added. Run the script from
the Windows

# DHCP server with the scope you are adding to.

$csv="mgmt-a.csv"

$ip=11

$macs = Import-Csv $csv | Where-Object {$_.Assigned -eq "yes"} | Select ID

Foreach ($mac in $macs) {

$mac = ($mac.ID -split '/')[3]

$mac = $mac -replace ":"

$address="192.168.10."+$ip

Write-Host "Creating reservation for MAC address:" $mac

Write-Host "Using IP address:" $address

Get-DhcpServerv4Scope -ScopeId 192.168.10.0 | Add-DhcpServerv4Reservation -

ClientId $mac -IPAddress $address

$ip++

Configure ESXi Hosts

# Configure_ESX_Hosts_HX.ps1

# Description: Configures ESXi options and adds the server to vCenter

# Usage: Modify the variables to specify the vCenter server address, user,
password, ESXi

# host root password, vCenter cluster name, and the servers to be configured.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false | Out-

Null

$vc = "vcenter1.hx.lab.cisco.com"

$user = "hx\1"

$password = "1"

$rootpw = "Cisco123"

$cluster = "Cluster1"

$servers="hx220-01.hx.lab.cisco.com","hx220-02.hx.lab.cisco.com","hx220-
03.hx.lab.cisco.com","hx220-04.hx.lab.cisco.com","hx220-
05.hx.lab.cisco.com","hx220-06.hx.lab.cisco.com","hx220-
07.hx.lab.cisco.com","hx220-08.hx.lab.cisco.com"
# connect to each ESXi host and configure their base settings

Foreach ($server in $servers) {

write-host "Configuring ESXi host:" $server

# connect to the ESXi host

Connect-VIServer -server $server -user root -password $rootpw

$vmhost = Get-VMHost -Name $server

# configure NTP

write-host "Configuring NTP:"

$myntpserver1="192.168.10.4"

Add-VmHostNtpServer -VMHost $vmhost -NtpServer $myntpserver1

Get-VmHostService -VMHost $vmhost | Where-Object {$_.key -eq "ntpd"} | Start-

VMHostService

Get-VmHostService -VMHost $vmhost | Where-Object {$_.key -eq "ntpd"} | Set-

VMHostService -policy "on"

# suppress SSH warnings

write-host "Configuring SSH:"

$vmhost | Set-VMHostAdvancedConfiguration UserVars.SuppressShellWarning 1

# enter maintenance mode

write-host "Enter maintenance mode:"

$vmhost | Set-VMHost -State Maintenance

# disconnect from the ESXi host

Disconnect-VIServer -server $server -Confirm:$False

# Connect to vCenter

Connect-VIServer -server $vc -user $user -password $password

# Add the host to vCenter

Add-VMHost $server -Location $cluster -User root -Password $rootpw -Force

Disconnect-VIServer -server $vc -Confirm:$False

Configure ESXi Post Install

# Configure_ESX_post_install.ps1

# Description: Configures ESXi options and settings after HyperFlex installation.

# Usage: Modify the variables to specify the ESXi root password, the servers to
be

# configured, the guest VLAN ID, and the IP addresses used for the vMotion
vmkernel

# interfaces.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false | Out-

Null

$rootpw = "Cisco123"

$servers="hx220-01.hx.lab.cisco.com","hx220-02.hx.lab.cisco.com","hx220-
03.hx.lab.cisco.com","hx220-04.hx.lab.cisco.com","hx220-
05.hx.lab.cisco.com","hx220-06.hx.lab.cisco.com","hx220-
07.hx.lab.cisco.com","hx220-08.hx.lab.cisco.com"

$ip=11

Foreach ($server in $servers) {

# connect to the ESXi host server

Connect-VIServer -server $server -user root -password $rootpw

$vmhost = Get-VMHost -Name $server

# retrieve the virtual switch configurations

$vswitch2 = Get-VirtualSwitch -VMHost $vmhost -Name vswitch-hx-vm-network

$vswitch3 = Get-VirtualSwitch -VMHost $vmhost -Name vmotion

# create a port group for the guest VMs

New-VirtualPortGroup -VirtualSwitch $vswitch2 -Name "VM Network" -VLanID 100

# create the vmotion port group and vmkernel interface

$vmip="192.168.200."+$ip

New-VMHostNetworkAdapter -VMHost $vmhost -VirtualSwitch $vswitch3 -PortGroup

"vmotion" -Mtu 9000 -VMotionEnabled $true -IP $vmip -SubnetMask 255.255.255.0 -
Confirm:$false

$ip=$ip+1

# configure syslog traffic to send to vCenter

Set-VMHostSysLogServer -SysLogServer '192.168.10.101:514' -VMHost $vmhost

Disconnect-VIServer -server $server -Confirm:$False

C: Example Cisco Nexus 9372 Switch Configurations

Switch A
hostname HX-9K-A

no feature telnet

no telnet server enable

cfs eth distribute

feature interface-vlan

feature lacp

feature vpc

ip domain-lookup

ip domain-list cisco.com

ip name-server 171.70.168.183 173.36.131.10

logging event link-status default

policy-map type network-qos jumbo

class type network-qos class-default

mtu 9216

system qos

service-policy type network-qos jumbo

clock timezone PST -8 0

clock summer-time PST

ntp server 171.68.38.65

ntp server 171.68.38.66

vrf context management

vlan 1

vlan 10

name Management

vlan 51

name HXCluster1

vlan 100

name VM-Prod-100

vlan 200

name VMotion

cdp enable

vpc domain 50

role priority 10

peer-keepalive destination 10.29.133.102 source 10.29.133.101

auto-recovery

delay restore 150

interface Vlan1

interface port-channel50

description VPC-Peer

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

spanning-tree port type network

vpc peer-link

interface port-channel10

description VPC to 6248-A

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type edge trunk

spanning-tree bpduguard enable

mtu 9216

vpc 10

interface port-channel20

description VPC to 6248-B

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type edge trunk

spanning-tree bpduguard enable

mtu 9216

vpc 20

interface Ethernet1/1

description uplink

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type network

interface Ethernet1/2

description NX9372-A_P1/2--UCS6248-A_2/1

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

channel-group 10 mode active

interface Ethernet1/4

description NX9372-A_P1/4--UCS6248-B_2/1

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

channel-group 20 mode active

interface Ethernet1/47

description NX9372-A_P1/47--NX9372-B_P1/47

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

channel-group 50 mode active

interface Ethernet1/48

description NX9372-A_P1/48--NX9372-B_P1/48

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

channel-group 50 mode active

interface mgmt0

ip address 10.29.133.101/24

vrf context management

ip route 0.0.0.0/0 10.29.133.1

Switch B
hostname HX-9K-B

no feature telnet

no telnet server enable

cfs eth distribute

feature interface-vlan

feature lacp

feature vpc

ip domain-lookup

ip domain-list cisco.com

ip name-server 171.70.168.183 173.36.131.10

logging event link-status default

policy-map type network-qos jumbo

class type network-qos class-default

mtu 9216

system qos

service-policy type network-qos jumbo

clock timezone PST -8 0

clock summer-time PST

ntp server 171.68.38.65

ntp server 171.68.38.66

vrf context management

vlan 1

vlan 10

name Management

vlan 51

name HXCluster1

vlan 100

name VM-Prod-100

vlan 200

name VMotion

cdp enable

vpc domain 50

role priority 10

peer-keepalive destination 10.29.133.101 source 10.29.133.102

auto-recovery

delay restore 150

interface Vlan1

interface port-channel50

description VPC-Peer

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

spanning-tree port type network

vpc peer-link

interface port-channel10

description VPC to 6248-A

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type edge trunk

spanning-tree bpduguard enable

mtu 9216

vpc 10

interface port-channel20

description VPC to 6248-B

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type edge trunk

spanning-tree bpduguard enable

mtu 9216

vpc 20

interface Ethernet1/1

description uplink

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

spanning-tree port type network

interface Ethernet1/2

description NX9372-A_P1/2--UCS6248-A_2/3

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

channel-group 10 mode active

interface Ethernet1/4

description NX9372-A_P1/4--UCS6248-B_2/3

switchport mode trunk

switchport trunk allowed vlan 10,51,100,200

channel-group 20 mode active

interface Ethernet1/47

description NX9372-B_P1/47--NX9372-A_P1/47

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

channel-group 50 mode active

interface Ethernet1/48

description NX9372-B_P1/48--NX9372-A_P1/48

switchport mode trunk

switchport trunk allowed vlan 1,10,51,100,200

channel-group 50 mode active

interface mgmt0

ip address 10.29.133.102/24

vrf context management

ip route 0.0.0.0/0 10.29.133.1

About Authors
Brian Everitt, Technical Marketing Engineer, Cisco Systems, Inc.

Brian Everitt is a Technical Marketing Engineer with the Cisco UCS Data Center Engineering Solutions group.
He is an IT industry veteran with over 18 years of experience deploying server, network, and storage
infrastructures for companies around the world. During his tenure at Cisco, he has previously been a lead
Advanced Services Solutions Architect for Microsoft solutions, virtualization, and SAP Hana on UCS.
Currentl
Infrastructure solutions.

Acknowledgements
Jeffrey Fultz, Technical Marketing Engineer, Cisco Systems, Inc.
Resources

Glossary

API Application Programming Interface. A set of remote calls, verbs, queries or actions
exposed to external users for automation of various application tasks.

ASIC Application Specific Integrated Circuit. An integrated circuit specially designed for a
specific use versus a general purpose processor.

CDP Cisco Discovery Protocol. A Cisco developed protocol to share information about
directly connected networking devices.

CLI Command Line Interface. A text based method of entering commands one-by-one in
successive lines.

DHCP Dynamic Host Configuration Protocol. A protocol for dynamically assigning and
distributing network settings to clients on demand.

DNS Domain Name System. A hierarchical service primarily used for mapping of host and
domain names to the IP addresses of the computers configured to use those names
and addresses.

FC Fibre Channel. A high speed, lossless networking protocol, using fixed packet sizes
and a worldwide unique identification technique, primarily used to connect computing
systems to storage devices.

FC-AL Fibre Channel Arbitrated Loop. A Fibre Channel protocol topology where the devices
are connected via a high speed, one-way ring network.

FCoE Fibre Channel over Ethernet. Fibre channel traffic encapsulated in Ethernet frames,
carried over Ethernet networks.

FI Fabric Interconnect. The primary component of the Cisco UCS system, connecting all
end devices and networks, and providing management services.

GbE Gigabit Ethernet. One gigabit of network transmission rate is equal to one billion bits
transferred per second.

GUI Graphical User Interface. In contrast to a CLI, a user interface where the primary
interaction between the user and the computer is via a series of graphical windows and
buttons, typically using an on screen pointer controlled by a computer mouse.

HBA Host Bus Adapter. A hardware device that connects the computer host to an external
device, such as a storage system or other network device.

HDD Hard Disk Drive. A computer storage device which stores data on rotating magnetic
discs or platters, read and written to by a moving magnetic head.

IPMI Intelligent Platform Management Interface. A specification for an independent computer

subsystem that provides management and monitoring capabilities separate from the
host system's CPU and operating system.
IO Input/Output.

IOM IO Module. A common name for Cisco Fabric Extender modules installed in the rear of
a Cisco UCS 5108 blade chassis.

IOPS Input/Output operations per second.

KVM Keyboard/Video/Mouse. The three primary methods of human to computer interaction.

LACP Link Aggregation Control Protocol. A specification for bundling multiple physical
network interfaces into a single logical interface, providing additional bandwidth and
failure tolerance.

LAN Local Area Network. Interconnection of computer devices within a limited area, in
contrast to a WAN.

MTU Maximum Transmission Unit. The maximum size of the data unit transmittable by the
network protocol.

NAND Negative-AND logic gate. A Boolean function circuit design used in SSD devices,
emulating the bit state of traditional magnetic media as used in hard disk drives.

NAS Network Attached Storage. A storage system providing file-level access to resources
via a network protocol, in contrast to a block-level storage system.

NFS Network File System. A file system protocol for accessing files via a server across a
network.

NTP Network Time Protocol. A protocol for synchronizing computer system time clocks over
data networks.

NIC Network Interface Controller. A hardware component connecting a computer to a data

network.

OUI Organizationally Unique Identifier. A registry of 24-bit (3-byte) identifiers maintained by

the Institute of Electrical and Electronics Engineers (IEEE) Registration Authority to
uniquely identify vendors and manufacturers.

OVA Open Virtual Appliance. A compressed open virtualization format (OVF) file containing
all the files necessary to deploy and run a pre-created virtual machine.

PCI Peripheral Component Interconnect. A computer bus interface for connecting internal
hardware components.

PCIe PCI Express. A revised standard replacing PCI, that uses a serial architecture, higher
speeds and fewer pins on the devices.

QoS Quality of Service. The overall performance of the network as seen by the end users
and computers. Various standards and methods have been developed to provide
different levels of control and prioritization of network traffic used by specific devices
and applications.

RBAC Role-based Access Control. Restricting system access to authorized users, and
granting them privileges in the system based on a defined functional or job role.

RF Replication Factor. The number of times a written block of data is replicated across
independent nodes in a HyperFlex storage cluster.
RU Rack Unit. EIA-310 specification define the height of each unit of height in a 19 inch

SAN Storage Area Network. A data network, typically using Fibre Channel protocol, to
provide computer hosts access to centralized storage devices.

SAS Serial Attached SCSI. A serial computer bus for connecting storage devices using SCSI
commands.

SCSI Small Computer Systems Interface. A set of standards for connections between
computers and storage devices and peripherals, using a common command set.

SD Secure Digital. A portable and non-volatile format for memory card storage devices.

SDS Software Defined Storage. A method for providing access to and provisioning of
computer storage resources that are independent of the underlying physical storage
resources.

SoL Serial over LAN. Transmission of serial port output via a LAN versus the physical serial
port.

SSD Solid State Disk. A computer data storage device typically using NAND based flash
memory for persistent data storage, versus spinning magnetic media as used in a hard
disk.

SSH Secure Shell. An encrypted protocol allowing remote login across unsecured networks.

STP Spanning-Tree Protocol. A network protocol that creates loop-free networks by

establishing preferred network pathways in a LAN.

Cisco UCS Cisco Unified Computing System. The Cisco product line combining rack-mount
servers, blade servers, and Fabric Interconnects into a single domain, with integrated
networking and management.

UCSM Cisco UCS Manager. The management GUI and CLI software for control of a Cisco UCS
domain.

VIC Virtual Interface Card. A Cisco product offering the ability to create multiple virtual
network or storage interfaces on a single physical hardware card.

VLAN Virtual LAN. A partitioned and isolated LAN segment, breaking a flat network into
multiple subdivided networks.

VM Virtual Machine. A running operating system atop a set of virtualized hardware

resources, abstracted from the real underlying physical components.

VMDK Virtual Machine Disk. A file format for storing the contents of a typical hard disk as a
file, used by virtual machines as their hard disk contents.

VMFS filesystem used to store virtual

machine definitions, VMDK files, snapshots and configuration files.

vNIC Virtual NIC. The virtualized definition of a traditional NIC running in the Cisco UCS
system. The vNIC is defined in a service profile, and dynamically programmed to a VIC.

VPC Virtual Port Channel. A Cisco technology for connecting network devices to multiple
partner devices without creating loops and without using STP.
WAN Wide Area Network. Interconnection of computer devices across long distances and
geographically large areas, in contrast to a LAN.
References
1. Cisco HyperFlex HX220c M4 Node Installation Guide:
http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX220c_M4/HX220c.h
tml

2. Cisco HyperFlex HX240c M4 Node Installation Guide:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX240c_M4/HX240c.h
tml

3. Cisco HyperFlex HX220c M4 Node Specifications:

http://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-infrastructure/hyperflex-
hx-series/datasheet-c78-736817.pdf

4. Cisco HyperFlex HX240c M4 Node Specifications:

http://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-infrastructure/hyperflex-
hx-series/datasheet-c78-736818.pdf

5. Cisco HyperFlex Systems Getting Started Guide:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftw
are/GettingStartedGuide/b_HyperFlexSystems_GettingStartedGuide.html

6. Cisco HyperFlex Systems Pre-Install Checklist:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftw
are/HyperFlex_preinstall_checklist/b_HyperFlexSystems_checklist_2.html

7. Cisco HyperFlex Systems Administration Guide:

http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftw
are/AdminGuide/b_HyperFlexSystems_AdminGuide.html

8. Cisco HyperFlex Systems Command Line Interface Reference:

http://www.cisco.com/c/dam/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatform
Software/CLIGuide/HX_Data_Platform_CLIReferenceGuide_171.pdf

9. HyperFlex Hardware and Software Interoperability Matrix:

http://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-
software/products-technical-reference-list.html