VMware ESXI - 6.7 Instruction Manual

Download

Page 1

vSphere Monitoring and Performance

17 APR 2018 VMware vSphere 6.7 VMware ESXi 6.7 vCenter Server 6.7

Page 2

vSphere Monitoring and Performance

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to

docfeedback@vmware.com

VMware, Inc.

3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

VMware, Inc. 2

Page 3

About vSphere Monitoring and Performance 5

Monitoring Inventory Objects with Performance Charts 7

Performance Chart Types 8

Data Counters 8

Metric Groups in vSphere 10

Data Collection Intervals 10

Data Collection Levels 11

View Performance Charts 12

Performance Charts Options Available Under the View Menu 13

Overview Performance Charts 14

Working with Advanced and Custom Charts 106

Troubleshoot and Enhance Performance 109

Monitoring Guest Operating System Performance 116

Enable Statistics Collection for Guest Operating System Performance Analysis 116

View Performance Statistics for Windows Guest Operating Systems 116

Monitoring Host Health Status 118

Monitor Health Status in the vSphere Client 119

Monitor Hardware Health Status 119

Reset Health Status Sensors in the vSphere Client 120

Reset Health Status Sensors 120

VMware, Inc.

Monitoring vSphere Health 121

Check vSphere Health in vSphere Client 121

Monitoring Events, Alarms, and Automated Actions 123

View Events 125

View System Logs 125

Export Events Data 126

Streaming Events to a Remote Syslog Server 126

Retention of Events in the vCenter Server Database 128

View Triggered Alarms and Alarm Definitions 130

Live Refresh of Recent Tasks and Alarms 130

Set an Alarm in the vSphere Web Client 131

Set an Alarm in the vSphere Client 141

Acknowledge Triggered Alarms 144

Page 4

vSphere Monitoring and Performance

Reset Triggered Event Alarms 144

Preconfigured vSphere Alarms 145

Monitoring Solutions with the vCenter Solutions Manager 151

View Solutions and vServices 151

Monitoring Agents 152

Monitoring vServices 152

Monitoring the Health of Services and Nodes 154

View the Health Status of Services and Nodes 154

Performance Monitoring Utilities: resxtop and esxtop 156

Using the esxtop Utility 156

Using the resxtop Utility 157

Using esxtop or resxtop in Interactive Mode 158

Using Batch Mode 173

Using Replay Mode 174

Using the vimtop Plug-In to Monitor the Resource Use of Services 177

Monitor Services By Using vimtop in Interactive Mode 177

Interactive Mode Command-Line Options 177

Interactive Mode Single-Key Commands for vimtop 178

Monitoring Networked Devices with SNMP and vSphere 180

Using SNMP Traps with vCenter Server 180

Configure SNMP for ESXi 182

SNMP Diagnostics 193

Monitor Guest Operating Systems with SNMP 193

VMware MIB Files 193

SNMPv2 Diagnostic Counters 195

System Log Files 197

View System Log Entries 197

View System Logs on an ESXi Host 197

System Logs 198

Export System Log Files 199

ESXi Log Files 200

Upload Logs Package to a VMware Service Request 200

Configure Syslog on ESXi Hosts 201

Configuring Logging Levels for the Guest Operating System 202

Collecting Log Files 204

Viewing Log Files with the Log Browser 208

VMware, Inc. 4

Page 5

About vSphere Monitoring and Performance

VMware provides several tools to help you monitor your virtual environment and to locate the source of

potential issues and current problems.

Performance charts Allow you to see performance data on a variety of system resources

including CPU, Memory, Storage, and so on.

Performance

monitoring command-

line utilities

Host health Allows you to quickly identify which hosts are healthy and which are

Events, alerts, and

alarms

System Log Files System logs contain additional information about activities in your vSphere

Allow you to access detailed information on system performance through

the command line.

experiencing problems.

Allow you to configure alerts and alarms and to specify the actions the

system should take when they are triggered.

environment.

Intended Audience

The content in this section is intended for vSphere administrators who perform the following tasks:

Monitor the health and performance of physical hardware backings for the virtual environment.

Monitor the health and performance of virtual devices in the virtual environment.

Troubleshoot problems in the system.

Configure alarms.

Configure SNMP messages.

Virtual machine administrators also might find the section on Chapter 2 Monitoring Guest Operating

System Performance helpful.

vSphere Web Client and vSphere Client

Instructions in this guide reflect the vSphere Client (an HTML5-based GUI). You can also use the

instructions to perform most of the tasks by using the vSphere Web Client (a Flex-based GUI).

VMware, Inc.

Page 6

vSphere Monitoring and Performance

Tasks for which the workflow differs significantly between the vSphere Client and the vSphere Web Client

have duplicate procedures that provide steps according to the respective client interface. The procedures

that relate to the vSphere Web Client, contain vSphere Web Client in the title.

Note In vSphere 6.7, most of the vSphere Web Client functionality is implemented in the vSphere Client.

For an up-to-date list of the unsupported functionality, see Functionality Updates for the vSphere Client.

VMware Technical Publications Glossary

VMware Technical Publications provides a glossary of terms that might be unfamiliar to you. For

definitions of terms as they are used in VMware technical documentation, go to

http://www.vmware.com/support/pubs.

VMware, Inc. 6

Page 7

Monitoring Inventory Objects

with Performance Charts 1

The vSphere statistics subsystem collects data on the resource usage of inventory objects. Data on a

wide range of metrics is collected at frequent intervals, processed, and archived in the vCenter Server

database. You can access statistical information through command-line monitoring utilities or by viewing

performance charts in the vSphere Web Client.

Counters and Metric Groups

vCenter Server systems and hosts use data counters to query for statistics. A data counter is a unit of

information relevant to a given inventory object or device. Each counter collects data for a different

statistic in a metric group. For example, the disk metric group includes separate data counters to collect

data for disk read rate, disk write rate, and disk usage. Statistics for each counter are rolled up after a

specified collection interval. Each data counter consists of several attributes that are used to determine

the statistical value collected.

For a complete list and description of performance metrics, see the vSphere API Reference.

Note Counters that are introduced in later versions might not contain data from hosts of earlier versions.

For details, see the VMware Knowledge Base.

Collection Levels and Collection Intervals

Collection levels determine the number of counters for which data is gathered during each collection

interval. Collection intervals determine the time period during which statistics are aggregated, calculated,

rolled up, and archived in the vCenter Server database. Together, the collection interval and collection

level determine how much statistical data is collected and stored in your vCenter Server database.

Data Availability

Real-time data appears in the performance charts only for hosts and virtual machines that are powered

on. Historical data appears for all supported inventory objects, but might be unavailable during certain

circumstances.

VMware, Inc.

Page 8

vSphere Monitoring and Performance

This chapter includes the following topics:

Performance Chart Types

Data Counters

Metric Groups in vSphere

Data Collection Intervals

Data Collection Levels

View Performance Charts

Performance Charts Options Available Under the View Menu

Overview Performance Charts

Working with Advanced and Custom Charts

Troubleshoot and Enhance Performance

Performance Chart Types

Performance metrics are displayed in different types of charts, depending on the metric type and object.

Table 1‑1. Performance Chart Types

Chart Type Description

Line chart Displays metrics for a single inventory object. The data for each performance counter is plotted on a separate

line in the chart. For example, a network chart for a host can contain two lines: one showing the number of

packets received, and one showing the number of packets transmitted.

Bar chart Displays storage metrics for datastores in a selected data center. Each datastore is represented as a bar in the

chart. Each bar displays metrics based on the file type: virtual disks, snapshots, swap files, and other files.

Pie chart Displays storage metrics for a single object, based on the file types, or virtual machines. For example, a pie

chart for a datastore can display the amount of storage space occupied by the virtual machines taking up the

largest space.

Stacked chart Displays metrics for the child objects that have the highest statistical values. All other objects are aggregated,

and the sum value is displayed with the term Other. For example, a host's stacked CPU usage chart displays

CPU usage metrics for the 10 virtual machines on the host that are consuming the most CPU. The Other

amount contains the total CPU usage of the remaining virtual machines.

The metrics for the host itself are displayed in separate line charts.

Stacked charts are useful in comparing the resource allocation and usage across multiple hosts or virtual

machines. By default, the 10 child objects with the highest data counter values are displayed.

Data Counters

Each data counter includes several attributes that are used to determine the statistical value collected.

See the vSphere API Reference for a complete list and description of supported counters.

VMware, Inc. 8

Page 9

vSphere Monitoring and Performance

Table 1‑2. Data Counter Attributes

Attribute Description

Unit of Measurement Standard in which the statistic quantity is measured.

Kilobytes (KB) – 1024 bytes

Kilobytes per second (KBps) – 1024 bytes per second

Kilobits (kb) – 1000 bits

Kilobits per second (kbps) – 1000 bits per second

Megabytes (MB)

Megabytes per second (MBps)

Megabits (Mb), megabits per second (Mbps)

Megahertz (MHz)

Microseconds (µs)

Milliseconds (ms)

Number (#)

Percent (%)

Seconds (s)

Description Text description of the data counter.

Statistics Type Measurement used during the statistics interval. Related to the unit of measurement.

Rate – Value over the current statistics interval

Delta – Change from previous statistics interval.

Absolute – Absolute value (independent of the statistics interval).

Rollup Type Calculation method used during the statistics interval to roll up data. Determines the type of statistical

values that are returned for the counter.

Average – Data collected during the interval is aggregated and averaged.

Minimum – The minimum value is rolled up.

Maximum – The maximum value is rolled up.

The Minimum and Maximum values are collected and displayed only in statistics level 4. Minimum and

maximum rollup types are used to capture peaks in data during the interval. For real-time data, the

value is the current minimum or current maximum. For historical data, the value is the average

minimum or average maximum.

For example, the following information for the CPU usage chart shows that the average is collected at

statistics level 1. The minimum and maximum values are collected at statistics level 4.

Counter: usage

Unit: Percentage (%)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Summation – Data collected is summed. The measurement displayed in the chart represents the sum

of data collected during the interval.

Latest – Data collected during the interval is a set value. The value displayed in the performance charts

represents the current value.

Collection level Number of data counters used to collect statistics. Collection levels range from 1 to 4, with 4 having the

most counters.

Note Be careful when you set a higher collection level, as the process requires significant increase of

resource usage. For more information, see Data Collection Levels.

VMware, Inc. 9

Page 10

vSphere Monitoring and Performance

Metric Groups in vSphere

The performance data collection subsystem for vSphere collects performance data on various inventory

items and their devices. Data counters define individual performance metrics. Performance metrics are

organized into logical groups based on the object or object device. Statistics for one or more metrics can

be displayed in a chart.

Table 1‑3. Metric Groups

Metric group Description

Cluster Services Performance statistics for clusters configured by using vSphere Distributed Resource Scheduler, vSphere High

Availability, or both.

CPU CPU utilization per host, virtual machine, resource pool, or compute resource.

Datastore Statistics for datastore utilization.

Disk Disk utilization per host, virtual machine, or datastore. Disk metrics include I/O performance, such as latency and

read/write speeds, and utilization metrics for storage as a finite resource.

Memory Memory utilization per host, virtual machine, resource pool, or compute resource. The value obtained is one of

the following:

For virtual machines, memory refers to the guest physical memory. Guest physical memory is the amount of

physical memory presented as a virtual-hardware component to the virtual machine, at creation time, and

made available when the virtual machine is running.

For hosts, memory refers to the machine memory. Machine memory is the RAM that is installed on the

hardware that comprises the host.

Network Network utilization for both physical and virtual network interface controllers (NICs) and other network devices.

The virtual switches that support connectivity among all components, such as hosts, virtual machines, VMkernel.

Power Energy usage statistics per host.

Storage Adapter Data traffic statistics per host bus adapter (HBA).

Storage Path Data traffic statistics per path.

System Overall system availability, such as the system heartbeat and uptime. These counters are available directly from

hosts and from vCenter Server.

Virtual Disk Disk utilization and disk performance metrics for virtual machines.

Virtual Flash Virtual flash counters.

Virtual Machine

Operations

vSphere

Replication

Virtual machine power and provisioning operations in a cluster or data center.

Statistics for the virtual machine replication performed by VMware vCenter Site Recovery Manager.

Data Collection Intervals

Collection intervals determine the duration for which statistics are aggregated, calculated, rolled up, and

archived. Together, the collection interval and collection level determine how much statistical data is

gathered and stored in your vCenter Server database.

VMware, Inc. 10

Page 11

vSphere Monitoring and Performance

Table 1‑4. Collection Intervals

Collection

Interval/Archive Length

1 Day 5 Minutes Real-time statistics are rolled up to create one data point every 5 minutes. The

1 Week 30 Minutes 1-Day statistics are rolled up to create one data point every 30 minutes. The result

1 Month 2 Hours 1-Week statistics are rolled up to create one data point every 2 hours. The result

1 Year 1 Day 1-Month statistics are rolled up to create one data point every day. The result is

Collection

Frequency Default Behavior

result is 12 data points every hour and 288 data points every day. After 30

minutes, the six data points collected are aggregated and rolled up as a data point

for the 1-Week time range.

You can change the interval duration and archive length of the 1-Day collection

interval by configuring the statistics settings.

is 48 data points every day and 336 data points every week. Every 2 hours, the 12

data points collected are aggregated and rolled up as a data point for the 1-Month

time range.

You cannot change the default settings of the 1-Week collection interval.

is 12 data points every day and 360 data points every month (assuming a 30-day

month). After 24 hours, the 12 data points collected are aggregated and rolled up

as a data point for the 1-Year time range.

You cannot change the default settings of the 1-Month collection interval.

365 data points each year.

You can change the archive length of the 1-Year collection interval by configuring

the statistics settings.

Note If you change the duration of data collection intervals you might need to allocate more storage

resources.

Data Collection Levels

Each collection interval has a default collection level that determines the amount of data gathered and

which counters are available for display in the charts. Collection levels are also referred to as statistics

levels.

VMware, Inc. 11

Page 12

vSphere Monitoring and Performance

Table 1‑5. Statistics Levels

Level Metrics Best Practice

Level 1

Level 2

Level 3

Cluster Services (VMware Distributed Resource Scheduler) – all

metrics

CPU – cpuentitlement, totalmhz, usage (average), usagemhz

Disk – capacity, maxTotalLatency, provisioned, unshared, usage

(average), used

Memory – consumed, mementitlement, overhead, swapinRate,

swapoutRate, swapused, totalmb, usage (average), vmmemctl

(balloon)

Network – usage (average), IPv6

System – heartbeat, uptime

Virtual Machine Operations – numChangeDS, numChangeHost,

numChangeHostDS

Level 1 metrics

CPU – idle, reservedCapacity

Disk – All metrics, excluding numberRead and numberWrite.

Memory – All metrics, excluding memUsed and maximum and

minimum rollup values.

Virtual Machine Operations – All metrics

Level 1 and Level 2 metrics

Metrics for all counters, excluding minimum and maximum rollup

values.

Device metrics

Use for long-term performance monitoring

when device statistics are not required.

Level 1 is the default Collection Level for

all Collection Intervals.

Use for long-term performance monitoring

when device statistics are not required

but you want to monitor more than the

basic statistics.

Use for short-term performance

monitoring after encountering problems or

when device statistics are required.

Because of the large quantity of

troubleshooting data retrieved and

recorded, use level 3 for the shortest time

period (Day or Week collection interval).

Level 4 All metrics supported by the vCenter Server, including minimum and

maximum rollup values.

Use for short-term performance

monitoring after encountering problems or

when device statistics are required.

Because of the large quantity of

troubleshooting data retrieved and

recorded, use level 4 for the shortest

amount of time.

Note When you increase the collection level, the storage and system requirements might change. You

might need to allocate more system resources to avoid a decrease in the performance.

View Performance Charts

The vCenter Server statistics settings, the type of object selected, and the features that are enabled on

the selected object determine the amount of information displayed in charts. Charts are organized into

views. You can select a view to see related data together on one screen. You can also specify the time

range, or data collection interval. The duration extends from the selected time range to the present time.

VMware, Inc. 12

Page 13

vSphere Monitoring and Performance

Overview charts display multiple data sets in one panel to evaluate different resource statistics, display

thumbnail charts for child objects. It also displays charts for a parent and a child object. Advanced charts

display more information than overview charts, are configurable, and can be printed or exported. You can

export data in the PNG, JPEG, or CSV formats. See View Advanced Performance Charts.

Procedure

1 Select a valid inventory object in the vSphere Web Client.

Overview and advanced performance charts are available for datacenter, cluster, host, resource pool,

vApp, and virtual machine objects. Overview charts are also available for datastores and datastore

clusters. Performance charts are not available for network objects.

2 Click the Monitor tab, and click Performance.

3 Select a view.

Available views depend on the type of object. For views that might contain many charts in a large

environment, the vSphere Web Client displays the charts distributed on multiple pages. You can use

the arrow buttons to navigate between pages.

4 Select a predefined or custom time range.

Performance Charts Options Available Under the View Menu

The performance chart options that you can access under the View menu vary depending on the type of

inventory object you select.

For example, the Virtual Machines view is available when you view host performance charts only if there

are virtual machines on the selected host. Likewise, the Fault Tolerance view for virtual machine

performance charts is available only when that feature is enabled for the selected virtual machine.

Table 1‑6. Performance Chart Views by Inventory Object

Object View List Items

Data center

Datastore and

datastore cluster

Storage - space utilization charts for datastores in the data center, including space by file type and storage

space used by each datastore in the data center.

Clusters - thumbnail CPU and memory charts for each cluster, and stacked charts for total CPU and

memory usage in the data center. This view is the default.

Space - space utilization charts for the datastore:

space utilization by file type

space utilization by virtual machine

space usage

Performance - performance charts for the datastore or datastore cluster and for virtual machine disks on

the resource.

Note The Performance view for datastores is only available when all hosts that are connected to the datastores

are ESX/ESXi 4.1 or greater. The Performance view for datastore clusters is only available when the Storage

DRS is enabled.

VMware, Inc. 13

Page 14

vSphere Monitoring and Performance

Table 1‑6. Performance Chart Views by Inventory Object (Continued)

Object View List Items

Cluster

Host

Resource Pool

and vApps

Virtual Machine

Home - CPU and memory charts for the cluster.

Resource Pools & Virtual Machines - thumbnail charts for resource pools and virtual machines, and

stacked charts for total CPU and memory usage in the cluster.

Hosts - thumbnail charts for each host in the cluster, and stacked charts for total CPU, memory, disk usage,

and network usage.

Home - CPU, memory, disk, and network charts for the host.

Virtual Machines - thumbnail charts for virtual machines, and stacked charts for total CPU usage and total

memory usage on the host.

Home - CPU and memory charts for the resource pool.

Resource Pools & Virtual Machines - thumbnail charts for resource pools, and virtual machines and

stacked charts for CPU and memory usage in the resource pool or vApp.

Storage - space utilization charts for the virtual machine: space by file type, space by datastore, and total

gigabytes.

Fault Tolerance - CPU and memory charts that display comparative metrics for the fault-tolerant primary

and secondary virtual machines.

Home - CPU, memory, network, host (thumbnail charts), and disk usage charts for the virtual machine.

Overview Performance Charts

The overview performance charts display the most common metrics for an object in the inventory. Use

these charts to monitor and troubleshoot performance problems.

The metrics provided in Overview performance charts are a subset of those collected for hosts and the

vCenter Server. For a complete list of all metrics collected by hosts and the vCenter Server, see the

vSphere API Reference.

Clusters

The cluster charts contain information about CPU, disk, memory, and network usage for clusters. The

help topic for each chart contains information about the data counters displayed in that chart. The

collection level set for vCenter Server determines the available counters.

CPU (MHz)

The CPU (MHz) chart displays CPU usage for the cluster.

Cluster Counters

This chart is located in the Home view of the Cluster Performance tab.

VMware, Inc. 14

Page 15

vSphere Monitoring and Performance

Table 1‑7. Data Counters

Chart Label Description

Usage Sum of the average CPU usage values, in Megahertz, of all virtual machines in the cluster.

Counter: usagemhz

Stats Type: Rate

Unit: Megahertz (MHz)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Total Total amount of CPU resources available in the cluster. The maximum value is equal to the number of

cores multiplied by the frequency of the processors.

For example, a cluster has two hosts, each of which has four CPUs that are 3 GHz each, and one virtual

machine that has two virtual CPUs.

VM totalmhz = 2 vCPUs * 3000 MHz = 6000 MHz

Host totalmhz = 4 CPUs * 3000 MHz = 12000 MHz

Cluster totalmhz = 2 x 4 * 3000 MHz = 24000 MHz

Counter: totalmhz

Stats Type: Rate

Unit: Megahertz (MHz)

Rollup Type: Summation

Collection Level: 1

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of cluster resources. However, if

the value is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high

CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the

hosts in the cluster.

If performance is impacted, consider taking the following actions.

Table 1‑8. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks:

1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4 Click Turn ON vSphere DRS, and click OK.

3 If the cluster is a DRS cluster:

Increase the number of hosts, and migrate one or more virtual machines to the new host.

Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the

cluster.

4 Migrate one or more virtual machines to a new host.

5 Upgrade the physical CPUs or cores on each host in the cluster if necessary.

VMware, Inc. 15

Page 16

vSphere Monitoring and Performance

Table 1‑8. CPU Performance Enhancement Advice (Continued)

# Resolution

6 Enable CPU-saving features, such as TCP Segmentation Offload.

7 Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

CPU Usage

The cluster CPU Usage charts monitor the CPU utilization of the hosts, resource pools, and virtual

machines in the cluster. This chart displays the 10 child objects in the cluster with the most CPU usage.

This chart is located in the Resource Pools and Virtual Machines view of the Cluster Performance tab.

Table 1‑9. Data Counters

Chart Label Description

<host>, <resource pool>, or

Amount of CPU actively used by the host, resource pool, or virtual machine in the cluster.

Counter: usagemhz

Stats Type: Rate

Unit: MegaHertz (MHz)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of cluster resources. However, if

the value is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high

CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the

hosts in the cluster.

If performance is impacted, consider taking the following actions.

Table 1‑10. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks:

1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4 Click Turn ON vSphere DRS, and click OK.

3 If the cluster is a DRS cluster:

Increase the number of hosts, and migrate one or more virtual machines to the new host.

Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the

cluster.

4 Migrate one or more virtual machines to a new host.

5 Upgrade the physical CPUs or cores on each host in the cluster if necessary.

VMware, Inc. 16

Page 17

vSphere Monitoring and Performance

Table 1‑10. CPU Performance Enhancement Advice (Continued)

# Resolution

6 Enable CPU-saving features, such as TCP Segmentation Offload.

7 Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

Disk (KBps)

The Disk (KBps) chart displays the disk I/O of the 10 hosts in the cluster with the most disk usage.

This chart is located in the Hosts view of the cluster Performance tab.

Table 1‑11. Data Counters

Chart Label Description

host_name Average data I/O rate across all hosts in the cluster.

Counter: usage

Stats Type: Rate

Unit: Kilobytes per second (KBps)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example,

you might notice a performance degradation with applications that frequently read from and write to the

hard disk. If you see a spike in the number of disk read or write requests, check whether any such

applications were running then.

The best ways to determine if your vSphere environment is experiencing disk problems is to monitor the

disk latency data counters. You can use the advanced performance charts to view these statistics.

The kernelLatency data counter measures the average amount of time, in milliseconds, that the

VMkernel spends processing each SCSI command. For best performance, the value must be 0 -1

milliseconds. If the value is greater than 4 ms, the virtual machines on the host are trying to send

more throughput to the storage system than the configuration supports. Check the CPU usage, and

increase the queue depth.

The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a

SCSI command from the physical device. Depending on your hardware, a number greater than 15 ms

indicates probable problems with the storage array. Move the active VMDK to a volume with more

spindles or add disks to the LUN.

The queueLatency data counter measures the average amount of time taken per SCSI command in

the VMkernel queue. This value must always be zero. If not, the workload is too high and the array

cannot process the data fast enough.

If the disk latency values are high, or if you notice other problems with disk I/O performance, consider

taking the following actions.

VMware, Inc. 17

Page 18

vSphere Monitoring and Performance

Table 1‑12. Disk I/O Performance Enhancement Advice

# Resolution

1 Increase the virtual machine memory. It allows more operating system caching, which reduces I/O activity. Note: It might

require you to increase the host memory. Increasing memory might reduce the need to store data because databases can

utilize the system memory to cache data and avoid disk access.

To verify that virtual machines have adequate memory, check swap statistics in the guest operating system. Increase the

guest memory, but not to an extent that leads to excessive host memory swapping. Install VMware Tools so that memory

ballooning can occur.

2 Defragment the file systems on all guests.

3 Disable antivirus on-demand scans on the VMDK and VMEM files.

4 Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access

common elements on an array, the disks might have trouble keeping up. To increase throughput, consider array-side

improvements.

5 Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.

6 Balance the disk load across all physical resources available. Spread heavily used storage across LUNs that are accessed

by different adapters. Use separate queues for each adapter to improve disk efficiency.

7 Configure the HBAs and RAID controllers for optimal use. Verify that the queue depths and cache settings on the RAID

controllers are adequate. If not, increase the number of outstanding disk requests for the virtual machine by adjusting the

Disk.SchedNumReqOutstanding parameter. For more information, see vSphere Storage.

8 For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system

page file. It alleviates disk spindle contention during periods of high use.

9 On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX

file.

10 If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.

11 For ESXi hosts, create virtual disks as preallocated. When you create a virtual disk for a guest operating system, select

Allocate all disk space now. The performance degradation associated with reassigning additional disk space does not

occur, and the disk is less likely to become fragmented.

12 Use the most current host hardware.

Memory (MB)

The Memory (MB) chart displays consumed memory for the cluster. The chart appears only at collection

level 1.

This chart is located in the Home view of the cluster Performance tab.

VMware, Inc. 18

Page 19

vSphere Monitoring and Performance

Table 1‑13. Data Counters

Chart Label Description

Consumed Amount of host machine memory used by all powered on virtual machines in the cluster. A cluster's

consumed memory consists of virtual machine consumed memory and overhead memory. It does not

include host-specific overhead memory, such as memory used by the service console or VMkernel.

Counter: consumed

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Total Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory

(physical memory for use by the Guest OS) and virtual machine overhead memory.

Memory Total = Aggregate host machine memory - (VMkernel memory + Service Console memory +

other service memory)

Note The totalmb data counter is the same as the effectivemem data counter, which is supported only

for backward compatibility.

Counter: totalmb

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

Memory usage is not an indicator of performance problems. Memory can be high if a host is swapping or

ballooning, which can result in virtual machine guest swapping. In such cases, check for other problems,

such as CPU over-commitment or storage latencies.

If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking the

following actions.

Table 1‑14. Memory Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to performance. The

VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, it does not impact virtual

machine performance.

3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource pools on

the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine. If free memory is

available on the hosts and the virtual machines are experiencing high swap or balloon memory, the virtual machine (or

resource pool, if it belongs to one) has reached its resource limit. Check the maximum resource limit set on that host.

VMware, Inc. 19

Page 20

vSphere Monitoring and Performance

Table 1‑14. Memory Performance Enhancement Advice (Continued)

# Resolution

4 If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks:

1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4 Click Turn ON vSphere DRS, and click OK.

5 If the cluster is a DRS cluster:

Increase the number of hosts, and migrate one or more virtual machines to the new host.

Check the aggressiveness threshold. If the value is low, increase the threshold. It might help avoid hot spots in the

cluster.

6 Add more physical memory to one or more hosts.

Memory (MB)

The Memory (MB) chart displays memory data counters for clusters. The chart appears at all collection

levels except level 1.

Description

This chart is located in the Home view of the cluster Performance tab.

Note These data counter definitions are for hosts. At the cluster level, the values are collected and

totaled. The counter values in the chart represent the aggregate amounts of the host data. The counters

that appear in the chart depend on the collection level set for your vCenter Server.

Table 1‑15. Data Counters

Chart Label Description

Active Sum of the active guest physical memory of all powered on virtual machines on the host,

plus memory used by basic VMkernel applications. Active memory is estimated by the

VMkernel.

Counter: active

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 2 (4)

Balloon Sum of the guest physical memory reclaimed by the balloon driver for all powered on virtual

machines on the host.

Counter: vmmemctl

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

VMware, Inc. 20

Page 21

vSphere Monitoring and Performance

Table 1‑15. Data Counters (Continued)

Chart Label Description

Consumed Amount of machine memory used on the host.

Consumed memory includes virtual machine memory, service console memory, and

VMkernel memory.

consumed memory = total host memory - free host memory

Counter: consumed

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Granted Sum of the guest physical memory granted for all powered on virtual machines. Granted

memory is mapped to the host's machine memory.

Granted memory for a host includes the shared memory of each virtual machine on the host.

Counter: granted

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 2 (4)

Swap Used Sum of the memory swapped by all powered on virtual machines on the host.

Counter: swapused

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 2 (4)

Total Aggregate total memory available to the cluster.

Counter: totalmb

Stats Type: Absolute

Unit: Megabytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

To ensure best performance, the host memory must be large enough to accommodate the active memory

of the virtual machines. The active memory can be smaller than the virtual machine memory size. It

allows you to over-provision memory, but still ensures that the virtual machine active memory is smaller

than the host memory.

Transient high-usage values usually do not cause performance degradation. For example, memory usage

can be high when several virtual machines are started at the same time or when a spike occurs in virtual

machine workload. However, a consistently high memory usage value (94% or greater) indicates that the

host is probably lacking the memory required to meet the demand. If the active memory size is the same

as the granted memory size, the demand for memory is greater than the memory resources available. If

the active memory is consistently low, the memory size might be too large.

VMware, Inc. 21

Page 22

vSphere Monitoring and Performance

If the memory usage value is high, and the host has high ballooning or swapping, check the amount of

free physical memory on the host. A free memory value of 6% or less indicates that the host cannot

handle the demand for memory. It leads to memory reclamation, which might degrade performance.

If the host has enough free memory, check the resource shares, reservation, and limit settings of the

virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower

than those set for the virtual machines.

If the host has little free memory available, or if you notice a degradation in performance, consider taking

the following actions.

Table 1‑16. Memory Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical

to performance.

2 Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and

swapping. Generally, it does not impact virtual machine performance.

3 Reduce the memory space on the virtual machine, and correct the cache size if it is too large. This frees up memory for other

virtual machines.

4 If the memory reservation of the virtual machine is set to a value much higher than its active memory, decrease the

reservation setting so that the VMkernel can reclaim the idle memory for other virtual machines on the host.

5 Migrate one or more virtual machines to a host in a DRS cluster.

6 Add physical memory to the host.

Memory Consumed

The Memory Consumed chart displays memory usage for the 10 child objects in the cluster with the most

consumed memory.

For resource pools and virtual machines in a cluster, this chart is located in the Resource Pools &

Virtual Machines view of the cluster Performance tab. For hosts in a cluster, this chart is located in the

Hosts view of the cluster Performance tab.

Table 1‑17. Data Counters

Chart Label Description

resource_pool,

virtual_machine, or

host

Amount of machine memory used by all resource pools and virtual machines in the cluster or by all hosts

in the cluster, depending on the cluster view.

Consumed memory includes virtual machine memory, service console memory, and VMkernel memory.

consumed memory = total host memory - free host memory

Counter: consumed

Stats Type: Absolute

Unit: MegaBytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

VMware, Inc. 22

Page 23

vSphere Monitoring and Performance

Chart Analysis

Memory usage is not an indicator of performance problems. Memory can be high if a host is swapping or

ballooning, which can result in virtual machine guest swapping. In such cases, check for other problems,

such as CPU over-commitment or storage latencies.

If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking the

following actions.

Table 1‑18. Memory Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to performance. The

VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, it does not impact virtual

machine performance.

3 If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource pools on

the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine. If free memory is

available on the hosts and the virtual machines are experiencing high swap or balloon memory, the virtual machine (or

resource pool, if it belongs to one) has reached its resource limit. Check the maximum resource limit set on that host.

4 If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks:

1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4 Click Turn ON vSphere DRS, and click OK.

5 If the cluster is a DRS cluster:

Increase the number of hosts, and migrate one or more virtual machines to the new host.

Check the aggressiveness threshold. If the value is low, increase the threshold. It might help avoid hot spots in the

cluster.

6 Add more physical memory to one or more hosts.

Network (Mbps)

The Network (Mbps) chart displays network speed for the 10 hosts in the cluster with the most network

usage.

This chart is located in the Hosts view of the Cluster Performance tab.

Table 1‑19. Data Counters

Chart Label Description

<host> Average rate at which data is transmitted and received across all NIC instances on the host.

Counter: usage

Stats Type: Rate

Unit: Megabits per second (Mbps)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

VMware, Inc. 23

Page 24

vSphere Monitoring and Performance

Chart Analysis

Network performance depends on the application workload and network configuration. Dropped network

packets indicate a bottleneck in the network. To determine whether packets are being dropped, use

esxtop or the advanced performance charts to examine the droppedTx and droppedRx network counter

values.

If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check

the size of the network packets and the data receive and transfer rates. In general, the larger the network

packets, the faster the network speed. When the packet size is large, fewer packets are transferred,

which reduces the amount of CPU required to process the data. When network packets are small, more

packets are transferred but the network speed is slower because more CPU is required to process the

data.

Note In some instances, large packets might result in a high network latency. To check the network

latency, use the VMware AppSpeed performance monitoring application or a third-party application.

If packets are not being dropped and the data receive rate is slow, the host is probably lacking the CPU

resources required to handle the load. Check the number of virtual machines assigned to each physical

NIC. If necessary, perform load balancing by moving virtual machines to different vSwitches or by adding

more NICs to the host. You can also move virtual machines to another host or increase the host CPU or

virtual machine CPU.

If you experience network-related performance problems, also consider taking the following actions.

Table 1‑20. Networking Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance.

3 If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid

transferring packets over the physical network.

4 Assign each physical NIC to a port group and a vSwitch.

5 Use separate physical NICs to handle the different traffic streams, such as network packets generated by virtual machines,

iSCSI protocols, vMotion tasks.

6 Ensure that the physical NIC capacity is large enough to handle the network traffic on that vSwitch. If the capacity is not

enough, consider using a high-bandwidth physical NIC (10 Gbps). Alternatively, consider moving some virtual machines to a

vSwitch with a lighter load or to a new vSwitch.

7 If packets are being dropped at the vSwitch port, increase the virtual network driver ring buffers where applicable.

8 Verify that the reported speed and duplex settings for the physical NIC match the hardware expectations and that the

hardware is configured to run at its maximum capability. For example, verify that NICs with 1 Gbps are not reset to 100 Mbps

because they are connected to an older switch.

9 Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a

lower speed or half duplex mode.

10 Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where

possible.

VMware, Inc. 24

Page 25

vSphere Monitoring and Performance

Data centers

The data center charts contain information about CPU, disk, memory, and storage usage for data centers.

The help topic for each chart contains information about the data counters displayed in that chart. The

counters available are determined by the collection level set for vCenter Server.

CPU (MHz)

The CPU (MHz) chart displays CPU usage for the 10 clusters in the data center with the most CPU

usage.

This chart is located in the Clusters view of the Datacenters Performance tab.

Table 1‑21. Data Counters

Chart Label Description

<cluster> Amount of CPU currently in use by the cluster. The active CPU usage is approximately equal to the ratio of

the used CPU cycles to the available CPU cycles.

The maximum possible value is the frequency of the processors multiplied by the number of cores. For

example, a two-way SMP virtual machine using 4000MHz on a host that has four 2GHz processors is using

50% of the CPU (4000 ÷ 4 × 2000) = 0.5).

Counter: usagemhz

Stats Type: Rate

Unit: MegaHertz (MHz)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of cluster resources. However, if

the value is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high

CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the

hosts in the cluster.

If performance is impacted, consider taking the following actions.

Table 1‑22. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2 If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks:

1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4 Click Turn ON vSphere DRS, and click OK.

VMware, Inc. 25

Page 26

vSphere Monitoring and Performance

Table 1‑22. CPU Performance Enhancement Advice (Continued)

# Resolution

3 If the cluster is a DRS cluster:

Increase the number of hosts, and migrate one or more virtual machines to the new host.

Check the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the

cluster.

4 Migrate one or more virtual machines to a new host.

5 Upgrade the physical CPUs or cores on each host in the cluster if necessary.

6 Enable CPU-saving features, such as TCP Segmentation Offload.

7 Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

Memory (MB)

The Memory (MB) chart displays the average amount of consumed memory for the 10 clusters in the data

center with the most consumed memory.

This chart is located in the Clusters view of the Datacenters Performance tab.

Table 1‑23. Data Counters

Chart Label Description

<cluster> Amount of host machine memory used by all powered on virtual machines in the cluster.

Counter: consumed

Stats Type: Absolute

Unit: MegaBytes (MB)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A cluster's consumed memory consists of virtual machine consumed memory and overhead memory. It

does not include host-specific overhead memory, such as memory used by the service console or

VMkernel.

If you experience problems with cluster memory usage, use the thumbnail cluster charts to examine

memory usage for each cluster and increase memory resources if needed.

If the cluster is a DRS cluster, check the aggressiveness threshold. If the value is low, increase the

threshold. Increasing the threshold might help avoid hot spots in the cluster.

Space used by Datastore in GB

The Space used by Datastore in GB chart displays the 10 datastores in the data center with the most

used disk space.

This chart is located in the Storage view of the Datacenter Performance tab.

VMware, Inc. 26

Page 27

vSphere Monitoring and Performance

Table 1‑24. Data Counters

Chart Label Description

<datastore> Amount of used storage space on the 10 datastores with the most used space.

Counter: used

Stats Type: Absolute

Unit: GigaBytes (GB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Space Utilization By File Type

The Space Utilization By File Type chart displays datastore space usage for virtual disks, swap files,

snapshot files, and other virtual machine files.

Note This chart does not show historical statistics. It only shows the most recently available data, which

may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics

are not collected across all datastores at one time. They are collected asynchronously.

The Space Utilization by File Type chart is located in the Storage view of the data center Performance

tab.

VMware, Inc. 27

Page 28

vSphere Monitoring and Performance

Datastore Counters

Table 1‑25. Data Counters

File Type Description

Virtual Disks Amount of disk space used by virtual disk files.

Virtual disk files store the contents of the virtual machine's hard disk drive. It includes information that you

write to a virtual machine's hard disk, such as the operating system, program files, and data files. The files

have the extension .vmdk and appear as a physical disk drive to a guest operating system.

Note Delta disks, which also have an extension .vmdk, are not included in this file type.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Swap Files Amount of disk space used by swap files.

Swap files back up the virtual machine's physical memory.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Snapshots Amount of disk space used by virtual machine snapshot files.

Snapshot files store information about virtual machine snapshots. They include snapshot state files and

delta disk files. A snapshot state file stores the running state of the virtual machine at the time of the

snapshot. It has the extension .vmsn. A delta disk file stores the updates made by the virtual machine to

the virtual disks after a snapshot is taken.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Other VM Files Amount of disk space used by all other virtual machine files, such as configuration files and log files.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Other Amount of disk space used by all other non-virtual machine files, such as documentation files and backup

files.

Free Space Amount of disk space not currently in use.

Total Space Amount of disk space available to the datastore. It defines the datastore capacity. The chart displays the

information for datastores but not for data centers.

total space = virtual disk space + swap file space + snapshot space + other VM file space + other space +

free space

VMware, Inc. 28

Page 29

vSphere Monitoring and Performance

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Datastores and Datastore Clusters

The datastore charts contain information about disk usage for datastores or the datastores that are part of

a cluster. The help topic for each chart contains information about the data counters displayed in that

chart. The counters available are determined by the collection level set for vCenter Server.

Space in GB

The Space in GB chart displays space usage data counters for datastores.

This chart is located in the Space view of the datastore or datastore cluster Performance tab.

VMware, Inc. 29

Page 30

vSphere Monitoring and Performance

Table 1‑26. Data Counters

Chart Label Description

Allocated Amount of physical space provisioned by an administrator for the datastore. It is the storage size up to

which files on the datastore can grow. Allocated space is not always in use.

Counter: provisioned

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1

Used Amount of physical datastore space in use.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1

Capacity Maximum capacity of the datastore.

capacity = virtual machine file space + non-virtual machine file space + free space

Note Storage data is collected and updated in the overview charts every 30 minutes. Therefore, if you

refresh the datastore, the capacity value might only be updated in the datastore Summary tab, and not in

the overview charts.

Counter: capacity

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Space Utilization By File Type

The Space Utilization by File Type chart displays space used by virtual disks, swap files, snapshot files,

and other virtual machine files on the datastore or the datastore cluster.

Note This chart does not show historical statistics. It only shows the most recently available data, which

may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics

are not collected across all datastores at one time. They are collected asynchronously.

VMware, Inc. 30

Page 31

vSphere Monitoring and Performance

The Space Utilization by File Type chart is located in the Storage view of the datastore Performance tab.

The counters can also be displayed for datastore cluster charts.

Datastore Counters

Table 1‑27. Data Counters

File Type Description

Virtual Disks Amount of disk space used by virtual disk files.

Virtual disk files store the contents of the virtual machine's hard disk drive. It includes information that you

write to a virtual machine's hard disk, such as the operating system, program files, and data files. The files

have the extension .vmdk and appear as a physical disk drive to a guest operating system.

Note Delta disks, which also have an extension .vmdk, are not included in this file type.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Swap Files Amount of disk space used by swap files.

Swap files back up the virtual machine's physical memory.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Snapshots Amount of disk space used by virtual machine snapshot files.

Snapshot files store information about virtual machine snapshots. They include snapshot state files and

delta disk files. A snapshot state file stores the running state of the virtual machine at the time of the

snapshot. It has the extension .vmsn. A delta disk file stores the updates made by the virtual machine to

the virtual disks after a snapshot is taken.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Other VM Files Amount of disk space used by all other virtual machine files, such as configuration files and log files.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1 (4)

Other Amount of disk space used by all other non-virtual machine files, such as documentation files and backup

files.

VMware, Inc. 31

Page 32

vSphere Monitoring and Performance

Table 1‑27. Data Counters (Continued)

File Type Description

Free Space Amount of disk space not currently in use.

Total Space Amount of disk space available to the datastore. It defines the datastore capacity. The chart displays the

information for datastores but not for data centers.

total space = virtual disk space + swap file space + snapshot space + other VM file space + other space +

free space

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Space used by Datastore in GB

The Space used by Datastore in GB chart displays the 10 datastores in the data center with the most

used disk space.

This chart is located in the Storage view of the Datacenter Performance tab.

Table 1‑28. Data Counters

Chart Label Description

<datastore> Amount of used storage space on the 10 datastores with the most used space.

Counter: used

Stats Type: Absolute

Unit: GigaBytes (GB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

VMware, Inc. 32

Page 33

vSphere Monitoring and Performance

Space Utilization by Virtual Machine

The Space Utilization by Virtual Machine chart displays the amount of space used by the five virtual

machines with the most space used on the datastore or the datastores in the cluster.

Note This chart does not show historical statistics. It only shows the most recently available data, which

may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics

are not collected across all datastores at one time. They are collected asynchronously.

The Space Utilization by Virtual Machine chart is located in the Space view of the datastore Performance

tab. The counter can also be displayed for datastore cluster charts.

Table 1‑29. Data Counters

Chart Label Description

virtual_machine Amount of datastore space used by the five virtual machines with the most used datastore

space.

Counter: used

Stats Type: Absolute

Unit: Gigabytes (GB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Space Allocated by Datastore in GB

The Space allocated by Datastore in GB displays the top 10 datastores, virtual machines in the datastore-

cluster with most provisioned space.

This chart is located in the Space view of the Datacenter Performance tab.

VMware, Inc. 33

Page 34

vSphere Monitoring and Performance

Table 1‑30. Data Counters

Chart Label Description

<datastore> Amount of provisioned storage space on the top 10 datastores with the most provisioned

space.

Counter: provisioned

Stats Type: Absolute

Unit: KiloBytes (KB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Space Capacity by Datastore in GB

The Space capacity by Datastore in GB displays the top 10 configured size of the datastores in the

datastore cluster.

This chart is located in the Space view of the Datacenter Performance tab.

Table 1‑31. Data Counters

Chart Label Description

<datastore> Configured size of the datastores in the datastore cluster.

Counter: capacity

Stats Type: Absolute

Unit: KiloBytes (KB)

Rollup Type: Latest

Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be

larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If

possible, you can provision more space to the datastore, or you can add disks to the datastore or use

shared datastores.

VMware, Inc. 34

Page 35

vSphere Monitoring and Performance

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when

they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the

snapshots from the vSphere Web Client user interface. For information about consolidating the data

center, see the vSphere documentation.

Storage I/O Control Normalized Latency

This chart displays the normalized latency in microseconds on the datastore. Storage I/O Control

monitors latency to detect congestion on the datastore. This metric computes a weighted response time

across all hosts and VMs accessing the datastore. I/O count is used as the weight for the response time.

It captures the device level latency and does not include any queuing inside the hypervisor storage stack

or inside the VM. It is adjusted for the I/O size. High latencies that are the result of large I/Os are

discounted so as not to make the datastore seem slower than it really is. Data for all virtual machines is

combined. This chart displays zero values when Storage I/O Control is disabled.

This chart is located in the Performance view of the datastore Performance tab. The

sizeNormalizedDatastoreLatency counter can also be displayed for datastore cluster charts.

Table 1‑32. Data Counters

Chart Label Description

Storage I/O Control

Normalized Latency

Storage I/O Control monitors latency to detect congestion on the datastore.

Counter: sizeNormalizedDatastoreLatency

Stats Type: Absolute

Unit: Microseconds

Rollup Type: Average

Collection Level: 3

Storage I/O Control Aggregate IOPs

This chart displays the number of I/O operations per one second on the datastore, aggregated across all

hosts, and virtual machines accessing this datastore. The chart displays zero values when Storage I/O

Control is disabled.

This chart is located in the Performance view of the datastore or the datastore cluster Performance tab.

The counter can be displayed for datastore and datastore cluster charts.

Table 1‑33. Data Counters

Chart Label Description

Storage I/O Control Aggregate IOPs Number of I/O operations per second on the datastore, aggregated across all hosts, and

virtual machines accessing the datastore.

Counter: datastoreIops

Stats Type: Absolute

Unit: Number

Rollup Type: Average

Collection Level: 3

VMware, Inc. 35

Page 36

vSphere Monitoring and Performance

Storage I/O Control Activity

This chart displays the percentage of time during which Storage I/O Control actively controlled latency of

the datastore.

This chart is located in the Performance views of the datastore Performance tabs. The counter can also

be displayed for datastore cluster charts.

Table 1‑34. Data Counters

Chart Label Description

Storage I/O Control

Activity

This is the percentage of time during which the Storage I/O Control actively controlled the I/O latency

for the datastore.

Counter: siocActiveTimePercentage

Stats Type: Absolute

Unit: Percent

Rollup Type: Average

Collection Level: 3

Average Device Latency per Host

This chart displays the average amount of latency on a host device. The chart displays the 10 hosts with

the highest device latency.

This chart is located in the Performance view of the datastore Performance tab.

Table 1‑35. Data Counters

Chart Label Description

Average Device Latency

per Host

Measures the amount of time, in milliseconds, to complete a SCSI command issued from the physical

device.

Counter: deviceLatency

Stats Type: Absolute

Unit: Milliseconds (ms)

Rollup Type: Average

Collection Level: 3

Maximum Queue Depth per Host

This chart displays maximum queue depth that hosts are currently maintaining for the datastore. When

Storage I/O is enabled, queue depth can change over time when congestion is detected at the array.

This chart is located in the Performance view of the datastore Performance tab. The chart displays

information about the ten hosts with the highest values.

VMware, Inc. 36

Page 37

vSphere Monitoring and Performance

Table 1‑36. Data Counters

Chart Label Description

Max Queue Depth per Host Maximum queue depth. Queue depth is the number of commands the SCSI driver queues to

the HBA.

Counter: maxQueueDepth

Stats Type: Absolute

Unit: Number

Rollup Type: Average

Collection Level: 3

Read IOPs per Host

This chart displays the per-host disk read rates for a datastore. The chart displays information about the

ten hosts with the highest values.

This chart is located in the Performance view of the datastore Performance tab.

Table 1‑37. Data Counters

Chart Label Description

Read IOPs per Host Number of disk read commands completed on each disk on the host, per second.

Read rate = blocks read per second × block size

Counter: numberReadAveraged

Stats Type: Rate

Unit: Number

Rollup Type: Average

Collection Level: 3

Write IOPs Per Host

This chart displays the per-host disk write rates for a datastore. The chart displays information about the

10 hosts with the highest values.

This chart is located in the Performance view of the datastore Performance tab.

Table 1‑38. Data Counters

Chart Label Description

Write IOPs per Host Number of disk write commands completed on each disk on the host, per second.

Write rate = blocks written per second × block size

Counter: numberWriteAveraged

Stats Type: Rate

Unit: Number

Rollup Type: Average

Collection Level: 3

VMware, Inc. 37

Page 38

vSphere Monitoring and Performance

Average Read Latency per Virtual Machine Disk

This chart displays the top ten virtual machine disks with the highest average read latency in milliseconds.

Data is not displayed when the virtual machine is powered off.

This chart is located in the Performance view of the datastore Performance tab. The counter can also

be displayed for datastore cluster charts.

Table 1‑39. Data Counters

Chart Label Description

Average Read Latency

per Virtual Machine Disk

Latency measures the time used to process a SCSI command issued by the guest OS to the virtual

machine. The kernel latency is the time VMkernel takes to process an I/O request. The device latency

is the time it takes the hardware to handle the request.

Total latency = kernelLatency + deviceLatency.

Counter: totalReadLatency

Stats Type: Absolute

Unit: Milliseconds (ms)

Rollup Type: Average

Collection Level: 3

Average Write Latency per Virtual Machine Disk

This chart displays the top ten virtual machine disks with the highest average write latency in

milliseconds. Data is not displayed when the virtual machine is powered off.

This chart is located in the Performance view of the datastore Performance tab. The counter can also

be displayed for datastore cluster charts.

Table 1‑40. Data Counters

Chart Label Description

Average Write Latency

per Virtual Machine Disk

Latency measures the time used to process a SCSI command issued by the guest OS to the virtual

machine. The kernel latency is the time VMkernel takes to process an I/O request. The device latency

is the time it takes the hardware to handle the request.

Total latency = kernelLatency + deviceLatency.

Counter: totalWriteLatency

Stats Type: Absolute

Unit: Milliseconds (ms)

Rollup Type: Average

Collection Level: 3

Read IOPs per Virtual Machine Disk

This chart displays the top ten virtual machines with the highest number of read operations. Data is not

displayed when the virtual machine is powered off.

This chart is located in the Performance view of the datastore Performance tab. The counter can also

be displayed for datastore cluster charts.

VMware, Inc. 38

Page 39

vSphere Monitoring and Performance

Table 1‑41. Data Counters

Chart Label Description

Read IOPs per Virtual

Machine Disk

Number of disk read commands completed on each virtual machine disk, per second.

Read rate = blocks read per second × block size

Counter: numberReadAveraged

Stats Type: Rate

Unit: Number

Rollup Type: Average

Collection Level: 3

Write IOPs Per Virtual Machine Disk

This chart displays the 10 virtual machines with the highest number of write operations. Data is not

displayed when the virtual machine is powered off.

This chart is located in the Performance view of the datastore Performance tab. The counter can also

be displayed for datastore cluster charts.

Table 1‑42. Data Counters

Chart Label Description

Write IOPs per Virtual

Machine Disk

Number of disk write commands completed on each virtual machine disk on the host.

Write rate = blocks read per second × block size

Counter: numberWriteAveraged

Stats Type: Rate

Unit: Number

Rollup Type: Average

Collection Level: 3

Virtual Machine Observed Latency per Datastore

This chart displays the average datastore latency as observed by the virtual machines.

This chart is located in the Performance view of the datastore cluster Performance tab.

Table 1‑43. Data Counters

Chart Label Description

VM observed latency

report per Datastore

VMware, Inc. 39

This is the average datastore latency as observed by the virtual machines in the datastore cluster.

Counter: datastoreVMObservedLatency

Stats Type: Absolute

Unit: Microseconds

Rollup Type: Latest

Collection Level: 3

Page 40

vSphere Monitoring and Performance

Hosts

The hosts charts contain information about CPU, disk, memory, network, and storage usage for hosts.

The help topic for each chart contains information about the data counters displayed in that chart. The

counters available are determined by the collection level set for vCenter Server.

CPU (%)

The CPU (%) chart displays CPU usage for the host.

This chart is located in the Home view of the Host Performance tab.

Table 1‑44. Data Counters

Chart Label Description

Usage Actively used CPU, as a percentage of the total available CPU, for each physical

CPU on the host.

Active CPU is approximately equal to the ratio of the used CPU to the available CPU.

Available CPU = # of physical CPUs × clock rate.

100% represents all CPUs on the host. For example, if a four-CPU host is running a

virtual machine with two CPUs, and the usage is 50%, the host is using two CPUs

completely.

Counter: usage

Stats Type: Rate

Unit: Percentage (%)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of the host resources. However, if

the value is constantly high, the host is probably lacking the CPU required to meet the demand. A high

CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the

host.

If performance is impacted, consider taking the following actions.

Table 1‑45. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on every virtual machine on the host.

2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles required.

3 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For example, a

single-threaded application on a four-way virtual machine only benefits from a single vCPU. But the hypervisor's maintenance

of the three idle vCPUs takes CPU cycles that could be used for other work.

4 If the host is not already in a DRS cluster, add it to one. If the host is in a DRS cluster, increase the number of hosts and

migrate one or more virtual machines onto the new host.

VMware, Inc. 40

Page 41

vSphere Monitoring and Performance

Table 1‑45. CPU Performance Enhancement Advice (Continued)

# Resolution

5 Upgrade the physical CPUs or cores on the host if necessary.

6 Use the newest version of hypervisor software, and enable CPU-saving features such as TCP Segmentation Offload, large

memory pages, and jumbo frames.

CPU (MHz)

The CPU (MHz) chart displays CPU usage for the host.

This chart is located in the Home view of the Host Performance tab.

Table 1‑46. Data Counters

Chart Label Description

Usage The sum, in megahertz, of the actively used CPU of all powered on virtual machines

on a host.

The maximum possible value is the frequency of the processors multiplied by the

number of processors. For example, if you have a host with four 2GHz CPUs running a

virtual machine that is using 4000MHz, the host is using two CPUs completely.

4000 ÷ (4 × 2000) = 0.50

Counter: usagemhz

Stats Type: Rate

Unit: MegaHertz (MHz)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of the host resources. However, if

the value is constantly high, the host is probably lacking the CPU required to meet the demand. A high

CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the

host.

If performance is impacted, consider taking the following actions.

Table 1‑47. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on every virtual machine on the host.

2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles required.

3 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For example, a

single-threaded application on a four-way virtual machine only benefits from a single vCPU. But the hypervisor's maintenance

of the three idle vCPUs takes CPU cycles that could be used for other work.

4 If the host is not already in a DRS cluster, add it to one. If the host is in a DRS cluster, increase the number of hosts and

migrate one or more virtual machines onto the new host.

VMware, Inc. 41

Page 42

vSphere Monitoring and Performance

Table 1‑47. CPU Performance Enhancement Advice (Continued)

# Resolution

5 Upgrade the physical CPUs or cores on the host if necessary.

6 Use the newest version of hypervisor software, and enable CPU-saving features such as TCP Segmentation Offload, large

memory pages, and jumbo frames.

CPU Usage

The CPU Usage chart displays CPU usage of the 10 virtual machines on the host with the most CPU

usage.

This chart is located in the Virtual Machines view of the host Performance tab.

Table 1‑48. Counters

Name Description

virtual_machine Amount of CPU actively being used by each virtual machine on the host. 100% represents all CPUs.

For example, if a virtual machine has one virtual CPU that is running on a host with four CPUs and

the CPU usage is 100%, the virtual machine is using one CPU resource.

virtual CPU usage = usagemhz ÷ (number of virtual CPUs × core frequency)

Note The host's view of the CPU usage, not the guest operating system view.

Counter: usage

Stats Type: Rate

Unit: Percentage (%). Precision is to 1/100%. A value between 0 and 100.

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

A short spike in CPU usage or CPU ready indicates that you are making the best use of the virtual

machine resources. However, if the CPU usage value for a virtual machine is above 90% and the CPU

ready value is above 20%, performance is being impacted.

If performance is impacted, consider taking the following actions.

Table 1‑49. CPU Performance Enhancement Advice

# Resolution

1 Verify that VMware Tools is installed on every virtual machine on the host.

2 Set the CPU reservations for all high-priority virtual machines to guarantee that they receive the CPU cycles required.

3 Compare the CPU usage value of a virtual machine with the CPU usage of other virtual machines on the host or in the

resource pool. The stacked line chart on the host's Virtual Machine view shows the CPU usage for virtual machines on the

host.

4 Determine whether the high ready time for the virtual machine resulted from its CPU usage time reaching the CPU limit

setting. If so, increase the CPU limit on the virtual machine.

5 Increase the CPU shares to give the virtual machine more opportunities to run. The total ready time on the host might remain

at the same level if the host system is constrained by CPU. If the host ready time doesn't decrease, set the CPU reservations

for high-priority virtual machines to guarantee that they receive the required CPU cycles.

VMware, Inc. 42

Page 43

vSphere Monitoring and Performance

Table 1‑49. CPU Performance Enhancement Advice (Continued)

# Resolution

6 Increase the amount of memory allocated to the virtual machine. This decreases disk and or network activity for applications

that cache. This might lower disk I/O and reduce the need for the host to virtualize the hardware. Virtual machines with

smaller resource allocations generally accumulate more CPU ready time.

7 Reduce the number of virtual CPUs on a virtual machine to only the number required to execute the workload. For example, a

single-threaded application on a four-way virtual machine only benefits from a single vCPU. But the hypervisor's maintenance

of the three idle vCPUs takes CPU cycles that could be used for other work.

8 If the host is not already in a DRS cluster, add it to one. If the host is in a DRS cluster, increase the number of hosts and

migrate one or more virtual machines onto the new host.

9 Upgrade the physical CPUs or cores on the host if necessary.

10 Use the newest version of hypervisor software, and enable CPU-saving features such as TCP Segmentation Offload, large

memory pages, and jumbo frames.

Disk (KBps)

The Disk (KBps) chart displays disk I/O of the host.

This chart is located in the Home view of the host Performance tab.

Table 1‑50. Data Counters

Chart Label Description

Usage Average data I/O rate across all LUNs on the host.

Counter: usage

Stats Type: Rate

Unit: Kilobytes per second (KBps)

Rollup Type: Average (Minimum/Maximum)

Collection Level: 1 (4)

Chart Analysis

Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example,

you might notice a performance degradation with applications that frequently read from and write to the

hard disk. If you see a spike in the number of disk read or write requests, check whether any such

applications were running then.

The best ways to determine if your vSphere environment is experiencing disk problems is to monitor the

disk latency data counters. You can use the advanced performance charts to view these statistics.

The kernelLatency data counter measures the average amount of time, in milliseconds, that the

VMkernel spends processing each SCSI command. For best performance, the value must be 0 -1

milliseconds. If the value is greater than 4 ms, the virtual machines on the host are trying to send

more throughput to the storage system than the configuration supports. Check the CPU usage, and

increase the queue depth.

VMware, Inc. 43

Page 44

vSphere Monitoring and Performance

The deviceLatency data counter measures the average amount of time, in milliseconds, to complete a

SCSI command from the physical device. Depending on your hardware, a number greater than 15 ms

indicates probable problems with the storage array. Move the active VMDK to a volume with more

spindles or add disks to the LUN.

The queueLatency data counter measures the average amount of time taken per SCSI command in

the VMkernel queue. This value must always be zero. If not, the workload is too high and the array

cannot process the data fast enough.

If the disk latency values are high, or if you notice other problems with disk I/O performance, consider

taking the following actions.

Table 1‑51. Disk I/O Performance Enhancement Advice