VMware ESXI - 6.7 Instruction Manual

Name: VMware ESXi - 6.7
Brand: VMware
SKU: 2840574
Rating: 4 (3 reviews)

4 (3)

vSphere Monitoring and

Performance

17 APR 2018

VMware vSphere 6.7

VMware ESXi 6.7

vCenter Server 6.7

vSphere Monitoring and Performance

You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to docfeedback@vmware.com

VMware, Inc.

3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

VMware, Inc.

About vSphere Monitoring and Performance 5
1 Monitoring Inventory Objects with Performance Charts		7
Performance Chart Types	8
Data Counters 8
Metric Groups in vSphere	10
Data Collection Intervals	10
Data Collection Levels 11
View Performance Charts	12
Performance Charts Options Available Under the View Menu		13
Overview Performance Charts 14
Working with Advanced and Custom Charts 106
Troubleshoot and Enhance Performance 109

2	Monitoring Guest Operating System Performance							116
	Enable Statistics Collection for Guest Operating System Performance Analysis 116
	View Performance Statistics for Windows Guest Operating Systems 116
3	Monitoring Host Health Status			118
	Monitor Health Status in the vSphere Client					119
	Monitor Hardware Health Status			119
	Reset Health Status Sensors in the vSphere Client						120
	Reset Health Status Sensors		120
4	Monitoring vSphere Health		121
	Check vSphere Health in vSphere Client				121
5	Monitoring Events, Alarms, and Automated Actions							123
	View Events 125
	View System Logs	125
	Export Events Data	126
	Streaming Events to a Remote Syslog Server					126
	Retention of Events in the vCenter Server Database						128
	View Triggered Alarms and Alarm Definitions					130
	Live Refresh of Recent Tasks and Alarms				130
	Set an Alarm in the vSphere Web Client				131
	Set an Alarm in the vSphere Client 141
	Acknowledge Triggered Alarms			144

VMware, Inc.

vSphere Monitoring and Performance

Reset Triggered Event Alarms	144
Preconfigured vSphere Alarms	145

6 Monitoring Solutions with the vCenter Solutions Manager 151

View Solutions and vServices 151 Monitoring Agents 152 Monitoring vServices 152

7	Monitoring the Health of Services and Nodes 154
	View the Health Status of Services and Nodes 154
8	Performance Monitoring Utilities: resxtop and esxtop 156
	Using the esxtop Utility		156
	Using the resxtop Utility		157
	Using esxtop or resxtop in Interactive Mode 158
	Using Batch Mode	173
	Using Replay Mode	174

9 Using the vimtop Plug-In to Monitor the Resource Use of Services 177

Monitor Services By Using vimtop in Interactive Mode 177 Interactive Mode Command-Line Options 177

Interactive Mode Single-Key Commands for vimtop 178

10 Monitoring Networked Devices with SNMP and vSphere 180

Using SNMP Traps with vCenter Server 180 Configure SNMP for ESXi 182

SNMP Diagnostics 193

Monitor Guest Operating Systems with SNMP 193 VMware MIB Files 193

SNMPv2 Diagnostic Counters 195

11 System Log Files		197
View System Log Entries			197
View System Logs on an ESXi Host 197
System Logs	198
Export System Log Files			199
ESXi Log Files	200
Upload Logs Package to a VMware Service Request 200
Configure Syslog on ESXi Hosts 201
Configuring Logging Levels for the Guest Operating System 202
Collecting Log Files		204

Viewing Log Files with the Log Browser 208

VMware, Inc.

About vSphere Monitoring and

Performance

VMware provides several tools to help you monitor your virtual environment and to locate the source of potential issues and current problems.

Performance charts	Allow you to see performance data on a variety of system resources
	including CPU, Memory, Storage, and so on.
Performance	Allow you to access detailed information on system performance through
monitoring command-	the command line.
line utilities
Host health	Allows you to quickly identify which hosts are healthy and which are
	experiencing problems.
Events, alerts, and	Allow you to configure alerts and alarms and to specify the actions the
alarms	system should take when they are triggered.
System Log Files	System logs contain additional information about activities in your vSphere
	environment.

Intended Audience

The content in this section is intended for vSphere administrators who perform the following tasks:

nMonitor the health and performance of physical hardware backings for the virtual environment.

nMonitor the health and performance of virtual devices in the virtual environment.

nTroubleshoot problems in the system.

nConfigure alarms.

nConfigure SNMP messages.

Virtual machine administrators also might find the section on Chapter 2 Monitoring Guest Operating System Performance helpful.

vSphere Web Client and vSphere Client

Instructions in this guide reflect the vSphere Client (an HTML5-based GUI). You can also use the instructions to perform most of the tasks by using the vSphere Web Client (a Flex-based GUI).

VMware, Inc.

vSphere Monitoring and Performance

Tasks for which the workflow differs significantly between the vSphere Client and the vSphere Web Client have duplicate procedures that provide steps according to the respective client interface. The procedures that relate to the vSphere Web Client, contain vSphere Web Client in the title.

Note In vSphere 6.7, most of the vSphere Web Client functionality is implemented in the vSphere Client. For an up-to-date list of the unsupported functionality, see Functionality Updates for the vSphere Client.

VMware Technical Publications Glossary

VMware Technical Publications provides a glossary of terms that might be unfamiliar to you. For definitions of terms as they are used in VMware technical documentation, go to http://www.vmware.com/support/pubs.

VMware, Inc.

Monitoring Inventory Objects	1
with Performance Charts

The vSphere statistics subsystem collects data on the resource usage of inventory objects. Data on a wide range of metrics is collected at frequent intervals, processed, and archived in the vCenter Server database. You can access statistical information through command-line monitoring utilities or by viewing performance charts in the vSphere Web Client.

Counters and Metric Groups

vCenter Server systems and hosts use data counters to query for statistics. A data counter is a unit of information relevant to a given inventory object or device. Each counter collects data for a different statistic in a metric group. For example, the disk metric group includes separate data counters to collect data for disk read rate, disk write rate, and disk usage. Statistics for each counter are rolled up after a specified collection interval. Each data counter consists of several attributes that are used to determine the statistical value collected.

For a complete list and description of performance metrics, see the vSphere API Reference.

Note Counters that are introduced in later versions might not contain data from hosts of earlier versions. For details, see the VMware Knowledge Base.

Collection Levels and Collection Intervals

Collection levels determine the number of counters for which data is gathered during each collection interval. Collection intervals determine the time period during which statistics are aggregated, calculated, rolled up, and archived in the vCenter Server database. Together, the collection interval and collection level determine how much statistical data is collected and stored in your vCenter Server database.

Data Availability

Real-time data appears in the performance charts only for hosts and virtual machines that are powered on. Historical data appears for all supported inventory objects, but might be unavailable during certain circumstances.

VMware, Inc.

vSphere Monitoring and Performance

This chapter includes the following topics:

nPerformance Chart Types

nData Counters

nMetric Groups in vSphere

nData Collection Intervals

nData Collection Levels

nView Performance Charts

nPerformance Charts Options Available Under the View Menu

nOverview Performance Charts

nWorking with Advanced and Custom Charts

nTroubleshoot and Enhance Performance

Performance Chart Types

Performance metrics are displayed in different types of charts, depending on the metric type and object.

Table 1 1. Performance Chart Types

Chart Type	Description

Line chart	Displays metrics for a single inventory object. The data for each performance counter is plotted on a separate
	line in the chart. For example, a network chart for a host can contain two lines: one showing the number of
	packets received, and one showing the number of packets transmitted.

Bar chart	Displays storage metrics for datastores in a selected data center. Each datastore is represented as a bar in the
	chart. Each bar displays metrics based on the file type: virtual disks, snapshots, swap files, and other files.

Pie chart	Displays storage metrics for a single object, based on the file types, or virtual machines. For example, a pie
	chart for a datastore can display the amount of storage space occupied by the virtual machines taking up the
	largest space.

Stacked chart	Displays metrics for the child objects that have the highest statistical values. All other objects are aggregated,
	and the sum value is displayed with the term Other. For example, a host's stacked CPU usage chart displays
	CPU usage metrics for the 10 virtual machines on the host that are consuming the most CPU. The Other
	amount contains the total CPU usage of the remaining virtual machines.
	The metrics for the host itself are displayed in separate line charts.
	Stacked charts are useful in comparing the resource allocation and usage across multiple hosts or virtual
	machines. By default, the 10 child objects with the highest data counter values are displayed.

Data Counters

Each data counter includes several attributes that are used to determine the statistical value collected. See the vSphere API Reference for a complete list and description of supported counters.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 2. Data Counter Attributes

Attribute	Description

Unit of Measurement	Standard in which the statistic quantity is measured.
	n Kilobytes (KB) – 1024 bytes
	n Kilobytes per second (KBps) – 1024 bytes per second
	n Kilobits (kb) – 1000 bits
	n Kilobits per second (kbps) – 1000 bits per second
	n	Megabytes (MB)
	n Megabytes per second (MBps)
	n Megabits (Mb), megabits per second (Mbps)
	n	Megahertz (MHz)
	n	Microseconds (µs)
	n	Milliseconds (ms)
	n	Number (#)
	n	Percent (%)
	n	Seconds (s)

Description	Text description of the data counter.

Statistics Type	Measurement used during the statistics interval. Related to the unit of measurement.
	n Rate – Value over the current statistics interval
	n Delta – Change from previous statistics interval.
	n Absolute – Absolute value (independent of the statistics interval).

Rollup Type	Calculation method used during the statistics interval to roll up data. Determines the type of statistical
	values that are returned for the counter.

nAverage – Data collected during the interval is aggregated and averaged.

nMinimum – The minimum value is rolled up.

nMaximum – The maximum value is rolled up.

The Minimum and Maximum values are collected and displayed only in statistics level 4. Minimum and maximum rollup types are used to capture peaks in data during the interval. For real-time data, the value is the current minimum or current maximum. For historical data, the value is the average minimum or average maximum.

For example, the following information for the CPU usage chart shows that the average is collected at statistics level 1. The minimum and maximum values are collected at statistics level 4.

n Counter: usage

n Unit: Percentage (%)

n Rollup Type: Average (Minimum/Maximum) n Collection Level: 1 (4)

nSummation – Data collected is summed. The measurement displayed in the chart represents the sum

of data collected during the interval.

nLatest – Data collected during the interval is a set value. The value displayed in the performance charts represents the current value.

Collection level	Number of data counters used to collect statistics. Collection levels range from 1 to 4, with 4 having the
	most counters.

	Note Be careful when you set a higher collection level, as the process requires significant increase of
	resource usage. For more information, see Data Collection Levels.

VMware, Inc.

vSphere Monitoring and Performance

Metric Groups in vSphere

The performance data collection subsystem for vSphere collects performance data on various inventory items and their devices. Data counters define individual performance metrics. Performance metrics are organized into logical groups based on the object or object device. Statistics for one or more metrics can be displayed in a chart.

Table 1 3. Metric Groups

Metric group	Description

Cluster Services	Performance statistics for clusters configured by using vSphere Distributed Resource Scheduler, vSphere High
	Availability, or both.

CPU	CPU utilization per host, virtual machine, resource pool, or compute resource.

Datastore	Statistics for datastore utilization.

Disk	Disk utilization per host, virtual machine, or datastore. Disk metrics include I/O performance, such as latency and
	read/write speeds, and utilization metrics for storage as a finite resource.

Memory	Memory utilization per host, virtual machine, resource pool, or compute resource. The value obtained is one of
	the following:
	n For virtual machines, memory refers to the guest physical memory. Guest physical memory is the amount of
	physical memory presented as a virtual-hardware component to the virtual machine, at creation time, and
	made available when the virtual machine is running.
	n For hosts, memory refers to the machine memory. Machine memory is the RAM that is installed on the
	hardware that comprises the host.

Network	Network utilization for both physical and virtual network interface controllers (NICs) and other network devices.
	The virtual switches that support connectivity among all components, such as hosts, virtual machines, VMkernel.

Power	Energy usage statistics per host.

Storage Adapter	Data traffic statistics per host bus adapter (HBA).

Storage Path	Data traffic statistics per path.

System	Overall system availability, such as the system heartbeat and uptime. These counters are available directly from
	hosts and from vCenter Server.

Virtual Disk	Disk utilization and disk performance metrics for virtual machines.

Virtual Flash	Virtual flash counters.

Virtual Machine	Virtual machine power and provisioning operations in a cluster or data center.
Operations

vSphere	Statistics for the virtual machine replication performed by VMware vCenter Site Recovery Manager.
Replication

Data Collection Intervals

Collection intervals determine the duration for which statistics are aggregated, calculated, rolled up, and archived. Together, the collection interval and collection level determine how much statistical data is gathered and stored in your vCenter Server database.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 4. Collection Intervals

Collection	Collection
Interval/Archive Length	Frequency	Default Behavior

1 Day	5 Minutes	Real-time statistics are rolled up to create one data point every 5 minutes. The
		result is 12 data points every hour and 288 data points every day. After 30
		minutes, the six data points collected are aggregated and rolled up as a data point
		for the 1-Week time range.
		You can change the interval duration and archive length of the 1-Day collection
		interval by configuring the statistics settings.

1 Week	30 Minutes	1-Day statistics are rolled up to create one data point every 30 minutes. The result
		is 48 data points every day and 336 data points every week. Every 2 hours, the 12
		data points collected are aggregated and rolled up as a data point for the 1-Month
		time range.
		You cannot change the default settings of the 1-Week collection interval.

1 Month	2 Hours	1-Week statistics are rolled up to create one data point every 2 hours. The result
		is 12 data points every day and 360 data points every month (assuming a 30-day
		month). After 24 hours, the 12 data points collected are aggregated and rolled up
		as a data point for the 1-Year time range.
		You cannot change the default settings of the 1-Month collection interval.

1 Year	1 Day	1-Month statistics are rolled up to create one data point every day. The result is
		365 data points each year.
		You can change the archive length of the 1-Year collection interval by configuring
		the statistics settings.

Note If you change the duration of data collection intervals you might need to allocate more storage resources.

Data Collection Levels

Each collection interval has a default collection level that determines the amount of data gathered and which counters are available for display in the charts. Collection levels are also referred to as statistics levels.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 5. Statistics Levels

Level	Metrics	Best Practice

Level 1	n	Cluster Services (VMware Distributed Resource Scheduler) – all
		metrics
	n CPU – cpuentitlement, totalmhz, usage (average), usagemhz
	n	Disk – capacity, maxTotalLatency, provisioned, unshared, usage
		(average), used

Use for long-term performance monitoring when device statistics are not required. Level 1 is the default Collection Level for all Collection Intervals.

nMemory – consumed, mementitlement, overhead, swapinRate, swapoutRate, swapused, totalmb, usage (average), vmmemctl (balloon)

nNetwork – usage (average), IPv6

nSystem – heartbeat, uptime

nVirtual Machine Operations – numChangeDS, numChangeHost, numChangeHostDS

Level 2	n Level 1 metrics
	n CPU – idle, reservedCapacity
	n Disk – All metrics, excluding numberRead and numberWrite.
	n Memory – All metrics, excluding memUsed and maximum and
	minimum rollup values.
	n Virtual Machine Operations – All metrics

Use for long-term performance monitoring when device statistics are not required but you want to monitor more than the basic statistics.

Level 3	n	Level 1 and Level 2 metrics
	n Metrics for all counters, excluding minimum and maximum rollup
		values.
	n	Device metrics

Use for short-term performance monitoring after encountering problems or when device statistics are required. Because of the large quantity of troubleshooting data retrieved and recorded, use level 3 for the shortest time period (Day or Week collection interval).

Level 4	All metrics supported by the vCenter Server, including minimum and
	maximum rollup values.

Note When you increase the collection level, the storage and system requirements might change. You might need to allocate more system resources to avoid a decrease in the performance.

View Performance Charts

The vCenter Server statistics settings, the type of object selected, and the features that are enabled on the selected object determine the amount of information displayed in charts. Charts are organized into views. You can select a view to see related data together on one screen. You can also specify the time range, or data collection interval. The duration extends from the selected time range to the present time.

VMware, Inc.

vSphere Monitoring and Performance

Overview charts display multiple data sets in one panel to evaluate different resource statistics, display thumbnail charts for child objects. It also displays charts for a parent and a child object. Advanced charts display more information than overview charts, are configurable, and can be printed or exported. You can export data in the PNG, JPEG, or CSV formats. See View Advanced Performance Charts.

Procedure

1Select a valid inventory object in the vSphere Web Client.

Overview and advanced performance charts are available for datacenter, cluster, host, resource pool, vApp, and virtual machine objects. Overview charts are also available for datastores and datastore clusters. Performance charts are not available for network objects.

2Click the Monitor tab, and click Performance.

3Select a view.

Available views depend on the type of object. For views that might contain many charts in a large environment, the vSphere Web Client displays the charts distributed on multiple pages. You can use the arrow buttons to navigate between pages.

4Select a predefined or custom time range.

Performance Charts Options Available Under the View

The performance chart options that you can access under the View menu vary depending on the type of inventory object you select.

For example, the Virtual Machines view is available when you view host performance charts only if there are virtual machines on the selected host. Likewise, the Fault Tolerance view for virtual machine performance charts is available only when that feature is enabled for the selected virtual machine.

Table 1 6. Performance Chart Views by Inventory Object

Object	View List Items

Data center	n Storage - space utilization charts for datastores in the data center, including space by file type and storage
		space used by each datastore in the data center.
	n Clusters - thumbnail CPU and memory charts for each cluster, and stacked charts for total CPU and
		memory usage in the data center. This view is the default.

Datastore and	n Space - space utilization charts for the datastore:
datastore cluster		n space utilization by file type
		n space utilization by virtual machine
		n space usage
	n Performance - performance charts for the datastore or datastore cluster and for virtual machine disks on
		the resource.

		Note The Performance view for datastores is only available when all hosts that are connected to the datastores
	are ESX/ESXi 4.1 or greater. The Performance view for datastore clusters is only available when the Storage
	DRS is enabled.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 6. Performance Chart Views by Inventory Object (Continued)

Object	View List Items

Cluster	n Home - CPU and memory charts for the cluster.
	n Resource Pools & Virtual Machines - thumbnail charts for resource pools and virtual machines, and
	stacked charts for total CPU and memory usage in the cluster.
	n Hosts - thumbnail charts for each host in the cluster, and stacked charts for total CPU, memory, disk usage,
	and network usage.

Host	n Home - CPU, memory, disk, and network charts for the host.
	n Virtual Machines - thumbnail charts for virtual machines, and stacked charts for total CPU usage and total
	memory usage on the host.

Resource Pool	n Home - CPU and memory charts for the resource pool.
and vApps	n Resource Pools & Virtual Machines - thumbnail charts for resource pools, and virtual machines and
	stacked charts for CPU and memory usage in the resource pool or vApp.

Virtual Machine	n Storage - space utilization charts for the virtual machine: space by file type, space by datastore, and total
	gigabytes.
	n Fault Tolerance - CPU and memory charts that display comparative metrics for the fault-tolerant primary
	and secondary virtual machines.
	n Home - CPU, memory, network, host (thumbnail charts), and disk usage charts for the virtual machine.

Overview Performance Charts

The overview performance charts display the most common metrics for an object in the inventory. Use these charts to monitor and troubleshoot performance problems.

The metrics provided in Overview performance charts are a subset of those collected for hosts and the vCenter Server. For a complete list of all metrics collected by hosts and the vCenter Server, see the vSphere API Reference.

Clusters

The cluster charts contain information about CPU, disk, memory, and network usage for clusters. The help topic for each chart contains information about the data counters displayed in that chart. The collection level set for vCenter Server determines the available counters.

CPU (MHz)

The CPU (MHz) chart displays CPU usage for the cluster.

Cluster Counters

This chart is located in the Home view of the Cluster Performance tab.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 7. Data Counters

Chart Label	Description

Usage	Sum of the average CPU usage values, in Megahertz, of all virtual machines in the cluster.
	n	Counter: usagemhz
	n	Stats Type: Rate
	n	Unit: Megahertz (MHz)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 1 (4)

Total	Total amount of CPU resources available in the cluster. The maximum value is equal to the number of
	cores multiplied by the frequency of the processors.
	For example, a cluster has two hosts, each of which has four CPUs that are 3 GHz each, and one virtual
	machine that has two virtual CPUs.
	VM totalmhz = 2 vCPUs * 3000 MHz = 6000 MHz
	Host totalmhz = 4 CPUs * 3000 MHz = 12000 MHz
	Cluster totalmhz = 2 x 4 * 3000 MHz = 24000 MHz
	n	Counter: totalmhz
	n	Stats Type: Rate
	n	Unit: Megahertz (MHz)
	n	Rollup Type: Summation
	n	Collection Level: 1

Chart Analysis

A short spike in CPU usage indicates that you are making the best use of cluster resources. However, if the value is constantly high, the CPU demanded is likely greater than the CPU capacity available. A high CPU usage value can lead to increased ready time and processor queuing of the virtual machines on the hosts in the cluster.

If performance is impacted, consider taking the following actions.

Table 1 8. CPU Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine.

2If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks: 1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4Click Turn ON vSphere DRS, and click OK.

3If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.

nCheck the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the cluster.

4Migrate one or more virtual machines to a new host.

5Upgrade the physical CPUs or cores on each host in the cluster if necessary.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 8. CPU Performance Enhancement Advice (Continued)

#Resolution

6Enable CPU-saving features, such as TCP Segmentation Offload.

7Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

CPU Usage

The cluster CPU Usage charts monitor the CPU utilization of the hosts, resource pools, and virtual machines in the cluster. This chart displays the 10 child objects in the cluster with the most CPU usage.

This chart is located in the Resource Pools and Virtual Machines view of the Cluster Performance tab.

Table 1 9. Data Counters

Chart Label	Description

<host>, <resource pool>, or

Amount of CPU actively used by the host, resource pool, or virtual machine in the cluster.

nCounter: usagemhz

nStats Type: Rate

nUnit: MegaHertz (MHz)

nRollup Type: Average (Minimum/Maximum)

nCollection Level: 1 (4)

Chart Analysis

If performance is impacted, consider taking the following actions.

Table 1 10. CPU Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine.

2If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks: 1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4Click Turn ON vSphere DRS, and click OK.

3If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.

nCheck the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the cluster.

4Migrate one or more virtual machines to a new host.

5Upgrade the physical CPUs or cores on each host in the cluster if necessary.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 10. CPU Performance Enhancement Advice (Continued)

#Resolution

6Enable CPU-saving features, such as TCP Segmentation Offload.

7Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

Disk (KBps)

The Disk (KBps) chart displays the disk I/O of the 10 hosts in the cluster with the most disk usage.

This chart is located in the Hosts view of the cluster Performance tab.

Table 1 11. Data Counters

Chart Label	Description

host_name	Average data I/O rate across all hosts in the cluster.
	n	Counter: usage
	n	Stats Type: Rate

n Unit: Kilobytes per second (KBps)

n Rollup Type: Average (Minimum/Maximum) n Collection Level: 1 (4)

Chart Analysis

Use the disk charts to monitor average disk loads and to determine trends in disk usage. For example, you might notice a performance degradation with applications that frequently read from and write to the hard disk. If you see a spike in the number of disk read or write requests, check whether any such applications were running then.

The best ways to determine if your vSphere environment is experiencing disk problems is to monitor the disk latency data counters. You can use the advanced performance charts to view these statistics.

nThe kernelLatency data counter measures the average amount of time, in milliseconds, that the VMkernel spends processing each SCSI command. For best performance, the value must be 0 -1 milliseconds. If the value is greater than 4 ms, the virtual machines on the host are trying to send more throughput to the storage system than the configuration supports. Check the CPU usage, and increase the queue depth.

nThe deviceLatency data counter measures the average amount of time, in milliseconds, to complete a SCSI command from the physical device. Depending on your hardware, a number greater than 15 ms indicates probable problems with the storage array. Move the active VMDK to a volume with more spindles or add disks to the LUN.

nThe queueLatency data counter measures the average amount of time taken per SCSI command in the VMkernel queue. This value must always be zero. If not, the workload is too high and the array cannot process the data fast enough.

If the disk latency values are high, or if you notice other problems with disk I/O performance, consider taking the following actions.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 12. Disk I/O Performance Enhancement Advice

#Resolution

1Increase the virtual machine memory. It allows more operating system caching, which reduces I/O activity. Note: It might require you to increase the host memory. Increasing memory might reduce the need to store data because databases can utilize the system memory to cache data and avoid disk access.

To verify that virtual machines have adequate memory, check swap statistics in the guest operating system. Increase the guest memory, but not to an extent that leads to excessive host memory swapping. Install VMware Tools so that memory ballooning can occur.

2Defragment the file systems on all guests.

3Disable antivirus on-demand scans on the VMDK and VMEM files.

4Use the vendor's array tools to determine the array performance statistics. When too many servers simultaneously access common elements on an array, the disks might have trouble keeping up. To increase throughput, consider array-side improvements.

5Use Storage vMotion to migrate I/O-intensive virtual machines across multiple hosts.

6Balance the disk load across all physical resources available. Spread heavily used storage across LUNs that are accessed by different adapters. Use separate queues for each adapter to improve disk efficiency.

7Configure the HBAs and RAID controllers for optimal use. Verify that the queue depths and cache settings on the RAID controllers are adequate. If not, increase the number of outstanding disk requests for the virtual machine by adjusting the Disk.SchedNumReqOutstanding parameter. For more information, see vSphere Storage.

8For resource-intensive virtual machines, separate the virtual machine's physical disk drive from the drive with the system page file. It alleviates disk spindle contention during periods of high use.

9On systems with sizable RAM, disable memory trimming by adding the line MemTrimRate=0 to the virtual machine's VMX file.

10If the combined disk I/O is higher than a single HBA capacity, use multipathing or multiple links.

11For ESXi hosts, create virtual disks as preallocated. When you create a virtual disk for a guest operating system, select Allocate all disk space now. The performance degradation associated with reassigning additional disk space does not occur, and the disk is less likely to become fragmented.

12Use the most current host hardware.

Memory (MB)

The Memory (MB) chart displays consumed memory for the cluster. The chart appears only at collection level 1.

This chart is located in the Home view of the cluster Performance tab.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 13. Data Counters

Chart Label	Description

Consumed	Amount of host machine memory used by all powered on virtual machines in the cluster. A cluster's
	consumed memory consists of virtual machine consumed memory and overhead memory. It does not
	include host-specific overhead memory, such as memory used by the service console or VMkernel.
	n	Counter: consumed
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 1 (4)

Total	Total amount of machine memory of all hosts in the cluster that is available for virtual machine memory
	(physical memory for use by the Guest OS) and virtual machine overhead memory.
	Memory Total = Aggregate host machine memory - (VMkernel memory + Service Console memory +
	other service memory)

	Note The totalmb data counter is the same as the effectivemem data counter, which is supported only
	for backward compatibility.
	n	Counter: totalmb
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)

n Rollup Type: Average (Minimum/Maximum)

n Collection Level: 1 (4)

Chart Analysis

Memory usage is not an indicator of performance problems. Memory can be high if a host is swapping or ballooning, which can result in virtual machine guest swapping. In such cases, check for other problems, such as CPU over-commitment or storage latencies.

If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking the following actions.

Table 1 14. Memory Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine.

2Verify that the balloon driver is enabled. The balloon driver is installed with VMware Tools and is critical to performance. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, it does not impact virtual machine performance.

3If the balloon value is high, check the resource shares, reservations, and limits for the virtual machines and resource pools on the hosts. Verify that the host's settings are adequate and not lower than those set for the virtual machine. If free memory is available on the hosts and the virtual machines are experiencing high swap or balloon memory, the virtual machine (or resource pool, if it belongs to one) has reached its resource limit. Check the maximum resource limit set on that host.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 14. Memory Performance Enhancement Advice (Continued)

#Resolution

4If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks: 1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4Click Turn ON vSphere DRS, and click OK.

5If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.

nCheck the aggressiveness threshold. If the value is low, increase the threshold. It might help avoid hot spots in the cluster.

6Add more physical memory to one or more hosts.

Memory (MB)

The Memory (MB) chart displays memory data counters for clusters. The chart appears at all collection levels except level 1.

Description

This chart is located in the Home view of the cluster Performance tab.

Note These data counter definitions are for hosts. At the cluster level, the values are collected and totaled. The counter values in the chart represent the aggregate amounts of the host data. The counters that appear in the chart depend on the collection level set for your vCenter Server.

Table 1 15. Data Counters

Chart Label	Description

Active	Sum of the active guest physical memory of all powered on virtual machines on the host,
	plus memory used by basic VMkernel applications. Active memory is estimated by the
	VMkernel.
	n	Counter: active
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 2 (4)

Balloon	Sum of the guest physical memory reclaimed by the balloon driver for all powered on virtual
	machines on the host.
	n	Counter: vmmemctl
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)

n Rollup Type: Average (Minimum/Maximum)

n Collection Level: 1 (4)

VMware, Inc.

vSphere Monitoring and Performance

Table 1 15. Data Counters (Continued)

Chart Label	Description

Consumed	Amount of machine memory used on the host.
	Consumed memory includes virtual machine memory, service console memory, and
	VMkernel memory.
	consumed memory = total host memory - free host memory
	n	Counter: consumed
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 1 (4)

Granted	Sum of the guest physical memory granted for all powered on virtual machines. Granted
	memory is mapped to the host's machine memory.
	Granted memory for a host includes the shared memory of each virtual machine on the host.
	n	Counter: granted
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 2 (4)

Swap Used	Sum of the memory swapped by all powered on virtual machines on the host.
	n	Counter: swapused
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)
	n Rollup Type: Average (Minimum/Maximum)
	n Collection Level: 2 (4)

Total	Aggregate total memory available to the cluster.
	n	Counter: totalmb
	n	Stats Type: Absolute
	n	Unit: Megabytes (MB)

n Rollup Type: Average (Minimum/Maximum)

n Collection Level: 1 (4)

Chart Analysis

To ensure best performance, the host memory must be large enough to accommodate the active memory of the virtual machines. The active memory can be smaller than the virtual machine memory size. It allows you to over-provision memory, but still ensures that the virtual machine active memory is smaller than the host memory.

Transient high-usage values usually do not cause performance degradation. For example, memory usage can be high when several virtual machines are started at the same time or when a spike occurs in virtual machine workload. However, a consistently high memory usage value (94% or greater) indicates that the host is probably lacking the memory required to meet the demand. If the active memory size is the same as the granted memory size, the demand for memory is greater than the memory resources available. If the active memory is consistently low, the memory size might be too large.

VMware, Inc.

vSphere Monitoring and Performance

If the memory usage value is high, and the host has high ballooning or swapping, check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host cannot handle the demand for memory. It leads to memory reclamation, which might degrade performance.

If the host has enough free memory, check the resource shares, reservation, and limit settings of the virtual machines and resource pools on the host. Verify that the host settings are adequate and not lower than those set for the virtual machines.

If the host has little free memory available, or if you notice a degradation in performance, consider taking the following actions.

Table 1 16. Memory Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine. The balloon driver is installed with VMware Tools and is critical to performance.

2Verify that the balloon driver is enabled. The VMkernel regularly reclaims unused virtual machine memory by ballooning and swapping. Generally, it does not impact virtual machine performance.

3Reduce the memory space on the virtual machine, and correct the cache size if it is too large. This frees up memory for other virtual machines.

4If the memory reservation of the virtual machine is set to a value much higher than its active memory, decrease the reservation setting so that the VMkernel can reclaim the idle memory for other virtual machines on the host.

5 Migrate one or more virtual machines to a host in a DRS cluster.

6Add physical memory to the host.

Memory Consumed

The Memory Consumed chart displays memory usage for the 10 child objects in the cluster with the most consumed memory.

For resource pools and virtual machines in a cluster, this chart is located in the Resource Pools & Virtual Machines view of the cluster Performance tab. For hosts in a cluster, this chart is located in the Hosts view of the cluster Performance tab.

Table 1 17. Data Counters

Chart Label	Description

resource_pool, virtual_machine, or host

Amount of machine memory used by all resource pools and virtual machines in the cluster or by all hosts in the cluster, depending on the cluster view.

Consumed memory includes virtual machine memory, service console memory, and VMkernel memory. consumed memory = total host memory - free host memory

nCounter: consumed

nStats Type: Absolute

nUnit: MegaBytes (MB)

nRollup Type: Average (Minimum/Maximum)

nCollection Level: 1 (4)

VMware, Inc.

vSphere Monitoring and Performance

Chart Analysis

If you have constantly high memory usage in a cluster, resource pool, or vApp, consider taking the following actions.

Table 1 18. Memory Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine.

4If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks: 1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4Click Turn ON vSphere DRS, and click OK.

5If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.

nCheck the aggressiveness threshold. If the value is low, increase the threshold. It might help avoid hot spots in the cluster.

6Add more physical memory to one or more hosts.

Network (Mbps)

The Network (Mbps) chart displays network speed for the 10 hosts in the cluster with the most network usage.

This chart is located in the Hosts view of the Cluster Performance tab.

Table 1 19. Data Counters

Chart Label	Description

<host>	Average rate at which data is transmitted and received across all NIC instances on the host.
	n	Counter: usage
	n	Stats Type: Rate

n Unit: Megabits per second (Mbps)

n Rollup Type: Average (Minimum/Maximum) n Collection Level: 1 (4)

VMware, Inc.

vSphere Monitoring and Performance

Chart Analysis

Network performance depends on the application workload and network configuration. Dropped network packets indicate a bottleneck in the network. To determine whether packets are being dropped, use esxtop or the advanced performance charts to examine the droppedTx and droppedRx network counter values.

If packets are being dropped, adjust the virtual machine shares. If packets are not being dropped, check the size of the network packets and the data receive and transfer rates. In general, the larger the network packets, the faster the network speed. When the packet size is large, fewer packets are transferred, which reduces the amount of CPU required to process the data. When network packets are small, more packets are transferred but the network speed is slower because more CPU is required to process the data.

Note In some instances, large packets might result in a high network latency. To check the network latency, use the VMware AppSpeed performance monitoring application or a third-party application.

If packets are not being dropped and the data receive rate is slow, the host is probably lacking the CPU resources required to handle the load. Check the number of virtual machines assigned to each physical NIC. If necessary, perform load balancing by moving virtual machines to different vSwitches or by adding more NICs to the host. You can also move virtual machines to another host or increase the host CPU or virtual machine CPU.

If you experience network-related performance problems, also consider taking the following actions.

Table 1 20. Networking Performance Enhancement Advice

#Resolution

1 Verify that VMware Tools is installed on each virtual machine.

2If possible, use vmxnet3 NIC drivers, which are available with VMware Tools. They are optimized for high performance.

3If virtual machines running on the same host communicate with each other, connect them to the same vSwitch to avoid transferring packets over the physical network.

4Assign each physical NIC to a port group and a vSwitch.

5Use separate physical NICs to handle the different traffic streams, such as network packets generated by virtual machines, iSCSI protocols, vMotion tasks.

6Ensure that the physical NIC capacity is large enough to handle the network traffic on that vSwitch. If the capacity is not enough, consider using a high-bandwidth physical NIC (10 Gbps). Alternatively, consider moving some virtual machines to a vSwitch with a lighter load or to a new vSwitch.

7If packets are being dropped at the vSwitch port, increase the virtual network driver ring buffers where applicable.

8Verify that the reported speed and duplex settings for the physical NIC match the hardware expectations and that the hardware is configured to run at its maximum capability. For example, verify that NICs with 1 Gbps are not reset to 100 Mbps because they are connected to an older switch.

9Verify that all NICs are running in full duplex mode. Hardware connectivity problems might result in a NIC resetting itself to a lower speed or half duplex mode.

10Use vNICs that are TCP Segmentation Offload (TSO)-capable, and verify that TSO-Jumbo Frames are enabled where possible.

VMware, Inc.

vSphere Monitoring and Performance

Data centers

The data center charts contain information about CPU, disk, memory, and storage usage for data centers. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server.

CPU (MHz)

The CPU (MHz) chart displays CPU usage for the 10 clusters in the data center with the most CPU usage.

This chart is located in the Clusters view of the Datacenters Performance tab.

Table 1 21. Data Counters

Chart Label	Description

<cluster>	Amount of CPU currently in use by the cluster. The active CPU usage is approximately equal to the ratio of
	the used CPU cycles to the available CPU cycles.
	The maximum possible value is the frequency of the processors multiplied by the number of cores. For
	example, a two-way SMP virtual machine using 4000MHz on a host that has four 2GHz processors is using
	50% of the CPU (4000 ÷ 4 × 2000) = 0.5).
	n	Counter: usagemhz
	n	Stats Type: Rate
	n	Unit: MegaHertz (MHz)

n Rollup Type: Average (Minimum/Maximum)

n Collection Level: 1 (4)

Chart Analysis

If performance is impacted, consider taking the following actions.

Table 1 22. CPU Performance Enhancement Advice

#Resolution

1Verify that VMware Tools is installed on each virtual machine.

2If the cluster is not a DRS cluster, enable DRS. To enable DRS, perform the following tasks: 1 Select the cluster, and click the Configure tab.

2 Under Services, click vSphere DRS.

3 click Edit.

An Edit Cluster Settings dialog box opens.

4Click Turn ON vSphere DRS, and click OK.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 22. CPU Performance Enhancement Advice (Continued)

#Resolution

3If the cluster is a DRS cluster:

n Increase the number of hosts, and migrate one or more virtual machines to the new host.

nCheck the aggressiveness threshold. If the value is low, increase the threshold. This might help avoid hot spots in the cluster.

4Migrate one or more virtual machines to a new host.

5 Upgrade the physical CPUs or cores on each host in the cluster if necessary.

6Enable CPU-saving features, such as TCP Segmentation Offload.

7Replace software I/O with the dedicated hardware, such as iSCSI HBAs or TCP Segmentation Offload NICs.

Memory (MB)

The Memory (MB) chart displays the average amount of consumed memory for the 10 clusters in the data center with the most consumed memory.

This chart is located in the Clusters view of the Datacenters Performance tab.

Table 1 23. Data Counters

Chart Label	Description

<cluster>	Amount of host machine memory used by all powered on virtual machines in the cluster.
	n	Counter: consumed
	n	Stats Type: Absolute
	n	Unit: MegaBytes (MB)

n Rollup Type: Average (Minimum/Maximum)

n Collection Level: 1 (4)

Chart Analysis

A cluster's consumed memory consists of virtual machine consumed memory and overhead memory. It does not include host-specific overhead memory, such as memory used by the service console or VMkernel.

If you experience problems with cluster memory usage, use the thumbnail cluster charts to examine memory usage for each cluster and increase memory resources if needed.

If the cluster is a DRS cluster, check the aggressiveness threshold. If the value is low, increase the threshold. Increasing the threshold might help avoid hot spots in the cluster.

Space used by Datastore in GB

The Space used by Datastore in GB chart displays the 10 datastores in the data center with the most used disk space.

This chart is located in the Storage view of the Datacenter Performance tab.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 24. Data Counters

Chart Label	Description

<datastore>	Amount of used storage space on the 10 datastores with the most used space.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: GigaBytes (GB)
	n	Rollup Type: Latest
	n	Collection Level: 1

Chart Analysis

The datastore is at full capacity when the used space is equal to the capacity. Allocated space can be larger than datastore capacity, for example, when you have snapshots and thin-provisioned disks. If possible, you can provision more space to the datastore, or you can add disks to the datastore or use shared datastores.

If snapshot files are consuming high datastore space, consider consolidating them to the virtual disk when they are no longer needed. Consolidating the snapshots deletes the redo log files and removes the snapshots from the vSphere Web Client user interface. For information about consolidating the data center, see the vSphere documentation.

Space Utilization By File Type

The Space Utilization By File Type chart displays datastore space usage for virtual disks, swap files, snapshot files, and other virtual machine files.

Note This chart does not show historical statistics. It only shows the most recently available data, which may be up to 30 minutes late, depending on when the last statistics rollup occurred. In addition, statistics are not collected across all datastores at one time. They are collected asynchronously.

The Space Utilization by File Type chart is located in the Storage view of the data center Performance tab.

VMware, Inc.

vSphere Monitoring and Performance

Datastore Counters

Table 1 25. Data Counters

File Type	Description

Virtual Disks	Amount of disk space used by virtual disk files.
	Virtual disk files store the contents of the virtual machine's hard disk drive. It includes information that you
	write to a virtual machine's hard disk, such as the operating system, program files, and data files. The files
	have the extension .vmdk and appear as a physical disk drive to a guest operating system.

	Note Delta disks, which also have an extension .vmdk, are not included in this file type.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n Collection Level: 1 (4)

Swap Files	Amount of disk space used by swap files.
	Swap files back up the virtual machine's physical memory.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n Collection Level: 1 (4)

Snapshots	Amount of disk space used by virtual machine snapshot files.
	Snapshot files store information about virtual machine snapshots. They include snapshot state files and
	delta disk files. A snapshot state file stores the running state of the virtual machine at the time of the
	snapshot. It has the extension .vmsn. A delta disk file stores the updates made by the virtual machine to
	the virtual disks after a snapshot is taken.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n Collection Level: 1 (4)

Other VM Files	Amount of disk space used by all other virtual machine files, such as configuration files and log files.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n Collection Level: 1 (4)

Other	Amount of disk space used by all other non-virtual machine files, such as documentation files and backup
	files.

Free Space	Amount of disk space not currently in use.

Total Space	Amount of disk space available to the datastore. It defines the datastore capacity. The chart displays the
	information for datastores but not for data centers.

total space = virtual disk space + swap file space + snapshot space + other VM file space + other space + free space

VMware, Inc.

vSphere Monitoring and Performance

Chart Analysis

Datastores and Datastore Clusters

The datastore charts contain information about disk usage for datastores or the datastores that are part of a cluster. The help topic for each chart contains information about the data counters displayed in that chart. The counters available are determined by the collection level set for vCenter Server.

Space in GB

The Space in GB chart displays space usage data counters for datastores.

This chart is located in the Space view of the datastore or datastore cluster Performance tab.

VMware, Inc.

vSphere Monitoring and Performance

Table 1 26. Data Counters

Chart Label	Description

Allocated	Amount of physical space provisioned by an administrator for the datastore. It is the storage size up to
	which files on the datastore can grow. Allocated space is not always in use.
	n	Counter: provisioned
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n	Collection Level: 1

Used	Amount of physical datastore space in use.
	n	Counter: used
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n	Collection Level: 1

Capacity	Maximum capacity of the datastore.
	capacity = virtual machine file space + non-virtual machine file space + free space

	Note Storage data is collected and updated in the overview charts every 30 minutes. Therefore, if you
	refresh the datastore, the capacity value might only be updated in the datastore Summary tab, and not in
	the overview charts.

	n	Counter: capacity
	n	Stats Type: Absolute
	n	Unit: Gigabytes (GB)
	n	Rollup Type: Latest
	n	Collection Level: 1

Chart Analysis

Space Utilization By File Type

The Space Utilization by File Type chart displays space used by virtual disks, swap files, snapshot files, and other virtual machine files on the datastore or the datastore cluster.

VMware, Inc.

+ 182 hidden pages

VMware ESXI - 6.7 Instruction Manual

Contents

Performance Chart Types

Data Counters

Metric Groups in vSphere

Data Collection Intervals

Data Collection Levels

View Performance Charts

Overview Performance Charts

Clusters

Data centers

Datastores and Datastore Clusters