VMware vRealize Configuration Manager - 5.8 Troubleshooting Guide

VMware vRealize Configuration Manager

Troubleshooting Guide

vRealize Configuration Manager 5.8

This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see

http://www.vmware.com/support/pubs.

EN-001818-00

VCM Troubleshooting Guide

You can find the most up-to-date technical documentation on the VMware Web site at:

http://www.vmware.com/support/

The VMware Web site also provides the latest product updates.

If you have comments about this documentation, submit your feedback to:

docfeedback@vmware.com

http://www.vmware.com/go/patents.

VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

VMware, Inc.

3401 Hillview Ave. Palo Alto, CA 94304 www.vmware.com

VMware, Inc.

About This Book 7

Troubleshooting Overview 9

Troubleshooting Workflow 9 Verifying That Behavior Is Negative 10 Isolating Behavior 10 Identifying External Factors 10 Checking VCM Logs 11

Info Messages 11 Warning Messages 11 Error Messages 12 Exception Messages 12

Types of Problems 13

User Interface Problems 13 Security and Authentication Problems 14 SQL Server Problems 14 VCM Agent Problems 15 UNIX Agent Problems 15 Report Server Problems 15 Internet Information Services Problems 16 Network Connectivity Problems 16 Hardware and Performance Problems 16

Gathering Diagnostic Information 19

What to Send to VMware Technical Support 19 Capture a Desktop Image 20 Capture a Window Image 20 Set the Debug Log to Store all Message Types 21 Extract the Debug Log 21 Extract SQL Server Logs 22 Collect IIS Logs 22 Collect ARS Files 23 Collect the UNIX Syslog Messages 24 Collect Import/Export Tool Logs 24 Extract Windows Event Logs 24 Extract Windows System Information 25 Collect UNIX ETL Logs 25 Collect VCM Installation Logs 25 Enable VCM Patching Logging 25 Collect VCM Patching Logs 26 Collect Agent Logging 26

Troubleshooting Problems with VCM 29

Patch Content Does Not Download for Red Hat and SUSEMachines 29 Signed Patch Content Cannot Be Validated 30 Mismatched Security Setting for AIX Patch Staging with NFS 30 UNIX Patch Deployment Fails 31 UNIX Patch Assessment Returns No Results 31

VMware, Inc.

VCM Troubleshooting Guide

Patch Deployment Jobs Might Time Out 32 UNIX Bulletins Missing from the Required Location 33 Report and Node Summary Errors 33 Report Parameter Errors 34 Protected Storage Errors 35 SSL Becomes Disabled 36

Troubleshooting the vSphere Client VCM Plug-In 37

vSphere Client VCM Plug-In Is Not Enabled 37 Cannot Register the vSphere Client VCM Plug-In 37 Invalid Certificate on a vSphere Client 38 Collector Not Running 38 HTTPS/SSL Is Not Configured on the Collector 38 Collection Unsuccessful 39 Machines Not Listed in the Collect Available List 39 Machines Not Listed in the Available List for Any Action 39 ESX Servers Are Not Displayed 39

VCM Windows Agent 41

Windows Agent Installation Environment 41

Windows Agent Installation Networking Requirements 41 Windows Agent Installation Collector Credentials 42

Windows Agent Installation Process 42

Detect Previous Install 42 Validate Installation Environment 43 Interrogate Target Environment 43 Resolve Uninstall Dependencies 44 Uninstall Module 44 Uninstall Module Installer 44 Install Simple Installer 44 Install Module Installer 46 Resolve All Versions of Modules Based on Highest Version Number 46 Install Module 47 Fully Release the Synchronization Lock on the Target Machine 47 Submit Request to Agent 47 Check If Request Is Complete 47 Transfer Request Results 47 Acknowledge Successful Data Transfer 47 Prepare Request Results for Insert 48 Insert Data Into Database 48 Transform Inserted Data 48 Cleanup Machine Data 48 Partially Release the Synchronization Lock on the Target Machine 48 Cleanup Request Data 48

Windows Agent Uninstallation Process 48

Detect Previous Install 49 Validate Installation Environment 49 Interrogate Target Environment 50 Resolve Uninstall Dependencies 50 Uninstall Module 50 Uninstall Module Installer 50 Fully Release the Synchronization Lock on the Target Machine 51 Partially Release the Synchronization Lock on the Target Machine 51

Cleanup Request Data 51 Windows Agent Upgrade Process 51 Windows Agent Manual Installation Process 52

No Manual Upgrade 52

VMware, Inc.

Contents

Manual Window Agent Installation 52 Windows Agent Communication Protocols 53 Communication Protocol Change Process 54

Detect Previous Install 54

Uninstall Agent 54

Uninstall Package Installer 54

Uninstall Basic Installer 55

Validate Installation Environment 55

Install Simple Installer 55

Store Installation Data in the Database 56

Install Module Installer 56

Fully Release the Synchronization Lock On the Target Machine 56

Submit Request to Agent 56

Check If Request Is Complete 56

Transfer Request Results 56

Acknowledge Successful Data Transfer 57

Prepare Request Results For Insert 57

Insert Data Into Database 57

Transform Inserted Data 57

Cleanup Machine Data 57

Partially Release the Synchronization Lock on the Target Machine 57

Cleanup Request Data 57 Debug Window Agent Installations 58

VCM UNIX Agent 59

UNIX Agent Directory Structure After Installation 59

/opt/CMAgent 59

/opt/CMAgent/Agent 60

/opt/CMAgent/CFC 61

/opt/CMAgent/data 63

/opt/CMAgent/data/db 63

/opt/CMAgent/data/db/DtmDB/RDM 63

/opt/CMAgent/data/db/PDS 63

/opt/CMAgent/data/db/SM/RDM 64

/opt/CMAgent/ECMu 64

/opt/CMAgent/ECMu/x.x/bin 65

/opt/CMAgent/ECMu/x.x/scripts 65

/opt/CMAgent/install 66

/opt/CMAgent/Installer 68

/opt/CMAgent/ThirdParty 68

/opt/CMAgent/ThirdParty/x.x/PatchAssessment 68

/opt/CMAgent/uninstall 69 Directories Created During an Inspection 69 Directory of Executed Scripts and Results 70 Collector Certificates 70 Patch Assessment 71 Exploratory UNIX Agent Troubleshooting 71

Installation Errors 71

Collector Cannot Contact the Agent 72

Agent is Unresponsive 72

Collections Return No Data 73

Monitoring Network Traffic 78

Index 81

VMware, Inc.

VCM Troubleshooting Guide

VMware, Inc.

About This Book

The VMware vRealize Configuration Manager Troubleshooting Guide explains problems that might occur with VMware vRealize Configuration Manager.

In addition, parts of this document describe how to find diagnostic information to help you or VMware Technical Support analyze problems.

Intended Audience

This information is for experienced Windows, Linux, UNIX, or Mac OS X system administrators who are familiar with managing network users and resources, and with performing system maintenance.

To use this information effectively, you must have a basic understanding of how to configure network resources, install software, and administer operating systems. You also need to fully understand your network topology and resource naming conventions.

Document Feedback

VMware welcomes your suggestions for improving our documentation. If you have comments, send your feedback to docfeedback@vmware.com.

Technical Support and Education Resources

The following technical support resources are available to you. To access the current version of this book and other books, go to http://www.vmware.com/support/pubs.

Online and Telephone Support

Support Offerings To find out how VMware support offerings can help meet your business

VMware Professional Services

VMware, Inc.

To use online support to submit technical support requests, view your product and contract information, and register your products, go to

http://www.vmware.com/support.

Customers with appropriate support contracts should use telephone support for priority 1 issues. Go to http://www.vmware.com/support/phone_

support.html.

needs, go to http://www.vmware.com/support/services.

VMware Education Services courses offer extensive hands-on labs, case study examples, and course materials designed to be used as on-the-job reference tools. Courses are available onsite, in the classroom, and live online. For onsite pilot programs and implementation best practices, VMware Consulting

VCM Troubleshooting Guide

Services provides offerings to help you assess, plan, build, and manage your virtual environment. To access information about education classes, certification programs, and consulting services, go to

http://www.vmware.com/services.

VMware, Inc.

Troubleshooting Overview

Before making changes to your environment to solve a troubleshooting situation, learn as much as possible about the problem.

This chapter includes the following topics:

Troubleshooting Workflow 9

Verifying That Behavior Is Negative 10

Isolating Behavior 10

Identifying External Factors 10

Checking VCM Logs 11

Troubleshooting Workflow

Many troubleshooting situations follow a workflow that might allow you to discover the cause of simple problems and correct them on your own.

In other cases, the workflow involves gathering information in the form of files or other evidence, and contacting VMware Technical Support for further assistance.

Figure 1–1. Troubleshooting Workflow

VMware, Inc.

VCM Troubleshooting Guide

Verifying That Behavior Is Negative

Sometimes, an apparent error might not be a real error.

n A message might seem like an error when it is only a warning.

n A designed behavior might seem like an error if the result is not what you expect.

For example, in compliance, VCM highlights noncompliant systems because they fail to meet rules, but this convention might be the opposite of your own thinking. In other words, you might define rules that look for incorrect settings to eliminate, instead of correct settings to keep. When that happens, VCM marks machines noncompliant that you thought were configured properly. VCM is performing as designed though: the machines failed to have the incorrect settings.

The best first step to troubleshooting is to verify that you have a real problem. Click the context-sensitive Help buttons on the Console or wizards, and determine whether a message is only a warning, or whether the suspect behavior is an operation that is running the way that it should.

Isolating Behavior

After determining that you have a real problem, try to reduce the number of factors that might be causing the problem.

Eliminate, one by one, conditions that might be contributing to the behavior until a minimum of factors are still present when the error occurs.

For example, if you see a problem on a single machine during a collection on 50 machines, with 10 data types per machine, the first step is to eliminate the 49 machines that did not exhibit the problem and rerun the collection only against the problem machine. You can usually determine which machine failed by viewing the details of the job from the Job History screen under Administration. Assuming that the problem still exists, you then eliminate data types, one or two at a time, until you find the data type that is causing the behavior. Again, the job details might indicate which data types fail.

With process of elimination, you can understand and document when the undesired behavior occurs and you can reproduce it consistently.

Identifying External Factors

Sometimes a behavior cannot be predicted or isolated in VCM alone. When that happens, note the environment at the time that the behavior occurs.

Performance

System load can indicate performance problems.. VCM processes a large amount of data, and the servers that run the database and main application are subject to periods of high resource consumption. For Windows machines, the Performance tab in Windows Task Manager provides this kind of information.

On the Collector server and the Agent machines, note the system load, memory usage, network traffic, time of day, other running applications, or other details that might decrease performance. With this approach, you might be able to find a specific time or a set of other conditions that must be present for the undesired behavior to occur.

For example, every time a collection stalls at a specific step in the process, you notice that a single CPU is running at 100% capacity. The problem might be caused by a number of factors, but knowing that the CPU always correlates with the behavior is a significant piece of external evidence.

VMware, Inc.

Operating System Logs

The operating system event log can identify external factors that might cause problems..You can detect simple problems by reviewing the security and application logs. Common errors include authentication problems:

Report Server (MSSQLSERVER) cannot connect to the report server database.

However, any message that occurs during the time frame surrounding the undesired behavior is suspect, especially if the same error or warning always accompanies the behavior.

Hardware

Errors and behaviors caused by hardware problems are among the most erratic. You might see a series of seemingly unrelated errors in sequence, or you might see the same error at random times. You might also see VCM operation degrade over time. This sort of irregular evidence might indicate a need for system hardware diagnostics.

Checking VCM Logs

During normal operation, VCM writes files with the .dbe extension to various locations, which store debug information about VCM.

The Collector writes additional debug information to the VCM database whenever jobs are running in the VCM Console.

Troubleshooting Overview

The Collector log is vital because it records all primary functions of VCM, and when VMware Technical Support engineers mention the debug log in the singular, they mean this Collector log.

Debug logs are a common troubleshooting tool for VMware Technical Support, so if you have called for assistance before, you might already be familiar with the process of gathering them. You might even recognize certain messages that appear in the logs. Even if that is the case, be aware that the debug logs were not designed for the average user. The messages were written to help engineers understand why VCM is operating the way that it is. You are free to review the logs yourself, but usually you forward the logs in their entirety to VMware Technical Support for analysis.

While it is not necessary to know every message that a debug log might contain, it is helpful to know about the types of messages.

Info Messages

Although VCM is not configured to save them by default, info messages are the most common message category. VCM usually discards info messagesto reduce debug file size, but you temporarily enable the saving of info messages to help you are look for clues related to a problem.

After enabling info messages, nearly every action that VCM performs is recorded in debug logs. Even though the messages are only informational, they provide a context for error behavior and can be important in solving a troubleshooting problem.

Warning Messages

Warnings indicate an unexpected situation but do not necessarily indicate a permanent problem. Many common situations such as network timeouts or authentication failures produce warnings.

VMware, Inc.

Warning messages usually do not prevent VCM from normal operation. They are intended to alert engineers about a possible problem.

VCM Troubleshooting Guide

Error Messages

Error messages indicate serious problems. VCM processes do not stop running because of an error, but VCM might not continue normally.

For example, if the collection of a specific data type produces a value that is not of the type that the database expects, VCM might have to discard the collected information for that entire data type. Discarding collected information produces an error message that indicates that something did not work properly. VCM can recover and continue, but the data you wanted for that collection is not correct.

Exception Messages

Exception messages indicate a problem that unexpectedly stops VCM because of a complete failure of the current process.

For example, the following exception might occur when there is a communication problem between the Collector and Agent.

A connection with the server could not be established HRESULT 0x80072efd;

VMware, Inc.

Types of Problems

VCM problems usually fall into common categories.

This chapter includes the following topics:

User Interface Problems 13

Security and Authentication Problems 14

SQL Server Problems 14

VCM Agent Problems 15

UNIX Agent Problems 15

Report Server Problems 15

Internet Information Services Problems 16

Network Connectivity Problems 16

Hardware and Performance Problems 16

User Interface Problems

Unwanted VCM behaviors often reveal themselves in the user interface, but the user interface is rarely the root cause.

VMware, Inc.

Common problems directly associated with the user interface are often display anomalies. Duplicate columns in the data grid, inappropriately enabled or disabled fields, and poor formatting are all examples of user interface problems. In this example, the root nodes of a report are incorrectly compressed into a space that is too narrow.

VCM Troubleshooting Guide

Figure 2–1. User Interface Formatting Problem

Security and Authentication Problems

VCM acts as a multiple-system administration tool and must have administrator authority on all the machines that it manages.

The necessary authority might be accidentally removed or restricted. Common authentication problems include changed passwords, dropped administrator access, or security measures such as a proxy server, that are added without updating VCM. In addition to managed machine access, VCM also requires access to its SQL Server database with the authority to insert, modify, and delete data.

Both kinds of access problems might reveal themselves in the user interface, in the VCM debug logs, or in the operating system event log:

The VCM Collector service failed to start due to the following error:

The service did not start due to a logon failure.

SQL Server Problems

Nearly all data associated with VCM is stored in its SQL Server databases.

Datab ase Description

VCM Contains the collected managed machine data gathered from VCM

VCM_Coll Stores information about the user interface, such as Collector settings

VCM_Raw Performance-enhancing database that temporarily holds collection data

Agents

and options.

before bulk insertion into the VCM database

VCM_UNIX Contains collected managed machine data from any UNIX Agents in

the environment

VMware, Inc.

SQL Server errors might include resource, disk space, or authentication problems, among others. Poor tuning of your SQL Server might also cause performance problems such as data bottlenecks. Evidence of SQL Server trouble might appear in the user interface or the debug logs, but messages directly related to SQL Server typically appear directly in the SQL Server logs:

INSERT statement conflicted with COLUMN FOREIGN KEY constraint 'fk_vcm_

sysdat_role_rules_role_id'. The conflict occurred in database 'VCM', table

'vcm_sysdat_rules', column 'rule_id'.

VCM Agent Problems

The VCM Agent is the locally installed mechanism by which VCM collects information from a managed machine. If the Agent is not functioning properly, collections from that machine might fail.

In the VCM Console, the Running Jobs display is the first place to check when looking for problems with an Agent. A typical error message for an Agent that cannot start collecting might be:

There was a problem parsing the agent instructions document.

The message indicates that a conflict occurred in the instruction set sent to the Agent, the Agent could not process its instructions, and collection failed as a result.

UNIX Agent Problems

Types of Problems

UNIX operates differently from Windows. To accommodate the differences, VCM uses a separate Agent for Linux or UNIX based managed machines.

Troubleshooting UNIX Agent problems might be easier than Windows, because certain operations available for Windows Agents are not available on the UNIX side. For example, compliance enforcement is not available for UNIX, so you do not need to investigate compliance messages when you troubleshoot on UNIX.

Report Server Problems

The Report Server is responsible for the graphical display of information in VCM and for VCM scheduled reports. A Report Server error usually appears in the user interface.

An error has occurred during report processing.

System.Web.Services.Protocols.SoapException: An error has occurred during

report processing -

Microsoft.Reporting.Services.Diagnostics.Utilities.RSException: An error has

occurred during report processing -

. . .

You can determine if a problem exists with the Report Server by navigating to the Report Server home page to see if the native interface for the Report Server is working. The URL is usually the name or IP address of the Report Server machine followed by Reports:

http://report-server-name-or-IP/Reports

VMware, Inc.

If the same error seen in the VCM user interface also occurs directly in the Report Server interface, the problem is almost certainly with the Report Server itself, and not with VCM. If the Report Server interface does not show the error, the cause is likely in VCM.

VCM Troubleshooting Guide

Internet Information Services Problems

The VCM user interface is hosted on Microsoft Internet Information Server (IIS), which might be incorrectly configured.

Sometimes the VCM user interface displays an error similar to what you see when you fail to connect to a Web page. Error messages such as 404 File Not Found or 403 Forbidden are typical IIS messages, and indicate improper configuration of IIS in relation to VCM or Report Server pages. You might also see ASP.NET errors:

Server object error 'ASP 0177:8007007e'

Server.CreateObject Failed

/vcm5/L1033/SumAdminDeployment.asp, line 68

8007007e

Network Connectivity Problems

The VCM Collector must have a network connection to all the Agents that send data. In addition, split installations require network connections between the separate VCM servers: Collector, SQL database, and Web server.

Failed network connections do not usually occur between all systems at the same time, so network issues are often easier to diagnose. A failed connection typically appears in the user interface as a failure of a VCM or VCM Patching job on a single or subset of managed machines. In the Jobs Detail display, a

PingFailed message indicates connectivity problems.

A way to test network connectivity is to ping the Agent from the Collector. From the Collector, it might be necessary to connect to the HTTP port of an Agent if any of the following are true:

n The environment contains a firewall.

n The Agent is a UNIX Agent.

n The Agent machine is using HTTP instead of the DCOM protocol for its primary communication

method.

To connect to the HTTP port, type telnet

agent-machi ne-name -or-IP

A successful command returns only a blank screen, which indicates that the managed machine has answered the connection request and awaits further instructions. You can break the connection and exit telnet by typing Ctrl+] and then quit.

If you send the telnet command and receive any other message (such as Connection refused or

Connection timeout) that is a good indication of a network problem. Failed network connections must

be resolved with the help of local support before further VCM troubleshooting. Otherwise, you cannot know whether or not the problems you are seeing are network related.

Hardware and Performance Problems

Hardware and performance problemsare among the most difficult to diagnose because they are often intermittent and random.

Sometimes, the only common denominator is that the problem is random. When that happens, look for possible hardware or performance evidence.

26542 at the command prompt.

Running out of disk space is the most common hardware problem.On the VCM servers, use the operating system file or disk management tools to check that enough disk space is available for the database and for the VCM application itself.

VMware, Inc.

Types of Problems

Next, make sure that enough memory and CPU cycles exist for VCM services and processes to start and continue properly. If VCM is competing with other, non-VCM processes, performance might degrade to the point that errors and exceptions appear in the debug log.

Finally,you might need to run diagnostics on hardware components, such as memory chips, processors, or system boards.

For the recommended hardware sizing and configuration needed to run VCM, see the installation documentation.

VMware, Inc.

VCM Troubleshooting Guide

VMware, Inc.

Gathering Diagnostic Information

To solve a problem, you need to collect information for your own analysis or for forwarding to VMware Technical Support.

This chapter includes the following topics:

What to Send to VMware Technical Support 19

Capture a Desktop Image 20

Capture a Window Image 20

Set the Debug Log to Store all Message Types 21

Extract the Debug Log 21

Extract SQL Server Logs 22

Collect IIS Logs 22

Collect ARS Files 23

Collect the UNIX Syslog Messages 24

Collect Import/Export Tool Logs 24

Extract Windows Event Logs 24

Extract Windows System Information 25

Collect UNIX ETL Logs 25

Collect VCM Installation Logs 25

Enable VCM Patching Logging 25

Collect VCM Patching Logs 26

Collect Agent Logging 26

What to Send to VMware Technical Support

To help VMware Technical Support analyze a VCM problem, you often need to gather and send files, exports of system logs, message text, or interface images.

Table 3–1. What to Send to VMware Technical Support

Type of Problem What to Send

User interface Screenshots

Security and authentication

VMware, Inc.

Screenshots Error messages VCM debug logs

VCM Troubleshooting Guide

Type of Problem What to Send

SQL Server SQL Server logs

VCM Agent Collector debug log

UNIX Agent Collector debug log

Report server Screenshots

IIS Screenshots

Network connectivity Network connectivity problems are usually investigated by local site

Windows system and application event logs VCM debug logs

Agent ARS files Windows system and application event logs

UNIX Agent debug log UNIX Agent ZRP files UNIX system log

SQL profiler trace files Collector debug log

Entries from IIS logs

support.

Hardware and performance

Capture a Desktop Image

Desktop screenshots capture the exact behavior that you see, in the broad context of your entire workspace.

Use desktop screenshots alone or in a series to capture error messages, changes in behavior over time, or to verify data that you enter in wizards or other interfaces.

Procedure

1. On the keyboard, press Print Screen (PrtScn).

2. Open a new message or document.

3. Press Ctrl+v to paste the image into the message or document.

Capture a Window Image

Window screenshots capture the exact behavior that you see, with the focus on one interface.

Use window screenshots alone or in a series to capture error messages, changes in behavior over time, or to verify data that you enter in wizards or other interfaces.

Procedure

Hardware and performance problems are usually investigated by local site support.

1. To bring it into focus, click the window that you want.

2. On the keyboard, press Alt+Print Screen (PrtScn).

3. Open a new message or document.

4. Press Ctrl+v to paste the image into the message or document.

VMware, Inc.

Set the Debug Log to Store all Message Types

Because Info messages might contain important troubleshooting clues, turn them on before extracting the log.

By default, the Collector debug log saves on performance and space by not storing Info messages. After extracting the debug log, turn Info messages back off.

CAUTION This procedure involves using the regedit command to open the Windows Registry and edit the settings.

Procedure

1. In the VCM Console, click Administration, and select Settings > General Settings > Collector.

2. In the Description column, select Type of information that should be logged, and click Edit Settings.

3. Select all message types: Exception, Error, Warning, and Info.

4. Follow the prompts to finish turning on the messages, and click Finish.

5. Repeat steps 1 through 4 for the following Administration settings:

n Settings > General Settings > Database

GatheringDiagnostic Information

n Settings > Windows > Agent - General

n Settings > UNIX > Agent - General

6. On Windows Agent machines, edit the Windows Registry to create a DWORD under the following Registry key.

HKEY_LOCAL_MACHINE\SOFTWARE\Configuresoft\CSI\5.0\Common\DebugEvent

DWORD name = Filter DWORD value = 0000003C

7. On the Collector, in the Windows Services Manager, restart the VCM Collector service.

What to do Next

Extract the log.

Extract the Debug Log

To create a DBEfile from your Collector debug log messages, extract the log from the database.

Prerequisites

Configure the debug log to capture all messages, including Info messages. See"Set the Debug Log to Store

all Message Types" on page 21.

Procedure

VMware, Inc.

1. To put the full set of messages into the log, rerun the job that caused the problem you are troubleshooting.

2. After the job completes, wait five minutes before proceeding.

3. In Windows, navigate to the following VCM tools folder.

(By default) C:\Program Files (x86)\VMware\VCM\Tools

VCM Troubleshooting Guide

4. To open the debug log viewer, right-click the following executable, andRun As Administrator.

ECMDebugEventViewer.exe

5. Click Filter Settings.

6. In the Message Type and Message Source areas, select all of the check boxes and click OK.

7. In the Data Source area, type the names of the servers and databases and click OK.

8. Click Date/Time, and select the between option.

9. Specify the start and end times when the job ran, and pad the times with an additional five or more minutes at each end.

10. Click File, and select Fetch.

The displayed data refreshes.

11. Click File, and select Fetch Next.

Continue the Fetch Next process until no additional data is added to the displayed debug log.

12. Click File, and select Save as DBE.

13. Name the DBE file and note where it is saved.

What to do Next

Using "Set the Debug Log to Store all Message Types" on page 21 as a guideline, restore the original logging levels. Usually, you only log Exception, Error, and Warning messages.

Extract SQL Server Logs

To save messages about database operations, extract SQL Server logs.

Procedure

1. On your SQL Server machine, open SQL Server Management Studio.

2. Select the server name and authentication method, and click Connect.

3. In the Object Explorer pane, expand server-name > Management > SQL Server Logs.

4. Right-click a log, and select View SQL Server Log.

The Log File Viewer displays the logs.

5. Click Export.

The Export Last Fully Retrieved Log dialog box appears.

6. Save the logs.

Give each saved file a meaningful name and note where it is saved.

Collect IIS Logs

To save messages about VCM Web server operations, extract Internet Information Services (IIS) Server logs.

VMware, Inc.

Procedure

1. On the Web server, select Start > Administrative Tools > Internet Information Services (IIS) Manager.

2. Expand Internet Information Services > server-name > Web Sites.

3. Right-click Default Web Site, and select Properties.

4. Verify that the Enable Logging check box is selected.

5. In the Active log format drop-down menu, select W3C Extended Log File Format, and click Properties.

6. Make a note of where the logs are stored.

The default location is C:\WINDOWS\system32\LogFiles\W3SVC1\

Each log file has the notation u_exyymmdd.log where yymmdd represents the date.

7. In Windows, copy the log files for the timeframe that you want.

Collect ARS Files

ARS files are raw data files sent to the Collector by the Agent. The Collector deletes the files after processing the data, but you can configure the Collector to keep the files.

GatheringDiagnostic Information

ARS files might hold valuable troubleshooting clues.

CAUTION This procedure involves using the regedit command to open the Windows Registry and edit the settings.

Procedure

1. On the Collector, type regedit at the command prompt to open the Registry.

2. Navigate to the following key:

(32-bit) \HKEY_LOCAL_MACHINE\Software\Configuresoft\ECM\4.0\Agent (64-bit) \HKEY_LOCAL_MACHINE\Software\Wow6432Node\Configuresoft\ECM\4.0\Agent

If the Agent key does not exist, navigate to 4.0, right-click, and select New > Key to create it.

3. Under Agent, verify that the AreResultsSaved key value is 1.

If AreResultsSaved does not exist, create it as a new DWORD and set its value to 1.

4. Navigate to the following key:

\HKEY_LOCAL_MACHINE\Software\WOW6432Node\Configuresoft\ECM\4.0\Collector

If the Collector key does not exist, navigate to 4.0, right-click, and select New > Key to create it.

5. Under Collector, verify that the AreResultsSaved key value is 1.

If AreResultsSaved does not exist, create it as a new DWORD and set its value to 1.

VMware, Inc.

6. Close the Registry.

7. Stop all running jobs, or make sure that no jobs are currently running.

8. Navigate to the following VCM folder:

The default location is C:\Program Files (x86)\VMware\VCM\CollectorData

9. Wait until no jobs are running, and delete any existing folders under the directory.

VCM Troubleshooting Guide

10. Start a collection, and note its Job ID as seen in the Running Jobs window.

11. After the job is finished, look for a CollectorData subfolder named with the Job ID of the job that just finished.

12. Create a ZIP file of the entire subfolder.

The ZIP file is what you send to VMware Technical Support.

What to do Next

Reopen the Registry, and set the two AreResultsSaved values to 0.

Collect the UNIX Syslog Messages

The UNIX syslog functions like the Event Log in Windows, because it records a large amount of detailed system messages.

NOTE The messages file location varies.

Procedure

1. Log in to the UNIX Agent system as the root user.

2. Use catto view the contents of the /etc/syslog.conf file.

In the file, look for an entry that indicates where the messages file is located.

3. Find the messages file itself, and transfer a copy of it to your local workstation or Collector.

4. Look for log files that have been rotated, and copy those as well.

Rotated files typically have some value appended to their name, such as /var/log/messages.1.

Collect Import/Export Tool Logs

The VCM import/export tool creates a debug file that might provide troubleshooting clues to data import or export problems.

Procedure

1. Log in to the VCM machine where the import/export tool is installed.

2. Navigate to the following VCM folder:

The default location is C:\Program Files (x86)\VMware\VCM\Tools\ImportExport

3. Copy any DBE files in the folder.

Extract Windows Event Logs

The Windows Event Log categories are Application, Security, and System. VCM errors almost always appear in the Application or System categories.

Procedure

1. On the Collector, select Start > Administrative Tools > Event Viewer.

2. On the left, locate and highlight the Application or System log.

3. From the pull-down menus, select Action > Save Log File As.

4. Type an appropriate name and click Save.

VMware, Inc.

Extract Windows System Information

Windows includes an executable application file msinfo32.exe that can provide a detailed snapshot of the current state of a system.

Procedure

1. On the Collector, click Start > Run.

2. Type msinfo32.exein the text box.

3. From the pull-down menus, click File > Save.

4. Type a meaningful file name, and click Save.

The save process might take a few minutes to create the NFO file that holds the detailed snapshot.

Collect UNIX ETL Logs

UNIX extract, transform, load (ETL) logs record the detailed workings of the ETL service.

Procedure

1. On the Collector, navigate to the following folder:

GatheringDiagnostic Information

The default location is C:\ProgramData\Configuresoft\ECM\ExceptionLog

2. Copy the log files that begin with etl.

Collect VCM Installation Logs

When installing VCM, the installer script writes log files that record the progress and status of the installation.

Procedure

1. On the Collector, log in as the user who installed VCM.

2. Select Start > Run.

3. In the text box, type %TEMP% to open the \Temp directory.

If you are not taken to the \Temp directory, it might be because of the FlatTempDir Registry key. See Microsoft Knowledge Base article 243215 for more information.

4. Make a ZIP file of all the files in the \_csi_installation folder.

Enable VCM Patching Logging

The VCM Patching module creates its own logs, which supplement the VCM debug file.VCM Patching logging is disabled by default.

Procedure

VMware, Inc.

1. On your SQL Server machine, open SQL Server Management Studio.

2. Connect to the VCM database using your preferred authentication method.

3. From the toolbar, click New Query.

A blank query pane appears.

4. From the toolbar drop-down menu, select the VCM database.

VCM Troubleshooting Guide

5. To enable VCM Patching debugging, type and run the following query:

update csi_hf_settings set val = '1' where setting = 'debug'

6. In Windows, in the Services Manager, restart the VCM Patch Management service.

What to do Next

Collect the VCM Patching logs.

Collect VCM Patching Logs

For problemswith the VCM Patching process, collect the VCM Patching log files.

Prerequisites

Turn on VCM Patching logging. See "Enable VCM Patching Logging" on page 25.

Procedure

1. With logging on, rerun the patching that produced the unwanted behavior.

2. On the Collector, navigate to the following VCM folder.

The default location is C:\Program Files (x86)\VMware\VCM\SUM\Collector

3. Copy the following TXT files.

CSISUMWorker_SumDBDebug.txt

CSISUMSvc_SumDBDebug.txt

CSISUMSvc_debug.txt

What to do Next

Using "Enable VCM Patching Logging" on page 25 as a guideline, turn VCM Patching logging back off. In the query, set the value to zero (0).

Collect Agent Logging

To collect Agent logs, modify the logging configuration file on the Agent.

Procedure

1. Locate the following file.

The default location is agent-path\Installers\Providers\Logging.conf

If you are on a Collector, the default path for the same file is:

C:\Program Files (x86)\VMware\VCM\Installer\Providers\Logging.conf

2. Make a copy of Logging.conf so you can restore it later.

3. Open Logging.conf in a text editor.

4. Locate the [handler_fileHandler] section.

5. Change the args setting in that section to the directory where you want to store logs. For example:

args=('C:\tmp\provider.log', 'a')

6. From the Collector, rerun the action that caused the problem.

7. Review the log stored in the directory that you specified.

The directory contains all output generated during precollection, the action itself, and post-collection.

VMware, Inc.

What to do Next

Using the backup copy, restore Logging.conf to its previous state.

GatheringDiagnostic Information

VMware, Inc.

VCM Troubleshooting Guide

VMware, Inc.

Troubleshooting Problems with VCM

This information describes troubleshooting situations that occur in general VCM operations such as installing, upgrading, patching, or reporting.

This chapter includes the following topics:

Patch Content Does Not Download for Red Hat and SUSEMachines 29

Signed Patch Content Cannot Be Validated 30

Mismatched Security Setting for AIX Patch Staging with NFS 30

UNIX Patch Deployment Fails 31

UNIX Patch Assessment Returns No Results 31

Patch Deployment Jobs Might Time Out 32

UNIX Bulletins Missing from the Required Location 33

Report and Node Summary Errors 33

Report Parameter Errors 34

Protected Storage Errors 35

SSL Becomes Disabled 36

Patch Content Does Not Download for Red Hat and SUSEMachines

Patch assessment content fails to download on certain Red Hat and SUSE managed machines.

VCM 5.6 and later selectively download bulletins. For example, if the inventory contains no Red Hat machines, VCM does not download Red Hat bulletins.

Problem

The log file reports missing bulletins or filters for patch assessments on Red Hat and SUSE machines that run a VCM Agent older than version 5.4.1.

Level: Error

Message: No filters found for filter set 66 [ RequestId = your-request-ID,

Requestor = Collector-name ] HRESULT 0x00040ec6;

Level: Warning

Message: No available bulletins matched filter for machine id: machine-ID.

Setting filter to: !/*

Cause

VCM is not configured to enable patch assessment for Agents older than version 5.4.1.

VMware, Inc.

VCM Troubleshooting Guide

Solution

Upgrade to a supported VCM Agent.

Signed Patch Content Cannot Be Validated

VCM signed patch content cannot be validated with the new signing certificate.

Problem

Signed patch content cannot be validated with the new signing certificate for VCM instances of 5.5.0, 5.5.1, and 5.6.0. VCM patch content was previously signed with a certificate from Verisign, but is now signed with a certificate from DigiCert.

Cause

When you download VCM patch content that was signed with the new DigiCert certificate on a VCM collector that is not updated with the new DigiCert issuer reference, and you run Check for update, the DBE entries indicate that the downloaded cab file signatures cannot be validated.

VCM is configured to validate patch content certificates by default, and stores the list of certificate issuers that are used to validate signed content in the database table named ecm_sysdat_certificate_ configuration_settings_ui. For VCM to properly validate patch content in instances of 5.5.0, 5.5.1, and 5.6.0, you must add the new issuer entry to this table.

Solution

Update the certificate information so that VCM recognizes the patch content as valid. Contact VMware Customer Service for assistance.

Mismatched Security Setting for AIX Patch Staging with NFS

A mismatched security setting causes the patch staging action to fail.

Problem

When an AIX managed machine uses NFSto stage patches from the Linux patching repository or Linux alternate location machine, a mismatched security setting on the AIX managed machine causes the patch staging action to fail.

Cause

In some situations, the patch staging action cannot connect to the patching repository or alternate location machine because of a mismatched security setting on the AIX managed machine. The Linux NFS Server displays the following error:

mount: 1831-008 giving up on:

vmount: Operation not permitted.

Solution

n On the AIX managed machine, run the following command to reserve a port for the NFS client:

nfso -o nfs_use_reserved_ports=1

VMware, Inc.

n To make the change permanent, so that the change persists after a reboot of the AIX managed

machine, include the -p option:

nfso -p -o nfs_use_reserved_ports=1

For more information, see the online IBM information for the vmount: Operation not permitted error from the Linux NFS Server.

UNIX Patch Deployment Fails

Reading or retrieving a patch bulletin name causes UNIX patch deployment to fail.

Problem

During UNIX patch deployment, VCM reports an error similar to the following message:

spiChangeResultsHelper->GetChangeActionInfo HRESULT 0x80070057 = The

parameter is incorrect.

In addition, the following exception messages appear in the Collector debug log.

Invalid Argument because an STL container is empty: m_deqBulletins HRESULT

0x80070057 = The parameter is incorrect.

spiChangeRequestPopulator->Initialize() HRESULT 0x80070057 = The parameter is

incorrect.

Troubleshooting Problems with VCM

Cause

The patch bulletin name contains information that VCM cannot process.

Solution

Because this failure originates in the patching content, contact VMware Technical Support, and provide details about the bulletin being used.

UNIX Patch Assessment Returns No Results

UNIXpatch assessments do not finish successfully.

Problem

UNIXpatch assessments do not finish successfully and do not display assessment results. Assessments must succeed before VCM Patchingfor UNIXcan install patches.

Cause

The problem can occur because of any of the following situations:

n The assessment template contains patch bulletins that do not match the selected machine type.

n The selected patch is for the wrong machine architecture, 32-bit or 64-bit.

n You defined a custom filter for Patch Assessment that caused bulletins to not match the selected

machine type.

VMware, Inc.

For example, a filter attribute such as Severity might not apply to bulletins for some platforms. If you defined a filter based on Severity, the bulletins do not match platforms that do not use Severity.

n For VCM 5.1 Agents or earlier, you have not run a Machines - General collection.

n The bulletins are missing from their required location on the UNIX Agent machine.

VCM Troubleshooting Guide

Solution

Use these corrective measures, respectively:

n Review the patch bulletins to make sure they match the selected machine type.

n Review the patch bulletins to make sure they match the selected machine architecture, 32-bit or 64-bit.

n Review or change the custom filter to remove attributes that the selected machine type does not

support.

n VCM supports UNIX Patch Assessment custom filters, which can be used when assessing older Agents.

For 5.1 or earlier Agents, first collect the Machines - General data class so that the assessment can succeed.

For 5.1.x or later Agents, you do not need to first perform a Machines - General collection.

n See "UNIX Bulletins Missing from the Required Location" on page 33.

Patch Deployment Jobs Might Time Out

Patching many machines simultaneously or downloading patches at run time might exceed the patching window.

Problem

When you patch many machines, or if you selected the option to download the patches at run time, the patching jobs might exceed the available time.

Cause

When VCM deploys patches to managed machines, VCM creates a job for each machine. When a reboot of the managed machine is required, two jobs are created for the machine, with the same start time. Depending on the number of machines being patched and the number of jobs, the time allowed for the patching window might expire before the patching jobs are finished. Either the request might have become stale or the number of maximum concurrent VCMAgent installations might be set too low.

Solution

If patching jobs time out, take the following actions.

1. Increase the time before a request is considered stale.

a. Click Administration and select Settings > General Settings > Collector.

b. Select How long before a request will be considered stale (minutes).

c. Click Edit Setting, and increase the number of minutes.

2. Increase the number of maximum concurrent VCM Agent installs.

a. Click Administration and select Settings > General Settings > Patching > Windows >

Deployment > Collector Option.

b. Click Edit and increase the number of maximum concurrent VCM Agent installations.

3. Reschedule the patch assessment or deployment jobs.

a. Click Patching and select Job Management > UNIX or Windows > Job Manager > Scheduled >

Assessments or Deployments.

b. Reschedule the job.

VMware, Inc.

UNIX Bulletins Missing from the Required Location

UNIXpatch bulletins are not in the required location on the Agent machine.

Problem

UNIXpatch bulletins are not in the required location on the Agent machine, which causes UNIXpatch assessments to fail and not display assessment results.

Cause

The problem can occur because of any of the following situations:

n The Agent version does not match the UNIX platform support for Patch Assessment.

n The process of distributing the bulletin information to the UNIX Agent machine failed.

n The bulletin information was removed from the UNIX Agent machine.

n Bulletin information is not loaded on the Collector.

n A Collector upgrade failed to reprocess bulletin information.

Solution

Use these corrective measures, respectively:

Troubleshooting Problems with VCM

n Make sure the Agent version is supported for how you are performing UNIX Patch Assessment:

Agent earlier than 5.0. No support.

Version 5.0 Agent. Support for some UNIX platforms, but requires manual distribution of

bulletin information to the UNIX Agent machine.

Version 5.1 Agent and later. Support for additional UNIX platforms, along with automated distribution of bulletin information to the UNIX Agent machine.

n Retry the process of distributing the bulletin information to the UNIX Agent machine.

n Add the bulletin information back to the UNIX Agent machine.

n Check for updates on the Collector. If the check reports that no updates are available, run Check for

Updates with the Force option.

n The Collector should reprocess bulletin information during an upgrade. If it did not, running Check for

Updates with the Force option might correct the problem.

Report and Node Summary Errors

You see error messages when looking at reports and node summaries.

Problem

After installing or upgrading VCM, you see any of the following errors when you look at reports and node summaries.

VMware, Inc.

Server Unavailable

The web application you are attempting to access on this web server is

currently unavailable.

Client found response content type of "text/html" but expected "text/xml".

No results returned for specified parameters.

The last error might occur even if the report appears to run, and part of the report appears.

VCM Troubleshooting Guide

Cause

Problems with Visual Studio 2005 and the .NET Framework are responsible for these errors.

Solution

1. Go to the Microsoft Web site.

2. Search for Knowledge Base article KB913384.

The article describes a hotfix for the following problem:

A .NET Framework 2.0 application that runs under a user account context

when no user profile is associated with the user account context might

crash, or you might receive an access violation error message.

3. Download and apply the hotfix for your machine.

4. Restart the machine.

Report Parameter Errors

A report does not contain the correct parameters.

Problem

After upgrading VCM, a report contains parameters that are out of date or wrong.

Cause

Report parameter values changed, but Report Manager did not handle the changes. The report was not uploaded correctly, and the error occurs because the report is overwritten instead of being first removed in Report Manager.

Solution

Remove the existing report, and upload it again.

VMware, Inc.

Troubleshooting Problems with VCM

1. On the Collector, open Report Manager:

http://collector-name-or-IP-address/Reports.

2. In Report Manager, open the folder that holds the affected report.

Folder Report

ECMAD Active Directory

ECMu UNIX

RSCA RSCA

Service Desk Service Desk and Change Reconciliation

SMS Systems Management Server

Standard Windows reports, and Change Management and Compliance

SUM VCM Patching

Virtualization Virtualization

3. On the right, click Show Details.

4. Select the check box next to the affected report.

5. Click Delete and click OK.

6. Click Upload File.

7. On the Upload File page, next to the File to Upload text box, select Browse.

8. From the reports directory, select the report.

9. Click OK.

The report now includes all of the new parameter modifications.

Protected Storage Errors

Trying to generate key pairs on the Agent Proxy machine results in a protected storage error.

Problem

When you attempt to generate key pairs on the Agent Proxy machine, a protected storage error similar to the following message appears:

CsiCommProxyUtil::wmain(): Failed to get protected storage for VCMv. HRESULT

0x8009000b = Key not valid for use in specified state.

Cause

Files that are not accessible because they reside in a protected folder are preventing the key generation process from succeeding.

VMware, Inc.

VCM Troubleshooting Guide

Solution

1. On the Agent Proxy machine, open the command prompt.

2. Change directory to the following VCM folder.

The default location is C:\Program Files (x86)\VMware\VCM\AgentData\protected

3. Delete the following files:

ECMv.csi.pds

ECMv.csi.pds.lck

4. Run the following command:

GenerateAgentProxyKeys.cmd

5. Verify that the following files were generated:

agent-proxy-machine-name_securecomm_public_key.txt

agent-proxy-machine-name_ssh_public_key.txt

6. Run the following command:

DatabaseUploadKey.cmd agent-proxy-machine-name_securecomm_public_key.txt

SSL Becomes Disabled

During a Collector upgrade, the option for secure socket layer (SSL) communication is turned off.

Problem

The check box to require a secure SSL channel becomes unchecked in the VCM virtual directory properties. After upgrading, you are then logging in to VCM over an unsecured connection without HTTPS.

Cause

The upgrade process causes this problem on a VCM Collector that is using SSL, even though the settings for secure SSL were in effect before starting the upgrade.

Solution

Restore the VCM virtual directory setting:

1. From a command prompt on the VCM Web server, type compmgmt.msc.

2. Expand the Services and Applications node, and expand Internet Information Services > Web Sites > Default Web Site.

3. Right-click the VCM virtual directory, and select Properties.

4. Click the Directory Security tab, and in the Secure Communications panel, click Edit.

5. Select the Require secure channel (SSL) check box, and click OK twice.

Restore the IIS setting in VCM:

1. On the VCM Collector, select Administration.

2. Select Settings > General Settings > Database.

3. Select IIS HTTP string http or https.

4. Click Edit Setting and change the IIS HTTP string setting to https.

VMware, Inc.

Troubleshooting the vSphere Client VCM Plug-In

When troubleshooting vSphere Client VCM Plug-In issues, you must consider the vSphere Client, the plug-in integration, and VCM.

vSphere Client VCM Plug-In Is Not Enabled 37

Cannot Register the vSphere Client VCM Plug-In 37

Invalid Certificate on a vSphere Client 38

Collector Not Running 38

HTTPS/SSL Is Not Configured on the Collector 38

Collection Unsuccessful 39

Machines Not Listed in the Collect Available List 39

Machines Not Listed in the Available List for Any Action 39

ESX Servers Are Not Displayed 39

vSphere Client VCM Plug-In Is Not Enabled

If the VCM Summary and VCM Actions tabs are not displayed, the plug-in is not properly configured.

Problem

In the vSphere Client, you cannot see the VCM Summary or VCM Actions tabs.

Cause

The plug-in is not enabled in the vSphere Client.

Solution

1. In the vSphere Client, select Plug-ins > Manage Plug-ins.

2. In the Installed Plug-ins area, right-click the vCenter Configuration Manager Extension plug-in, and select Enable.

3. Close the Plug-in Manager.

When the tabs appear, you are ready to use the vSphere Client VCM Plug-In.

Cannot Register the vSphere Client VCM Plug-In

When trying to register the vSphere Client VCM Plug-In, you cannot perform the action.

Problem

Cannot register the vSphere Client VCM Plug-In.

VMware, Inc.

VCM Troubleshooting Guide

Cause

You are not running vCenter 4 Server.

Solution

Install vCenter 4 Server. vCenter 4 Server is required for vSphere Client Plug-ins, including the vSphere Client VCM Plug-In.

Invalid Certificate on a vSphere Client

The vSphere Client connects to the vCenter Server using the SSL certificate and displays the datacenters, hosts, and any clusters.

Problem

When logging into a vSphere Client for the first time, if the certificate is not valid, a security warning about the SSL certificate appears.

Cause

The certificate is not valid.

Solution

1. Select the Install this certificate and do not display any security warnings for <vCenter_Server_ Instance> option.

2. Click Ignore.

Collector Not Running

If the VCM Summary and VCM Actions tabs are not displayed, the Collector is not running.

Problem

In the vSphere Client, you cannot see the VCM Summary or VCM Actions tabs.

Cause

The VCM Collector is not running.

Solution

Contact your VCM Administrator to verify the status of VCM.

HTTPS/SSL Is Not Configured on the Collector

If the VCM Summary and VCM Actions tabs are not displayed, the settings are improperly configured.

Problem

In the vSphere Client, you cannot see the VCM Summary or VCM Actions tabs.

Cause

If Use SSL was selected during VCM installation, the https/SSL is not properly configured on the Collector.

Solution

1. Open the .xml file specified during the registration.

2. Edit the file to reflect the configured connection method, either http or https.

VMware, Inc.

Collection Unsuccessful

The collection of data from one or more machines fails.

Problem

Collection did not run.

Cause

Machine is licensed, but the Agent is not installed.

Solution

Review the Getting Started information for licensing and installing the Agent on the target machine type.

Machines Not Listed in the Collect Available List

Machines are listed when viewing Compliance, Patching, and Reports, but not when viewing Collect.

Problem

Machine names are displayed in the Available list for Compliance, Patching, and Reports, but not when you select Collect.

Troubleshooting the vSphere Client VCM Plug-In

Cause

The VMs are not powered on or VMware Tools is not installed on the VMs.

Solution

Turn on the machines and verify that VMware Tools is installed on all VMs.

Machines Not Listed in the Available List for Any Action

The Available list does not display any machine names.

Problem

No machines are displayed in the Available list for any selected actions.

Cause

You are logged out of VCM

Solution

Click a blank area on the VCM Actions tab, and then press CTRL-F5 to log back in.

ESX Servers Are Not Displayed

When collecting, the ESX servers are to in the Available list.

VMware, Inc.

Problem

ESX servers are not displayed on the VCM Actions page when trying to Collect.

Cause

The ESX server was added to VCM using a name different from the name used in the vSphere Client.

VCM Troubleshooting Guide

Solution

In VCM, delete the ESXserver, and then re-add using the name used to manage the machine in the vSphere Client.

VMware, Inc.

VCM Windows Agent

To troubleshoot problems with your VCM Windows Agent, you must understand the Agent requirements to operate in your network, to communicate with your Collector. You must also understand how the Agent is installed so that you can trace the process and identify possible failure points.

The VCM Windows Agent is installed on managed Windows physical and virtual machines. The Agent is used to collect information from the machines. If the Agent is not functioning properly, VCM cannot collect data from the machine.

This chapter includes the following topics:

Windows Agent Installation Environment 41

Windows Agent Installation Process 42

Windows Agent Uninstallation Process 48

Windows Agent Upgrade Process 51

Windows Agent Manual Installation Process 52

Windows Agent Communication Protocols 53

Communication Protocol Change Process 54

Debug Window Agent Installations 58

Windows Agent Installation Environment

To install the VCM Windows Agent, your target Windows machines must have the required components and services, and your Collector must be configured with the correct network credentials.

n "Windows Agent Installation Networking Requirements" on page 41

The Windows machine on which you install the VCM Agent must have the required components and services to support the installation process and the Agent.

n "Windows Agent Installation Collector Credentials" on page 42

The Collector must have domain accounts with the necessary domain authority so that you can install the VCM Windows Agent on your target physical or virtual machines.

Windows Agent Installation Networking Requirements

The Windows machine on which you install the VCM Agent must have the required components and services to support the installation process and the Agent.

VMware, Inc.

VCM Troubleshooting Guide

n IPC$share must exist.

RPC uses IPC$.

n Windows Server Service must be running.

The Collector uses the server services to resolve the target machine's share to a local path to register the bootstrap service, the Registration Service, with the Remote Service Control Manager (SCM).

n You must be able to attach to the share files using the credentials provided during installation.

n DNS must correctly resolve the machine and share to the appropriate machine.

n Collector service authority must correctly attach to the SCM.

n DCOM must be set to receive DCOM creation requests for various components. DCOMrequires the

following settings.

n DCOM must be enabled on the target machine. On Windows 2003 Servers, this option is turned off

by default.

n On machines running the Windows NT operating systems, configure the default DCOM security

authorities for Access, Launch, and Configuration.

n Verify that the default authentication level is Connect and that the default Impersonation level is

Identify.

Windows Agent Installation Collector Credentials

The Collector must have domain accounts with the necessary domain authority so that you can install the VCM Windows Agent on your target physical or virtual machines.

In VCM, define one or more domain administrator accounts in the Available Accounts data grid. These accounts are associated By Domain or By Machine Group with each target machine. You can configure multiple accounts that are specified in priority order.

When you install the Agent, the Collector cycles through these accounts until it locates the account needed to establish communication with each target machine.

Configure the accounts in VCM in the network authority available account settings.

Windows Agent Installation Process

When you install the VCM Windows Agent from the Collector using install Agent action rather than a manual action, the installation process comprises many steps, the success and failure of which are tracked in the Jobs Manager. If an Agent installation fails, you can evaluate the job as it is running or view the details in the history to determine which part of the process you must evaluate more closely.

Review the job history, identify the step that failed, and determine what might be impeding the installation of the Agent.

Detect Previous Install

The detect previous install action evaluates the target Windows machine to determine whether a previous version of the VCM Agent is installed.

When installing VCM 4.11.x Agent or later, if a previous Agent version is detected, VCM must remove it before installing the new Agent.

VMware, Inc.

VCM Windows Agent

The detect previous install action determines if a previous Agent is present by attempting to connect to the Agent installation DCOM components, the Basic and Agent Installers. If a connection is made to either of these components, the detect previous install action sets a state member that the Validate Installation Environment, Interrogate Target Environment, and Resolve Uninstall Dependencies actions use to determine if they should run.

These actions remove the previous Agents and are included in all types of install and uninstall requests.

Validate Installation Environment

The validate installation environment action ensures that the target Windows machine is available for the installation of the VCM Agent.

1. If the validate job can contact the Module Installer, it calls methods on that component to obtain all the installation manifests on the target. It also checks the registry for conditions that might prevent the installation.

This job fails if the following conditions are satisfied:

n The Agent is locked. The HKEY_LOCAL_

MACHINE\SOFTWARE\Configuresoft\ECM\4.0\Agent\IsLocked, for 32-bit Windows servers, or HKEY_LOCAL_ MACHINE\SOFTWARE\Wow6432Node\Configuresoft\ECM\4.0\Agent\IsLocked, for 64-bit

Windows servers, the registry entry is evaluated by this job to determine if the Agent is locked. If this key exists and has a value other than 0, then the Agent is locked.

The installation infrastructure cannot remove or modify this value.

n The Agent is a Collector. The HKEY_LOCAL_

MACHINE\SOFTWARE\Configuresoft\ECM\4.0\Installer\CDInstallor or HKEY_LOCAL_

MACHINE\SOFTWARE\Wow6432Node\Configuresoft\ECM\4.0\Installer\CDInstall

registry entry is evaluated by this job to determine if the Agent is on a Collector machine. If this key exists and has a value of 2, then the target Windows machine is identified as a Collector machine.

The installation infrastructure cannot remove or modify this value.

2. If the validate job cannot communicate with the installation infrastructure, it flags the installation as valid and the request continues.

3. If an error is generated by an attempt to communicate with the installation infrastructure, the validation fails, as does the installation process, usually with an error that mandates that an uninstall action must occur before installing. An error is any result other than Install required or Success.

4. If the validation job can communicate with the Module installer, it gathers all the module information available from the various installation manifests and stores this information as a state variable so that other installation jobs can use the information to determine what was installed. The validation job does this in the RecordModuleInstallerVersion method and stores the module names and versions in the install_module state parameter . The module installer job uses this parameter to determine which parts of the installation infrastructure must to be installed.

Interrogate Target Environment

VMware, Inc.

The interrogate target environment action runs on the target Windows machine after the installation infrastructure is deployed to install the various product modules during the VCM Agent installation process.

The following actions determine which product modules must be deployed to the target machine.

VCM Troubleshooting Guide

1. A connection is made to the module installer.

2. The runtime Agent lock is updated.

3. The module installer returns all modules currently installed.

4. The product modules are recorded in the database.

A failure in this action causes the installation to fail.

Resolve Uninstall Dependencies

The resolve uninstall dependencies action reconciles the product modules that were found on the target Windows machine by the interrogation action and the product modules that must be deployed for the VCM Agent.

The result of the action is a list of product modules that might need to be uninstalled and a list of product modules that must be installed. Any product modules that require deployment are recorded in the state matrix.

A failure in this action causes the installation to fail.

Uninstall Module

The uninstall module action removes all product modules that must be uninstalled from the target Windows machine before the process installs the VCM Agent.

1. A list of product modules is obtained from the database.

2. A connection is made to the module installer.

3. The runtime Agent lock is updated.

4. Each module is uninstalled in turn by the module installer.

A failure in this action causes the installation to fail.

Uninstall Module Installer

The uninstall module installer action completes the removal of a previous Windows Agent as preparation for installing the new VCM Agent.

1. If you are uninstalling the Agent, this action tries all of the following methods to completely remove the Agent.

n Asynchronous command using the Registration Service.

n Command to simple installer

n Remote share.

2. If the current request is for any other kind of installation action, the action does nothing.

Install Simple Installer

The install simple installer action uses a service to deploy the installation infrastructure modules to the target Windows machines so that the infrastructure can manage the installation of the VCM Agent.

The installation infrastructure modules included in this action are ECMCommon.exe, ECMSimpleInstaller.exe, and ECMComSocketListener.exe.

VMware, Inc.

VCM Windows Agent

1. An attempt is made to contact the simple installer on the target Windows machine. If this action is successful, then this step is complete. If the action is not successful, the process continues.

2. An attempt is made to attach to the share on the target that was specified in the user interface and using the specified authority.

If this action fails, then the installation fails.

3. If the attach to share action succeeds, the process determines if the Agent has a runtime lock. This lock, which is not the same as the IsLocked registry entry, is used by the installation process to prevent two or more Collectors from installing on the same target machine to the same location at the same time.

The lock manifests as a ECMMachineActionLock.dat file on the target machine. This lock file is located in the root directory of the installation file and contains the type of action being performed. The types of actions include installing, uninstalling, or collecting. The lock file includes the time stamp and the request ID of the action doing the work. If the lock file exists on the target machine, and it contains a valid action with a request ID and a time stamp that has not expired, the target machine is considered locked and the installation fails.

When the target machine is considered locked, the lock is partially unlocked. To partially unlock the lock, the time stamp is zeroed and the request ID is purged. These actions allow a subsequent installation to continue if it was not relocked by another Collector. In all other cases, the lock file is zeroed and the installation process is allowed to continue.

4. A preinstallation check ensures that the subsequent installation will succeed. If components, for example, ComSocketServiceListener, the debug event .dll, or the subsystem singleton .dll are located, they are removed.

5. The directory structure is created and the following components are copied to the target machine.

ECMColInstallAgtRegistrationService.exe

This binary file is the bootstrap service that is invoked by the target machine’s Remote Service Control Manager (SCM).

Psapi.dll

This file is a dependency of the Registration Service.

ECMTargetShareInfo.dat

This file is generated and contains the share that is used for this installation as well as the relative path to the Agent.

6. An attempt is made to connect to the SCMusing the same authority as the Collector service.

If this action fails, the installation fails.

7. The ECMColInstallAgtRegistrationService.exe is registered with the SCM on the target machine.

8. The following loop runs until all the installation infrastructure modules are deployed.

n Copy an installation infrastructure module to the target.

VMware, Inc.

n Start the registration service.

This service runs the module and waits until it is fully expanded.

n If the module reports an error, terminate the installation.

A log file is left on the target machine that specifies the nature of the problem.

n Stop the service.

VCM Troubleshooting Guide

9. The registration service is unregistered with VCM.

10. An attempt is made to contact the simple install using DCOM.

If this action succeeds, this step in the process is finished.

If the process fails on any of the actions, the simple installer job enters the rollback state. All modules are uninstalled, all files are removed, and the registry is purged. In this state, the debug event files are not deleted on the target machine. The file might contain useful information about the error.

Install Module Installer

The module installer action manages the installation of the product modules when you install the VCM Agent on target Windows machines.

The installer is called on to terminate the Windows Agent installation process when an inspection job is canceled. It can update the installation infrastructure. The module installer completely removes the remainder of the VCM files during the uninstallation process.

1. Determine if the module installer component was deployed. If it exists, a check is preformed to see if it needs to be upgraded.

2. If the module installer needs to be installed, the following actions are run.

n A connection is made to the simple installer.

n The runtime install lock is updated.

n The module installer is deployed and run by the simple installer.

3. Determine if any installation infrastructure modules must be upgraded.

Outdated components are listed and upgraded.

4. If a component of the installation infrastructure, other than the module installer, must be updated, the following actions run.

n A connection is made to the module installer.

n The registration service is copied if it is not present on the target Windows machine.

n All infrastructure modules that must be updated are copied to the target machine.

n A call is made to the module installer to install the installation infrastructure modules. This call is

received and delegated to the registration service to perform the action.

n A connection is made to the simple installer.

n The runtime install lock is updated.

n The module installer is pushed out and run by the simple installer.

5. Any installation infrastructure modules that must be removed are removed.

If an action fails at any point, the changes are rolled back and the installation fails.

Resolve All Versions of Modules Based on Highest Version Number

The resolve all versions of modules action evaluates the modules to be installed against those already on the target Windows machine and determines what must be installed for the VCM Agent.

The action uses the resolve highest version modules resolution algorithm.

VMware, Inc.

VCM Windows Agent

This algorithm is provided with a requested install module list (RIML), the already installed module list (AIML), the module dependency graph, and the module updates map. The algorithm uses the map to determine what must be installed and uninstalled on the target machine to achieve the latest versions of modules based on the modules provided in the RIML and the AIML.

Install Module

The install module action installs all product modules related to the VCM Agent.

1. A list of product modules is obtained from the database.

2. A connection is made to the module installer.

3. The runtime Agent lock is updated.

4. Each module is installed by the module installer.

A failure in this action causes the installation to fail.

Fully Release the Synchronization Lock on the Target Machine

The fully release the synchronization lock action clears the runtime Agent lock. Clearing the Agent lock allows the VCM Agent to inspect the target Windows machine.

1. A connection is made to the module installer.

2. Unlock is called.

If this action is not run because of a previous failure, the installation is considered invalid. Attempts to collect from the Agent results in failure because the installation process failed.

Submit Request to Agent

The submit request to Agent action is the first VCM Agent inspection on the Windows machine.

After installing the Agent, a machine environment inspection is performed.

Check If Request Is Complete

The check if request is complete action verifies that the Collector successfully collected from the VCM Agent on the Windows machine.

The Collector checks to determine if the Windows machine environment collection from the Agent is finished.

Transfer Request Results

The transfer request results action sends the results of the inspection on the Windows machine to the Collector as part of the VCM Agent installation process.

After the Agent completes the machine environment inspection on the Windows machine, the results are sent back to the Collector.

VMware, Inc.

Acknowledge Successful Data Transfer

The acknowledge successful data transfer action is a record of the data from the VCM Agent inspection on the Windows machine during the installation of the Agent.

The Collector records the receipt of the data from the Windows machine environment inspection.

VCM Troubleshooting Guide

Prepare Request Results for Insert

The prepare request results for insert action prepares the data from the Windows machine inspection for insertion in the VCM database during the VCM Agent installation process.

A bulk insert of the machine environment inspection data is prepared for insertion in the VCM database.

Insert Data Into Database

The insert data into database action adds the data collected from the Windows machine by the VCM Agent to the database as a step in the Agent installation process.

The data from the machine environment inspection is inserted in the VCM database.

Transform Inserted Data

The transform inserted data action takes data collected from the Windows machine by the VCM Agent during the Agent installation process and transforms it from the temp tables to the proper database format for the VCM database.

The inserted machine environment data is transformed in the database from temp tables and the meta data is updated

Cleanup Machine Data

The cleanup machine data action cleans up the files on the target Windows machine after installing the VCM Agent.

The data on the target Windows machine is cleaned up.

Partially Release the Synchronization Lock on the Target Machine

The partial release of the synchronization lock on the target Windows machine action is a finalization job that runs even if any of the previous steps fail during the VCM Agent installation process.

This action clears out the Agent runtime lock so that if another installation of the Agent is attempted, it will succeed.

This step does not clear the action type in the lock. Because this condition is a result of an invalid installation, this action is a partial release that renders the Agent incapable of doing useful work.

The Agent is available for collections only if the fully release the synchronization lock on the Agent action runs successfully.

Cleanup Request Data

The cleanup request data action removes unneeded and transform data on the Collector as the final action in the VCM Agent installation process.

The VCM Collector cleans up the inserted and transformed data, the request is removed, and the state of the request is complete.

Windows Agent Uninstallation Process

The Windows Agent uninstallation process removes the VCM Agent from Windows machines. This action removes VCM products and the installation infrastructure. The result of uninstall is a clean Windows machine with no VCM remnants.

VMware, Inc.

VCM Windows Agent

To troubleshoot this process, review the job history, identify the step that failed, and determine what might be impeding the removal of the Agent.