IBM Problem Determination User Manual

Download

ivoli

Version 5.1

Intelligent Orchestrator



Problem

Determination and Troubleshooting Guide

SC32-2216-00

ivoli

Version 5.1

Intelligent Orchestrator



Problem

Determination and Troubleshooting Guide

SC32-2216-00

Note:

Before using this information and the product it supports, read the information in Appendix C, “Notices,” on page 229.

First Edition, July 2006

US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Preface . . . . . . . . . . . . . . vii

Who should read this book . . . . . . . . . vii

Publications . . . . . . . . . . . . . . vii

Tivoli Intelligent Orchestrator library . . . . . vii

Prerequisite publications . . . . . . . . . viii

Related publications and resources . . . . . viii

Accessing publications online . . . . . . .ix

Ordering publications . . . . . . . . . . .ix

Accessibility . . . . . . . . . . . . . .ix

Tivoli software training . . . . . . . . . . .x

Support information . . . . . . . . . . . .x

Conventions used in this book . . . . . . . .x

Typeface conventions . . . . . . . . . .x

Chapter 1. Introduction to

troubleshooting . . . . . . . . . . .1

Product and system overview . . . . . . . .1

Product comparison . . . . . . . . . . .2

Product architecture . . . . . . . . . . . .3

Tivoli Intelligent Orchestrator components . . .3

Prerequisites installed with Tivoli Intelligent

Orchestrator . . . . . . . . . . . . .4

Built-in troubleshooting features . . . . . . . .5

Problem classification . . . . . . . . . . .5

Product maintenance . . . . . . . . . . .5

Backing up the system . . . . . . . . . .5

Performing periodic checks and maintenance . .6

Problem resolution . . . . . . . . . . . .6

Chapter 2. Problem determination essentials for Tivoli Intelligent

Orchestrator . . . . . . . . . . . . .7

Problem determination methodologies . . . . . .7

Defining the problem . . . . . . . . . .7

Identifying the cause of the problem . . . . .8

Fixing the problem . . . . . . . . . . .9

Troubleshooting methods for Tivoli Intelligent

Orchestrator and prerequisites . . . . . . . .9

Troubleshooting the prerequisite installation . .10

Tivoli Intelligent Orchestrator installation and

subcomponent configuration verification . . .17

Tivoli Intelligent Orchestrator runtime problem

determination . . . . . . . . . . . .21

Accessing the knowledge bases . . . . . . .26

Using the available command line tools . . . .26

Chapter 3. Log file types . . . . . . .35

Message logs . . . . . . . . . . . . . .35

Message ID format . . . . . . . . . . .35

Message elements . . . . . . . . . . .36

Trace logs . . . . . . . . . . . . . . .37

Common Base Event logs . . . . . . . . . .37

Chapter 4. Locating and configuring

log files . . . . . . . . . . . . . .39

Log locations . . . . . . . . . . . . . .39

Log directory files . . . . . . . . . . .39

Logging levels . . . . . . . . . . . . .40

Configuring logs with log4j . . . . . . . . .40

log4j.prop . . . . . . . . . . . . . .40

Log level defaults . . . . . . . . . . .42

RollingFileAppender defaults . . . . . . .42

Configuring log4j dynamically . . . . . . .42

Preparing log files for review by IBM Tivoli

Support . . . . . . . . . . . . . .43

Chapter 5. Tivoli Intelligent

Orchestrator log files . . . . . . . .45

Log files for product setup . . . . . . . . .45

Log files for the uninstall process . . . . . . .45

Log files for starting and stopping the server . . .46

Log files for the We b interface . . . . . . . .47

Deployment engine log files . . . . . . . . .47

Workflow logs . . . . . . . . . . . . .48

Logs for data collection and decision making . . .49

Automation package logs . . . . . . . . . .49

Chapter 6. Middleware prerequisite log

files . . . . . . . . . . . . . . . .51

Installation log files for prerequisites . . . . . .51

Installation logs for individual prerequisites . .51

Runtime logs for prerequisites . . . . . . . .52

WebSphere Application Server . . . . . . .52

DB2 Universal Database . . . . . . . . .53

Directory Server . . . . . . . . . . . .53

Tivoli Agent Manager . . . . . . . . . .54

Chapter 7. Common problems and known limitations in Tivoli Intelligent

Orchestrator . . . . . . . . . . . .55

Problems with installation . . . . . . . . .55

In Linux, you cannot create the user tioadmin . .55

The system cannot connect to the IBM Tivoli

Directory Server during the Tivoli Intelligent

Orchestrator installation . . . . . . . . .56

The installation of the DB2 Universal Database

Client on Windows 2003 system fails . . . . .56

Installing the common agent and the agent

manager is not supported . . . . . . . .57

The system cannot connect to the database server

during the Tivoli Intelligent Orchestrator

installation . . . . . . . . . . . . .57

The Microsoft Active Directory installation fails 57

The Microsoft Active Directory installation fails

with invalid certificate value . . . . . . . .58

The silent installation program for Tivoli Intelligent Orchestrator exits before the

installation is completed . . . . . . . . .58

Environment variables not set for user tioadmin 58 The installation of IBM Tivoli NetView on Windows fails if the password for creating a user account does not meet the system requirements .59 The agent manager installation fails during the Tivoli Intelligent Orchestrator installation . . .59 An error occurs when running the rpm -qa command after the DB2 Universal Database, IBM Tivoli Directory Server, and WebSphere Application Server prerequisites are installed on

Linux . . . . . . . . . . . . . . .62

Problems with logging in . . . . . . . . . .62

You cannot log in . . . . . . . . . . .62

You cannot change your password . . . . .63

Problems with workflows . . . . . . . . .64

DB2 Universal Database errors occur when you

deploy resources . . . . . . . . . . . .64

DB2 Universal Database creates a database state

error . . . . . . . . . . . . . . . .64

DB2 Universal Database deadlocks occur when

the system runs a logical operation . . . . .65

Shell command error: Resource temporarily

unavailable . . . . . . . . . . . . .65

Shell command error: Exit value=1, Error

stream=", Result stream="no bash in ..." . . . .65

IBM Tivoli Software Support personnel have

requested a copy of a workflow . . . . . .65

A workflow does not install . . . . . . . .67

Cannot run workflows if the PS1 environment

variable is altered . . . . . . . . . . .68

Other common problems . . . . . . . . . .68

On Windows, Tivoli Intelligent Orchestrator does

not start . . . . . . . . . . . . . .69

Problems occur when administering Tivoli Intelligent Orchestrator from the Web interface

while a local firewall is enabled . . . . . .69

Incorrect SOAP parameters generate detailed

Java exception messages . . . . . . . . .70

XML import does not work properly . . . . .70

DB2 Universal Database does not work properly

with terminal server . . . . . . . . . .71

On UNIX, Tivoli Intelligent Orchestrator logs fill

up the file system . . . . . . . . . . .71

On Windows 2000, the embedded messaging

feature does not work . . . . . . . . . .71

WebSphere Application Server on Linux remains paused when you try to stop the JMS pub sub

broker . . . . . . . . . . . . . . .72

On Windows, the remote connection to the database hangs when the database server is on a

multiprocessor machine . . . . . . . . .72

For languages other than English, the text in CSV reports imported in Microsoft Excel is garbled . .73 An error occurs if you use special characters to

create a new user or security role . . . . . .74

Common agents unable to communicate with the

agent manager . . . . . . . . . . . .75

Limitations when trying to manually associate a discovered software resource with a software

definition . . . . . . . . . . . . . .75

Some patch installations might fail when trying

to perform a multiple patch installation . . . .76

There is a maximum size for returnResult

parameter storage . . . . . . . . . . .76

Device Manager limitations . . . . . . . . .76

Running lightweight management server with DB2 Universal Database generates exceptions . .76 Agent receives an HTTP Unauthorized (401)

response code . . . . . . . . . . . .77

Known report limitations . . . . . . . . . .77

Running any report with too many objects

selected fails . . . . . . . . . . . . .77

Chapter 8. Troubleshooting the agent

manager . . . . . . . . . . . . . .79

Log files for the agent manager . . . . . . . .79

Packaging the log files . . . . . . . . . .79

Locating the installation log files . . . . . .79

Locating the uninstallation log files . . . . .81

Locating the remote registry logs . . . . . .81

Locating the runtime log files . . . . . . .82

Locating the DB2 Universal Database runtime

logs . . . . . . . . . . . . . . . .82

Determining the version of agent manager . . . .83

Installation and uninstallation verifications . . . .83

Verifying that the agent manager service is

running . . . . . . . . . . . . . . .83

Verifying the installation of WebSphere

Application Server . . . . . . . . . . .84

Verifying the installation of DB2 Universal Database Enterprise Server Edition (ESE) . . .84 Errors starting the agent manager application

server . . . . . . . . . . . . . . .84

Errors starting the agent manager application . .84

Verifying the connection to the registry . . . .84

Enabling and disabling tracing . . . . . . .85

Recovering from an incomplete uninstallation of

the agent manager . . . . . . . . . . .86

Common problems . . . . . . . . . . . .87

Registration and communication failures . . .87 Problems using the WebSphere Administrative

Console . . . . . . . . . . . . . . .88

Chapter 9. Contacting IBM Tivoli

Software Support . . . . . . . . . .91

General data to collect for Tivoli Support . . . .91

packageLog ZIP tool . . . . . . . . . .92

Level Reporting tool for the WebSphere

Application Server platform . . . . . . . .92

Chapter 10. Tivoli Intelligent

Orchestrator messages . . . . . . .95

Common subsystem messages . . . . . . . .95

Deployment engine subsystem messages . . . . 124

Job Distribution Service (JDS) subsystem messages 135

J2ee subsystem messages . . . . . . . . . 135

Policy engines subsystem messages . . . . . . 150

iv Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Common agent subsystem messages . . . . . 153

TC driver manager subsystem messages . . . . 154

Utilities subsystem messages . . . . . . . . 164

Compliance Messages . . . . . . . . . . 166

Group Messages . . . . . . . . . . . . 168

Networking Messages . . . . . . . . . . 169

RXA Messages . . . . . . . . . . . . . 171

Software Distribution Messages . . . . . . . 173

Chapter 11. Tivoli Common Agent

Services messages . . . . . . . . . 179

Common Agent messages . . . . . . . . . 179

Agent Manager messages . . . . . . . . . 202

Appendix A. Common Agent

serviceability tool . . . . . . . . . 221

Components . . . . . . . . . . . . . . 221

Running the serviceability tool . . . . . . . 221

Contents of the CASservice.zip file . . . . . . 221

Appendix B. Support information . . . 225

Searching knowledge bases . . . . . . . . . 225

Search the information center on your local

system or network . . . . . . . . . . . 225

Search the Internet . . . . . . . . . . 225

Obtaining fixes . . . . . . . . . . . . . 225

Contacting IBM Software Support . . . . . . 226

Determine the business impact of your problem 227

Describe your problem and gather background

information . . . . . . . . . . . . . 227

Submit your problem to IBM Software Support 227

Appendix C. Notices . . . . . . . . 229

Trademarks . . . . . . . . . . . . . . 230

Index . . . . . . . . . . . . . . . 231

Contents v

vi Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Preface

This guide describes how to identify and resolve problems that might occur when you use Tivoli® Intelligent Orchestrator or Tivoli Provisioning Manager.

The instructions in this book apply to both Tivoli Intelligent Orchestrator and Tivoli Provisioning Manager and will clearly identify any information or tasks that apply only to one application or the other.

Who should read this book

This book should be read by system administrators, operators, or anyone else responsible for performing orchestration and provisioning tasks with the application. The book is also addressed to all Tivoli Customer Support service engineers or developers who are responsible for providing customer support and for troubleshooting Tivoli Intelligent Orchestrator.

People reading this book should have knowledge in the following areas:

v Windows® 2000 Server or Windows Server 2003

v Linux® on Intel

v IBM® AIX® operating system

v Sun Solaris operating system

v SUSE Linux Enterprise Server 8 for Linux on iSeries

v Networking concepts

v Basic operating system commands

v DB2 Universal Database™ Enterprise Server Edition or Oracle9i Database

operation, configuration, and maintenance

v WebSphere® Application Server

v Basic SQL commands

v XML

v The Internet

™

to the online help for more information on configuring and administering

Refer Tivoli Intelligent Orchestrator.

Publications

This section lists publications in the Tivoli Intelligent Orchestrator library and related documents. It also describes how to access Tivoli publications online and how to order Tivoli publications.

Read the descriptions in the Tivoli Intelligent Orchestrator library, the prerequisite publications, and the related publications to determine which publications you might find helpful. After you determine the publications you need, refer to the instructions for accessing publications online.

Tivoli Intelligent Orchestrator library

The publications in the Tivoli Intelligent Orchestrator library are:

v Tivoli Intelligent Orchestrator Installation Guide

Explains how to install and upgrade Tivoli Intelligent Orchestrator software. This guide also provides information about troubleshooting installation problems.

v Tivoli Intelligent Orchestrator Migration Guide

This book describes the steps to migrate from a previous version of Tivoli Intelligent Orchestrator.

v Tivoli Intelligent Orchestrator Release Notes

These release notes describe last minute product changes and known problems in the Tivoli Intelligent Orchestrator product release.

v Tivoli Intelligent Orchestrator Problem Determination Guide

This guide describes how to identify and resolve problems that occur when you use Tivoli Intelligent Orchestrator. The instructions in this book apply to both Tivoli Intelligent Orchestrator and Tivoli Provisioning Manager and will clearly identify any information or tasks that apply only to one application or the other.

remaining product documentation is available in the online help, which can be

The launched from the We b interface for the product.

Prerequisite publications

To use the information in this book effectively, you must have some prerequisite knowledge, which you can obtain from the following publications:

v WebSphere Application Server Information Center , available from

www.ibm.com/websphere.

You can also download the WebSphere Application Server documentation plug-in and install it in the Tivoli Intelligent Orchestrator. This plug-in enables you to view the WebSphere Application Server documentation in the same Information Center as the Tivoli Intelligent Orchestrator help.

v DB2 Universal Database Information Center, available from www.ibm.com/db2. v IBM Directory Server documentation., available from http://

publib.boulder.ibm.com/tividd/td/tdprodlist.html.

Related publications and resources

Information related to Tivoli Intelligent Orchestrator is available in the following locations:

v The IBM Orchestration and Provisioning Automation Library is available online

at: http://catalog.lotus.com/wps/portal/tpm. The IBM Orchestration and Provisioning Automation Library delivers the tools that you need to build your business around on demand automation. New workflows and automation packages and updated readme files for automation packages that are included with Tivoli Intelligent Orchestrator will be available in the IBM Orchestration and Provisioning Automation Library. This site will also contain the most up-to-date information about the full list of available workflows.

Note: Many workflows, for example the network devices and storage

workflows, cannot be categorized by platform. However, the workflow for each readme will document any platform-specific information.

v The IBM On Demand Automation Catalog is available online at

http://www-306.ibm.com/software/tivoli/features/opal/. This site is a resource for partners who want to develop workflows to be put into the Orchestration and Provisioning Library.

v Redbooks™ related to Tivoli Intelligent Orchestrator are available at

www.redbooks.ibm.com

viii Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

v The Tivoli Software Library provides a variety of Tivoli publications such as

white papers, datasheets, demonstrations, redbooks, and announcement letters. The Tivoli Software Library is available on the Web at: http:// publib.boulder.ibm.com/tividd/td/tdprodlist.html

v The Tivoli Software Glossary includes definitions for many of the technical terms

related to Tivoli software. The Tivoli Software Glossary is available at http://publib.boulder.ibm.com/tividd/glossary/tivoliglossarymst.htm

Accessing publications online

The publications for this product are available online in Portable Document Format (PDF) or Hypertext Markup Language (HTML) format, or both in the Tivoli software library: http://publib.boulder.ibm.com/tividd/td/tdprodlist.html

To locate product publications in the library, click the first letter of the product name or scroll until you find the product name. Then, click the product name.

Product publications include release notes, installation guides, user’s guides, administrator’s guides, and developer’s references.

Note: To ensure proper printing of PDF publications, select the Fit to page check

box in the Adobe Acrobat Print window (which is available when you click File → Print).

In addition to accessing the PDF publications, you can also access the online help which is shipped with Tivoli Intelligent Orchestrator. The online help is available at the external IBM Tivoli Information Center, located at: http:// publib.boulder.ibm.com/infocenter/tivihelp/index.jsp.

Ordering publications

You can order many Tivoli publications online at the following Web site:

http://www.elink.ibmlink.ibm.com/public/applications/ publications/cgibin/ pbi.cgi

You can also order by telephone by calling one of these numbers:

v In the United States: 800-879-2755

v In Canada: 800-426-4968

In other countries, see the following We b site for a list of telephone numbers:

http://www.ibm.com/software/tivoli/order-lit/

Accessibility

Accessibility features help a user who has a physical disability, such as restricted mobility or limited vision, to use software products successfully. With this product, you can use assistive technologies to hear and navigate the interface. Yo u also can use the keyboard instead of the mouse to operate all features of the graphical user interface.

Preface ix

Tivoli software training

For Tivoli software training information, refer to the IBM Tivoli Education Web site: http://www.ibm.com/software/tivoli/education.

Support information

If you have a problem with your IBM software, you want to resolve it quickly. IBM provides the following ways for you to obtain the support you need:

v Searching knowledge bases: You can search across a large collection of known

problems and workarounds, Technotes, and other information.

v Obtaining fixes: You can locate the latest fixes that are already available for your

product.

v Contacting IBM Software Support: If you still cannot solve your problem, and

you need to work with someone from IBM, you can use a variety of ways to contact IBM Software Support.

more information about these three ways of resolving problems, see

For Appendix B, “Support information,” on page 225.

Conventions used in this book

This reference uses several conventions for special terms and actions and for operating system-dependent commands and paths.

Typeface conventions

The following typeface conventions are used in this reference:

Bold Lowercase commands or mixed case commands that are difficult to

distinguish from surrounding text, graphical user interface (GUI) controls such as names of fields, icons, or menu choices, keywords, parameters, options, names of Java™ classes, and objects are in bold.

Italic Variables, titles of publications, and special words or phrases that are

emphasized are in italic. Italics also indicate names for which you must substitute the appropriate values for your system.

Monospace

Code examples, command lines, screen output, file and directory names that are difficult to distinguish from surrounding text, system messages, text that the user must type, and values for arguments or command options are in monospace.

x Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 1. Introduction to troubleshooting

Problem determination, or troubleshooting, is a process of determining why a product is not functioning in the expected manner. This guide provides information to help you identify and resolve problems that you encounter when using Tivoli Intelligent Orchestrator.

Product and system overview

Tivoli Intelligent Orchestrator is an automated resource management solution for corporate and Internet data centers. Through orchestrated provisioning, it provides the ability to manage the IT environment in real time, according to defined business policies, to achieve desired business goals.

Many organizations face challenges in managing their existing IT resources, evaluating their effectiveness in supporting business needs, and determining how to meet future needs with existing or new resources. To effectively respond to change, financial pressures, and competition, businesses need to maintain an environment that is flexible, responsive, and makes optimal use of available resources. There are two aspects to creating such an environment:

Creating a flexible business model

Many organizations have difficulty responding to change because business processes. In some organizations, processes might be too rigid and difficult to adapt to different situations. Conversely, other organizations struggle with processes that are unclear, inconsistent, or duplicated by participants in the processes. To effectively respond to change, businesses must clearly define their end-to-end business processes and subprocesses. For maximum flexibility and consistent implementation, subprocesses are designed as components that can be customized or reused in larger processes.

Creating

a flexible IT environment

To support changing business needs and priorities, the IT infrastructure that supports the operations and applications in a business must be optimized for allocation based on demand and re-purposing in different business contexts.

Tivoli Intelligent Orchestrator helps you to develop an optimized IT environment

by:

Modeling your business processes

You can capture your business processes in workflows. Workflows automatically and consistently perform the configuration tasks that you currently perform manually. Tivoli Intelligent Orchestrator provides workflows that you can customize, or you can create new ones to support your existing tools and processes. Workflows can automate processes from configuring and allocating servers, to installing, configuring, and patching software, and can be either large and complex or can consist of a single command. The automation of configuration tasks is called automated provisioning.

Modeling

your data center assets

Tivoli Intelligent Orchestrator manages a virtual representation of your data center in a data center model. Virtualization exists on several levels to provide flexible configuration in different contexts:

v Each asset is represented by a data center object. When you make a

change to an asset with Tivoli Intelligent Orchestrator, the data center object is updated in the data center model. If a change is made outside of Tivoli Intelligent Orchestrator, the external change can be identified by comparing it with the data center object in the data center model.

v For some data center assets, the data center model stores data about the

asset and data about deploying or provisioning the asset separately to provide a range of implementation options. For example, when a software package is added to the Tivoli Intelligent Orchestrator software catalog, you define the software package as a installable file. Yo u can then create software definitions that describe different configuration requirements and configuration options for installing the same software package.

v Yo u can create templates that define standard configurations. For

example, you can create a server template that includes the routing, software, and storage configuration for a particular application tier. When a server is added to the application tier, the defined configuration is automatically applied.

v Yo u can define application topologies that describe requirements for an

application. You can then deploy an application based on the defined application requirements.

Orchestrating

provisioning

You can configure Tivoli Intelligent Orchestrator to dynamically allocate resources to applications based on demand or priority. Applications can share pools of available resources to optimize utilization and eliminate idle resources or excess allocation of resources.

Product comparison

Tivoli Intelligent Orchestrator includes Tivoli Provisioning Manager, a standalone product that can be purchased separately, based on your data center needs. Tivoli Provisioning Manager provides core automated deployment capability, while Tivoli Intelligent Orchestrator adds policy-based decision-making capabilities.

Tivoli Intelligent Orchestrator uses historical and current demand and performance data, as well as defined business policies to determine where and when to allocate resources. When a deployment decision is made, Tivoli Intelligent Orchestrator runs workflows to automatically make the required configuration and allocation changes. You can specify the degree of automated provisioning that you require at the global, application, and tier level.

The following table summarizes the complementary roles of the two products:

Tivoli Intelligent Orchestrator Tivoli Provisioning Manager

Orchestrates automation Coordinates provisioning

Determines why, when, and where: Senses why an action is necessary, anticipates when to start provisioning, and prioritizes where to put resources.

Coordinates what and how: Determines what data center assets to provision and provisions them based on your defined business processes.

2 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Product architecture

Tivoli Intelligent Orchestrator includes the following main components:

Device-specific protocols (For ex., SNMP)

WebSphere Application Server

SOAP

IIOP/RMI

Policy Engine

JDBC

Command Line Interface (CLI)

JDBC

Web User Interface

IIOP/RMI

JDBC

EJB (

Soap Service (Web Container)

LDAP

Managed Data Center

WebSphere

Application Server

EJB Container)

IIOP/RMI

LDAP

Device-specific protocols

Queue (

WebSphere Application Server

JDBC

MQ)

Tivoli Intelligent Orchestrator components

data center model

The data center model is an abstract representation of all physical and logical assets in a data center that Tivoli Intelligent Orchestrator manages, and their relationships. The data center model is stored in the Tivoli Intelligent Orchestrator database. It communicates directly with all other Tivoli Intelligent Orchestrator components and facilitates communication between product components.

Database

Device-specific protocol

Deployment Engine

JDBC

Automation Packages (TC Drivers)

Discovery

policy

engine

The policy engine is responsible for orchestration capability in Tivoli Intelligent Orchestrator. It encompasses:

data acquisition engine

This component gathers and preprocesses performance metrics

Chapter 1. Introduction to troubleshooting 3

from each managed application environment in the data center. It captures and filters data from the application, operating system, and infrastructure layers and communicates the data to other components.

objective

global resource manager

deployment

The deployment engine is responsible for automated provisioning. It creates, stores, and runs workflows and communicates their success or failure in performing an operation. Workflows can be triggered by commands from an administrator, by external tools that send SOAP commands to Tivoli Intelligent Orchestrator, or by recommendations from the policy engine.

automation

An automation package is a collection of workflows, scripts, and other commands and tools that apply to the operation of a specific type of software component or a physical device. The deployment engine manages the deployment of workflows and associated workflow components in an automation package.

analyzer

This component determines the resource requirements of each application, and identifies trends and peaks in resource use. Each managed application environment has an associated objective analyzer.

This component receives requirements for servers or network devices from all the objective analyzers, and manages the overall optimization. It has two primary responsibilities: making optimal resource allocation decisions, and maintaining a stable application infrastructure. By considering the different server requirements for each application environment, it determines where to allocate resources.

engine

packages

discovery technologies

A discovery technology is an application that performs configuration change detection. For example Tivoli NetView can discover new hardware that is added to the data center and is not defined in the data center model. For hardware that is already defined in the data center model, it can also identify configuration changes that occur outside of Tivoli Intelligent Orchestrator.

management interface

Tivoli Intelligent Orchestrator has two management interfaces: a Web interface, and a command-line interface. The We b interface provides a graphical representation of the data center, includes wizards to simplify configuration, and offers features such as reporting and task status monitoring that are not available from the command-line interface. The command-line interface provides access to Tivoli Intelligent Orchestrator features with SOAP. Administrators have the flexibility to perform tasks such as creating scripts that run specific SOAP commands or setting up external tools to send SOAP commands in response to an event.

Prerequisites installed with Tivoli Intelligent Orchestrator

Tivoli Intelligent Orchestrator interacts with other components that are installed with it.

4 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

WebSphere Application Server

WebSphere Application Server hosts management interfaces and coordinates interaction between other system components.

Database

server

The database server stores the data center model and report and audit data.

Built-in troubleshooting features

Tivoli Intelligent Orchestrator records system activity and events in message logs, trace logs, and Common Base Event logs. You can view the contents of these log files in a text editor.

Problem classification

This guide provides information about Tivoli Intelligent Orchestrator tools, resources, and techniques that can help you identify and resolve the following types of problems:

v “Problems with installation” on page 55

v “Problems with logging in” on page 62

v “Problems with workflows” on page 64

v “Other common problems” on page 68

For details about problems and resolutions for these issues, see Chapter 7, “Common problems and known limitations in Tivoli Intelligent Orchestrator,” on page 55.

Product maintenance

Problems can often be avoided with planning and preparation before you deploy Tivoli Intelligent Orchestrator. Before you install the software, review the Tivoli

Intelligent Orchestrator Release Notes, the Tivoli Intelligent Orchestrator Installation Guide, and the Tivoli Intelligent Orchestrator Migration Guide. These documents

contain the following important information:

v Supported operating system levels

v Prerequisite software requirements

v Required software fix packs

v Minimum and recommended memory requirements

v Disk space requirements

v Upgrade considerations

v Known problems, limitations, and recovery procedures

Backing up the system

After you have installed Tivoli Intelligent Orchestrator, ensure that you have a comprehensive backup and system recovery strategy in place. Because Tivoli Intelligent Orchestrator does not include tools for backing up or restoring your system, you should back up your system in accordance with the documentation that is provided with your operating system or with any specialized backup and restore software that you use.

Chapter 1. Introduction to troubleshooting 5

You might want to consider the following general recommendations for backing up your Tivoli Intelligent Orchestrator system:

v Perform a full backup of your existing DB2 Universal Database or Oracle

database. General guidelines for backing up DB2 Universal Database and Oracle9i Database data are provided in the Tivoli Intelligent Orchestrator Version

5.1 Migration Guide. For more details, refer to the DB2 Universal Database information center available at

http://publib.boulder.ibm.com/infocenter/db2help/index.jsp

or to the Oracle9i documentation available at

http://www.oracle.com/technology/documentation/oracle9i.html

v Back up your directory server database. Considerations for Microsoft Active

Directory and general guidelines for backing up an IBM Tivoli Directory Server database are provided in the Tivoli Intelligent Orchestrator Version 5.1 Migration Guide. For more backup and migration details, refer to the IBM Tivoli Directory Server Installation and Configuration documentation available at:

http://publib.boulder.ibm.com/tividd/td/IBMDirectoryServer5.2.html

v Back up your automation packages (.tcdriver files) using any of the standard

backup and restore tools that you are currently using. For more details on automation packages, refer to the Developing automation packages section of the Tivoli Intelligent Orchestrator information center.

For Tivoli Intelligent Orchestrator publications:

1. Go to http://www.ibm.com/software/tivoli/products/intell-orch/

2. Click Technical documentation on the left.

For Tivoli Provisioning Manager publications:

1. Go to http://www.ibm.com/software/tivoli/products/prov-mgr/

2. Click Technical documentation on the left.

Performing periodic checks and maintenance

In addition to the preceding backup guidelines, here are several best practices that can help you do even more to prevent problems:

v On Windows, periodically back up the user registry following the instructions

provided by the user registry vendor.

v Periodically check that all systems that are running Tivoli Intelligent

Orchestrator have sufficient disk space for runtime and problem determination data. As your security policy grows, and the number of users, groups, and protected objects increase, the space requirements for the policy databases, message logs, trace logs, Common Base Event logs, and any auditing information can increase as well. For more information, see Chapter 4, “Locating and configuring log files,” on page 39.

v Regularly check for the availability of fix packs and install them as they become

available. Information on fix packs and other usual information can be found on the Tivoli Intelligent Orchestrator support Web site, at:

http://www.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html

Problem resolution

Use the information in this guide to identify and resolve potential problems. If you cannot correct the problem, gather the relevant diagnostic information as described in this guide and then use the information in Chapter 9, “Contacting IBM Tivoli Software Support,” on page 91 to get further assistance.

6 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator

This chapter describes how to approach troubleshooting in Tivoli Intelligent Orchestrator. Providing methodologies and troubleshooting methods for identifying problems in Tivoli Intelligent Orchestrator will help you to reduce the time spent finding resolutions to these problems.

The following sections are described:

v “Problem determination methodologies”

v “Troubleshooting methods for Tivoli Intelligent Orchestrator and prerequisites”

on page 9

Problem determination methodologies

The keys to the problem determination process are understanding the problems and identifying the causes of problems. Problem determination in Tivoli Intelligent Orchestrator might be quite a difficult process at times, due to the complexity of the product, the fact that it integrates with many different products, the number of software prerequisites the product relies on, and also to the variety of platforms the product can be installed and configured on.

The following steps are required to identify and resolve a problem in a Tivoli Intelligent Orchestrator environment in a systematic manner:

v “Defining the problem”

v “Identifying the cause of the problem” on page 8

v “Fixing the problem” on page 9

Defining the problem

The first step is made when you determine that the product is not functioning in the expected manner. Then, you try to correctly identify the problem, by gathering information about the context that triggered the problem and about the problem itself. This phase requires the following steps:

v Identifying the problem symptoms

v Providing a complete description of the error messages generated by the system

During

v How did you first notice the problem?

v Where were you in the process when the problem occurred?

this phase, you must be prepared to answer questions like:

– Are you a first-time user of Tivoli Intelligent Orchestrator?

– We re you installing a prerequisite?

– We re you configuring a prerequisite?

– We re you installing Tivoli Intelligent Orchestrator? If yes, was this the first

installation?

– We re you performing post-installation configurations?

– We re you performing a runtime task?

Have you previously performed the same procedure without getting errors?

What was done differently this time?

v What was the first symptom of the problem? Have you noticed other symptoms

occurring simultaneously?

v Does the problem affect only one system or multiple systems?

v Did you receive an error message indicating what the problem is?

v If the problem can be reproduced, what are the steps that you need to perform

to recreate it?

Identifying the cause of the problem

The information you gathered during the previous phase helped you to correctly identify the problem and describe the context that triggered it. To be able to identify and locate the cause of the problem, it is very important that you understand the Tivoli Intelligent Orchestrator architecture and the interaction between its components. This will help you to determine the specific component that is involved with the problem:

v Data center model

v Policy engine

v Deployment engine

v Automation packages

v Discovery technologies

v Management interface

Problems

limited to a single product component have causes that are easier to

identify. Other problems might affect various components and their causes might be subtle and more difficult to identify. Furthermore, different problems may have the same symptoms, but different causes. Ideally, you should be able to approach the problem in such a way so that you can isolate it to a single component.

If you cannot identify the cause of a problem, you might want to seek the assistance of the Tivoli Support team, who will be able to pinpoint the cause of the problem and recommend ways to recover from specific situations. For more information on how to contact the IBM Tivoli Software Support, refer to Chapter 9, “Contacting IBM Tivoli Software Support,” on page 91.

As Tivoli Intelligent Orchestrator integrates many different products, some of the problem determination issues that might be considered difficult at times are:

Determining where the problem lies and what it is

The first challenge would be to determine if the problem is a customer error, triggered by an incorrect use of the product, or an error of the product itself, that is, a defective piece of software or hardware. Errors encountered at startup are often missed, resulting in symptoms that are encountered down the road. It is always worthwhile verifying whether problems or exceptions are encountered at startup, before debugging a symptom found down the road. Any problem found in a startup trace should be addressed first.

Verifying whether the problem can be replicated

You need a non-production environment on which to replicate the problem and do all tracing and analysis. Having a non-production environment available would also allow you to test the problem on different platforms.

Checking

whether the problem has occurred before

If the same problem or a similar problem have been dealt with before, it is very likely that extensive support documentation is provided. Begin by

8 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

searching the available IBM technotes for your product and the available knowledge bases to determine whether the resolution to your problem is already documented.

Analyzing

the actions that led up to the problem and generating an action plan

The approach that you should take must make use of all the available resources and tools. You should start by verifying whether all the hardware and software prerequisites have been met. While focusing on the problem itself, you should always take into account the bigger picture of the Tivoli Intelligent Orchestrator environment.

Fixing the problem

Once the problem is well defined and understood and its cause is correctly identified, corrective steps are required to solve the problem. As an additional step to this phase, it is always worthwhile documenting the problem for the future, to expedite the resolution of further possible problems.

Troubleshooting methods for Tivoli Intelligent Orchestrator and prerequisites

Based on the feedback received from our Support teams, the following methods have been proposed for troubleshooting Tivoli Intelligent Orchestrator and its prerequisites. Used in addition to your own product knowledge and experience, these troubleshooting methods might help you to shorten the time spent looking for a problem resolution, by better orienting you inside the Tivoli Intelligent Orchestrator problem determination process.

At all times during the problem determination process, keep the following points in mind:

v The WebSphere Application Server is central to Tivoli Intelligent Orchestrator

and its components. All components, DB2 Universal Database, IBM Tivoli Directory Server, SOAP, user interface, deployment engine, policy engine, command line tools, interact with the WebSphere Application Server. If you do not know where to start with a problem that you have encountered, start with WebSphere Application Server. Check the SystemOut.log file for errors. The log file is located in the following directory:

– On Windows: %WAS_HOME%\logs\<server-name> – On UNIX: $WAS_HOME/logs/<server-name>

v Isolate the root error using the proposed troubleshooting methods to find a

resolution.

v Take screen shots of specific steps and errors received along the way

v Utilize and exploit the knowledge bases.

methods that are provided for troubleshooting the Tivoli Intelligent

The Orchestrator are series of tests or questions. Troubleshooting methods are organized so that you can start with general questions and then continue with more specific questions that allow you to narrow down the problem space. Based on the answers to these questions, you can isolate the problem and find a resolution to it.

Depending on the stage you are encountering the problem, you must identify the starting troubleshooting method within that stage, and then follow it to find the resolution to your problem. Many troubleshooting methods will point you to the knowledge bases.

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 9

The following stages are described to the Tivoli Intelligent Orchestrator problem determination process:

1. “Troubleshooting the prerequisite installation”

2. “Tivoli Intelligent Orchestrator installation and subcomponent configuration

verification” on page 17

3. “Tivoli Intelligent Orchestrator runtime problem determination” on page 21

4. “Accessing the knowledge bases” on page 26

5. “Using the available command line tools” on page 26

Troubleshooting the prerequisite installation

The following steps are required during this phase:

1. Ensure that the operating system has all required fixes and levels. For a

complete list of operating system environments that are supported by Tivoli Intelligent Orchestrator, refer to the Tivoli Intelligent Orchestrator Installation Guide, Version 5.1.

2. For your specific server topology, ensure that the prerequisite installer installed

the required versions and patches for the prerequisites and their related components, as described in detail in the Tivoli Intelligent Orchestrator Installation Guide. Tivoli Intelligent Orchestrator requires the following prerequisites:

v DB2 Universal Database Enterprise Server Edition, Version 8.2, Fix Pack 11 or

2000Solaris

: Oracle 9i

v IBM Tivoli Directory Server, Version 6.0, Fix Pack 1

v WebSphere Application Server, Version 6.0, Fix Pack 2.

After the prerequisite installation, you might want to follow these steps:

v Ensure that the configuration of the IBM Tivoli Directory Server database

was successful. If not, refer to the “IBM Tivoli Directory Server verification” on page 11 section.

v Perform a subcomponent verification, as described in the following

subsections:

– “DB2 Universal Database verification”

– “IBM Tivoli Directory Server verification” on page 11

– “WebSphere Application Server verification” on page 13

DB2 Universal Database verification

Follow these steps to ensure that the DB2 Universal Database was successfully started:

1. Use the following command to start the database:

v On UNIX: $ db2start

For example, db2start on AIX has the following output:

$ db2start 07/13/2005 11:54:07 0 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful

v On Windows:

– From the services window, start the service DB2 - <instance_name>. – The default service name in Tivoli Intelligent Orchestrator 5.1 is DB2 -

DB2.

Verify whether the designated port is available by running this command:

netstat -an |grep 50000

3. Verify if all required DB2 Universal Database processes are up and running:

10 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

v On Windows, launch Task Manager and check the Processes window.

v On UNIX, run the following command to list all DB2 Universal

Databaseprocesses: db2_local_ps

required DB2 Universal Database processes include:

The

v db2fmd

v db2dasrrm

v db2fmcd

v db2wdog

v db2sysc

v db2ckpwd

v db2gds

v db2syslog

v db2ipccm

v db2tcpcm

v Db2resync

v db2srvlst

v db2agent (LDAP)

v db2fmp

v db2agent (TIOINTER)

v db2loggr (TC)

v db2loggw (TC)

v db2dlock (TC)

v db2pfchr

v db2pclnr

v db2event

4. Check the db2diag.log file for errors. The file is located in the following

directory:

v On Windows: <DB2_Instance_Home_Directory> v On UNIX: <instance-owner-home>/sqllib/db2dump

Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

IBM Tivoli Directory Server verification

The verification requires these steps:

1. Ensure that the IBM Tivoli Directory Server database was successfully created.

Check the ldapcfg.stat file to see the syntax used at the time of the database creation. The file is located in the following directory:

v On Windows: C:\IBM\ldap\tmp v On UNIX: /usr/ldap/tmp

should see an output similar to the following:

You

v Windows:

C:\IBM\ldap\bin\ldapcfg -n -a db2inst1 -w password -d LDAP -l C: -c -f C:\IBM\ldap\tmp\ldapcfg.dat

v UNIX:

/usr/ldap/bin/ldapcfg -n -a db2inst1 -w th1nk -d ldap -l /home/db2inst1 -c -f /usr/ldap/tmp/ldapcfg.dat

2. Ensure that the IBM Tivoli Directory Server was successfully started:

a. Verify whether the IBM Tivoli Directory Server database was successfully

started, using the ibmdirctl command line utility:

ibmdirctl -D cn=root -w <password> status

The ibmdirctl command is located in the following directory:

v On Windows: C:\IBM\ldap\bin v On UNIX: /usr/ldap/bin

To be able to use the ibmdirctl command, the IBM Directory

Note:

Administrator service must be running.

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 11

Use the Windows Services panel, or issue the ibmdiradm command on UNIX to verify if the service is running. IBM Directory Administrator uses the port number 3538 for requests.

You can also start and stop IBM Tivoli Directory Server using the ibmdirctl command:

v Starting the IBM Tivoli Directory Server:

ibmdirctl -D cn=root -w <password> start

v Stopping the IBM Tivoli Directory Server:

ibmdirctl -D cn=root -w <password> stop

3. You can also use ldapsearch command line utility to verify whether IBM Tivoli

Directory Server is running, configured properly, and responding to queries:

ldapsearch -h <ldapserver> -p 389 -D cn=root -w <password>

-b "" -s base objectclass=*

The ldapsearch command is also located in the following directory:

v On Windows: C:\IBM\ldap\bin v On UNIX: /usr/ldap/bin

For more information on command line tools that can be used during the IBM Tivoli Directory Server problem determination process, refer to “IBM Tivoli Directory Server command line tools” on page 33.

4. Verify whether the DB2 database is successfully started. For more information,

refer to “DB2 Universal Database verification” on page 10.

5. Check the runtime IBM Tivoli Directory Server log files for errors. The logs are

stored in the following directory:

v Windows: <LDAP_Install_Directory>\var\ v UNIX: /var/ldap/

<LDAP_Install_Directory> is the installation directory for the LDAP

where server.

The log files include:

v ibmslapd.log v audit.log v db2diag.log v ibmds_config.log v ibmds_config_err.log

If required, edit the ibmslapd.conf file, which is the IBM Tivoli Directory

Server configuration file that controls the server operation. The file is located in the following directory:

v On Windows: C:\IBM\ldap\etc v On UNIX: /usr/ldap/etc

IBM Tivoli Directory Server must be restarted for the edits to the

Note:

ibmslapd.conf file to take effect.

The ibmslapd.conf file can be used for:

v Changing the IBM Tivoli Directory Server log locations

v Adding and removing IBM Tivoli Directory Server suffixes

v Controlling server startup parameters and database connection parameters

v Changing administrator user credentials

v Modifying SSL security settings

12 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

7. Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

WebSphere Application Server verification

The verification requires the following steps:

1. Perform a WebSphere Application Server installation verification test, by

running the following script:

v On Windows: %WAS_HOME%\bin\ivt.bat v On UNIX: $WAS_HOME/bin/ivt.sh

the ivt.log file for errors. The log file is located in the following

Check directory:

v On Windows: %WAS_HOME%\logs v On UNIX: $WAS_HOME/logs

2. Ensure that the correct WebSphere Application Server version was installed,

and verify the WebSphere Application Server status: v Use the versionInfo command line utility to display the current level of

WebSphere Application Server. The command is located in the following directory:

– On Windows: %WAS_HOME%\bin – On UNIX: $WAS_HOME/bin

Use the serverStatus command line utility to display the WebSphere

Application Server status:

serverStatus -all -user <username> -password <password>

The command is located in the following directory: – On Windows: %WAS_HOME%\bin – On UNIX: $WAS_HOME/bin

information is logged in the serverStatus.log file, located in the

Tool %WAS_HOME%\logs directory on Windows, $WAS_HOME/logs on UNIX.

For more information on command line tools that can be used during the WebSphere Application Server problem determination process, refer to “WebSphere Application Server command line tools” on page 32.

3. If required, edit the WebSphere Application Server configuration files. The

master configuration repository is located at $WAS_HOME/config. The configuration files contain the following information:

v Cell configuration information in XML files v Applications held as EAR files and deployment descriptors v Templates as XML files that can be used for creating resources

Two administration clients are provided to edit the configuration of all components:

v The administrative console, available at http://<WAS

hostname.domainname>:9090/admin/

v The wsadmin shell scripting client

The following is a list of useful configuration files:

Security.xml

This file contains server security configuration, and is located in the $WAS_HOME/config/cells/<cell> directory. When encountering problems logging in to the WebSphere Application Server

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 13

administrative console, security can be disabled in this file by changing the value of the enabled parameter in the second line from true to false.

By default, <cell> represents the host name.

Server.xml

This file contains server-specific configuration information, and is located in the $WAS_HOME/config/cells/<cell>/nodes/<node>/servers/ server1 directory.

Java.security

This file contains security properties used by the java.security classes. It is located in the $WAS_HOME/java/jre/lib/security directory.

Sas.client.props

This file contains security settings that indicate the Java Client how to access secured applications. It is located in the $WAS_HOME/properties directory.

Check the runtime WebSphere Application Server log files for errors. The logs

are stored in the following directory:

v On Windows: %WAS_HOME%\logs\<server-name> v On UNIX: $WAS_HOME/logs/<server-name>

<server-name> is by default server1.

where

The runtime WebSphere Application Server log files include:

SystemOut.log

This log file contains messages generated when the applications running inside the application server are being started or stopped. When startingTivoli Intelligent Orchestrator, use the following command to monitor this log file for any problems:

tail -f SystemOut.log

SystemErr.log

This log file contains Java exceptions and stack traces caused by applications and their applications.

startServer.log

This log file contains errors that occur during the startup of the WebSphere Application Server. Look for the following output for a successful startup:

Server <server-name> open for e-business.

stopServer.log

This log file contains errors that occur during the shutdown of the WebSphere Application Server. Look for the following output for a successful shutdown:

Server <server-name> stop completed.

5. Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

Problem

determination using SystemOut.log: The following steps are required

when troubleshooting the WebSphere Application Server using SystemOut.log:

1. Look for any error during the WebSphere Application Server startup.

v If you see an error before the point where you see ...ready for e-business,

check the WebSphere Application Server configuration.

14 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

v If no error can be found, look for other symptoms in the log file.

Find the first occurrence of the error in the log file. You will either see an error

message or an error exception with a stack trace.

3. Review the proper error message. It will be either from WebSphere Application

Server or from the application that caught and properly handled the exception. Review the message and investigate.

4. Check the Java exception with the stack trace:

v Check the stack trace and follow the classes, with the package names, and

try to identify them.

v If all the classes belong to the application server and you can easily identify

them by package names, you probably have a configuration error.

v Tr y to detect the source of the problem based on class names, which are

usually very descriptive.

v If you find any class in the stack trace that belongs to the enterprise

application, you need to check the application.

SystemOut.log

successful startup breakdown:

Administration service

AdminInitiali A ADMN0015I: AdminService initialized

Secure Authentication Services (LDAP)

Configuration A SECJ0215I: Successfully set JAAS login provider

configuration class to com.ibm.ws.security.auth.login.Configuration.

SecurityDM I SECJ0231I: The Security component’s FFDC Diagnostic Module

com.ibm.ws.security.core.SecurityDM

registered successfully: true. SecurityCompo I SECJ0309I: Java 2 Security is disabled. SecurityCompo I SECJ0212I: WCCM JAAS configuration information successfully

pushed to login provider class.

SecurityCompo I SECJ0240I: Security service initialization completed

successfully SASRas A JSAS0001I: Security configuration initialized. SASRas A JSAS0002I: Authentication protocol: CSIV2/IBM SASRas A JSAS0003I: Authentication mechanism: LTPA SASRas A JSAS0004I: Principal name: todaix05.think.lab.austin.-

ibm.com:389/wasadmin SASRas A JSAS0005I: SecurityCurrent registered. SASRas A JSAS0006I: Security connection interceptor initialized. SASRas A JSAS0007I: Client request interceptor registered. SASRas A JSAS0008I: Server request interceptor registered. SASRas A JSAS0009I: IOR interceptor registered.

Map Tivoli Intelligent Orchestrator resource references

ResourceMgrIm I WSVR0049I: Binding DataSource as jdbc/KulaDB ResourceMgrIm I WSVR0049I: Binding Default Datasource as

ResourceMgrIm I WSVR0049I: Binding QueueConnectionFactory as

ResourceMgrIm I WSVR0049I: Binding DeploymentEnginePrivateRequests

ResourceMgrIm I WSVR0049I: Binding DeploymentEngineHighLevelOutput

ResourceMgrIm I WSVR0049I: Binding TopicConnectionFactory as jms/factory/TopicConnectionFactory ResourceMgrIm I WSVR0049I: Binding DeploymentEngineHighLevelOutputForTEC as jms/factory/DeploymentEngine/HighLevelOutputForTEC ResourceMgrIm I WSVR0049I: Binding DeploymentEnginePrivateRequests as queue/thinkcontrol/DeploymentEngine/PrivateRequests ResourceMgrIm I WSVR0049I: Binding DeploymentEngineHighLevelOutput as topic/thinkcontrol/DeploymentEngine/HighLevelOutput ResourceMgrIm I WSVR0049I: Binding ThinkControlInternalMessages as topic/thinkcontrol/Invocation

DefaultDatasource

jms/factory/QueueConnectionFactory

as jms/factory/DeploymentEngine/ PrivateRequests

as jms/factory/DeploymentEngine/ HighLevelOutput

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 15

ResourceMgrIm I WSVR0049I: Binding Default_CF as eis/DefaultDatasource_CMP ResourceMgrIm I WSVR0049I: Binding AgentRegistry as jdbc/AgentRegistry

Simple Object Access Protocol ( SOAP)

JMXSoapAdapte A ADMC0013I: SOAP connector available at port 8880

Applications

ApplicationMg A WSVR0200I: Starting application: DefaultApplication EJBContainerI I WSVR0207I: Preparing to start EJB jar: Increment.jar EJBContainerI I WSVR0037I: Starting EJB jar: Increment.jar . . . ApplicationMg A WSVR0221I: Application started: DefaultApplication

ApplicationMg A WSVR0200I: Starting application: ivtApp EJBContainerI I WSVR0207I: Preparing to start EJB jar: ivtEJB.jar EJBContainerI I WSVR0037I: Starting EJB jar: ivtEJB.jar . . . ApplicationMg A WSVR0221I: Application started: ivtApp

ApplicationMg A WSVR0200I: Starting application: adminconsole WebContainer A SRVE0169I: Loading Web Module: adminconsole. . . . ApplicationMg A WSVR0221I: Application started: adminconsole

TCEAR Application

ApplicationMg A WSVR0200I: Starting application: TCEAR EJBContainerI I WSVR0207I: Preparing to start EJB jar: tcEJB.jar EJBContainerI I WSVR0037I: Starting EJB jar: tcEJB.jar EJBContainerI I WSVR0207I: Preparing to start EJB jar: jdsEJB.jar EJBContainerI I WSVR0037I: Starting EJB jar: jdsEJB.jar WebContainer A SRVE0169I: Loading Web Module: tcWebUI.war. . . . WebContainer A SRVE0169I: Loading Web Module: tcSoap. ... WebContainer A SRVE0169I: Loading Web Module: odiWebServices. ... WebContainer A SRVE0169I: Loading Web Module: odiWebServicesClient. ... WebContainer A SRVE0169I: Loading Web Module: jdsWAR. ... MDBListenerIm I WMSG0042I: MDB Listener RecommendationsMDBPort started successfully for JMSDestination topic/thinkcontrol/DeploymentEngine/ HighLevelOutput MDBListenerIm I WMSG0042I: MDB Listener TECEventSenderMDBPort started successfully for JMSDestination topic/thinkcontrol/DeploymentEngine/ HighLevelOutput MDBListenerIm I WMSG0042I: MDB Listener DEAdaptorMDBPort started successfully for JMSDestination topic/thinkcontrol/DeploymentEngine/ HighLevelOutput ApplicationMg A WSVR0221I: Application started: TCEAR

Agent Manager application

ApplicationMg A WSVR0200I: Starting application: AgentManager WebContainer A SRVE0169I: Loading Web Module: AgentManager. ... ApplicationMg A WSVR0221I: Application started: AgentManager

ApplicationMg A WSVR0200I: Starting application: AgentRecoveryService WebContainer A SRVE0169I: Loading Web Module: AgentRecovery. ... ApplicationMg A WSVR0221I: Application started: AgentRecoveryService

HTTP Transport Service

HttpTransport A SRVE0171I: Transport http is listening on port 9,080. HttpTransport A SRVE0171I: Transport https is listening on port 9,443. HttpTransport A SRVE0171I: Transport http is listening on port 9,090.

16 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

HttpTransport A SRVE0171I: Transport https is listening on port 9,043. HttpTransport A SRVE0171I: Transport https is listening on port 9,511. HttpTransport A SRVE0171I: Transport https is listening on port 9,512. HttpTransport A SRVE0171I: Transport http is listening on port 9,513.

Remote Method Invocation (RMI)

RMIConnectorC A ADMC0026I: RMI Connector available at port 2809

Startup Complete

WsServer A WSVR0001I: Server server1 open for e-business

Tivoli Intelligent Orchestrator installation and subcomponent configuration verification

Tivoli Intelligent Orchestrator configures all of the subcomponents during its installation. You might also choose to manually configure all or some of the subcomponents, as required. For more information on Tivoli Intelligent Orchestrator installation log files, refer to Chapter 5, “Tivoli Intelligent Orchestrator log files,” on page 45.

Tivoli Intelligent Orchestrator installation

If you are installing Tivoli Intelligent Orchestrator using the Tivoli Intelligent Orchestrator installer, use the Tivoli Intelligent Orchestrator installation wizard to enter connection information such as user IDs, passwords, paths, fully qualified domain information, and ports, to connect to the subcomponents. Tivoli Intelligent Orchestrator will check the validity of this information. Yo u can review the configuration and installation information that is displayed at the end of the installation wizard, when the progress indicator is shown.

The Tivoli Intelligent Orchestrator installer completes the following operations:

v Creates subfolders for the Tivoli common directory. v Creates the tioadmin user or group if it does not already exist.

v Sets up the license files.

v Copies the Tivoli Intelligent Orchestrator file structure to the Tivoli Intelligent

Orchestrator installation directory.

v Installs the Tivoli Intelligent Orchestrator files. v Copies the automation packages to <TIO HOME>/drivers.

v Copies the Eclipse plug-ins for online help system.

v Installs the language files.

v Installs GUID.

v If the Install Netview check box is selected, the Netview installer is launched.

v If the Copy Repository CD check box is selected, disk 9 that contains the agent

installers is copied to the specified directory.

v Defines the environment variables.

v Sets up the configuration data. Configuration information entered into the Tivoli

Intelligent Orchestrator wizard is stored in the %TIO_HOME%\config directory on Windows ($TIO_HOME/config on UNIX). By editing the XML files in this directory, it ensures that the placeholders are replaced by correct values.

v On Windows, it creates Tivoli Intelligent Orchestrator start and stop desktop

icons.

v Generates the agent manager response file template.

v Sets up the DB2 database.

v Configures the IBM Tivoli Directory Server.

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 17

v Starts the WebSphere Application Server.

v Configures the WebSphere Application Server, including:

– WebSphere Application Server configuration

– WebSphere Application Server deployment

– WebSphere Application Server setup on UNIX, to run as user configuration

v Enables security for the WebSphere Application Server.

v On Windows, it creates the Windows Service.

v On Windows, it creates keys in Windows registry.

v Reinitializes the database.

v The Tivoli Intelligent Orchestrator uninstaller is automatically created by the

Install Shield MultiPlatform (ISMP).

v If the Install Agent Manager check box is selected, the agent manager installer

is launched.

Verifying subcomponent configuration

The following procedures are required to confirm that Tivoli Intelligent Orchestrator and subcomponents have been successfully configured:

v Verifying Tivoli Intelligent Orchestrator’s connection to DB2 Universal Database.

See “DB2 Universal Databaseconfiguration verification.”

v Verifying the IBM Tivoli Directory Server configuration. See “IBM Tivoli

Directory Server configuration verification” on page 19.

v Verifying Tivoli Intelligent Orchestrator’s connection to IBM Tivoli Directory

Server. See “IBM Tivoli Directory Server connection verification” on page 20.

v Verifying the WebSphere Application Server configuration. See “WebSphere

Application Server configuration verification” on page 21.

Universal Databaseconfiguration verification: Tivoli Intelligent Orchestrator

DB2

uses two mechanisms to connect to the database:

1. Java connection using the information in the dcm.xml file, located at %TIO_HOME%\config on Windows ($TIO_HOME/config on UNIX):

<database> <type>db2</type> <driver>COM.ibm.db2.jdbc.app.DB2Driver</driver> <url>jdbc:db2:TC</url> <name>TC</name> <username>db2inst1</username> <password>QJsdAjUEHi8=</password> </database>

</config>

2. Kula user through the WebSphere Application Server.

In the administrative console, go to Security –> JAAS Configuration –> J2C Authentication Data to connect to the database.

troubleshooting the DB2 Universal Database connection with Tivoli

When Intelligent Orchestrator, follow these steps:

1. Try to manually connect to the database, and run the following command to list all DB2 Universal Database tables:

list tables

18 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Verify if the DCM_OBJECT table is listed. If not, determine what user ID is required to connect to see the DCM_OBJECT table, and update the dcm.xml file and the kula user appropriately.

If the table list is successfully created, verify the DB2 Universal Database connection information in the dcm.xml file.

2. If you cannot connect manually to the database, check if the database is started

successfully. See “DB2 Universal Database verification” on page 10. You might also want to run the following command to verify if the designated port is available:netstat -an grep| 50000.

3. Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

Tivoli Directory Server configuration verification: The following steps are

IBM

required:

Note that all ldapsearch commands are based on the default tree path,

Note:

dc=ibm,dc=com. If this is changed, then you should use your custom tree path.

1. Verify whether the IBM Tivoli Directory Server was successfully started, using

the ibmdirctl command line utility. For more information, see “IBM Tivoli Directory Server verification” on page 11.

2. If the IBM Tivoli Directory Server was successfully started, check the IBM

Tivoli Directory Server’s basic functionality, using the ldapsearch command line utility:

ldapsearch -h <ldapserver> -p 389 -D cn=root

-w <password> -b "" -s base objectclass=*

This search should return about sixty lines of ldap information. On Windows, you must use the IBM ldapsearch command, not the Cygwin

OpenLDAP ldapsearch command. The IBM ldapsearch command is located in the following directory:

v On Windows: C:\IBM\ldap\bin v On UNIX: /usr/ldap/bin

Check the IBM Tivoli Directory Server suffixes, using the ldapsearch command:

ldapsearch -v -h <ldapserver> -D cn=root -w <password>

-p 389 -b "" -s base objectclass=* | grep namingcontexts

Note: Not supplying the -b option is resource-intensive.

4. Check the IBM Tivoli Directory Server schema:

ldapsearch -b cn=schema -s base objectclass=* | grep -i think

5. Check all users and groups using the ldapsearch command. The following

steps are required:

a. Confirm proper import of all user and group entries:

ldapsearch -v -h < ldapserver> -D cn=root

-w <password> -p 389 -b "dc=ibm,dc=com" cn=*

The output should include over one hundred and ninety matches.

b. Confirm the proper import of all Tivoli Intelligent Orchestrator user entries:

ldapsearch -v -h < ldapserver > -D cn=root

-w <password> -p 389 -b "dc=ibm,dc=com" cn=tio*

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 19

The output should return the following entries: tioldap, tiointernal, and tioappadmin.

c. Confirm that a particular user or group has been imported:

ldapsearch -v -h < ldapserver > -D cn=root

-w <password> -p 389 -b "dc=ibm,dc=com" cn=<username>

This should return the full output of the specified user.

d. Ensure the proper functionality with the wasadmin user:

ldapsearch -v -h < ldapserver > -w wasadmin

-D "cn=wasadmin,dc=ibm,dc=com" -p 389 -b "dc=ibm,dc=com" cn=tio*

The output should return the following entries: tioldap, tiointernal, tioappadmin.

Check for proper tree ownership, to ensure that Tivoli Intelligent Orchestrator

is able to navigate the user and group structure accordingly:

ldapsearch -v -h <ldapserver> -p 389 -D cn=root -w <password>

-b "dc=ibm,dc=com" -s base "objectclass=*" entryowner ownerpropagate

The output should return the following: ownerpropagate=TRUE, and entryowner=access-id:CN=TIOLDAP,DC=IBM,DC=COM

7. Search the knowledge bases and, if necessary, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

Tivoli Directory Server connection verification: The following steps are

IBM

required:

1. Check Tivoli Intelligent Orchestrator’s connection to the IBM Tivoli Directory Server:

a. Check the user-factory.xml file for changes that were made during the

Tivoli Intelligent Orchestrator installation. The user-factory.xml file is Tivoli Intelligent Orchestrator’s configuration file for accessing the IBM Tivoli Directory Server, and is located in the following directory:

v Windows: %TIO_HOME%\config v UNIX or Linux: $TIO_HOME/config

Changes to the server name, port, root suffix, principal user, internal user, encrypted passwords, and SSL-binding are shown as follows:

<server>todaix03.think.lab.austin.ibm.com</server> <ldap-port>389</ldap-port> <root>dc=ibm,dc=com</root> <principal>tioldap</principal> <internal-user>tiointernal</internal-user>

2. In the WebSphere administrative console, go to Security –> User Registries –> LDAP, to check the WebSphere Application Server connection to the IBM Tivoli

Directory Server.

3. Verify the IBM Tivoli Directory Server connection:

a. Check the IBM Tivoli Directory Server basic functionality using the

following command:

ldapsearch -v -h <ldapserver> -p 389 -D cn=root

-w <password> -b "" -s base objectclass=*

b. If the ldapsearch search returns a command not found, install the IBM Tivoli

Directory Server Client, following the instructions provided in the Tivoli Intelligent Orchestrator Installation Guide.

20 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

c. If the ldapsearch search returns a connection error, start a Telnet session

and try to connect to the IBM Tivoli Directory Server using the port number

389.

d. If communication can be established with a Telnet session between the IBM

Tivoli Directory Server and Tivoli Intelligent Orchestrator, check the IBM Tivoli Directory Server administrative password.

4. Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

WebSphere

Application Server configuration verification: The following steps

are required:

1. Check the SystemOut.log file for errors. For more information, refer to

“Problem determination using SystemOut.log” on page 14.

2. Verify the components in the WebSphere administrative console. If there are

missing components, search the knowledge bases and, if necessary, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

Tivoli Intelligent Orchestrator runtime problem determination

The runtime problem determination process for Tivoli Intelligent Orchestrator can be divided into component-based problem determination, and task-based problem determination. Depending on the type of error you are encountering, you must determine where to start the problem determination process.

v “Component-based problem determination”

v “Task-based problem determination” on page 23

Component-based problem determination

This is the base level of problem determination in the Tivoli Intelligent Orchestrator runtime environment. After you isolate the problem to a specific component, you can then refer to that component problem determination method, as follows:

v “DB2 Universal Database error problem determination”

v “DB2 Universal Database constraint problem determination”

v “IBM Tivoli Directory Server authentication problem determination” on page 22

v “WebSphere Application Server error problem determination” on page 23

Universal Database error problem determination: If you have encountered

DB2

a DB2 Universal Databaseerror, follow these steps:

1. If it is a DB2 Universal Database connection problem, refer to “DB2 Universal

Databaseconfiguration verification” on page 18.

2. If it is a constraint error or exception, refer to “DB2 Universal Database

constraint problem determination.”

3. If the problem you have encountered is neither a connection problem nor a

constraint error or exception, search the knowledge bases for the keyword error and, if necessary, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

Universal Database constraint problem determination: If you have

DB2

encountered a DB2 Universal Database constraint error or exception, follow these steps:

1. Using the error message you have received, identify the DB2 Universal

Database table with the constraint.

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 21

Constraints are listed in the $TIO_HOME/sql/db2/dbschema.sql file as foreign keys and primary keys.

2. Identify the type of constraint, by specifying whether it is an object addition or an object removal.

3. In the dbschema.sql file, search the ALTER TABLE constraint for the table that interests you. Use the table name from the error message and the type of constraint identified.

4. If it is a primary key constraint, use the constraint statement and the object you are inserting and look in the database to see whether an object with those values already exists. If it does, change the object you are inserting so that to make it unique.

5. If it is a foreign key constraint, identify the second table and the value that is involved in the constraint, and then delete the associated object that is described by the constraint, and then you should be able to delete your object.

Tivoli Directory Server authentication problem determination: If you have

IBM

encountered an IBM Tivoli Directory Server authentication problem on the Tivoli Intelligent Orchestrator server, follow these steps:

1. Check the IBM Tivoli Directory Server for root authentication:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=root"

-w <password> -b "dc=ibm,dc=com" cn=wasadmin

If it is a connection problem, refer to “IBM Tivoli Directory Server connection verification” on page 20.

2. Check the IBM Tivoli Directory Server for wasadmin authentication:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=wasadmin,dc=ibm,dc=com"

-w <password> -b "dc=ibm,dc=com" cn=wasadmin

If it is an authentication problem, perform the following:

v Verify the user credentials v Synchronize password using changePassword.sh

v Refer to “IBM Tivoli Directory Server connection verification” on page 20

Check the tioldap user:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=wasadmin,dc=ibm,dc=com"

-w <password> -b "dc=ibm,dc=com" cn=tioldap

4. Check the tioldap authentication:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=tioldap,dc=ibm,dc=com"

-w <password> -b "dc=ibm,dc=com" cn=tioldap

If it is an authentication problem, perform the following:

v Verify the user credentials v Synchronize password using changePassword.sh

v Refer to “IBM Tivoli Directory Server connection verification” on page 20

5. Check the tiointernal user:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=wasadmin,dc=ibm,dc=com"

-w <password> -b "dc=ibm,dc=com" cn=tiointernal

6. Check the tioappadmin user:

ldapsearch -v -h < ldapserver > -p 389 -D "cn=wasadmin,dc=ibm,dc=com"

-w <password> -b "dc=ibm,dc=com" cn=tioappadmin

7. If the encountered error is not related to either tioldap, tiointernal, or tioappadmin, compare the user output against ldap.ldif.

22 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

8. Search the knowledge bases and, if necessary, contact Support. For more

information, refer to Appendix B, “Support information,” on page 225.

WebSphere

Application Server error problem determination: If you have

encountered a WebSphere Application Server error, use SystemOut.log and follow these steps:

1. In the SystemOut.log file, find the last startup section by going to the end of the file and then searching upwards for "End Display". If you cannot find it, start the WebSphere Application Server. For more information, refer to “WebSphere Application Server verification” on page 13.

2. If you found the startup section, search for errors between "End Display" and open for e-business. If errors are found, go to the WebSphere Application

Server startup verification. See “WebSphere Application Server verification” on page 13.

3. If no errors are found, search for errors at the time stamp of the encountered problem. If no errors are found, go to the Tivoli Intelligent Orchestrator “Task-based problem determination.”

4. If errors are found, match the error found with any of the following LDAP keywords: SEC*, LDAP, LTPA, security. If there is an error match, go to “IBM Tivoli Directory Server authentication problem determination” on page 22.

5. If no match is found for the error, search the Tivoli Intelligent Orchestrator Support Knowledge Base for the error message. If the error message is found, resolve the issue using the knowledge entry.

6. If no error message is found, check the WebSphere Application Server Support Knowledge Base for the error message. If the error message is found, resolve the issue using the knowledge entry.

7. If no error message is found, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

Task-based problem determination

The task-based troubleshooting methodologies and methods described below can help you to either solve or isolate your problem to a particular component. To find a resolution to your problem, you might need to return to the component-based troubleshooting methods or to the verification troubleshooting methods.

Problem determination methods are proposed for the following runtime tasks:

v “Starting Tivoli Intelligent Orchestrator”

v “Tivoli Intelligent Orchestrator login” on page 24

v “Navigating the Tivoli Intelligent Orchestrator user interface” on page 24

v “User interface errors” on page 24

v “Running workflows” on page 25

v “Running commands on managed servers” on page 25

Starting

starting Tivoli Intelligent Orchestrator:

1. Verify whether the DB2 Universal Database is successfully started. If DB2

2. If DB2 Universal Database is started, check whether the IBM Tivoli Directory

Tivoli Intelligent Orchestrator: The following steps are required for

Universal Database is not started, start it. For more information, refer to “DB2 Universal Database verification” on page 10.

Server is started. If the IBM Tivoli Directory Server is not started, start it. For more information, refer to “IBM Tivoli Directory Server verification” on page

11.

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 23

3. If the IBM Tivoli Directory Server is started, check the SystemOut.log file for the following types of exceptions. For more information on the SystemOut.log file, refer to “Problem determination using SystemOut.log” on page 14:

a. If authentication exceptions are found in the log file, go to “IBM Tivoli

Directory Server authentication problem determination” on page 22.

b. If XA connection exceptions are found, go to “DB2 Universal Database error

problem determination” on page 21.

c. If other exceptions are found, go to “WebSphere Application Server error

problem determination” on page 23.

Intelligent Orchestrator login: The following steps are required for

Tivoli

troubleshooting the Tivoli Intelligent Orchestrator login:

1. If the login screen cannot be loaded, verify whether you can ping the Tivoli Intelligent Orchestrator server.

v If you can ping the server successfully, go to “WebSphere Application Server

verification” on page 13.

v If you cannot ping the server, resolve the required network issues.

2. If the login screen is loaded successfully, use the appropriate credentials to log in to the Tivoli Intelligent Orchestrator server.

v If the login fails, check the Tivoli Intelligent Orchestrator Knowledge Base

for firewall issues. If firewall issues are found, follow the resolution using the knowledge entry.

v If no firewall issues are found, go to “IBM Tivoli Directory Server

authentication problem determination” on page 22.

Navigating

the Tivoli Intelligent Orchestrator user interface: The following steps

are required for troubleshooting the Tivoli Intelligent Orchestrator user interface navigation:

1. If a COPXXXX error is encountered, go to “User interface errors.”

2. If you encounter a navigation problem, go to “IBM Tivoli Directory Server

authentication problem determination” on page 22.

3. If the error you have encountered cannot be included in any of the above categories, search the error on the Tivoli Intelligent Orchestrator Knowledge Base. If the error is documented on the knowledge base, follow the resolution using the knowledge entry.

4. If the error cannot be found on the knowledge base, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

interface errors: The following steps are required for troubleshooting the

User

Tivoli Intelligent Orchestrator user interface errors:

1. If a user interface error is encountered, search for the error in the %TIO_LOGS%\j2ee\console.log file using either the user interface error number or the error time stamp.

For example, if the following error message is received in the Tivoli Intelligent Orchestrator user interface:

COPCOM093E The JDBC driver caused an SQL exception. COPJEE272E (ProblemID: UI640236)

You would then search the j2ee/console.log file for the Problem ID: UI640236. In this case, you would find the corresponding error stack trace:

2005-05-02 10:24:11,108 ERROR [Servlet.Engine.Transports : 2] (Location.java:454) webui.Location: UI Exception [640236], user=null

24 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

2005-05-02 10:24:11,123 ERROR [Servlet.Engine.Transports : 2] (Location.java:506) webui.Location: Error Code = COPCOM093EdcmSqlException

ERROR MESSAGE = COPCOM093E The JDBC driver caused an SQL exception.

1. NESTED EXCEPTION: MESSAGE = [IBM][CLI Driver][DB2/6000]

SQL0532N A parent row cannot be deleted

because the relationship "DB2INST1.RECOMMENDATION_REQUEST.SQL050324085328960"

restricts the deletion.

SQLSTATE=23504 DETAILS: DB2INST1.RECOMMENDATION_REQUEST(POOL_ID) ->

DB2INST1.SPARE_POOL(POOL_ID) IP = 9.3.127.47

THREAD = Servlet.Engine.Transports : 2 STACK TRACE ==> com.thinkdynamics.kanaha.datacentermodel.DataCenter-

SystemException: COPCOM093E The JDBC driver caused an SQL exception....

2. Determine whether there is a root cause shortly above the error.

3. Use the logged error or the root cause to research your problem.

4. If a solution is still not found, try matching the error with the following

keywords: LTPA, LDAP, authentication, credential, token. If a match is found, go to “IBM Tivoli Directory Server authentication problem determination” on page 22.

5. If no match is found, try matching the error with the following keywords:

constraint, DB2, SQL, relationship, row, table. If a match is found, go to “DB2 Universal Database error problem determination” on page 21.

6. If no match is found, check the WebSphere Application Server SystemOut.log

file for errors. For more information, go to “WebSphere Application Server error problem determination” on page 23.

Running

workflows: The following steps are required for troubleshooting

workflows:

1. Try running a noop workflow. If you cannot run the noop workflow, the

deployment engine is not running. Restart Tivoli Intelligent Orchestrator.

2. Try running workflows as logical device operations.

v If you cannot run workflows as logical device operations, check if errors are

encountered against a particular object.

v If no errors are encountered against a particular object, check if errors occur

while running the workflow. If such errors are found, go to “Running commands on managed servers.”

v If no errors occur while running the workflow, search the Tivoli Intelligent

Orchestrator Knowledge Base. If the error is documented on the knowledge base, follow the resolution using the knowledge entry.

v If the error cannot be found on the knowledge base, contact Support. For

more information, refer to Appendix B, “Support information,” on page 225.

3. Verify if the faulty workflow is associated with a data center model object.

v If the workflow is associated with a data center model object, go to“Running

commands on managed servers.”

v If the workflow is not associated with a data center model object, associate it

with a data center model object.

Running

commands on managed servers: The following steps are required for

troubleshooting executing commands on managed servers:

1. Try isolating the error using the log info statements, or set the log level to DEBUG.

2. Try a simplified command such as uname -a against the managed server.

3. If trying a simplified command works, try running a command using the

command line interface:

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 25

ssh -l username IP ’command’

4. If running the command using the command line interface works, it is an

environment issue. In your workflow source, check your .profile first.

5. If running the command using the command line interface fails, run the

command directly on the managed server.

6. If running the command on the managed server works, debug the SSH

configuration issue. For more information on SSH configurations, refer to the Security section of the Tivoli Intelligent Orchestrator information center. If you need more information on how to address the SSH configuration issue, search the Tivoli Intelligent Orchestrator Knowledge Base. If information is found on the knowledge base, follow the resolution using the knowledge entry. If no information is found, contact Support. For more information, refer to Appendix B, “Support information,” on page 225.

7. If running the command on the managed server fails, debug the command on

the managed server.

Accessing the knowledge bases

At any time during the problem determination process, when a resolution to your problem cannot be found, check the knowledge bases for Tivoli Intelligent Orchestrator and its prerequisites to determine whether the resolution to your problem is already documented.

The following knowledge bases are available:

v For Tivoli Intelligent Orchestrator:

http://www-306.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html

v For DB2 Universal Database: http://www-306.ibm.com/software/data/db2/udb/

support/

v For the IBM Tivoli Directory Server:

http://www-306.ibm.com/software/sysmgmt/products/support/ IBMDirectoryServer.html

v For the WebSphere Application Server:

http://www-306.ibm.com/software/webservers/appserv/ was/support/

For more Support information, refer to Appendix B, “Support information,” on page 225.

Using the available command line tools

This section describes a number of command line tools that are most frequently used during the problem determination process of Tivoli Intelligent Orchestrator and its prerequisites. The following command line tools are described:

v “Tivoli Intelligent Orchestrator command line tools”

v “WebSphere Application Server command line tools” on page 32

v “IBM Tivoli Directory Server command line tools” on page 33

Tivoli Intelligent Orchestrator command line tools

The following table lists a number of command line tools that can be used during the Tivoli Intelligent Orchestrator problem determination process:

26 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Note: The command line tools can be found on TIO_HOME/tools

Table 1. Tivoli Intelligent Orchestrator command line tools

Name Location Description Syntax

applyNewEncryptKey.cmd applyNewEncryptKey.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Migrates all of the existing credentials from the TIO database tables based on a new encryption key. When switching to a new encryption key, a user must also

applyNewEncryptKey.cmd/sh [new_encryption_key] applyNewEncryptKey.sh [new_encryption_key]

change encrypted data inside the database tables to re-encrypt them using the new encryption key.

cancel-all-inprogress.cmd cancel-all-inprogress.sh

v On

Windows:

%TIO_HOME%\ tools

Cancels all of the deployment requests that are in progress.

cancel-all-inprogress.cmdcancel-allin-progress.sh

v On UNIX:

$TIO_HOME/ tools

changePassword.cmd changePassword.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Changes the default Tivoli Intelligent Orchestrator password for the following users:

wasadmin, tioldap, tioappadmin, tiodb, tiointernal. The

changePassword.cmd <user id> <your new password> <current wasadmin password> changePassword.sh <user id> <your new password> <current wasadmin password>

tiointernal

password is used by the agents to register with the agent manager.

clean-up-deploymentrequests.cmd clean-up-deploymentrequests.sh

v On

Windows:

%TIO_HOME%\ tools

Cleans up all workflows that are in the in-progress state.

clean-up-deploymentrequests.cmd clean-up-deploymentrequests.sh

v On UNIX:

$TIO_HOME/ tools

clean-up-taskshistory.cmd clean-up-taskshistory.sh

v On

Windows:

%TIO_HOME%\ tools

Deletes the task history.

clean-up-tasks-history.cmd [date] clean-up-tasks-history.sh [date]

v On UNIX:

$TIO_HOME/ tools

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 27

Table 1. Tivoli Intelligent Orchestrator command line tools (continued)

Name Location Description Syntax

dcmExport.cmd dcmExport.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Example: dcmExport.cmd c:/myDirectory/ myOutput.xml If outputFilename is not specified, the default output file name is dcmExport.xml in the

dcmExport.cmd [-d] [outputFilename] dcmExport.sh [-d] [outputFilename]

current directory). Use of option [-d] outputs protected data in clear text (default is encrypted).

dcmQueryCommand.cmd dcmQueryCommand.sh

v On

Windows:

%TIO_HOME%\ tools

Expects full path name of the input file.

dcmQueryCommandt.cmd [input_file] dcmQueryCommandt.sh [input_file]

v On UNIX:

$TIO_HOME/ tools

engineStatus.cmd engineStatus.sh

init.cmd init.sh

v On

Windows:

v On UNIX:

$TIO_HOME/ tools

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

Verifies the status of the deployment engine, policy engine, and AgentShell Server.

Initializes the database and installs the automation packages with the installDriver option.

init.cmd [dcmcfg_file] init.sh [dcmcfg_file]

$TIO_HOME/ tools

packageLogs.cmd packageLogs.sh

reinit.cmd reinit.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Packages the Tivoli Intelligent Orchestrator logs. The resulting zip file is saved by default in your current directory.

Initializes the database and installs the automation packages with the

forceInstallDriver

option.

packageLogs.cmd packageLogs.sh [dcmcfg_file]

reinit.cmd reinit.sh

28 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Table 1. Tivoli Intelligent Orchestrator command line tools (continued)

Name Location Description Syntax

retrieveDetails.cmd retrieveDetails.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Retrieves and displays a data center model object based on the specified identifier.

retrieveDetails.cmd

-i<deviceId> [-i<deviceId>]* retrieveDetails.sh

-i<deviceId> [-i<deviceId>]*

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 29

Table 1. Tivoli Intelligent Orchestrator command line tools (continued)

Name Location Description Syntax

siregister.cmd siregister.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Extracts installable files from Macrovision Solution Installation packages and adds them to the Tivoli Intelligent Orchestrator software catalog.

siregister.cmd [package_location] [repository_name] [repository_mount_point] [mount_path] [repository_host_name] [user_name] siregister.sh [package_location] [repository_name]

package_location The

URL of the Solution Installation package on the Tivoli

[repository_mount_point] [mount_path] [repository_host_name] [user_name]

Intelligent Orchestrator server. For example, file:S:/solutioninstall/ sipackage/acme.zip.

repository_name The name of a file repository in Tivoli Intelligent Orchestrator where you want to store the package. For example, si-file-repository.

repository_mount_point

The mount point to mount the file repository. For example, /si/export. mount_path The relative path to the package, starting with the specified mount point. For example, if the mount point is /si/export and the mount path is sipackage, then the whole path is /si/export/ sipackage.

repository_host_name

The host name of the file repository for the package.

user_name The SSH user name configured for accessing the file repository.

30 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Table 1. Tivoli Intelligent Orchestrator command line tools (continued)

Name Location Description Syntax

tc-drivermanager.cmd tc-driver-manager.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Installs, uninstalls, gets information, and gets the device model for the specified automation package. Available commands are:

tc-driver-manager.cmd [command] tc-driver-manager.sh [command]

<forceInstallDriver, fid> <driverName | '*'> [options]

tio.cmd tio.sh

v On

Windows:

%TIO_HOME%\ tools

Starts and stops the Tivoli Intelligent Orchestrator.

v On UNIX:

$TIO_HOME/ tools

tioStatus.cmd tioStatus.sh

updateCredential.cmd updateCredential.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Verifies the status of the deployment engine, policy engine, AgentShell Server, and WebSphere Application Server.

Updates the value of the internal-user and

internal-password

tag in the user-factory.xml file. It also updates the JMS login user in the WebSphere administrative

updateCredential.cmd internal_user_password [WAS_Admin_User] [WAS_Admin_password] [WAS_CELL_NAME] updateCredential.sh internal_user_password [WAS_Admin_User] [WAS_Admin_password] [WAS_CELL_NAME]

console.

workflowLogExport.cmd workflowLogExport.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Exports the run history of one or more specified workflows to a log file. The default location for this log file is in the %TIO_LOGS% directory.

workflowLogExport.cmd (-n [workflow name] or -r [request id]) -f [output file name] -i [input file name] workflowLogExport.sh (-n [workflow name] or -r [request id]) -f [output file name] -i [input file name]

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 31

Table 1. Tivoli Intelligent Orchestrator command line tools (continued)

Name Location Description Syntax

xmlImport.cmd xmlImport.sh

v On

Windows:

%TIO_HOME%\ tools

v On UNIX:

$TIO_HOME/ tools

Imports objects from a specified XML file into the data center model. The XML source file must be validated against the xmlimport.dtd file (located in the

xmlImport.cmd file_URL [file_URL] xmlImport.sh file_URL [file_URL]

$TIO_HOME/xml

directory), that defines the legal building blocks of the XML document.

WebSphere Application Server command line tools

The following table lists command line tools that can be used during the WebSphere Application Server problem determination process:

Table 2. WebSphere Application Server command line tools

Name Location Description Syntax

setupCmdLine.bat setupCmdLine.sh

v On

Windows:

%WAS_HOME%\ bin

Sets the environment variables needed by the WebSphere Application Server.

v On UNIX:

$WAS_HOME/ bin

collector

v On

Windows:

%WAS_HOME%\ bin

v On UNIX:

$WAS_HOME/ bin

Gathers logs, property files, configuration files, operating system and Java data and puts into a JAR file. The JAR file name is based on the host name and package of the server, in the format:

hostname-ND|BaseWASenv.jar.

versionInfo

v On

Windows:

%WAS_HOME%\ bin

Displays information about the current level of the WebSphere Application Server.

v On UNIX:

$WAS_HOME/ bin

serverStatus

v On

Windows:

%WAS_HOME%\ bin

Displays the current status of the WebSphere Application Server.

v On UNIX:

$WAS_HOME/ bin

32 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Table 2. WebSphere Application Server command line tools (continued)

Name Location Description Syntax

backupConfig restoreConfig

was_security.bat was_security.sh

v On

Windows:

%WAS_HOME%\ bin

v On UNIX:

$WAS_HOME/ bin

v On

Windows:

%TIO_HOME%\ bin

Archives the contents of the

%WAS_HOME%\config

directory and subdirectories to a specified ZIP file, or toWebSphereConfig_<yyyy- mm-dd>.zip, created in the same directory in which the command is run.

Can be used to enable the WebSphere Application Server security in case it is disabled.

v On UNIX:

$TIO_HOME/ bin

IBM Tivoli Directory Server command line tools

The following table lists command line tools that can be used during the IBM Tivoli Directory Server problem determination process:

Table 3. IBM Tivoli Directory Server command line tools

Name Location Description

ibmdirctl

v On Windows:

C:\IBM\ldap\bin

v On UNIX:

/usr/ldap/bin

Verifies whether the IBM Tivoli Directory Server database was successfully started. Can be used to start and stop the IBM Tivoli Directory Server. To be able to use the ibmdirctl command, the IBM Directory Administrator service must be running.

ibmdiradm

ldapsearch

v On UNIX:

/usr/ldap/bin

v On Windows:

C:\IBM\ldap\bin

v On UNIX:

Verifies whether the IBM Directory Administrator service is running.

Verifies whether IBM Tivoli Directory Server is running, configured properly, and responding to queries.

/usr/ldap/bin

update_xmi.cmd update_xmi.sh

v On Windows:

%TIO_HOME%\tools

Updates the base domain, basedn, for the IBM Tivoli Directory Server.

v On UNIX:

$TIO_HOME/tools

updateLDAPHostname.cmd updateLDAPHostname.sh

v On Windows:

%TIO_HOME%\tools

Updates the IBM Tivoli Directory Server host name.

v On UNIX:

$TIO_HOME/tools

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 33

34 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 3. Log file types

Tivoli Intelligent Orchestrator produces message logs, trace logs, and Common Base Event logs that can be configured dynamically.

Message logs

Message logs record Tivoli Intelligent Orchestrator system events as text messages so they can be reviewed later by customers or by IBM Tivoli Software Support. Typically, messages are used to provide information about how the system or application is performing, and to alert the system administrator to exceptional conditions when they occur. If an error occurs, you can check the message logs for information about the error, the cause of the error, and possible resolutions for the error.

Message logs have the file name msg.log, and are stored in the subfolder for each software component. Messages are localized based on the locale configured on the Tivoli Intelligent Orchestrator server. There are multiple levels of message logs, and they can be filtered by message log level and software component.

For a complete listing of the Tivoli Intelligent Orchestrator messages, see Chapter 10, “Tivoli Intelligent Orchestrator messages,” on page 95.

Message ID format

The message ID consists of 10 alphanumeric characters that uniquely identify the message. Each message ID includes:

v A 3-character product identifier

v A 3-character component or subsystem identifier

v A 3-digit serial or message number

v A 1-character type code indicating the severity of the message

sequence of the alphanumeric characters in the message ID is COPYYY###Z

The where:

v COP is a three-character release-independent product identifier. The system uses

this product identifier to identify the relevant subdirectory that contains serviceability information when using the Tivoli Common Directory.

v YYY is the subsystem code, including the following subsystem codes:

Code Subsystem

COM Common

DEX Deployment engine

JEE j2ee — the Tivoli Intelligent Orchestrator code that runs on the WebSphere

Application Server.

PEZ Policy engine

TCA Tivoli common agent

TDM TC driver manager

JDS Job Distribution Service

UTL Utilities

v ### is 3-digit unique serial or message number.

v Z is the severity code indicator, including the following indicators:

Severity

Description

code

I Informational message. The message provides information or feedback

about normal events that have occurred or are occurring, but do not require you to take action. The message might also request that you provide information in instances where the outcome of the information you provide will not be negative.

W Warning message. The message indicates that potentially undesirable

conditions have occurred or could occur, but the program can continue. Warning messages might ask you to make a decision before processing continues.

E Error message. The message indicates an error that requires an intervention

or correction before the program can continue.

Message elements

Each message includes a message ID, message text, and explanation text. Some messages also include action text when the user can take an action.

Example:

COPDEX101E A security exception occurred while

the system parsed the profile XML file.

Explanation: The ITM Obtain OS Profiles workflow

generates an XML file that contains the list of profiles installed in Tivoli Management Agent. This profile XML file is copied to the IBM Tivoli Provisioning Manager and then it is parsed by the Profile XML Parser. The XML parser cannot read the profile XML

file because a security error occurred. This is a file permissions issue. Either the parent directory of the profile XML file is missing appropriate file permissions, or the profile XML file is missing the appropriate file permissions.

Operator response: Verify that the login user is assigned read and write access to the parent directory of the ProfileXMLParser.

Message ID

String of 10 alphanumeric characters that uniquely identifies the message. In the preceding example, the message ID is COPDEX101E.

Message text

Explains the reason for the message, what the message means, and possible causes of the message with recommended steps you can take (for those messages that require some action on your part). In the preceding example, the message text is: A security exception occurred while the system parsed the profile XML file.

Explanation

Contains additional information about the cause of the message, and describes the action that the system took or will take. In the example, the explanation follows the label Explanation.

Action

Describes what you must do to proceed, to recover from the error, or to prevent a problem from occurring. Actions can include:

Action title Description

System Action Describes the reaction of the system to the condition that caused it

to display this message.

Operator Response Describes what response you might be able to take.

36 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Action title Description

Administrator Response

Programmer Response

Describes what response a system administrator might be able to take.

Describes what response a system programmer might be able to take.

In the example, the action follows the label Operator Response.

Trace logs

Tivoli Intelligent Orchestrator provides configurable tracing capabilities that can help you determine the cause of a problem. Trace logs and first-failure data capture (FFDC) are built into the software to assist the IBM Customer Support for Tivoli software personnel in gathering information to determine why a problem is occurring, and might be requested as part of diagnosing a reported problem.

v Trace logs capture operating environment details, including the starting and

stopping of processes, data transfer between software components, activity that occurs while the software is running, and errors that occur when the code fails to operate as intended.

v First-failure data capture captures and stores the tracing information preceding

an error message.

Trace logs are used to determine the causes of Tivoli Intelligent Orchestrator problems and to debug errors. Error messages can include information about the cause of the error and possible resolutions for the error. Trace logs are intended to be used by Tivoli Customer Support service engineers or developers. If you encounter an issue, Support representatives might ask you to obtain information from trace logs so that they can review the details.

Trace logs have the file name trace.log, and are stored in the subfolder for each software component. Trace logs are in English only, localization support (national language support) is not provided for trace entries. By default, tracing is enabled at the minimal level that is required for FFDC, and can be changed as required.

Common Base Event logs

Trace logs are automatically translated to Common Base Event, which is a standardized format for logging and tracing events in a multi-application environment. By using the Common Base Event format, Tivoli Intelligent Orchestrator facilitates the integration of problem determination data, and makes it possible to rapidly identify an event, collect and analyze problem determination data, establish the event priority, plan appropriate responses, and make the necessary adjustments to resolve it. An XML-based standard, the Common Base Event guarantees consistency of interpretation for all of the messages generated between applications, and offers consistent and complete information about the following:

v The system component that reports a particular error

v The components affected by this error, which might be the same as the

component reporting the error

v The error itself

Common

Base Event logs have the file name cbe.log, and are stored in the

subfolder for each software component.

Chapter 3. Log file types 37

38 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 4. Locating and configuring log files

This release of Tivoli Intelligent Orchestrator produces logs that follow the Tivoli Common Directory standard.

Log locations

To provide a consistent mechanism for locating serviceability information, Tivoli products and applications have implemented the Tivoli Common Directory. The Tivoli Common Directory represents a central location on systems running Tivoli software for storing serviceability-related files.

The location of the message logs, trace logs, and Common Base Event logs for Tivoli Intelligent Orchestrator follow the Tivoli Common Directory standard. The log locations are shown in the following table.

Directory name Description Path

j2ee WebSphere JVM %TIO_LOGS%\j2ee

deploymentengine Deployment Engine JVM %TIO_LOGS%\deploymentengine

policyengine Policy Engine JVM %TIO_LOGS%\policyengine

tcdrivermanager Workflows %TIO_LOGS%\tcdrivermanager

install Installation %TIO_LOGS%\install

uninstall Uninstall process %TIO_LOGS%\uninstall

agentshellserver Agent Shell Server %TIO_LOGS%\agentshellserver

wswb WebSphere Studio

%TIO_LOGS%\wswb

WorkBench (WSWB)

where %TIO_LOGS% takes the following default values:

v Windows: C:\Program Files\ibm\tivoli\common\COP\logs

v UNIX® or Linux: /var/ibm/tivoli/common/COP/logs

The

log files that are generated by all the utility scripts, such as %TIO_HOME%\tools\cancel-

all-in-progress.cmd,

%TIO_HOME%\tools\changepassword.cmd, and so on, are located in the %TIO_LOGS% directory on Windows systems ($TIO_LOGS on UNIX or Linux).

Log directory files

Each Tivoli Intelligent Orchestrator JVM directory includes the following log files:

console.log

Stores all event logs including messages, traces, and debugging information.

msg.log

Stores the globalized event messages so the user can understand a problem and take action to try and resolve the problem.

trace.log

Stores errors that are reviewed by IBM Tivoli Support.

cbe.log

Stores all error messages in Common Base Event format.

Logging levels

This section details the behavior of Tivoli Intelligent Orchestrator logs based on how the logging levels are defined.

The logging level hierarchy for Tivoli Intelligent Orchestrator is shown in the following table:

Logging level Description

msg_error#com.thinkdynamics.kanaha. util.logging.MessageLevel

msg_warn#com.thinkdynamics.kanaha. util.logging.MessageLevel

msg_info#com.thinkdynamics.kanaha. util.logging.MessageLevel

error Default log4j level — trace logs

warn Default log4j level — trace logs

info Default log4j level — trace logs

debug Default log4j level — trace logs

The extended logging levels are the highest levels in the logging hierarchy. These levels are set exclusively for the messages, which are globalized and distinct from the trace logs. Trace logs are available only in English.

Configuring logs with log4j

Log data in Tivoli Intelligent Orchestrator is managed by log4j, an established open source logging tool. This section details the default log4j settings, the customized Tivoli Intelligent Orchestrator settings, and how you can modify settings dynamically. For complete log4j documentation, go to http://logging.apache.org/ log4j/docs/documentation.html.

Extended for message logs

The data from the console.log, msg.log, trace.log, and cbe.log files are recorded based on the default logging levels and configuration parameters set in the log4j.prop file or in the log4j-util.prop. Use the log4j-util.prop file to configure the logging for all scripts located in the %TIO_HOME%\tools directory.

log4j.prop

The log4j.prop file, shown below, defines the default configuration for message, trace, and Common Base Event logs.

The location of this file is as follows:

Windows

%TIO_HOME%\config

UNIX or Linux

$TIO_HOME/config

# output directory. Can be overwritten with -Dkanaha.logs= # kanaha.logs=logs

# message formats # normal used to write to console.log # error used to format error messages (prints location of a problem)

40 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

# module is meant for messages written module specific files # output.normal=%d{ISO8601} %-5p [%t] (%13F:%L) %c{2}: %m%n output.error=%d{ISO8601} %-5p [%t] (%13F:%L): %m%n output.module=%d{ISO8601} %-5p [%t] (%13F:%L): %m%n

# # configure root category # note that this configuration is inherited by all other categories (see # example below if you want to suppress this behaviour) # log4j.rootCategory=DEBUG, consolefile, errorfile, messagefile

# everything goes to console.log # rolling by log size. For other rolling options, see # http://logging.apache.org/log4j/docs/api/index.html # log4j.appender.consolefile=org.apache.log4j.RollingFileAppender log4j.appender.consolefile.MaxFileSize=100MB log4j.appender.consolefile.MaxBackupIndex=10 log4j.appender.consolefile.File=${kanaha.logs}/console.log log4j.appender.consolefile.layout=org.apache.log4j.PatternLayout log4j.appender.consolefile.layout.ConversionPattern=${output.normal} log4j.appender.consolefile.threshold=info log4j.appender.consolefile.append=true

# errors to trace log file, for FFDC # rolling by log size # log4j.appender.errorfile=org.apache.log4j.RollingFileAppender log4j.appender.errorfile.MaxFileSize=10MB log4j.appender.errorfile.MaxBackupIndex=10 log4j.appender.errorfile.File=${kanaha.logs}/trace.log log4j.appender.errorfile.layout=org.apache.log4j.PatternLayout log4j.appender.errorfile.layout.ConversionPattern=${output.error} log4j.appender.errorfile.threshold=error log4j.appender.errorfile.append=true

# globalized message log to msg.log (user log) # rolling by log size # log4j.appender.messagefile=org.apache.log4j.RollingFileAppender log4j.appender.messagefile.MaxFileSize=10MB log4j.appender.messagefile.MaxBackupIndex=10 log4j.appender.messagefile.File=${kanaha.logs}/msg.log log4j.appender.messagefile.layout=org.apache.log4j.PatternLayout log4j.appender.messagefile.layout.ConversionPattern=${output.normal} log4j.appender.messagefile.threshold=MSG_INFO#com.thinkdynamics.kanaha. util.logging.MessageLevel log4j.appender.messagefile.append=true

# suppress annoying messages from datacentermodel log4j.category.com.thinkdynamics.kanaha.datacentermodel=INFO

# # examples #

# suppress writing dcm messages to console.log # #log4j.category.com.thinkdynamics.kanaha.datacentermodel=INFO, datacentermodel, errorfile #log4j.additivity.com.thinkdynamics.kanaha.datacentermodel=false #log4j.appender.datacentermodel=org.apache.log4j.FileAppender #log4j.appender.datacentermodel.File=${kanaha.logs}/datacentermodel.log

Chapter 4. Locating and configuring log files 41

#log4j.appender.datacentermodel.layout=org.apache.log4j.PatternLayout #log4j.appender.datacentermodel.layout.ConversionPattern=${output.module} #log4j.appender.datacentermodel.append=false

Log level defaults

The initial log4j log level configuration for each log file is set to info. As defined in the log4j.prop file above, each log file is set to a unique log level for Tivoli Intelligent Orchestrator using the

log4j.appender.filename.threshold= parameter.

The following is a list of the default logging levels for each of the log file types:

console.log

info

If you want more log information for troubleshooting purposes, set the log level to debug.

trace.log

error

msg.log

MSG_INFO#com.thinkdynamics.kanaha.util.logging.MessageLevel

cbe.log

error

RollingFileAppender defaults

The RollingFileAppender defines the autoarchiving behavior of the log4j tool. As defined in the log4j.prop file above, the default settings for each Tivoli Intelligent Orchestrator log file are:

console.log

v Maximum number of archived log files: 10

v Maximum log file size: 100MB

trace.log

msg.log

v Maximum number of archived log files: 10

v Maximum log file size: 10MB

v Maximum number of archived log files: 10

v Maximum log file size: 10MB

cbe.log

v Maximum number of archived log files: 10

v Maximum log file size: 10MB

Configuring log4j dynamically

To modify the log4j.prop or log4j-util.prop file:

1. Open the properties file in a text editor.

2. Modify settings as required.

3. Save the file.

42 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

The Tivoli Intelligent Orchestrator application server automatically reloads the log4j configurations 60 seconds after you save the properties file. Yo u do not need to restart the Tivoli Intelligent Orchestrator server. The updated log4j configuration will implement after 60 seconds.

Example:

To enable the TC driver manager to append to its log file located in the %TIO_LOGS%\tcdrivermanager directory, complete these steps:

1. Open log4j-util.prop in a text editor.

2. Change the default log4j.appender.file.append=false to

log4j.appender.file.append=true.

3. Save the file.

Preparing log files for review by IBM Tivoli Support

Tivoli Intelligent Orchestrator includes packageLog, a log management tool that simplifies the problem determination process. For more information, see “packageLog ZIP tool” on page 92.

Chapter 4. Locating and configuring log files 43

44 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 5. Tivoli Intelligent Orchestrator log files

This chapter describes the directory locations for the following log files:

v “Log files for product setup”

v “Log files for the uninstall process”

v “Log files for starting and stopping the server” on page 46

v “Log files for the Web interface” on page 47

v “Deployment engine log files” on page 47

v “Workflow logs” on page 48

v “Logs for data collection and decision making” on page 49

v “Automation package logs” on page 49

Log files for product setup

These log files are for installation and data center model initialization.

Table 4. Product setup logs

File name Location Description

reinit.log

Windows

or Linux

UNIX

%TIO_LOGS%

Default value for %TIO_LOGS%:

C:\Program Files\ibm\tivoli\ common\COP\logs

$TIO_LOGS

Default value for $TIO_LOGS:

/var/ibm/tivoli/common/COP/logs

This log file is created when you initialize the database and load the default automation packages, workflows, and data center model. It is also used whenever you run the reinit command to reload your automation packages and data center model. It contains SQL statements used to initialize the database, and output data resulting from the loading of automation packages and data center model.

tcinstall.log

Windows

or Linux

UNIX

%TIO_LOGS%\install

This is the InstallShield log for Tivoli Intelligent Orchestrator installation.

$TIO_LOGS/install

Log files for the uninstall process

When you uninstall Tivoli Intelligent Orchestrator, the log files for the uninstall process follow the Tivoli Common Directory standard, and are located in the following directory:

Windows

%TIO_LOGS%\uninstall

UNIX or Linux

$TIO_LOGS/uninstall

Log files for starting and stopping the server

Log files for starting and stopping the Tivoli Intelligent Orchestrator server are stored in the %TIO_LOGS% directory. Log files specific to WebSphere Application Server are located in the %WAS_HOME%\logs\server1 directory, where %WAS_HOME% is the default WebSphere Application Server home directory.

Table 5. Server startup and shutdown logs

File name Location Description

tio_start.log

Windows

or Linux

UNIX

%TIO_LOGS%

$TIO_LOGS

This is the log file of the most recent startup of the Tivoli Intelligent Orchestrator server. This log is overwritten each time the server is started.

If startup happens with no errors, this file is very short. If there are errors, you might need to consult the WebSphere Application Server logs for additional information.

tio_stop.log

startServer.log

Windows

or Linux

UNIX

Windows

or Linux

UNIX

%TIO_LOGS%

$TIO_LOGS

%WAS_HOME%\logs\server1

This is the log file of the most recent shutdown of the Tivoli Intelligent Orchestrator server. This file is overwritten each time the server is stopped.

This is the log file for the startup of the WebSphere Application Server.

$WAS_HOME/logs/server1

stopServer.log

Windows

or Linux

UNIX

%WAS_HOME%\logs\server1

$WAS_HOME/logs/server1

This is the log file for the shutdown of the WebSphere Application Server. WebSphere is the last service to stop when you shut down Tivoli Intelligent Orchestrator, so this log file is useful for verifying whether the shutdown completed.

SystemOut.log

Windows

or Linux

UNIX

%WAS_HOME%\logs\server1

$WAS_HOME/logs/server1

This is the log file for WebSphere Application Server output. You might want to refer to this file when WebSphere Application Server does not start.

46 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Log files for the Web interface

Log files for the We b interface and data center model changes, and other operations associated with the We b interface are stored in the %TIO_LOGS%\j2ee directory.

Table 6. Web interface logs

File name Location Description

console.log

Windows

or Linux

UNIX

%TIO_LOGS%\j2ee

This log file stores all event logs for the Web interface including messages, traces, and debugging information.

$TIO_LOGS/j2ee

msg.log

Windows

or Linux

UNIX

%TIO_LOGS%\j2ee

This log file stores the globalized event messages for the We b interface and data center model changes.

$TIO_LOGS/j2ee

trace.log

Windows

or Linux

UNIX

%TIO_LOGS%\j2ee

This log file stores error messages that can be reviewed by Tivoli Software Support.

$TIO_LOGS/j2ee

cbe.log

Windows

or Linux

UNIX

%TIO_LOGS%\j2ee

This log file stores all error messages for the Web interface in Common Base Event format.

$TIO_LOGS/j2ee

Deployment engine log files

Log files for workflow and deployment requests are stored in the %TIO_LOGS%\deploymentengine directory.

Table 7. Deployment engine logs

File name Location Description

console.log

msg.log

trace.log

Windows

or Linux

UNIX

Windows

or Linux

UNIX

Windows

or Linux

UNIX

%TIO_LOGS%\deploymentengine

$TIO_LOGS/deploymentengine

%TIO_LOGS%\deploymentengine

$TIO_LOGS/deploymentengine

%TIO_LOGS%\deploymentengine

$TIO_LOGS/deploymentengine

This log file stores all event logs for workflow and deployment requests including messages, traces, and debugging information.

This log file stores the globalized event messages for the deployment engine component.

This log file stores error messages that can be reviewed by Tivoli Software Support.

Chapter 5. Tivoli Intelligent Orchestrator log files 47

Workflow logs

Table 7. Deployment engine logs (continued)

File name Location Description

cbe.log

Windows

or Linux

UNIX

%TIO_LOGS%\deploymentengine

$TIO_LOGS/deploymentengine

This log file stores all event logs for the deployment engine component in Common Base Event standard format.

deploymentengine_start.log

Windows

or Linux

UNIX

%TIO_LOGS%\deploymentengine

$TIO_LOGS/deploymentengine

This is the log file of the most recent startup of the deployment engine. This log is overwritten each time the deployment engine is started.

If you want to determine why a particular workflow has failed, you can use the Web interface to display the run history for that workflow. You can also export the log files of your workflow history using the workflowLogExport command, as described below. The command script is located in the %TIO_HOME%\tools directory, where %TIO_HOME% is the Tivoli Intelligent Orchestrator home directory.

v On a Windows platform, type the following command:

workflowLogExport.cmd (-n <workflow_name>* | -r <request_id>) [-f <export_filename>] [-i <input_filename>]

v On a UNIX or Linux platform, type the following command:

workflowLogExport.sh (-n <workflow_name>* | -r <request_id>) [-f <export_filename>] [-i <input_filename>]

where: – <workflow_name> is the name of the workflow whose run history you want to

export to a log file;

– <request_id> indicates the deployment request identifier; – [-f <export_filename>] specifies the fully qualified path and file name

where the workflow log will be exported. By default, the file format for the exported log file is XML, and the default file name is workflowLogExport.xml. If you do not specify a different file format, the workflow log file will be exported by default to XML. Also, if no file location is specified, the log file will be exported by default to the %TIO_LOGS% directory.

– [-i <input_filename>] specifies the name of the file that lists the names of all

workflows whose run history you want to export to the same log file.

48 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Logs for data collection and decision making

Log files for the policy engine component (data acquisition engine, application controllers, global resource manager) are stored in the %TIO_LOGS%\policyengine directory.

Table 8. Policy engine logs

File name Location Description

console.log

Windows

or Linux

UNIX

%TIO_LOGS%\policyengine

$TIO_LOGS/policyengine

This log file stores all event logs for the policy engine component including messages, traces, and debugging information.

msg.log

Windows

or Linux

UNIX

%TIO_LOGS%\policyengine

This log file stores the globalized event messages for the policy engine component.

$TIO_LOGS/policyengine

trace.log

Windows

or Linux

UNIX

%TIO_LOGS%\policyengine

This log file stores error messages that can be reviewed by Tivoli Software Support.

$TIO_LOGS/policyengine

cbe.log

Windows

or Linux

UNIX

%TIO_LOGS%\policyengine

$TIO_LOGS/policyengine

This log file stores all event logs for the policy engine component in Common Base Event standard format.

policyengine_start.log

Windows

or Linux

UNIX

%TIO_LOGS%\policyengine

$TIO_LOGS/policyengine

This is the log file of the most recent startup of the policy engine. This log is overwritten each time the policy engine is started.

Automation package logs

Log files for orchestration packages are stored in the %TIO_LOGS%\tcdrivermanager directory.

Table 9. Automation package logs

File name Location Description

console.log

msg.log

Windows

or Linux

UNIX

Windows

or Linux

UNIX

%TIO_LOGS%\tcdrivermanager

$TIO_LOGS/tcdrivermanager

%TIO_LOGS%\tcdrivermanager

$TIO_LOGS/tcdrivermanager

Chapter 5. Tivoli Intelligent Orchestrator log files 49

This log file stores all event logs for orchestration packages including messages, traces, and debugging information.

This log file stores the globalized event messages for orchestration packages.

Table 9. Automation package logs (continued)

File name Location Description

trace.log

Windows

or Linux

UNIX

%TIO_LOGS%\tcdrivermanager

This log file stores error messages that can be reviewed by Tivoli Software Support.

$TIO_LOGS/tcdrivermanager

cbe.log

Windows

or Linux

UNIX

%TIO_LOGS%\tcdrivermanager

This log file stores all event logs for orchestration packages in Common Base Event standard format.

$TIO_LOGS/tcdrivermanager

50 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 6. Middleware prerequisite log files

This chapter describes the directory locations for the following log files:

v “Installation log files for prerequisites”

v “Runtime logs for prerequisites” on page 52

Installation log files for prerequisites

When you install the prerequisite software for Tivoli Intelligent Orchestrator using the graphical installer, the log files for the installation process are created in the Tivoli_common_directory\GPX directory, where:

v The default value for the Tivoli_common_directory is:

Windows

C:\Program Files\ibm\tivoli\common

UNIX or Linux

/var/ibm/tivoli/common

GPX is the three-digit product code for the prerequisite installer.

Installation logs for individual prerequisites

The installation logs for each software prerequisite are also located in the Tivoli_common_directory\GPX directory.

WebSphere Application Server

The WebSphere Application Server logs are:

v base.log

v websphere.stdout

v websphere.stderr

v websphereFixPack.stdout

DB2 Universal Database

The log files for IBM DB2 Universal Database Enterprise Edition 8.2 are:

v db2server.log

v db2server.trace

v db2Server.stderr

v db2Server.stdout

v db2ServerHotfix.stderr

v db2ServerHotfix.stdout

v db2client.log

v db2client.trace

v db2Client.stderr

v db2Client.stdout

v db2wi.log

Directory Server

The log files for the IBM Tivoli Directory Server are:

v ldap.stderr

v ldap.stdout

v ldapinst.log

Runtime logs for prerequisites

For additional information on errors that do not pertain to Tivoli Intelligent Orchestrator, you might want to consult the error and trace logs for the middleware prerequisites. The names and the locations of these log files are provided below.

WebSphere Application Server

The runtime WebSphere Application Server logs are located in the %WAS_HOME%\logs and the %WAS_HOME%\logs\server1 directories, where %WAS_HOME% is the WebSphere Application Server home directory. The runtime transaction logs for WebSphere Application Server are stored in the %WAS_HOME%\tranlog directory.

The default value for %WAS_HOME% is:

Windows

C:\Program Files\WebSphere\AppServer

/usr/WebSphere/AppServer

AIX

and Solaris

Linux

/opt/WebSphere/AppServer

the following log files for errors:

Check

v SystemOut.log

v startServer.log

v stopServer.log

v SystemErr.log

SystemOut.log

This is the log file for WebSphere output. It contains messages that are generated when the applications running inside the WebSphere Application Server are being started or stopped. You might want to refer to this file when WebSphere does not start. The log file is located in the %WAS_HOME%\logs\server1 directory on Windows systems ($WAS_HOME/logs/server1 on UNIX or Linux).

After the Tivoli Intelligent Orchestrator is successfully started, use the tail

-f SystemOut.log command to monitor this log file for any problems that might occur.

startServer.log

This is the log file for the startup of the WebSphere Application Server, located in the %WAS_HOME%\logs\server1 directory on Windows systems ($WAS_HOME/logs/server1 on UNIX or Linux).

Look for open for e-business for a successful startup of the WebSphere Application Server.

stopServer.log

This is the log file for the shutdown of the WebSphere Application Server.

52 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

WebSphere Application Server is the last service to stop when you shut down Tivoli Intelligent Orchestrator, so this log file is useful for verifying whether the shutdown completed. It is located in the %WAS_HOME%\logs\ server1 directory on Windows systems ($WAS_HOME/logs/server1 on UNIX or Linux).

Look for Server <server-name> stop completed for a successful shutdown of the WebSphere Application Server.

SystemErr.log

This is the log file that contains Java exceptions and stack traces caused by the enterprise applications. It is located in the %WAS_HOME%\logs\server1 directory on Windows systems ($WAS_HOME/logs/server1 on UNIX or Linux).

DB2 Universal Database

The DB2 Universal Database error logs are stored in the %DB2_HOME%\DB2 directory on Windows® systems ($DB2_HOME/db2dump on UNIX or Linux), where %DB2_HOME% is the default DB2 home directory.

Check the following log file for errors:

db2diag.log

When an error occurs, the db2diag.log is updated with information about the error. This is the primary log to use when debugging DB2 Universal Database problems.

db2alert.log

dump

If an error is determined to be an alert, then an entry is made in the db2alert.log file and to the operating system or native logging facility.

files

For some error conditions, additional information is logged in external binary dump files named after the failing process ID. These files are intended for DB2 Universal Database Customer Support.

files

trap

The database manager generates a trap file if it cannot continue processing because of a trap, segmentation violation, or exception. Trap files contain a function flow of the last steps that were executed before a problem occurred.

useful DB2 Universal Databasecommands include:

Other

db2trc This command gives you the ability to control tracing.

db2support

This command collects environment information and log files and places them into a compressed archive file.

Directory Server

Trace logs for the LDAP server are stored in the <LDAP_Install_Directory>\ appsrv\logs

directory for the LDAP server.

Check the following log files for errors:

v ibmslapd.log v audit.log

directory, where <LDAP_Install_Directory> is the installation

Chapter 6. Middleware prerequisite log files 53

v db2diag.log v ibmds_config.log v ibmds_config_err.log

Tivoli Agent Manager

Trace logs for the agent manager are stored in the Agent_Manager_install_dir\

directory, where Agent_Manager_install_dir is the installation directory for

logs

the agent manager. The log files for the Tivoli Intelligent Orchestrator installed on a managed server are stored in the installation directory on the target server.

For more information on agent manager logs, refer to Chapter 8, “Troubleshooting the agent manager,” on page 79.

54 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator

This chapter describes how to recover from the following types of Tivoli Intelligent Orchestrator problems, and describes a number of known limitations for the Tivoli Intelligent Orchestrator reports:

v “Problems with installation”

v “Problems with logging in” on page 62

v “Problems with workflows” on page 64

v “Other common problems” on page 68

v “Known report limitations” on page 77

Problems with installation

This section includes troubleshooting scenarios for the following problems:

v “In Linux, you cannot create the user tioadmin”

v “The system cannot connect to the IBM Tivoli Directory Server during the Tivoli

Intelligent Orchestrator installation” on page 56

v “The installation of the DB2 Universal Database Client on Windows 2003 system

fails” on page 56

v “Installing the common agent and the agent manager is not supported” on page

v “The system cannot connect to the database server during the Tivoli Intelligent

Orchestrator installation” on page 57

v “The Microsoft Active Directory installation fails” on page 57

v “The Microsoft Active Directory installation fails with invalid certificate value”

on page 58

v “The silent installation program for Tivoli Intelligent Orchestrator exits before

the installation is completed” on page 58

v “Environment variables not set for user tioadmin” on page 58

v “The installation of IBM Tivoli NetView on Windows fails if the password for

creating a user account does not meet the system requirements” on page 59

v “The agent manager installation fails during the Tivoli Intelligent Orchestrator

installation” on page 59

v “An error occurs when running the rpm -qa command after the DB2 Universal

Database, IBM Tivoli Directory Server, and WebSphere Application Server prerequisites are installed on Linux” on page 62

In Linux, you cannot create the user tioadmin

The installation appears to proceed successfully to completion, but when you check to find the user tioadmin, it has not been created.

Cause

This is a known InstallShield Multi-Platform issue. Because the user tioadmin is not created, the server environment is unstable. Various elements of the installation expect tioadmin and require it to exist. For example, because WebSphere Application Server is configured to start with tioadmin, WebSphere Application Server cannot start.

In Linux, the tioadmin user creation fails if the tioadmin group already exists before you install. No exception is thrown, so the installer cannot detect that the user creation operation has failed, and the system cannot return to a previous stage of the installation.

Solution

Reinstall Tivoli Intelligent Orchestrator. However, before you start the installation, ensure that the tioadmin user and tioadmin group either both exist or both do not exist.

The system cannot connect to the IBM Tivoli Directory Server during the Tivoli Intelligent Orchestrator installation

Cause

During the Tivoli Intelligent Orchestrator installation, the system might indicate that it cannot connect to the IBM Tivoli Directory Server. This error occurred because the directory server was not started before running the installer. The directory server must be started before you install Tivoli Intelligent Orchestrator, so the installer can connect to it.

Solution

1. Ensure that the directory server is installed:

a. If the installation destination directory was created, check the installation

log file for the directory server:

v Windows: C:\IBM\LDAP\ldapinst.log v UNIX or Linux: /usr/ldap/ldapinst.log

After the installation of the directory server is complete, an LDAP database must be created within DB2 Universal Database. For more information, refer to the Installation Guide for Tivoli Intelligent Orchestrator 5.1.

The ldapcfg.stat file shows the syntax that was used at the time of the database creation:

C:\IBM\ldap\bin\ldapcfg -n -a db2inst1 -w password -d LDAP -l C: -c -f C:\IBM\ldap\tmp\ldapcfg.dat

The ldapcfg.stat file is located in the following directory:

v On Windows: C:\IBM\ldap\tmp\ldapcfg.stat v On UNIX or Linux: /usr/ldap/tmp/ldapcfg.stat

2. Verify the status of the directory server, using the ibmdirctl tool, located in the C:\IBM\ldap\bin directory on Windows or in the /usr/ldap/bin directory on

UNIX or Linux. Type the following command to check the directory server status:

ibmdirctl -D cn=root -w <password> status

3. If the directory server is not started, start it by using the following command:

start: ibmdirctl -D cn=root -w <password> start

The installation of the DB2 Universal Database Client on Windows 2003 system fails

Cause

The CD used for installing the DB2 Universal Database Client, DB2 Universal Database Administration Client, Version 8.1 on 64–bit systems, is not supported on Windows 2003 operating systems.

56 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Solution

Use the DB2 Universal Database Administration Client, Version 8.1 for Windows operating systems on 32–bit systems CD instead.

Installing the common agent and the agent manager is not supported

Cause

Installing the common agent on the Tivoli Intelligent Orchestrator server, where the agent manager is also installed, is not supported.

Solution

Manually uninstall the common agent.

The system cannot connect to the database server during the Tivoli Intelligent Orchestrator installation

Cause

During the installation, the system might indicate that it cannot connect to the database. This error occurred because the database was not started before running the installer. The database server must be started before you install Tivoli Intelligent Orchestrator, so that the installer can connect to it.

Solution

Ensure that the database server is installed. Verify the status of the database server. If it is not started, start it.

Use the following commands to start the DB2 Universal Database server:

v On Windows: DB2 - <instance_name> v On UNIX: $db2start

If the database server was successfully started, you should see the following output:

db2start 12-21-2004 14:44:01 0 0 SQL1063N DB2START processing was successful. SQL1063N DB2START processing was successful

You should also verify whether the required port is available:

netstat -an |grep 50000

The Microsoft Active Directory installation fails

Cause

The Microsoft® Active Directory SSL certificate is missing. If you run the Tivoli Intelligent Orchestrator installer without the SSL certificate, the Microsoft Active Directory configuration will fail.

Solution

This is a manual configuration step that you must complete before you install Tivoli Intelligent Orchestrator.

1. Generate the SSL certificate on the Microsoft Active Directory server.

2. Install the SSL certificate on the client server.

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 57

3. Import the schema.ldif and ldap.ldif files into the Microsoft Active Directory server. Instructions for this step are found in Tivoli Intelligent Orchestrator Installation Guide Version 5.1.

4. Run the reinit command.

The Microsoft Active Directory installation fails with invalid certificate value

Cause

The Microsoft Active Directory certificate is missing but the user enters a value for the certificate location during the install. If the file does not exist on the Tivoli Intelligent Orchestrator server, then the Tivoli Intelligent Orchestrator exits with error 1005. There is no other error information available with this error code.

The silent installation program for Tivoli Intelligent Orchestrator exits before the installation is completed

Cause

WebSphere Application Server was not started before the silent installation started.

Solution

Before you run the silent installation:

1. Ensure that WebSphere Application Server is started.

2. Ensure that WebSphere Application Server security is not running.

Environment variables not set for user tioadmin

You are using a login window manager such as CDE, and when you start Tivoli Intelligent Orchestrator you get a message that informs you that WAS_HOME is not set.

Cause

The login window manager might have bypassed the required user profile file.

User tioadmin uses the bash shell as the login shell, which is supported for a line-mode login (for example, via telnet). If you use a login window manager such as CDE, it might bypass the .profile file for user tioadmin. When the profile file is bypassed, the system cannot create a complete login environment.

Solution

1. Create the .bashrc file in the tioadmin home directory, and insert the following single line: . $HOME/.profile

2. Save the file.

3. Edit the .dtprofile in the tioadmin home directory and remove the comment

from the line: DTSOURCEPROFILE=true. This file is created automatically when user tioadmin logs in to CDE for the first time.

4. Login as user tioadmin again to the Common Desktop Environment.

58 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

The installation of IBM Tivoli NetView on Windows fails if the password for creating a user account does not meet the system requirements

Cause

When installing IBM Tivoli NetView on Windows, the NetView installer creates a user account on your system for the NetView service. If the password that you are prompted to enter for creating this user account does not conform with the password policy of your system, the NetView installation fails.

Solution

Ensure that the password for creating the NetView user account conforms with the password policy of your system.

The agent manager installation fails during the Tivoli Intelligent Orchestrator installation

The installation of the agent manager fails during the Tivoli Intelligent Orchestrator installation with the following warning:

WARNING: Failed to install agent manager. View log C:\Program Files\IBM\tivoli\thinkcontrol\AM\logs\am_install.log for more details.

Cause

You might want to consult the agent manager logs to identify the cause of this problem. The agent manager log files are located in the C:\Program Files\IBM\tivoli\thinkcontrol\AM\logs directory. Possible causes might include:

v The port required for the agent manager installation might be busy.

v The agent manager has already been installed on your system.

Solution 1

If no agent manager installation has been performed on the server prior to the Tivoli Intelligent Orchestrator installation, follow these steps:

1. Consult the agent manager log files and make all the necessary changes

following the instructions in the logs.

2. Retry the Tivoli Intelligent Orchestrator installation. The Tivoli Intelligent

Orchestrator installer will detect that Tivoli Intelligent Orchestrator is already installed and will install only agent manager.

more details on the agent manager reinstallation, refer to the Tivoli Intelligent

For Orchestrator Installation Guide.

Solution 2

If the agent manager was previously installed on the Tivoli Intelligent Orchestrator installation, you should uninstall the agent manager first, and then retry the Tivoli Intelligent Orchestrator installation.

Uninstalling the agent manager: Uninstalling the agent manager uninstalls the application from your operating system and removes the agent manager servlets from WebSphere Application Server. The uninstallation wizard does not remove the registry. Removing the registry is an optional step that you can perform after running the wizard. The wizard also does not uninstall WebSphere Application Server or DB2 Universal Database Enterprise Server Edition, even if those prerequisite products were installed along with the agent manager.

To prevent the loss of data:

Note:

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 59

v Do not uninstall the agent manager until all products that use it have

been uninstalled.

v Do not clean the agent manager tables from the registry or drop the

registry database until all products that use the registry are uninstalled.

uninstall the agent manager, follow these steps:

1. Stop the agent manager server if it is running:

v As a Windows service:

To stop the agent manager server if the installation program created a Windows service, use the Windows Services window or the Services folder in the Microsoft Management Console to stop the service with the following name:

IBM WebSphere Application Server V5 - Tivoli Agent Manager

v On Windows but not as a service:

To stop the agent manager server if it is not a Windows service:

a. Open a command prompt window.

b. Run the following command:

WAS_install_dir\bin\stopServer app_server_name

where app_server_name is the name of the application server where the agent manager is installed. By default, this is AgentManager. The server name is case-sensitive.

When the agent manager server is stopped, the following message is generated:

ADMU4000I: Server app_server_name stop completed.

v On AIX, Linux, or Solaris systems

To stop the agent manager server on AIX, Linux, or Solaris systems, run the following command:

WAS_install_dir/bin/stopServer.sh app_server_name

When the agent manager server is stopped, the following message is generated:

ADMU4000I: Server app_server_name stop completed.

2. Optionally, remove agent manager objects from the registry database.

v If the registry database is used only by the agent manager and is not shared

with another program, drop the database using the database administration tools for your type of database. If the registry is in a remote database, you might have to perform this step on the remote database server instead of on the agent manager server.

v If the database is shared with other programs, remove the agent

manager-specific tables from the database by following the procedure for your database type. Yo u can do this step on the agent manager server, even if the registry is in a remote database:

– DB2 Universal Database

a. In a command line window, change to the

Agent_Manager_install_dir/db/db2 directory.

b. Run the following command:

- On Windows

db2cmd /c /i /w "RemoveCASTables.bat database_password"

- On AIX, Linux, and Solaris

60 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Agent_Manager_install_dir/bin/ RunInDbEnv.sh

RemoveCASTables.sh database_password

Replace database_password with the DB2 database password.

Oracle

–

a. In a command line window, change to the

Agent_Manager_install_dir/db/oracle directory.

b. Run the following command:

- On Windows

./RemoveCASTables.bat database_password

- On AIX, Linux, and Solaris

./RemoveCASTables.sh database_password

Replace database_password with the Oracle database password.

Start the uninstallation program for your operating system:

v On Windows systems, either use the Add/Remove Programs window to

uninstall the agent manager, or run the following command from a command prompt:

Agent_Manager_install_dir\_uninst\uninstall.exe

v On AIX, Linux, and Solaris systems, run the following command from the

Agent_Manager_install_dir/_uninst directory:

java -jar uninstall.jar

Or, to uninstall silently:

java -jar uninstall.jar -silent

The program does not delete the registry database or files created in the agent manager installation directory after installation.

4. If the agent manager application server is not named AgentManager, determine

whether other Web applications are using that application server. If no other applications are using that application server, you can optionally delete the application server.

5. If you do not need the uninstallation logs, optionally delete the agent manager

installation directory. By default, this is the following directory:

v On Windows systems: C:\Program Files\IBM\AgentManager v On AIX, Linux, and Solaris systems: /opt/IBM/AgentManager

On a Windows system, you might have to restart the system before you

Note:

can delete the agent manager installation directory.

6. If the registry is in a remote database, run the following command on the

remote database server to uninstall the agent manager from that system:

java -jar "Agent_Manager_install_dir/_uninstDS/uninstall.jar" -silent

7. If you do not need the uninstallation logs on the remote database server,

optionally delete the agent manager installation directory. By default, this is the following directory:

v On Windows systems: C:\Program Files\IBM\AgentManager v On AIX, Linux, and Solaris systems: /opt/IBM/AgentManager

On a Windows system, you might have to restart the system before you

Note:

can delete the agent manager installation directory.

8. If you will not be reinstalling the agent manager on this system, remove the

definition of TivoliAgentRecovery from your DNS servers.

agent manager is now uninstalled.

The

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 61

Reinstalling Tivoli Intelligent Orchestrator: Retry the Tivoli Intelligent Orchestrator installation. The Tivoli Intelligent Orchestrator installer will detect that Tivoli Intelligent Orchestrator is already installed, and will install only agent manager.

An error occurs when running the rpm -qa command after the DB2 Universal Database, IBM Tivoli Directory Server, and WebSphere Application Server prerequisites are installed on Linux

After successfully installing the Tivoli Intelligent Orchestrator prerequisites on Linux, the following error might occur when running the rpm -qa command. This command queries the packages that are installed on the RedHat Advanced Server.

error: db4 error(-30989) from dbcursor->c_get: DB_PAGE_NOTFOUND: Requested page not found

Cause

This is a known problem for RedHat Enterprise Linux version 3. A workaround is documented on the RedHat We b site as defect number 149974. For more details, see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=149974.

Solution

Run the following command first, to remove the cache files where the page is not found:

rm -f /var/lib/rpm/__db*

The cache will be re-created as needed.

Run the rpm -qa command afterwards, to obtain a complete listing of installed packages on the RedHat Advanced Server, without returning errors.

Problems with logging in

This section includes troubleshooting scenarios for the following problems:

v “You cannot log in”

v “Problems with logging in”

You cannot log in

Cause

The LDAP server might be down.

Solution 1

Run the ibmdirctl command to check the status of the LDAP server. From LDAP_install_directory\appsrv\bin, run the ibmdirctl command using the following syntax:

ibmdirctl -D username -w password status

The returned value is ibmslapd process is running when the server is running. If the system does not confirm that the process is running, start the server.

To start the server, run ibmdirctl again and use the following syntax:

ibmdirctl -D username -w password start

62 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Solution 2

Run the Tivoli Directory Server Web Administration tool to check the status of the LDAP server.

This solution is provided for a two-node directory server installation.

Note:

1. From LDAP_Install\appsrv\bin, run:

v startServer.sh in UNIX v startServer.bat in Windows

and use the following syntax:

startServer server1

2. In a Web browser, go to http://hostname:9080/IDSWebApp/IDSjsp/Login.jsp

3. Log in with the appropriate user name and password. The Tivoli Directory

Server We b Administration tool starts.

4. In the navigation bar click Server administration. The status of the directory

server appears in the main window.

5. Click Start or Stop or Restart to change the status of the server.

more information on scenarios that can affect login, see the Tivoli Intelligent

For Orchestrator support Web site:

http://www.ibm.com/software/sysmgmt/ products/support/IBMTivoliIntelligentOrchestrator.html

In the Choose a Product list, select Tivoli Intelligent Orchestrator.

You cannot change your password

Cause

The password policy parameters are incorrect (such as minimum length, minimum number of alphabetic characters, and other parameters), so the LDAP server cannot validate the password.

Solution

Run the Tivoli Directory Server Web Administration tool to check the password policy.

This solution is provided for a two-node directory server installation.

Note:

1. From LDAP_Install\appsrv\bin, run:

v startServer.sh in UNIX v startServer.bat in Windows

and use the following syntax:

startServer server1

2. In a Web browser, go to http://hostname:9080/IDSWebApp/IDSjsp/Login.jsp

3. Log in with the appropriate user name and password. The Tivoli Directory

Server We b Administration tool starts.

4. In the navigation bar click Server administration and then click Manage

security properties.

5. In the main window:

a. Click Password policy to check whether the password syntax is enabled or

not.

b. Click Password validation to check the password syntax.

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 63

6. Change your password again based on the specified password syntax.

Problems with workflows

This section includes troubleshooting scenarios for the following problems:

v “DB2 Universal Database errors occur when you deploy resources”

v “DB2 Universal Database creates a database state error”

v “DB2 Universal Database deadlocks occur when the system runs a logical

operation” on page 65

v “Shell command error: Resource temporarily unavailable” on page 65

v “IBM Tivoli Software Support personnel have requested a copy of a workflow”

on page 65

v “A workflow does not install” on page 67

v “Cannot run workflows if the PS1 environment variable is altered” on page 68

DB2 Universal Database errors occur when you deploy resources

When you run a workflow to deploy resources, you might see a DB2 Universal Database error telling you that an SQL exception has occurred.

Cause

This error can occur when the application heap size is not large enough to process the request.

Solution

Increase the application heap size, rebind the packages, terminate all connections, and restart the Tivoli Intelligent Orchestrator server. To complete this task:

1. Log on to the DB2 Universal Database server as the DB2 Universal Database instance owner.

2. Open a DB2 Universal Database command window and enter the following command to increase the application heap size:

db2 update db cfg for $DB2_DB_NAME using applheapsz 3072

where $DB2_DB_NAME is the name of the database you want to modify.

3. Under the sqllib\bnd directory, run the following command to rebind the packages:

db2 connect to $DB2_DB_NAME db2 "bind @db2ubind.lst blocking all grant public" db2 "bind @db2cli.lst blocking all grant public CLIPKG 6"

4. Run the following command to terminate all connections:

db2 force applications all

5. Restart the Tivoli Intelligent Orchestrator server.

DB2 Universal Database creates a database state error

Tivoli Intelligent Orchestrator displays a DB2 Universal Database error. For example:

com.ibm.tivoli.orchestrator.de.dao.PersistentStateException: COPDEX038E A persistent/database state error: [IBM][CLI Driver][DB2/NT] SQL0302N EXECUTE <DB text> SQLSTATE=22001 occurred.

64 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Cause

The workflow variable values are limited to 4000 bytes. When a scriptlet or Java plug-in returns a value longer than 4000 bytes, DB2 Universal Database will throw an exception.

Solution

Keep the length of your workflow variables to less than 4000 bytes.

DB2 Universal Database deadlocks occur when the system runs a logical operation

Cause

The DB2 database configuration parameter (lock list) value is not large enough.

Solution

Adjust the size of the lock list value from 50 to 2000. Depending on the volume of traffic on the servers and the size of your data center model, you might want to select a more appropriate value.

For more information on setting the lock list value, go to

http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/ support/v8document.d2w/report?fn=r0000267.htm

Shell command error: Resource temporarily unavailable

If you get this error: bash: fork: Resource temporarily unavailable, it might be accompanied by this Tivoli Intelligent Orchestrator message:

COPCOM123E This shell command error occured: Exit value=128, Error stream= "/usr/bin/bash: fork: Resource temporarily unavailable ", Result stream="".

Solution

Increase the maximum number of processes allowed per user on your system.

For instructions on setting the maximum number of processes allowed per user, refer to the documentation for your operating system.

Shell command error: Exit value=1, Error stream=", Result stream="no bash in ..."

When you are running a workflow, you might see the following error:

COPCOM123E This shell command error occurred: Exit value=1, Error stream="", Result stream="no bash in /bin/usr/bin /usr/sbin /usr/local/bin . /usr/bin / etc /usr/sbin /usr/ucb /usr/bin//usr/java131/jre/bin /usr/java131/bin / nmlprod/common/scripts /nmlprod/common/bin /opt/seos/bin".

Solution

The automation package that you are running requires bash on the endpoint.

IBM Tivoli Software Support personnel have requested a copy of a workflow

Solution: workflowLogExport tool

This command line utility exports the run history of one or more specified workflows to a log file that can be either in XML format (the default option) or in any other file format of your choice.

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 65

This command line utility exports the workflow execution logs into an XML file. The tool simplifies the process of sending workflows to the Software Support team so they can access the log information that they need to diagnose a workflow problem without accessing your computer remotely. The user can export the run history of individual workflows, or the run history of more logs into a log file whose name, file format, and location you can specify.

The command line utility is available at the following location:

Windows

%TIO_HOME%\tools

where %TIO_HOME% is the Tivoli Intelligent Orchestrator home directory.

or Linux

UNIX

$TIO_HOME/tools

Syntax

for Windows: Enter the following command on one line:

workflowLogExport.cmd (-n <workflow_name>* | -r <request_id>) [-f <export_filename>] [-i <input_filename>]

where: v [-n workflow_name]* is a command option that specifies the name of the

workflow whose run history you want to export to a log file. The asterisk (*) indicates that you can include multiple workflow names in this variable. If you do not specify one or more workflow names, workflowLogExport will dump the workflow run history for all workflows.

v [-r <request_id>] is a command option that indicates the deployment request

identifier.

v [-f export_filename] is a command option that specifies the fully qualified

path and file name of the workflow log. By default, the file format for the exported log file is XML, and the default file name is workflowLogExport.xml. If you do not specify a different file format, the workflow log file will be exported by default to XML. Also, if no file location is specified, the log file will be exported by default to the %TIO_LOGS% directory.

v [-i input_filename] is a command option that specifies the name of the file

that lists the names of all workflows whose run history you want to export to the same log file.

Examples:

workflowLogExport.cmd -n MyWorkflow1 -n MyWorkflow2 -r 10017 -f "c:/myDirectory/ myWorkflowExport.xml" -i "c:/myDirectory/myExportWorkflowList.txt" workflowLogExport.cmd -i "c:/myDirectory/myExportWorkflowList.txt" workflowLogExport.cmd -i "c:/myDirectory/myExportWorkflowList.txt" -n MyWorkflow101

Syntax for UNIX:

workflowLogExport.cmd (-n <workflow_name>* | -r <request_id>) [-f <export_filename>] [-i <input_filename>]

where the command options are the same as the command options for Windows.

Returned file: The workflow run history will be exported into a log file with a file name and format that you can specify. If you do not specify them, the workflow run history will be exported by default to workflowLogExport.xml in the %TIO_LOGS% directory.

66 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Log content structure: workflow –> deployment request –> workflow execution

log –> workflow execution log detail

Example workflow log export:

<?xml version="1.0" encoding="UTF-8"?> <workflow-execution-history>

<deployment-requestid="10020">

<execution-log id="21" date="Mar 10, 2005 11:48:54 AM" position="1" call-stack-level="1" lo

<log-details position="2" name="Test Details Name"> Test Details Values</log-details>

</execution-log>

</deployment-request>

</workflow>

</workflow-execution-history>

A workflow does not install

Cause

This error can occur when workflows, parameters, and variables are named incorrectly.

Solution

To ensure that your workflows can be installed properly, ensure that you use valid names for workflows, parameters and variables.

Only the following characters can be used in workflow names: a-z, A-Z, 0-9, underscore (_), and period (.).

v The first character can be any of: a-z, A-Z, or underscore (_). It cannot be 0-9 or

a period (.)

v The last character in the name cannot be a period (.)

v Intervening characters between the first character and the last character in the

workflow can be: a-z, A-Z, 0-9, underscore (_), or period (.)

v Workflow names can be 255 characters in length.

the following characters can be used for workflow parameter and variable

Only names: a-z, A-Z, 0-9, and underscore (_).

v The first character can only be: a-z, A-Z, or underscore (_). It cannot be 0-9.

v The remaining characters can be: a-z, A-Z, 0-9, or underscore (_).

Workflow

keywords cannot be used for variable names. If your workflows contain

variables with any of the following names, you must rename them.

in, inout, catch, catchall, CheckDeviceLocale, credentialskey, DCMDelete, DCMInsert, DCMQuery, DCMUpdate, do, done, encrypted, else, endif, endtry, error, finally, foreach, if, implements, info, java, Java, Jython, language, LocaleInsensitive, log, out, noop, parent, rethrow, scriptlet, then, timeout, try, target, throw, var, warning, while, workflow

Note: Any Unicode character can be used to enter comments in workflow scripts.

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 67

Cannot run workflows if the PS1 environment variable is altered

Cause

When running scriptlets in Tivoli Intelligent Orchestrator, the system will expect that the command prompt ends in $ or #. The PS1 environment variable governs what your prompt looks like. If you change this variable on your managed-to or managed-from system, you might run into problems running workflows.

For example, on most UNIX systems, when you are logged in as tioadmin, by default, you see the following command prompt:

If you change the PS1 variable for tioadmin to be tioadmin>, then you will see the following as the command prompt:

tioadmin>

Solution

To be able to run scriptlets in Tivoli Intelligent Orchestrator, ensure that the command prompt ends in $ or #.

Here are examples of acceptable command prompts:

command$ $ # command#

For more information on troubleshooting workflows, refer to the Developing automation packages section in the Tivoli Intelligent Orchestrator information center.

Device Manager limitations

The following are known limitations for the Device Manager:

Running lightweight management server with DB2 Universal Database generates exceptions

Some device classes (plug-ins) are not configured with lightweight device management server. You can receive exceptions for the missing device classes. However, lightweight device management server will start and run with its more limited set of device classes.

76 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Agent receives an HTTP Unauthorized (401) response code

Agent can receive an Unauthorized (401) response code for a number of reasons, such as:

v The agent did not provide a client certificate as part of the SSL handshake,

which can be caused by the agent connecting in on the wrong port. The agent must connect in on the port which is configured for SSL client authentication.

v The client certificate provided as part of the SSL handshake revoked.

Known report limitations

The following are known limitations for the reporting capabilities of the Tivoli Intelligent Orchestrator:

Running any report with too many objects selected fails

Running any report with too many objects selected generates the following error:

COPJEE389E: The report contains too many objects. Retry with fewer selections.

Cause

Based on the information in the log files, this error is triggered by the following DB2 Universal Database error:

DB2 error - SQL0101N The statement is too long or too complex.

Two causes have been identified:

v The application heap size and statement heap size are not large enough to

process the request.

v The report contains too many selected objects, which generates an SQL

statement that is too large to be handled in one single report.

Solution

1. Increase the Java memory to prevent your computer from becoming slow or

unresponsive. Increase the application heap size and the statement heap size incrementally by 1024, until the problem is alleviated. To complete this task:

a. Open a DB2 Universal Database command window and enter the following

commands to increase the application heap size and statement heap size:

db2 update db cfg for $DB2_DB_NAME using applheapsz <size> db2 update db cfg for $DB2_DB_NAME using stmtheap <size>

where $DB2_DB_NAME is the name of the DB2 database that you want to modify.

2. Instead of selecting individual objects, try grouping objects first, and then select

the object groups. For example, you might want to try selecting groups of systems or entire application tiers or resource pools rather than selecting individual systems. Grouping objects will reduce the number of selections, making the report more manageable.

Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 77

78 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Chapter 8. Troubleshooting the agent manager

This chapter contains the following sections to help you resolve problems with the agent manager:

v “Log files for the agent manager”

v “Determining the version of agent manager” on page 83

v “Installation and uninstallation verifications” on page 83

v “Common problems” on page 87

Log files for the agent manager

The following sections describe how to locate and use the agent manager log files:

v “Packaging the log files”

v “Locating the installation log files”

v “Locating the uninstallation log files” on page 81

v “Locating the remote registry logs” on page 81

v “Locating the runtime log files” on page 82

v “Locating the DB2 Universal Database runtime logs” on page 82

Packaging the log files

If you encounter a problem that you cannot resolve immediately, collect a copy of the log files and system configuration information. This preserves the information you need to perform detailed problem determination and prevents you from having to scan back through messages and trace events that were recorded after the problem occurred. If you contact IBM Customer Support, you must provide this package.

To package the logs and configuration information into a compressed, or zipped, file for IBM Customer Support, use the log collection tool that is available in the toolkit subdirectory of the agent manager installation directory (Agent_Manager_install_dir).

Locating the installation log files

The log files generated during the installation and initial configuration of the agent manager are located in the Agent_Manager_install_dir\logs directory. The default directory for the agent manager installation is %TIO_HOME%\AgentManager on Windows ($TIO_HOME/AgentManager on UNIX or Linux). The following table lists the installation logs.

Table 10. Agent manager installation log files

Log file name Description

AMReturnValues.log Summary of return values of the steps of the

agent manager installation. If the agent manager was installed silently, check this log for the results of each step of the installation. If you installed using the agent manager installation wizard, this information is displayed automatically. If you are performing a silent installation, check this log for a step-by-step summary of the installation’s results.

am_install.log InstallShield MultiPlatform (ISMP) log for the

installation of the agent manager. Check this log first to verify that the agent manager installed properly and that the agent manager is started.

am_upgrade.log Information about whether a new installation

was performed or an existing version of the agent manager was found on the system and upgraded.

auth_stdout.log

Standard output and standard error logs for the AuthXMLUpdate program.

auth_stderr.log

certGen_stdout.log

Standard output and standard error logs for generating the root certificate for the agent

certGen_stderr.log

datastore_stdout.log

manager certificate authority.

Standard output and standard error logs for creating and initializing the tables in the

datastore_stderr.log

db_stdout.log

registry database.

Standard output and standard error logs for creating the registry database.

db_stderr.log

encrypt_stdout.log

Standard output and standard error logs for the EncryptAMProps program.

encrypt_stderr.log

guid_install.log

Standard output and standard error logs for installing the Tivoli globally unique identifier

guid_stdout.log

(GUID).

guid_stderr.log

msg_EPM_Install.log

Messages and trace information generated during the installation and configuration of the

trace_EPM_Install.log

agent manager applications in WebSphere Application Server.

startserver_stdout.log

Standard output and standard error logs for starting the agent manager application server

startserver_stderr.log

jacl/amApp_out.log

under WebSphere.

Standard output and standard error logs generated while installing the AgentManager

jacl/amApp_err.log

and AgentRecoveryService applications and .WAR files. These logs are generated by the EPMInstallApp.jacl configuration script.

80 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Table 10. Agent manager installation log files (continued)

Log file name Description

jacl/appServer_out.log

Standard output and standard error logs generated while installing the application

jacl/appServer_err.log

server for the agent manager. These logs are generated by the EPMAppServer.jacl script

jacl/checkcell_out.log

Standard output and standard error logs for verifying the cell for the WebSphere

jacl/checkcell_err.log

configuration. These logs are generated by the EPMValidate.jacl script.

jacl/checknode_out.log

Standard output and standard error logs for verifying the node for the WebSphere

jacl/checknode_err.log

configuration. These logs are generated by the EPMValidate.jacl script.

jacl/jdbc_out.log

Standard output and standard error logs for configuring the WebSphere JDBC provider,

jacl/jdbc_err.log

data source, and J2C Authentication Data Entry. These logs are generated by the EPMJdbcProvider.jacl script.

jacl/ssl_out.log

Standard output and standard error logs for SSL configuration. These logs are generated by

jacl/ssl_err.log

jacl/virHost_out.log

the EPMSSLConfig.jacl script.

Standard output and standard error logs for creating the WebSphere virtual host. These logs

jacl/virHost_err.log

are generated by the EPMVirtualHost.jacl script.

Locating the uninstallation log files

The log files generated when you uninstall the agent manager are located in the Agent_Manager_install_dir\logs directory. The following table lists the agent manager uninstallation logs:

Table 11. Agent manager uninstallation log files

Log file name Description

uninstall.log InstallShield MultiPlatform (ISMP) log for

uninstalling the agent manager.

AMUninstallReturnValues.log Summary of return values of the steps of the

agent manager uninstallation.

msg_EPM_Install.log

Messages and trace information generated when uninstalling the agent manager

trace_EPM_Install.log

applications in WebSphere Application Server.

Locating the remote registry logs

If the registry is on a system other than the agent manager server, install a subset of the agent manager files on the database server to create and initialize the registry database. The following table lists the logs that are created when creating the remote registry database.

Chapter 8. Troubleshooting the agent manager 81

Table 12. Remote registry database installation log files

Log File Description

datastore.out A log of the SQL statements that are run to create the registry

database and its tables.

ds_install.log An ISMP log for installing the files necessary to create the registry

database.

db_stdout.log

Standard output and standard error logs for creating the registry database. Check db_stdout.log first to see if the registry was

db_stderr.log

created successfully.

Locating the runtime log files

The runtime logs for the agent manager are located in the was_install_dir\logs\ app_server_name

server. By default, this is Agent Manager.

The runtime logs for WebSphere are located in the WAS_install_dir\logs\server1 directory.

Additional logs are in the was_install_dir/logs/ffdc directory.

directory, where app_server_name is the name of the application

Locating the DB2 Universal Database runtime logs

DB2 Universal Database provides several First Failure Data Capture (FFDC) facilities that log information as errors occur. The DB2 Universal Database FFDC facilities are used with CLI and DB2 traces to diagnose problems. The information captured by DB2 Universal Database for FFDC includes:

db2diag.log

When an error occurs, the db2diag.log is updated with information about the error. This is the primary log to use when debugging DB2 Universal Database problems.

db2alert.log

dump files

files

trap

If an error is determined to be an alert, then an entry is made in the db2alert.log file and to the operating system or native logging facility.

For some error conditions, additional information is logged in external binary dump files named after the failing process ID. These files are intended for DB2 Universal Database Customer Support.

useful DB2 Universal Database commands include:

Other

db2trc This command gives you the ability to control tracing.

db2support

This command collects environment information and log files and places them into a compressed archive file.

82 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Determining the version of agent manager

To determine the version of the agent manager, use the GetAMInfo command, which is located in the Agent_Manager_install_dir/bin directory. On Windows systems, it has the .cmd extension (GetAMInfo.cmd). On AIX, Linux, and Solaris systems, is has the .sh extension (GetAMInfo.sh).

To display the version of the agent manager and agent recovery service applications, run the following command. This example shows the command for a AIX, Linux, or Solaris system:

Agent_Manager_install_dir/GetAMInfo.sh

To display the version of the agent manager application, run the following command. This example shows the command for a Windows system:

Agent_Manager_install_dir\GetAMInfo.bat AgentManager

Note: The application name, AgentManager, is not related to the name of

application server in which you install agent manager. For example, if you install the agent manager into the application server named server1, the application name remains AgentManager.

To display the version of the agent recovery service, run the following command. This example shows the command for a AIX, Linux, or Solaris system:

Agent_Manager_install_dir/GetAMInfo.sh AgentRecoveryService

Installation and uninstallation verifications

The following sections are described:

v “Verifying that the agent manager service is running”

v “Verifying the installation of WebSphere Application Server” on page 84

v “Verifying the installation of DB2 Universal Database Enterprise Server Edition

(ESE)” on page 84

v “Errors starting the agent manager application server” on page 84

v “Errors starting the agent manager application” on page 84

v “Verifying the connection to the registry” on page 84

v “Enabling and disabling tracing” on page 85

v “Recovering from an incomplete uninstallation of the agent manager” on page

Verifying that the agent manager service is running

If you suspect that the agent manager server is not running, use the health check tools that are available in the toolkit subdirectory of the agent manager installation directory (Agent_Manager_install_dir). The WebSphere Application Server serverStatus command tells whether the application server is started, but does not indicate whether it is fully functional. For example, if a remote registry database is inaccessible because of a network or authorization problem, the serverStatus command reports that the application server is operational. However, the agent manager server cannot register agents or resource managers without a connection to the registry.

Chapter 8. Troubleshooting the agent manager 83

Verifying the installation of WebSphere Application Server

To verify the installation of WebSphere Application Server, use the firststeps tool, located in the WAS_install_dir/firststeps directory. Run the program for your operating system:

v Windows: firststeps.bat v AIX, Linux, or Solaris: firststeps.sh

Verifying the installation of DB2 Universal Database Enterprise Server Edition (ESE)

If the registry is in a DB2 database and the agent manager server does not install or start properly, or if you receive errors about problems accessing the registry, verify that DB2 Universal Database ESE installed properly and that the DB2 Universal Database server that controls the registry database is running.

To verify that DB2 Universal Database ESE installed properly, refer to the DB2 Universal Databasedocumentation for your operating system. The installation documentation contains information about verifying the installation using either the command line processor (CLP) or the First Steps GUI. The general procedure is the same for both methods:

1. Create the SAMPLE database.

2. Connect to the SAMPLE database.

3. Execute a query against the SAMPLE database.

4. Drop the SAMPLE database.

Errors starting the agent manager application server

If you cannot start the agent manager application server:

v If you get a NoServerDefinedException when you use the WebSphere

startServer command to start the agent manager application server, ensure that

you specified the correct server name. Although the default name of the agent manager application server is AgentManager, the agent manager might be installed to a different application server, such as server1.

v If you have reconfigured WebSphere Application Server on UNIX or Linux to

run as a user other than root and get a FileNotFoundException exception or a “Permission denied” error, make sure that the user ID that runs the WebSphere processes has permission to read the agent manager files, particularly the files in the Agent_Manager_install_dir/certs directory.

Errors starting the agent manager application

If the logs indicate a problem starting the certificate authority, a common cause is a problem with the certificates. To correct this, you can replace your certificates.

Verifying the connection to the registry

If any errors indicate a problem accessing the registry, verify that the registry is correctly configured and available by performing these steps:

1. Open the WebSphere Administrative Console, and then expand Resources and then click JDBC Providers.

2. In the Server field on the JDBC Providers panel, type the name of the application server where the agent manager is installed or click Browse to select it from a list. By default, this is AgentManager.

84 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

3. Click Apply to list the JDBC providers for the agent manager application

server.

4. In the list of providers, click AgentJDBCProvider.

5. In the Additional Properties area at the bottom of the Configuration page of the

JDBC provider, click Data Sources.

6. In the Data Sources page, check the box next to AgentRegistry and then click

Test Connection.

7. Review the status message at the top of the Data Sources panel. A message

similar to the following indicates that the JDBC connection is correctly configured:

Test connection for datasource AgentRegistry on server server1 at node WAShostname was successful.

Enabling and disabling tracing

You can control whether trace information is captured and the amount of details in agent manager trace levels. By default, tracing is disabled.

Enabling tracing

To enable tracing for the agent manager:

1. Start the WebSphere Administrative Console.

2. Click Servers –> Application Servers –> app_server_name –> Logging and

Tracing –> Diagnostic Trace.

Replace app_server_name with the name of the application server where the agent manager applications are installed, for example, AgentManager.

3. In the General Properties area of the Configuration page, make sure that

Enable Trace is checked.

4. In the Trace Specification field on the Runtime page, type one of the values

specified in the table below..

Table 13. Setting trace levels

Task Value in Trace Specification text box

Turn on all tracing mgr.*=all=enabled

Turn on the entry and exit tracing mgr.*=entryexit=enabled

Turn on the highest level of tracing mgr.*=debug=enabled

Turn on tracing for warnings mgr.*=event=enabled

The Trace Output area shows the directory and file name where the trace information is written. A typical location is ${SERVER_LOG_ROOT}/trace.log, where the WebSphere environment variable ${SERVER_LOG_ROOT} represents the runtime log directory, WAS_install_dir/logs/app_server_name.

5. Click OK.

6. In the Taskbar, click Save to save the change to the master configuration.

7. In the Save to Master Configuration area, click Save again to confirm the

change.

Disabling tracing

To disable tracing for the agent manager:

1. Start the WebSphere Administrative Console.

Chapter 8. Troubleshooting the agent manager 85

2. Click Servers –> Application Servers –> app_server_name –> Logging and Tracing –> Diagnostic Trace. Replace app_server_name with the name of the

application server where the agent manager applications are installed, for example, AgentManager.

3. Ensure that Enable Trace is not checked.

4. Click OK.

Recovering from an incomplete uninstallation of the agent manager

This section describes how to manually remove the agent manager from your environment if the uninstallation wizard does not complete successfully. Use this procedure only after running the wizard. Some of these steps might not be necessary, depending on how much of the uninstallation wizard was completed.

1. Stop the agent manager server if it is running. Run the following command:

v On Windows systems: WAS_install_dir\bin\stopServer.bat AgentManager v On AIX, Linux, or Solaris systems: WAS_install_dir/bin/stopServer.sh

AgentManager

If the agent manager server is configured to start automatically after a

Windows system restarts, delete the Windows service for the agent manager, using the following command:

WAS_install_dir\bin\WASService.exe -remove "Tivoli Agent Manager"

3. If the agent manager server is configured to start automatically after an AIX, Linux, or Solaris system restarts, remove the entries in the /etc/inittab file that start the server:

v Remove the entry that starts the agent manager server. This entry is added

on all systems.

am:2345:once:/opt/IBM/WebSphere/AppServer/bin/rc.am >/dev/console 2>&1

In this example, WebSphere Application Server is installed in the /opt/IBM/WebSphere/AppServer directory.

v If the registry is in a local DB2 database, remove the entry that starts the DB2

server. The entry is not created if the registry is in a remote DB2 database, or a local or remote Oracle database.

amdb:2345:once:su - db2inst1 -c db2start >/dev/console 2>&1

In this example, db2inst1 is the DB2 user name for accessing the registry.

Remove the agent manager from WebSphere Application Server:

a. Open the WebSphere Administrative Console.

b. In the navigation tree on the left side of the console, expand Environment,

then click Virtual Hosts, and then delete AgentManagerHost.

c. Expand Security, then click SSL, and then delete AgentManagerSSL and

AgentManagerClientAuthSSL.

d. Expand Resources, and then click JDBC Providers. Using the filter table,

set the scope to the server AgentManager, and then delete AgentJDBCProvider.

e. Expand Applications, then click Enterprise Applications, and then uninstall

AgentManager and AgentRecoveryService.

f. Expand Servers, then click Application Servers, and then delete

AgentManager.

g. In the Message(s) area of the console, click Save to save the changes to the

configuration.

86 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

h. In the Save to Master Configuration area , click Save again to exit the

Master Configuration window.

Optionally, clean up the agent manager objects in the database.

Important:

Remove the agent manager tables from the database or drop the

registry database only if all products that use the registry are uninstalled.

a. Remove the agent manager tables from the registry database by running the

following command:

v On Windows systems:

Agent_Manager_install_dir\db\database_type\RemoveCASTables.bat

v On AIX, Linux, and Solaris systems:

Agent_Manager_install_dir/db/database_type\RemoveCASTables.sh

Replace variable database_type with one of the following values to identify the database type:

– db2

– oracle

If the database is used only by the agent manager and is not shared with

another program, optionally drop the registry database using the database administration tools for your type of database.

6. Delete the agent manager installation directory. By default, this is the following

directory:

v On Windows systems: C:\Program Files\IBM\AgentManager v On AIX, Linux, and Solaris systems: /opt/IBM/AgentManager

On a Windows system, you might have to restart the system before you

Note:

can delete the agent manager installation directory.

Common problems

You might encounter the following problems:

v “Registration and communication failures”

v “Problems using the WebSphere Administrative Console” on page 88

Registration and communication failures

The WebSphere log SystemOut.log contains a record of agents that contact the agent recovery service. Common reasons for an agent to be unable to register or communicate with the agent manager include the following problems:

v The agent manager properties file, which the agent manager sends to the agent

when it first connects, might contain a short host name in the URL for the Web services instead of a fully qualified host name. For example, the person who installs the agent manager might specify the host name as myserver instead of myserver.ibm.com. If an agent is in a network that cannot resolve the short host name to the fully qualified host name, it cannot access the agent manager services. Because the person who installs the agent can specify the fully qualified host name for the initial connection to the agent manager server, the problem can be discovered after the initial connection.

v The wrong truststore file was used when installing the agent or resource

manager.

v The wrong agent registration password was used when installing the agent.

Chapter 8. Troubleshooting the agent manager 87

v The wrong resource manager registration user or password was used when

installing the resource manager.

v Agent manager configuration files, such as AgentManager.properties or

Authorization.xml, are missing or have been altered.

v Security credentials are invalid, expired, or revoked.

v An agent is attempting to re-register, but duplicate registration is disabled.

Problems using the WebSphere Administrative Console

This section describes common problems that you might encounter with the WebSphere Administrative Console.

Cannot start the WebSphere Administrative Console

You receive a ″Page cannot be displayed″ message when you start the WebSphere Administrative Console.

Possible problems: This error can be caused by the following problems:

v The wrong port number is being used.

v The WebSphere server that runs the WebSphere Administrative Console is

stopped.

Solution:

Tr y the following solutions:

v Ensure that the port number shown in the browser Address field shows the

correct port number. Typically, this is port 9090, but it can be different, particularly on AIX systems.

v Start the WebSphere server that runs the WebSphere Administrative Console:

As a Windows service

To start the WebSphere server as a Windows service, use the Windows Services window or the Services folder in the Microsoft Management Console to start the service with the following name:

IBM WebSphere Application Server V5 - server1

Using a Windows command

To start the WebSphere using a Windows command:

1. Open a command prompt window.

2. Change to the WAS_install_dir\bin directory.

3. Run the following command:

startServer server1

Note: The service name, server1, is case-sensitive.

4. Check the log to see if any errors occurred. The log is in the

WAS_install_dir\server1\logs\startServer.log file.

AIX, Linux, or Solaris systems

To start the WebSphere server on AIX, Linux, or Solaris systems, run the following command:

startServer server1

Configuring the WebSphere Administrative Console to use a different port

By default, the HTTP Transport uses port 9090 to communicate with the WebSphere Administrative Console. Other processes, such as the Web-based System Manager on AIX systems, might also use port 9090. To resolve this conflict,

88 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

IBM Problem Determination User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

Preface

Who should read this book

Publications

Tivoli Intelligent Orchestrator library

Prerequisite publications

Related publications and resources

Accessing publications online

Ordering publications

Accessibility

Tivoli software training

Support information

Conventions used in this book

Typeface conventions

Chapter 1. Introduction to troubleshooting

Product and system overview

Product comparison

Product architecture

Tivoli Intelligent Orchestrator components

Prerequisites installed with Tivoli Intelligent Orchestrator

Built-in troubleshooting features

Problem classification

Product maintenance

Backing up the system

Performing periodic checks and maintenance

Problem resolution

Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator

Problem determination methodologies

Defining the problem

Identifying the cause of the problem

Fixing the problem

Troubleshooting methods for Tivoli Intelligent Orchestrator and prerequisites

Troubleshooting the prerequisite installation

DB2 Universal Database verification

IBM Tivoli Directory Server verification

WebSphere Application Server verification

Tivoli Intelligent Orchestrator installation and subcomponent configuration verification

Tivoli Intelligent Orchestrator installation

Verifying subcomponent configuration

Tivoli Intelligent Orchestrator runtime problem determination

Component-based problem determination

Task-based problem determination

Accessing the knowledge bases

Using the available command line tools

Tivoli Intelligent Orchestrator command line tools

WebSphere Application Server command line tools

IBM Tivoli Directory Server command line tools

Chapter 3. Log file types

Message logs

Message ID format

Message elements

Trace logs

Common Base Event logs

Chapter 4. Locating and configuring log files

Log locations

Log directory files

Logging levels

Configuring logs with log4j

log4j.prop

Log level defaults

RollingFileAppender defaults

Configuring log4j dynamically

Preparing log files for review by IBM Tivoli Support

Chapter 5. Tivoli Intelligent Orchestrator log files

Log files for product setup

Log files for the uninstall process

Log files for starting and stopping the server

Log files for the Web interface

Deployment engine log files

Workflow logs

Logs for data collection and decision making

Automation package logs

Chapter 6. Middleware prerequisite log files

Installation log files for prerequisites

Installation logs for individual prerequisites

WebSphere Application Server