IBM E027SLL-H, Tivoli Monitoring 6.2.3 FP1 Troubleshooting Manual
IBM Tivoli Monitoring
Version 6.2.3 FP1
Troubleshooting Guide
GC32-9458-05
IBM Tivoli Monitoring
Version 6.2.3 FP1
Troubleshooting Guide
GC32-9458-05
Note
Before using this information and the product it supports, read the information in “Notices” on page 267.
This edition applies to version 6, release 2, modification 3, fix pack 1 of IBM Tivoli Monitoring (product number
5724-C04 ) and to all subsequent releases and modifications until otherwise indicated in new editions.
© Copyright IBM Corporation 2005, 2012.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
Contents
Tables ...............xiii
About this information ........xv
Chapter 1. Introduction to
troubleshooting ...........1
Sources of troubleshooting information .....1
Problem classification ...........1
Viewing the IBM Support Portal ........2
Subscribing to IBM support notifications .....2
Chapter 2. Logs and data collection for
troubleshooting ...........5
Appropriate IBM Tivoli Monitoring RAS1 trace
output ................5
Running snapcore to collect information .....5
Locating the core file ...........6
Getting Dr. Watson dumps and logs ......7
KpcCMA.RAS files ............7
Sources of other important information .....8
Chapter 3. Common problem solving . . 9
About the tools .............9
I am trying to find out what software is supported . 9
Workspaces are missing or views are empty . . . 10
Diagnosing that workspaces are missing or
empty ...............10
Resolving application support problems ....11
Resolving monitoring server problems ....12
Resolving monitoring agent problems ....12
Status of a monitoring agent is mismatched between
the portal client and tacmd command .....13
Diagnosing that the status of a monitoring agent
is mismatched between the portal client and
tacmd command............13
Resolving monitoring agent problems ....14
Resolving monitoring server problems ....14
The portal server does not start or stops responding 15
Diagnosing that the portal server does not start
or stops responding ..........15
Resolving database problems - missing table or
portal server database ..........16
Resolving database problems - user ID and
password ..............16
Resolving database problems - instance not
started ...............17
Diagnosing that portal server logon fails....18
The portal client does not respond.......18
Diagnosing that the portal client does not
respond ..............18
Resolving storage or memory problems ....19
Resolving client configuration problems ....19
Historical data is missing or incorrect .....20
Diagnosing that historical data is missing or
incorrect ..............20
Resolving warehouse proxy connection problems 21
Resolving warehouse proxy agent problems -
configuration .............21
Resolving warehouse proxy agent problems -
connectivity .............22
Resolving summarization and pruning agent
problems ..............22
Resolving persistent data store for z/OS
problems ..............23
Historical data does not get collected for some
monitoring server attribute groups .....25
A situation does not raise when expected ....25
Diagnosing that a situation does not raise when
expected ..............25
Resolving situation-specific problems .....26
A reflex automation script does not run when it
should ................28
Diagnosing that a reflex automation script does
not run when it should .........28
Resolving format and variable problems ....28
High CPU usage on a distributed system ....29
Diagnosing high CPU usage on a distributed
system ...............29
Resolving situation problems - diagnostic actions 30
Resolving situation problems - corrective actions 31
Resolving firewall problems - diagnostic actions 31
Resolving firewall problems - corrective actions 31
Resolving Oracle DB Agent problems - diagnostic
actions ...............32
Resolving Oracle DB Agent problems - corrective
actions ...............32
Chapter 4. Tools...........35
Trace logging ..............35
Log file locations ...........35
Installation log files...........39
Reading RAS1 logs ...........42
Setting traces .............43
Dynamically modify trace settings for an IBM Tivoli
Monitoring component ..........53
Using the IBM Tivoli Monitoring Service Console . 55
Starting the IBM Tivoli Monitoring service
console ...............56
Blocking access to the IBM Tivoli Monitoring
Service Console ............56
Displaying portal server tasks in the command
prompt ................57
KfwSQLClient utility ...........57
Clearing the JAR cache ..........58
Using the UAGENT application .......59
pdcollect tool ..............59
ras1log tool ..............60
Backspace Check utility ..........60
© Copyright IBM Corp. 2005, 2012 iii
Build TEPS Database utility .........61
IBM Tivoli Monitoring Operations Logging ....61
Windows and UNIX systems .......61
z/OS systems ............62
ITMSuper ...............63
Chapter 5. Installation and
configuration troubleshooting .....65
Frequently asked questions .........65
General installation frequently asked questions 65
Windows installation frequently asked questions 65
UNIX-based systems installation frequently asked
questions ..............66
General installation problems and resolutions . . . 68
Agent Builder application support is not
displayed in listappinstallrecs output if it is
manually installed without recycling the
monitoring server ...........68
Debugging mismatched application support files 69
Startup Center fails to reset the sysadmin
password on the hub Tivoli Enterprise
Monitoring Server configuration panel ....69
Startup Center fails to create the Tivoli
Warehouse database and user .......69
On UNIX systems, a new user is not created or a
password is not reset in the Startup Center when
you use a non-root user to install Warehouse
Proxy Agent and Tivoli Enterprise Portal Server . 69
On Windows systems, a Tivoli Monitoring
Warehouse DSN is not created in the Startup
Center ...............69
Startup Center fails to test DSN with database
connectivity .............69
Startup Center shows some system types as
“Unknown Operating System” .......70
Tivoli Enterprise Monitoring Agents .....70
Upgrade SQL file not found when installing
application support on the standby hub ....73
Many files in the First Failure Data Capture log
directory ..............74
Monitoring agents fail to start after agent support
or multi-instance agents are installed .....74
Incorrect behavior after an uninstallation and
re-installation.............75
Where Remote Deployment of agents is not
supported ..............75
Application Support Installer hangs .....75
An agent bundle is not visible from the Tivoli
Enterprise Portal............75
Agent Management Services fails after
deployment on Linux Itanium and xLinux with
kernel 2.4 systems ...........76
Watchdog utility requires Windows Script Host
5.6................76
Unable to deploy monitoring agents from the
Tivoli Enterprise Portal .........76
Installing application support with a silent
installation response file fails .......76
Unable to run gsk7ikm.exe ........77
*_cq_*.log files appear ..........77
SPD: Installing a bundle on the wrong operating
system, architecture, or kernel .......77
Installing a Software Package Block (SPB) on top
of an existing, running IBM Tivoli Monitoring
agent ...............78
Problems with the SPB file ........78
Installation was halted and receive message
about active install ...........78
Receive an install.sh error when installing two
components or agents in the same installation
directory ..............78
When attempting to install IBM Java 1.5.0 on
Windows 64 bit system nothing happens . . . 78
Backup failure message during a remote
monitoring server upgrade ........79
Remote configuration of deployed Monitoring
Agent for DB2 agent fails .........79
Monitoring Server cannot find your deployment
depot ...............79
The agent installation log shows error
AMXUT7502E ............80
Failure occurs when sharing directories for the
agent deploy depot ...........80
You receive a KFWITM290E error when using
deploy commands with a z/OS monitoring
server ...............80
Running deployment in a hot-standby
environment .............80
Difficulty with default port numbers .....81
Selecting Security Validation User displays a
blank popup .............81
When installing a monitoring agent on top of the
Systems Monitor Agent, you receive an error . . 81
Some rows do not display in an upgraded table 81
The monitoring server and portal server
automatically start after running Application
Support Installer............82
Errors occur during installation of Event IBM
Tivoli Monitoring Event Forwarding tool . . . 82
Missing LSB tags and overrides warning message
at the end of installation .........82
Self-describing capability .........82
Windows installation problems and resolutions . . 84
On Windows systems, the installation fails
randomly when installing different features . . 84
Problems that are cleared by rebooting the
Windows system ...........85
When installing and configuring the Tivoli
Enterprise Monitoring Server on Windows Server
2008, a number of popups and errors occur. . . 85
After an upgrade, the Tivoli Enterprise Portal
Server is in the 'stop pending' state and cannot
be manually started ..........86
When running the setup.exe, an unknown
publisher error message displays ......86
The error “Could not open DNS registry key”
occurs ...............86
Agent not connecting to Tivoli Enterprise
Monitoring Server ...........86
iv IBM Tivoli Monitoring: Troubleshooting Guide
InstallShield displays the error “1607: Unable to
install InstallShield Scripting Runtime” during
installation on Windows from a
network-mounted drive .........87
Installation on a Windows 2003 server fails with
Error Number: 0x80040707 ........87
Extracting the nls_replace script causes remote
deployment to fail ...........87
Upgrade tool deploys agent to the wrong
directory ..............87
Deploying an agent instance gives a
KUICAR020E error ...........88
Uninstallation is not available for Application
Support on Windows systems .......88
Problems installing directly from the .zip file . . 88
Installation hangs or loops after presenting initial
splash screen .............88
Unable to discover systems within a specified IP
range when running the Startup Center from
eclipse.exe ..............88
UNIX-based system installation problems and
resolutions...............88
Self-describing capability might be overwritten
by UNIX monitoring server application support . 89
On a RHEL6 64-bit system, the Tivoli Monitoring
installer fails with errors .........91
Application agent remote deployment on
workload partition fails .........91
Message is received about the Korn Shell after
running the install.sh file ........92
AIX stat_daemon memory leak .......92
Manage Tivoli Enterprise Monitoring Services
does not start on AIX V6.1 ........92
UNIX and Linux install.sh command fails with
error code: 99 and error code: 4.......93
Receive KUIC02101W error ........93
Receive JVMDG080 or JVMXM012 Java errors . . 93
On HP-UX systems, where the host name does
not equal the nodename, the upgrade installation
fails to stop running processes .......94
EIF Slot Customization does not work on
upgraded zlinux systems .........94
Running of the KfwSQLClient binary fails on
Linux and AIX systems .........94
Failed to attach to the DB2 instance db2inst1
ERROR: Unable to create TEPS, return code = 3 . 94
Installation on SLES9 terminates with install.sh
failure:KCI1008E terminating... license declined . 95
Command line interface program of the
Application Support Installer is not currently
available ..............95
Silent installation on UNIX-based systems returns
an encryption key setting error .......95
The error “Unexpected Signal: 4 occurred at
PC=0xFEC3FDE4” occurs during installation . . 95
Installing IBM Tivoli Monitoring on Red Hat 5
and see the following error: “KCI1235E
terminating ... problem with starting Java Virtual
Machine” ..............95
Installation on the Linux S390 R2.6 64-bit
operating system fails with the message “LINUX
MONITORING AGENT V610Rnnn unable to
install agent” where nnn is the release number . 96
Troubleshooting z/OS-based installations ....96
Tivoli Monitoring z/OS initialization checklist. . 96
z/OS-based installations problems and
resolutions .............104
Uninstallation problems and workarounds . . . 107
Unable to uninstall multi-instance agent from a
managed system on windows 64bit .....108
Prompted for .msi file during uninstallation
process started from 'Add/Remove Programs'
on systems with v6.2.2 installed ......108
Uninstallation is blocked by another process
that is using the IBM Tivoli Monitoring Eclipse
help server .............108
Uninstallation of an agent produces help errors 108
Uninstallation of an agent occurring more than
once stops the OS agent .........109
After uninstallation, Tivoli Enterprise
Monitoring Server folder is not deleted....109
Removing a failed installation on Windows . . 109
Incorrect behavior after an uninstallation and
reinstallation.............113
Tivoli Data Warehouse database does not
uninstall ..............113
The agent installation log shows error
AMXUT7512E ............113
Prompted to uninstall a database that was not
running during uninstallation .......114
Chapter 6. Connectivity
troubleshooting ..........115
Cannot log on to the portal server ......115
Cannot connect to the portal server ......117
Cannot launch the portal client on Windows XP
after installation (message KFWITM215E) ....120
Portal server is initializing and is not ready for
communications ............121
Portal server is unavailable during a portal client
work session .............121
Portal server does not start after installation . . . 121
Portal server is not connecting with the hub
monitoring server ............121
DB2 errors when opening a Tivoli Enterprise Portal
workspace ..............122
A monitoring process fails to start on Linux or
UNIX after changing a .profile for root .....123
Heartbeat issues when running on a Linux guest
using VMware .............124
Chapter 7. Portal client
troubleshooting ..........127
Cannot select the Create new group icon within the
Object group editor ...........127
Cannot load product configuration data after
changing warehouse database from Oracle to DB2
on Linux or UNIX ............127
Contents v
Data in the Tivoli Enterprise Portal is missing and
you receive an error ...........127
JavaWebStart Tivoli Enterprise Portal fails to
display help screens ...........127
Client allows you to save a situation with an
invalid character ............128
Tivoli Enterprise Portal or the browser displays the
yen symbol as a backslash in Japanese .....128
Using an administrator name with non-latin1
characters, cannot log onto the Tivoli Enterprise
Portal ................128
Non-ASCII characters are not accepted in the user
ID or the distinguished name field ......128
The Tivoli Enterprise Portal desktop does not work
when exporting DISPLAY .........129
Some attribute groups showing a different name in
the Tivoli Enterprise Portal .........129
Monitoring agents show in an unexpected position
in the navigation tree...........129
Tivoli Enterprise Portal Desktop called through
Java Web Start does not work properly after
adding agents support for Tivoli Enterprise Portal
Server ................131
Receive a Loading Java TM0 Applet Failed... error 131
tacmd createUser output indicates that the path to
the Java home directory was not found .....131
Cannot launch the Tivoli Enterprise Portal help 132
On an Active Directory Server, sysadmin cannot
logon to the Tivoli Enterprise Portal client ....132
Several enterprise workspaces are returning an
error, KFWITM217E:Request
Error,SQL1_CreateRequest Failed, rc=350 ....133
You cannot paste non-ASCII characters in the
Situation editor .............133
Situation editor cannot display advanced advice
help files ...............133
After acknowledging a situation event and
selecting the link for that situation, you might
receive a message ............133
Password problem using the LDAP Security option
on Active Directory system .........134
There is a memory leak in the Tivoli Enterprise
Portal browser client when the number of
workspace switches increases ........134
Help index and search text entry fields are
disabled ...............134
Java exception logging onto the Tivoli Enterprise
Portal from a browser ..........134
On Linux, IBM Tivoli Enterprise Monitoring Agent
topics do not display in the Help Contents or
Index tabs ..............135
Navigator items are listed in an unexpected order 135
Clicking on the Timespan icon for one view brings
up the data for another view ........135
HEAPDUMPs and JAVACore files are placed on
desktops when running in browser mode ....135
Java errors occur with the IBM v1.4.2 JRE ....137
Web Portal Client does not work with Sun JRE . . 137
Tivoli Enterprise Portal has high memory usage
and poor response time ..........138
Tivoli Enterprise Portal has high memory usage 138
Data is not returned to the portal client ....139
DirectDraw thread loops infinitely causing poor
portal client performance .........139
Workflow Editor is disabled and the following
tools do not display: Event Console, Graphic View,
Edit Navigator View (Navigator view toolbar) . . 140
Situations are not firing ..........140
Historical UADVISOR situations are started on the
agent if historical collection is configured to collect
data ................140
At the bottom of each view, you see a historical
workspace KFWITM217E error .......141
Installation of situation data fails due to I/O on
VSAM data sets ............141
kshsoap client fails because of missing libraries on
UNIX-based systems ...........142
Category and Message field of the universal
message does not accept DBCS .......142
An error occurs when remotely removing an
instance on Windows...........142
Agents display offline in the portal client but fire
situations and agent logs report that they are
running ...............143
Navigator view sorts erratically when you remove
multiple managed systems simultaneously . . . 143
Multiple events that occur at the same time are
loaded too slowly ............143
Desktop client performs poorly after installing
Language Packs for IBM Tivoli Monitoring . . . 143
Existing OMEGAMON product imagery displays
after upgrading to IBM Tivoli Monitoring V6.1 . . 144
The Warehouse Proxy Agent started, but does not
appear in the Managed System Status list on the
Tivoli Enterprise Portal ..........144
Cannot start or stop agents from the Navigator
view ................144
Cannot load a ws_pres.css file in order to select a
language other than English ........145
Chapter 8. Portal server
troubleshooting ..........147
Performance impacts of the HTTP and HTTPS
protocols ...............147
Users who run the IBM HTTP Server do not have
permission to the content directory ......147
tacmd exportWorkspaces or importWorkspaces
receives an out of memory error .......147
The portal server and Warehouse Proxy Agent fail
to connect to the database on a 64-bit Windows
system ...............148
Failed to log on as sysadmin with portal server
LDAP enabled .............148
On AIX systems, newly created users with
auto-expire passwords cause installation failures . 148
Linux portal server unable to FTP catalog/attribute
files ................148
Upgrading the Tivoli Enterprise Portal Server takes
a long time ..............148
Running the Tivoli Management Services
Discovery Library Adapter, results in a book that
does not contain the fully qualified host name . . 148
vi IBM Tivoli Monitoring: Troubleshooting Guide
Portal server performance is slow ......149
Cannot create a Tivoli Enterprise Portal Server
database ...............149
You receive a KFW error when a query is sent to
more than 200 managed systems .......150
Non-hub situations are not associated at the Tivoli
Enterprise Portal Server level ........151
Starting and stopping the Eclipse Help Server . . 151
Non-root stopping or starting agents causes
problems ...............151
Root password is not accepted during non-root
Tivoli Enterprise Portal Server configuration . . . 151
Corba user exception is included in the portal
server log when creating situations ......152
Stopping or starting the eWAS subcomponent of
the portal server ............152
Chapter 9. Monitoring server
troubleshooting ..........153
Messages related to the index file are displayed
when the agent fails back to a remote monitoring
server ................153
A generic RPC communications error is received
when issuing a long-running tacmd execute
command ..............153
Troubleshooting monitoring server problems on
distributed systems ...........153
The CT_GET request method fails in SOAP
queries with a V6.2.3 hub monitoring server, a
remote hub monitoring server earlier than
V6.2.3, and an agent connected to a remote
monitoring server ...........154
Exposure of passwords in the clear .....154
Receive a seeding failed message ......154
High monitoring server CPU after restarting
with Warehouse Proxy Agents configured . . . 154
Upgrade inconsistency between the History and
Object windows ...........155
Attribute groups started for collection on the
managed systems should not be available on the
monitoring server list..........155
To decrypt a password,
KDS_VALIDATE_EXT='Y' is required ....156
Remote Tivoli Enterprise Monitoring Server
consumes high CPU when large number of
agents connect ............156
Unable to start the Tivoli Enterprise Monitoring
Server after the kdsmain process is terminated
abnormally .............156
THRESHOLDS.XML and Tivoli Enterprise
Monitoring Server table not cleaned when
managed system override is removed ....157
Situations fail to trigger for attributes by
applying group function.........157
Monitoring server application support
completes all seeding functions but might crash
as the program is exiting ........157
tacmd login fails when monitoring server is
configured with LDAP authentication ....158
Some agents are displayed in the Service
Console list that are not accessible from that
user interface ............159
tacmd login fails after hub monitoring server is
recycled ..............159
tacmd and SOAP are not able to connect . . . 160
The system crashes when attempting a bulk
import or export command........160
Monitoring server fails to start, but then does
after a reboot ............160
Remote monitoring server lost connection to the
hub and all agents display offline .....161
After the set timeout, the Tivoli Enterprise
Monitoring Server is still pending .....161
Providing the wrong path to configuration files
during LDAP configuration causes the Tivoli
Enterprise Portal login window to hang . . . 161
Crash on Linux remote monitoring server
during hub failover to Hot Standby .....162
HUB Tivoli Enterprise Monitoring Server
quiesce prevents the display of the data
collected by the attached Tivoli Enterprise
Monitoring Agents ..........162
During installation of a remote Tivoli Enterprise
Monitoring Server on a Windows system, the
agent support is applied, but fails .....162
Using a Deploy Group with addSystem or
updateAgent commands.........163
Tivoli Enterprise Monitoring Server requires
restart if you issue itmcmd server stop/start
commands when you are already logged on . . 163
Log indicates hub monitoring servers are down
when they are up ...........163
The Platform view in the Manage Tivoli
Enterprise Monitoring Services panel shows the
Tivoli Enterprise Monitoring Server as running
as a 32 bit application, but my agents are shown
as running as 64 bit applications ......164
Tivoli Enterprise Monitoring Server does not
release memory after running a large SQL query 164
SQL queries with more than 200 OR predicates
do not complete ...........164
Tivoli Enterprise Monitoring Server aborts
unexpectedly when exiting the telnet session
used to start it ............165
KCIIN0084E Timeout appears while waiting for
Tivoli Enterprise Monitoring Server to start on
AIX5.3..............165
In a hot standby (FTO) environment, commands
to a mirror hub might not return hub records
after reconnection ...........165
A deleted object is redisplayed when two hot
standby (FTO) hubs reconnect .......166
Troubleshooting monitoring server problems on
z/OS systems .............166
Receive Program KDFCINIT and Program
FAXCMON messages..........166
Contents vii
The Tivoli Enterprise Monitoring Server start
task (CANSDSST default) encountered error
message 'KLVST044 LOADLIST MEMBER NOT
FOUND IN RKANPAR DATASET (KDSLLIST)
KppLLIST KLVST001 CANDLE ENGINE
INITIALIZATION ERROR(S), ABEND U0012' in
the RKLVLOG at startup ........167
KDS Parameters not generated from the batch
parm deck .............167
Cannot encrypt text. A call to CSNBSYE failed.
Cannot encrypt contents of keyfile .....168
The error “KLVST005 MVS JOBSTEP
AUTHORIZATION REQUIRED KLVST001
CANDLE ENGINE INITIALIZATION
ERROR(S), ABEND U0012 CSV019I - Required
module KLVSTWTO not accessed, it is not APF
Authorized (RKANMODL) CSV028I - ABEND
306-0C” occurs in the z/OS monitoring server
RKLVLOG during startup ........168
The error “KLVSQ000 carved mode in effect for
extended storage” occurred in the RKLVLOG
during startup ............168
Error message 'KDSMA013 OPEN VTAM for
VDM1APPL failed with status 8' occurs in the
Tivoli Enterprise Monitoring Server start task
(CANSDSST default) ..........169
Chapter 10. Monitoring agent
troubleshooting ..........171
Command-line interface ..........171
OS agents ..............171
Linux OS agent fails to start .......172
OS agent start command fails .......172
Specific events are not monitored by the
Windows OS agent ..........172
Take action commands and reflex automation . . 172
Warehouse agents ............173
Unable to configure the Warehouse Proxy agent
with modified parameters from the Tivoli
Enterprise Portal GUI .........174
Workspaces ..............174
A workspace view is showing an error ....174
Local history migration tools move the agent
operation logs to multiple agent history locations . 175
Unreadable tool tip information for Available EIF
Receivers list of the Situation editor ......175
32-bit Agent Builder agent will not start on 64-bit
Windows with System Monitor Agent-installed OS
Agent ................175
Unable to locate the file name of an exported
situation that begins with numerals ......176
Tivoli Enterprise Portal data for UNIX OS and
Linux OS agents is not updated after stopping the
disk ................176
Testing the connection to the Tivoli Data
Warehouse database is valid even with an invalid
password...............176
Configured non-root user agent starts up as root 176
Large historical collections slow monitoring agents 176
Unable to access History Collection Configuration
for any agent .............176
Agent names and icons are displayed incorrectly 177
64 bit monitoring agents are not started ....177
Errors in the configuration xml file ......177
Subnode Limitations for autonomous function . . 178
Binary Path attribute of the Windows OS agent
does not show a value ..........179
Installing pre-v6.2.1 Monitoring Agent for
Windows OS onto a v6.2.1 or later monitoring
server inadvertently unconfigures the monitoring
server ................179
OS agent restarted unexpectedly on heavily loaded
systems ...............179
Calendar entries overlap..........180
Receive an error when deploying an System
Service Monitor agent ..........180
The Agent Service Interface is not globalized . . . 180
Some attribute group names are unintelligible from
the History Collection Configuration window . . 180
History collection fails to display the most recent
24 hours of data ............180
Situations with attributes from more than 1 group
not supported with autonomous agent .....181
Failure when importing situation xml file edited
with WordPad .............181
Printer details of another system are displayed on
the Tivoli Enterprise Portal .........181
CTIRA_MAX_RECONNECT_TRIES environment
variable is now obsolete ..........181
Agent goes offline after removing history path . . 182
Override button is not present for a situation. . . 182
Agent's Management Definition View columns are
not showing data ............182
There is a situation distribution discrepancy if
there is a hub monitoring server outage when one
or more remote monitoring servers remain active . 182
Installing v6.2.2 agent application support on a
monitoring server for a prior release causes agents
to fail ................182
Installing backlevel Windows OS agent on existing
environment causes monitoring server not to start . 182
SNMP trap Sendto fails ..........183
Situation overrides cannot be used to disable
situations on specific systems at specific times . . 183
Situation or calendar name in thresholds.xml file
appears incorrect ............183
BAROC file is missing for IBM Tivoli Monitoring
5.x Endpoint situations ..........183
The target host name, platform, and version
information is not displayed for the deployment
status in the CLI or the workspace ......184
Agent upgrade and restart using non-root ....184
After installing and configuring a monitoring
agent, it fails to start ...........186
situation_fullname slot missing for delete events 186
Logs are using the situation ID string instead of
the display name ............186
If a managed system list is removed for a situation,
the situation stops ............187
Descriptions are not displayed for default
situations...............187
viii IBM Tivoli Monitoring: Troubleshooting Guide
Agent configuration failed on remote deployment
while using single quotes for configuration
properties ..............187
New attributes missing ..........187
Unable to receive summarized data for the last
hour in the Tivoli Enterprise Portal ......188
Summarization for CCC logs is not allowed . . . 188
Receive errors when modifying the JAVA HEAP
SIZE for the Summarization and Pruning Agent . . 188
When associating situations, they fire, but cannot
be viewed ..............188
The Summarization and Pruning agent fails when
processing an index created in a previous release of
the product ..............188
Summarization and Pruning agent schedule not
affected by daylight saving time .......189
Attribute names must be kept under 28 characters
long ................189
Agent deployment operations are not completing
before the TIMEOUT expires ........189
Deploy cannot tell if the installation failed....190
An agent does not display in the portal client or in
the output from the listSystems command....190
One monitoring agent's workspaces are listed
under another agent node on the portal client . . 192
Issues with starting and stopping an agent as a
non-Administrator user ..........193
UNIX-based systems Log agent was deployed,
configured, and started but returns the
KFWITM290E error ...........193
KDY1024E error is displayed when configuring the
run-as user name for an agent ........193
Interface unknown messages in ras1 logs ....193
When upgrading a System Service Monitors agent
from 3.2.1 to 4.0, receive KDY3008E message . . . 194
The Tivoli Data Warehouse fails and you either
lose data or have memory problems......194
Error list appears in warehouse logs ......195
When configuring the Monitoring Agent for Sybase
and the Warehouse Proxy Agent, receive message
to use CandleManage ..........196
listSit command with the type option fails with a
KUIC02001E message on Japanese Systems . . . 196
Creating a situation from a group member does
not copy the distribution list ........196
A changed situation name does not show up . . . 196
New agents do not display in the portal client
Navigator view .............196
An agent displays unavailable in the portal client 196
CTIRA_HOSTNAME has no effect on log file
names ................197
The Summarization and Pruning Agent and the
Warehouse Proxy Agent do not work with DB2 9.1
Fix Pack 2 ..............197
An error of 'can bind a LONG value only for
insert' appears .............197
Errors in either the Warehouse Proxy Agent or
Summarization and Pruning Agent logs ....197
Receive a message saying that the statement
parameter can only be a single select or a single
stored procedure ............197
Custom defined workspace views do not handle
symbol substitution as expected .......197
Unresolved variables in custom queries ....198
A message appears after accepting the license . . 199
Adding agent help files requires a restart of the
Eclipse Help Server and the Tivoli Enterprise
Portal Server .............200
Unable to create historical collection directory for
ud:db2inst1 ..............200
Receive a large amount of data back from the
warehouse for a baseline command ......200
Chapter 11. Command
troubleshooting ..........201
On Solaris 8 operating systems, checkprereq
processes do not complete .........201
Situations deleted from the CLI are still listed on
Tivoli Enterprise Portal Situation editor.....201
The tacmd addBundles command returns an
unexpected KUICAB010E error message ....201
tacmd removeBundles command returns
unnexpected KUICRB010E error message ....201
tacmd executecommand command run against
subnode fails .............202
The krarloff command returns an error message 202
Unexpected KUIC02013E error message ....202
Missing options for login -stdin results in
unexpected behavior ...........203
A system error occurs with the tacmd
editsystemlist -e command .........203
Problem running the tacmd listsystemlist -d
command on Linux systems ........203
Commands with embedded single quotation marks
fail .................204
tacmd exportnavigator -o not behaving correctly 204
TACMD xxxxAction commands fail on Japanese
systems ...............204
Overrides set against an agent cannot be deleted
from the command line ..........204
tacmd listsit -m UX Managed System gives no
result ................204
Receive a busy monitoring server message when
using the getfile, putfile, or executecommand
commands ..............205
Reconfiguring an agent and then getting the
deploy status yields a port number message . . . 205
tacmd getfile or putfile command is failing . . . 205
Temporary files remain when tacmd getfile or
putfile is interrupted ...........205
Receive an OutOfMemory exception when using
the import or export commands .......206
The suggestbaseline or acceptbaseline commands
fail .................206
Problems with Take Action commands and curly
brackets ...............206
When configuring the Monitoring Agent for Sybase
and the Warehouse Proxy Agent, receive message
to use CandleManage ..........206
listSit command with the type option fails with a
KUIC02001E message on Japanese Systems . . . 206
Contents ix
Take Action command names do not accept
non-English characters ..........207
addBundles command times out .......207
createNode command fails .........207
tacmd suggestbaseline minimum, maximum, and
average function values are ignored ......207
tacmd suggestbaseline command receives an error 208
When using the listSystems command, the last two
digits for the version appear as 'XX' ......208
The command tacmd restartAgent fails if the agent
is already stopped ............208
Using the kinconfig command and remotely
starting, stopping or recycling agents fails on
Windows 2000 systems ..........209
You receive a message when using a tacmd
command related to agents .........209
You receive a message when trying to use the
tacmd maintagent command ........209
Endpoint fails to connect to monitoring server
when running createnode from a monitoring server
in a different domain ...........209
Take Action commands do not work if unrequired
values are left blank ...........210
Take Action commands do not display messages
when run from a Navigator Item or from a
workspace view ............210
Corrupted tacmd responses are displayed in the
command-line interface ..........210
The listSystems command consumes high CPU in
enterprise environments ..........211
Improving tacmd command response time when
using VMWare .............211
The addSystem command fails with error message
KUICCR099E .............212
The tacmd getdeploystatus command is not
returning status return codes ........212
tacmd createSit does not send errors if you mistype
the name of an attribute ..........213
wsadmin commands' output indicates the wrong
server name ..............213
When using the viewuser command, you receive a
message that an option is repeating ......213
Commands fail when a situation name consists of
characters ..............213
tacmd addSystem fails if agent already exists. . . 213
Installing an exported agent bundle using install.sh
causes an error .............214
The addbundles command fails .......214
The exportBundles command does not work for
patches ...............214
Chapter 12. Performance Analyzer
troubleshooting ..........215
Enabling logging for the agent .......215
Enabling logging for the monitoring portal . . . 216
Installation and configuration issues......216
Problems after upgrading .........217
Tivoli Performance Analyzer graphical user
interface for Tivoli Enterprise Portal fails when
downloading tasks list ..........218
When tasks are started and when you should see
data in the workspaces ..........218
No data is displayed in the workspaces ....219
The Tivoli Performance Analyzer workspaces are
not available or not displayed ........219
No chart is visible on the Forecast Details
workspace ..............219
The Performance Analyzer Agent Statistics
workspace shows database errors indicating that
some tables or views are missing .......219
Nonlinear tasks take too long to complete ....220
Agent never connects to the monitoring server . . 221
The Tivoli Enterprise Monitoring Server does not
restart after installation of Domain Support . . . 221
Chapter 13. Database troubleshooting 223
Data loss prevention ...........223
Backing up the database for recovery purposes 223
Restoring the original database contents . . . 223
If you modify your password or if it expires . . . 223
DB2 pureScale environment ........224
Receive First Steps error at the end of a DB2
installation ..............225
Windows portal server cannot connect to the
database ...............225
Oracle problem with JDBC drivers prior to 11.1.0.7 226
Database contents are incorrect after installation 226
The error SQL0443N with 'SYSIBM:CLI:-805' occurs
after upgrading to DB2 UDB Version 8.1 Fix Pack
10.................227
Using DB2 v8.1, Warehouse Proxy Agent crashes 227
Using DB2 V9.1 for z/OS, Warehouse Proxy agent
encounters a large number of disconnections . . . 227
Historical data is not warehoused ......228
Historical data for logs is incorrect ......228
Warehouse Proxy Agent or Summarization and
Pruning Agent fails due to DB2 transaction log full. 228
Incorrect data is collected in the warehouse for
filtering if using a wildcard.........228
Too much historical data is collected .....229
Warehouse Proxy agent failed to export data . . . 229
There are ORACLE or DB2 errors in the
khdras1.log file .............229
SQL0552N “ITMUSER” does not have the privilege
to perform operation “CREATE BUFFERPOOL”
SQLSTATE=42502 ............230
Chapter 14. Event synchronization
troubleshooting ..........231
Event synchronization installation and
configuration troubleshooting ........231
Errors occur during installation of IBM Tivoli
Monitoring event synchronization .....231
Netcool/OMNIbus Probe for Tivoli EIF does not
start after configuring the probe to use
monitoring rules ...........231
Netcool/OMNIbus integration troubleshooting . . 232
Log files for Netcool/OMNIbus Event
Synchronization ...........232
x IBM Tivoli Monitoring: Troubleshooting Guide
Unable to send situation events from the hub
monitoring server to Netcool/OMNIbus . . . 233
Event status updates in Netcool/OMNIbus are
not forwarded to Tivoli Monitoring .....235
Monitoring events in Netcool/OMNIbus do not
have expected values for the Summary attribute
or other attributes set by the IBM Tivoli
Monitoring probe rules .........239
After an event is cleared in Netcool/OMNIbus,
the event's severity is changed back to its
original severity ...........241
Tivoli Enterprise Console integration
troubleshooting.............242
General event synchronization troubleshooting . . 242
Editing the default destination server
information from the command line does not
work ...............242
tacmd refreshTECinfo -t all shows no results on
console ..............243
Changing the TCP/IP timeout setting on your
event server .............243
Cognos reports are displayed as a blank page . . 248
You are missing drivers after the Tivoli Common
Reporting installation...........248
Documentation for the base agents ......249
The report fails to generate because the SQL query
was not valid .............249
Message “SQL Statement does not return a
ResultSet object” displayed .........250
Your report fails to generate with unexpected error
messages displayed ...........250
The generated report displays the message “SQL
Error”................251
The report fails with a SQLSTATE:22003 arithmetic
overflow error .............251
No data is plotted in graph, or some columns in
the table are blank............252
The generated report displays the message “The
requested data is not available” .......253
You receive the message “serverName is unknown
host” ................253
You receive the message “Empty Data Set” . . . 254
Chapter 15. Tivoli Common Reporting
troubleshooting ..........245
Locations of log files ...........245
Java out of memory error after installation . . . 245
Running OS Cognos Reports with Tivoli Common
Reporting 2.1.1 on 64-bit AIX 6.1 results in error
DPR-ERR-2056 .............246
Displaying data for Situations History report
results in error .............247
Date and time format in IBM Tivoli Monitoring OS
Agents reports not localized ........247
Prompted to select report type when installing
reports with CLI ............247
Cognos Query Studio displays Japanese text within
the Thai web browser ..........247
The prompt page of a Cognos report within the
Tivoli Common Reporting tool displays strings that
are not translated ............247
The generated report displays an incorrect date
and time format ............247
The generated report does not display report
legend................248
Receive a 'statement is too long' error message
when running a report ..........248
Running COGNOS reports against a DB2 database
is slow ...............248
Chapter 16. Tivoli Audit Facility
troubleshooting ..........255
Audit Log workspace shows only 100 of the most
recent audit records ...........255
Audit Log workspace does not display records
before the latest component startup ......255
Appendix. IBM Tivoli Monitoring
processes .............257
Documentation library .......259
IBM Tivoli Monitoring library ........259
Documentation for the base agents .....260
Related publications ...........261
Other sources of documentation .......261
Support information ........263
Notices ..............267
Glossary .............271
Index ...............285
Contents xi
xii IBM Tivoli Monitoring: Troubleshooting Guide
Tables
1. Location of log files for the IBM Tivoli
Monitoring components. ........36
2. Installation log files ..........39
3. Upgrading from Tivoli Distributed
MonitoringTivoli log file ........41
4. Setting the trace option for the Tivoli
Monitoring upgrade toolkit .......50
5. General frequently asked questions ....65
6. Windows installation frequently asked
questions .............65
7. Frequently asked questions for UNIX-based
systems installation ..........66
8. lcfd log file.............80
9. Removing a failed installation on Windows 109
10. Installation logs ...........113
11. Uninstall OS command ........114
12. Cannot log in to the Tivoli Enterprise Portal
Server ..............115
13. Cannot connect to Tivoli Enterprise Portal
Server ..............118
14. The Tivoli Enterprise Portal Server does not
start after installation .........121
15. Control interface publishing ......122
16. Resolutions for agent deployment operations
that TIMEOUT ...........189
17. createNode command fails .......207
18. Utilities for backing up the database ....223
19. Resolving problems sending events to
Netcool/OMNIbus..........233
20. Event status updates in Netcool/OMNIbus
are not forwarded to Tivoli Monitoring . . . 236
21. Monitoring events in Netcool/OMNIbus do
not have expected values .......239
22. IBM Tivoli Monitoring processes by operating
system ..............257
© Copyright IBM Corp. 2005, 2012 xiii
xiv IBM Tivoli Monitoring: Troubleshooting Guide
About this information
This guide provides problem determination and resolution information for the
issues most commonly encountered with IBM®Tivoli®Monitoring components and
related products.
You can use this guide in conjunction with the other books for your product.
© Copyright IBM Corp. 2005, 2012 xv
xvi IBM Tivoli Monitoring: Troubleshooting Guide
Chapter 1. Introduction to troubleshooting
To troubleshoot a problem, you typically start with a symptom or set of symptoms
and trace back to the cause.
Troubleshooting is not the same as problem solving, although during the process of
troubleshooting, you can obtain enough information to solve a problem, such as
with end-user errors, application programming errors, and system programming
errors.
You might not always be able to solve a problem yourself after determining its
cause. For example, a performance problem might be caused by a limitation of
your hardware. If you are unable to solve a problem on your own, contact IBM
Software Support for a solution. See Chapter 2, “Logs and data collection for
troubleshooting,” on page 5 for information on the types of data to collect before
contacting Support.
Sources of troubleshooting information
The primary troubleshooting feature is logging. Logging refers to the text messages
and trace data generated by the software. Messages and trace data are sent to an
output destination, such as a console screen or a file.
Typically, text messages relay information about the state and performance of a
system or application. Messages also alert the system administrator to exceptional
conditions when they occur. Consult the explanation and operator response
associated with the displayed messages to determine the cause of the failure. See
the document IBM Tivoli Monitoring Messages for message information.
Trace data capture transient information about the current operating environment
when a component or application fails to operate as designed. IBM Software
Support personnel use the captured trace information to determine the source of
an error or unexpected condition. See “Trace logging” on page 35 for more
information about tracing.
Problem classification
The first task in troubleshooting is to determine the origin of the problem, or
which component or function is experiencing a problem. To assist you in
determining the origin of the problem, collect documentation at the time of the
error.
You might experience problems with IBM Tivoli Monitoring in the following areas:
v Installation
v Upgrading
v Configuration
v Connectivity
v Tivoli Enterprise Portal
v Tivoli Enterprise Portal Server
v Tivoli Enterprise Monitoring Server
v Tivoli Enterprise Monitoring Agent deployment
© Copyright IBM Corp. 2005, 2012 1
v Databases
v Tivoli Data Warehouse
v Universal Agent
v IBM Tivoli Enterprise Console
Viewing the IBM Support Portal
The IBM Support Portal is a unified, customizable view of all technical support
tools and information for your IBM systems, software, and services. It brings all
the support resources available for IBM hardware and software offerings together
in one place.
About this task
Perform the following actions to access technotes for this product:
Procedure
1. Open the http://ibm.com website and select Support & downloads >
Technical support. You can also launch an IBM support website, such as
http://www.ibm.com/support/us.
2. Enter your IBM user ID when prompted or, in the Quick start page or Support
home, click Sign in to sign in with your IBM user ID or to register if you have
not yet registered.
3. Enter a keyword or keywords for the information you want to find in the
Quick Find or Search support fields. You can also browse through the other
Support tabs.
®
Subscribing to IBM support notifications
You can subscribe to e-mail notification about product tips and newly published
fixes through the Support portal.
In the Support portal, you can specify the products for which you want to receive
notifications; choose from flashes, downloads, and technotes; and set up to receive
email updates.
About this task
Perform the following actions to subscribe to Support emails.
Procedure
1. Open the http://ibm.com website and select Support & downloads >
Technical support. You can also launch an IBM support website, such as
http://www.ibm.com/support/us.
2. In the Quick start page or Support home, click Sign in to sign in or to register
if you have not yet registered.
3. In the Notifications area of Support home, click Manage all my subscriptions.
4. In the Subscribe and My defaults tabs, select a product family and continue
setting your preferences to specify the information you want in your emails.
5. If you have not yet added an email address to your profile, click My IBM >
Profile > Edit and add it to your personal information.
2 IBM Tivoli Monitoring: Troubleshooting Guide
Results
You begin receiving “IBM My notifications” emails about the products you have
selected and at the interval you specified.
Chapter 1. Introduction to troubleshooting 3
4 IBM Tivoli Monitoring: Troubleshooting Guide
Chapter 2. Logs and data collection for troubleshooting
If you have a problem that you are unable to solve using the information in this
guide or on the IBM Support Portal, gather the information that relates to the
problem and contact IBM Software Support for further assistance.
Appropriate IBM Tivoli Monitoring RAS1 trace output
IBM Software Support uses the information captured by trace logs to trace a
problem to its source or to determine why an error occurred.
The reliability, availability, and serviceability (RAS) trace logs are available on the
Tivoli Enterprise Monitoring Server, the Tivoli Enterprise Portal Server, and the
monitoring agent. By default, the logs are stored in the installation path for IBM
Tivoli Monitoring.
The following links to sections in this document supply more information on these
files:
v For information on where they are stored, see “Log file locations” on page 35
v For information on setting the trace option for an IBM Tivoli Monitoring
component, see “Setting traces” on page 43.
v For information on dynamically setting the trace settings, see “Dynamically
modify trace settings for an IBM Tivoli Monitoring component” on page 53.
v For information on reading RAS1 logs, see “Reading RAS1 logs” on page 42.
v For information on the ras1log tool, see “ras1log tool” on page 60.
Running snapcore to collect information
Use the snapcore command for collecting information for use in identifying and
resolving problems with an application.
The snapcore command gathers a core file, program, and libraries used by the
program and compresses the information into a pax file. The file can then be
downloaded to disk or tape, or transmitted to a remote system.
About this task
Take the following steps to run the snapcore command and collect information
you might need to debug and analyze the problem:
Procedure
1. Change to the directory where the core dump file is located:
#ls-l
total 84176
-rw-r--r-- 1 root system 2704 Feb 21 09:52 core.18048.01084144
2. Run the snapcore command to collect all needed files:
# snapcore -d /tmp/myDir core.18048.01084144
The snapcore command gathers all information and creates a new compressed
pax archive in the/tmp/myDir directory. If you do not specify a special directory
© Copyright IBM Corp. 2005, 2012 5
using the -d flag, the archive will be stored in the/tmp/snapcore directory. The
new archive file will be named as snapcore_$pid.pax.Z:
# ls -l /tmp/myDir
total 5504
-rw-r--r-- 1 root system 2815081 Feb 21 09:56 snapcore_20576.pax.Z
3. To check the content of the pax archive, run the uncompress command:
# uncompress -c snapcore_20576.pax.Z | pax core.18048.01084144
README
lslpp.out
errpt.out
vi
./usr/lib/libc.a
./usr/lib/libcrypt.a
./usr/lib/libcurses.a
./usr/lib/nls/loc/en_US
./usr/lib/libi18n.a
./usr/lib/libiconv.
Locating the core file
You can read the core file for information related to system stops on UNIX-based
systems. Use the errpt -a command to get a summary of the most recent system
stoppages and the location of the core file.
If the system stops on UNIX-based systems, collect the core file from the directory
that stores the binary file, to which the process belongs. For example, if the failing
process is the Tivoli Enterprise Portal Server server process, KfwServices, the core
is created in the /opt/IBM/ITM/archtype/cq/bin/ directory.
Procedure
To retrieve information on where the core file is created, enter the errpt -a
command.
Results
A summary of information is displayed about the most recent crashes and also the
location of the core file:
------------LABEL: CORE_DUMP
IDENTIFIER: A63BEB70
Date/Time: Tue Jun 30 15:38:47 DFT 2009
Sequence Number: 1229
Machine Id: 0056536D4C00
Node Id: nc114062
Class: S
Type: PERM
Resource Name: SYSPROC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
Recommended Actions
CORRECT THEN RETRY
6 IBM Tivoli Monitoring: Troubleshooting Guide
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
USER’S PROCESS ID:
FILE SYSTEM SERIAL NUMBER
INODE NUMBER
PROCESSOR ID
CORE FILE NAME
/opt/IBM/ITM/aix533/cq/bin/core
PROGRAM NAME
KfwServices
STACK EXECUTION DISABLED
---------------
11
32248
10
655367
0
Getting Dr. Watson dumps and logs
Use the Dr. Watson debugger to get the information needed by IBM Support to
diagnose problems on Windows systems.
If you encounter errors or failures on your Windows system, collect the
drwtsn32.log and user.dmp files if they are available. The drwtsn32.log and
user.dmp files are located in: \Documents and Settings\All Users\Documents\
DrWatson.
About this task
Take the following steps to enable Dr. Watson and configure it to create a detailed
dump file:
Procedure
1. To enable Dr. Watson as the default debugger, at the command prompt, enter
2. To open the Dr. Watson configuration dialog, at the command prompt, enter the
3. Set the following fields:
KpcCMA.RAS files
the following command: drwtsn32 –i.
following command: drwtsn32
a. Set the Crash dump Type to FULL.
b. Clear the Dump Symbol Table check box.
c. Enable the Dump all Thread Contexts check box.
d. Enable the Create Crash Dump File check box.
IBM Tivoli Monitoring on Windows systems has ( where pc is the two-character
product or component code) KpcCMA.RAS files in the c:\windows\system32 directory
to collect information about monitoring process failures.
Chapter 2. Logs and data collection for troubleshooting 7
For example, KNTCMA.RAS is the Monitoring Agent for Windows OS the reliability,
availability, and serviceability file. These files contain system dump information
similar to the drWatson.log, but are generated by the IBM Tivoli Monitoring
infrastructure.
Sources of other important information
You can collect important information from log files, such as trace or message logs
that report system failures. Also, application information provides details on the
application that is being monitored, and you can obtain information from messages
or information on screen.
The following sources provide additional information to aid in troubleshooting:
v Monitored application file as specified on the SOURCE FILE statement, if
applicable.
v Description of the operation scenario that led to the problem.
v Incorrect output, such as Tivoli Enterprise Portal screen captures or a description
of what you observed, if applicable.
v Log files collected from failing systems. You can collect all logs or logs of a
certain type such as, RAS trace logs or message logs.
v Messages and other information displayed on the screen.
v Information about the application that you are monitoring, such as DB2 or SAP.
This information includes the version number, patch level, and a sample
application data file if you are monitoring a file.
v Operating system version number and patch level.
v Version number of the following members of the monitoring environment:
– IBM Tivoli Monitoring and the patch level, if available.
– Monitoring Agent version number .
– Tivoli Enterprise Portal (Select Help > About Tivoli Enterprise Portal)
Note: The version number of the Tivoli Enterprise Portal and theTivoli
Enterprise Portal Server must always be synchronized.
8 IBM Tivoli Monitoring: Troubleshooting Guide
Chapter 3. Common problem solving
Customers using IBM Tivoli Monitoring products or the components of Tivoli
Management Services can encounter problems such as missing workspaces or
historical data, or a reflex automation script that does not run when it should. In
many cases you can recover from these problems by following a few steps.
Note: Use the trace settings indicated in these troubleshooting instructions only
while you are trying to diagnose a specific problem. To avoid generating excessive
trace data, go back to the default trace settings as soon as the problem is solved.
About the tools
You can access several troubleshooting tools, such as the Log analyzer or pdcollect
tool to help you troubleshoot your IBM Tivoli Monitoring product or the
components of Tivoli Management Services.
ITMSuper Tools
The ITMSUPER Tools give you information about the health of your
managed systems, situations, and environment configuration. You can find
the tools by searching for “ITMSUPER” in the IBM Integrated Service
Management Library (http://www.ibm.com/software/brandcatalog/
ismlibrary).
pdcollect tool
The pdcollect tool collects the most commonly used information from a
system. It gathers log files, configuration information, version information,
and other data. You can also use this tool to manage the size of trace data
repositories. For more information see “pdcollect tool” on page 59.
IBM Support Assistant
The IBM Support Assistant is a free, stand-alone application that you can
install on any workstation. Then, you can enhance the application by
installing product-specific plug-in modules for the IBM products you use.
For more information see “Support information” on page 263.
I am trying to find out what software is supported
Use resources in the IBM Tivoli Monitoring Installation and Setup Guide and the IBM
website to determine the software that is supported. This enables you to find
platform or database information for specific products.
The following resources are available to determine the software that is supported:
v For specific information about the supported software for IBM Tivoli
Monitoring, see “Hardware and software requirements” in the IBM Tivoli
Monitoring Installation and Setup Guide
v For platform and database support information for most Tivoli products, consult
the matrix at Tivoli Supported Platforms (http://www-306.ibm.com/software/
sysmgmt/products/support/Tivoli_Supported_Platforms.html)
© Copyright IBM Corp. 2005, 2012 9
Workspaces are missing or views are empty
You can encounter a problem that workspaces are missing or views are empty. For
example, you may have workspaces that return no data.
Symptoms of the problem:
v The workspaces return no data.
v There are no child Navigator items under the agent node in the Navigator view.
See “Resolving application support problems” on page 11.
v The Navigator items are labeled with internal names, such as Knt:KNT1076
instead of the correct names (such as Disk). See “Resolving application support
problems” on page 11.
v You receive message KFWITM217E: Request error: SQL1_CreateRequest failed,
rc=209. See “Resolving application support problems” on page 11.
v You receive message KFWITM220E: Request failed during execution.See
“Resolving monitoring agent problems” on page 14.
For more information on workspaces that relate to historical data, see “Historical
data is missing or incorrect” on page 20.
To diagnose the problem that workspaces are missing or empty, see “Diagnosing
that workspaces are missing or empty.”
Diagnosing that workspaces are missing or empty
You can diagnose that workspaces are missing or empty by verifying that the
monitoring agent has been started and that the configuration is correct.
You can also check that application support has been added.
About this task
To diagnose that workspaces are missing or empty, perform the following steps:
Procedure
Preliminary diagnostics
1. Refresh the Navigator by clicking View > Refresh.
2. Verify that the monitoring agent has been started. Restart if necessary. In the
Tivoli Enterprise Portal, right-click the Navigator item of the monitoring agent
and click Start or Restart
3. Verify that the monitoring agent configuration is correct.
4. If your data is missing in an Oracle Agent workspace, see “Resolving Oracle
DB Agent problems - diagnostic actions” on page 32. Similar problems might
exist for other monitoring agents.
5. Check that application support has been added. See “Resolving application
support problems” on page 11.
10 IBM Tivoli Monitoring: Troubleshooting Guide
What to do next
For more information on actions that relate to these diagnostics, see the problem
resolution tasks.
Resolving application support problems
Application support problems are caused by a lack of application support or an
application support level mismatch among the components: monitoring server,
portal server, desktop and Java Web Start clients, and monitoring agents. Check the
installed level of application support or run the ITMSUPER Tivoli Enterprise
Monitoring Server analysis tool (or both) to get more information.
Before you begin
Complete one or both of the following tasks to ensure that this is an application
support problem:
v “Diagnosing that workspaces are missing or empty” on page 10
v “Diagnosing that a situation does not raise when expected” on page 25
About this task
To resolve application support problems, you perform diagnostic and corrective
actions. These actions include checking application support on the servers and
client and running the Tivoli Enterprise Monitoring Server tool to ensure that
application support is installed consistently in your environment.
Procedure
Diagnostic and corrective actions
1. Check application support on the monitoring server, portal server, and portal
client:
v
v
v
2. You can also run the Tivoli Enterprise Monitoring Server analysis tool provided
by ITMSUPER against the hub monitoring server to ensure that application
support is installed consistently throughout your environment.
3. If application support is missing, add the appropriate application support to
the portal server and monitoring server for the monitoring agents.
4. If the desktop client or Java Web Start client is being used, ensure application
support is installed on the portal client.
Run the kincinfo.exe -i command in the %CANDLE_HOME\InstallITM
directory to show what is installed.
Run the ./cinfo –i command in the $CANDLEHOME/bin directory to
show what is installed.
(monitoring server) Look in the &rhilev.&rte.RKANDATV data set where
&rhilev is the high-level qualifier and &rte is the mid-level qualifier of the
libraries for the runtime environment where the monitoring server is
configured for files named KppCATand KppATR where pp is the two-character
product or component code.
What to do next
For more information and instructions on installing application support see
“Configuring application support for nonbase monitoring agents” in the IBM Tivoli
Monitoring Installation and Setup Guide. For instructions on installing application
Chapter 3. Common problem solving 11
support on a z/OS monitoring server, see “Adding application support to a
monitoring server on z/OS” in Configuring IBM Tivoli Enterprise Monitoring Server
on z/OS.
Resolving monitoring server problems
Monitoring server problems are caused by a monitoring server that is not started
or connectivity that is lost either between servers or between servers and agents.
You can restart the Tivoli Enterprise Monitoring Server, and you can also run the
ITMSUPER Topology tool to get more information.
About this task
To resolve monitoring server problems, you perform diagnostic and corrective
actions. These actions include running tools, such as the Topology or Connectivity
tool and correcting communication failures in logs.
Procedure
Diagnostic and corrective actions
1. If you are an administrator, restart the monitoring server. Otherwise, notify an
administrator and wait for the monitoring server to be restarted.
2. Running the following ITMSUPER tools might also provide more information:
v Topology tool
v Connectivity tool
v Tivoli Enterprise Monitoring Server analysis tool
v Tivoli Enterprise Portal Server
3. Check the portal server logs for messages indicating communication failures to
the monitoring server.
4. Check the monitoring server logs for messages indicating communication
failures to the remote monitoring servers or to monitoring agents.
5. Correct the communication failures indicated in the logs.
Resolving monitoring agent problems
If the monitoring agent is running but data is not being returned or if you receive
an error message from an agent log, such as Endpoint unresponsive, verify that the
agent is connected and online. You can also verify that application support has
been installed correctly.
About this task
To resolve monitoring agent problems, you perform diagnostic and corrective
actions. These actions include verifying that the agent is running and that
application support has been installed correctly. For information on monitoring
agents on z/OS
Procedure
Diagnostic and corrective actions
1. Verify that the agent is connected. Check the monitoring server log for
messages similar to Remote node <SYS:MQIRA> is ON-LINE.
2. If the agent is online, check to see whether subnodes are online in the agent
log. For example: KMQMI171I Node JSG1:SYS:MQESA is online.
12 IBM Tivoli Monitoring: Troubleshooting Guide
®
see each product's Problem Determination Guide.
Loading...