Cisco Customer Response User Manual

Download

Cisco Customer Response Solutions Servicing and Troubleshooting Guide

Cisco Unified Contact Center Express, Cisco Unified IP IVR, and Cisco Unified Queue Manager, Release 5.0(1)

June 2007

Americas Headquarters

Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000

800 553-NETS (6387)

Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

CCVP, the Cisco logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco

Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo,

Cisco

IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco

All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0705R)

Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Systems, Inc. and/or its affiliates in the United States and certain other countries.

IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE

Preface i

Purpose i

Audience i

Organization i

Purpose

Audience

Preface

The Cisco CRS Servicing and Troubleshooting Guideprovides instructions for using the CRS Serviceability tools and helps you resolve any problems you might experience with the CRS system.

The Cisco CRS Servicing and Troubleshooting Guide is written for administrators and others who are responsible for managing and troubleshooting the Cisco CRS system.

Organization

Part/Chapter Title Description

Part 1 Serviceablity

Chapter 1 About Serviceability Provides an overview of the Cisco CRS serviceability

Chapter 2 Simple Network

Chapter 3 Alarm Service Describes how to configure the Cisco CRS Alarm Service

Chapter 4 Trace Describes how to configure the component trace file, set

Chapter 5 Cisco Discovery

Chapter 6 Cisco Support Tools Provides an overview of Cisco Support Tools that are

Management Protocol Support

Protocol Support

support and the support provided when an expansion server is used.

Describes how you can use SNMP to monitor and manage your CRS system.

and view alarm messages.

trace level options, and collect, view, and interpret trace log files.

Describes how to use the CDP Driver, view the CDP Driver properties, and locate the CDP Driver and interface files.

supported by Cisco CRS.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Part/Chapter Title Description

Part 2 Troubleshooting

Chapter 7 Diagnosing and

Chapter 8 Troubleshooting Tips Provides troubleshooting tips for the various elements of

Conventions

This manual uses the following conventions.

Convention Description

boldface font Boldface font is used to indicate commands, such as user entries,

keys, buttons, and folder and submenu names. For example:

• Choose Edit > Find.

• Click Finish.

italicfont Italic font is used to indicate the following:

• To introduce a new term. Example: A skill group is a

collection of agents who share similar skills.

• For emphasis. Example: Do not use the numerical naming

convention.

• A syntax value that the user must replace. Example: IF

(condition, true-value, false-value)

• A book title. Example: See the Cisco CRS Installation Guide.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

Convention Description

window font Window font, such as Courier, is used for the following:

< > Angle brackets are used to indicate the following:

Obtaining Documentation

Cisco documentation and additional literature are available on Cisco.com. This section explains the product documentation resources that Cisco offers.

• Text as it appears in code or that the window displays.

Example: <html><title>Cisco Systems,Inc. </title></html>

• For arguments where the context does not allow italic, such as

ASCII output.

• A character string that the user enters but that does not appear

on the window such as a password.

Cisco.com

You can access the most current Cisco documentation at this URL:

http://www.cisco.com/techsupport

You can access the Cisco website at this URL:

http://www.cisco.com

You can access international Cisco websites at this URL:

http://www.cisco.com/public/countries_languages.shtml

Product Documentation DVD

The Product Documentation DVD is a library of technical product documentation on a portable medium. The DVD enables you to access installation, configuration, and command guides for Cisco hardware and software products. With the DVD, you have access to the HTML documentation and some of the PDF

files found on the Cisco website at this URL:

http://www.cisco.com/univercd/home/home.htm

The Product Documentation DVD is created and released regularly. DVDs are available singly or by subscription. Registered Cisco.com users can order a Product Documentation DVD (product number DOC-DOCDVD= or DOC-DOCDVD=SUB) from Cisco Store at this URL:

Marketplace at the Product Documentation

http://www.cisco.com/go/marketplace/docstore

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

iii

Ordering Documentation

You must be a registered Cisco.com user to access Cisco Marketplace. Registered users may order Cisco documentation at the Product Documentation Store at this URL:

http://www.cisco.com/go/marketplace/docstore

If you do not have a user ID or password, you can register at this URL:

http://tools.cisco.com/RPF/register/register.do

Documentation Feedback

You can provide feedback about Cisco technical documentation on the Cisco Support site area by entering your comments in the feedback form available in every online document.

Cisco Product Security Overview

Preface

Cisco provides a free online Security Vulnerability Policy portal at this URL:

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

From this site, you will find information about how to do the following:

• Report security vulnerabilities in Cisco products

• Obtain assistance with security incidents that involve Cisco products

• Register to receive security information from Cisco

A current list of security advisories, security notices, and security responses for Cisco products is available at this URL:

http://www.cisco.com/go/psirt

To see security advisories, security notices, and security responses as they are updated in real time, you can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS) feed. Information about how to subscribe to the PSIRT RSS feed is found at this URL:

http://www.cisco.com/en/US/products/products_psirt_rss_feed.html

Reporting Security Problems in Cisco Products

Cisco is committed to delivering secure products. We test our products internally before we release them, and we strive to correct all vulnerabilities quickly. If you think that you have identified a vulnerability in a Cisco product, contact PSIRT:

• For emergencies only — security-alert@cisco.com

An emergency is either a condition in which a system is under active attack or a condition for which a severe and urgent security vulnerability should be reported. All other conditions are considered nonemergencies.

• For nonemergencies — psirt@cisco.com

In an emergency, you can also reach PSIRT by telephone:

• 1 877 228-7302

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

• 1 408 525-6532

Tip We encourage you to use Pretty Good Privacy (PGP) or a compatible product (for example, GnuPG) to

encrypt any sensitive information that you send to Cisco. PSIRT can wo rk with information that has been encrypted with PGP versions

Never use a revoked encryption key or an expired encryption key. The correct public key to use in your correspondence with PSIRT is the one linked in the Contact Summary section of the Security Vulnerability Policy page at this

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

The link on this page has the current PGP key ID in use.

If you do not have or use PGP, contact PSIRT to find other means of encrypting the data before sending any sensitive material.

2.x through 9.x.

URL:

Product Alerts and Field Notices

Modifications to or updates about Cisco products are announced in Cisco Product Alerts and Cisco Field Notices. You can receive these announcements by using the Product Alert Tool on Cisco.com. This tool enables you to create a profile and choose those products for which you want to receive information.

To access the Product Alert Tool, you must be a registered Cisco.com user. Registered users can access the tool at this URL:

http://tools.cisco.com/Support/PAT/do/ViewMyProfiles.do?local=en

To register as a Cisco.com user, go to this URL:

http://tools.cisco.com/RPF/register/register.do

Obtaining Technical Assistance

Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco

Support website on Cisco.com features extensive online support resources. In addition, if you

have

a valid Cisco service contract, Cisco Technical Assistance Center (TAC) engineers provide

telephone support. If you do not have a valid Cisco service contract, contact your reseller.

Cisco Support Website

The Cisco Support website provides online documents and tools for troubleshooting and resolving technical issues with Cisco products and technologies. The website is available 24 this

URL:

http://www.cisco.com/en/US/support/index.html

Access to all tools on the Cisco Support website requires a Cisco.com user ID and password. If you have a valid service contract but do not have a user ID or password, you can register at this URL:

http://tools.cisco.com/RPF/register/register.do

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

hours a day at

Preface

Note Before you submit a request for service online or by phone, use the Cisco Product Identification Tool

to locate your product serial number. You can access this tool from the Cisco Support website by

clicking the Get Tools & Resources link, clicking the All Tools (A-Z) tab, and then choosing

Cisco

Product Identification Tool from the alphabetical list. This tool offers three search options:

product ID or model name; by tree view; or, for certain products, by copying and pasting show command output. Search results show an illustration of your product with the serial number label location highlighted. Locate the serial number label on your product and record the information before placing a service call.

Tip Displaying and Searching on Cisco.com

If you suspect that the browser is not refreshing a web page, force the browser to update the web page by holding down the Ctrl key while pressing F5.

To find technical information, narrow your search to look in technical documentation, not the entire

Cisco.com website. After using the Search box on the Cisco.com home page, click the

Advanced Tec hn ic al

To provide feedback about the Cisco.com website or a particular technical document, click

Contacts & Feedback at the top of any Cisco.com web page.

Search link next to the Search box on the resulting page and then click the

Support & Documentation radio button.

Submitting a Service Request

Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3 and S4 service requests are those in which your network is minimally impaired or for which you require product information.) After you describe your situation, the TAC recommended solutions. If your issue is not resolved using the recommended resources, your service request is assigned to a Cisco engineer. The TAC Service Request Tool is located at this URL:

http://www.cisco.com/techsupport/servicerequest

For S1 or S2 service requests, or if you do not have Internet access, contact the Cisco TAC by telephone. (S1 or S2 service requests are those in which your production network is down or severely degraded.) Cisco engineers are assigned immediately to S1 and S2 service requests to help keep your business operations running smoothly.

To open a service request by telephone, use one of the following numbers:

Asia-Pacific: +61 2 8446 7411 Australia: 1 800 805 227 EMEA: +32 2 704 55 55 USA: 1 800 553 2447

For a complete list of Cisco TAC contacts, go to this URL:

http://www.cisco.com/techsupport/contacts

Service Request Tool provides

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

Definitions of Service Request Severity

To ensure that all service requests are reported in a standard format, Cisco has established severity definitions.

Severity 1 (S1)—An existing network is “down” or there is a critical impact to your business operations. You and Cisco will commit all necessary resources around the clock to resolve the situation.

Severity 2 (S2)—Operation of an existing network is severely degraded, or significant aspects of your business operations are negatively affected by inadequate performance of Cisco products. You and Cisco will commit full-time resources during normal business hours to resolve the situation.

Severity 3 (S3)—Operational performance of the network is impaired while most business operations remain functional. You and Cisco will commit resources during normal business hours to restore service to satisfactory levels.

Severity 4 (S4)—You require information or assistance with Cisco product capabilities, installation, or configuration. There is little or no effect on your business operations.

Obtaining Additional Publications and Information

Information about Cisco products, technologies, and network solutions is available from various online and printed sources.

• The Cisco Online Subscription Center is the website where you can sign up for a variety of Cisco

e-mail newsletters and other communications. Create a profile and then select the subscriptions that you would like to receive. To visit the Cisco

http://www.cisco.com/offer/subscribe

• The Cisco Product Quick Reference Guide is a handy, compact reference tool that includes brief

product overviews, key features, sample part numbers, and abbreviated technical specifications for many Cisco the latest Cisco channel product offerings. To order and find out more about the Cisco Reference Guide, go to this URL:

http://www.cisco.com/go/guide

• Cisco Marketplace provides a variety of Cisco books, reference guides, documentation, and logo

merchandise. Visit Cisco

http://www.cisco.com/go/marketplace/

• Cisco Press publishes a wide range of general networking, training, and certification titles. Both new

and experienced users will benefit from these publications. For current Cisco information, go to Cisco

http://www.ciscopress.com

• Internet Protocol Journal is a quarterly journal published by Cisco for engineering professionals

involved in designing, developing, and operating public and private internets and intranets. You can access the Internet Protocol Journal at this URL:

products that are sold through channel partners. It is updated twice a year and includes

Marketplace, the company store, at this URL:

Press at this URL:

Online Subscription Center, go to this URL:

Product Quick

Press titles and other

http://www.cisco.com/ipj

• Networking products offered by Cisco, as well as customer support services, can be obtained at

this

URL:

http://www.cisco.com/en/US/products/index.html

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

vii

Preface

• Networking Professionals Connection is an interactive website where networking professionals

share questions, suggestions, and information about networking products and technologies with Cisco experts and other networking professionals. Join a discussion at this URL:

http://www.cisco.com/discuss/networking

• “What’s New in Cisco Documentation” is an online publication that provides information about the

latest documentation releases for Cisco products. Updated monthly, this online publication is organized by product category to direct you quickly to the documentation for your products. You can view the latest release of “What’s New in Cisco

http://www.cisco.com/univercd/cc/td/doc/abtunicd/136957.htm

• World-class networking training is available from Cisco. You can view current offerings at

this

URL:

http://www.cisco.com/en/US/learning/index.html

Documentation” at this URL:

viii

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

ART

Serviceability

CHA PTER

About Serviceability

Cisco Customer Response Solutions (CRS) Serviceability enables remote network management support for the Cisco CRS system. Serviceability enables this support through CiscoWorks and through any third-party network management system (NMS) that uses standard protocols. These protocols include Syslog, Simple Network Management Protocol (SNMP), XML, and HTTP.

This section contains the following topics:

• About CRS Serviceability, page 1-1

• Serviceability Support, page 1-1

• CiscoWorks Support, page 1-2

• Syslog Support, page 1-3

• Remote Serviceability, page 1-3

• Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support, page 1-4

About CRS Serviceability

Serviceability Support

Serviceability allows you to monitor and discover the status of the installed components of your Cisco CRS system, its subsystems, and its services from any NMS. You can use the information that you obtain through serviceability to troubleshoot system problems. (For additional troubleshooting information, refer to Part 2 of this guide.)

Serviceability support includes:

• SNMP Support—Provides integration with CiscoWorks or another SNMP-based network

management system (NMS). SNMP agents provide monitoring of network devices through MIBs (Management Information Bases). For more information, see

Simple Network Management Protocol Support, page 2-1.

• SNMP Traps—Provides notification messages of high-severity Cisco CRS Engine errors. For more

information, see

Simple Network Management Protocol Support, page 2-1

CiscoWorks Support, page 1-2 and

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

1-1

CiscoWorks Support

Chapter 1 About Serviceability

• Alarms—Use Alarms to obtain the run-time status and state of the Cisco CRS system and to take

corrective action to fix detected problems. You can forward alarms to a Syslog server, to an SNMP trap subagent, or to a Windows Event Log. For more information, see

• Trace—Provides specific, detailed Cisco CRS information for troubleshooting system problems.

You can also send alarms to a trace file for further analysis and you can specify what level of event information is sent to the trace file. For more information, see

• Cisco Discovery Protocol (CDP) Support—Sends messages containing system information to a

designated multicast address. For more information, see

Trace, page 4-1.

Cisco Discovery Protocol Support,

page 5-1.

• Syslog Support—Sends common event logging messages in standard Syslog format to any

third-party Syslog server. For more information, see

• Cisco Support Tools—The Node Agent utility of Cisco Support Tools helps you collect log

Alarm Service, page 3-1.

information and troubleshoot Cisco CRS servers. For more information, see

page 6-1.

You can obtain additional system troubleshooting information using the following tools:

• CiscoWorks—Provides a suite of web-based applications for managing Cisco devices. For more

information, see

• Third-Party Network Management Systems—Provide Simple Network Management Protocol-based

CiscoWorks Support, page 1-2.

browser, Syslog support, and other system management tools.

Alarm Service, page 3-1.

Cisco Support Tools,

• Microsoft Windows 2003 Performance Monitoring—Allows you to monitor the performance of the

Cisco CRS system. For more information, refer to your Microsoft Windows documentation.

• Microsoft Windows 2003 Terminal Service—Provides remote systems with access to

Windows-based applications through terminal emulation. Windows 2003 Server Terminal Services are integrated with the Windows 2003 operating system. For more information, refer to your Microsoft Windows documentation.

CiscoWorks Support

CiscoWorks, available as a separate package, provides a suite of web-based applications for managing Cisco devices. It is the network management system (NMS) of choice for the Cisco CRS system and for other Cisco devices.

The Cisco CRS system integrates with these CiscoWorks applications:

• Cisco Unified Operations Manager (Operations Manager)—Operations Manager tracks the health

of Cisco Unified Communications environments by proactively monitoring Cisco voice elements in the network to alert operations personnel to potential problems and to help minimize Unified Communications service downtime.

• Resource Management Essentials (RME)—Providestools for collecting Syslog messages from

multiple sources for system-level fault monitoring and analysis.

• Campus Manager—Provides network topology services, user tracking, and path analysis. Campus

Manager Topology Services can display a map of your network and it can display a variety of information about each device on the network. It provides version, run-time status, and URLs of the applications on the devices and it provides filtering to display only specified devices. User Tracking provides a tool that tracks IP telephones on a Voice over IP (VoIP) network. Path Analysis provides a diagnostic application that traces connectivity between two specified points on a network and analyzes physical and logical paths.

1-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 1 About Serviceability

For more information about CiscoWorks, refer to the documentation available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps3996/tsd_products_support_series_home.html

Syslog Support

In addition to writing information to a trace file, the Cisco CRS system sends standard event logging messages to a Syslog server through the Alarm Service. These messages contain information about the activities of the Cisco CRS Engine and its subsystems. You can use any Syslog server to analyze these messages.

For analyzing Syslog messages, the Cisco CRS system integrates with CiscoWorks Resource Management Essentials (RME). The RME Cisco Syslog Analyzer controls and displays all event messages so that they can easily be read, interpreted, filtered, and used for system maintenance and troubleshooting. In the Syslog Analyzer, these reports are available under WorkFlow Report. You can also adapt Syslog output from the Cisco CRS system for use with other network management systems that have standard Syslog receiving capability.

For information about configuring a Syslog server, see Configuring the Alarm Service, page 3-3.

For more information about CiscoWorks, refer to the documentation available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps3996/tsd_products_support_series_home.html

Syslog Support

Remote Serviceability

Many of the serviceability tools can be used by a Cisco Service Engineer (CSE) to assist you with the management and administration of your Cisco CRS system. These tools allow CSEs to remotely gather system and debugging information if you require help with troubleshooting or system diagnostics.

With your permission, CSEs can log on to a Cisco CRS server and obtain a desktop or shell that allows them to perform any function that could be performed locally.

Tools that assist with remote serviceability include:

• CiscoWorks—Provides remote management capabilities for the Cisco CRS system and Cisco CRS

network. For more information, see

• Microsoft Windows 2003 Performance Monitoring—Allows monitoring the performance counters

of the Cisco CRS system from local or from remote systems. For more information, refer to your Microsoft Windows documentation.

• Microsoft Windows 2003 Terminal Services—Provides remote systems with access to

Windows-based applications through terminal emulation. Windows 2003 Server Terminal Services are integrated with the Windows 2003 operating system.

• Virtual Network Computing (VNC) isa desktop protocol to remotely control another computer. It

transmits the keyboard presses and mouse clicks from one computer to another, relaying the screen updates back in the other direction, over a network. There are many variants of freeware VNC available today.

CiscoWorks Support, page 1-2.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

1-3

Chapter 1 About Serviceability

Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support

You can set up a dedicated server for monitoring, recording, and maintaining Unified CCX statistics. Such a server is called an Unified CCX Call Statistics, Recording, and Monitoring Server. You can also set up a dedicated server or servers for monitoring. These servers are called Unified CCX Call Monitoring Servers. The CRS installation process automatically sets up and configures serviceability on these dedicated servers.

Serviceability enables CiscoWorks support and third-party NMS support for the servers and includes:

• Cisco Discover Protocol (CDP) support, which enables the Media Convergence Server (MCS) to be

discovered automatically by CiscoWorks.

• CISCO-CDP-MIB support.

• SYSAPPL-MIB support, which provides run-time status, version information, and application

discovery for voice recording and monitoring services.

• Standard third-party MIB support.

• EMBLEM support for CiscoWorks.

1-4

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Simple Network Management Protocol Support

This section contains the following topics:

• About Simple Network Management Protocol (SNMP), page 2-1SNMP Basics, page 2-1

• SNMP Agent and Subagents, page 2-2

• SNMP Management Information Base (MIB), page 2-2

• SYSAPPL-MIB, page 2-2

• CISCO-VOICE-APPS-MIB, page 2-6

• CISCO-CDP-MIB, page 2-6

• SNMP Traps, page 2-6

• SNMP Trap Messages, page 2-6

• Setting up SNMP Traps, page 2-7

About Simple Network Management Protocol (SNMP)

Simple Network Management Protocol (SNMP) isan industry-standard interface for exchanging management information between network devices. SNMP and its components provide you with information about your Cisco CRS system. You can refer to this information to monitor and manage the status of the Cisco CRS system, its subsystems, and its related installed components. You can also use this information to troubleshoot problems, if they arise.

You can set up SNMP traps to automatically notify you of high-severity messages and errors that are generated by the Cisco CRS system.

SNMP Basics

A network management system (NMS) uses SNMP to exchange management information between devices on a network. An SNMP-managed network is made up of the following main components:

• Managed devices—Network nodes, each containing an SNMP agent. Managed devices collect and

store information and make this information available using SNMP.

• Agents—Network management software that resides on a managed device. An agent contains local

knowledge of management information and translates the information into a form that is compatible with SNMP.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-1

Chapter 2 Simple Network Management Protocol Support

SNMP Agent and Subagents

• Management stations—Computers on which the NMS is installed and from which system

administrators can retrieve and evaluate information from managed devices.

SNMP Agent and Subagents

The Microsoft Windows SNMP service (referred to as the SNMP Service) provides a framework for SNMP and provides the SNMP agent that interfaces with SNMP subagents.

SNMP Service starts automatically when the system starts. You can restart or stop the SNMP Service if a problem occurs or if it did not start automatically.

For more information, see Starting, Stopping, and Confirming the SNMP Service, page 2-9.

For information on configuring SNMP Service, see SNMP Traps, page 2-6.

SNMP Management Information Base (MIB)

A Management Information Base (MIB) designates a collection of information that is organized hierarchically. You access MIBs with SNMP. MIBs are made up of managed objects, which are identified by object identifiers. Managed objects are made up of one or more object instances, which are essentially variables. MIBs provide status monitoring, provisioning and notification.

The Cisco CRS system supports these MIBs:

• SYSAPPL-MIB—Provides system information, such as installed applications, application

components, product version, processes that are running, and process start time. For more information, see

SYSAPPL-MIB, page 2-2.

• CISCO-VOICE-APPS-MIB—Contains information about supported SNMP traps. For more

information, see

• CISCO-CDP-MIB—Provides information about device identifications, CDP (Cisco Discovery

Protocol) running status, CDP transmitting frequency, and the time for the receiving device to hold CDP messages (time to live). For more information, see

page 5-1.

Standard third-party MIBs, including:

–

SNMP Community Names authenticate access to MIB objects and serve as passwords for SNMP information. A system can exchange SNMP information only with systems in the same community. For more information on setting up communities, see

For additional information about MIBs, refer to this URL:

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

SYSAPPL-MIB

The SYSAPPL-MIB provides system information about installed packages, including product name, product version, URL of the Cisco CRS Administration page, run-time status, application start time, and currently running processes.

CISCO-VOICE-APPS-MIB, page 2-6.

Cisco Discovery Protocol Support,

Standard Microsoft MIBs, such as MIB II

Compaq Insight Agent MIBS for Compaq MCS 78xx platforms

IBM UM MIB for IBM 3xx MCS platforms

Setting the SNMP Trap Receiver, page 2-7.

2-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 2 Simple Network Management Protocol Support

The SYSAPPL-MIB allows you to use CiscoWorks or a third-party NMS browser to remotely access information about the Cisco CRS components including:

• Cisco CRS Administration

• Cisco CRS Node Manager

• Cisco CRS Engine

• Cisco CRS Repository Datastore

• Cisco CRS Historical Datastore

• Cisco CRS Config Datastore

• Cisco CRS Agent Datastore

• Cisco Recording

• Cisco Monitoring

The SYSAPPL-MIB also provides access to the Cisco CRS Services, including:

• Cisco CRS Cluster View Daemon

including but not limited to:

–

Manager Manager

SYSAPPL-MIB

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

Socket Manager

• Cisco CRS Administration

including but not limited to:

–

Manager Manager

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

File Manager

–

Prompt Manager

–

Grammar Manager

–

Document Manager

–

Resource Manager

–

Script Manager

–

Expression Manager

–

Socket Manager

• Cisco CRS Engine

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-3

SYSAPPL-MIB

including but not limited to:

–

Manager Manager

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

File Manager

–

Prompt Manager

–

Grammar Manager

–

Document Manager

–

Resource Manager

–

Script Manager

–

Expression Manager

Chapter 2 Simple Network Management Protocol Support

–

Socket Manager

–

RTP Port Manager

–

Contact Manager

–

Channel Manager

–

Session Manager

–

ICM Subsystem

–

JTAPI Subsystem

–

CMT Subsystem

–

MRCP ASR Subsystem

–

MRCP TTS Subsystem

–

eMail Subsystem

–

RmCm Subsystem

–

Voice Browser Subsystem

–

Core Real-Time Reporting Subsystem

–

Enterprise Server Data Subsystem

–

Database Subsystem

–

VoIP Monitor Subsystem

–

HTTP Subsystem

–

Outbound Subsystem

–

SIP Subsystem

2-4

–

• Cisco Desktop License and Resource Manager

• Cisco Desktop Call and Chat Service

• Cisco Desktop Enterprise Service

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 2 Simple Network Management Protocol Support

• Cisco Desktop IP Phone Agent Service

• Cisco Desktop Recording and Statistics Service

• Cisco Desktop VoIP Monitor Service

• Cisco Desktop Recording Service

• Cisco Desktop LDAP Monitor Service

• CRS SQL Server--Repository

• CRS SQL Server--Historical

• CRS SQL Server--Config

• CRS SQL Server--Agent

• Microsoft SQL Agent

• Microsoft Distributed Transaction Coordinator

The SYSAPPL-MIB also allows you to use CiscoWorks or a third-party Network Management System (NMS) to remotely access information about these Unified CCX Standard and Unified CCX Enhanced packages:

• Cisco CRS Recording and Statistics (RAS) Server

• Cisco CRS Telephony Agent Interface (TAI) Server

SYSAPPL-MIB

Note The TAI Server is also called the Cisco Desktop IP Phone Agent Service.

• Cisco CRS Enterprise Server

• Cisco CRS VoIP Monitor Server

• Cisco CRS Chat Server

The SYSAPPL-MIB also allows you to use CiscoWorks or a third-party NMS to remotely access information about these services on an Unified CCX Call Statistics, Recording, and Monitoring Server, or on an Unified CCX Call Monitoring Server:

• Cisco CRS Recording and Statistics (RAS) Server

• Cisco CRS VoIP Monitor Server

The SYSAPPL-MIB also allows you to use CiscoWorks or a third-party NMS to remotely access information about the status of the SQL services MSSQLService and SQLServerAgent. For a standalone CRS server (a server on which CRS but not Cisco Unified Communications Manager is installed), and for a Database Expansion Server, this information appears as "Cisco CRS Database."

The SYSAPPL-MIB uses SNMP to organize and distribute the information that it gathers from your network. The Cisco CRS system supports these SYSAPPL-MIB tables:

• SysApplInstallPkgTable—Provides installed application information, including manufacturer,

product name, product version, date installed, and location, which is a partial URL for accessing the associated Cisco CRS Administration web page (when applicable)

• SysApplRunTable—Describes the application starting time and run-time status

• SysApplInstallElmtTable—Describes the individual application elements or the associated

executables that make up the applications defined in the SysApplInstallPkgTable

• SysApplElmtRunTable—Describes the processes that are currently running on the host system,

similar to the processes that the Windows Task Manager displays

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-5

CISCO-VOICE-APPS-MIB

For more information about the SYSAPPL-MIB, refer to this URL:

ftp://ftp.cisco.com/pub/mibs/v2/SYSAPPL-MIB.my

CISCO-VOICE-APPS-MIB

The CISCO-VOICE-APPS-MIB provides information about supported SNMP traps. For more information about the CISCO-VOICE-APPS-MIB, refer to this URL:

ftp://ftp.cisco.com/pub/mibs/v2/CISCO-VOICE-APPS-MIB.my

CISCO-CDP-MIB

The CISCO-CDP-MIB provides information about device identifications, CDP running status, CDP transmitting frequency, and the time for the receiving device to hold CDP messages (time to live). This MIB stores information in a table called cdpGlobalInfo.

For more information about the CISCO-CDP-MIB, refer to this URL:

ftp://ftp.cisco.com/pub/mibs/v2/CISCO-CDP-MIB.my

Chapter 2 Simple Network Management Protocol Support

SNMP Traps

You can set up SNMP traps to automatically notify you of high-severity messages and errors that come from the Cisco CRS Engine. An SNMP agent can send traps that identify these important system events. Traps can also come from the Alarm Service. The Alarm Service forwards messages to the SNMP trap subagent, which sends the messages to the SNMP trap receiver in the proper format.

SNMP Trap Messages

Table 2-1 shows the Cisco CRS SNMP trap messages that are sent to an NMS specified as a trap receiver.

These trap messages can be sent for each subsystem shown in SYSAPPL-MIB, page 2-2.

Ta b l e 2-1 SNMP Trap Message Descriptions

Message Description

cvaProcessStart A Windows process associated with the Cisco CRS server started. The

cvaProcessStop A Windows process associated with the Cisco CRS server stopped or

cvaModuleStart A subsystem started successfully and became in-service. The trap

cvaModuleStop A subsystem stopped. The trap includes the severity level and the

cvaModuleRunTimeFailure A run-time failure occurred. The trap includes the severity level and

processId parameter specifies the Windows process ID.

aborted. The processId parameter specifies the Windows process ID.

includes the severity level and the module name.

module name. The cvaModuleFailureCause parameter specifies the cause, if available.

module name. The cvaModuleRunTimeFailureCause parameter specifies the cause, if available.

2-6

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 2 Simple Network Management Protocol Support

When an SNMP agent detects an alarm condition, it generates a trap (also called a notification) that is sent to configured IP addresses. To set up SNMP traps, see

Failover Traps

Cisco CRS failover traps are sent using CvaProcessStart trap with cvaModuleName description as "New Master xxx" where xxx = Process name. Below is a snapshot of a CRS Engine failover trap.

12/8/2006 13.51:28 SERVER-NAME Trap: P3 cvaProcessStart, ent=ciscoVoiceAppsMIBNotifications, comm-public, cvaAlarmSeverity=notice, cvaModuleName=New Master Engine, cvaProcessId=0

Setting up SNMP Traps

To use SNMP traps, you must designate the SNMP trap destination for the trap messages.

You can specify the following security options for the SNMP traps to ensure that only authorized systems have access to SNMP trap information:

• Community strings—Serve as passwords for SNMP information. A system can exchange SNMP

information only with systems in the same community.

Setting up SNMP Traps

Setting up SNMP Traps, page 2-7.

• Valid sources for SNMP requests.

• Read/write privileges—Whether systems can only read SNMP information or can read and write

information.

For additional information about SNMP security, refer to your Microsoft Windows documentation.

To configure the SNMP trap sender, see these sections:

• Setting the SNMP Trap Receiver, page 2-7

• Setting the SNMP Community Names, page 2-8

Setting the SNMP Trap Receiver

The trap receiver is the network management system (NMS) that receives the SNMP traps. This NMS must have the same SNMP community string as the trap sender. The Cisco CRS system sends traps that can be received by CiscoWorks and by standard third-party NMSs.

To set the SNMP trap receiver, follow these steps:

Step 1 From the Windows desktop, choose Start > Settings > Control Panel.

Step 2 Double-click Administrative Tools.

Step 3 Double-click Services.

The Services window appears.

Step 4 Right-click SNMP Services and choose Properties.

Step 5 Click the Tra p s tab.

Step 6 In the Community name field, enter the community name to which this computer will send trap

messages.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-7

Setting up SNMP Traps

You must configure at least one community string or SNMP will not respond to requests. Community name is case-sensitive

Step 7 Click Add to List.

Step 8 Under the Trap destinations field, clickAdd.

Step 9 In the SNMP Service Configuration dialog box, enter the IP address or the host name of the trap

destination.

Step 10 In the SNMP Service Configuration dialog box, click Add.

Step 11 Repeat Step 7 through Step 10 for each trap destination required.

Step 12 Click OK to apply your changes and exit the SNMP Service Properties window.

Setting the SNMP Community Names

You can configure security settings for the SNMP traps to ensure that only authorized system can access information that is sent to the traps. SNMP community names serve as passwords for SNMP information. You can set valid sources for SNMP requests and specify whether systems can only read information, or both read and write information. For more information about SNMP security, refer to your Microsoft Windows documentation.

Chapter 2 Simple Network Management Protocol Support

To set up community names and privileges, follow these steps:

Step 1 From the Windows desktop, choose Start > Settings > Control Panel.

Step 2 Double-click Administrative Tools.

Step 3 Double-click Services

The Services window appears.

Step 4 Right-click SNMP Services and chooseProperties.

Step 5 Click the Security tab.

Step 6 In the Accepted Community Names pane, click Add.

The SNMP Service Configuration dialog box appears.

Step 7 In the Community Name field, enter the name of the community.

Step 8 If you need write privileges for the community, choose READ WRITE from the Community Rights

drop-down list.

Step 9 On the SNMP Service Configuration dialog box, click Add.

Step 10 Repeat Step 6 through Step 9 as needed to add other community names.

Step 11 If you want to allow only specific NMS hosts to query the SNMP subagent, follow these steps:

a. Click the Accept SNMP packets from these hosts radio button.

b. In the Accept SNMP packets from these hosts pane, click Add.

c. In the SNMP Service Configuration dialog box, enter the IP address or the host name of the host that

is allowed to query the SNMP subagent.

d. In the SNMP Service Configuration dialog box, click Add.

2-8

e. Repeat Steps a through d as needed.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 2 Simple Network Management Protocol Support

Step 12 Click OK to apply your changes and exit the SNMP Service Properties window.

Starting, Stopping, and Confirming the SNMP Service

In general, the SNMP Service will always be running. To confirm that the SNMP Service is running and to restart it or stop it, if necessary, follow these steps:

Step 1 From the Windows desktop, chooseStart > Settings > Control Panel

Step 2 Double-click Administrative Tools.

Step 3 Double-click Services.

The Services window appears.

Step 4 Look at the Status field in the SNMP Service row.

If this field displays "Started," the SNMP Service is running. If this field is blank, the SNMP Service is not running.

To start the SNMP Service, right-click SNMP Service and choose Start.

Setting up SNMP Traps

To stop t h e SNMP Service, right-click SNMP Service and choose Stop.

Snapshot of Traps During Startup

The following example shows a snapshot of traps generated by CRS during startup. To view this file more clearly in Acrobat, use the Zoom In icon on the Acrobat menu bar to increase the text size.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-9

Setting up SNMP Traps

Figure 2-1 Traps During Startup

Chapter 2 Simple Network Management Protocol Support

Snapshot of Traps During Shutdown

The following example shows a snapshot oftraps generated by CRS during shutdown. To view this file more clearly in Acrobat, use the Zoom In icon on the Acrobat menu bar to increase the text size.

Figure 2-2 Traps During Shutdown

2-10

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Alarm Service

This section contains the following topics:

• About Alarms, page 3-1

• Cisco CRS Alarm Service, page 3-1

• Starting and Confirming the Alarm Service, page 3-2

• Configuring the Alarm Service, page 3-3

• Viewing Alarm Messages, page 3-3

• Viewing Alarm Messages Sent to a Syslog Server, page 3-3

• Viewing Alarm Messages Sent to an SNMP Trap Receiver, page 3-5

• Viewing Alarm Messages Sent to a Windows Event Log, page 3-5

• Alarm Definitions, page 3-5

• Finding Information About an Alarm, page 3-5

About Alarms

Alarms provide information about the Cisco CRS system activities. You can use this information to monitor the status and the state of the system and to determine actions to take if a problem occurs. By default, the Cisco CRS system also writes alarm information to trace files. You can use the information in a trace file for further analysis of a problem.

Cisco CRS Alarm Service

The Cisco CRS Alarm Service is installed as part of the Cisco CRS installation process. It is a Windows service that receives alarms about system events from the Cisco CRS Engine, Cisco CRS Node Manager, Cisco CRS Administration, Cisco CRS Repository Datastore, Cisco CRS Historical Datastore, Cisco CRS Config Datastore, Cisco CRS Agent Datastore, Cisco Recording, and Cisco Monitoring components. These alarms are defined in XML format in files called catalogs. Catalogs are set up as part of the Cisco CRS installation process.

Based on catalogs, the Cisco CRS Alarm Service forwards the alarms that it receives to one or more of the following destinations:

• Syslog Server—Forwards alarms as standard Syslog-format messages to CiscoWorks or any

third-party Syslog server. For related information, see

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CiscoWorks Support, page 1-2.

3-1

Cisco CRS Alarm Service

Chapter 3 Alarm Service

• SNMP Trap Subagent—Processes alarms and sends them as traps to a configured trap receiver, such

as the Voice Health Monitor (VHM) in CiscoWorks. For more information, see

page 1-2 and SNMP Traps, page 2-6.

• Windows Event Log—Sends alarms that can be viewed with the Windows Event Viewer. For more

information, see

Viewing Alarm Messages Sent to a Windows Event Log, page 3-5.

You can specify the severity level of the alarm that the Cisco CRS Alarm Service sends to a Syslog server. Alarm severity levels are described in the following table. For more information, see

the Alarm Service, page 3-3.

Ta b l e 3-1 Alarm Severity Levels

Severity Level

Name Explanation

0 EMERGENCY_ALARM System emergency

1 ALERT_ALARM Situation where the application will continue to run but

not all functions are available

2 CRITICAL_ALARM Critical failure that prevents the application from

accomplishing a task

3 ERROR_ALARM Critical failure that prevents the application from

accomplishing a task

4 WARNING_ALARM Problem exists but it does not prevent the application

from completing its tasks

5 NOTICE_ALARM Notification of a normal but significant condition

6 INFORMATIONAL_ALARM Information that does not relate to errors, warnings,

audits, or debugging

7 DEBUG_ALARM Detailed information regarding system errors and

processing status

CiscoWorks Support,

Configuring

Starting and Confirming the Alarm Service

In general, the Cisco CRS Alarm Service is always running.

To confirm that the Alarm Service is running and to restart it, if necessary, follow these steps:

Step 1 From the Windows desktop, choose Start > Settings > Control Panel.

Step 2 Double-click Administrative Tools.

Step 3 Double-click Services.

The Services window appears.

Step 4 Look at the Status field in the Cisco CRS Alarm Service row.

If this field displays “Started”, the Alarm Service is running. If this field is blank, start the Alarm Service by right-clicking Cisco CRS Alarm Service in the Name field, and then choosing Start.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

3-2

Chapter 3 Alarm Service

Configuring the Alarm Service

When you configure the Alarm Service, you provide the Cisco CRS system with information about how to handle alarms. To configure the Alarm Service, perform the following steps.

If you will be entering information in the Syslog Server Name field or in the Syslog Message Filtering Level field, as explained in Step 3, make sure that the Alarm Service is running before following these steps. (See

Step 1 From the Cisco CRS Administration menu, choose System > Alarm Configuration.

The Alarm and Tracing Configuration page appears.

Step 2 Choose Alarm Server Tracing from the navigation bar.

Step 3 Enter information in the fields under Alarm Service as follows:

• Alarm Server—IP address or name of the server on which the Cisco CRS Alarm Service is running.

• Alarm Server Port—Port on the Alarm Server to which alarm messages are sent. This information

Starting and Confirming the Alarm Service, page 3-2 for more information.)

By default, the Alarm Server is “localhost,” meaning that the Alarm Service is running on the Cisco CRS server. You cannot change this information.

is entered as part of the installation process. The default value is 1444. You cannot change this information.

Viewing Alarm Messages

• Catalog Directory—Directory in which the catalogs of alarm messages are stored. The default is

“catalog”. This information is entered as part of the installation process. You cannot change this information.

• Syslog Server Name—Enter the IP address or the host name of the Syslog server to which alarm

messages are be sent. If you are using CiscoWorks, enter the IP address or the host name of the CiscoWorks server. If this field is blank, the system sends alarm messages to the Cisco CRS server.

• Syslog Message Filtering Level—Click the drop-down arrow and choose the severity level of alarm

messages that you want sent to the Syslog server. Syslog messages range from severity 0 (most severe) to severity 7 (least severe). When you choose a severity level, all messages of that severity level and higher will be sent. For example, if you choose ERROR_ALARM (Severity 3), all messages of severity 3, severity 2, severity 1, and severity 0 will be sent. The default is “DEBUG_ALARM (Severity 7)”, which will send messages of all severity levels.

Step 4 Click Update.

Viewing Alarm Messages

The way in which you view alarm messages depends on the destination to which messages were sent. Each alarm message that you view will include an alarm name. To find information about the alarm name that appears in an alarm message, see

Alarm Definitions, page 3-5.

Viewing Alarm Messages Sent to a Syslog Server

To view alarm messages that were sent to a CiscoWorks Resource Management Essentials (RME) Syslog server, refer to the CiscoWorks documentation, available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps2073/tsd_products_support_series_home.html

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

3-3

Viewing Alarm Messages Sent to a Syslog Server

To view alarm messages that were sent to a third-party Syslog server, refer to the documentation for that system.

Table 3-2 describes the fields found in Syslog messages.

Ta b l e 3-2 Syslog Message Format

Field Example Description

<pri> <128> This field is added so that syslog can read the

n: 100: This field mimics the Solaris syslogd, which

MMM DD Aug 09 Abbreviated month day as known at the source.

hh:mm:ss.mmm 19:20:10.209 Time at source device. The UTC time is used to

TimeZone UTC Abbreviated time zone defined in the device, such

% FACILITY Allowed characters A-Z 0-9 _

[SUBFACILITY-] A-Z 0-9 _

SEVERITY 0 A single-digit code from 0 to 7 that reflects the

MNEMONIC BADIPALIGN: Invalid

Message-text Module Failure

CDP (Cisco Discovery Protocol), ALIGN (Memory optimization in RISC)

CLAW (Common Link Access for Workstations)

alignment in packet for IP.

Cause=Unknown

Chapter 3 Alarm Service

severity level. Syslogd looks for this pri value which is set to LOCAL0|SEVERITY by default.

prefixes the syslog message with an internal counter szi. It has no significance to the SAC. The number is parsed out by the SAC.

avoid any time zone name discrepancy.

as GMT. This field is always set to UTC to avoid any time zone name discrepancy.

A code consisting of two or more uppercase letters that indicate the facility to which the message refers. A facility can be a hardware device, a protocol, or a module of the system software. Note that this is not the same as the UNIX Syslog server logging facility.

Subfacility Code. This field is optional.

severity of the condition. The lower the number, the more serious the situation. Severity also maps to logging level.

The mnemonic code uniquely identifies the error message. This code is used by CiscoWorks to associate the syslog message with the message information in the message catalog.

A text string describing the condition.

3-4

Sample Format:

<LOCAL7|SEVERITY>51:Oct 18 03:28:29.327 PDT: %MIVR-GENERIC-1-ModuleStop: Module has stopped; Module Name=HTTP SubSystem; Module Failure Cause=Unknown

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 3 Alarm Service

Viewing Alarm Messages Sent to an SNMP Trap Receiver

To view alarm messages that were sent to the CiscoWorks Voice Health Monitor (VHM), refer to the CiscoWorks documentation, available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps2431/tsd_products_support_eol_series_home.ht ml

To view alarm messages that were sent to a third-party SNMP trap receiver, refer to the documentation for that system.

Viewing Alarm Messages Sent to a Windows Event Log

You use the Windows Event Viewer to view alarm messages that were sent to a Windows event log. To use the Windows Event Viewer, perform the following steps. For additional information about the Windows Event Viewer, refer to your Microsoft Windows documentation.

Step 1 From the Windows desktop, choose Start > Setting > Control Panel.

Step 2 Double-click Administrative Tools.

Step 3 Double-click Event Viewer.

Step 4 On the Tree pane, click the item for which you want to view information.

Alarm Definitions

Cisco CRS maintains a list of alarm catalogs. Each of these catalogs contains a list of alarms. Each alarm contains a definition of the alarm, which includes the alarm name, a description, an explanation, recommended actions, and related information.

An alarm name appears in an alarm message as follows:

• Trace file—Alarm name follows the severity level.

• CiscoWorks RME—Alarm name appears in the Mnemonic field on the Syslog WorkFlow report.

• Third-party Syslog server—Alarm name follows the reason. If a reason is not shown the alarm

message name follows the module name.

• Windows Event Viewer—Alarm name follows the severity level.

Finding Information About an Alarm

To use the alarm catalog to find information about an alarm message name, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose Tools > Alarm Definition.

The Alarm Definitions web page appears.

Locate information for the alarm message name as follows:

• For a list of all alarm message names, make sure that All appears in the Catalog field, and then click

Search.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

3-5

Alarm Definitions

Step 2 To see a detailed explanation of an alarm message name, click the alarm message name.

Chapter 3 Alarm Service

• For a list of alarm message names that relate to a specific facility and subfacility, click the Catalog

drop-down arrow, choose the desired item, and then click Search.

• For a specific alarm message name, type the name of the alarm in the Enter Alarm Name field, and

then click Search.

A list of the alarm message names that you requested appears. If the list contains more than one page, you can click First, Previous, Next, or Last to move through the list. You can also type a page number in the Page field and click Page to move to that page.

3-6

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Trace

This section contains the following topics:

• About Trace Files, page 4-1

• The Component Trace File, page 4-2

• Configuring the Component Trace File, page 4-2

• Trace Level Options, page 4-3

• Setting Trace Level Options, page 4-7

• Viewing and Interpreting the Trace Files, page 4-8

• Displaying a Trace File, page 4-8

• Interpreting a Trace File, page 4-8

• The Thread Dump Trace File, page 4-8

• Writing to the Thread Dump Trace file, page 4-9

• Displaying the Thread Dump Trace File, page 4-9

• The CRS Log Files, page 4-9

• Cisco Desktop Product Suite Installation Logs, page 4-11

• CRS Log Collection Tool, page 4-11

About Trace Files

A trace file is a log file that records activity from the Cisco CRS components. Trace files let you obtain specific, detailed information about the system that can help you troubleshoot problems.

The Cisco CRS system can generate trace information for every component subfacility. This information is stored in a trace file. To help you control the size of an trace file, you specify the component and subfacilities for which you want to collect information and the level of information that you want to collect.

The Cisco CRS system also generates information about all threads that are running on the system. This information is stored in the thread dump trace file and is useful for troubleshooting.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-1

The Component Trace File

You can create a trace file for any of the following Cisco CRS components:

• Cisco CRS Engine

• Cisco CRS Administration

• Cisco CRS Editor

• Cisco CRS Node Manager

• Cisco CRS SQL Server

The component trace file contains information about each of the component's subfacilities. To set up this trace file, you perform the following general procedures:

• Configuring the Component Trace File, page 4-2

• Trace Level Options, page 4-3

For information about reading the trace file, see the Viewing and Interpreting the Trace Files, page 4-8

Chapter 4 Trace

Configuring the Component Trace File

By default, the Cisco CRS system sends information about subfacilities toa trace file, for example, CiscoMIVRnn.log. The system replaces nn with a number, starting with 01. You can configure the size of the trace file. When the size you configured is reached, or if a Cisco CRS component is restarted, the system creates a new trace file, incrementing nn by one. After creating the tenth trace file (by default), the trace file begins overwriting existing files, starting with the first trace file created.

Note that the examples shown here are for the Cisco CRS Engine component. Follow the same procedures for the other components, substituting the component's name.

To change any of these default trace file parameters, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose System > Tracing.

The Trace Configuration page appears.

Step 2 Choose and expand a component from the navigation bar.

Step 3 Change the following information under Trace File as needed:

• Trace File Output—Check this check box to send information to a trace file. Uncheck this box if you

do not want to send information to a trace file. By default, this check box is checked.

• File Name—Enter the base name and the extension of the trace file. A trace file name is made up of

the base facility name, the file number, and the extension (for example, CiscoMIVR01.log). The default file name is Cisco<facility_code>.log

where the <facility_code> could be MIVR, MCVD, MADM, MEDT, or MARC.

• Number of Trace Files—Enter the number of trace files that the system will create before starting

to overwrite existing files. The system will create a new trace file each time the existing one reaches the size specified in the Trace File Size field. The default number of trace files is 10.

• Trace File Size—You can configure the file size, or you can enter the maximum size, in bytes, of the

trace file. The default files size is 1048576.

4-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 4 Trace

Step 4 Click Update.

Trace Level Options

A trace file is a log file that records activity from the Cisco CRS component subsystems and steps. Trace files let you obtain specific, detailed information about the system that can help you troubleshoot problems.

The Cisco CRS system can generate trace information for every subfacility. This information is stored in an engine trace file. To help you control the size of an engine trace file, you specify the subfacilities for which you want to collect information and the level of information that you want to collect.

The Cisco CRS system also generates information about all threads that are running on the system. This information is stored in the thread dump trace file and is useful for troubleshooting.

A trace file that records all information for a component, such as the Cisco CRS Engine, can become large and difficult to read. To help you manage the trace file, the Cisco CRS system lets you specify the subfacilities for which you want to record information. These subfacilities are shown in the following table.

For each subfacility, you can select a trace level of Debugging, Alarm Tracing, both selections, or no selections. These selections specify the messages that the system sends to a trace file. the effect of each trace level settings. For an explanation of message severity levels, see Cisco CRS

Alarm Service, page 3-1.

Trace Level Options

Table 4-1 shows

Warning

Ta b l e 4-1 Messages Sent to a Trace File

Severity Level of

Selection

Messages Sent

Explanation

Debugging 0, 1, 2, 3, 7 Sends detailed, verbose information. To be used

primarily for debugging and troubleshooting.

Alarm Tracing 0, 1, 2, 3, 4, 5, 6 Sends messages of all severity levels except

detailed debugging information.

Debugging and Alarm

0, 1, 2, 3, 4, 5, 6, 7 Sends messages of all severity levels.

Tracing

No selections 0, 1, 2, 3 Sends high-priority notifications, errors, and

alerts.

The Trace Configuration pane groups trace level options into these lists:

• Active trace level options—Facilities and subfacilities that are running on your system

• Inactive trace level options—Facilities and subfacilities that are not running on your system

If you make a change under an active facility, the trace file will reflect your change immediately. If you make a change under an inactive subfacility, the change will take effect when the subfacility becomes active.

Level 7 traces are debug only and do not reflect a system issue.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-3

Trace Level Options

Chapter 4 Trace

All applications that use the CRS Trace library use a Syslog Facility Code. Tab le 4-2 lists the Facilities and Descriptions for the Trace Files

Ta b l e 4-2 Trace File Facilities

Facility Code Description

MIVR Workflow Application Framework

MCVD Cluster Framework

MADM CRS Administration page

MEDT Editor

Table 4-3 describes the Trace file subfacilities.

Ta b l e 4-3 Trace File Subfacilities

Subfacility Code Description

AC_CLUSTER Archive Cluster Component

AC_CONFIG Archive Configuration Component

AC_DATABASE Archive Database Component

AC_JTAPI JTAPI Archive Component

AC_OS Archive Operating System Component

AC_SPANLINK CAD/CSD Archive Component

ADM Administration Client

ADM_CFG Administration Configuration

APP_MGR Applications Manager

ARCHIVE_MGR Archive Manager

AW_C FG Restore Administration Configuration

BARBI_CLI Backup and Restore Client Interface

BOOTSTRAP_MGR CRS Bootstrap Manager

CFG_MGR Configuration Manager

CHANNEL_MGR Channel Manager

CLUSTER_MGR Cluster Manager

CONTACT_MGR Contact Manager

CONTACT_STEPS Contact Steps

CRA_CMM CRS ClusterMsgMgr Component

CRA_HRDM CRS Historical Reporting Data Manager

CVD Cluster View Daemon

DB Database

DBPURGE_MGR Database Purge Manager

DESKTOP CRS Editor Desktop

DOC_MGR Document Manager

4-4

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 4 Trace

Trace Level Options

Table 4-3 Trace File Subfacilities (continued)

Subfacility Code Description

EDT CRS Editor general

ENG CRS Engine

EXECUTOR_MGR Executor Manager

EXPR_MGR Expression Manager

FILE_MGR File Manager

GENERIC Generic catalog for a facility

GRAMMAR_MGR Grammar Manager

GRP_CFG Group Configuration

HOLIDAY_MGR Holiday Manager

HR_MGR Historical Reports Manager

ICD_CTI Unified CCX CTI Server

ICD_HDM Unified CCX Historical Data Manager

ICD_RTDM Unified CCX Real-Time Data Manager

IO_ICM Unified ICME Input/Output

JAS MIN Java Signaling and Monitoring Interface

LIB_APPADMININTERCEPTOR CRS Administration Interceptor Library

LIB_AXL AXL Library

LIB_CFG Configuration Library

LIB_CRTP CRTP Library

LIB_DATABASE Database Library

LIB_DIRECTORY Directory Access Library

LIB_EVENT Event Message Library

LIB_ICM Unified ICME Library

LIB_JASPER Jasper Tomcat Library

LIB_JCUP JavaCup Library to parse expressions

LIB_JDBC JDBC Library

LIB_JINI JINI Services

LIB_JMAIL Java Mail Library

LIB_JLEX JLEX Library used to parse expressions

LIB_LICENSE License Library

LIB_MEDIA Media Library

LIB_RMI Java Remote Method Invocation Library

LIB_SERVLET Servlet Library

LIB_TC To m cat Lib r ary

LOG_MGR Log Manager

MRCP_CFG MRCP Configuration

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-5

Trace Level Options

Chapter 4 Trace

Table 4-3 Trace File Subfacilities (continued)

Subfacility Code Description

MGR_MGR Manager Manager

NODE_MGR Node Manager

PALETTE Editor Palette

PROMPT_MGR Prompt Manager

PURGING Purging

RPT Reporting

RTPPORT_MGR RTP Manager

SCRIPT_MGR Script Manager

SESSION_MGR Session Manager

SIP_STACK SIP Stack logging

SOCKET_MGR Socket Manager

SS_APP Application Subsystem

SS_CM Contact Manager Subsystem

SS_CMT Cisco Media Termination Subsystem

SS_DB Database Subsystem

SS_EMAIL E-mail Subsystem

SS_ENT_SRV Enterprise Server Subsystem

SS_HTTP HTTP Subsystem

SS_ICM Unified ICME Subsystem

SS_MRCP_ASR MRCP ASR Subsystem

SS_MRCP_TTS MRCP TTS Subsystem

SS_OUTBOUND Preview Outbound Dialer Express Subsystem (uses MIVR log

file)

SS_RM Resource Manager Subsystem

SS_RMCM Resource Manager Contact Manager Subsystem

SS_RTR Real-Time Reporting Subsystem

SS_SIP SIP Subsystem

SS_TEL JTAPI Subsystem (Telephony)

SS_VB Voice Browser Subsystem

SS_VOIPMON_SRV Voice over IP Monitor Server Subsystem

STEP_CALL_CONTROL Call Control Steps

STEP_ENT_SRV Enterprise Server Steps

STEP_MEDIA_CONTROL Media Control Steps

STEP_SESSION Sessions Steps

STEP_SESSION_MGMT Session Management Steps

STEP_USER User Steps

4-6

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 4 Trace

Trace Level Options

Table 4-3 Trace File Subfacilities (continued)

Subfacility Code Description

STEP_CALL_CONTACT Call Contact Steps

STEPS_CONTACT Contact Steps

STEPS_DB Database Steps

STEPS_DOCUMENT Document Steps

STEPS_EMAIL E-mail Steps

STEPS_GENERAL General Steps

STEPS_GRAMMAR Grammar Steps

STEPS_HTTP HTTP Steps

STEPS_ICM Unified ICME Steps

STEPS_IPCC_EXP Unified CCX Steps

STEPS_JAVA Java Steps

STEPS_PROMPT Prompt Steps

STEPS_SESSION Session Steps

STEPS_USER.ALARM User Alarm Steps

USR_MGR User Manager

WEB_STEPS HTTP Contact Steps

When the Unified CCX product is running on a 7845 machine and tracing is ON (the default), limit the Busy Hour Call Completions (BHCC) to 4500 calls per hour. If you want to run a higher BHCC, turn the debug traces OFF. The trace subfacilities to be turned OFF are ICD_CTI, SS_TEL, SS_RM, SS_CM, and SS_RMCM.

Setting Trace Level Options

To set trace level options, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose System > Tracing.

The Trace Configuration web page appears.

Step 2 Under a specific CRS Component, choose Trace File Configuration from the navigation bar.

Step 3 Check or uncheck the desired boxes in the Active trace level option list and in the Inactive trace level

option list.

Step 4 Click Update.

Step 5 If you made any changes in the Inactive trace level option list, stop and restart the Cisco CRS Engine to

reflect your changes in the trace file.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-7

Viewing and Interpreting the Trace Files

The Cisco CRS server stores the trace files in the Log directory under the directory in which you installed the Cisco CRS component. From the Cisco CRS Administration menu, you can view a list of all trace files and display the contents of any trace file.

Displaying a Trace File

To display a CRS component trace file, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose System > Tracing.

The Trace Configuration web page appears.

Step 2 Select and expand a component from the navigation bar and select Trace Configuration.

A list of subfacility categories appears.

Step 3 Expand the category of subfacility, select the levels of debugging for specific subfacilities, and click

Update.

The trace file appears in a separate window.

Chapter 4 Trace

Interpreting a Trace File

The trace files contain information in standard Syslog format. The file includes some or all of the following information for each event that it records:

• Line number

• Date and time the event occurred

• Facility and subfacility name

• Severity level

• Message name

• Explanation

• Parameters and values

The Thread Dump Trace File

The thread dump trace file is named JVM.log. It is stored on the Cisco CRS server in the Log directory under the directory in which you installed the Cisco CRS Engine. This file contains stack trace information about all threads that are running on the Cisco CRS system. You can write information to this file when you need it. In addition, the system writes information to this file automatically if the system detects a severe system problem. When new information is generated, it is appended to the existing thread dump file.

4-8

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 4 Trace

Note There is also a Memory Dump file. It is located in CRS Administration in the same place as the Thread

Dump file. It creates a memory dump file of the typememory<timestamp>.log.

Writing to the Thread Dump Trace file

To manually write to the thread dump trace file, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose System > Control Center.

The Control Center web page appears.

Step 2 Click Servers and choose the server hostname from the navigation bar (if it is not the selected server).

Step 3 Click Server Traces (at the top), and choose the component for which you want to enable the thread

dump.

Step 4 Click Dump Threads Trace.

The CRS Log Files

Displaying the Thread Dump Trace File

To display the thread dump trace file, follow these steps:

Step 1 From the Cisco CRS Administration menu, choose System > Control Center.

The Control Center web page appears.

Step 2 Click Servers and choose the server hostname from the navigation bar (if it is not the selected server).

Step 3 Click Server Traces (at the top), and choose the component for which you want to enable the thread

dump.

Step 4 Click Dump Threads Trace.

Step 5 In the File Name column, click JVM.log.

The trace file appears in a separate window.

The CRS Log Files

The CRS log files can help you troubleshoot problems. Table 4-4 provides information about the log files for the various CRS components and points you to the log file path locations.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-9

Chapter 4 Trace

The CRS Log Files

Ta b l e 4-4 CRS Log Files

Component Path File/Extension

MSI Installer \ (root) • CRSMsiInstallLog.txt

• CalInstall.log

• CRSMsdeInstallLog.txt

• CRSAutorun.log

• CRS-BARSJVMinstallLog.txt

• SQLInstallLog.txt

• CRSMsiUnregister.log

• UpdateTool.log

• CRSPatchInstallLog.txt

• CRSJREInstallLog.txt

Cluster View Daemon (CVD) \Program Files\wfavvid\log\MCVD\ *.log

Database

ADS / HDS / RDS

\Program Files\wfavvid\log\ReplLogs\

\Program Files\Microsoft SQL

*.log

CRS SQL Server Logs

Server\MSSQL$CRSSQL\LOG\

*.log

CRS Administration \Program Files\wfavvid\log\MADM\ *.log

Engine, driverManager \Program Files\wfavvid\log\MIVR\ *.log

JTAPI \Program Files\wfavvid\log\JTAPI\ *.log

CRS Editor \Program Files\wfavvid\log\MEDT\ *.log

Archive Tool \%TEMP%\ log\MARC\ *.log

Serviceability components:

1. Alarm Service

2. SNMP SYSApp

3. SNMP CDP

1. \Program Files\Cisco\AlarmService\AlarmServiceLog\

2. \Program Files\Cisco\AlarmService\Log \ALARM\

1. *. log

2. SnmpSysAppImpl.log

3. SnmpCdpImpl.log

3. \Program Files\Cisco\bin\

Transaction components:

1. Transaction Manager's Persistence directory (Do Not delete or modify this directory

1. \Program Files\wfavvid \txnMgrPersistence

2. \Program Files\wfavvid \txnMgrPersistence_old

3. \Program Files\wfavvid \BSTxState

1. everything inside

2. everything inside

3. *.per and *.old

2. Transaction Manager's Persistence Backup directory

3. Transaction State directory (Do Notdelete or modify this directory)

4-10

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 4 Trace

Cisco Desktop Product Suite Installation Logs

If you need to locate the Cisco Desktop Product Suite, Version 6.4, log files, this section helps you to locate them.

Here are the locations of the various log files:

• The Install Manager log files are located at the root of the C: drive:

The files are:

–

IM<number>.dbg - where <number> ranges between 0001 & 0010, (i.e IM0001.dbg)

–

IM<number>.log

• The InstallShield silent install file is located at C:\Winnt:

–

splk_<project>.log - where <project> is a Desktop installation project, such as splk_base.log.

• The InstallShield install / uninstall debug files are located at <Program Files>\Cisco\Desktop\IM:

The files are:

–

splkInstall_<version>.dbg - where <version> is a Desktop software version, such as splkInstall_6.4.0.20.dbg

–

splkInstall_Obj_<version>.dbg - where <version> is a Desktop software version, such as splkInstall_Obj_6.4.0.20.dbg.

After you uninstall the Desktop, the log files are located at:

The CRS Log Files

• Install Manager files exist in the location defined above until the uninstall reboot when they are

removed from the system.

• InstallShield silent uninstall file is at the root of the BootUp drive.

• InstallShield install / uninstall debug files are at the root of the C: drive.

CRS Log Collection Tool

The CRS Log Collection Tool provides a way for you to collect all of the log files you want to view into one zip file. The tool also provides a way for you to run it remotely and to move the zip file off of the CRS server to your own desktop or to a network drive.

To use the log collection tool to collect log files into a zip file, complete the following steps:

Step 1 To access the CRS Log Collection Tool, go to Start > Programs > Cisco CRS Administrator > Cisco

CRS Log Collection Tool.

The following warning message appears:

Warning The Log Collection Tool might impact system performance, so run this tool during off peak hours. Do

not run this tool during a system backup or restore. To save disk space on the CRS server, write the zip file to a network drive; otherwise, remove the zip file from the CRS server once captured. To limit the size of the zip file, use the Log Collection Tool Advanced Options to select start and end times and any subset of components.

Step 2 After reading and adhering to the message, click OK.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

4-11

The CRS Log Files

Step 3 Enter the path and name of the zip file you want to create and click Save. This collects all the log files

Step 4 Enter the information to limit the collection of log files by date and time. Check the check boxes of the

Step 5 Click Yes to save the file to the CRS system, or click No to go back and select another location on a

Step 6 ClickYes to continue, or click No if you want to stop the collection of log files into the zip file.

Chapter 4 Trace

The CRS Log Collection Tool dialog box appears.

on the system into the zip file. If you want to limit the number of files by date, time, and component, and if you want to select another location for the zip file, check the Advanced Options check box, and the dialog box appears with more options.

components for which you want to collect log files, and then browse to a location where you want to move the zip file, by clicking the... button next to the Source Drive field. Select the location for the zip file; it then appears in the Source Drive field. Then click Save.

If you choose a location on the CRS system instead of on a network drive, a warning message appears asking if you want to continue.

network drive.

When you click Ye s, the tool displays a dialog box with the estimated disk space to be used by the zip file before actually writing the zip file.

While the tool is collecting the log files, a Progress dialog box appears.

4-12

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHAP T E R

Cisco Discovery Protocol Support

This section contains the following topics:

• About the Cisco Discovery Protocol (CDP), page 5-1

• Using the CDP Driver, page 5-2

• Accessing CDP Driver Control, page 5-2

• Installing the CDP Protocol Driver, page 5-2

• Starting the CDP Protocol Driver, page 5-2

• Enabling the CDP Protocol Driver, page 5-3

• Showing the CDP Protocol Driver Properties, page 5-3

• Updating an IP Address for the CDP Protocol Driver, page 5-3

• Locating Updated CDP Driver and Interface Files, page 5-4

• Default CDP Settings, page 5-4

About the Cisco Discovery Protocol (CDP)

The Cisco CRS system uses the Cisco Discovery Protocol (CDP) to periodically send out CDP messages to a designated multicast address. These messages contain information such as device identification, interface name, system capabilities, SNMP agent address, and time-to-live. Any Cisco device with CDP support can locate a Cisco CRS server by monitoring these periodic messages.

Using information provided through CDP, the CiscoWorks server discovers your Cisco CRS server and the Campus Manager application Topology Services builds topology maps that display the CRS server and other Cisco devices.

CDP is enabled on the Cisco CRS system by default. You must have the CDP driver enabled at all times for CiscoWorks to discover the CRS server.

Note The Windows 2003 CDP Protocol Driver is designed to run with Cisco CRS on a Cisco Media

Convergence Server (MCS) with a 10/100BaseT Ethernet network interface card under Windows 2003 Server. It does not support other media, such as Token Ring, ATM, or Windows NT platforms (including Windows 98 or Windows NT 4.0).

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

5-1

Using the CDP Driver

Starting a system on which Cisco CRS is installed enables the CDP driver. You can use CDP to allow CiscoWorks to discover and manage your Cisco CRS systems.

CiscoWorks uses the CDP cache MIB of the direct neighboring device to discover the Cisco CRS server. You can use CiscoWorks to query other Cisco CRS-supported MIBs for provisions or statistics.

Accessing CDP Driver Control

You can control the CDP driver using the CISCO-CDP-MIB.

Chapter 5 Cisco Discovery Protocol Support

Warning

Alter the CDP setting only in special cases. For example, you might restart the CDP driver from the Control Panel at run time to pick up the latest IP configuration changes without resetting the system.

Installing the CDP Protocol Driver

The Cisco CRS installation process installs the CDP protocol driver. After completion of a successful Cisco CRS installation, the CDP protocol driver resides in the list of device drivers under the Windows Control Panel.

Starting the CDP Protocol Driver

To start the CDP protocol driver, follow these steps:

Step 1 Choose Start > Settings > Control Panel.

Step 2 Double-click System.

Step 3 Click the Hardwaretab.

Step 4 Click Device Manager.

The Device Manager window appears.

Step 5 Choose View > Devices by connection.

Step 6 Choose View > Show hidden devices.

5-2

Step 7 Double-click CDP Protocol Driver.

Step 8 Click the Driver tab.

Step 9 Click Start to enable the driver (Default = Start).

Step 10 Click OK.

Note Choosing Startup Type=Demand keeps Start setting after a restart

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 5 Cisco Discovery Protocol Support

Enabling the CDP Protocol Driver

Toenable the CDP protocol driver, follow these steps.

Step 1 Choose Start > Settings > Control Panel.

Step 2 Double-click System.

Step 3 Click the Hardware tab.

Step 4 Click the Device Manager button.

The Device Manager window appears.

Step 5 Choose View > Devices by connection.

Step 6 Choose View > Show hidden devices.

Step 7 Double-click CDP Protocol Driver.

Step 8 Click the Driver tab.

Step 9 Choose Enable Device.

Step 10 Click Next, and then click Finish to enable the device.

Step 11 Click Close and restart the system.

Updating an IP Address for the CDP Protocol Driver

Showing the CDP Protocol Driver Properties

To show CDP protocol driver properties, follow these steps:

Step 1 Choose Start > Run.

Step 2 In the Run field, type \WINNT\system32\drivers.

Step 3 Click OK.

Step 4 Right-click cdp.sys.

Step 5 Choose Properties to show CDP driver properties.

Step 6 Click OK.

Updating an IP Address for the CDP Protocol Driver

The CDP protocol driver runs on top of the existing Ethernet network interface card. You can restart CDP when a new IP address is configured at run time.

To update the CDP protocol driver, restart CDP using the Windows Device Manager to update the CDP driver with the new IP address information. You do not have to reset the system after updating.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

5-3

Chapter 5 Cisco Discovery Protocol Support

Locating Updated CDP Driver and Interface Files

Installing Cisco CRS updates these components:

• The CDP driver (cdp.sys) updates to the Windows 2003 driver directory

(WINNT\System32\Drivers\cdp.sys).

• The CDP Interface Library (cdpintf.dll) updates to the Windows 2003 System32 directory

(\WINNT\System32\cdpintf.dll).

• A Backup Regedit export file for reinstalling CDP registries updates to the bin directory (\Program

Files\Cisco\Bin\cdp2k101.reg). Use this file to restore the CDP registry in case it becomes

corrupted. This file restores the CDP registry to the HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\CDP directory.

• After running the cdp2k101.reg file, you must reset the system to restore the CDP registries.

Default CDP Settings

Table 5-1 shows the default CDP settings.

Ta b l e 5-1 Default CDP Setting Values

Description Default Value

Default Transmit Frequency 60 seconds

Default Time to Live 180 seconds

Default State CDP advertisement enabled

5-4

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Cisco Support Tools

This section contains the following topics:

• About Cisco Support Tools with Cisco CRS, page 6-1

• Accessing Cisco Support Tools, page 6-1

About Cisco Support Tools with Cisco CRS

Cisco Support Tools can help you manage and troubleshoot the Cisco CRS servers. Cisco Support Tools is a suite of utilities, but not every utility in the suite is supported by Cisco CRS.

Cisco CRS supports two components of Cisco Support Tools:

• Node Agent Service

The Node Agent Service is bundled with the CRS installer and is automatically installed on every CRS machine when you install the CRS software.

• Server

The Support Tools Server must be installed separately on a different machine. It provides a web server and the Support Tools Dashboard user interface.

Accessing Cisco Support Tools

Although the Support Tools Node Agent Service is automatically installed with CRS, before you can use it, you also need the Support Tools Server software. You must purchase the server software separately from your Cisco Representative.

The documentation that provides instructions on how to use Cisco Support Tools can be found on the Cisco website at

http://www.cisco.com/en/US/products/ps5905/tsd_products_support_series_home.html

When you purchase Support Tools from Cisco, the documentation is also included on the CD with the Support Tools Server software.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

6-1

Accessing Cisco Support Tools

Chapter 6 Cisco Support Tools

6-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

ART

Troubleshooting

Diagnosing and Correcting Cisco CRS Problems

The troubleshooting section describes problems that you might encounter when using the Cisco Customer Response Solutions (CRS) system. For each problem, this manual lists symptoms, possible causes, and corrective actions that you can take.

This section assumes that you are familiar with the CRS Administration web interface, CRS trace and log files, and various Windows administrative tasks. For more information, refer to the Cisco Customer Response Solutions Administration Guide and your Windows documentation.

This chapter contains the following topic:

• General Troubleshooting Steps, page 7-1

General Troubleshooting Steps

The following troubleshooting steps can help you diagnose most problems with your Cisco CRS products:

CHA PTER

Step 1 Verify that Cisco Unified Communications Manager is running.

Step 2 Verify that the Cisco CRS Node Manager service is registered.

Step 3 Verify that you uploaded the application.aef files to the repository using the Script Management page

and that you refreshed the CRS Engine after making a change to an application.

Step 4 Refer to the Release Notes for known problems.

Step 5 Verify that the Cisco CRS Node Manager service is running under a user account with Administrator

privileges.

Step 6 Stop and start the Internet Information Server (IIS).

Step 7 Save log files to prevent them from being overwritten.

Step 8 Save the application (.aef) file.

Step 9 Before debugging CRS Administration problems, turn on the Debugging trace level option for the ADM

subfacility.

The detailed output will be in the following file:

c:\program files\wfavvid\log\MADM\jvm.stdout

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

7-1

General Troubleshooting Steps

The error output will be in the following file:

c:\program files\wfavvid\log\MADM\jvm.stderr

Chapter 7 Diagnosing and Correcting Cisco CRS Problems

7-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Troubleshooting Tips

The following sections provide help in correcting problems with Cisco CRS software.

If you experience problems when using the Cisco Agent Desktop or the Cisco Supervisor Desktop, see the Troubleshooting section of the Cisco CAD Service Information Guide book, located at

http://www.cisco.com/univercd/cc/td/doc/product/voice/sw_ap_to/apps_5_0/english/agents/cad641si.p df.

If you are using Unified CCX with Unified ICME as part of the IPCC Gateway Solution and you experience any problems, see the troubleshooting information in the

Guide

Note The following troubleshooting tips are also accessible from the CRS Administration user interface. To

access them from the Main menu, select Tools > Troubleshooting Tips.

The tips are divided into the following categories:

• Installation Problems, page 8-2

Cisco IPCC Gateway Deployment

• Backup, Restore, and Update Problems, page 8-2

• CME Telephony subsystem problems, page 8-8

• Cisco Unified Communications Manager Automated Attendant problems, page 8-9

• Cisco Unified Communications Manager Express Problems, page 8-10

• Cisco Unified CCX Problems, page 8-10

• Cisco Unified IP IVR Problems, page 8-17

• CRS Administration Problems, page 8-18

• CRS Admin Utility Problems, page 8-23

• CRS Database Problems, page 8-25

• CRS Editor Problems, page 8-28

• CRS Engine Problems, page 8-29

• CRS Real-Time Reporting Problems, page 8-40

• CRS Historical Reporting Problems, page 8-41

• Automatic Speech Recognition (ASR) Problems, page 8-53

• Outbound Problems, page 8-56

• Text-to-Speech (TTS) Problems, page 8-60

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-1

Chapter 8 Troubleshooting Tips

Installation Problems

• Serviceability Problems, page 8-64

• CRS Internationalization Problems, page 8-68

• VXML Problems, page 8-69

• High Availability and Bootstrap, page 8-71

• High Availability and Failover, page 8-72

• VoIP Monitor Problems, page 8-76

Installation Problems

This section contains the following troubleshooting tips on installation problems:

• One node on a CRS 5.0 two-node cluster crashes beyond repair, page 8-2

One node on a CRS 5.0 two-node cluster crashes beyond repair

Symptom You have a CRS 5.0 two-node cluster and one node crashes beyond repair.

Error Message None

Possible Cause The cause is unknown.

Recommended Action For information on installation instructions, see the Cisco Customer Response

Solutions Installation Guide.

Do the following:

Step 1 If necessary, switch the DB publisher from the crashed node to the working node.

Step 2 Remove the crashed node from the cluster by executing the Remove option in Control Center Server

Configuration page in the CRS Application Administration web interface on the non-crashed node.

Step 3 For instructions, see "Removing a Server" in the Cisco Customer Response Solutions Administration

Guide.

Step 4 Re-image the crashed node.

Step 5 Re-install CRS on the crashed node, and execute the Add To Cluster selection as part of the CRS

Administration configuration.

Backup, Restore, and Update Problems

This section contains the following troubleshooting tips on Backup, Restore, and Update problems:

• Backup, Restore, and Upgrade cannot be started from a client desktop, page 8-3

• During Backup, Restore, or Upgrade, an exception is seen in UI, page 8-3

• Backup failed for a One or Two-Node system, page 8-4

• CRS 4.5 profile name is missing, page 8-4

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-2

Chapter 8 Troubleshooting Tips

Backup, Restore, and Update Problems

• Page Not Found message is displayed during Restore or Upgrade, page 8-4

• Restore fails due to a file not being found, page 8-5

• Restore failed for a one-node system, page 8-5

• Restore failed on a two-node system that had run before the Restore, page 8-6

• Restore failed on a two-node system that was re-imaged, page 8-7

• Some RmCm configuration is missing after Upgrade, page 8-8

Backup, Restore, and Upgrade cannot be started from a client desktop

Symptom Backup, Restore, and Upgrade cannot be started from a client desktop.

Error Message Backup and Restore or Upgrade displays an exception or a 'Page Not

Found' message.

Possible Cause When an exception is displayed by Backup and Restore or Upgrade, please check the logs

in the c:\program files\wfavvid\log\MCVD folder and search for the keyword: 'backup_fail'. This will show the cause of the failure. If the message is not understandable, please contact TAC. If the 'Page Not Found' message is displayed, please contact TAC since the CRS Node Manager service has restarted for some reason.

Recommended Action Start Backup and Restore or Upgrade from the CRS server desktop.

During Backup, Restore, or Upgrade, an exception is seen in UI

Symptom During Backup, Restore, or Upgrade, an exception is seen.

Error Message To see the error message, open the C:\Program

Files\wfavvid\log\MCVD\MCVDXXX.log where the time of the failure occurred. Search for the keyword: 'BACKUP_FAILED', 'RESTORE_FAILED', or 'UPGRADE_FAILED' based on type of failure. An exception with stack trace will be shown next to this text.

From the error message, go down to the last exception shown and look for the following keyword to see which component failed:

• com.cisco.archive.* - Indicates general issue with ArchiveManager.

• com.cisco.archive.impl.component.config.* - Indicates issue with saving or restoring configuration

such as properties files.

• com.cisco.database.* - Indicates issue with database.

• com.cisco.wf.spanlinkBackupRestore.* - Indicates issues with Spanlink components.

• com.cisco.wf.jtapi.archive.* - Indicates issue with JTAPI configuration.

• com.cisco.wf.cme.archive.* - Indicates issue with CME configuration.

• com.cisco.restoreadmin.jtapiresyncwizard.* - Indicates issue with JTAPI wizard synchronization of

Route Points, CTI Ports.

• com.cisco.restoreadmin.cmevalidate.* - Indicates issues with CME validation wizard.

Recommended Action Please contact TAC based on information in Cause to find appropriate specialist.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-3

Backup, Restore, and Update Problems

Backup failed for a One or Two-Node system

Symptom Backup failed for a one or two-node system.

Error Message Backup and Restore displays an exception or a 'Page Not Found' message.

Possible Cause When an exception is displayed by Backup and Restore, please check the logs in the

c:\program files\wfavvid\log\MCVD folder and search for the keyword: 'backup_fail'. This will show the cause of the failure. If the message is not understandable, please contact TAC. If the 'Page Not Found' message is displayed, please contact TAC since the CRS Node Manager service has restarted for some reason.

Recommended Action Check and make sure that the 'Backup Storage Location' is set correctly with the

right credentials.

CRS 4.5 profile name is missing

Symptom You are prompted to select the CRS 4.5 profile name during the 4.5 to 5.0 upgrade, but that

name is missing from the pulldown menu list during restore.

Chapter 8 Troubleshooting Tips

Error Message None

Possible Cause This can happen if an initial attempt to restore was unsuccessful due to the CRS Node

Manager abruptly shutting down or restarting. The CRS 4.5 profile name may have been migrated to the 5.0 cluster ID (that is, a long number) during the restore attempt. However, because the CRS Node Manager service abruptly shut down, the name itself remained migrated in the CCM table.

Recommended Action When prompted for the CRS 4.5 profile name, please choose the long integer

number from the menu pulldown list.

Page Not Found message is displayed during Restore or Upgrade

Symptom During Restore or Upgrade, the message "Page Not Found" is displayed.

Error Message Check the log in c:\program files\wfavvid\log\MCVD and search for

keyword 'reboot_on' for the error message.

Possible Cause Most likely, the CRS Node Manager has restarted during Restore or Upgrade due to an

abnormal shutdown. This can be checked if you see a new MCVD log file (in the c:\program files\wfavvid\log\MCVD folder) created during the time of the restore or if you see that the CRS Node Manager Service is no longer running.

Recommended Action

8-4

Step 1 Analyze the 'reboot_on' message to see which service went down. If this is intermittent issue, redo the

Restore by first doing the following two steps. Contact TAC if condition persists

Step 2 Stop the CRS Node Manager Service if it is running.

Step 3 Replace the c:\program files\wfavvid\ClusterData folder with the original ClusterData folder that was

copied before the restore.

Step 4 Remember to keep the original copy around just in case the process needs to be repeated.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

If Restore still fails, please contact TAC.

Restore fails due to a file not being found

Symptom Restore fails due to "<file-name> file is not found."

Error Message "<file-name> file is not found" message is displayed in the Restore

Pop-up UI.

Possible Cause The <file-name> file cannot be found by the Restore process.

Recommended Action Do the following:

Step 1 Delete the staging directory (C:\STI).

Step 2 Manually stop the Node Manager Service by selecting Start > Programs > Administrative Tools >

Services, and then stopping the 'Cisco CRS Node Manager' service.

Backup, Restore, and Update Problems

Step 3 Delete the C:\Program Files\wfavvid\ClusterData folder.

Step 4 Copy C:\BackupClusterData\ClusterData folder (which was backed up before) to C:\Program

Files\wfavvid\.

Step 5 Reboot the machine.

Step 6 Re-run the restore.

Restore failed for a one-node system

Symptom

Error Message Backup and Restore displays an exception or a 'Page Not Found' message.

Possible Cause When an exception is displayed by Backup and Restore, please check the logs in the

Recommended Action Check failure. If failure is irrecoverable, please contact TAC. If failure is due to

intermittent issue, retry restore by doing one of the following.

If you have a copy of original ClusterData folder:

Step 1 Stop the CRS Node Manager Service.

Step 2 Remove the C:\Program Files\wfavvid\ClusterData folder.

Step 3 Copy the original ClusterData folder to the C:\Program Files\wfavvid\ folder.

Step 4 Start the CRS Node Manager Service.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-5

Chapter 8 Troubleshooting Tips

Backup, Restore, and Update Problems

Step 5 Redo the restore.

If you don't have a copy of original ClusterData folder:

Step 1 Reinstall the CRS server using Win2K3 OS.

Step 2 Fresh install the server using the CRS installer.

Step 3 Rerun the Restore.

Restore failed on a two-node system that had run before the Restore

Symptom Restore failed on a two-node system. The system had already been configured as a cluster and

was running successfully before the restore.

Error Message Backup and Restore displays an exception or a 'Page Not Found' message.

Possible Cause When an exception is displayed by Backup and Restore, please check the logs in the

Recommended Action Check failure. If failure is irrecoverable, please contact TAC. If failure is due to

intermittent issue, retry restore by doing the following:

Step 1 Shutdown the CRS Node Manager Service on both nodes.

Step 2 You must have a copy of ClusterData folder saved on both nodes:

a. Remove the C:\Program Files\wfavvid\ClusterData folder on both nodes.

b. Copy the original ClusterData folder to the C:\Program Files\wfavvid folder on both nodes.

Step 3 If you don't have a copy of ClusterData folder saved on both nodes:

a. Reinstall both CRS servers using Win2K3 OS.

b. Fresh install both servers using CRS installer.

c. Rerun the Restore.

Step 4 Restart CRS Node Manager on both nodes.

Step 5 Rerun the Restore again.

8-6

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Restore failed on a two-node system that was re-imaged

Symptom Restore failed on a two-node system. The system was re-imaged and had not been configured

to run as a cluster.

Error Message Backup and Restore displays an exception or a 'Page Not Found' message.

Possible Cause When an exception is displayed by Backup and Restore, please check the logs in the

Recommended Action Check failure. If failure is irrecoverable, please contact TAC. If failure is due to

intermittent issue, retry restore by doing the following:

Step 1 Shutdown CRS Node Manager Service on both nodes.

Step 2 If you have a copy of ClusterData folder saved on both nodes:

a. Remove the C:\Program Files\wfavvid\ClusterData folder on both nodes.

Backup, Restore, and Update Problems

b. Copy the original ClusterData folder to the C:\Program Files\wfavvid folder on both nodes.

c. On the 2nd node, use regedit and check to make sure the 'com.cisco.cluster.node.id' is set to '1' for

the following:

\HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems, Inc.\CRS\Properties\application.MADM.properties

Step 3 If you do not have a copy of the ClusterData folder saved on both nodes:

a. Reinstall both CRS servers using Win2K3 OS.

b. Fresh install both servers using CRS installer.

c. Rerun the Restore.

Step 4 Restart the CRS Node Manager on both nodes.

Step 5 Rerun the Restore again.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-7

CME Telephony subsystem problems

Some RmCm configuration is missing after Upgrade

Symptom After the 4.5 to 5.0 Upgrade is successful, the system is missing some RmCm configuration

(that is, resource skills group, CSQ configuration, and so on).

Error Message None

Possible Cause This can happen when an Upgrade was initially triggered, but failed due to the CRS Node

Manager restarting in the middle of the Restore. During the successful attempt for Restore, the CRS

4.5 user profile name has already been changed to a long integer by the 1st attempt to Restore.

Recommended Action To reset the profileID = 1 for the default profilename, do the following:

Step 1 Open the SQL query analyzer and type the following:

a. Run SELECT * FROM db_cra.dbo.profileIDMapping

You should see 2 records (one from 4.5 and the default for 5.0). Note the CRS4.5_profilename which is NOT the default. You will need this.

b. Run DELETE FROM db_cra.dbo.profileIDMapping where profileName='CRS4.5_profilename

Make sure you see 1 row affected in the result window after executing the preceding command.

c. Run UPDATE db_cra.dbo.profileIDMapping SET profileID =1

Chapter 8 Troubleshooting Tips

Make sure you see 1 row affected after executing the preceding command.

Step 2 SELECT * FROM db_cra.dbo.profileIDMapping

Step 3 You should get only one record with profilename 'default' and profileID=1

CME Telephony subsystem problems

This section contains the following troubleshooting tips on CME Telephony problems:

• A functional routing point stopped working or the CME Telephony subsystem is in partial service,

page 8-8

A functional routing point stopped working or the CME Telephony subsystem is in partial service

Symptom A functional routing point stopped working or the CME Telephony subsystem is in partial

service

Error Message None

Possible Cause This can happen if some one manually deleted the route point DN on the router side.

8-8

Recommended Action Run the configuration validator tool and check for any warning messages. A report

should indicate that there is an RP DN mismatch. Check on the router configuration whether or not the DN exists or not. If confirmed, then visit the Trigger page and click the update button to fix and recreate the DN on the router side.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Cisco Unified Communications Manager Automated Attendant problems

This section contains the following troubleshooting tips on Cisco Unified Communications Manager (Unified CM) Automated Attendant (AA) problems:

• Dial by name does not find the specified server, page 8-9

• Automated Attendant prompt is not played, page 8-9

Dial by name does not find the specified server

Symptom The Cisco Unified Communications Manager Automated Attendant cannot find a user that a

caller specifies when dialing by name.

Error Message None.

Possible Cause The extension of the requested user is not valid because the user does not have a primary

extension assigned in Cisco Unified Communications Manager, or the ccndir.ini file is missing information.

Recommended Action Complete the following steps:

Step 1 In the Cisco Unified Communications Manager User Information web page, verify that the user has an

entry in the AutoAttendant Dialing field, that the User record has an associated phone, and that the Primary Extension radio button is selected.

Step 2 On the CRS server, verify that the ccndir.ini file contains the correct userbase and profilebase

information. For example:

# Base DN for CCN APPS CCNAPPSBASE "ou=CCN Apps, o=cisco.com" # CCN Cluster Profile name CCNCLUSTERPROFILE "johndoe_test" # Base DN for Users USERBASE "ou=Users, o=cisco.com"

Automated Attendant prompt is not played

Symptom The Cisco Unified Communications Manager Automated Attendant prompt is not played.

Error Message None.

Possible Cause An incorrect welcome prompt is specified in the welcomePrompt field in the Cisco Script

Application web page.

Recommended Action From the CRS Administration web page, choose Applications > Prompt

Management. Click the Upload New Prompts link to upload the Welcome prompt.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-9

Chapter 8 Troubleshooting Tips

Cisco Unified Communications Manager Express Problems

This section contains the following troubleshooting tips on Cisco Unified Communications Manager Express (Unified CME) problemss:

• Agent cannot log in on shared line, page 8-10

• Agent cannot log in on restricted line, page 8-10

• When agent drops from conference, all parties on conference are dropped, page 8-10

Agent cannot log in on shared line

Symptom Agent is unable to log in on the Cisco Agent Desktop (CAD).

Error Message CAD displays Unable to login agent because line is shared.

Possible Cause The extension specified during login is a shared extension.

Recommended Action Make sure the extension only exists on one device.

Agent cannot log in on restricted line

Symptom Agent is unable to log in on CAD.

Error Message CAD displays The line is restricted.

Possible Cause The extension specified during login cannot be monitored.

Recommended Action Make sure the agent's extension is configured with "allow-watch" on Unified CME.

When agent drops from conference, all parties on conference are dropped

Symptom When an agent drops from a conference, all parties on the conference are dropped as well.

Error Message None.

Possible Cause The agent device is not configured with "keep-conference."

Recommended Action Make sure the agent's device is configured with "keep-conference" on Unified

CME.

Cisco Unified CCX Problems

8-10

This section contains the following troubleshooting tips on Cisco Unified Contact Center Express (Unified CCX) problems:

• RmCm subsystem is out of service, page 8-11

• RmCm subsystem remains INITIALIZING, page 8-11

• RmCm remains in Initializing state, page 8-12

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

• Agents, Supervisors, or Teams are out of synch, page 8-12

• Agent or CSQ does not appear in Cisco Desktop Administrator (CDA), page 8-12

• Agents do not appear in the Resources area in the Unified CCX Configuration web page, page 8-13

• You cannot select the order of agents, page 8-13

• Agent does not go to Work state after handling a call, page 8-13

• A media step causes a Could not create PlayPromptDialog Object exception, page 8-14

• Unable to make any Unified CCX configuration changes, page 8-14

• Some resource selection criteria are missing, page 8-14

• Unable to record an agent, page 8-15

• Sometimes the supervisor can monitor and record an agent and sometimes he cannot, page 8-15

• Calls to Unified CCX route points are disconnected, page 8-15

• Calls are not routed to agents, page 8-15

• Agents do not show in a CSQ, page 8-16

• Caller gets dropped when an attempt is made to recall a Unified CCX agent extension after the agent

Cisco Unified CCX Problems

previously parked the call, page 8-16

• Updating a NIC driver disables silent monitoring and recording, page 8-16

RmCm subsystem is out of service

Symptom The Resource Manager Contact Manager (RmCm) subsystem is out of service.

Error Message None.

Possible Cause The RM JTAPI user in Cisco Unified Communications Manager is not configured

properly.

Recommended Action Complete the following steps:

Step 1 From the CRS Administration web page, choose Subsystems > RmCm.

Step 2 Click the RM JTAPI Provider hyperlink.

Step 3 Make sure that the information in the RM JTAPI User ID and Password fields matches the information

for the RM JTAPI user in Cisco Unified Communications Manager.

RmCm subsystem remains INITIALIZING

Symptom The Resource Manager Contact Manager (RmCm) subsystem remains in INITIALIZING state.

Error Message None.

Possible Cause Could not load the default scripts CM.aef and RM.aef.

Complete the following steps:

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-11

Cisco Unified CCX Problems

Step 1 Check the RM JTAPI provider configuration and then stop and restart the CRS engine.

Step 2 Check to be sure the workflow scripts CM.aef and RM.aef are present on the Script Management page

on CRS Administration. They are needed for the RmCm subsystem to be in service. If either of these scripts are deleted, missing, or corrupted, the RmCm subsystem will not go IN SERVICE. To recover, upload new copies of these scripts from C:\Program Files\wfavvid\scripts\system\default\rmcm, and restart the CRS Engine.

RmCm remains in Initializing state

Symptom The RmCm subsystem remains in the Initializing state after the CRS Engine starts.

Error Message None.

Possible Cause The RmCm subsystem is unable to read any configuration information.

Recommended Action Make sure there is at least one "CRS SQL Server - Config" service running in the

cluster. If the service is stopped, start it.

Chapter 8 Troubleshooting Tips

Agents, Supervisors, or Teams are out of synch

Symptom Agents, Supervisors, or Teams are out of synch between Unified CCX and the Cisco Desktop

Administrator (CDA).

Error Message None.

Possible Cause The automatic synchronization between Unified CCX and the CDA failed.

Recommended Action Launch a manual synch from the CDA by selecting the Logical Call Center (usually

the CRS profile name) and clicking Setup > Synchronize Directory Services.

Agent or CSQ does not appear in Cisco Desktop Administrator (CDA)

Symptom After adding an agent or a contact service queue (CSQ) in CRS Administration, the agent or

the CSQ does not appear in the CDA.

Error Message None.

Possible Cause The RmCm subsystem has not synchronized the agents.

Recommended Action Go to the Resources link under Subsystems > RmCm. This will force the RmCm

subsystem to synchronize the agents.

8-12

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Cisco Unified CCX Problems

Agents do not appear in the Resources area in the Unified CCX Configuration web page

Symptom No agents appear in the Resources area in the Unified CCX Configuration web page.

Error Message None.

Possible Cause To appear as an agent in this area, a user must be configured as a Unified CCX agent in

the Cisco Unified Communications Manager User Information web page.

Recommended Action In Cisco Unified Communications Manager, verify configuration information in the

User Information web pages. For each user, under Associated Devices, verify that a phone is associated, and verify that the Unified CCX extension radio button is selected.

You cannot select the order of agents

Symptom When you configure a resource group, the system does not allow you to select the order of

agents.

Error Message None.

Possible Cause You order agents at the CSQ level.

Recommended Action When you configure the CSQ and select the desired Resource Group, click Show

Resources and order the agents as desired.

Agent does not go to Work state after handling a call

Symptom An agent does not go to Work State after handling a call, even though the CSQ is configured

with Auto Work turned on.

Error Message None.

Possible Cause An agent will not go to Work State after handling a call if the agent presses the Ready

button while in Talk state. In addition, if the agent services multiple CSQs, Auto Work may not be configured for each CSQ. The agent will only go to Work State if the call comes from a CSQ where Auto Work is enabled.

Recommended Action None.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-13

Chapter 8 Troubleshooting Tips

Cisco Unified CCX Problems

A media step causes a Could not create PlayPromptDialog Object exception

Symptom Any media step except SendDigitString causes the following exception in the CRS trace files.

Error Message Could not create PlayPromptDialog Object:

Exception=com.cisco.channel.ChannelUnsupportedException: com.cisco.dialog.PlayPromptDialog is not supported.

Possible Cause A Primary Dialog Group was not specified when a trigger was defined.

Recommended Action After you add an application in the CRS Application Configuration web page, you

must define a trigger. When you define a trigger for the application, you must define both a Call Control Group and a Primary Dialog Group in the JTAPI Trigger Configuration window.

Unable to make any Unified CCX configuration changes

Symptom When trying to save Unified CCX configuration changes, CRS Administration shows an error

message.

Error Message There was an error reading/updating the database. Please contact your administrator.

Possible Cause All "CRS SQL Server - Config" services need to be IN-SERVICE in order to make

Unified CCX configuration changes. If one or more services are down, no Unified CCX configuration update is allowed.

Recommended Action Check the state of all "CRS SQL Server - Config" services in the cluster. If a service

is stopped, start it. Make sure the "CRS Config Datastore" component is activated.

Some resource selection criteria are missing

Symptom When trying to configure a CSQ, CRS Administration does not show all the resource selection

criteria.

Error Message None.

Possible Cause The CSQ is resource-group based. A resource-group based CSQ has Longest Available,

Most Handled Contact, Shortest Average Handle Time, Linear and Circular criteria. A skills-based CSQ has Longest Available, Most Handled Contact, Shortest Average Handle Time, Most Skilled, Least Skilled, Most Skilled by weight, Least Skilled by Weight, Most Skilled by Order, and Least Skilled by Order criteria.

Recommended Action You might want to use a skills-based CSQ in order to use a specific resource

selection criteria.

8-14

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Cisco Unified CCX Problems

Unable to record an agent

Symptom A supervisor is unable to record an agent's call. Clicking on Record pops up a message dialog

box.

Error Message Unable to record agent.

Possible Cause The recording count is set to 0.

Recommended Action Go to CRS Administration. Select System > System Parameters and set the

number of the recording count appropriately.

Sometimes the supervisor can monitor and record an agent and sometimes he cannot

Symptom Sometimes the supervisor can monitor and record an agent and sometimes he cannot.

Error Message None

Possible Cause Currently, CAD supports only the G.711 and the G.729 codeces. If your codex setting is

different in the Cisco unified Communications Manager, for example, if your setting is G.722, then you can experience these problems.

Recommended Action Make sure you have disabled "Advertise G 722 codex" on the agent phone and make

sure your settings in Unified CM are for the G.711 or the G.,729 codex. Although Unified CM 6.0 supports the G 722 codex, CAD does not.

Calls to Unified CCX route points are disconnected

Symptom Callers are disconnected when calling Unified CCX route points.

Error Message None.

Possible Cause The CSQ parameter is not correctly defined in the Cisco Script Application web page.

Recommended Action From the CRS Administration web page, chooseApplications > Application

Management, click the name of the script that corresponds to Unified CCX, and then enter the name of the configured CSQ in the CSQ field.

Calls are not routed to agents

Symptom Calls are not routed to agents even though the agents are configured with the skills of the CSQ.

Error Message None.

Possible Cause The skill levels of the agents are not equal to or higher than the skill levels of the CSQ.

Recommended Action Click Show Resources on the CSQ configuration page to determine that agents are

part of the CSQ. If agents do not appear, verify that the skill levels of the agents are equal to or higher than the skill levels of the CSQ.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-15

Chapter 8 Troubleshooting Tips

Cisco Unified CCX Problems

Agents do not show in a CSQ

Symptom A CSQ is configured with a group of agents for Skill A and a group of agents for Skill B;

however, the agents do not show up in the CSQ.

Error Message None.

Possible Cause Agents do not have all the skill levels of the CSQ or the skill level of the agents do not

have equal or higher skill levels than that of the CSQ.

Recommended Action Verify that agents have all the skill levels of the CSQ and that the agents have equal

or higher skill levels than that of the CSQ.

Caller gets dropped when an attempt is made to recall a Unified CCX agent extension after the agent previously parked the call

Symptom Agent A gets a Unified CCX call and parks that call. After the parked call times out, a recall

is attempted to the Agent A extension (if no other agent has picked up the call). If Agent A is busy handling another call on that Unified CCX extension while the previously parked call is being routed, the caller gets dropped.

Error Message None.

Possible Cause A parked call gets dropped if an attempt is made to place the call again to a busy line that

is not set with forward busy.

Recommended Action Configure the Unified CCX extension of Agent A withForward Busy to a

non-Unified CCX line on the same phone.

Also, configure this line as Forward Busy to the Unified CCX route point. When an attempt is made to recall the Unified CCX extension of Agent A, the call is forwarded to the non-Unified CCX line if the extension is busy. If the non-Unified CCX line is busy, the call is forwarded to the Unified CCX route point and gets queued again instead of being dropped. You can set up the workflow of the Unified CCX route point to increase the priority of the call.

Updating a NIC driver disables silent monitoring and recording

Symptom After updating a network interface card (NIC) driver, the Cisco Supervisor Desktop and Cisco

Agent Desktop Silent Monitoring and Recording features do not work.

Error Message None.

Possible Cause This problem can occur if you have updated a NIC driver on a server on which you

checked the VoIP Monitor Server check box during the installation of Cisco CRS.

Recommended Action Reinstall Cisco CRS. Make sure to check the VoIP Monitor Server check box in the

Component Distribution pane.

8-16

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Cisco Unified IP IVR Problems

This section contains the following troubleshooting tips on Unified IP IVR problems:

• Cisco Unified IP IVR drops callers when transferring to an extension, page 8-17

• Prompts play in language, page 8-17

• Some prompts do not play, page 8-18

• Some prompts in a script play in the language specified and other prompts play in English, page 8-18

• A prompt plays phrases in the wrong order, page 8-18

Cisco Unified IP IVR drops callers when transferring to an extension

Symptom After Cisco Unified IP IVR transfers a call to an extension, the called party hears a busy signal

when taking the call and the caller is dropped.

Error Message None.

Possible Cause If a call gets dropped, one potential cause is a codec mismatch between the endpoint and

the CRS Server. The CRS Server supports either the G.729 or the G.711 protocol, but not both simultaneously. To support these protocols, a transcoder is required.

Recommended Action Install a transcoder for Cisco Unified IP IVR.

Prompts play in language

Symptom A script was assigned to a language at the route point but it plays prompts in another language.

Error Message None.

This problem can be caused by the following situations:

• The system default language is set incorrectly.

• The language specified in the Set Contact step is incorrect.

• The language specified in the Play Prompt step is incorrect.

Recommended Action Verify that system default language is set correctly. Verify that the correct language

is set in the Set Contact step or the Play Prompt step if these steps are used.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-17

Chapter 8 Troubleshooting Tips

CRS Administration Problems

Some prompts do not play

Symptom A prompt in a script does not play. The script may or may not continue executing.

Error Message None.

Possible Cause A prompt is missing in the language directory for the language used by the script. By

default, the Play Prompt step is set to continue if it encounters an error and the script will continue to play if it encounters a missing prompt. If you have changed the Play Prompt step to not continue if it encounters an error, the script will stop executing.

Recommended Action Refer to the CRS trace files to find the missing prompt. Provide the missing prompt

in the language folder shown in the CRS trace files.

Some prompts in a script play in the language specified and other prompts play in English

Symptom A script is set to a language other than US English, but some prompts play in US English.

Error Message None.

Possible Cause A prompt is missing in the language directory for the language used by the script. If the

default language for the script uses the same rules as US English, the system will automatically replace the missing prompt with a US English prompt.

Recommended Action Refer to the CRS trace files to find the missing prompt and provide the missing

prompt in the language folder shown in the CRS trace files.

A prompt plays phrases in the wrong order

Symptom A prompt played by the Create Generated Prompt step plays the correct language but plays

phrases in the wrong order. For example, a prompt that you expect to play as “month, day, year” plays as “year, month, day.”

Error Message None.

Possible Cause The Create Generated Prompt step is using incorrect rules for the language.

Recommended Action If creating a new language or adapting an existing language for a new locale, check

the PromptGenerator.properties file and make sure that it is using the correct rules for the language.

CRS Administration Problems

8-18

This section contains the following troubleshooting tips on CRS Administration problems:

• The CRS Administration Authentication web page is not available, page 8-19

• Uploading a license file can result in a warning message, page 8-20

• User cannot log in to the CRS web page, page 8-20

• Refreshing subflow script does not update parent scripts, page 8-20

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

CRS Administration Problems

• Unified Communications Manager users display in random order, page 8-20

• CRS Supervisor web page cannot be viewed from CRS Server, page 8-21

• Database table fields used by wallboard store data in milliseconds, page 8-21

• Management pages display error message when selected, page 8-21

• Zip file does not auto unzip on Document Management page, page 8-22

• Invalid files message displays while uploading a zip file of prompts, page 8-22

• A Component Manager goes into partial service when uploading a zip file, page 8-23

• High call rejection rate under heavy load, page 8-23

The CRS Administration Authentication web page is not available

Symptom You cannot browse to the CRS Administration URL and a Page Cannot be Displayed error

appears.

Error Message None.

Possible Cause The system cannot access the CRS Administration web page. A required service may not

be running or required files may be missing.

Make sure the following services are running:

• Check that the CRSJavaAdmin.exe is running (in Windows Task Manager).

• IIS Admin service

• World Wide Web Publishing service

Recommended Action If these services are running, verify that files exist in the

install_directory\tomcat_appadmin\webapps\appadmin\ directory, where install_directory is the folder in which the CRS system is installed. (By default, the CRS system is installed in the c:\Program Files\wfavvid folder.)

If no files exist in this directory, perform the following steps:

Step 1 Stop the Cisco CRS Node Manager service.

Step 2 Delete the appadmin folder from the tomcat_appadmin\webapps folder in the folder in which you

installed the CRS system. (By default, the CRS system is installed in the c:\Program Files\wfavvid folder.)

Step 3 Start the Cisco CRS Node Manager service.

Step 4 Wait for a few minutes and try to browse to the URL again.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-19

CRS Administration Problems

Uploading a license file can result in a warning message

Symptom The user gets a warning message when uploading license files using CRS Administration.

Error Message The license file, <filename>, if uploaded will change the package from <existing license

package> to <new license package>. Please click OK to continue or CANCEL to abort.

Possible Cause This warning is only displayed if a user tries to upload licenses which change the existing

license package of the CRS cluster to a different package.

Recommended Action The user needs to determine if he or she really wants to change the license package

as described in the warning message. If yes, clicking OK will change the package. If it was a user error, clicking CANCEL will keep the license package unchanged.

User cannot log in to the CRS web page

Symptom A user cannot log in to the Cisco CRS web pages after the CRS Administration times out.

Error Message None.

Chapter 8 Troubleshooting Tips

Possible Cause If you perform no activity for 30 minutes, the CRS system automatically logs you out.

Recommended Action Log in again to continue.

Refreshing subflow script does not update parent scripts

Symptom Refreshing a subflow script does not update its parent scripts.

Error Message None.

Possible Cause If a script is referenced in other scripts, refreshing a subflow script does not update its

parent scripts.

Recommended Action Manually refresh all parent scripts.

Unified Communications Manager users display in random order

Symptom On the CRS User Maintenance window, Unified Communications Manager users display in

random order when the number of users returned during the search is greater than 75.

Error Message None.

Possible Cause The User Maintenance window on Cisco CRS limits the display to 75 Unified

Communications Manager users. If that number is exceeded, the Unified Communications Manager users display in a random order. Instead of the usual logical order of 39001, 39002, 39003, 39004, the list contains 39001, 39003, with 39002 and 39004 not shown.

8-20

Recommended Action Narrow the search by adding additional characters to be matched.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

CRS Administration Problems

CRS Supervisor web page cannot be viewed from CRS Server

Symptom The Cisco CRS Supervisor web page cannot be viewed from the Cisco CRS Server.

Error Message None.

Possible Cause If the Cisco CRS Supervisor is running on a computer with the language set to Simplified

Chinese, you cannot view the Cisco CRS Supervisor web page from the Cisco CRS Server.

Recommended Action Use a client computer to view the Cisco CRS Supervisor web page.

Database table fields used by wallboard store data in milliseconds

Symptom Some database table fields used by a wallboard store data in milliseconds instead of in

HH:MM:SS.

Error Message None.

Possible Cause The avgTalkDuration, avgWaitDuration, longestTalkDuration, longestWaitDuration, and

oldestContact database table fields in the RtCSQsSummary and the RtICDStatistics database tables store date in milliseconds.

If you want to include information described by these fields on your wallboard, use the following fields, which store the same information but use the HH:MM:SS format:

• Instead of avgTalkDuration, use convAvgTalkDuration.

• Instead of avgWaitDuration, use convAvgWaitDuration.

• Instead of longestTalkDuration, use convLongestTalkDuration.

• Instead of longestWaitDuration, use convLongestWaitDuration.

• Instead of oldestContact, use convOldestContact.

Management pages display error message when selected

Symptom The Prompt Management, Grammar Management, or Document Management pages show an

error message when selected.

Error Message com.cisco.file.FileException: Unable to list files; Repository Data Store not initialized

Possible Cause This error occurs when there is no master selected for the Repository Datastore

component. This can happen due to one of the following reasons:

• Repository Datastore component is not activated at all in the cluster.

• Repository Datastore component activated node is not up or SQL Services are not running on this

node.

• In the case of high availability, the Repository Datastore component is activated, but the Publisher

activation has not yet completed from the Datacontrol Center page of CRS Administration.

Recommended Action To resolve the problem, do one of the following:

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-21

CRS Administration Problems

• To activate the Repository Datastore component, from CRS Administration select Control Center

> Component Activation page. Check to be sure the nodes with Repository Datastore components are up and running. If the nodes are up, check that all the SQL Services, including Microsoft SQL Server and Microsoft SQL Agent services are up and running.

• In the case of high availability, from CRS Administration, go to Datacontrol Center > Publisher

Activation page to check that the Publisher is activated . If active, you see the Publication Snapshot

Agent in STOPPED state and the Subscription Agent in RUNNING state.

Zip file does not auto unzip on Document Management page

Symptom On the Document Management page when a zip file is uploaded, it does not get unzipped

automatically as a zip file does on the Prompt Management and Grammar Management pages.

Error Message None.

Possible Cause While uploading a zip file in Document Management, the user has the option of storing

it as a zip file without unzipping or unzipping the file before it gets stored.

Recommended Action Be sure the Unzip after uploading check box is selected if that is the intention.

Chapter 8 Troubleshooting Tips

Invalid files message displays while uploading a zip file of prompts

Symptom Uploading a zip file of prompts (or grammars or documents) at the root level in Prompts

Management (or Grammar Management or Document Management) shows an error message in the MADM log files.

Error Message Invalid files...

Possible Cause This problem could occur for one of the following reasons:

• At the root level only language folders can exist.

• Prompt Management and Grammar Management pages except files of valid extension only.

Recommended Action To correct the problem, do the following:

• Check that your zip file does not contain any files that do not belong to a folder while uploading at

the root level.

• Check that all the files have a valid extension.

8-22

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

CRS Admin Utility Problems

A Component Manager goes into partial service when uploading a zip file

Symptom When uploading a file or zip file from Prompt Management, Grammar Management, or

Document Management in CRS Administration, the CRS Engine component Prompt Manager (or Grammar Manager or Document Manager) is shown in PARTIAL_SERVICE state.

Error Message PARTIAL_SERVICE

Possible Cause The Prompt Manager, Grammar Manager, or Document Manager are put in

PARTIAL_SERVICE by File Manager while it synchronizes the uploaded files from the Repository Datastore to the local disk. Once the synchronization is complete, they are put back into INSERVICE state.

Recommended Action None.

High call rejection rate under heavy load

Symptom With a heavy load of over 200 agents a high call rejection or aborted rate occurs.

Error Message None.

Possible Cause Writing the Unified Communications Manager and CTI Manager traces to the local drive

leads to call failures due to the increased load of tracing.

Recommended Action CTI Manager and Unified Communications Manager traces need to be directed to

another hard drive. Here is an example of how to set things up. Note that you need to create the directory structure shown in the F:\ drive:

Unified Communications Manager SDL Trace Directory Path = F:\Program Files\Cisco\ \Trace\SDL\

Unified Communications Manager SDI Trace output setting, File Name = F:\Program Files\Cisco\Trace\CCM\ccm.txt

CTIManager SDL Trace Directory Path = F:\Program Files\Cisco\ Trace\SDL\

CTIManager SDI Trace output setting, File Name = F:\Program Files\Cisco\Trace\CCM\cti.txt

The AntiVirus has been set not to scan the following folders:

C:\Program Files\Cisco\Trace\ F:\Program Files\Cisco\Trace\

CRS Admin Utility Problems

This section contains the following troubleshooting tips on CRS Admin Utility problems:

• The cluster is not in synchronization, page 8-24

• CRS Admin Utility exits or does not come up after login., page 8-24

• The CRS Admin Utility fails due to data corruption, page 8-24

• The CRS Admin Utility will not run on a none bootstrap node, page 8-25

• The CRS Admin Utility will not run since the Node Manager hung, page 8-25

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-23

CRS Admin Utility Problems

The cluster is not in synchronization

Symptom The cluster is not in synchronization.

Error Message Rerun the CRS Admin Utility again to bring the cluster back into

synchronization.

Possible Cause This can happen due to a variety of reasons: The password setting or synchronization

failed in the middle of the process. The update was done on a bootstrap data store but not on a Windows database. The update was done on one node but not on the second node.

Recommended Action Rerun the CRS Admin Utility again to bring the cluster back into synchronization.

CRS Admin Utility exits or does not come up after login.

Symptom CRS Admin Utility exits or does not come up after login.

Error Message None

Chapter 8 Troubleshooting Tips

Possible Cause The CRS Admin Utility runs on a system where CRS has been properly installed and

configured.

Possible Cause Check the status of the configuration manager (and the bootstrap manager for both nodes,

if you have a two node system) from the CRS Admin control center.

Possible Cause The CRS Admin Utility needs to have the CRS configuration manager In_service for a

single node system or the CRS configuration manager and bootstrap manager In_service for a multiple node system.

Recommended Action Properly install and configure the CRS system.

The CRS Admin Utility fails due to data corruption

Symptom The CRS Administration Utility fails due to data corruption.

Error Message None

Possible Cause The CRS Admin Utility fails when there is data corruption in the bootstrap data store.

Any process running on the node on which the Admin Utility is running and that has access to the bootstrap data store can cause data corruption. For example, the Admin Utility, the CRS engine, CONFINGAPI, and so on, many cause data corruption.

Recommended Action When the Admin Utility fails on one node due to data corruption, try to run it from

another bootstrap server node. If the Admin Utility fails agin, contact TAC.

8-24

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

The CRS Admin Utility will not run on a none bootstrap node

Symptom The CRS Admin Utility will not run on a none bootstrap node.

Error Message Local system is not a Bootstrap Server Node. Please run CRS Admin

Utility on a Bootstrap Server Node.

Possible Cause An attempt was made to run the CRS Admin Utility on a none bootstrap server node. ON

CRS 5.0 2-node systems, both nodes are bootstrap servers. A none bootstrap server will be a DB node or a VOP node.

Recommended Action You must run the CRS Admin Utility on a bootstrap server node. Only the bootstrap

server has the required bootstrap data storage installed.

The CRS Admin Utility will not run since the Node Manager hung

Symptom The CRS Admin Utility will not run since the Node Manager hung.

Error Message Failed to shut down remote bootstrap server Node Manager on

<server_name>. Please shut it down manually and try again.

CRS Database Problems

Possible Cause Anything running on the system could potentially hang the Node Manger and make it fail

to shutdown smoothy or the window could not bring the node manager down.

Recommended Action Manually shut down the Node Manager from the window service, or reboot the

system, or first power down the system and then reboot it.

CRS Database Problems

This section contains the following troubleshooting tips on CRS Database problems:

• Cannot configure Application or System parameters from their pages in CRS Administration,

page 8-26

• HR client login error, page 8-26

• Cannot activate DB components on HA node, page 8-26

• CRS Databases are not purged as expected, page 8-26

• Historical Database db_cra is full, page 8-27

• E-mail notification of database purging activities is not sent, page 8-27

• Syslog or SNMP trap notification of database purging activities is not sent, page 8-28

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-25

Chapter 8 Troubleshooting Tips

CRS Database Problems

Cannot configure Application or System parameters from their pages in CRS Administration

Symptom Cannot configure Application or System parameters from their pages in CRS Administration.

Error Message "ConfigException* occurred"

Possible Cause This can happen if CRS Config Datastore service is OOS when SQL Services or MSDTC

are not running.

Recommended Action Check in the control center that the Microsoft Distributed Transaction coordinator,

Microsoft SQL Agent services are running and the CRS Config Datastore is in INSERVICE. If it is an HA setup, ensure that both nodes have the above services up and running.

HR client login error

Symptom HR client login error.

Error Message An Error ocurred while attempting to communicate to Web Server. Check

your userid and pwd and try again (NO_HISTORICAL_REPORTING_CAPABILITY)

Possible Cause The userid is not assigned reporting capability in the CRS Administration User

Management web page.

Recommended Action Please give reporting capability and priveleges to the userid in the User

Management web page and relogin the HR client.

Cannot activate DB components on HA node

Symptom Cannot activate DB components on HA node.

Error Message Error related to DB Engine Version or DB Engine Edition mimatch during

activation of DB components.

Possible Cause HA requires both the nodes to have SQL 2K installed and the SQL 2k should be of same

version on both nodes.

Recommended Action Ensure both nodes have the same version of SQL 2K installed.

CRS Databases are not purged as expected

8-26

Symptom The CRS databases are not automatically purged as expected.

Error Message None.

Possible Cause The cause of this problem could be one of the following:

• Automatic purging is not configured properly.

• You have changed the system clock on the CRS server.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

• You have altered the size of the CRS databases.

Recommended Action Depending upon the cause of the problem, do one of the following:

• From the CRS Administration web page, choose Tools > Historical Reporting and configure

automatic purging.

• If you change the size of the CRS databases, make sure that the CRS database size is equal to the

maxsize.

Historical Database db_cra is full

Symptom Historical data is not getting written into the database, db_cra.

Error Message Could not allocate space for object in database db_cra because the

PRIMARY file group is full in the SQL log file, MIVR log file.

Possible Cause The db_cra database is full.

Recommended Action Do one of the following:

CRS Database Problems

Step 1 Try to start purging using the CRS Administration web page. Choose Tools > Historical Reporting >

Purge Now.

Step 2 Check the db_cra database size to make sure that it is of the proper size for the call volume generated.

E-mail notification of database purging activities is not sent

Symptom The CRS system does not send e-mail notification of database purging activities.

Error Message None.

Possible Cause The Email subsystem is not configured or e-mail notification is not set up properly in

CRS Administration.

Complete the following steps:

Step 1 From the CRS Administration web page, choose Subsystems > eMail and make sure that correct

information is entered in the Mail Server and eMail Address fields.

Step 2 From the CRS Administration web page, choose Tools > Historical Reporting and click the Purge

Schedule Configuration hyperlink.

Step 3 If multiple e-mail addresses are specified in the Send Email Notifications To field, make sure that each

address is separated with a semicolon (;), comma (,), or space.

Step 4 Make sure that the Send Email Notifications To field contains no more than 255 characters.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-27

Chapter 8 Troubleshooting Tips

CRS Editor Problems

Syslog or SNMP trap notification of database purging activities is not sent

Symptom The CRS system does not send Syslog notification or SNMP trap notification of purging

activities.

Error Message None.

Possible Cause The Cisco CRS Alarm Service is not running, Syslog is not configured, or SNMP service

is not configured.

Complete the following steps:

Step 1 Make sure that the Cisco CRS Alarm Service is running.

Step 2 Make sure that Syslog is properly configured.

Step 3 Make sure that SNMP service is properly configured on the CRS server.

CRS Editor Problems

This section contains the following troubleshooting tips on CRS Editor problems:

• Change a string variable to an integer, page 8-28

• Accept step error during active debug, page 8-28

• Error occurs with Reactive Debugging Tool, page 8-29

Change a string variable to an integer

Symptom You want to change a string variable to an integer.

Error Message None.

Possible Cause None.

Recommended Action Use the Set step, which supports the conversion of a string to any numerical type.

Accept step error during active debug

Symptom While debugging an application, the following message appears, where n is the task ID:

Error Message Ta sk : nAccept Step: Trigger is not a Contact Application trigger.

8-28

Possible Cause The debugger encountered the Accept step in the application but there was no call to

answer.

Recommended Action Debug the application as a Reactive Application and make the call before the

Reactive Application times out.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

Error occurs with Reactive Debugging Tool

Symptom An error occurs when using the Reactive Debugging tool.

Error Message Not defined.

Possible Cause Using the CRS Editor Reactive Debugging tool on a translation routed call can cause an

error.

Recommended Action From the ICM Configuration Manager, choose Tools > List Tools > Network VRU

Script List and temporarily increase the value in the Timeout field for the script.

CRS Engine Problems

This section contains the following troubleshooting tips on CRS Engine problems

• Agent cannot go Ready after logging in, page 8-30

• Voice Browser step throws an exception, page 8-30

• CRS Engine does not start and an RMI port in use error appears, page 8-30

• Attempting to start the Cisco CRS Node Manager service causes an error 1067, page 8-31

CRS Engine Problems

• Attempting to start the Cisco CRS Node Manager service causes an error 1069, page 8-31

• Application subsystem is in partial service, page 8-31

• CRS Engine is running but calls are not answered, page 8-32

• Changing the time on CRS machines results in agents getting logged off, page 8-32

• An error message plays when calling a CTI route point, page 8-33

• Changes to applications do not register, page 8-33

• Call drops during transfer over gateway, page 8-34

• H.323 client DTMF digits not detected, page 8-34

• Redirected call is disconnected, page 8-34

• The CRS server runs out of disk space, page 8-35

• CRS Server runs at 100% capacity or is very slow, page 8-35

• Database Subsystem goes into partial service, page 8-36

• JTAPI subsystem is in partial service, page 8-37

• Unable to connect to JTAPI provider, page 8-37

• The Simple Recognition step takes the unsuccessful branch, page 8-38

• Calling party and CRS do not have common codec, page 8-38

• Prompts with incorrect codec being played out, page 8-38

• Prompt Exception in CRS Engine log file, page 8-39

• CRS Engine does not start, page 8-39

• Application subsystem in partial service and application running for an unexpectedly long time,

page 8-39

• CRS Server and Active Directory integration results in some services being unregistered, page 8-40

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-29

CRS Engine Problems

Agent cannot go Ready after logging in

Symptom Agent cannot go Ready after logging in.

Error Message The Cisco Agent Desktop says that the resource's device is off and the

agent extension is out of service.

Possible Cause The agent's ephone does not have a session-server configured.

Recommended Action Make sure the session server of the agent's ephone is set to the the CRS session

server.

Voice Browser step throws an exception

Symptom When the URL specified in Voice Browser step uses "ServerName" instead of IPAddress, the

step throws an exception, "UnknownHostException."

Error Message None.

Chapter 8 Troubleshooting Tips

Possible Cause The Java Virtual Machine (JVM) caches a previously resolved entry, that is no longer

correct, until the JVM is restarted.

Recommended Action Restart the CRS Engine.

CRS Engine does not start and an RMI port in use error appears

Symptom The CRS Engine does not start and a Remote Method Invocation (RMI) port in use error

appears in the CRS trace files.

Error Message RMI port in use.

Possible Cause Another process is using the port that the CRS Engine is attempting to use.

From the CRS Administration web page, complete the following steps:

Step 1 Choose System > System Parameters.

Step 2 Enter a different port in the RMI Port Number field.

Step 3 Stop and then restart the CRS Engine.

Step 4 If CRS Engine is shown "Invalid" from CRS Administration, see the troubleshooting tip "Service

constantly shows Invalid".

8-30

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

CRS Engine Problems

Attempting to start the Cisco CRS Node Manager service causes an error 1067

Symptom You attempt to start the Cisco CRS Node Manager service in the Windows Services window

and the following message appears.

Error Message Could not start the Cisco CRS Node Manager service on local computer. Error 1067: The

process terminated unexpectedly.

Possible Cause There is an internal error in the Cisco CRS Node Manager.

Recommended Action Refer to Cisco Customer Response Solutions Administration Guide or the

Administration online help for information about properly setting up the CRS Node Manager service.

Attempting to start the Cisco CRS Node Manager service causes an error 1069

Symptom You attempt to start the Cisco CRS Node Manager service in the Windows Services window

and the following message appears.

Error Message Could not start the Cisco CRS Node Manager service on local computer. Error 1069: The service did not start due to a logon failure.

Possible Cause When you install Cisco Unified Communications Manager or Cisco CRS, the Windows

2003 administrator password that you enter overwrites the existing Windows 2003 administrator password. Also, if you enter a password that includes spaces, it may not be recorded properly.

Perform the following steps to change the password for the CRS Node Manager service:

Step 1 On the CRS Server, choose Start > Settings > Control Panel > Administrative Tools > Services.

Step 2 Double-click Cisco CRS Node Manager.

Step 3 Choose the Log On tab.

Step 4 Enter and confirm the Windows 2003 administrator password and click Apply. Do not include spaces in

the password.

Application subsystem is in partial service

Symptom The Engine Status area in the Engine web page shows that the Application subsystem is in

partial service.

Error Message None.

Possible Cause Some applications are invalid.

Performs these actions:

Step 1 Refer to the CRS trace files to identify the invalid application.

Step 2 Validate the corresponding script using the CRS Editor.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-31

CRS Engine Problems

CRS Engine is running but calls are not answered

Symptom The CRS Engine is running but the CRS system does not answer calls.

Error Message None.

Possible Cause The JTAPI subsystem is out of service, the trigger is disabled, the application is disabled,

the maximum number of sessions or maximum number of tasks were exceeded, or no CTI ports or media channels are available for the trigger.

Complete the following steps:

Step 1 From the CRS Administration web page, choose System > Control Center, pick up the servers on the

left panel, and expand the CRS Engine to verify that the JTAPI subsystem is in service.

If the JTAPI subsystem is in partial service, see the “JTAPI subsystem is in partial service” troubleshooting tip in this guide.

If the JTAPI subsystem out of service, refer to the “CRS Provisioning for Cisco Unified Communications Manager” section in the Cisco Customer Response Solutions Administration Guide for information about configuration.

Step 2 From the CRS Administration web page, choose Subsystems > JTAPI and click the JTAPI Triggers

hyperlink. If False appears in the Enabled column for the trigger, double-click the trigger, click the Enabled Yes radio button, and then click Update.

Chapter 8 Troubleshooting Tips

Step 3 From the CRS Administration web page, choose Applications > Configure Applications. If No appears

in the Enabled column for the application, double-click the application, click the Enabled Yes radio button, and then click Update.

Step 4 In the CRS trace files, verify that the calls do not exceed the maximum number of allowed sessions.

Step 5 In the CRS trace files, verify that the calls do not exceed the maximum number of allowed tasks.

Step 6 In the CRS trace files, make sure that there are no messages regarding insufficient free CTI ports or

media channels.

Changing the time on CRS machines results in agents getting logged off

Symptom Agents got logged off and Cisco Agent Desktop out-of-service and wrap-up timer delay

expired when the time was changed on CRS.

Error Message None.

Possible Cause If wrap-up timers are being used on Cisco Agent Desktops, changing the CRS time can

cause erroneous firings of the timers.

Recommended Action Do not change the system time on CRS machines.

8-32

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

An error message plays when calling a CTI route point

Symptom Callers hear a message when calling a CTI route point. The JTAPI subsystem might also be in

partial service because the CTI route point cannot load the associated application script.

Error Message I'm sorry, we are currently experiencing system problems.

Possible Cause The application script associated with the CTI route point did not load correctly.

Recommended Action Validate the application script in the CRS Editor as follows:

Step 1 From the CRS Administration web page, choose Applications > Script Management.

Step 2 Click the script and download it from the Repository.

Step 3 Open the script in the CRS Editor.

Step 4 Validate the script and save it.

Step 5 Choose Applications > Script Management and upload the script to the Repository.

Step 6 When prompted, clickYes to refresh both script and applications.

CRS Engine Problems

Step 7 Refer to the CRS trace files to verify that the application script was loaded successfully.

Step 8 If a script has been validated, saved, and uploaded to the repository, and still will not load, verify that

any other dependencies are met. For example, if the script references a custom class, make sure that the class is available to the CRS Engine.

Changes to applications do not register

Symptom You make changes to an application script but the changes are not apparent to callers.

Error Message None.

Possible Cause The application script was not uploaded to the repository and refreshed.

Recommended Action After making a change to an application script, perform the following steps:

Step 1 Save the application script.

Step 2 From the CRS Administration web page, choose Applications > Script Management and upload the

application script to the repository.

Step 3 When prompted, click Yes to refresh both script and applications.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-33

CRS Engine Problems

Call drops during transfer over gateway

Symptom When the CRS system receives a call made over a gateway, the CRS system drops the call if

the call is transferred.

Error Message None.

Possible Cause The H.323 client does not support the Empty Capability Service and the H.323 port on

the Cisco Unified Communications Manager is not configured to use a Media Termination Point (MTP).

Recommended Action Update the configuration of the Cisco Communications Manager H.323 port to

require an MTP and reset the H.323 port.

H.323 client DTMF digits not detected

Symptom When a call originates from an H.323 client, DTMF digits are not collected.

Error Message None.

Chapter 8 Troubleshooting Tips

Possible Cause The H.323 client only produces in-band DTMF signals. Cisco Unified Communications

Manager cannot detect in-band DTMF signals.

Recommended Action None.

Redirected call is disconnected

Symptom A redirected call disconnects or a redirected call does not ring the IP phone to which it was

directed.

Error Message None.

Possible Cause Some gateways do not support ringback.

Recommended Action Reconfigure the gateway and protocols so that they will support ringback.

Following are the gateways and the protocol for each gateway (note that the protocol is in parentheses):

• 26XX FXO (Media Gateway Control)

• 36XX FXO (36XX FXO Media Gateway Control)

• VG200 FXO (Media Gateway Control)

• DT-24+ (Skinny)

• WS-6608-T1—[Cat6K 8-port T1 PRI] (Skinny)

• WS-6608-E1 [Cat6K 8-port E1 PRI] (Skinny)

• DE-30+ (Skinny)

8-34

• AT-2, 4, 8 (AT-2, 4, 8 — )

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 8 Troubleshooting Tips

The CRS server runs out of disk space

Possible symptoms are:

• An out of memory error occurs on the CRS server.

• Accessing purging or synchronization pages on the Administration UI returns an error.

• Running historical reports returns SQL error 5048.

Error Message SQL error 5048.

Possible Cause The CRS database log files, the tempdb database, or the tempdb log files have grown

large.

Recommended Action There are two possible actions you can take:

Step 1 To manually shrink a CRS database log file, open a command window on the CRS server and type the

following commands:

osql -Usa -Ppassword -ddb_cra, where password is the password for the sa log in to the CRS database.

USE database_name, where database_name is db_cra.

CRS Engine Problems

DBCC SHRINKFILE (database_name_log.mdf), where database_name is db_cra.

Step 2 Alternatively, you can shrink the log files by running the batch file runTruncateHistDBLogs.bat,

installed under the wfavvid directory. Depending on the arguments, it shrinks the log files of db_cra or tempdb.

Examples:

• Truncate the log files for db_cra to 10MB:

• runTruncatedHistDBLogs “sa” “sa_password” “db_cra_all” 10

• Truncate the tempdb transaction log:

• runTruncateHistDBLogs “sa” “sa_password” “tempdb”

Additional information can be found about truncating logs in the Managing Historical Reporting Databases section of the Cisco CRS Administration Guide.

Note This troubleshooting tip also applies to the CRS Database Problems section and the CRS Historical

Reporting Problems section.

CRS Server runs at 100% capacity or is very slow

Symptom The CRS server CPU works at or close to 100 percent capacity. DTMF digits are delayed.

Error Message None.

One of the following configurations might be causing this problem:

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

8-35

CRS Engine Problems

Step 1 Turn off debugging as a trace level option. Debugging consumes substantial server resources. Only use

Step 2 Turn CRS debugging off if the system is running BHCC higher then 4500.

Step 3 If you have a very high load with 300 agents, disable all the logs and traces.

Step 4 If you have very high load with 300 agents, you should redirect logs (for example Unified CM logs) to

Step 5 Turn off Cisco Unified Communications Manager polling. Polling enables JTAPI (and therefore the

Step 6 If you are using a smaller system with many applications running at the same time, install the different

Step 7 Defragment the hard disk at regular interval on the CRS machine if the load is very high.

Chapter 8 Troubleshooting Tips

• Trace settings include debugging.

• Cisco Unified Communications Manager polling is enabled. (Polling is enabled by default, but it can

consume server resources.)

• You are running many applications on a smaller system simultaneously. For example, you are

running Cisco Unified Communications Manager, and Cisco CRS Server, all on a low-end MCS.

Recommended Action Complete the following steps:

debugging as a trace level option when you are actively debugging Cisco CRS.

a different drive where CRS and SQL is installed.

telephony applications that use JTAPI, such as CRS) to detect the addition of devices to an application or user’s controlled list. For example, polling can detect when an agent is added to a call center or a CTI port is added to the CRS Engine. If you do turn off polling, Cisco Unified Communications Manager does not update new devices automatically. For example, you must restart the CRS Server after adding a new CTI port or route point to Cisco Unified Communications Manager.

telephony applications on separate servers or use an MCS-7835.

Database Subsystem goes into partial service

Symptom The Database subsystem is in partial service when the Cisco CRS system is configured to use

a Sybase database.

Error Message None.

Possible Cause If the Sybase datasource name that you enter in the Cisco CRS Administration Enterprise

Database Subsystem Configuration web page does not match exactly the datasource name in the Windows ODBC DSN configuration window, the database connection will fail and the database will go into partial service.

Recommended Action Be sure the Sybase datasource name on the Administration Enterprise Database

Subsystem Configuration web page matches the Windows ODBC datasource name.

8-36

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Cisco Customer Response User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

CONTENTS

Purpose

Audience

Preface

Organization

Related Documents

Conventions

Obtaining Documentation

Cisco.com

Product Documentation DVD

Ordering Documentation

Documentation Feedback

Cisco Product Security Overview

Reporting Security Problems in Cisco Products

Product Alerts and Field Notices

Obtaining Technical Assistance

Cisco Support Website

Submitting a Service Request

Definitions of Service Request Severity

Obtaining Additional Publications and Information

Serviceability

About Serviceability

About CRS Serviceability

Serviceability Support

CiscoWorks Support

Syslog Support

Remote Serviceability

Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support

Simple Network Management Protocol Support

About Simple Network Management Protocol (SNMP)

SNMP Basics

SNMP Agent and Subagents

SNMP Management Information Base (MIB)

SYSAPPL-MIB

CISCO-VOICE-APPS-MIB

CISCO-CDP-MIB

SNMP Traps

SNMP Trap Messages

Failover Traps

Setting up SNMP Traps

Setting the SNMP Trap Receiver

Setting the SNMP Community Names

Starting, Stopping, and Confirming the SNMP Service

Snapshot of Traps During Startup

Snapshot of Traps During Shutdown

Alarm Service

About Alarms

Cisco CRS Alarm Service

Starting and Confirming the Alarm Service

Configuring the Alarm Service

Viewing Alarm Messages

Viewing Alarm Messages Sent to a Syslog Server

Viewing Alarm Messages Sent to an SNMP Trap Receiver

Viewing Alarm Messages Sent to a Windows Event Log

Alarm Definitions

Finding Information About an Alarm

Trace

About Trace Files

The Component Trace File

Configuring the Component Trace File

Trace Level Options

Setting Trace Level Options

Viewing and Interpreting the Trace Files

Displaying a Trace File

Interpreting a Trace File

The Thread Dump Trace File

Writing to the Thread Dump Trace file

The CRS Log Files

Displaying the Thread Dump Trace File

Cisco Desktop Product Suite Installation Logs

CRS Log Collection Tool

Cisco Discovery Protocol Support

About the Cisco Discovery Protocol (CDP)

Using the CDP Driver

Accessing CDP Driver Control

Installing the CDP Protocol Driver