Cisco Customer Response User Manual

Cisco Customer Response Solutions Servicing and Troubleshooting Guide

Cisco Unified Contact Center Express, Cisco Unified IP IVR, and Cisco Unified Queue Manager, Release 5.0(1)

June 2007

Americas Headquarters

Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000

800 553-NETS (6387)

Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

CCVP, the Cisco logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco

Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo,

Cisco

IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco

All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0705R)

Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Systems, Inc. and/or its affiliates in the United States and certain other countries.

IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE

Preface i

Purpose i

Audience i

Organization i

Related Documents ii

Conventions ii

Obtaining Documentation iii

Cisco.com iii Product Documentation DVD iii Ordering Documentation iv

Documentation Feedback iv

PART

I Serviceability

CHAPTER

1 About Serviceability 1-1

Cisco Product Security Overview iv

Reporting Security Problems in Cisco Products iv

Product Alerts and Field Notices v

Obtaining Technical Assistance v

Cisco Support Website v Submitting a Service Request vi Definitions of Service Request Severity vii

Obtaining Additional Publications and Information vii

About CRS Serviceability 1-1

Serviceability Support 1-1

CiscoWorks Support 1-2

Syslog Support 1-3

Remote Serviceability 1-3

CHAPTER

Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support 1-4

2 Simple Network Management Protocol Support 2-1

About Simple Network Management Protocol (SNMP) 2-1

SNMP Basics 2-1

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Contents

SNMP Agent and Subagents 2-2

SNMP Management Information Base (MIB) 2-2

SYSAPPL-MIB 2-2

CISCO-VOICE-APPS-MIB 2-6

CISCO-CDP-MIB 2-6

SNMP Traps 2-6

SNMP Trap Messages 2-6

Failover Traps 2-7

Setting up SNMP Traps 2-7

Setting the SNMP Trap Receiver 2-7 Setting the SNMP Community Names 2-8 Starting, Stopping, and Confirming the SNMP Service 2-9 Snapshot of Traps During Startup 2-9 Snapshot of Traps During Shutdown 2-10

CHAPTER

3 Alarm Service 3-1

About Alarms 3-1

Cisco CRS Alarm Service 3-1

Starting and Confirming the Alarm Service 3-2 Configuring the Alarm Service 3-3

Viewing Alarm Messages 3-3

Viewing Alarm Messages Sent to a Syslog Server 3-3

Viewing Alarm Messages Sent to an SNMP Trap Receiver 3-5

Viewing Alarm Messages Sent to a Windows Event Log 3-5

Alarm Definitions 3-5

Finding Information About an Alarm 3-5

4 Trace 4-1

About Trace Files 4-1

The Component Trace File 4-2

Configuring the Component Trace File 4-2

Trace Level Options 4-3

Setting Trace Level Options 4-7

Viewing and Interpreting the Trace Files 4-8

Displaying a Trace File 4-8

Interpreting a Trace File 4-8

The Thread Dump Trace File 4-8

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Writing to the Thread Dump Trace file 4-9 Displaying the Thread Dump Trace File 4-9

The CRS Log Files 4-9

Cisco Desktop Product Suite Installation Logs 4-11 CRS Log Collection Tool 4-11

Contents

CHAPTER

PART

5 Cisco Discovery Protocol Support 5-1

6 Cisco Support Tools 6-1

II Troubleshooting

About the Cisco Discovery Protocol (CDP) 5-1

Using the CDP Driver 5-2

Accessing CDP Driver Control 5-2

Installing the CDP Protocol Driver 5-2

Starting the CDP Protocol Driver 5-2 Enabling the CDP Protocol Driver 5-3 Showing the CDP Protocol Driver Properties 5-3

Updating an IP Address for the CDP Protocol Driver 5-3

Locating Updated CDP Driver and Interface Files 5-4

Default CDP Settings 5-4

About Cisco Support Tools with Cisco CRS 6-1

Accessing Cisco Support Tools 6-1

CHAPTER

7 Diagnosing and Correcting Cisco CRS Problems 7-1

General Troubleshooting Steps 7-1

8 Troubleshooting Tips 8-1

Installation Problems 8-2

One node on a CRS 5.0 two-node cluster crashes beyond repair 8-2

Backup, Restore, and Update Problems 8-2

Backup, Restore, and Upgrade cannot be started from a client desktop 8-3 During Backup, Restore, or Upgrade, an exception is seen in UI 8-3 Backup failed for a One or Two-Node system 8-4 CRS 4.5 profile name is missing 8-4 Page Not Found message is displayed during Restore or Upgrade 8-4 Restore fails due to a file not being found 8-5 Restore failed for a one-node system 8-5

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

iii

Contents

Restore failed on a two-node system that had run before the Restore 8-6 Restore failed on a two-node system that was re-imaged 8-7 Some RmCm configuration is missing after Upgrade 8-8

CME Telephony subsystem problems 8-8

A functional routing point stopped working or the CME Telephony subsystem is in partial service

Cisco Unified Communications Manager Automated Attendant problems 8-9

Dial by name does not find the specified server 8-9 Automated Attendant prompt is not played 8-9

Cisco Unified Communications Manager Express Problems 8-10

Agent cannot log in on shared line 8-10 Agent cannot log in on restricted line 8-10 When agent drops from conference, all parties on conference are dropped 8-10

Cisco Unified CCX Problems 8-10

RmCm subsystem is out of service 8-11 RmCm subsystem remains INITIALIZING 8-11 RmCm remains in Initializing state 8-12 Agents, Supervisors, or Teams are out of synch 8-12 Agent or CSQ does not appear in Cisco Desktop Administrator (CDA) 8-12 Agents do not appear in the Resources area in the Unified CCX Configuration web page 8-13 You cannot select the order of agents 8-13 Agent does not go to Work state after handling a call 8-13 A media step causes a Could not create PlayPromptDialog Object exception 8-14 Unable to make any Unified CCX configuration changes 8-14 Some resource selection criteria are missing 8-14 Unable to record an agent 8-15 Sometimes the supervisor can monitor and record an agent and sometimes he cannot 8-15 Calls to Unified CCX route points are disconnected 8-15 Calls are not routed to agents 8-15 Agents do not show in a CSQ 8-16 Caller gets dropped when an attempt is made to recall a Unified CCX agent extension after the agent

previously parked the call Updating a NIC driver disables silent monitoring and recording 8-16

8-8

8-16

Cisco Unified IP IVR Problems 8-17

Cisco Unified IP IVR drops callers when transferring to an extension 8-17 Prompts play in language 8-17 Some prompts do not play 8-18 Some prompts in a script play in the language specified and other prompts play in English 8-18 A prompt plays phrases in the wrong order 8-18

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CRS Administration Problems 8-18

The CRS Administration Authentication web page is not available 8-19 Uploading a license file can result in a warning message 8-20 User cannot log in to the CRS web page 8-20 Refreshing subflow script does not update parent scripts 8-20 Unified Communications Manager users display in random order 8-20 CRS Supervisor web page cannot be viewed from CRS Server 8-21 Database table fields used by wallboard store data in milliseconds 8-21 Management pages display error message when selected 8-21 Zip file does not auto unzip on Document Management page 8-22 Invalid files message displays while uploading a zip file of prompts 8-22 A Component Manager goes into partial service when uploading a zip file 8-23 High call rejection rate under heavy load 8-23

CRS Admin Utility Problems 8-23

The cluster is not in synchronization 8-24 CRS Admin Utility exits or does not come up after login. 8-24 The CRS Admin Utility fails due to data corruption 8-24 The CRS Admin Utility will not run on a none bootstrap node 8-25 The CRS Admin Utility will not run since the Node Manager hung 8-25

Contents

CRS Database Problems 8-25

Cannot configure Application or System parameters from their pages in CRS Administration 8-26 HR client login error 8-26 Cannot activate DB components on HA node 8-26 CRS Databases are not purged as expected 8-26 Historical Database db_cra is full 8-27 E-mail notification of database purging activities is not sent 8-27 Syslog or SNMP trap notification of database purging activities is not sent 8-28

CRS Editor Problems 8-28

Change a string variable to an integer 8-28 Accept step error during active debug 8-28 Error occurs with Reactive Debugging Tool 8-29

CRS Engine Problems 8-29

Agent cannot go Ready after logging in 8-30 Voice Browser step throws an exception 8-30 CRS Engine does not start and an RMI port in use error appears 8-30 Attempting to start the Cisco CRS Node Manager service causes an error 1067 8-31 Attempting to start the Cisco CRS Node Manager service causes an error 1069 8-31 Application subsystem is in partial service 8-31 CRS Engine is running but calls are not answered 8-32

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Contents

Changing the time on CRS machines results in agents getting logged off 8-32 An error message plays when calling a CTI route point 8-33 Changes to applications do not register 8-33 Call drops during transfer over gateway 8-34 H.323 client DTMF digits not detected 8-34 Redirected call is disconnected 8-34 The CRS server runs out of disk space 8-35 CRS Server runs at 100% capacity or is very slow 8-35 Database Subsystem goes into partial service 8-36 JTAPI subsystem is in partial service 8-37 Unable to connect to JTAPI provider 8-37 The Simple Recognition step takes the unsuccessful branch 8-38 Calling party and CRS do not have common codec 8-38 Prompts with incorrect codec being played out 8-38 Prompt Exception in CRS Engine log file 8-39 CRS Engine does not start 8-39 Application subsystem in partial service and application running for an unexpectedly long time 8-39 CRS Server and Active Directory integration results in some services being unregistered 8-40

CRS Real-Time Reporting Problems 8-40

Attempting to run a real-time report causes an error 8-40 After installing JRE, the user receives a message from real-time reporting saying to install JRE 8-41

CRS Historical Reporting Problems 8-41

Exported PDF report does not print in landscape orientation 8-42 User login missing in Windows XP after installing HR client 8-42 Client and Server security policies do not match 8-43 Charts do not appear properly in MS Excel format 8-43 Columns of data missing in report in MS Excel format 8-43 Records truncated in report in MS Excel format 8-43 Agent names overwritten on charts 8-44 RTF Report containing charts has tabular report headings 8-44 Scheduler icon does not appear on Terminal Services client 8-44 Reports do not execute at scheduled times 8-44 Search dialog box and Preview tab appear in English on Windows system with locale set to

German

8-45

Dialog box does not appear as expected when report is exported 8-45 Error when choosing an option from the Historical Reporting web page 8-45 Truncated report description in Historical Reports client 8-46 Scheduled Historical Reports do not run 8-46 The SQL Command Failed dialog box appears when you try to generate a historical report 8-46 Some information appears in English on a German system 8-47

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Contents

The Historical Reports client computer cannot connect to the CRS server 8-47 A Database Connection Error 5051 error appears 8-47 Export file name does not appear in Export dialog box 8-48 Cannot point to local applications from the Database Server Configuration page 8-48 Attempt to log in to the CRS Server from the Historical Reporting client fails and an error message is

returned Only three report templates available for Unified CCX Standard 8-49 Discrepancy in number of ACD calls shown on custom reports 8-50 Priority Summary Activity Report chart prints only partly in color 8-50 Scheduled Historical Reports do not run and message appears in CiscoSch.log file 8-50 Historical Reporting Client window shows nothing in user drop-down menu 8-51 Historical Reporting Client stops working; attempt to log in again results in error messages 8-51 Scheduler DOS exception error received when running a custom report 8-52 Columns displaced in Excel spreadsheet when exporting a report 8-52 Scheduler icon does not appear in Windows status bar 8-52 Error message appears indicating connection with database is broken 8-53

8-49

Automatic Speech Recognition (ASR) Problems 8-53

Names are not recognized 8-53 Recognition never times out 8-54 Alternate pronunciations and nicknames are not recognized 8-54 Reduced call completion rate under heavy load while using an MRCP ASR Group 8-54 MRCP ASR subsystem is out of service 8-55 Changes, additions, or deletions to MRCP ASR Providers, MRCP Servers, or Groups do not take

effect

8-55

Calling a route point with an MRCP ASR Dialog Group results in default treatment 8-56

Outbound Problems 8-56

Agent does skip or skip-close but does not stay reserved 8-57 Agent is not getting callbacks 8-57 Agent is ready but does not get an Outbound call for up to Two minutes 8-57 Errors placing Outbound calls 8-58 Not all contacts get imported 8-58 On the Campaigns Configuration web page, the available CSQs list is empty even though there are

CSQs configured under the RmCm subsystem

8-58

Outbound buttons do not show up on CAD 8-58 Outbound buttons show up but are disabled on CAD 8-59 Outbound calls are not getting dialed 8-59 Outbound call volume is low 8-59 Outbound System Service is not in service 8-60 RTR Outbound reports do not show all possible reclassification 8-60

Text-to-Speech (TTS) Problems 8-60

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

vii

Contents

Provider becomes IN_SERVICE immediately 8-61 A TTS Prompt will not play 8-61 A TTS prompt is not recognizable 8-62 MRCP TTS subsystem is out of service 8-62 Long TTS prompts consume significant memory on CRS Server 8-62 Non-UTF-8 characters needed for some languages 8-63 A .wav file prompt playback is garbled when played by a TTS server 8-63 Changes, additions, or deletions to MRCP TTS Providers, MRCP Servers, locales, or genders do not

take effect

Serviceability Problems 8-64

SNMP-based network management tools cannot monitor CRS components 8-65 File Manager in partial service 8-65 SNMP traps do not arrive at the trap receiver 8-65 Syslog messages not received by receiver 8-66 The Alarm Service does not start 8-67 Serviceability does not uninstall completely 8-67 Updating Data with the Serviceability Tool on One Node Does Not Update Other Nodes 8-67 Virus Scan software slows Call Completion Rate 8-68

8-64

CRS Internationalization Problems 8-68

Results not as expected for first name and last name in Chinese, Japanese, and Korean 8-68 Language specified is not accepted or played 8-69

VXML Problems 8-69

Voice Browser Step troubleshooting steps 8-69 Timeout attribute for non-input does not work 8-70 Menu Choice DTMF does not work 8-70

High Availability and Bootstrap 8-71

Transaction Manager cannot start 8-71 Have an exception on startup with a message like "unable to recover transaction" or an error message

related to reading or modifying the "Tx.per" file.

8-71

High Availability and Failover 8-72

Previously configured log file size is not preserved after system upgrade 8-72 Conflicts in Datastore Control Center history 8-73 Cannot make configuration changes in HA cluster 8-73 Cannot make configuration changes in RmCm Subsystem 8-73 Service constantly shows Invalid 8-74 CRS server keeps rebooting due to CRS Node Manager failure 8-74 Cluster is in partial service 8-74 Server is in partial service 8-75 CRS does not accept call or function properly 8-75 Service Master/Slave status is not shown on CRS Administration Control Center 8-75

viii

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

NDEX

Contents

Cluster time synch fails 8-76 CRS Servers respond slowly in HA environment 8-76 Multilple failovers with high CPU usage 8-76

VoIP Monitor Problems 8-76

VoIP monitor does does not work correctly 8-77 CRS fails to start 8-77

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Contents

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Purpose

Audience

Preface

The Cisco CRS Servicing and Troubleshooting Guideprovides instructions for using the CRS Serviceability tools and helps you resolve any problems you might experience with the CRS system.

The Cisco CRS Servicing and Troubleshooting Guide is written for administrators and others who are responsible for managing and troubleshooting the Cisco CRS system.

Organization

Part/Chapter Title Description

Part 1 Serviceablity

Chapter 1 About Serviceability Provides an overview of the Cisco CRS serviceability

Chapter 2 Simple Network

Chapter 3 Alarm Service Describes how to configure the Cisco CRS Alarm Service

Chapter 4 Trace Describes how to configure the component trace file, set

Chapter 5 Cisco Discovery

Chapter 6 Cisco Support Tools Provides an overview of Cisco Support Tools that are

Management Protocol Support

Protocol Support

support and the support provided when an expansion server is used.

Describes how you can use SNMP to monitor and manage your CRS system.

and view alarm messages.

trace level options, and collect, view, and interpret trace log files.

Describes how to use the CDP Driver, view the CDP Driver properties, and locate the CDP Driver and interface files.

supported by Cisco CRS.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Part/Chapter Title Description

Part 2 Troubleshooting

Chapter 7 Diagnosing and

Chapter 8 Troubleshooting Tips Provides troubleshooting tips for the various elements of

Conventions

This manual uses the following conventions.

Convention Description

boldface font Boldface font is used to indicate commands, such as user entries,

keys, buttons, and folder and submenu names. For example:

• Choose Edit > Find.

• Click Finish.

italicfont Italic font is used to indicate the following:

• To introduce a new term. Example: A skill group is a

collection of agents who share similar skills.

• For emphasis. Example: Do not use the numerical naming

convention.

• A syntax value that the user must replace. Example: IF

(condition, true-value, false-value)

• A book title. Example: See the Cisco CRS Installation Guide.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

Convention Description

window font Window font, such as Courier, is used for the following:

< > Angle brackets are used to indicate the following:

Obtaining Documentation

Cisco documentation and additional literature are available on Cisco.com. This section explains the product documentation resources that Cisco offers.

• Text as it appears in code or that the window displays.

Example: <html><title>Cisco Systems,Inc. </title></html>

• For arguments where the context does not allow italic, such as

ASCII output.

• A character string that the user enters but that does not appear

on the window such as a password.

Cisco.com

You can access the most current Cisco documentation at this URL:

http://www.cisco.com/techsupport

You can access the Cisco website at this URL:

http://www.cisco.com

You can access international Cisco websites at this URL:

http://www.cisco.com/public/countries_languages.shtml

Product Documentation DVD

The Product Documentation DVD is a library of technical product documentation on a portable medium. The DVD enables you to access installation, configuration, and command guides for Cisco hardware and software products. With the DVD, you have access to the HTML documentation and some of the PDF

files found on the Cisco website at this URL:

http://www.cisco.com/univercd/home/home.htm

The Product Documentation DVD is created and released regularly. DVDs are available singly or by subscription. Registered Cisco.com users can order a Product Documentation DVD (product number DOC-DOCDVD= or DOC-DOCDVD=SUB) from Cisco Store at this URL:

Marketplace at the Product Documentation

http://www.cisco.com/go/marketplace/docstore

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

iii

Ordering Documentation

You must be a registered Cisco.com user to access Cisco Marketplace. Registered users may order Cisco documentation at the Product Documentation Store at this URL:

http://www.cisco.com/go/marketplace/docstore

If you do not have a user ID or password, you can register at this URL:

http://tools.cisco.com/RPF/register/register.do

Documentation Feedback

You can provide feedback about Cisco technical documentation on the Cisco Support site area by entering your comments in the feedback form available in every online document.

Cisco Product Security Overview

Preface

Cisco provides a free online Security Vulnerability Policy portal at this URL:

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

From this site, you will find information about how to do the following:

• Report security vulnerabilities in Cisco products

• Obtain assistance with security incidents that involve Cisco products

• Register to receive security information from Cisco

A current list of security advisories, security notices, and security responses for Cisco products is available at this URL:

http://www.cisco.com/go/psirt

To see security advisories, security notices, and security responses as they are updated in real time, you can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS) feed. Information about how to subscribe to the PSIRT RSS feed is found at this URL:

http://www.cisco.com/en/US/products/products_psirt_rss_feed.html

Reporting Security Problems in Cisco Products

Cisco is committed to delivering secure products. We test our products internally before we release them, and we strive to correct all vulnerabilities quickly. If you think that you have identified a vulnerability in a Cisco product, contact PSIRT:

• For emergencies only — security-alert@cisco.com

An emergency is either a condition in which a system is under active attack or a condition for which a severe and urgent security vulnerability should be reported. All other conditions are considered nonemergencies.

• For nonemergencies — psirt@cisco.com

In an emergency, you can also reach PSIRT by telephone:

• 1 877 228-7302

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

• 1 408 525-6532

Tip We encourage you to use Pretty Good Privacy (PGP) or a compatible product (for example, GnuPG) to

encrypt any sensitive information that you send to Cisco. PSIRT can wo rk with information that has been encrypted with PGP versions

Never use a revoked encryption key or an expired encryption key. The correct public key to use in your correspondence with PSIRT is the one linked in the Contact Summary section of the Security Vulnerability Policy page at this

http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

The link on this page has the current PGP key ID in use.

If you do not have or use PGP, contact PSIRT to find other means of encrypting the data before sending any sensitive material.

2.x through 9.x.

URL:

Product Alerts and Field Notices

Modifications to or updates about Cisco products are announced in Cisco Product Alerts and Cisco Field Notices. You can receive these announcements by using the Product Alert Tool on Cisco.com. This tool enables you to create a profile and choose those products for which you want to receive information.

To access the Product Alert Tool, you must be a registered Cisco.com user. Registered users can access the tool at this URL:

http://tools.cisco.com/Support/PAT/do/ViewMyProfiles.do?local=en

To register as a Cisco.com user, go to this URL:

http://tools.cisco.com/RPF/register/register.do

Obtaining Technical Assistance

Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco

Support website on Cisco.com features extensive online support resources. In addition, if you

have

a valid Cisco service contract, Cisco Technical Assistance Center (TAC) engineers provide

telephone support. If you do not have a valid Cisco service contract, contact your reseller.

Cisco Support Website

The Cisco Support website provides online documents and tools for troubleshooting and resolving technical issues with Cisco products and technologies. The website is available 24 this

URL:

http://www.cisco.com/en/US/support/index.html

Access to all tools on the Cisco Support website requires a Cisco.com user ID and password. If you have a valid service contract but do not have a user ID or password, you can register at this URL:

http://tools.cisco.com/RPF/register/register.do

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

hours a day at

Preface

Note Before you submit a request for service online or by phone, use the Cisco Product Identification Tool

to locate your product serial number. You can access this tool from the Cisco Support website by

clicking the Get Tools & Resources link, clicking the All Tools (A-Z) tab, and then choosing

Cisco

Product Identification Tool from the alphabetical list. This tool offers three search options:

product ID or model name; by tree view; or, for certain products, by copying and pasting show command output. Search results show an illustration of your product with the serial number label location highlighted. Locate the serial number label on your product and record the information before placing a service call.

Tip Displaying and Searching on Cisco.com

If you suspect that the browser is not refreshing a web page, force the browser to update the web page by holding down the Ctrl key while pressing F5.

To find technical information, narrow your search to look in technical documentation, not the entire

Cisco.com website. After using the Search box on the Cisco.com home page, click the

Advanced Tec hn ic al

To provide feedback about the Cisco.com website or a particular technical document, click

Contacts & Feedback at the top of any Cisco.com web page.

Search link next to the Search box on the resulting page and then click the

Support & Documentation radio button.

Submitting a Service Request

Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3 and S4 service requests are those in which your network is minimally impaired or for which you require product information.) After you describe your situation, the TAC recommended solutions. If your issue is not resolved using the recommended resources, your service request is assigned to a Cisco engineer. The TAC Service Request Tool is located at this URL:

http://www.cisco.com/techsupport/servicerequest

For S1 or S2 service requests, or if you do not have Internet access, contact the Cisco TAC by telephone. (S1 or S2 service requests are those in which your production network is down or severely degraded.) Cisco engineers are assigned immediately to S1 and S2 service requests to help keep your business operations running smoothly.

To open a service request by telephone, use one of the following numbers:

Asia-Pacific: +61 2 8446 7411 Australia: 1 800 805 227 EMEA: +32 2 704 55 55 USA: 1 800 553 2447

For a complete list of Cisco TAC contacts, go to this URL:

http://www.cisco.com/techsupport/contacts

Service Request Tool provides

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Preface

Definitions of Service Request Severity

To ensure that all service requests are reported in a standard format, Cisco has established severity definitions.

Severity 1 (S1)—An existing network is “down” or there is a critical impact to your business operations. You and Cisco will commit all necessary resources around the clock to resolve the situation.

Severity 2 (S2)—Operation of an existing network is severely degraded, or significant aspects of your business operations are negatively affected by inadequate performance of Cisco products. You and Cisco will commit full-time resources during normal business hours to resolve the situation.

Severity 3 (S3)—Operational performance of the network is impaired while most business operations remain functional. You and Cisco will commit resources during normal business hours to restore service to satisfactory levels.

Severity 4 (S4)—You require information or assistance with Cisco product capabilities, installation, or configuration. There is little or no effect on your business operations.

Obtaining Additional Publications and Information

Information about Cisco products, technologies, and network solutions is available from various online and printed sources.

• The Cisco Online Subscription Center is the website where you can sign up for a variety of Cisco

e-mail newsletters and other communications. Create a profile and then select the subscriptions that you would like to receive. To visit the Cisco

http://www.cisco.com/offer/subscribe

• The Cisco Product Quick Reference Guide is a handy, compact reference tool that includes brief

product overviews, key features, sample part numbers, and abbreviated technical specifications for many Cisco the latest Cisco channel product offerings. To order and find out more about the Cisco Reference Guide, go to this URL:

http://www.cisco.com/go/guide

• Cisco Marketplace provides a variety of Cisco books, reference guides, documentation, and logo

merchandise. Visit Cisco

http://www.cisco.com/go/marketplace/

• Cisco Press publishes a wide range of general networking, training, and certification titles. Both new

and experienced users will benefit from these publications. For current Cisco information, go to Cisco

http://www.ciscopress.com

• Internet Protocol Journal is a quarterly journal published by Cisco for engineering professionals

involved in designing, developing, and operating public and private internets and intranets. You can access the Internet Protocol Journal at this URL:

products that are sold through channel partners. It is updated twice a year and includes

Marketplace, the company store, at this URL:

Press at this URL:

Online Subscription Center, go to this URL:

Product Quick

Press titles and other

http://www.cisco.com/ipj

• Networking products offered by Cisco, as well as customer support services, can be obtained at

this

URL:

http://www.cisco.com/en/US/products/index.html

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

vii

Preface

• Networking Professionals Connection is an interactive website where networking professionals

share questions, suggestions, and information about networking products and technologies with Cisco experts and other networking professionals. Join a discussion at this URL:

http://www.cisco.com/discuss/networking

• “What’s New in Cisco Documentation” is an online publication that provides information about the

latest documentation releases for Cisco products. Updated monthly, this online publication is organized by product category to direct you quickly to the documentation for your products. You can view the latest release of “What’s New in Cisco

http://www.cisco.com/univercd/cc/td/doc/abtunicd/136957.htm

• World-class networking training is available from Cisco. You can view current offerings at

this

URL:

http://www.cisco.com/en/US/learning/index.html

Documentation” at this URL:

viii

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

ART

Serviceability

CHA PTER

About Serviceability

Cisco Customer Response Solutions (CRS) Serviceability enables remote network management support for the Cisco CRS system. Serviceability enables this support through CiscoWorks and through any third-party network management system (NMS) that uses standard protocols. These protocols include Syslog, Simple Network Management Protocol (SNMP), XML, and HTTP.

This section contains the following topics:

• About CRS Serviceability, page 1-1

• Serviceability Support, page 1-1

• CiscoWorks Support, page 1-2

• Syslog Support, page 1-3

• Remote Serviceability, page 1-3

• Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support, page 1-4

About CRS Serviceability

Serviceability Support

Serviceability allows you to monitor and discover the status of the installed components of your Cisco CRS system, its subsystems, and its services from any NMS. You can use the information that you obtain through serviceability to troubleshoot system problems. (For additional troubleshooting information, refer to Part 2 of this guide.)

Serviceability support includes:

• SNMP Support—Provides integration with CiscoWorks or another SNMP-based network

management system (NMS). SNMP agents provide monitoring of network devices through MIBs (Management Information Bases). For more information, see

Simple Network Management Protocol Support, page 2-1.

• SNMP Traps—Provides notification messages of high-severity Cisco CRS Engine errors. For more

information, see

Simple Network Management Protocol Support, page 2-1

CiscoWorks Support, page 1-2 and

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

1-1

CiscoWorks Support

Chapter 1 About Serviceability

• Alarms—Use Alarms to obtain the run-time status and state of the Cisco CRS system and to take

corrective action to fix detected problems. You can forward alarms to a Syslog server, to an SNMP trap subagent, or to a Windows Event Log. For more information, see

• Trace—Provides specific, detailed Cisco CRS information for troubleshooting system problems.

You can also send alarms to a trace file for further analysis and you can specify what level of event information is sent to the trace file. For more information, see

• Cisco Discovery Protocol (CDP) Support—Sends messages containing system information to a

designated multicast address. For more information, see

Trace, page 4-1.

Cisco Discovery Protocol Support,

page 5-1.

• Syslog Support—Sends common event logging messages in standard Syslog format to any

third-party Syslog server. For more information, see

• Cisco Support Tools—The Node Agent utility of Cisco Support Tools helps you collect log

Alarm Service, page 3-1.

information and troubleshoot Cisco CRS servers. For more information, see

page 6-1.

You can obtain additional system troubleshooting information using the following tools:

• CiscoWorks—Provides a suite of web-based applications for managing Cisco devices. For more

information, see

• Third-Party Network Management Systems—Provide Simple Network Management Protocol-based

CiscoWorks Support, page 1-2.

browser, Syslog support, and other system management tools.

Alarm Service, page 3-1.

Cisco Support Tools,

• Microsoft Windows 2003 Performance Monitoring—Allows you to monitor the performance of the

Cisco CRS system. For more information, refer to your Microsoft Windows documentation.

• Microsoft Windows 2003 Terminal Service—Provides remote systems with access to

Windows-based applications through terminal emulation. Windows 2003 Server Terminal Services are integrated with the Windows 2003 operating system. For more information, refer to your Microsoft Windows documentation.

CiscoWorks Support

CiscoWorks, available as a separate package, provides a suite of web-based applications for managing Cisco devices. It is the network management system (NMS) of choice for the Cisco CRS system and for other Cisco devices.

The Cisco CRS system integrates with these CiscoWorks applications:

• Cisco Unified Operations Manager (Operations Manager)—Operations Manager tracks the health

of Cisco Unified Communications environments by proactively monitoring Cisco voice elements in the network to alert operations personnel to potential problems and to help minimize Unified Communications service downtime.

• Resource Management Essentials (RME)—Providestools for collecting Syslog messages from

multiple sources for system-level fault monitoring and analysis.

• Campus Manager—Provides network topology services, user tracking, and path analysis. Campus

Manager Topology Services can display a map of your network and it can display a variety of information about each device on the network. It provides version, run-time status, and URLs of the applications on the devices and it provides filtering to display only specified devices. User Tracking provides a tool that tracks IP telephones on a Voice over IP (VoIP) network. Path Analysis provides a diagnostic application that traces connectivity between two specified points on a network and analyzes physical and logical paths.

1-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 1 About Serviceability

For more information about CiscoWorks, refer to the documentation available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps3996/tsd_products_support_series_home.html

Syslog Support

In addition to writing information to a trace file, the Cisco CRS system sends standard event logging messages to a Syslog server through the Alarm Service. These messages contain information about the activities of the Cisco CRS Engine and its subsystems. You can use any Syslog server to analyze these messages.

For analyzing Syslog messages, the Cisco CRS system integrates with CiscoWorks Resource Management Essentials (RME). The RME Cisco Syslog Analyzer controls and displays all event messages so that they can easily be read, interpreted, filtered, and used for system maintenance and troubleshooting. In the Syslog Analyzer, these reports are available under WorkFlow Report. You can also adapt Syslog output from the Cisco CRS system for use with other network management systems that have standard Syslog receiving capability.

For information about configuring a Syslog server, see Configuring the Alarm Service, page 3-3.

For more information about CiscoWorks, refer to the documentation available at this URL:

http://www.cisco.com/en/US/products/sw/cscowork/ps3996/tsd_products_support_series_home.html

Syslog Support

Remote Serviceability

Many of the serviceability tools can be used by a Cisco Service Engineer (CSE) to assist you with the management and administration of your Cisco CRS system. These tools allow CSEs to remotely gather system and debugging information if you require help with troubleshooting or system diagnostics.

With your permission, CSEs can log on to a Cisco CRS server and obtain a desktop or shell that allows them to perform any function that could be performed locally.

Tools that assist with remote serviceability include:

• CiscoWorks—Provides remote management capabilities for the Cisco CRS system and Cisco CRS

network. For more information, see

• Microsoft Windows 2003 Performance Monitoring—Allows monitoring the performance counters

of the Cisco CRS system from local or from remote systems. For more information, refer to your Microsoft Windows documentation.

• Microsoft Windows 2003 Terminal Services—Provides remote systems with access to

Windows-based applications through terminal emulation. Windows 2003 Server Terminal Services are integrated with the Windows 2003 operating system.

• Virtual Network Computing (VNC) isa desktop protocol to remotely control another computer. It

transmits the keyboard presses and mouse clicks from one computer to another, relaying the screen updates back in the other direction, over a network. There are many variants of freeware VNC available today.

CiscoWorks Support, page 1-2.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

1-3

Chapter 1 About Serviceability

Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support

You can set up a dedicated server for monitoring, recording, and maintaining Unified CCX statistics. Such a server is called an Unified CCX Call Statistics, Recording, and Monitoring Server. You can also set up a dedicated server or servers for monitoring. These servers are called Unified CCX Call Monitoring Servers. The CRS installation process automatically sets up and configures serviceability on these dedicated servers.

Serviceability enables CiscoWorks support and third-party NMS support for the servers and includes:

• Cisco Discover Protocol (CDP) support, which enables the Media Convergence Server (MCS) to be

discovered automatically by CiscoWorks.

• CISCO-CDP-MIB support.

• SYSAPPL-MIB support, which provides run-time status, version information, and application

discovery for voice recording and monitoring services.

• Standard third-party MIB support.

• EMBLEM support for CiscoWorks.

1-4

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

CHA PTER

Simple Network Management Protocol Support

This section contains the following topics:

• About Simple Network Management Protocol (SNMP), page 2-1SNMP Basics, page 2-1

• SNMP Agent and Subagents, page 2-2

• SNMP Management Information Base (MIB), page 2-2

• SYSAPPL-MIB, page 2-2

• CISCO-VOICE-APPS-MIB, page 2-6

• CISCO-CDP-MIB, page 2-6

• SNMP Traps, page 2-6

• SNMP Trap Messages, page 2-6

• Setting up SNMP Traps, page 2-7

About Simple Network Management Protocol (SNMP)

Simple Network Management Protocol (SNMP) isan industry-standard interface for exchanging management information between network devices. SNMP and its components provide you with information about your Cisco CRS system. You can refer to this information to monitor and manage the status of the Cisco CRS system, its subsystems, and its related installed components. You can also use this information to troubleshoot problems, if they arise.

You can set up SNMP traps to automatically notify you of high-severity messages and errors that are generated by the Cisco CRS system.

SNMP Basics

A network management system (NMS) uses SNMP to exchange management information between devices on a network. An SNMP-managed network is made up of the following main components:

• Managed devices—Network nodes, each containing an SNMP agent. Managed devices collect and

store information and make this information available using SNMP.

• Agents—Network management software that resides on a managed device. An agent contains local

knowledge of management information and translates the information into a form that is compatible with SNMP.

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-1

Chapter 2 Simple Network Management Protocol Support

SNMP Agent and Subagents

• Management stations—Computers on which the NMS is installed and from which system

administrators can retrieve and evaluate information from managed devices.

SNMP Agent and Subagents

The Microsoft Windows SNMP service (referred to as the SNMP Service) provides a framework for SNMP and provides the SNMP agent that interfaces with SNMP subagents.

SNMP Service starts automatically when the system starts. You can restart or stop the SNMP Service if a problem occurs or if it did not start automatically.

For more information, see Starting, Stopping, and Confirming the SNMP Service, page 2-9.

For information on configuring SNMP Service, see SNMP Traps, page 2-6.

SNMP Management Information Base (MIB)

A Management Information Base (MIB) designates a collection of information that is organized hierarchically. You access MIBs with SNMP. MIBs are made up of managed objects, which are identified by object identifiers. Managed objects are made up of one or more object instances, which are essentially variables. MIBs provide status monitoring, provisioning and notification.

The Cisco CRS system supports these MIBs:

• SYSAPPL-MIB—Provides system information, such as installed applications, application

components, product version, processes that are running, and process start time. For more information, see

SYSAPPL-MIB, page 2-2.

• CISCO-VOICE-APPS-MIB—Contains information about supported SNMP traps. For more

information, see

• CISCO-CDP-MIB—Provides information about device identifications, CDP (Cisco Discovery

Protocol) running status, CDP transmitting frequency, and the time for the receiving device to hold CDP messages (time to live). For more information, see

page 5-1.

Standard third-party MIBs, including:

–

SNMP Community Names authenticate access to MIB objects and serve as passwords for SNMP information. A system can exchange SNMP information only with systems in the same community. For more information on setting up communities, see

For additional information about MIBs, refer to this URL:

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

SYSAPPL-MIB

The SYSAPPL-MIB provides system information about installed packages, including product name, product version, URL of the Cisco CRS Administration page, run-time status, application start time, and currently running processes.

CISCO-VOICE-APPS-MIB, page 2-6.

Cisco Discovery Protocol Support,

Standard Microsoft MIBs, such as MIB II

Compaq Insight Agent MIBS for Compaq MCS 78xx platforms

IBM UM MIB for IBM 3xx MCS platforms

Setting the SNMP Trap Receiver, page 2-7.

2-2

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

Chapter 2 Simple Network Management Protocol Support

The SYSAPPL-MIB allows you to use CiscoWorks or a third-party NMS browser to remotely access information about the Cisco CRS components including:

• Cisco CRS Administration

• Cisco CRS Node Manager

• Cisco CRS Engine

• Cisco CRS Repository Datastore

• Cisco CRS Historical Datastore

• Cisco CRS Config Datastore

• Cisco CRS Agent Datastore

• Cisco Recording

• Cisco Monitoring

The SYSAPPL-MIB also provides access to the Cisco CRS Services, including:

• Cisco CRS Cluster View Daemon

including but not limited to:

–

Manager Manager

SYSAPPL-MIB

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

Socket Manager

• Cisco CRS Administration

including but not limited to:

–

Manager Manager

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

File Manager

–

Prompt Manager

–

Grammar Manager

–

Document Manager

–

Resource Manager

–

Script Manager

–

Expression Manager

–

Socket Manager

• Cisco CRS Engine

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

2-3

SYSAPPL-MIB

including but not limited to:

–

Manager Manager

–

Log Manager

–

Config Manager

–

Executor Manager

–

Cluster Manager

–

Node Manager

–

File Manager

–

Prompt Manager

–

Grammar Manager

–

Document Manager

–

Resource Manager

–

Script Manager

–

Expression Manager

Chapter 2 Simple Network Management Protocol Support

–

Socket Manager

–

RTP Port Manager

–

Contact Manager

–

Channel Manager

–

Session Manager

–

ICM Subsystem

–

JTAPI Subsystem

–

CMT Subsystem

–

MRCP ASR Subsystem

–

MRCP TTS Subsystem

–

eMail Subsystem

–

RmCm Subsystem

–

Voice Browser Subsystem

–

Core Real-Time Reporting Subsystem

–

Enterprise Server Data Subsystem

–

Database Subsystem

–

VoIP Monitor Subsystem

–

HTTP Subsystem

–

Outbound Subsystem

–

SIP Subsystem

2-4

–

• Cisco Desktop License and Resource Manager

• Cisco Desktop Call and Chat Service

• Cisco Desktop Enterprise Service

Cisco Customer Response Solutions Servicing and Troubleshooting Guide, Release 5.0(1)

+ 118 hidden pages

Cisco Customer Response User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

CONTENTS

Purpose

Audience

Preface

Organization

Related Documents

Conventions

Obtaining Documentation

Cisco.com

Product Documentation DVD

Ordering Documentation

Documentation Feedback

Cisco Product Security Overview

Reporting Security Problems in Cisco Products

Product Alerts and Field Notices

Obtaining Technical Assistance

Cisco Support Website

Submitting a Service Request

Definitions of Service Request Severity

Obtaining Additional Publications and Information

Serviceability

About Serviceability

About CRS Serviceability

Serviceability Support

CiscoWorks Support

Syslog Support

Remote Serviceability

Unified CCX Call Statistics, Recording, and Monitoring Server Serviceability Support

Simple Network Management Protocol Support

About Simple Network Management Protocol (SNMP)

SNMP Basics

SNMP Agent and Subagents

SNMP Management Information Base (MIB)

SYSAPPL-MIB