Cisco Systems 30 VIP User Manual

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Cisco IP Telephony Troubleshooting Guide

for Cisco CallManager Release 3.0(1)

Contents

Purpose............................................................................................................................................4

Version............................................................................................................................................4

Topology .........................................................................................................................................4

Documentation Checklist............................................................................................................5

Glossary of Terms...........................................................................................................................6

Tools and Utilities to Monitor and Troubleshoot Cisco CallManager..........................................10

Cisco CallManager Administration Details ..............................................................................10

Microsoft Performance..............................................................................................................11

Opening Microsoft Performance...........................................................................................11

Customizing Performance.....................................................................................................11

Microsoft Event Viewer............................................................................................................12

Opening Event Viewer..........................................................................................................12

Detailed Information about Events .......................................................................................13

SDI Trace ..................................................................................................................................13

SDI Trace Output..................................................................................................................13

Configuring Traces................................................................................................................14

SDL Trace.................................................................................................................................15

Enabling SDL Trace..............................................................................................................15

Disk Space Warning..............................................................................................................17

Sniffer Trace..............................................................................................................................18

Sniffer Trace Applications ....................................................................................................18

Call Detail Records (CDR) and Call Management Records (CMR) ........................................18

Enabling or Disabling CDRs.................................................................................................19

CDRs.....................................................................................................................................20

Diagnostic CDRs (Also Known As CMRs)..........................................................................20

Problem Categories.......................................................................................................................21

Voice Quality ............................................................................................................................21

Lost or Distorted Audio.........................................................................................................21

i Button Help.........................................................................................................................24

Crackling...............................................................................................................................25

Check Your Loads.................................................................................................................25

Echo.......................................................................................................................................25

Check Your Loads.............................................................................................................26

One-Way Audio or No Audio...............................................................................................26

MTP and One-Way Audio ....................................................................................................27

Phone Resets .............................................................................................................................27

Dropped Calls............................................................................................................................28

Check Your Loads.................................................................................................................31

Cisco CallManager Feature Issues............................................................................................31

Codec/Regions: Codec Mismatch.........................................................................................31

Locations...............................................................................................................................32

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Conference Bridge.................................................................................................................32

Transcoding Problems...........................................................................................................33

MTP Resource Problems.......................................................................................................35

Dial Plans ..............................................................................................................................37

Partitions................................................................................................................................38

Security..................................................................................................................................40

Slow Server Response...............................................................................................................42

Reorder Tone Through Gateways .............................................................................................42

Gateway Registration Problems................................................................................................43

Gatekeeper Problems.................................................................................................................48

Inter-Cluster Trunks Only.....................................................................................................48

Admission Rejects (ARJ)......................................................................................................48

Registration Rejects (RRJ)....................................................................................................49

Intra-Cluster Cisco IP Phone-to-Cisco IP Phone Calls.................................................................50

Sample Topology ......................................................................................................................50

Cisco IP Phone Initialization Process........................................................................................50

Skinny Station Registration Process .........................................................................................52

Cisco IP Phone-to-Cisco IP Phone Call Flow within a Cluster ................................................53

Cisco IP Phone-to-Cisco IP Phone Exchange of Skinny Station Messages during Call Flow.54

Cisco CallManager Initialization Process.................................................................................55

Self-Starting Processes..............................................................................................................56

Cisco CallManager Registration Process..................................................................................57

Cisco CallManager KeepAlive Process ....................................................................................58

Cisco CallManager Intra-Cluster Call Flow Traces..................................................................59

Cisco IP Phone-to-Cisco IOS Gateway Calls ...............................................................................63

Sample Topology ......................................................................................................................63

Call Flow Traces .......................................................................................................................63

Debug Messages and Show Commands on the Cisco IOS Gatekeeper....................................66

Debug Messages and Show Commands on the Cisco IOS Gateway........................................67

Cisco IOS Gateway with T1/PRI Interface...............................................................................70

Cisco IOS Gateway with T1/CAS Interface .............................................................................71

Inter-Cluster Cisco IP Phone-to-Cisco IP Phone Calls.................................................................73

Sample Topology ......................................................................................................................73

Inter-Cluster H.323 Communication.........................................................................................73

Call Flow Traces .......................................................................................................................74

Failed Call Flow........................................................................................................................75

Call Detail Records (CDRs and CMRs)........................................................................................77

Writing Records ........................................................................................................................77

Reading Records .......................................................................................................................77

Removing Records....................................................................................................................78

Table Schema............................................................................................................................78

Known Issues ............................................................................................................................79

IP to Device Name Translation............................................................................................. 79

OnNet vs. OffNet ..................................................................................................................79

OffNet Digits Dialed.............................................................................................................79

Fields in a Call Detail Record...................................................................................................79

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Field Data Conversions.........................................................................................................79

Time Values ..........................................................................................................................79

Deciphering the Time Stamp.............................................................................................80

IP Addresses..........................................................................................................................80

Converting IP Addresses...................................................................................................80

CDR Field Definition............................................................................................................81

CMR Field Definitions..........................................................................................................85

Call Records Logged By Call Type..........................................................................................86

Normal Calls (Cisco IP Phone-to-Cisco IP Phone)...............................................................87

Abandoned Calls...................................................................................................................87

Forwarded or Redirected Calls..............................................................................................87

Calls With Busy or Bad Destinations....................................................................................87

Call Management Records Logged By Call Type ....................................................................88

Normal Calls .........................................................................................................................88

Abandoned Calls...................................................................................................................88

Forwarded Calls ....................................................................................................................88

Calls With Busy or Bad Destinations....................................................................................88

Codec Types (Compression / Payload types)............................................................................89

Cause Codes..............................................................................................................................89

Alarms.......................................................................................................................................91

Unable to write CDR data. (Alarm # 1711 - Major Alarm)..................................................91

Calling Cisco Technical Assistance Center (TAC).......................................................................92

Index.............................................................................................................................................. 93

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Purpose

This troubleshooting guide provides descriptions of the tools and utilities used to configure, monitor, and troubleshoot Cisco CallManager Release 3.0(1), Cisco IOS Gateways and Gatekeeper. Appendices provide detailed examples of three different call flows. In the first case study, a Cisco IP Phone calls another Cisco IP Phone within a cluster, which is called an intracluster call. In the second case study, a Cisco IP Phone calls through a Cisco IOS Gateway to a phone hanging off of a local PBX or somewhere on the PSTN. In the third case study, a Cisco IP Phone calls another Cisco IP Phone in a different cluster, which is called an intercluster call. Once you understand the call flow and debug traces, it will be easier to isolate a problem and determine which component is causing the problem. This document helps you understand the tools available to troubleshoot potential problems and to understand the call flows and series of events through the call traces and debug outputs.

In the event that you must contact the Cisco Technical Assistance Center (TAC), many of the tools explained here are instrumental in gathering the data required by TAC. Having this information before calling TAC assists with faster problem resolution.

Version

All discussions in this document are written for Cisco CallManager Release 3.0(1), unless otherwise stated.

Topology

It is very important to have an accurate topology of the network that contains the ports to which various components are connected, such as VLANs, routers, switches, gateways, and so on. Having a well-documented topology will assist you in troubleshooting problems with the system. You need to ensure that you have an accurate topology, access to all the network devices, and terminal services for management of the Cisco CallManager.

Adding IP telephony to a new or existing network requires significant planning to ensure success. Since real-time traffic has different requirements than data traffic, the network must be designed with low latency and quality-of-service (QoS) in mind. As with any network that carries mission-critical traffic, it is imperative that the network administrator maintains accurate, detailed diagrams of the network topology. In a crisis situation it is important to know not just the broad overview of the network, but also which ports are connected to network components, such as routers, switches, Cisco CallManager servers, gateways, and other critical devices. It is important to plan the network with redundancy and scalability in mind.

Caution: Cisco does not support using hubs for shared connectivity to the switches as they can interfere with correct operation of the IP telephony system.

When working with switched networks, knowing the state of the spanning-tree for redundancy is critical. The state of the network should be documented before any failure occurs.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Documentation Checklist

Use the following checklist to be sure you have the proper documentation on your network topology.

• Topology that shows all network devices and critical components with port/interface

numbers to which they are attached, and what VLAN (if applicable) to which they

belong. Special designations should be used for ports that are in trunking or channeling

mode.

• The root of the spanning-tree should be configured and all normally blocking ports

should be identified.

• Any WAN circuits should be identified with the amount of bandwidth (CIR in the case of

frame-relay).

Note: The Cisco IP Phone 7960 has a 10/100-switched network port and a 10/100 PC port. Cisco does not support “cascading” phones off of the PC port. We do not recommend attaching both the network and PC ports to a switch (thereby creating a physical loop in the network).

Any WAN interface will require special consideration, since this is a potential source of congestion. Cisco IP Phones and gateways set the RTP stream IP precedence field to five, however this only tags the RTP packet. It is up to the network administrator to ensure that the network is configured for prioritization and call admission control so that the Voice over IP (VoIP) traffic can be serviced with minimal delay and contention for resources. For additional information on this topic, see:

http://www.cisco.com/warp/public/793/voip/

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Glossary of Terms

Following are some common terms and acronyms that may be used in this document.

Glossary

Acronym/Term Definition .cnf

µ-law (“mu-law”)

A-law

ACF ANI ARQ B-Channel

Calling Search Space

CCAPi CCO

CDR

Cisco IOS

Cluster

CMR

codec

D-Channel

DCF DHCP

Configuration file used by devices.

Companding technique commonly used in North America. µ-law is standardized as a 64-kbps codec in ITU-T G.711. ITU-T companding standard used in the conversion between analog and digital signals in PCM systems. A-law is used primarily in European telephone networks and is similar to the North American µ-law standard. Admission Confirm. The calling number Admission Request. Bearer channel. In ISDN, a full-duplex, 64-kbps channel used to send user data. The Calling Search Space defines what directory numbers and route patterns a given device can call. It is a grouping of partitions to look through when making a call. For example, assume there are several Partitions in a Calling Search Space named “Executive.” If a Cisco IP Phone number is in the “Executive” Calling Search Space, then when initiating a call, it looks for the example “NYInternationalCall,” “NYLongDistance,” “NYLocalCall,” and “NY911” Partitions available to search through. A Cisco IP Phone number that has a “Guest” Calling Search Space, for example, might only be allowed to search through “NYLocalCall” and “NY911” Partitions, so that if the user tries to dial an international number, it won’t find a match and the call can’t be routed. Call Control API. Used by Cisco IOS to handle VoIP call processing. Cisco Connection Online (http://www.cisco.com information on Cisco products, technical support information, and technical documentation. Call Detail Record. Information about call origination, destination, and duration, used to create billing records. Cisco system software that provides common functionality, scalability, and security for all products under the CiscoFusion architecture. Cisco IOS allows centralized, integrated, and automated installation and management of internetworks, while ensuring support for a wide variety of protocols, media, services, and platforms. Cisco CallManager cluster. A logical grouping of several Cisco CallManager servers. Call Management Records, also known as Diagnostic CDRs. Records that contain the count of bytes sent, packets sent, jitter, latency, dropped packets, and so on. Coder-Decoder. A DSP software algorithm used to compress/decompress speech or audio signals. Data channel. Full-duplex, 16-kbps (BRI) or 64-kbps (PRI) ISDN channel. Used for signaling and control. Disengage Confirm. Dynamic Host Configuration Protocol. Provides a mechanism for allocating IP addresses dynamically so that addresses can be reused when hosts no longer need them. Directory Number. This is the phone number of an end device. It can be a number assi

ned to a Cisco IP Phone, a Cisco IP SoftPhone, fax

). Provides the latest

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Glossary

DNIS DNS

DRQ DTMF

Flow

Full duplex

G.711

G.729

H.225

H.245 H.323

Half Duplex

Hookflash

ICCP ISDN

Jitter

µ-law (“mu-law”)

MGCP

MTP Partition

PBX

machine, or analog phone attached to a gateway. Examples include 1000, 24231, and so on. Dialed Number Identification Service. Domain Name System. System used in the Internet for translating names of network nodes into addresses. Disengage Request. Dual tone multifrequency. Use of two simultaneous voice-band tones for dialing (such as touch tone). Stream of data traveling between two endpoints across a netw ork (for example, from one LAN station to another). Multiple flows can be transmitted on a single circuit. Capability for simultaneous data transmission from both a sending station and a receiving station. Describes the 64-kbps PCM voice coding technique. In G.711, encoded voice is already in the correct format for digital voice delivery in the PSTN or through PBXs. Described in the ITU-T standard in its G-series recommendations. Describes CELP compression where voice is coded into 8-kbps streams. There are two variations of this standard (G.729 and G.729 Annex A) that differ mainly in computational complexity; both provide speech quality similar to 32-kbps ADPCM. Described in the ITU-T standard in its G-series recommendations. An ITU standard that governs H.225 session establishment and packetization. H.225 actually describes several different protocols: RAS, use of Q.931, and use of RTP. An ITU standard that governs H.245 endpoint control. Extension of ITU-T standard H.320 that enables videoconferencing over LANs and other packet-switched networks, as well as video over the Internet. Capability for data transmission in only one direction at a time between a sending station and a receiving station. BSC is an example of a halfduplex protocol. Short on-hook period usually generated by a telephone-like device during a call to indicate that the telephone is attempting to perform a dial-tone recall from a PBX. Hookflash is often used to perform call transfer. Intra-Cluster Control Protocol Integrated Services Digital Network. Communication protocol, offered by telephone companies, that permits telephone networks to carry data, voice, and other source traffic. The variation in the arrival times of voice packets.

Companding technique commonly used in North America. µ-law is standardized as a 64-kbps codec in ITU-T G.711. Media Gateway Control Protocol. A protocol for Cisco CallManager to control VoIP gateways (MGCP endpoints). Media Termination Point. A Partition is a logical grouping of Directory Numbers and Route Patterns with similar reachability characteristics. For simplicity, these are usually named for their characteristic, such as "NYLongDistance", "NY911", etc. When a DN or Route Pattern is placed into a certain partition, this creates a rule for who can call that device or Route List. Private Branch Exchange. Digital or analog telephone switchboard located on the subscriber premises and used to connect private and public telephone networks.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Glossary

PRI

PSTN

Q.931

RAS

Route Filter

Route Group

Route List

Route Pattern

RRJ RTP

SEP

Silence Suppression (Voice Activation Detection)

SNMP

SQL

T1/CAS

PRI is Primary Rate Interface. Primary rate access consists of a single 64-Kbps D channel plus 23 (T1) or 30 (E1) B channels for voice or data. Public Switched Telephone Network. General term referring to the variety of telephone networks and services in place worldwide. ITU standard that describes ISDN signaling. The H.225.0 standard uses a variant of Q.931 to establish and disconnect H.323 sessions. Registration, Admission, and Status protocol. Protocol used in the H.323 protocol suite for discovering and interacting with a gatekeeper. A route filter can be used not only to restrict dialing, but also to identify a subset of a wildcard pattern (when using the @ wildcard in the North American Dialing Plan). For example, it could be used to block the dialing of 900 area codes. In can also be used in conjunction with Partitions and Calling Search Spaces to set up complex rules. For example, assume you have three user groups established, Executive, Staff, and Guest. A Route Filter can allow the Executive user group to dial international numbers; while the Staff user group can only dial local numbers or long distance calls; and the Guest user group can only dial local numbers, 911, and 800 numbers. A Route Group is a list of one or more gateways or ports on gateways that are seen as equal access. It is analogous to a trunk group in traditional PBX terminology. For instance, you may have two PRI circuits to the same carrier that can be used arbitrarily. A gateway (or a particular port on a gateway) can only be added to one Route Group. Formerly called Route Point, the Route List allows Cisco CallManager to hunt through a list of Route Groups in a configured order of preference. Multiple Route Lists can point to the same Route Groups. A specific number or, more commonly, a range of dialed numbers that will be used to route calls to a device (such as a Cisco Access DT-24+ Gateway or a voice-capable router) or indirectly via a Route List. For example, 1XXX signifies 1000 through 1999. The ’X’ in 1XXX signifies a single digit, a wildcard. There are other such wildcards (such as @, .,!, etc). A Route Pattern does not have to be unique within a partition as long as the Route Filter is different. Registration Reject. Real-Time Transport Protocol. One of the IPv6 protocols. RTP is designed to provide end-to-end network transport functions for applications transmitting real-time data, such as audio, video, or simulation data, over Multicast or Unicast network services. RTP provides services such as payload type identification, sequence numbering, time stamping, and delivery monitoring to real-time applications. Selsius Ethernet Phone. Acronym that precedes MAC Addresses on Cisco IP Phones, and represents a unique device identifier. Silence Suppression allows a Cisco IP Phone to detect the absence of audio and does not transmit packets over the network. The sound quality may be slightly degraded but the connection may also use less bandwidth. Silence Suppression is disabled by default. Simple Network Management Protocol. Network management protocol used almost exclusively in TCP/IP networks. SNMP provides a means to monitor and control network devices, and to manage configurations, statistics collection, performance, and security. Structured Query Language. International standard language for defining and accessing relational databases. T1 is a di

ital WAN carrier facility, transmitting DS-1-formatted data at

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Glossary

T1/PRI

TCP

TFTP

Translation Pattern

UDP

Voice Activation Detection (Silence Suppression)

VoIP VLAN

1.544 Mbps through the telephone-switching network, using AMI or B8ZS coding. CAS is a Channel Associated Signaling interface. T1 is a digital WAN carrier facility, transmitting DS-1-formatted data at

1.544 Mbps through the telephone-switching network, using AMI or B8ZS coding. PRI is Primary Rate Interface. Primary rate access consists of a single 64-Kbps D channel plus 23 (T1) or 30 (E1) B channels for voice or data. Transmission Control Protocol. Connection-oriented transport layer protocol that provides reliable full-duplex data transmission. TCP is part of the TCP/IP protocol stack. Trivial File Transfer Protocol. Simplified version of FTP that allows files to be transferred from one computer to another over a network. Used to translate called (DNIS) and calling (ANI) numbers before routing the call. For example, a call may come in to a set of numbers 919 392-3XXX that need to be translated to a set of Cisco IP Phones that are in the range of 2XXX. Cisco CallManager has a Translation Pattern set up for 919 392-3XXX. This pattern translates the leading 919 392-3 simply to 2 while leaving the remaining digits intact. Then the call is routed to the appropriate Cisco IP Phone. Translation Patterns are used only for true translations and should not be used for simple digit stripping and prefixing. User Datagram Protocol. Connectionless transport layer protocol in the TCP/IP protocol stack. UDP is a simple protocol that exchanges datagrams without acknowledgments or guaranteed delivery, requiring that error processing and retransmission be handled by other protocols. UDP is defined in RFC 768. Voice Activation Detection allows a Cisco IP Phone to detect the absence of audio and does not transmit packets over the network. The sound quality may be slightly degraded but the connection may also use less bandwidth. VAD/Silence Suppression is disabled by default. Voice over IP. virtual LAN. Group of devices on one or more LANs that are configured (using management software) so that they can communicate as if they were attached to the same wire, when in fact they are located on a number of different LAN segments. Because VLANs are based on logical instead of physical connections, they are extremely flexible.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Tools and Utilities to Monitor and Troubleshoot Cisco CallManager

This section addresses the tools and utilities to configure, monitor and troubleshoot Cisco CallManager.

Cisco CallManager Administration Details

Cisco CallManager Administration provides version information for the system, database and other components. On the opening page, press the Details button and write down the versions in use.

A more detailed explanation of Cisco CallManager Administration is available at the following location:

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/3_0/index.htm

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

View Report

and hold the Ctrl key

Microsoft Performance

Performance (Monitor) is a Windows 2000 server application that can display the activities and status of your Cisco CallManager system. It reports both general and specific information in real time. You can use Windows 2000 Performance to collect and display system and device statistics for any Cisco CallManager installation. This administrative tool allows you to gain a full understanding of a system without studying the operation of each of its components.

You can use Performance to monitor a variety of system variables in real time. After adding the Cisco CallManager parameters, you can define the terms under which Cisco CallManager will display statistics generated by the system. For example, you can monitor the number of calls in progress at any time, or the number of calls currently passing through a specific gateway. Performance shows both general and Cisco CallManager-specific status information in real-time.

Add Counters

Click the Add

Click the View button and then the Add button and make selections on the dialog box (press

button. The Add Counters dialog box is displayed.

to select multiple items in the list). Click Add and then Close and view the report in the window.

Opening Microsoft Performance

To open Performance on the server PC running Cisco CallManager, click Start > Settings > Control Panel > Administration Tools > Performance.

Customizing Performance

The Performance monitor must be customized to view the Cisco CallManager-related parameters that you wish to monitor. Choose the object, counter, and instance you want to include. Please refer to the Remote Serviceability documentation for instructions on how to use objects and counters to customize Microsoft Performance for Cisco CallManager operations.

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/3_0/service/index.htm

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Microsoft Event Viewer

Microsoft Event Viewer is a Windows NT Server application that displays system, security, and application events (including Cisco CallManager) for the Windows NT Server. If a service (including TFTP) cannot read the database (where it gets the trace configuration), it will add errors to the Event Viewer. The Event Viewer is the only place where these types of errors will appear. The following illustration shows the application logs running on a Windows NT Server.

Opening Event Viewer

To open the Event Log on the server PC running Cisco CallManager, click Start > Settings > Control Panel > Administrative Tools > Event Viewer. The Event Viewer provides error logs

for System, Security, and Applications. Cisco CallManager errors are logged under the Application log.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Detailed Information about Events

You can double-click an event in the log to learn more information about the event.

SDI Trace

SDI traces are local log files. The IP address, TCP handle, device name or the time stamp can be used when reviewing the SDI trace to monitor the occurrence or the disposition of a request. This device name could be tracked back to the building of the file, which shows the device pool and model. The device pool and model can be tracked back to the building of the configuration file prototype, which will list the network address of the Cisco CallManager(s) and the TCP connection port.

When observing SDI traces, notice that C++ class and routine names are included with most trace lines. Most routines associated with the serving of a particular request include the thread ID in a standard format.

SDI traces will be explained in detail in the case studies in the appendices.

SDI Trace Output

SDI traces generate files (for example, CCM000000000) that store traces of Cisco CallManager activities. These traces provide information about the Cisco CallManager initialization process, registration process, KeepAlive process, call flow, digit analysis, and related devices such as Cisco IP Phones, Gateways, Gatekeepers, and more. This information can help you isolate problems when troubleshooting Cisco CallManager. To properly track the information you

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

need—and only the information you need—it’s important to understand how to set the options on the trace configuration interface.

The trace files are stored in the following default location: C:\Program Files\cisco\bin. A new trace file is started each time Cisco CallManager restarts, or when the designated number of lines has been reached.

Following is an illustration of the Cisco CallManager Administration trace configuration interface. You must enable the trace, choose the level on information needed, and check the user mask to obtain the desired level of information.

If the trace is not configured properly, it will generate a large amount of information making it very difficult to isolate problems. The following section explains how to properly configure a useful trace.

Configuring Traces

Traces are composed of user mask flags (also known as bits) and trace levels. Open Cisco CallManager Administration. To turn on tracing, set your trace parameters (including configured service, bits, and so on) in the Service>Trace screen. Refer to the Cisco CallManager

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

documentation for complete information about turning tracing on and off, and for descriptions of the User Masks and Levels for each configured service, and more.

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/3_0/admin_gd/admin_gd/inde x.htm

Following are two examples of trace mask bits that would be enabled based on the particular problem.

• For normal message debugging, turn on subsystem bits 5, 6, 7, 8, 11, and 12

• For debugging gateways, turn on subsystem bits 3,4,5,6,7,8,9,11,12,13

Following are two examples of desired trace levels based on the particular problem

• For normal debugging, the trace level should be set to SDI_LEVEL_ARBITRARY

• For normal running system, the trace level should be set to SDI_LEVEL_ERROR

SDL Trace

Cisco engineers use SDL traces to find the cause of an error. You are not expected to understand the information contained in an SDL trace. However, while working with TAC, you may be asked to enable the SDL trace and provide it to the TAC. SDL trace files can be saved to local directories, the Windows NT Event Viewer, and CiscoWorks 2000. To avoid any performance degradation on the server, be sure that after the trace has been captured, you turn off SDL tracing.

SDL trace provides a C interface to trace and alarms. Alarms are used to inform the administrator of unexpected events, such as being unable to access a file, database, Winsock, or being unable to allocate other operating system resources.

Enabling SDL Trace

SDL traces are enabled in the Service > Service Parameter area in Cisco CallManager Administration. Remember that these traces should be turned on only when requested by a TAC engineer. Note the values chosen to turn on the SDL trace in the following illustration.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Once SDL traces are enabled, collect the traces. If the traces are being sent to the local drive, then you can retrieve them in the Cisco\Trace subdirectory. Alternatively, the trace files can be sent to an event log or to CiscoWorks 2000.

SDL flag bits described in the following table are set in the Service > Service Parameters area in Cisco CallManager Administration. Following are two examples of desired values based on the particular problem.

• The recommended value for normal call debugging is SdlTraceTypeFlags=0x00000b04

• The recommended value for low level debugging or debugging gateways is

SdlTraceTypeFlags=0x00004b05

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

SdlTraceTypeFlags Definitions

SDLTraceTypeFlag Value Definition

traceLayer1 = 0x00000001 All Layer 1 trace on TraceDetailLayer1 = 0x00000002 Detail Layer 1 trace on TraceSdlLinkAdmin = 0x00000004 Trace inter-Cisco CallManager links within a cluster traceUnused = 0x00000008 Not used traceLayer2 = 0x00000010 All Layer 2 trace on traceLayer2Interface = 0x00000020 Layer 2 interface trace on traceLayer2TCP = 0x00000040 Layer 2 TCP trace on TraceDetailLayer2 = 0x00000080 More detail dump of Layer 2 Frames. traceLayer3 = 0x00000100 All Layer 3 trace on traceCc = 0x00000200 All call control trace on traceMiscPolls = 0x00000400 Trace miscellaneous polls traceMisc = 0x00000800 Miscellaneous trace on (Database signals) traceMsgtrans = 0x00001000 Message Translation signals (TranslateIsdnToSdlReq,

TranslateIsdnToSdlRes TranslateSdlToIsdnReq,

TranslateSdlToIsdnRes) traceUuie = 0x00002000 UUIE output trace on traceGateway = 0x00004000 Gateway signals

Data bits described in the following table are set in the Service > Service Parameters area in Cisco CallManager Administration. Following are two examples of desired values based on the particular problem.

• The recommended value for normal system debugging is SdlTraceDataFlags=0x110

• The recommended value when tracking problems with SDL links is 0x13D (non-

compacted trace; if a compact trace is desired, bit 0x200 must be set. It can be set in combination with any other bits)

SDLTraceDataFlags Definitions

SDLTraceDataFlag Value Definition

TraceSdlLinkState = 0x001 Enable trace of SDL Link Initialization TraceSdlLowLevel = 0x002 Enable tracing of low-level SDL events, for example, fileOpen,

socket events, and so on TraceSdlLinkPoll = 0x004 Enable tracing of SDL Link Poll message TraceSdlLinkMsg = 0x008 Enable tracing of SDL Link Message traceRawData = 0x010 Enable raw signal data trace on all signals TraceSdlTagMap = 0x020 Enable tag mapping traceCreate = 0x100 Enable process create and stop traces TraceNoPrettyPrint = 0x200 Disable pretty printing of trace files

Disk Space Warning

IMPORTANT: Be advised that information obtained from this interface could be very detailed, and therefore consume a large amount of disk space. For this reason, we advise you to turn on the trace file for a specific amount of time, then review the information and turn off the trace.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Sniffer Trace

A Sniffer is a software application that monitors IP traffic on a network and provides information in the form of a trace. Sniffer traces provide information about the quantity and type of network traffic on your network. TCP/IP or UDP packets are protocols utilized by Cisco CallManager and endpoint devices such as phones and gateways. Sniffer traces can also help you identify high levels of broadcast traffic that could result in voice audio problems or dropped calls. Common Sniffer applications include Network Associates SnifferPro, Hewlett Packard Internet Advisor, and W&G Domino. Domino offers sniffing hardware and software solutions and a network analyzer. If you want to use Domino, we recommend using the analysis software to evaluate a captured sniffer file (such as from the SnifferPro application).

Sniffer Trace Applications

Use the following links to learn more about some available sniffer trace applications. Any sniffer application will work with Cisco CallManager.

• Network Associates SnifferPro: http://www.sniffer.com/

• Hewlett Packard Network Analyzer:

http://www.hp.com/rnd/products/netman/netmgt.htm

• W&G Domino Analyzer: http://www.wwgsolutions.com/products/domino/domino.html

Call Detail Records (CDR) and Call Management Records (CMR)

Call Detail Record (CDR) is a reporting option that logs every call made (or attempted) from any Cisco IP Phone. There are two kinds of CDRs—basic CDRs and Diagnostic CDRs, or CMRs. Once enabled, you can open CDRs or Diagnostic CDRs (CMRs) in the SQL Server Enterprise Manager. CDR files are saved in a SQL database that can be exported to nearly any application, including Microsoft Access or Excel.

CDR records contain information needed to generate billing records. In a distributed environment, all CDR data is collected in a central location, or a set of locations. The failure of a Cisco CallManager node does not make the CDR data associated with that node unavailable, since the data is no longer stored on the Cisco CallManager disk as a flat file, but is instead stored in a central database in tables.

If the Cisco CallManager fails before any records are written, then no record of the call will exist. This means that no record will be written for calls that are active on a given Cisco CallManager when it fails before the calls terminate.

Refer to the Appendix in the back of this book for detailed information about CDRs and CMRs. The information provided includes:

• Reading and Writing Records

• Known Issues

• List of record types generated

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

• List of fields contained in each record and a description of what that field represents

• Description of the types of calls logged, and the fields logged with each of them

• List of cause codes that may appear in the CDR records

Enabling or Disabling CDRs

CDR record creation is disabled by default when the system is installed. If you wish to have CDR data, you must enable CDRs in the Service > Service Parameters area of Cisco CallManager Administration. CDR processing can be enabled and disabled at any time while the system is in operation. You do not need to restart Cisco CallManager for the enabling or disabling of CDRs to take effect. The system will respond to all changes within a few seconds. CMR or diagnostic data is enabled separately from CDR data. CMR data will not be generated unless both CDRs and Call Diagnostics are enabled, but CDR data may be generated and logged without CMR data.

Use the following steps to enable CDRs.

1. Open Cisco CallManager Administration.

2. Select Service > Service Parameters.

3. Select the IP address of your Cisco CallManager installation.

4. From the list of Parameters, select CDREnabled.

5. Define type as boolean.

6. Select T for True.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

7. Update.

Result: Call Detail Records will start logging immediately.

Caution: Tracing voice connectivity requires that CDR logging be enabled on every

Cisco CallManager installation in a cluster.

CDRs

CDRs provide basic information that can help you understand the more detailed information contained in SDI traces. Basic CDRs provide information such as the calling number, called number, originating IP address, destination IP address, call duration, and so on. CDRs can help you troubleshoot phone problems. For example, if a user reports a problem with a call occurring at a specific time, you can consult the CDRs that occurred around the time indicated to learn additional information about that call and others. CDRs are commonly used for billing.

Diagnostic CDRs (Also Known As CMRs)

Diagnostic CDRs provide detailed call information, such as the number of packets sent, received, and lost, the amount of jitter and latency, and so on. This level of detail can provide explanations for some problems, such as one-way audio. For example, a one-way audio problem is indicated if a packet size of 10,000 is sent, but the received size is only 10.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Problem Categories

This section addresses some common problem categories that may occur with Cisco CallManager and related devices. Each problem category provides suggestions for the troubleshooting tools you should use to help isolate the problem. This document provides general categories of potential problems and suggestions about how to troubleshoot those problems. It does not provide an exhaustive list of problems and resolutions. If you encounter a problem that cannot be resolved using the tools and utilities described in this document, consult the Cisco Technical Assistance Center (TAC) for assistance. Be sure to have available the Cisco CallManager Administration Details, plus the diagnostic information (traces, etc.) you have gathered up to the point of calling the TAC.

Voice Quality

Voice quality issues include lost or distorted audio during phone calls. Common problems can be breaks in the sound which cause the audio to be intermittent (like broken words), or the presence of odd noises that distort the audio, such as echo, or effects that cause spoken words to sound watery or robotic. One-way audio, that is, a conversation between two people where only one person can hear anything, is not actually a voice quality issue, but will be discussed later in this section.

One or more of the following components could cause audio problems:

• Gateway

• Phone

• Network

To properly troubleshoot voice quality issues, you must consider the infrastructure and all the devices for drops and delays.

Lost or Distorted Audio

One of the most common problems encountered is a breaking up of audio (often described as garbled speech, or a loss of syllables within a word or sentence). There are two common causes for this: packet loss and/or jitter. Packet loss means that audio packets do not arrive at their destination because they were dropped or arrived too late to be useful. Jitter is the variation in the arrival times of packets. In the ideal situation, all VoIP packets from one phone to another would arrive exactly at a rate of 1 every 20 ms. Notice that this does not mention how long it takes for a packet to get from point A to point B, simply the variation in the arrival times. There are many sources of variable delay in a real network. Some of these cannot be controlled, and some can. Variable delay cannot be eliminated entirely in a packetized voice network. Digital Signal Processors (DSPs) on phones and other voice-capable devices are designed to buffer some of the audio, in anticipation of variable delay. This “dejittering” is done only when the audio packet has reached its destination and is now ready to be put into a conventional audio stream (to be played out into the user’s ear to be sent to the PSTN via a digital PCM stream). The Cisco IP Phone 7960 can buffer as much as one second of voice samples. The jitter buffer is adaptive, meaning if a burst of packets is received, the Cisco IP Phone 7960 can play them out in an attempt to control the jitter. The network administrator needs to minimize the variation

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

between packet arrival times by applying quality-of-service (QoS) and other measures in advance (especially if calls cross a wide-area network).

When faced with a lost or distorted audio problem, the first thing to do is to try to isolate the path of the audio. Try to identify each network device (switches and routers) in the path of the call’s audio stream. Keep in mind that the audio may be between two phones, or between a phone and a gateway, or it could have multiple legs (from a phone to a transcoding device and from there to another phone). Try to identify if the problem occurs only between two sites, only through a certain gateway, on a certain subnet, and so on. This will help narrow down which devices you need to look at more carefully. Next, if is often best to disable silence suppression (also known as Voice Activation Detection or VAD) if this hasn’t been done already. This mechanism does save bandwidth by not transmitting any audio when there is silence, but may cause noticeable clipping at the beginning of words that may be unacceptable. You can disable this in Cisco CallManager Administration, under Service > Service Parameters. From there, select the server and the Cisco CallManager service. Then set SilenceSuppressionSystemWide to “F” (alternatively you can set SilenceSuppressionWithGateways to “F”, but this does not apply to H.323 gateways or MGCP gateways). When in doubt, turn both off by selecting the Value F for each.

If a network analyzer is available, then a monitored call between two phones should have 50 packets per second (or 1 packet every 20 ms) when silence suppression is disabled. With proper filtering, it should be possible to identify if packets are being lost or delayed excessively.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

Remember that delay by itself won’t cause clipping, only variable delay will. Notice in the table below, which represents a perfect trace, the arrival times between the audio packets (which will have an RTP header), will be 20 ms. In a poor quality call (such as a call with a lot of jitter), the arrival times would vary greatly.

A Perfect Trace

Packet Number Time – absolute (ms) Time – delta (ms)

1 0 2 0.02 20 3 0.04 20 4 0.06 20 5 0.08 20

Placing the packet analyzer into various points in the network will help narrow down where the delay is coming from. If no analyzer is available, other methods will be required. It is important to examine interface statistics of each device in the path of the audio. Another tool for tracking calls with poor voice quality is the Diagnostic Call Detail Records (CDRs). See the CDR section in the Problem Categories section above, or Appendix D for more information about CDRs.

Then values for jitter and latency can be retrieved for all calls (but only after the call has terminated). Following is a sample Diagnostic CDR (CallDetailRecordDiagnostic is the actual table name). The number of packets sent, receive, lost, jitter, and latency are all recorded. The globalCallID value can be used to find the call in the regular CDR table so that the disconnect cause and other information can be obtained. The diagram below shows both tables open. Notice that in the Diagnostic CDR, every device that can possibly report this information is included. So if the problem is between two Cisco IP Phones, we see two table entries per call. If we have a call through a Cisco IOS Gateway, for example, we only see the diagnostic information from the Cisco IP Phone, not the gateway because there is no mechanism for it to notify the SQL database with this information.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

i Button Help

The Cisco IP Phone 7960 provides another tool for diagnosing possible audio problems. On an active call, you can press the i button twice rapidly and the phone will display an information screen that contains packet receive and transmit statistics, as well as average and maximum jitter counters. Note that on this screen, jitter is the average of the last 5 packets that arrived; the maximum jitter is the high-water mark for the average jitter.

The most common sources for delay and packet loss are devices where a higher speed interface feeds into a lower speed interface. For example: a router may have a 100 Mb fast Ethernet interface connected to the LAN and a slow frame-relay, for example, connected to the WAN. If the poor quality occurs only when communicating to the remote site (only the remote site may be reporting the poor voice quality while in the other direction everything appears to be fine), then the most likely causes of the problem include:

• The router has not been properly configured to give the voice traffic priority over the data

traffic

• There are too many calls active for the WAN to support (that is, there is no call admission

control to restrict the number of calls that can be placed)

• There are physical port errors

• There is congestion in the WAN itself

On the LAN, the most common problems are physical-level errors (such as CRC errors) caused by faulty cables, interfaces, or by incorrectly configured devices (such as a port speed or duplex

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

mismatch). Make sure that the traffic is not crossing any shared-media device, such as a hub. There could also be situations where the traffic is taking a slower path through the network than expected. If QoS has been configured correctly, then the possibility exists that there is no call admission control. Depending on your topology, this can be accomplished through the use of Locations in Cisco CallManager Administration configuration, or by using a Cisco IOS router as a gatekeeper. In any case, you should always know how many calls could be supported across your WAN. If possible, test this by disabling silence suppression as described earlier, then place calls between the two sites. Do not place the calls on hold or on mute, since this will stop packets from being transmitted. With the maximum number of calls across the WAN, the calls should all have acceptable quality. Test to make sure that a fast busy is returned when trying to make one more call.

Crackling

Another “poor quality” symptom may be a crackling, which is sometimes caused by a defective power supply or some kind of strong electrical interference close to the phone. Try swapping the power supply and moving the phone around.

Check Your Loads

You should also always check the phones and gateways to ensure the latest software loads are in use. When in doubt, check CCO (Cisco Connection Online at www.cisco.com) for the latest software loads, new patches, or release notes relating to the problem.

Echo

Echo (also known as “talker echo”) occurs when a talker’s speech energy, transmitted down the primary signal path, is coupled into the receive path from the far end. The talker then hears his or her own voice, delayed by the total echo path delay time.

John

Voice Network

Jane

In the diagram above, John’s voice (in blue) is being reflected back. This can happen but go unnoticed in a traditional voice network because the delay is so low. To the user, it sounds more like a side-tone than an echo. In a VoIP network, it will always be noticeable, since packetization and compression always contribute enough delay. The important thing to remember is that the cause of the echo is always with analog components and wiring. For instance, IP packets cannot simply turn around and go back to the source at a lower audio level. The same is impossible on digital T1/E1 circuits. So on a call from one Cisco IP Phone to another, there should never be any problem. The only exception may be if one party is using a speakerphone that has the volume set too high or some other situation where an audio loop is created.

When troubleshooting echo problems, make sure that the phones that are being tested or examined are not using the speakerphone, and that they have the headset volume to reasonable

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

levels (start with 50% of the maximum audio level). Most of the time, the problems will occur when attaching to the PSTN by way of a digital or analog gateway. Cisco IP Phone users may complain that they hear their own voice being reflected back to them. Now, although the true source of the problem is almost always at the far end, it is nearly always impossible to change anything in the PSTN. So the first step is to determine which gateway is being used. If a digital gateway is in use, then it may be possible to add additional padding in the transmit direction (towards the PSTN), in the hopes that the lower signal strength will yield less reflected energy. Additionally, you can adjust the receive level so that any reflected audio is reduced even further. It is very important to remember to make small adjustments at a time. Too much attenuation of the signal will make the audio impossible to hear on both sides. Alternatively, you can contact the carrier and request to have the lines checked. On a typical T1/PRI circuit in North America, the input signal should be –15 dB. If the signal level is much higher (-5 dB, for example), then echo will be the likely result.

A log should be kept of all calls that experience echo. The time of the problem, the source phone number, and the number called should all be recorded. Gateways have a fixed time of 16 ms of echo cancellation. If the delay in the reflected audio is longer than this, the echo chancellor will be unable to work properly. This should not be an issue for local calls, and long distance calls should have external echo chancellors built into the network at the Central Office. This is one of the reasons why it is important to note the external phone number of a call that experiences echo.

Check Your Loads

Gateway and phone loads should be verified. Check CCO (Cisco Connection Online at

www.cisco.com) for the latest software loads, new patches, or release notes relating to the

problem.

One-Way Audio or No Audio

One-way audio occurs when one person cannot hear another person during a call. This can be caused by an improperly configured Cisco IOS Gateway, a firewall, or a routing or default gateway problem, among other things.

There are a number of causes for one-way audio or no audio during a call. The most common cause is an improperly configured device. For instance, Cisco CallManager handles the call setup for a Cisco IP Phone. The actual audio stream occurs between the two Cisco IP Phones (or between the Cisco IP Phone and a gateway). So it is entirely possible that the Cisco CallManager is able to signal to a destination phone (making it ring) when the phone originating the call does not have an IP route to the destination phone. A common cause for this is when the default gateway in the phone is improperly configured manually or on the DHCP server.

If a call consistently has one-way audio, take a PC that is on the same subnet as the phone and has the same default gateway and try to ping the destination Cisco IP Phone. Then take a PC that is on the same subnet as the destination phone (with the same default gateway as the destination phone) and ping the source phone. Both of those tests should work. Other things can affect the audio traffic include a firewall or packet filter (such as access lists on a router) that may be blocking the audio in one or both directions. If the one-way audio occurs only through a voice-

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

enabled Cisco IOS Gateway, then check the configuration carefully. IP routing must be enabled (look at the configuration to make sure that “no ip routing” is not found near the beginning of the configuration). Also, make sure that if you’re using RTP header compression to save bandwidth across the WAN, that it is enabled on each router carrying voice traffic that attaches to the WAN circuit. There should not be a situation where the RTP header is compressed on one end but cannot be de-compressed on the other side of the WAN. A sniffer is a very useful tool when troubleshooting one-way audio problems, since you can then verify that the phone or gateway is actually sending or receiving packets. Diagnostic Call Detail Records (CDRs) are useful for determining if a call is experiencing one-way audio since they log transmitted and received packets (see “Lost or Distorted Audio” section). You can also press the i button twice quickly on a Cisco IP Phone 7960 during an active call to view details about transmitted and received packets.

Note: When a call is muted, no packets will be transmitted from the phone that has pressed the mute button. The Hold button stops the audio stream, so no packets are sent in either direction. When the Hold button is released, all the packet counters are reset. Remember that Silence Suppression must be disabled on both devices for the TX and RX counters to stay equal. Disabling Silence Suppression system-wide will not affect Cisco IOS Gateways.

MTP and One-Way Audio

If you are using Media Termination Point (MTP) in a call (to support supplementary services such as hold and transfer with H.323 devices that do not support H.323 version 2), check to see if the MTP allocated is working correctly. Cisco IOS routers support H.323 version 2 beginning in release 11.3(9)NA and 12.0(3)T. Starting with Cisco IOS release 12.0(7)T, the optional H.323 Open/Close LogicalChannel is supported, so that software-based MTP is no longer required for supplementary services.

The MTP device, as well as Conference Bridge and Transcoder, will bridge two or more audio streams. If the MTP, Conference Bridge, or Transcoder is not working properly, one-way audio or audio loss might be experienced. Shut down MTP to find out if MTP is causing the problem.

Phone Resets

Phones will power cycle or reset for two reasons: 1) TCP failure connecting to Cisco CallManager, or 2) failure to receive an acknowledgement to the phone’s KeepAlive messages.

Steps for troubleshooting phone resets:

1. Check the phones and gateways to ensure that you are using the latest software loads.

2. Check CCO (Cisco Connection Online at www.cisco.com) for the latest software loads, new patches, or release notes relating to the problem.

3. Check the Event Viewer for instances of phone(s) resetting. Phone resets are considered Information events, as shown in the following illustration.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

4. Look for these and any errors that may have occurred around the time that the phone(s) reset.

5. Start an SDI trace and try to isolate the problem by identifying any common characteristics in the phones that are resetting. For example, check whether they are all located on the same subnet, same VLAN, and so on. Look at the trace and determine:

• If the resets occur during a call or happen intermittently

• If there any similarities of phone model – Cisco IP Phone 7960, Cisco IP Phone 30VIP,

etc.

6. Start a Sniffer trace on a phone that frequently resets. After it has reset, look at the trace to determine if there are any TCP retries occurring. If so, this indicates a network problem. The trace may show some consistencies in the resets, such as the phone resetting every seven days. This might indicate DHCP lease expiration every seven days (this value is userconfigurable; could be every two minutes, etc.).

Dropped Calls

Dropped calls occur when a call is prematurely terminated. You can use CDRs to determine the possible cause of dropped calls, particularly if the problem is intermittent. Dropped calls can be the result of a phone or gateway resetting (see above section) or a circuit problem, such as incorrect PRI configuration or error.

The first step is to determine if this problem is isolated to one phone or a group of phones. Perhaps the affected phones are all on a particular subnet or location. The next step is to check the Event Viewer for phone or gateway resets.

Cisco IP Telephony Troubleshooting Guide for Cisco CallManager Release 3.0(1)

There should be one Warning and one Error message for each phone that resets. In this case, the problem is often that the phone cannot keep its TCP connection to the Cisco CallManager alive, so the Cisco CallManager resets the connection. This may be because a phone was turned off or there may be a problem in the network. If this is an intermittent problem, it may be useful to use Microsoft Performance to record phone registrations.

+ 67 hidden pages