Copyright 1996-2005 by EMC Corporation (“EMC”). All rights reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The Software and all intellectual property rights related thereto constitute trade secrets and proprietary data of EMC and any third party
from whom EMC has received marketing rights, and nothing herein shall be construed to convey any title or ownership rights to you. Your
right to copy the software and this documentation is limited by law. Making unauthorized copies, adaptations, or compilation works is
prohibited and constitutes a punishable violation of the law. Use of the software is governed by its accompanying license agreement.
The information in this publication is provided “as is” without warranty of any kind. EMC Corporation makes no representations or
warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties or merchantability
or fitness for a particular purpose. In no event shall EMC Corporation be liable for any loss of profits, loss of business, loss of use of data,
interruption of business, or for indirect, special, incidental, or consequential damages of any kind, arising from any error in this publication.
The InCharge™ products mentioned in this publication are covered by one or more of the following U.S. Patent Nos. or pending patent
applications: 5,528,516, 5,661,668, 6,249,755, 6,868,367 and 11/034,192.
“EMC," “InCharge,” the InCharge logo, “SMARTS,” the SMARTS logo, “Graphical Visualization,” “Authentic Problem,” “Codebook
Correlation Technology,” “Instant Results Technology,” “InCharge Viewlet,” and “Dashboard Viewlet” are trademarks or registered
trademarks of EMC. All other brand or product names are trademarks or registered trademarks of their respective companies or
organizations.
Additional copyright notices and license terms applicable to the software product are set forth in the Third-Party Copyright Read Me file
included on the accompanying software media.
Topology Created by ACM1
Faults Diagnosed by ACM2
Analysis Scenarios3
Single Application and Single TCP Check3
Single Application and Two TCP Checks6
Two Applications on a Single System9
Impact Analysis with Business Impact Manager13
2Viewing Analysis Results and Application Topology17
Analysis Results Displayed as Notifications17
About Notifications17
Viewing the Properties of a Notification18
This document describes the topology and notifications created by
Application Connectivity Monitor as it is displayed in the Global Console.
Intended Audience
This guide is intended for operators who are responsible for acting on the
analysis provided by Application Connectivity Monitor.
Prerequisites
This guide assumes that Application Connectivity Monitor and Service
Assurance Manager are properly deployed and that the user has access to
the Global Console.
Describes the notifications and topology
created by Application Connectivity
Monitor and presents several failure
scenarios.
Provides fundamental descriptions of
various Global Console functions.
Preface
Documentation Conventions
Several conventions may be used in this document as shown in Table 2.
Tab l e 2 : Documentation Conventions
CONVENTIONEXPLANATION
sample codeIndicates code fragments and examples in Courier font
keywordIndicates commands, keywords, literals, and operators in bold
%Indicates C shell prompt
#Indicates C shell superuser prompt
<parameter>Indicates a user-supplied value or a list of non-terminal items in
[option]Indicates optional terms in brackets
/InChargeIndicates directory path names in italics
yourDomainIndicates a user-specific or user-supplied value in bold, italics
File > OpenIndicates a menu path in italics
▼▲
angle brackets
Indicates a command is wrapped over one or more lines. The
command must be typed as one line.
Directory path names are shown with forward slashes (/). Users of the
Windows operating systems should substitute back slashes (\) for forward
slashes.
Also, if there are figures illustrating consoles in this document, they represent
the consoles as they appear in Windows. Under UNIX, the consoles appear
with slight differences. For example, in views that display items in a tree
hierarchy such as the Topology Browser, a plus sign displays for Windows
and an open circle displays for UNIX.
Finally, unless otherwise specified, the term InCharge Manager is used to
refer to EMC Smarts programs such as Domain Managers, Global
Managers, and adapters.
In this document, the term BASEDIR represents the location where the
Application Connectivity Monitor software is installed.
•For UNIX, this location is: /opt/InCharge<n>/<product>.
•For Windows, this location is: C:\InCharge<n>\<product>.
The <n> represents the software platform version number. The <product>
represents the product name. For example, on UNIX operating systems,
Application Connectivity Monitor is, by default, installed to:
/opt/InCharge6/ACM/smarts. On Windows operating systems, this
product is, by default, installed to: C:\InCharge6\ACM\smarts. This
location is referred to as BASEDIR/smarts.
Optionally, you can specify the root of BASEDIR to be something other than /opt/InCharge6 (on UNIX) or C:\InCharge6 (on Windows), but you cannot
change the <product> location under the root directory.
For more information about the software directory structure, refer to the EMC Smarts System Administration Guide.
Application Connectivity Monitor Products
Application Connectivity Monitor includes the following products:
•Application Connectivity Monitor
Additional Resources
In addition to this document, EMC Smarts provides the following resources.
Command Line Programs
Descriptions of command line programs are available as HTML pages. The
index.html file, which provides an index to the various commands, is located
in the BASEDIR/smarts/doc/html/usage directory.
For questions about technical support, call your local sales office or service
provider. For service, call one of the following numbers:
United States: 800.782.4362 (SVC.4EMC)
Canada: 800.543.4782 (543.4SVC)
Worldwide: 508.497.7901
EMC Powerlink
EMC Powerlink is the EMC Corporation’s secure extranet for customers and
partners. Powerlink is an essential tool for obtaining web-based support from
the EMC Corporation. Powerlink can be used to submit service or
information requests (tickets) and monitor their progress, to review the
knowledgebase for known problems and solutions, and to download
patches and SmartPacks.
From training on EMC products and technologies, to online support, product
announcements, software registration, technical white papers,
interoperability information, and a range of configuration tools, Powerlink
offers resources unavailable elsewhere.
For quickest access when you do not already have a Powerlink account, ask
your EMC representative for the access code for your company and register
at the Powerlink site. Visit the EMC Powerlink website at:
Application Connectivity Monitor provides root-cause analysis of application
failures. In addition, Application Connectivity Monitor determines the impact
of network and system failures on TCP-based applications.
Application Connectivity Monitor (ACM) discovers and monitors the
availability of applications that provide network services using the TCP
protocol. When a TCP-based application is not available, Application
Connectivity Monitor determines whether the problem is the result of a
network failure or an application failure.
Topology Created by ACM
Application Connectivity Monitor automatically discovers TCP-based
applications that are hosted by systems (for example, hosts, switches,
routers) that are present in the topology of the ACM Domain Manager.
For each discovered application, Application Connectivity Monitor creates
the following topology elements.
•Applications elements are represented by the Application class and its
subclasses; which includes subclasses such as HTTPService and
DatabaseServer. An application element is associated with the system it
runs on through the HostedBy relationship.
Application elements are visible in the Topology Browser and
Applications Map views of the Global Console when attached to the
Global Manager.
•TCPConnectivityCheck elements represent the TCP connection initiated
by the ACM Domain Manager to check the availability of monitored
applications. Application Connectivity Monitor uses the symptoms
reported by the TCP checks to perform its analysis.
TCPConnectivityCheck elements may be listed in the Notification Log
view or Codebook tab of the Notification Properties dialog of the
Global Console when attached to the Global Manager.
TCPConnectivityCheck elements are listed when an event occurs.
Faults Diagnosed by ACM
Application Connectivity Monitor notifies the following problems for
application elements.
Down
Down is a root-cause problem that indicates that the application is not
running; either because the process has died or stopped. Application
Connectivity Monitor diagnoses the Down problem when all TCP checks
return TimedOutSymptom events.
Malfunction
Malfunction is a root-cause problem that indicates that the application is not
working properly. The application may be providing service to some users
but not other users or it may not be providing the correct response.
Application Connectivity Monitor diagnoses the Malfunction problem when
one or more, but not all, TCP checks return TimedOutSymptoms events. In
addition, one or more TCP checks may return the SlowSymptom event.
Degraded
Degraded is a root-cause problem that indicates that the application is
running slowly. Application Connectivity Monitor diagnoses the Degraded
problem when a TCP check returns the SlowSymptom event.
System Unresponsive is a root-cause problem that indicates that a system’s
protocol endpoints are not responding to ICMP or SNMP polls and that
connections to applications hosted by the system are down. Application
Connectivity Monitor diagnoses the System Unresponsive problem when it
receives an UnresponsiveSymptom from IP Availability Manager.
Analysis Scenarios
This section uses three different topology scenarios to illustrate how
Application Connectivity Monitor (ACM) diagnoses problems. In each
scenario, Application Connectivity Monitor monitors the application by
regularly establishing a TCP connection with the managed application on
the specified TCP port. In addition, Application Connectivity Monitor
receives network and systems availability information from IP Availability
Manager.
Analysis Scenarios
Single Application and Single TCP Check
In the following scenario, a mail server, SMTPService, is hosted by
host1.example.com, as shown in Figure 1.
A Down problem indicates that the host the application is running on is
responding but the application is not available.
To diagnose Down in this scenario, ACM observes that the TCP checks sent
to SMTPService time out, resulting in a TimedOutSymptom event. A
TimedOutSymtpom event occurs when ACM does not receive a response to
a TCP check.
An Unresponsive problem indicates that the system on which the application
is running cannot be reached through the network.
To diagnose System Unresponsive in this scenario, Application Connectivity
Monitor receives a host1.example.com UnresponsiveSymptom from IP
Availability Manager. In addition, ACM may observe SlowSymptom or
TimedOutSymptoms events in response to TCP checks.
Application Connectivity Monitor must receive an UnresponsiveSymptom
from IP Availability Manager to diagnose an Unresponsive problem.
Mail Server Degraded
A Degraded problem indicates that the application is not performing as
expected. This means that the application is responding to TCP checks, but
not within the expected time period.
To diagnose Degraded in this scenario, Application Connectivity Monitor
observes a SlowSymptom event for a TCP check to SMTPService. A
SlowSymptom event occurs when an application responds to the TCP check
but the response time exceeds the threshold.
Figure 2 shows the Notification Properties dialog for the SMTPService
Degraded problem. Application Connectivity Monitor diagnosed a
Degraded problem with 100% certainty because it observed the
SlowSymptom event in response to a TCP check.
Figure 2: Notification Properties Of Degraded Problem
Single Application and Two TCP Checks
In the following scenario, a Web server, HTTPService, is hosted by
host1.example.com. The Web server is configured to serve content on two
separate ports. To ensure that the Web server is functioning properly, an
administrator has configured Application Connectivity Monitor to monitor
both ports on which the Web server provides content.
A Down problem indicates that the application is not responding to
connection requests to any of its ports.
To diagnose Down in this scenario, Application Connectivity Monitor must
observe that all TCP checks to both TCP ports in use by HTTPService time
out, causing TimedOutSymptom events. The TimedOutSymptom events
indicate that the Web server is not responding to connection requests.
A Malfunction problem indicates that the application is not functioning
properly because it responds correctly to certain connection requests and
incorrectly to other connection requests.
To diagnose Malfunction in this scenario, Application Connectivity Monitor
observes TimedOutSymptom events for TCP checks sent to one of the ports
on HTTPService. In addition, ACM may observe SlowSymptom events for
TCP checks sent to the ports used by webserver.example.com.
Figure 4 shows the Notification Properties for the HTTPService Malfunction
problem. Application Connectivity Monitor diagnosed the Malfunction
problem after observing a TimedOutSymptom event for a TCP check sent to
one of the ports of the Web server. Because one port is still responsive,
Application Connectivity Monitor does not diagnose this as a Down
problem.
Figure 4: Notifications Properties Of Malfunction Problem
A Host Unresponsive problem indicates that a network or system problem is
affecting the system on which the managed application is running.
To diagnose Host Unresponsive in this scenario, Application Connectivity
Monitor must receive an UnresponsiveSymptom from IP Availability
Manager. Application Connectivity Monitor may also observe
TimedOutSymptom or SlowSymptom events for TCP checks sent to the ports
used by HTTPService.
Web Server Degraded
A Degraded problem indicates that the application is not performing as
expected. The application is responding to TCP checks, but not within the
expected time period.
To diagnose Degraded in this scenario, Application Connectivity Monitor
observes a SlowSymptom event for one of the monitored ports of
HTTPService.
Two Applications on a Single System
In the following scenario, two applications, HTTPService and SMTPService,
are running on the same system, host2.example.com. Application
Connectivity Monitor monitors each application with a different TCP check.
Figure 5: Two Applications Hosted By A Single System
Mail Server Down
A Down problem indicates that the application is not responding to
connectivity polls from Application Connectivity Monitor.
To diagnose Mail Server Down in this scenario, Application Connectivity
Monitor observes that the TCP checks sent to SMTPService time out, resulting
in TimedOutSymptom events.
The analysis of Mail Server Down is independent of the analysis of other
applications running on host2.example.com.
Web Server Down
A Down problem indicates that the application is not responding to
connectivity polls from Application Connectivity Monitor.
To diagnose Web Server Down in this scenario, Application Connectivity
Monitor observes that TCP checks sent to HTTPService time out, resulting in
TimedOutSymptom events.
Figure 6 shows the Notification Properties for the HTTPService Down
problem. Application Connectivity Monitor diagnoses the Down problem
because it observes the TimedOutSymptom event in response to a TCP
check.
A Degraded problem indicates that the application is not performing as
expected. The application is responding to TCP requests, but not within the
expected time period.
To diagnose Mail Server Degraded in this scenario, Application Connectivity
Monitor observes SlowSymptom events in response to TCP checks sent to
SMTPService.
The analysis of Mail Server Degraded is independent of the analysis of other
applications running on host2.example.com.
Web Server Degraded
A Degraded problem indicates that the application is not performing as
expected. The application is responding to TCP requests, but not within the
expected time period.
To diagnose Web Server Degraded in this scenario, Application
Connectivity Monitor observes SlowSymptom events in response to TCP
checks sent to HTTPService.
Host Unresponsive
A Host Unresponsive problem indicates that a network or system problem is
affecting the system on which one or more managed applications are
running.
To diagnose Host Unresponsive in this scenario, Application Connectivity
Monitor must receive an UnresponsiveSymptom from IP Availability
Manager. Application Connectivity Monitor may also observe
TimedOutSymptom and SlowSymptom events for TCP connectivity checks
sent to HTTPService and SMTPService.
Figure 7 shows an Applications Map with two applications hosted by
host2.example.com. The icon representing host2.example.com is red,
indicating that this system is experiencing a problem. The status of the
applications running on host2.example.com are affected by this problem,
causing their icons to turn yellow.
When Application Connectivity Monitor is deployed in conjunction with
Business Impact Manager, problems diagnosed by ACM for applications
and systems result in Impact notifications for ServiceOffering and
ServiceSubscriber elements.
In the following scenario, a ServiceOffering, DNSService, consists of a DNS
server, DNSServer, that is hosted by host3.example.com. The DNS service,
is subscribed to by ACMECorp, a customer of the service provider. In this
scenario, Application Connectivity Monitor might diagnose the following
problems:
•DNS Server Down
•DNS Server Degraded
•Host Unresponsive
When Application Connectivity Monitor diagnoses one of these problems,
Service Assurance Manager notifies an Impact notification for the
DNSService and ACMECorp elements. The Impact notifications are
displayed in the Notification Log, the Business Services Map, and the Impact
tab of Notification Properties dialog.
Figure 8 shows a Business Services Map for this scenario. A problem in
DNSServer propagates to the related ServiceOffering and
ServiceSubscriber elements. The severity of the problem in DNSServer is
reflected in the status of the related business elements.
This chapter describes using the Global Console to view the analysis
provided by Application Connectivity Monitor. This includes a description of
the Applications maps, which provide a graphical representation of the
application topology.
For information about the Global Console see the EMC Smarts Service Assurance Manager Operator’s Guide.
Analysis Results Displayed as Notifications
Viewing notifications listed in the Global Console is a common method of
obtaining the results of Application Connectivity Monitor’s analysis. This
section briefly describes how to access the information included with a
notification.
2
About Notifications
Application Connectivity Monitor reports the results of its analysis to Service
Assurance Manager. Service Assurance Manager combines the analysis
from Application Connectivity Monitor with the analysis from other managed
domains and displays this information to users through the Global Console.
When Application Connectivity Monitor diagnoses a problem or detects an
abnormal condition, it generates a notification. The Global Console displays
notifications in several ways:
•In the tabular format of a Notification Log view.
•In the color state of icons on a map.
Viewing the Properties of a Notification
The Notification Properties dialog box provides detailed information about
individual notifications. There are several methods to display the Notification
Properties dialog box. The most common method is to double-click on a
notification or right-click on a map icon and select Notification Properties in
the context menu.
When open, the Notification Properties dialog box displays one or more of
the following tabs:
•General lists the properties of the notification. Notifications, similar to
topology elements, are objects in the repository of the Global Manager.
As such, their properties are codified as attributes. A notification’s
attributes identify the managed element where the problem occurred,
the severity of the problem, and the underlying domain, or source, that
diagnosed the problem.
•Audit Log lists any actions performed on the notification by the system
or by a user. Actions can include acknowledging, taking ownership,
and invoking a tool.
•Codebook lists the symptomatic events used to diagnose a root cause
problem. A separate Codebook tab is displayed for each underlying
domain that diagnoses the problem as a root cause.
•Impact lists managed elements, if any, that are affected by the
notification.
•Details lists information about the notification from the underlying
domain that sent the notification. Information may include attributes of
the affected element or threshold values and their related attributes.
•Caused By lists the problems or events that are causing the selected
notification. This is the inverse of the Codebook tab.
•Aggregates lists the component events of a notification.
For more information about the Notification Properties dialog box, see the
EMC Smarts Service Assurance Manager Operator’s Guide.
In addition to viewing notifications in a Notification Log, you can also view
the results of Application Connectivity Monitor’s analysis in a map. A map
provides a graphical representation of the topology, and the map icons
change color to reflect the status of elements.
Map Terminology
The following terms are used to describe map elements.
NodeA node is the iconic representation of a
topological element. Nodes that represent
composition, such as Groups, are drawn with
a dashed line. A node with a plus sign can be
expanded by right-clicking on the icon and
selecting Expand Node. A severity icon bar
underneath a node is colored to indicate that
its state is normal, affected by one or more
abnormal conditions, or unknown.
EdgeAn edge is a line between two nodes. It can
represent a connection or relationship between
two nodes or the participation in a logical
group.
Hop The distance between two nodes that are
logically or physical connected.
Opening an Applications Map
The Applications maps are available when you attach the Global Console to
a Global Manager. Several methods exist to access an Applications map,
the most common methods are:
•Open the Map Console from the Global Console (File > New > Map
Console). In the Topology tab, click on an application-related topology
instance to display an Applications map.
You can change the type of map that is displayed by right-clicking the
instance and selecting the map type in the context menu.
•Select the Show Map option from any opened console. For example, for
a selected application-related notification, the option appears in the
Event menu or the context menu if you right-click on it. The option also
appears in a context menu if you right-click on an instance in the
Topology Browser Console.
•Right-click on a node in a map and select Applications from the
submenu.
Topology Elements Displayed in an Application Map
An Applications map displays the topological elements related to the
delivery of application services. Such elements may include the systems that
host applications, and various types of applications, as shown in Figure 9. In
the Map Console, you can select Map > Map Legend to see a list of the
icons used to represent each type of topological element.