Juniper Junos Space Network Management Platform User Manual

Junos Space Network Management Platform

Monitoring and Troubleshooting Guide

Published

Release

2021-04-09

21.1

Juniper Networks, Inc. 1133 Innovation Way Sunnyvale, California 94089 USA

408-745-2000 www.juniper.net

JuniperNetworks,theJuniperNetworkslogo,Juniper,andJunosareregisteredtrademarksofJuniperNetworks,Inc. in theUnitedStatesandothercountries. Allothertrademarks,servicemarks,registeredmarks,orregisteredservicemarks are the property of their respective owners.

Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

JunosSpaceNetworkManagementPlatformMonitoringandTroubleshootingGuide

21.1

The information in this document is current as of the date on the title page.

YEAR 2000 NOTICE

Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related limitations through the year 2038. However, the NTP application is known to have some difficulty in the year 2036.

END USER LICENSE AGREEMENT

TheJuniperNetworksproductthatisthesubjectofthistechnicaldocumentationconsistsof(orisintendedforusewith) JuniperNetworkssoftware.UseofsuchsoftwareissubjecttothetermsandconditionsoftheEndUserLicenseAgreement (“EULA”) posted at https://support.juniper.net/support/eula/. By downloading, installing or using such software, you agree to the terms and conditions of that EULA.

iii

About the Documentation | vii

Documentation and Release Notes | vii

Documentation Conventions | vii

Documentation Feedback | x

Requesting Technical Support | x

Self-Help Online Tools and Resources | xi

Creating a Service Request with JTAC | xi

1Overview

Overview | 2

Monitoring Network Devices and Troubleshooting Software Issues with Junos Space Network

Management Platform | 2

Systems of Record | 3

System Snapshot | 3

Backup and Restore | 3

Maintenance Mode | 4

Audit Logs | 4

Jobs | 4

Secure Console | 5

Looking Glass | 5

Reports | 5

Junos Space Debug Utilities | 6

Overall System Condition and Fabric Load History Overview | 6

Overall System Condition | 6

Fabric Load History | 7

Active Users History | 8

Junos Space Network Management Platform Widgets | 9

Devices | 9

Device Templates | 9

CLI Configlets | 10

Images and Scripts | 10

Reports | 10

Network Monitoring | 11

Configuration Files | 11

Jobs | 11

Role Based Access Control | 12

Audit Logs | 12

Administration | 12

2Log Files and Debug Utilities

Troubleshooting Junos Space Network Management Platform Issues by Using Log

Files | 15

System Status Log File Overview | 15

System Status Log File | 15

Customizing Status Log File Content | 16

Downloading System Log Files for a Junos Space Appliance | 16

Customizing Log Files to Download | 17

Junos Space Network Management Platform Log Files Overview | 17

Apache Web Server Log Files | 18

JBoss Application Server Log Files | 18

MySQL Database Log Files | 20

Node Management Agent Log Files | 20

Troubleshooting Log File Overview | 21

Downloading the Troubleshooting Log File in Server Mode | 22

Downloading the Troubleshooting Log File in Maintenance Mode | 25

Downloading Troubleshooting System Log Files Through the Junos Space CLI | 26

Downloading a System Log File by Using a USB Device | 26

Downloading System Log File by Using SCP | 28

Customizing Node System Status Log Checking | 31

Customizing Node Log Files to Download | 32

Troubleshooting Network Devices by Using Junos Space Debug Utilties | 33

Junos Space Debug Utilities Overview | 33

Device-Connection Debug Scripts | 34

getDeviceInfo.sh | 34

DeviceDebugInfoCollector.sh | 34

getAllDeviceInfo.sh | 34 cleanupEditChannel.sh | 35

Device Import Scripts and Java Applications | 35 cleanupDeviceImportTables.sh | 35

DB-blob-reader.jar | 35

Job Management Scripts and Java Applications | 36

SystemLoadViewer.sh | 36 getJobThreadSump.sh | 36

JobInfoCollector.jar | 36

Usr/nma/bin/collectStuckJobLogFiles.pl | 36

HornetQ Scripts | 37

HornetQInfoProvider.sh | 37

HQMessageViewer.sh | 37

Compare.py | 37

Executing Device-Connection Debug Scripts | 38

Executing the Script to Collect Device-Connection Information | 38

Executing the Script to Collect Device Debug Information | 40

Executing the Script to Unlock the Device Configuration | 45

Executing the Script to Collect Node-Connection Information | 46

Executing Device Import Detail Script and Java Application | 52

Executing the Script to Delete Data from Device Import Tables | 52

Executing the Java Application to View Device XML | 53

Executing Job Management Scripts and Java Applications | 55

Executing the Java Application to Collect Job Information | 55

Executing the Script to View the Stack Trace of a Job | 59

Executing the Script to View Job Information on Nodes | 60

Executing HornetQ Scripts | 66

Executing the HornetQ Script to View all JBoss Queues | 66

Executing the HornetQ Script to List of Messages in a JBoss Queue | 68

3Troubleshooting Junos Space Platform Issues

Troubleshooting Login–Related Issues | 72

Troubleshooting the Not Able to Log In from the Junos Space Login Page Issue | 72

Troubleshooting Device Management–Related Issues | 74

Troubleshooting Device Discovery Failure | 74

Troubleshooting Device Data Collection Issue | 75

Troubleshooting Devices Discovered Twice Using the Device Discovery Workflow | 76

Troubleshooting Network Monitoring–Related Issues | 77

Troubleshooting the Network Monitoring Page Is Not Available Issue | 77

Troubleshooting DMI Schema–Related Issues | 78

Troubleshooting the Nondisplay of the DMI Schema Tree Issue | 78

vii

About the Documentation

IN THIS SECTION

Documentation and Release Notes | vii

Documentation Conventions | vii

Documentation Feedback | x

Requesting Technical Support | x

Use this guide to know about the various features, such as Systems of Record, System Snapshot, Audit Logs, Looking Glass, and so on, that you can use to monitor and troubleshoot devices; and also, know aboutprocedurestotroubleshootissues,suchaslogin-relatedissues,devicemanagement-relatedissues, DMI schema-related issues, and so on, in Junos Space Platform.

Documentation and Release Notes

To obtain the most current version of all Juniper Networks® technical documentation, see the product documentation page on the Juniper Networks website at https://www.juniper.net/documentation/.

Iftheinformationinthelatestreleasenotesdiffersfromtheinformationinthedocumentation,followthe product Release Notes.

Juniper Networks Books publishes books by Juniper Networks engineers and subject matter experts. These books go beyond the technical documentation to explore the nuances of network architecture, deployment, and administration. The current list can be viewed at https://www.juniper.net/books.

Documentation Conventions

Table 1 on page viii defines notice icons used in this guide.

viii

Table 1: Notice Icons
Icon	Meaning	Description
	Informational note	Indicates important features or instructions.
	Caution	Indicates a situation that might result in loss of data or hardware
		damage.
	Warning	Alerts you to the risk of personal injury or death.
	Laser warning	Alerts you to the risk of personal injury from a laser.
	Tip	Indicates helpful information.
	Best practice	Alerts you to a recommended use or implementation.

Table 2 on page viii defines the text and syntax conventions used in this guide.

Table 2: Text and Syntax Conventions

Convention	Description	Examples
Bold text like this	Represents text that you type.	To enter configuration mode, type
		the configure command:
		user@host> configure
Fixed-width text like this	Represents output that appears on	user@host> show chassis alarms
	the terminal screen.	No alarms currently active
		No alarms currently active

Italictextlikethis	• Introducesoremphasizesimportant
	new terms.
	• Identifies guide names.
	• Identifies RFC and Internet draft
	titles.

•A policy term is a named structure that defines match conditions and actions.

•JunosOSCLIUserGuide

•RFC 1997, BGPCommunities Attribute

Table 2: Text and Syntax Conventions (continued)

Convention	Description
Italictextlikethis	Represents variables (options for
	which you substitute a value) in
	commands or configuration
	statements.

Examples

Configure the machine’s domain name:

[edit]

root@# set system domain-name domain-name

Text like this	Represents names of configuration
	statements, commands, files, and
	directories; configuration hierarchy
	levels; or labels on routing platform
	components.

•To configure a stub area, include the stub statement at the [edit protocols ospf area area-id] hierarchy level.

•The console port is labeled

CONSOLE.

< > (angle brackets)	Encloses optional keywords or
	variables.
\| (pipe symbol)	Indicates a choice between the
	mutually exclusive keywords or
	variablesoneithersideofthesymbol.
	The set of choices is often enclosed
	in parentheses for clarity.

stub <default-metric metric>;

broadcast | multicast

(string1 | string2 | string3)

# (pound sign)

[ ] (square brackets)

Indention and braces ( { } )

; (semicolon)

Indicatesacommentspecifiedonthe	rsvp{#RequiredfordynamicMPLS
same line as the configuration	only
statement to which it applies.
Enclosesavariableforwhichyoucan	community name members [
substitute one or more values.	community-ids ]
Identifies a level in the configuration	[edit]
hierarchy.	routing-options {
	static {
Identifies a leaf statement at a	route default {
configuration hierarchy level.	nexthop address;
	retain;
	}
	}
	}

GUI Conventions

Table 2: Text and Syntax Conventions (continued)
Convention	Description	Examples
Bold text like this	Represents graphical user interface	• IntheLogicalInterfacesbox,select
	(GUI) items you click or select.	All Interfaces.
		• To cancel the configuration, click
		Cancel.
> (bold right angle bracket)	Separates levels in a hierarchy of	Intheconfigurationeditorhierarchy,
	menu selections.	select Protocols>Ospf.

Documentation Feedback

We encourage you to provide feedback so that we can improve our documentation. You can use either of the following methods:

•Online feedback system—Click TechLibrary Feedback, on the lower right of any page on the Juniper Networks TechLibrary site, and do one of the following:

•Click the thumbs-up icon if the information on the page was helpful to you.

•Click the thumbs-down icon if the information on the page was not helpful to you or if you have suggestions for improvement, and use the pop-up form to provide feedback.

•E-mail—Sendyourcommentstotechpubs-comments@juniper.net.Includethedocumentortopicname, URL or page number, and software version (if applicable).

Requesting Technical Support

TechnicalproductsupportisavailablethroughtheJuniperNetworksTechnicalAssistanceCenter(JTAC). If you are a customer with an active Juniper Care or Partner Support Services support contract, or are

covered under warranty, and need post-sales technical support, you can access our tools and resources online or open a case with JTAC.

•JTACpolicies—ForacompleteunderstandingofourJTACproceduresandpolicies,reviewtheJTACUser Guide located at https://www.juniper.net/us/en/local/pdf/resource-guides/7100059-en.pdf.

•Productwarranties—Forproductwarrantyinformation,visithttps://www.juniper.net/support/warranty/.

•JTAC hours of operation—The JTAC centers have resources available 24 hours a day, 7 days a week, 365 days a year.

Self-Help Online Tools and Resources

Forquickandeasyproblemresolution,JuniperNetworkshasdesignedanonlineself-serviceportalcalled the Customer Support Center (CSC) that provides you with the following features:

•Find CSC offerings: https://www.juniper.net/customers/support/

•Search for known bugs: https://prsearch.juniper.net/

•Find product documentation: https://www.juniper.net/documentation/

•Find solutions and answer questions using our Knowledge Base: https://kb.juniper.net/

•Download the latest versions of software and review release notes: https://www.juniper.net/customers/csc/software/

•Search technical bulletins for relevant hardware and software notifications: https://kb.juniper.net/InfoCenter/

•Join and participate in the Juniper Networks Community Forum: https://www.juniper.net/company/communities/

•Create a service request online: https://myjuniper.juniper.net

To verify service entitlement by product serial number, use our Serial Number Entitlement (SNE) Tool: https://entitlementsearch.juniper.net/entitlementsearch/

Creating a Service Request with JTAC

You can create a service request with JTAC on the Web or by telephone.

•Visit https://myjuniper.juniper.net.

•Call 1-888-314-JTAC (1-888-314-5822 toll-free in the USA, Canada, and Mexico).

For international or direct-dial options in countries without toll-free numbers, see https://support.juniper.net/support/requesting-support/.

PART

Overview

Overview | 2

CHAPTER 1

Overview

IN THIS CHAPTER

MonitoringNetworkDevicesandTroubleshootingSoftwareIssueswithJunosSpaceNetworkManagement

Platform | 2

Overall System Condition and Fabric Load History Overview | 6

Junos Space Network Management Platform Widgets | 9

Monitoring Network Devices and Troubleshooting Software Issues with Junos Space Network Management Platform

IN THIS SECTION

Systems of Record | 3

System Snapshot | 3

Backup and Restore | 3

Maintenance Mode | 4

Audit Logs | 4

Jobs | 4

Secure Console | 5

Looking Glass | 5

Reports | 5

Junos Space Debug Utilities | 6

Use the following features of Junos Space Network Management Platform to monitor devices and troubleshoot software issues:

Systems of Record

A network managed by Junos Space Platform contains two repositories of information about the devices inthenetwork:thedevicesthemselves(eachdevicedefinesandreportsitsofficialstate)andthedatabase (which contains information that is reported by the device during device discovery). This is known as systems of record.

Thesystemsofrecordoperateinthefollowingtwomodesdependingonwheretherepositoryofinformation is stored:

•Networkasasystemofrecord(NSOR)—Bydefault,thenetworkisthesystemofrecord(NSOR). Inthis mode, when a user commits a change in the configuration of a network device, the commit operation automatically triggers a report through the system log to Junos Space Platform.

•Junos Space as a system of record (SSOR)—In this mode, when you perform any out-of-band commit operation, Junos Space Platform receives a system log message from the device, but the values in the Junos Space Platform database are not automatically changed or synchronized with the values on the device. Instead, you can choose whether or not to overwrite the device's local changes by pushing the accepted configuration to the device from the Junos Space Platform database. For more information about systems of record in Junos Space Platform, see SystemsofRecordinJunosSpaceOverview.

System Snapshot

You can use the System Snapshot feature to create a snapshot of the current state of the Junos Space system. The snapshot includes all persistent data on the hard disk including data in the database, system and application configuration files, and application and Linux executables. You can roll back the Junos Spacesystemtoapredefinedstateoranolderreleaseifthesystemreachesanunrecoverableerrorstate caused by undesirable behavior due to corruption of system files, interruption of critical processes, and soon. TheSystemSnapshotisafabric-wideoperationthatmaintainsconsistencyofdataacrossallnodes in the fabric.

Youcancreateasnapshotbeforeasignificantactionisperformed—forexample,addingordeletingaJunos

Spacenode,installingaJunosSpaceapplication,andsoon—becausetheactioncanprecipitatethesystem into an undesirable state. You can delete the snapshot after you have ascertained that the action was performed successfully. For more information about system snapshots, see CreatingaSystemSnapshot.

Backup and Restore

You use the Backup and Restore feature to back up (or schedule the backup of) and restore the data in the Junos Space database. You can set up an hourly, daily, or weekly schedule. The database backup can

bestoredonthelocalJunosSpacesystemortransferredtoaremotesystemautomaticallyusingtheSecure Copy mechanism.

You can restore the backup in any of the following circumstances:

•Junos Space data is corrupted and you need to replace the corrupted data with uncorrupted data.

•Junos Space software is corrupted and unstable after a reinstallation or an upgrade and you need to populate the Junos Space database with uncorrupted data.

For more information about backup and restore operations, see BackingUpandRestoringtheDatabase Overview.

Maintenance Mode

Maintenance mode is a mode in which you can perform database restore and debugging tasks while all nodes in the fabric are shut down and the Junos Space Network Management Platform Web proxy is running. You need to be an authorized Junos Space administrator to put the system into maintenance mode. You can put the system into maintenance mode only after you initiate a restore task by using the Backup and Restore feature.

The Junos Space system goes into maintenance mode in the following situations:

•Junos Space Network Management Platform software goes down.

•You initiate a restore operation by using the Backup and Restore feature.

•You upgrade the Junos Space Network Management Platform software.

For more information about maintenance mode, see MaintenanceModeOverview.

Audit Logs

TheAuditLogsworkspaceofJunosSpacePlatformdisplaystheloginhistoryandtasksinitiatedbyalocal or remote user. Through this workspace, you can track login history, view the list of device management tasks,viewthelistofservicesthatwereprovisionedonthedevice,andsoon. However,tasksthatarenot initiated by users, such as device-driven activities (for example, resynchronization of network elements), and changes made from the Junos Space CLI are not recorded in audit logs. Audit logs can be used by administratorstoreviewevents—forexample,toidentifywhichuseraccountsareassociatedwithanevent, to determine the chronological sequence of events (that is, what happened before and during an event), and so on. For more information about audit logs, see JunosSpaceAuditLogsOverview.

Jobs

You use the Jobs workspace of Junos Space Platform to monitor the status of jobs that are run in Junos SpacePlatformandallJunosSpaceapplicationsinstalledonJunosSpacePlatform.Youcanviewthestatus of the jobs on the Job Management page. A job is a user-initiated action that is performed on any object

that is managed by Junos Space Platform, such as a device, service, or customer. Typical jobs in Junos SpaceNetworkManagementPlatformincludediscoveringdevices,deployingservices,prestagingdevices, and performing functional and configuration audits.

Youcantriggerjobsimmediatelyorschedulejobsforalaterdateandtime.JunosSpacePlatformmaintains a history of job statuses for all scheduled jobs. When a job is scheduled from a workspace, Junos Space PlatformassignsajobIDthatservestoidentifythejobontheJobManagementpage.Formoreinformation about jobs, see JobsOverview.

Secure Console

The Secure Console feature on the Devices workspace provides a secure remote access connection to managed and unmanaged devices. Secure Console initiates an SSH session from the Junos Space user interface by using the SSH protocol. Secure Console is a terminal window embedded in Junos Space Platform that eliminates the need for a third-party SSH client to connect to devices. Secure Console provides additional security while connecting to your devices by initiating an SSH session from the Junos SpaceserverratherthanfromyourWebbrowser. YoucanaccesstheSecureConsolefeatureeitherfrom the Device Management page or the Secure Console page. For more information about Secure Console, see SecureConsoleOverview.

Looking Glass

YouusetheLookingGlassfeaturefromtheDevicesworkspacetoviewdeviceconfigurationsbyexecuting basic CLI commands from the Junos Space user interface. You can execute these commands on multiple devices and compare the configurations and runtime information of these devices. You can execute the following types of commands by using Looking Glass: show, ping, test, and traceroute. The commands that are supported and stored in the Junos Space Platform database are displayed on the Looking Glass page. Whenyoutypethefirstfewlettersofthecommand,thesuggestionlistdisplaysthecommandsthat are supported, are stored, and begin with the letters that you typed. For more information about Looking Glass, see LookingGlassOverview.

Reports

WiththeReportsworkspaceofJunosSpacePlatform,youcangeneratecustomizedreportsformanaging theresourcesinyournetwork.Youcanusethesereportstogatherdeviceinventorydetails,jobexecution details, user accounts, and audit trails. You first create a report definition to specify what information to retrievefromtheJunosSpacePlatforminventorydatabase.Youthenusethisreportdefinitiontogenerate, export,andprintthereports. JunosSpacePlatformprovidessomepredefinedcategoriestocreatereport definitions. Youcancombinemultiplecategoriestocreateareportdefinition. Bydefault,apredefinedset of attributes is included in a report definition. You can choose to add or remove the attributes according towhatinformationyouwantfromthefinalgeneratedreport. Youcangroup,sort,orfilterdatabasedon specific attributes available with the report definition. For more information about reports, see Reports Overview.

Junos Space Debug Utilities

JunosSpacedebugutilitiesareacollectionofscriptsandJavaapplicationstofetchdetailsthatcannotbe viewed on the JBoss CLI or from the Junos Space user interface. These scripts and Java applications are storedat/var/log/space-debug/debug-utilitiesandcategorizedunderdeviceConnection,jobManagement, deviceImport,andHornetQdirectories.WhenyouexecutethesescriptsorJavaapplications,youcanview detailssuchasdevice-connectionornode-connectionissues,deviceXMLsfetchedfromtheJunosSpace Platform database, and jobs triggered and nodes that execute these jobs. For more information about Junos Space debug utilities, see “Junos Space Debug Utilities Overview” on page 33.

Overall System Condition and Fabric Load History Overview

You can view the overall Junos Space system condition and fabric load from the Junos Space Network Management Platform Dashboard or the Administration statistics page.

Overall System Condition

To calculate the overall Junos Space system condition, Junos Space Platform uses a formula based on cluster health and node-function health:

•Cluster health indicates the percentage of nodes in the fabric that are currently running. For example, if only three nodes are reachable in a four-node fabric, cluster health is 75%.

•Load-balancer health indicates the percentage of nodes (enabled for load balancing) that are running the load-balancing process.

For example, if two nodes are enabled for load balancing and the load-balancing process is running on only one node, the load-balancing health is 50%.

•Databasehealthindicatesthepercentageofnodes(enabledfordatabaserequests)thatarerunningthe database process.

For example, if two nodes are enabled as the database server and the database process is running on only one node, then database health is 50%.

•Application-logic health indicates the percentage of nodes (enabled for application logic (DML and business logic) that are running the application-logic process.

Forexample,ifthreenodesareenabledforapplicationlogicandtheapplication-logicprocessisrunning on only two nodes, then application-logic health is 67%.

JunosSpacePlatformretrievesdataonthenodesandthenodefunctionsthatarerunning,andthenapplies the following formula to determine the overall Junos Space system condition: Overall System Condition =[(NumberofNodesRunning)/(NumberofNodesinFabric)]*[(NumberofNodesRunningLoad_Balancing Process)/(NumberofNodesenabledforLoadBalancing)]*[(NumberofNodesRunningDatabase-Server Process)/(NumberofNodesEnabledAsDatabaseServer)]*[(NumberofNodesRunningApplication-Logic Process) / (Number of Nodes Enabled for Application Logic)]

TheoverallJunosSpacesystemconditionisexpressedasapercentage.Ifweusethevaluesinthepreceding examplesinthisformula,thentheoverallsystemconditionwouldbecalculatedas:OverallSystemCondition = 75% * 50%* 50% * 67% = 12.5%.

A value between 0 and 30% indicates that the system health is Poor, a value between 30% and 70% indicatesthatthesystemhealthisaverage,andavaluebetween70%and100%indicatesthatthesystem health is good. The Overall System Condition chart displays the system health as shown in

Figure 1 on page 7

Figure 1: Overall System Condition Gauge

The overall system health indicates 0% (Poor) when any one of the following conditions is detected:

•No nodes in the fabric are running.

•No nodes enabled for load balancing are running the load-balancing process.

•No nodes enabled for database requests are running the database process.

•No nodes enabled for application logic are running the application-logic process.

Fabric Load History

The Fabric Load History chart, as shown in Figure 2 on page 8, displays the average CPU usage across all nodes that are running in the fabric.

Figure 2: Fabric Load History Chart

Junos Space Platform uses the following formula to determine the fabric load: Fabric Load = (Total CPU Usage for All Nodes Running) / (Number of Nodes Running)

For example, for a fabric with three nodes running and CPU usage of 80%, 30%, and 10%, respectively, the fabric load is 40%.

Active Users History

The Active Users History chart, as shown in Figure 3 on page 8, displays the number of active users in the past one minute.

Figure 3: Active Users History Chart

Junos Space Network Management Platform Widgets

IN THIS SECTION

Devices | 9

Device Templates | 9

CLI Configlets | 10

Images and Scripts | 10

Reports | 10

Network Monitoring | 11

Configuration Files | 11

Jobs | 11

Role Based Access Control | 12

Audit Logs | 12

Administration | 12

This topic presents a list of workspaces in Junos Space Network Management Platform and the widgets that they display:

Devices

The Devices workspace displays the following widgets:

•Device Count by Platform—Number of Juniper Networks devices added per device platform

•Device Status—Percentage of devices with the UP, Down, or NA connection status

•Device Count by OS—Number of devices running a particular Junos OS version

•Device Count by Synchronization State—Device discovery targets that were discovered, failed, are managed

For more information about these widgets, refer to the ViewingDeviceStatistics topic in the JunosSpace NetworkManagementPlatformWorkspacesFeatureGuide.

Device Templates

The Device Templates workspace displays the following widgets:

• Template Status—Percentage of device templates with the Enabled and Need Review statuses

•TemplateDefinitionStatus—PercentageofdevicetemplatesthatarePublishedandUnpublishedstatuses

•Template Count by Device Family—Number of device templates created per device family

For more information about these widgets, refer to the ViewingDeviceTemplateStatistics and Viewing TemplateDefinitionStatistics topics in the JunosSpaceNetworkManagementPlatformWorkspacesFeature Guide.

CLI Configlets

The CLI Configlets workspace displays the following widgets:

•CLI Configlet Count by Device Family—Number of CLI configlets created per device family

•Configuration Viewer Count by Device Family—Number of configuration views per device family

For more information about these widgets, refer to the ViewingCLIConfigletStatistics and Viewing ConfigurationViewsStatistics topics in the JunosSpaceNetworkManagementPlatformWorkspacesFeature Guide.

Images and Scripts

The Images and Scripts workspace displays the following widgets:

•Device Image Count by Platform Group—Number of device images per platform group

•Device Images Count by Version—Number of device images created per Junos OS version

•Number of Scripts by Type—Number of scripts created per script type. The script types are : Commit, Op, and Event

•Number of Jobs per Script Action—Number of jobs triggered by different script-related actions

Formoreinformationaboutthesewidgets,refertotheViewingStatisticsforDeviceImagesandScriptstopic

in the JunosSpaceNetworkManagementPlatformWorkspacesFeatureGuide.

Reports

The Reports workspace displays the following widgets:

•Report Definition Count by User—Number of report definitions created per user

•Report Count by User—Number of reports created per user

Formoreinformationaboutthesewidgets,refertotheViewingReportStatisticsandViewingReportDefinition Statistics topics in the JunosSpaceNetworkManagementPlatformWorkspacesFeatureGuide.

Network Monitoring

The Network Monitoring workspace displays the following widgets:

•Nodes with Pending Problems—Nodes with outstanding alarms

•Nodes with Outages—Nodes that reported outages

•Availability Over the Past 24 hours—Number and percentage availability of the network interfaces of the devices that reported outages

•Notification—Checkfornotificationssenttoyou,allJunosSpacePlatformusers,andtheon-callschedule to fix outages.

•Resource Graphs—Search for resource graphs. Resource graphs display data collected from managed nodes throughout your network such as critical SNMP performance, response time, and so forth.

•KSC Reports—Search for key SNMP customized (KSC) reports. KSC reports enable you to create and view SNMP performance data using prefabricated graph types.

•QuickSearch—SearchfornodesbynodeID,nodelabel,IPaddress,orthetypeofservicewhetherICMP or SNMP.

For more information about these widgets, refer to the NetworkMonitoringReportsOverview topic in the

JunosSpaceNetworkManagementPlatformWorkspacesFeatureGuide.

Configuration Files

The Configuration Files workspace displays the following widgets:

•Configuration File Count by Device Family—Number of configuration files per device family

•DeviceswithmostFrequentlyRevisedConfigurationFiles—Deviceswhoseconfigurationfileshavebeen revised most number of times

For more information about these widgets, refer to the ViewingConfigurationFileStatistics topic in the

JunosSpaceNetworkManagementPlatformWorkspacesFeatureGuide.

Jobs

The Jobs workspace displays the following widgets:

•Job Types—Percentage of all jobs of a particular type that are run

•State of Jobs Run—Percentages of jobs that succeeded, are canceled, are in progress, or failed

•Average Execution Time per Completed Job— Each bar in the Average Execution Time per Completed Job bar chart represents a job type and the average execution time in seconds.

For more information about these widgets, refer to the ViewingStatisticsforJobs topic in the JunosSpace NetworkManagementPlatformWorkspacesFeatureGuide.

Role Based Access Control

The Role Based Access Control workspace displays the following widget:

• Number of Users Assigned by Role—Percentage and the number of users that are assigned to a role For more information about these widgets, refer to ViewingUserStatistics.

Audit Logs

The Audit Logs workspace displays the following widgets:

•Audit Log Statistical Graph—Tasks that are performed and logged in all Junos Space applications over a specific period of time

•Top10ActiveUsersin24hours—Toptenuserswhoperformedthemostnumberoftasksover24hours

Formoreinformationaboutthesewidgets,refertotheViewingAuditLogStatisticstopicintheJunosSpace

NetworkManagementPlatformWorkspacesFeatureGuide.

Administration

The Administration workspace displays the following widgets:

•System Health—Junos Space system condition, load on the fabric, and active users.

•SystemAlertMessagesinthelast30days—SMTPserveralertmessagescategorizedbyapplication,and when the error last occurred.

•System Health Report—Health and performance of the Junos Space nodes in your Junos Space setup and the processes on these nodes. Staring in Release 15.2R1, the Administration workspace displays the System Health Report widget.

For more information about these widgets, refer to the ViewingtheAdministrationStatistics topic in the

JunosSpaceNetworkManagementPlatformWorkspacesFeatureGuide.

Release History Table

Release Description

15.1R2	StaringinRelease15.2R1,theAdministrationworkspacedisplaystheSystemHealth
	Report widget.

Log Files and Debug Utilities

Troubleshooting Junos Space Network Management Platform Issues by Using Log Files | 15

Troubleshooting Network Devices by Using Junos Space Debug Utilties | 33

CHAPTER 2

Troubleshooting Junos Space Network Management Platform Issues by Using Log Files

IN THIS CHAPTER

System Status Log File Overview | 15

Junos Space Network Management Platform Log Files Overview | 17

Troubleshooting Log File Overview | 21

Downloading the Troubleshooting Log File in Server Mode | 22

Downloading the Troubleshooting Log File in Maintenance Mode | 25

Downloading Troubleshooting System Log Files Through the Junos Space CLI | 26

Customizing Node System Status Log Checking | 31

Customizing Node Log Files to Download | 32

System Status Log File Overview

The system writes a system log file for each fabric node to provide troubleshooting and monitoring information. See “System Status Log File” on page 15.

The System Administrator can customize the information that is collected in the system log file. See “Customizing Node System Status Log Checking” on page 31.

TheSystemAdministratorcandownloadthelatestlogfilesforeachfabricnodewhenloggedintoaJunos Space Appliance. See “Downloading System Log Files for a Junos Space Appliance” on page 16.

Ineachoperatingmode,theSystemAdministratorcancustomizethedefaultlogfilesthataredownloaded from a Junos Space Appliance. See “Customizing Node Log Files to Download” on page 32.

System Status Log File

Approximately once a minute, the system checks and writes a status log file SystemStatusLog for each fabric node by default. Each log file consists of system status, such as the disk, CPU, and memory usage information, as shown. Junos Space Network Management Platform writes each system status log file to

/var/log/SystemStatusLog

2009-08-10 11:51:48,673 DEBUG [net.juniper.jmp.cmp.nma.NMAResponse] (Thread-110:)

Node IP: 192.0.2.0Filesystem					1K-blocks		Used Available Use% Mounted
on
/dev/mapper/VolGroup00-LogVol00
			79162184 15234764 59841252 21% /
Cpu(s): 8.7%us, 1.1%sy, 0.0%ni, 90.0%id, 0.1%wa,							0.0%hi, 0.0%si, 0.0%st
Mem:	3866536k	total,	2624680k	used,	1241856k	free,	35368k	buffers
Swap:	2031608k	total,	941312k	used,	1090296k	free,	439704k	cached

CustomizingStatusLogFileContent

TheSystemAdministratorcancustomizetheinformationthatiswritteninafabricnodesystemstatuslog file. For more information, see “Customizing Node System Status Log Checking” on page 31.

Downloading System Log Files for a Junos Space Appliance

TheSystemAdministratorcandownloadthelatestlogfilesforeachfabricnodewhenloggedintoaJunos SpaceAppliance.Thesystemstatuslogfileandallotherthird-partylogfilesarecollectedandcompressed in a troubleshooting file.

Table 3 on page 16 lists the files included in the troubleshoot file.

Table 3: Log Files included in the troubleshoot File
Description	Location
System status log files	/var/log/SystemStatusLog
JBoss log files	/var/log/jboss/*
Service-provisioning data files	/var/tmp/jboss/debug/*
MySQL error log files	/var/log/mysqld.log
Log files for Apache, Node Management Agent (NMA), and	/var/log/httpd/*
Webproxy
Watchdog log files	/var/log/watchdog/*
System messages	/var/log/messages/*

The System Administrator can download log files in each operation mode as follows:

•Server mode (See “Downloading the Troubleshooting Log File in Server Mode” on page 22.)

•Maintenancemode(See“DownloadingtheTroubleshootingLogFileinMaintenanceMode”onpage25.)

+ 63 hidden pages

Juniper Junos Space Network Management Platform User Manual

Table of Contents

About the Documentation

Documentation and Release Notes

Documentation Conventions

Documentation Feedback

Requesting Technical Support

Self-Help Online Tools and Resources

Creating a Service Request with JTAC

Overview

Overview

Monitoring Network Devices and Troubleshooting Software Issues with Junos Space Network Management Platform

Systems of Record

System Snapshot

Backup and Restore

Maintenance Mode

Audit Logs

Jobs

Secure Console

Looking Glass

Reports

Junos Space Debug Utilities

Overall System Condition and Fabric Load History Overview

Overall System Condition

Fabric Load History

Active Users History

Junos Space Network Management Platform Widgets

Devices

Device Templates

CLI Configlets

Images and Scripts

Reports

Network Monitoring

Configuration Files

Jobs

Role Based Access Control

Audit Logs

Administration

Log Files and Debug Utilities

Troubleshooting Junos Space Network Management Platform Issues by Using Log Files

System Status Log File Overview

System Status Log File

Downloading System Log Files for a Junos Space Appliance