The information in this document is current as of the date on the title page.
ii
YEAR 2000 NOTICE
Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related
limitations through the year 2038. However, the NTP application is known to have some difficulty in the year 2036.
END USER LICENSE AGREEMENT
The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with)
Juniper Networks software. Use of such software is subject to the terms and conditions of the End User License Agreement
(“EULA”) posted at https://support.juniper.net/support/eula/. By downloading, installing or using such software, you
agree to the terms and conditions of that EULA.
Table of Contents
1
About the Documentation | v
Documentation and Release Notes | v
Documentation Conventions | v
Documentation Feedback | viii
Requesting Technical Support | viii
Self-Help Online Tools and Resources | ix
Creating a Service Request with JTAC | ix
Installing HealthBot
HealthBot Installation Overview | 11
iii
HealthBot Installation Requirements | 11
HealthBot Hardware Requirements | 12
HealthBot Software Requirements | 12
Web Browser Requirements | 13
Network Requirements | 14
Network Device Requirements | 15
Multi-Node Installation | 19
Using the Interactive Installers | 20
Installing HealthBot On Ubuntu | 22
Installing HealthBot On CentOS | 30
Using the Silent Installer | 39
Starting and Stopping HealthBot | 42
Checking HealthBot Service Status | 43
Upgrading From HealthBot 3.X to HealthBot Release 3.2 | 45
Migration from HealthBot Release 2.X to 3.X | 46
Overview | 46
Case 1: HealthBot 2.X Single-node (Docker-compose) to HealthBot 3.X Single-node
(Docker-compose) Migration | 47
Case 2: Automated Migration from 2.X to 3.X (Kubernetes) | 48
Case 3: Manual Migration from 2.X to 3.X (Kubernetes) | 49
2
Uninstalling or Reconfiguring HealthBot | 50
Uninstalling HealthBot | 50
Reconfiguring HealthBot | 52
HealthBot Command-Line Options | 53
Linux Kernel Upgrade Procedures
Ubuntu Kernel Upgrade | 57
CentOS Kernel Upgrade | 66
iv
About the Documentation
IN THIS SECTION
Documentation and Release Notes | v
Documentation Conventions | v
Documentation Feedback | viii
Requesting Technical Support | viii
Use this guide to install Contrail HealthBot on a Linux server.
v
Documentation and Release Notes
To obtain the most current version of all Juniper Networks®technical documentation, see the product
documentation page on the Juniper Networks website at https://www.juniper.net/documentation/.
If the information in the latest release notes differs from the information in the documentation, follow the
product Release Notes.
Juniper Networks Books publishes books by Juniper Networks engineers and subject matter experts.
These books go beyond the technical documentation to explore the nuances of network architecture,
deployment, and administration. The current list can be viewed at https://www.juniper.net/books.
Documentation Conventions
Table 1 on page vi defines notice icons used in this guide.
Table 1: Notice Icons
vi
DescriptionMeaningIcon
Indicates important features or instructions.Informational note
Caution
Indicates a situation that might result in loss of data or hardware
damage.
Alerts you to the risk of personal injury or death.Warning
Alerts you to the risk of personal injury from a laser.Laser warning
Indicates helpful information.Tip
Alerts you to a recommended use or implementation.Best practice
Table 2 on page vi defines the text and syntax conventions used in this guide.
Table 2: Text and Syntax Conventions
ExamplesDescriptionConvention
Fixed-width text like this
Italic text like this
Represents text that you type.Bold text like this
Represents output that appears on
the terminal screen.
Introduces or emphasizes important
•
new terms.
Identifies guide names.
•
Identifies RFC and Internet draft
•
titles.
To enter configuration mode, type
the configure command:
user@host> configure
user@host> show chassis alarms
No alarms currently active
A policy term is a named structure
•
that defines match conditions and
actions.
Junos OS CLI User Guide
•
RFC 1997, BGP Communities
•
Attribute
Table 2: Text and Syntax Conventions (continued)
vii
ExamplesDescriptionConvention
Italic text like this
Text like this
< > (angle brackets)
| (pipe symbol)
Represents variables (options for
which you substitute a value) in
commands or configuration
statements.
Represents names of configuration
statements, commands, files, and
directories; configuration hierarchy
levels; or labels on routing platform
components.
variables.
Indicates a choice between the
mutually exclusive keywords or
variables on either side of the symbol.
The set of choices is often enclosed
in parentheses for clarity.
Configure the machine’s domain
name:
[edit]
root@# set system domain-name
domain-name
To configure a stub area, include
•
the stub statement at the [edit
protocols ospf area area-id]
hierarchy level.
The console port is labeled
•
CONSOLE.
stub <default-metric metric>;Encloses optional keywords or
broadcast | multicast
(string1 | string2 | string3)
# (pound sign)
[ ] (square brackets)
Indention and braces ( { } )
; (semicolon)
GUI Conventions
Indicates a comment specified on the
same line as the configuration
statement to which it applies.
Encloses a variable for which you can
substitute one or more values.
Identifies a level in the configuration
hierarchy.
Identifies a leaf statement at a
configuration hierarchy level.
rsvp { # Required for dynamic MPLS
only
community name members [
community-ids ]
[edit]
routing-options {
static {
route default {
nexthop address;
retain;
}
}
}
Table 2: Text and Syntax Conventions (continued)
viii
ExamplesDescriptionConvention
Bold text like this
> (bold right angle bracket)
Represents graphical user interface
(GUI) items you click or select.
Separates levels in a hierarchy of
menu selections.
In the Logical Interfaces box, select
•
All Interfaces.
To cancel the configuration, click
•
Cancel.
In the configuration editor hierarchy,
select Protocols>Ospf.
Documentation Feedback
We encourage you to provide feedback so that we can improve our documentation. You can use either
of the following methods:
Online feedback system—Click TechLibrary Feedback, on the lower right of any page on the Juniper
•
Networks TechLibrary site, and do one of the following:
Click the thumbs-up icon if the information on the page was helpful to you.
•
Click the thumbs-down icon if the information on the page was not helpful to you or if you have
•
suggestions for improvement, and use the pop-up form to provide feedback.
E-mail—Send your comments to techpubs-comments@juniper.net. Include the document or topic name,
•
URL or page number, and software version (if applicable).
Requesting Technical Support
Technical product support is available through the Juniper Networks Technical Assistance Center (JTAC).
If you are a customer with an active Juniper Care or Partner Support Services support contract, or are
covered under warranty, and need post-sales technical support, you can access our tools and resources
online or open a case with JTAC.
JTAC policies—For a complete understanding of our JTAC procedures and policies, review the JTAC User
•
Guide located at https://www.juniper.net/us/en/local/pdf/resource-guides/7100059-en.pdf.
JTAC hours of operation—The JTAC centers have resources available 24 hours a day, 7 days a week,
•
365 days a year.
Self-Help Online Tools and Resources
For quick and easy problem resolution, Juniper Networks has designed an online self-service portal called
the Customer Support Center (CSC) that provides you with the following features:
Upgrading From HealthBot 3.X to HealthBot Release 3.2 | 45
Migration from HealthBot Release 2.X to 3.X | 46
Uninstalling or Reconfiguring HealthBot | 50
HealthBot Command-Line Options | 53
HealthBot Installation Overview
HealthBot is a highly automated and programmable device-level diagnostics and network analytics tool
that aggregates and correlates large volumes of time-sensitive telemetry data, providing a multidimensional
and predictive view of the network. HealthBot collects telemetry data using various methods, including
Junos Telemetry Interface, NETCONF, OpenConfig, SNMP, NetFlow, sFlow, and syslog.
This guide describes how to install HealthBot and is intended for network operators and administrators
who install, configure, and manage the network security infrastructure; it specifically explains how to:
Install HealthBot using the interactive or silent installer
•
Migrate HealthBot data from a previous (2.X) version
•
Start and stop HealthBot services, and check their status
•
Adjust HealthBot setup parameters
•
Reinstall and uninstall HealthBot
•
11
RELATED DOCUMENTATION
HealthBot Installation Requirements | 11
HealthBot Installation Requirements
IN THIS SECTION
HealthBot Hardware Requirements | 12
HealthBot Software Requirements | 12
Web Browser Requirements | 13
Network Requirements | 14
Network Device Requirements | 15
Multi-Node Installation | 19
For HealthBot to install successfully, the following hardware and software components are required on
the host machine.
HealthBot Hardware Requirements
You can install HealthBot on either a physical or a virtual machine.
Proof-of-concept (POC) system—supports up to two device groups and three devices per device group:
RAM: 20 GB
•
Disk space: 100 GB available on the /var/ partition
•
Free disk space must be at least 20% of total disk space at all times.
•
CPU cores: 8
•
Production system (minimum system requirements):
12
RAM: 32 GB
•
Disk space: 250 GB SSD available on the /var/ partition
•
Free disk space must be at least 20% of total disk space at all times.
•
Recommended minimum IOPS for the disk(s): 1000
•
CPU cores: 16
•
NOTE: HealthBot is a cloud-native application that leverages a microservices-based architecture
that allows scale-out and multinode deployment. Depending on your specific requirements and
use case, you can add more nodes to the Kubernetes cluster.
See the scaling tool at https://apps.juniper.net/hb-sizing/ for more information on server configurations
for a given use case.
HealthBot Software Requirements
HealthBot installs on Ubuntu, RedHat Enterprise Linux (RHEL), and CentOS versions of Linux.
For Ubuntu:
Ubuntu version 16.04.01 (Xenial Xerus) or 18.04.04 (Bionic Beaver)
•
For multi-node installation, the kernel version must be 4.4.19 or greater.
•
We recommend installing Ubuntu as one large disk partition.
•
If multiple partitions are used, HealthBot data is written to the /var/local/healthbot/ directory and
HealthBot log files are written to /var/lib/docker/containers.
Enter the following configuration line in the file /etc/sysctl.conf: vm.max_map_count=262144. Reboot
•
of the server may be required to make this setting take effect.
For RHEL:
RHEL version 7, Release 7.5 or later
•
The following system utilities must be installed manually if they are not already present:
An Internet connection is required for all nodes during the initial Ubuntu (.deb) or CentOS/RedHat
•
(.rpm) software extraction process. This is not a requirement for the healthbot setup portion of the
installation
One static IP address per node.
•
An SSH server must be running on all nodes.
•
All nodes must be in the same subnet
•
For a multi-node installation, a virtual (unused) IP address in the same subnet as the nodes is needed.
•
This is the address on which the Web GUI is accessed.
A common SSH user name and password is needed for all nodes. The healthbot setup command
•
(discussed later) must be run as this user.
Docker version 18.09.3 or later is required
•
The command: sysctl -w net.bridge.bridge-nf-call-iptables=1 must be run on all nodes.
•
You can add the net.bridge.bridge-nf-call-iptables=1 to the file /etc/sysctl.conf to ensure that it
persists across reboots of the server.
For non-Kubernetes based installations:
•
Static IP address for the server.
•
An SSH server must be running on the server.
•
DNS is not required.
•
The command: sysctl -w net.bridge.bridge-nf-call-iptables=1 must be run on all nodes.
•
You can add the net.bridge.bridge-nf-call-iptables=1 to the file /etc/sysctl.conf to ensure that it
persists across reboots of the server.
Open the following firewall ports, as appropriate:
•
JTI (native GPB), for telemetry collection - per your source and destination port settings
•
gRPC (OpenConfig), for telemetry collection - TCP port 32767
•
NETCONF/SSH, for telemetry collection - TCP port 830
•
SNMP, for telemetry collection - UDP port 161
•
Syslog messages - UDP port 514
•
NetFlow, for telemetry collection – UDP port of your choice.
•
Must be different for each NetFlow host.
NOTE: Default ports are listed above; adjust as needed if you use non-default ports.
15
Network Device Requirements
Junos Devices
HealthBot collects data from devices running Junos OS using multiple data collection methods, called
sensors. Each sensor type requires a certain Junos OS version, and configuration added to the devices, to
enable a connection to the HealthBot server.
Native GPB
Junos OS Version: 15.1 or later
•
Required configuration—configure a sensor profile for each relevant related rule in HealthBot:
•
##Streaming Server Profile
set services analytics streaming-server COLLECTOR-1 remote-address
<HealthBot-server-address>
set services analytics streaming-server COLLECTOR-1 remote-port 22000
##Export Profile
set services analytics export-profile EXP-PROF-1 local-address <local-router-IP>
set services analytics export-profile EXP-PROF-1 local-port 22001
set services analytics export-profile EXP-PROF-1 reporting-rate 30
set services analytics export-profile EXP-PROF-1 format gpb
set services analytics export-profile EXP-PROF-1 transport udp
##Sensor Profile
set services analytics sensor SENSOR-1 server-name COLLECTOR-1
set services analytics sensor SENSOR-1 export-name EXP-PROF-1
set services analytics sensor SENSOR-1 resource <resource> # example
/junos/system/linecard/interface/
See Configuring a Junos Telemetry Interface Sensor for more information.
NetFlow (IPFIX)
Junos OS Version: 14.1R1 or later for MX Series Routers. For complete Junos version and platform
•
support information, see:
Configuring Flow Aggregation on MX, M, vMX and T Series Routers and NFX250 to Use Version 9
•
Flow Templates
Configuring Flow Aggregation to Use IPFIX Flow Templates on MX, vMX and T Series Routers, EX
•
Series Switches and NFX250, and SRX Devices
16
Understanding Inline Active Flow Monitoring - TechLibrary
•
Required configuration—
•
Configure a NetFlow v9 or IPFIX template
•
Apply the template to enable traffic sampling
•
Associate the sampling instance with the FPC
•
Specify which traffic interface to sample
•
The following samples are for an IPFIX configuration. Lines that start with “##” are comments and are used
to point out details in the configuration.
IPFIX Template Configuration
set services flow-monitoring version-ipfix template IPv4-TEMPLATE ipv4-template
Apply IPFIX Template to Enable Traffic Sampling
set forwarding-options sampling instance IPFIX-IPv4-INSTANCE input rate 10
set forwarding-options sampling instance IPFIX-IPv4-INSTANCE family inet
output flow-server 10.102.70.200 port 2055
set forwarding-options sampling instance IPFIX-IPv4-INSTANCE family inet
output flow-server 10.102.70.200 version-ipfix template IPv4-TEMPLATE
set forwarding-options sampling instance IPFIX-IPv4-INSTANCE family inet
output inline-jflow source-address 198.51.100.1
## 10.102.70.200 = HealthBot server
## port 2055; use this value in HealthBot GUI (device group config)
## inline-jflow = Enable inline flow monitoring for traffic from the designated address
## 198.51.100.1 = traffic interface that does the exporting; use this value in HealthBot GUI (device config)
Associate Sampling Instance with the FPC
set chassis fpc 0 sampling-instance IPFIX-IPv4-INSTANCE
Specify the Traffic Interface to Sample and Direction of Sampled Traffic
set interfaces ge-0/0/0 unit 0 family inet sampling input
set interfaces ge-0/0/0 unit 0 family inet sampling output
OpenConfig
Junos OS Version: 16.1 or later
•
17
The OpenConfig sensor requires that the Junos device have the OpenConfig and network agent
•
packages installed. These packages are built into Junos OS Releases 18.2X75, 18.3, and later. For
releases between 16.1 and 18.2X75 or 18.3, you must install the packages.
To verify whether you have these packages, enter the following command:
user@host> show version | match "Junos:|openconfig|na telemetry"
Junos: 19.2R1.8
JUNOS na telemetry [19.2R1.8]
JUNOS Openconfig [19.2R1.8]
See Understanding OpenConfig and gRPC on Junos Telemetry Interface for more information.
Network agent is not supported on PPC platforms (MX104, MX80, and so on)
•
Required configuration:
•
set system services extension-service request-response grpc clear-text
iAgent (NETCONF)
Junos OS Version: 11.4 or later
•
Required configuration:
•
set system services netconf ssh
SNMP
Junos OS Version: Any release
•
Required configuration:
•
set snmp community public
Syslog
Junos OS Version: Any release
•
Required configuration:
•
set system syslog host 10.10.10.1 any any
set system syslog host 10.10.10.1 allow-duplicates
set system syslog host 10.10.10.1 structured-data
## 10.10.10.1 = HealthBot server
18
BEST PRACTICE: Structured syslog is highly recommended because it avoids text parsing by
the HealthBot server.
Cisco Device Support
HealthBot can collect telemetry data from Cisco IOS XR devices. To use these devices with HealthBot,
you must configure the grpc server and the openconfig-interfaces sensors. HealthBot does not automatically
configure these for you.
The following example shows a sensor group sg1 created for gRPC dial-in configuration with the YANG
model for interfaces. An hbot_interfaces subscription associates the sg1 sensor group with an interval of
10 seconds to stream data.
NOTE: The following reference configuration is loaded on a device running Cisco IOS XR software
version 6.3.2.
!
grpc
port 32767
!
telemetry model-driven
sensor-group sg1
sensor-path openconfig-interfaces:interfaces/interface
!
subscription hbot_interfaces_
sensor-group-id sg1 sample-interval 10000
!
!
ssh server v2
end
Multi-Node Installation
Starting with release 3.0.0 HealthBot uses Kubernetes for all HA/multi-node installations. In order to install
HealthBot on more than one server, you must install using Kubernetes. All of the needed configuration
for the clusters is performed by Kubernetes. During the setup phase of the installation process, you must
identify the Kubernetes master node and worker node(s). The following example shows the installation
command for an Ubuntu server.
$ sudo apt-get install -y /var/tmp/healthbot-3.0.0.deb
[sudo] password for user:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'healthbot' instead of '/var/tmp/healthbot-3.0.0.deb'
The following NEW packages will be installed:
healthbot
…
Unpacking healthbot (3.0.0) ...
…
Setting up healthbot (3.0.0) ...
…
RELATED DOCUMENTATION
Using the Interactive Installers | 20
Using the Silent Installer | 39
Using the Interactive Installers
IN THIS SECTION
Installing HealthBot On Ubuntu | 22
Installing HealthBot On CentOS | 30
The HealthBot software package is available for download as a Debian (.deb) file for installation on Ubuntu,
or a Red Hat Package Manager (.rpm) file for installation on CentOS and RedHat Enterprise Linux (RHEL).
Before You Begin
You will need the following details for the installation:
20
Deployment type—Single node or multinode installation. Multinode installations are useful for load
•
distribution and scaling.
SSL certificate and private key—Supply your own key, or have HealthBot create one for you.
•
Host IP address—The server IP address, for SSH and Web UI access.
•
If you are installing HealthBot into an existing Kubernetes cluster, you need information about the
•
existing Kubernetes cluster:
Docker registry name
•
The path to a kubeconfig file configured with the existing cluster details
•
A user account with administrator privileges for the kubernetes cluster
•
NOTE: The following points regarding installation on an existing Kubernetes cluster must be
considered:
HealthBot has only been qualified with the Calico container network interface (CNI). It should
•
work with other CNI plugins, but has not been tested.
If your Kubernetes cluster uses other CNI plugins, such as Flannel, you might notice long
•
playbook deployment times on some Kubernetes versions. One potential workaround, is to
disable transmit (tx) and receive (rx) checksum offloading on VXLAN interfaces. An example
of this, using Flannel: ethtool --offload flannel.1 tx off rx off.
For discussions regarding the latency issue, see:
Kubernetes Issue
•
Flannel Issue
•
LKML.ORG
•
If there are a large number of device groups in HealthBot, there is the possibility that some of
•
the Kubernetes pods might not get scheduled if the nodes are saturated by the maximum pods
per node limit. By default, this limit is 110 pods per node in most Kubernetes distributions.
21
As a workaround, you can increase the maximum pods per node setting. Refer to the
documentation from your Kubernetes distribution for details on how to modify this setting.
If your CNI plugin is configured with a static IP CIDR block for each node, make sure the
•
number of IP addresses in the block allocated to the node is at least double the size of the
maximum pods per node setting.
Additional requirements:
You must have a Juniper.net user account with permissions to download the HealthBot software package.
•
The HealthBot server must have access to the Internet during the software extraction process of
•
installation (.deb for Ubuntu or .rpm for CentOS/RedHat).
HealthBot creates a /var/local/healthbot/ directory as part of the installation process. This directory
contains all the HealthBot-related files. HealthBot also installs the minimum required version of the following
additional software packages, if they are not already present.
colorama
•
Docker Compose
•
Docker SDK for Python
•
Jinja2
•
pyaml
•
Python 3
•
pip3
•
tqdm
•
distro
•
jsonpath_rw
•
Installing HealthBot On Ubuntu
The general workflow for installing HealthBot on Ubuntu is shown in Figure 1 on page 22.
Figure 1: Installation Workflow - HealthBot on Ubuntu
22
NOTE: For multinode installations, perform the following tasks only on the primary node.
If you are logged in as root or using root privileges, remove the sudo portion of each command
below.
Part 1 - PREPARE
1. Ensure that you have at least 60GB of free disk space. We recommend 100GB of free disk space.
2. Set the environment variable, HB_EXTRA_MOUNT1, to allow HealthBot to access directories outside
3. Ensure that the Ubuntu package lists on your host system are current.
$ sudo apt-get update
4. (Optional) Install the wget package. This tool will be used later to download the HealthBot software
package. On some installations, wget is installed by default.
$ sudo apt-get install -y wget
5. Install Docker CE. The commands below will install the latest stable version on x86 machines. For full
instructions and information on other platforms, see Get Docker CE for Ubuntu.
6. (Optional) Add your user account to the Docker group. This is required for a later installation step.
If installing and running with root privileges, this step is not required.
$ sudo usermod -aG docker $USER
$ <Log out of the current session, and log back in>
Part 2 - DOWNLOAD
1. Download the Contrail HealthBot package from the Juniper Networks software download page to a
temporary directory (like /var/tmp/) on the server. Note that downloading software requires a Juniper.net
account.
HealthBot checks that prerequisite software is installed on your host device during this step. If any
required software is not found, HealthBot will prompt you before installing those missing software
packages.
For example:
root@ubuntu:/var/tmp#
apt-get install -y /var/tmp/healthbot-3.2.0-1.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'healthbot' instead of '/var/tmp/healthbot-3.2.0-1.deb'
The following NEW packages will be installed:
healthbot
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/6,599 MB of archives.
After this operation, 15.2 GB of additional disk space will be used.
Get:1 /var/tmp/healthbot-3.2.0-1.deb healthbot all 3.2.0 [6,599 MB]
Selecting previously unselected package healthbot.
(Reading database ... 121891 files and directories currently installed.)
Preparing to unpack /var/tmp/healthbot-3.2.0-1.deb ...
Unpacking healthbot (3.2.0) ...
Setting up healthbot (3.2.0) ...
25
NOTE: If you see the following error when running the sudo apt-get command, you can
ignore it:
“Can't drop privileges for downloading as file '/home/user/healthbot-3.0.0-1.deb' couldn't
be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)”