Juniper Upgrading Control Center from Version 2.34 User Manual

Upgrading Control Center from Version

Published

2021-03-09

2.34

RELEASE

3.0.0

Table of Contents

Introducon

Scenario A: Upgrade of Ubuntu 16.04 to Ubuntu 18.04

Scenario B: Fresh Ubuntu 18.04 Installaon

Troubleshoong

ii

Introducon

This document concerns upgrading of Paragon Acve Assurance Control Center from version 2.34 to a
later version.

The upgrade entails special procedures as it involves upgrading the Ubuntu OS from 16.04 to 18.04. The
document covers two scenarios:

• Upgrade of Ubuntu 16.04 (with Control Center installed) to Ubuntu 18.04.

• Fresh installaon of Ubuntu 18.04 followed by installaon of Control Center and transfer of backup
data from an old Control Center instance to the new instance.

For other upgrades, please refer to the Upgrade Guide.

1

Scenario A: Upgrade of Ubuntu 16.04 to Ubuntu

18.04

• Begin by disabling the apache2 and netrounds-callexecuter services:

sudo systemctl disable apache2 netrounds-callexecuter

• Stop all Paragon Acve Assurance services:

sudo systemctl stop "netrounds-*" apache2 openvpn@netrounds

• Take backups of Paragon Acve Assurance product data.

NOTE: This is the backup procedure described in the Lifecycle Management Guide, chapter

Backing Up Product Data, only more briey worded.

Run these commands:

# Back up the PostgreSQL database

pg_dump --help

pg_dump -h localhost -U netrounds netrounds > ncc_postgres.sql

# (Alternatively, to save in binary format:)

# pg_dump -h localhost -U netrounds -Fc netrounds > ncc_postgres.binary

# Back up OpenVPN keys

sudo tar -czf ncc_openvpn.tar.gz /var/lib/netrounds/openvpn

# Note: Be sure to store these in a safe place.

# Back up RRD files (metrics data)

# Check the file size before compressing the RRDs. Use of the tar command is

not

# recommended if the RRDs are larger than 50 GB; see note below.

du -hs /var/lib/netrounds/rrd

sudo tar -czf ncc_rrd.tar.gz /var/lib/netrounds/rrd

2

NOTE: The pg_dump command will ask for a password which can be found in /etc/
netrounds/netrounds.conf under "postgres database". The default password is "netrounds".

NOTE: For a large-scale setup (> 50 GB), making a tarball of the RRD les might take too long,

and taking a snapshot of the volume can be a beer idea. Possible soluons for doing this
include: using a le system that supports snapshots, or taking a snapshot of the virtual volume
if the server is running in a virtual environment.

• Check the integrity of the database using the supplied script netrounds_2.35_validate_db.sh.

WARNING: If this script outputs warnings, do not aempt the database migraon
procedure described "below" on page 4. Contact Juniper support by ling a cket at

hps://support.juniper.net/support/requesng-support (supplying the output from the

script) to have the problems with the database resolved before you proceed with the
upgrade.

• Take backups of the Control Center conguraon les:

• /etc/apache2/sites-available/netrounds-ssl.conf

• /etc/apache2/sites-available/netrounds.conf

• /etc/netrounds/netrounds.conf

• /etc/netrounds/probe-connect.conf

• /etc/netrounds/restol.conf

• /etc/netrounds/secret_key

• /etc/netrounds/test-agent-gateway.yaml

• /etc/openvpn/netrounds.conf

For example:

sudo cp /etc/apache2/sites-available/netrounds-ssl.conf /etc/apache2/sites-

available/netrounds-ssl.conf.old

• Upgrade Ubuntu to version 18.04. A typical upgrade procedure is as follows (adapted from hps://

wiki.ubuntu.com/BionicBeaver/ReleaseNotes):

3

• To upgrade on a server system:

• Install update-manager-core if it is not already installed.

• Make sure the Prompt line in /etc/update-manager/release-upgrades is set to 'lts' (to ensure
that the OS is upgraded to the 18.04, the next LTS version aer 16.04).

• Launch the upgrade tool with the command sudo do-release-upgrade.

• Follow the on-screen instrucons. As far as Paragon Acve Assurance is concerned, you can
keep the defaults throughout. (It may of course happen that you need to make dierent
choices for reasons unrelated to Paragon Acve Assurance.)

• Once Ubuntu has been upgraded, reboot the system. Then perform the following steps:

• Upgrade PostgreSQL.

• Update PostgreSQL database les from version 9.5 to version 10:

sudo pg_dropcluster 10 main --stop # Shut down server and completely

delete cluster# "main" version 10 (this prepares for the upgrade# in the

next command)

sudo pg_upgradecluster 9.5 main # Upgrade cluster "main" version 9.5

to latest# available version (10)

sudo pg_dropcluster 9.5 main # Completely delete cluster "main"

version 9.5

• Remove the outdated version of PostgreSQL:

sudo apt purge postgresql-9.5 postgresql-client-9.5 postgresql-contrib-9.5

• Update Paragon Acve Assurance packages.

• Compute the checksum for the tarball containing the new Control Center version and verify that
it is equal to the SHA256 checksum provided on the download page:

sha256sum paa-control-center_${CC_VERSION}.tar.gz

• Unpack the Control Center tarball:

export CC_VERSION=<enter new version here>

tar -xzf netrounds-control-center_${CC_VERSION}.tar.gz

4

• Install new Control Center packages:

sudo apt update

sudo apt install ./netrounds-control-center_${CC_VERSION}/*.deb

• Remove the obsolete packages:

NOTE: It is vital to remove these packages.

# Test Agent Lite support

sudo apt purge netrounds-agent-login

# Unsupported jsonfield package

sudo apt remove python-django-jsonfield

• Run the database migraon:

NOTE: Before doing the migraon, you must ensure that the database integrity check
described "above" on page 2 completes without error.

sudo ncc migrate

The ncc migrate command takes considerable me to execute (many minutes). It should print the
following (details omied below):

Migrating database...

Operations to perform:

<...>

Synchronizing apps without migrations:

<...>

Running migrations:

<...>

Creating cache table...

<...>

Syncing test scripts...

<Updating script ...>

5

•

(Oponal)

tar -xzf netrounds-confd_${NCC_VERSION}.tar.gz

sudo apt install ./netrounds-confd_${NCC_VERSION}\_all.deb

Update the ConfD package if you need ConfD:

• Compare the previously backed-up conguraon les with the newly installed ones, and manually
merge the contents of the two sets of les (they should remain in the same locaons).

• Enable the apache2, kaa, and netrounds-callexecuter services:

sudo systemctl enable apache2 kafka netrounds-callexecuter

• Start Paragon Acve Assurance services:

sudo systemctl start --all "netrounds-*" apache2 kafka openvpn@netrounds

• To acvate the new conguraon, you also need to run:

sudo systemctl reload apache2

• Install new Test Agent repositories:

TA_APPLIANCE_VERSION=<enter version number here>

TA_APPLICATION_VERSION=<enter version number here>

# For versions prior to 3.0:

# Verify the integrity of the repositories (response should be "OK")

shasum -c netrounds-test-agent_${TA_APPLIANCE_VERSION}_all.sha256

shasum -c netrounds-test-agent-application_$

{TA_APPLICATION_VERSION}.sha256.sum

6

# For version 3.0 and later:

# Compute checksums for the repositories and verify that they match the

# SHA256 checksums provided on the download page

sha256sum paa-test-agent_${TA_APPLIANCE_VERSION}_all.deb

sha256sum paa-test-agent-application_${TA_APPLICATION_VERSION}.tar.gz

# Start the installation

sudo apt-get install \

./netrounds-test-agent_${TA_APPLIANCE_VERSION}_all.deb

sudo cp netrounds-test-agent-application_${TA_APPLICATION_VERSION}.tar.gz \

/usr/lib/python2.7/dist-packages/netrounds/static/test_agent/

• Since support for Test Agent Lite was dropped in version 2.35, you should remove the old Test Agent
Lite packages if you have them installed:

sudo rm -rf /usr/lib/python2.7/dist-packages/netrounds/static/test_agent/

netrounds-test-agent-lite*

Scenario B: Fresh Ubuntu 18.04 Installaon

• On the Ubuntu 16.04 instance, take backups of Paragon Acve Assurance product data.

NOTE: This is the backup procedure described in the Lifecycle Management Guide, chapter
"Backing Up Product Data", only more briey worded.

Run these commands:

# Back up the PostgreSQL database

pg_dump --help

pg_dump -h localhost -U netrounds netrounds > ncc_postgres.sql

7

# (Alternatively, to save in binary format:)

# pg_dump -h localhost -U netrounds -Fc netrounds > ncc_postgres.binary

# Back up OpenVPN keys

sudo tar -czf ncc_openvpn.tar.gz /var/lib/netrounds/openvpn

# Note: Be sure to store these in a safe place.

# Back up RRD files (metrics data)

# Check the file size before compressing the RRDs. Use of the tar command is

not

# recommended if the RRDs are larger than 50 GB; see note below.

du -hs /var/lib/netrounds/rrd

sudo tar -czf ncc_rrd.tar.gz /var/lib/netrounds/rrd

NOTE: The pg_dump command will ask for a password which can be found in /etc/
netrounds/netrounds.conf under "postgres database". The default password is "netrounds".

NOTE: For a large-scale setup (> 50 GB), making a tarball of the RRD les might take too long,

and taking a snapshot of the volume can be a beer idea. Possible soluons for doing this
include: using a le system that supports snapshots, or taking a snapshot of the virtual volume
if the server is running in a virtual environment.

• On the Ubuntu 16.04 instance, take backups of the Control Center conguraon les:

• /etc/apache2/sites-available/netrounds-ssl.conf

• /etc/apache2/sites-available/netrounds.conf

• /etc/netrounds/netrounds.conf

• /etc/netrounds/probe-connect.conf

• /etc/openvpn/netrounds.conf

For example:

sudo cp /etc/apache2/sites-available/netrounds-ssl.conf /etc/apache2/sites-

available/netrounds-ssl.conf.old

• On the Ubuntu 16.04 instance, back up the license le.

8

• The new instance needs to sasfy at least the same hardware requirements as the old one.

• On the new instance, install Ubuntu 18.04. We recommend the following tutorial:

• hps://ubuntu.com/tutorials/install-ubuntu-server

As far as Paragon Acve Assurance is concerned, you can keep the defaults throughout. (It may of
course happen that you need to make dierent choices for reasons unrelated to Paragon Acve
Assurance.)

• Once Ubuntu 18.04 is installed, reboot the system.

• The following disk paroning is recommended, especially for snapshot backups (but it is up to

you as a user to decide):

• Recommended paroning for lab setup:

• /: Whole disk, ext4.

• Recommended paroning for producon setup:

• /: 10% of disk space, ext4.

• /var: 10% of disk space, ext4.

• /var/lib/netrounds/rrd: 80% of disk space, ext4.

• No encrypon

• Set the me zone to UTC, for example as follows:

sudo timedatectl set-timezone Etc/UTC

• Set all locales to en_US.UTF-8.

• One way to do this is to manually edit the le /etc/default/locale. Example:

LANG=en_US.UTF-8

LC_ALL=en_US.UTF-8

LANGUAGE=en_US.UTF-8

• Make sure the following line is NOT commented out in the /etc/locale.gen:

en_US.UTF-8 UTF-8

9

• Regenerate the locale les to make sure the selected language is available:

sudo apt-get install locales

sudo locale-gen

• Make sure that trac on the following ports are allowed to and from Control Center:

• Inbound:

• TCP port 443 (HTTPS): Web interface

• TCP port 80 (HTTP): Web interface (used by Speedtest, redirects other URLs to HTTPS)

• TCP port 830: ConfD (oponal)

• TCP port 6000: Encrypted OpenVPN connecon for Test Agent Appliances

• TCP port 6800: Encrypted WebSocket connecon for Test Agent Applicaons

• Outbound:

• TCP port 25 (SMTP): Mail delivery

• UDP port 162 (SNMP): Sending SNMP traps for alarms

• UDP port 123 (NTP): Time synchronizaon

• Install NTP:

• First disable medatectl:

sudo timedatectl set-ntp no

• Run this command:

timedatectl

and verify that

systemd-timesyncd.service active: no

• Now you can run the NTP installaon:

10

sudo apt-get install ntp

• Make sure that the congured NTP servers are reachable:

ntpq -np

The output should normally be "all ones" expressed in octal.

1

• Install PostgreSQL and set up a user for Control Center:

sudo apt-get update

sudo apt-get install postgresql

sudo -u postgres psql -c "CREATE ROLE netrounds WITH ENCRYPTED PASSWORD

'netrounds' SUPERUSER LOGIN;"

1

In the output, the "reach" value for the NTP servers is an octal value indicang the outcome of the
last eight NTP transacons. If all eight were successful, the value will be octal 377 (= binary
0b11111111). However, when you have just installed NTP, it is likely that fewer than eight NTP

transacons have occurred, so that the value will be smaller: one of 1, 3, 7, 17, 37, 77, or 177 if all
transacons were successful.

sudo -u postgres psql -c "CREATE DATABASE netrounds OWNER netrounds ENCODING

'UTF8' TEMPLATE 'template0';"

Using an external PostgreSQL server is not recommended.

• Install and congure an email server.

• Control Center will send emails to users:

• when they are invited to an account,

• when sending email alarms (i.e. if email rather than SNMP is used for this purpose), and

• when sending periodic reports.

• Run the command

sudo apt-get install postfix

11

• For a simple setup where postx can send directly to the desnaon email server, you can set

General type of mail conguraon to "Internet Site", and System mail name can usually be le as­is. Otherwise, postx needs to be congured according to the environment. For guidance, refer to
the ocial Ubuntu documentaon at hps://help.ubuntu.com/lts/serverguide/postx.html.

• Install Control Center on the Ubuntu 18.04 instance.

This procedure also installs the Paragon Acve Assurance REST API.

export CC_VERSION=<enter new version here>

# Compute the checksum for the tar file and verify that it is equal to the

SHA256

# checksum provided on the download page

sha256sum paa-control-center_${CC_VERSION}.tar.gz

# Unpack the tarball

tar -xzf netrounds-control-center_${CC_VERSION}.tar.gz

# Make sure packages are up to date

sudo apt-get update

# Start the installation

sudo apt-get install ./netrounds-control-center_${CC_VERSION}/*.deb

• Stop all Paragon Acve Assurance services:

sudo systemctl stop "netrounds-*" apache2 openvpn@netrounds

• Restore database backup:

sudo -u postgres psql --set ON_ERROR_STOP=on netrounds < ncc_postgres.sql

• Run the database migraon:

NOTE: This is a sensive command, and care should be taken when execung it on a remote
machine. In such a scenario it is strongly recommended that you use a program like screen or

tmux so that the migrate command will connue running even if the ssh session breaks.

12

sudo ncc migrate

The ncc migrate command takes considerable me to execute (many minutes). It should print the
following (details omied below):

Migrating database...

Operations to perform:

<...>

Synchronizing apps without migrations:

<...>

Running migrations:

<...>

Creating cache table...

<...>

Syncing test scripts...

<Updating script ...>

• Transfer the backup data to the 18.04 instance using scp or some other tool.

• Restore the OpenVPN keys:

# Remove any existing OpenVPN keys

sudo rm -rf /var/lib/netrounds/openvpn

# Unpack the backed-up keys

sudo tar -xzf ncc_openvpn.tar.gz -C /

• Restore RRD data:

# Remove any existing RRDs

sudo rm -rf /var/lib/netrounds/rrd

# Unpack the backed-up RRDs

sudo tar -xzf ncc_rrd.tar.gz -C /

• Compare the backed-up conguraon les with the newly installed ones, and manually merge the
contents of the two sets of les (they should remain in the same locaons).

• Acvate the product license using the license le taken from the old instance:

13

ncc license activate ncc_license.txt

• Start Paragon Acve Assurance services:

sudo systemctl start --all "netrounds-*" apache2 kafka openvpn@netrounds

• To acvate the new conguraon, you also need to run:

sudo systemctl reload apache2

• Install new Test Agent repositories:

TA_APPLIANCE_VERSION=<enter version number here>

TA_APPLICATION_VERSION=<enter version number here>

# For versions prior to 3.0:

# Verify the integrity of the repositories (response should be "OK")

shasum -c netrounds-test-agent_${TA_APPLIANCE_VERSION}_all.sha256

shasum -c netrounds-test-agent-application_$

{TA_APPLICATION_VERSION}.sha256.sum

# For version 3.0 and later:

# Compute checksums for the repositories and verify that they match the

# SHA256 checksums provided on the download page

sha256sum paa-test-agent_${TA_APPLIANCE_VERSION}_all.deb

sha256sum paa-test-agent-application_${TA_APPLICATION_VERSION}.tar.gz

# Start the installation

sudo apt-get install \

./netrounds-test-agent_${TA_APPLIANCE_VERSION}_all.deb

sudo cp netrounds-test-agent-application_${TA_APPLICATION_VERSION}.tar.gz \

/usr/lib/python2.7/dist-packages/netrounds/static/test_agent/

•

(Oponal)

Follow the NETCONF & YANG API Orchestraon Guide to install and congure ConfD if

you need it.

14

Troubleshoong

IN THIS SECTION

Problems Starng ConfD | 14

Problems Starng callexecuter | 15

Web Server Does Not Respond | 16

Restarng of Paragon Acve Assurance Services Fails | 17

Problems Starng ConfD

If you have problems starng ConfD aer the upgrade, please contact your Juniper partner or your local
Juniper account manager or sales representave in order to get a new subscripon.

Problems Starng callexecuter

Check the callexecuter logs with the command

sudo journalctl -xeu netrounds-callexecuter

You may see an error like the following:

Jun 03 09:53:27 myhost django-admin[6290]: ERROR netrounds.manager.callexecuter

Unhandled exception in CallExecuter.run [name=netrounds.manager.callexecuter,

thread=140364632504128, process=8238, funcName=handle, le

Jun 03 09:53:27 myhost django-admin[6290]: Traceback (most recent call last):

Jun 03 09:53:27 myhost django-admin[6290]: File "debian/tmp/usr/lib/python2.7/

dist-packages/netrounds/manager/management/commands/runcallexecuter.py", line

65, in handle

Jun 03 09:53:27 myhost django-admin[6290]: File "debian/tmp/usr/lib/python2.7/

dist-packages/netrounds/manager/calldispatcher.py", line 164, in run

Jun 03 09:53:27 myhost django-admin[6290]: File "debian/tmp/usr/lib/python2.7/

dist-packages/netrounds/manager/models.py", line 204, inwait

Jun 03 09:53:27 myhost django-admin[6290]: File "debian/tmp/usr/lib/python2.7/

dist-packages/netrounds/manager/models.py", line 42, in __unicode__

Jun 03 09:53:27 myhost django-admin[6290]: AttributeError: 'unicode' object has

no attribute 'iteritems'

15

What has happened is that the netrounds-callexecuter*.deb package was upgraded without making
sure the netrounds-callexecuter systemd service was stopped and disabled. The database is in the
wrong state; it needs to be restored from backup, and the upgrade needs to be repeated.

Do as follows to disable and stop the netrounds-callexecuter service:

sudo systemctl disable netrounds-callexecuter

sudo systemctl stop netrounds-callexecuter

Web Server Does Not Respond

Check the apache logs with the command

tail -n 50 /var/log/apache2/netrounds_error.log

If you see the following error, it means that Control Center version 2.34 is running on Ubuntu 18.04,
that is, Control Center has not been successfully upgraded. The soluon is to upgrade Control Center to
a later version as described in this document.

# Timestamps, pids, etc. stripped away below

Target WSGI script '/usr/lib/python2.7/dist-packages/netrounds/wsgi.py' cannot

be loaded as Python module.

Exception occurred processing WSGI script '/usr/lib/python2.7/dist-packages/

netrounds/wsgi.py'.

Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/netrounds/wsgi.py", line 6, in<module>

application = get_wsgi_application()

File "/usr/lib/python2.7/dist-packages/django/core/wsgi.py", line 13, in

get_wsgi_application

django.setup(set_prefix=False)

File "/usr/lib/python2.7/dist-packages/django/__init__.py", line 27, in setup

apps.populate(settings.INSTALLED_APPS)

File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 85, in

populate

app_config = AppConfig.create(entry)

File "/usr/lib/python2.7/dist-packages/django/apps/config.py", line 94, in

create

module = import_module(entry)

File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module

__import__(name)

File "/usr/lib/python2.7/dist-packages/grappelli/dashboard/__init__.py", line

1, in<module>

from grappelli.dashboard.dashboards import *

File "/usr/lib/python2.7/dist-packages/grappelli/dashboard/dashboards.py",

line 14, in<module>

from grappelli.dashboard import modules

File "/usr/lib/python2.7/dist-packages/grappelli/dashboard/modules.py", line

9, in<module>

from django.contrib.contenttypes.models import ContentType

16

File "/usr/lib/python2.7/dist-packages/django/contrib/contenttypes/models.py",

line 139, in<module>

class ContentType(models.Model):

File "/usr/lib/python2.7/dist-packages/django/db/models/base.py", line 110, in

__new__

app_config = apps.get_containing_app_config(module)

File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 247, in

get_containing_app_config

self.check_apps_ready()

File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 125, in

check_apps_ready

raise AppRegistryNotReady("Apps aren't loaded yet.")

AppRegistryNotReady: Apps aren't loaded yet.

Restarng of Paragon Acve Assurance Services Fails

17

Restarng the netrounds-* services with

sudo systemctl start --all "netrounds-*" apache2 openvpn@netrounds

produces the following message:

Failed to start netrounds-agent-ws-server.service: Unit netrounds-agent-ws-

server.service is masked.

Failed to start netrounds-agent-daemon.service: Unit netrounds-agent-

daemon.service is masked.

This means that the services menoned have been masked in the course of the package removal process
and require manual cleanup. The cleanup procedure is shown below:

sudo apt-get purge netrounds-agent-login

sudo find /etc/systemd/system -name "netrounds-agent-*.service" -delete

sudo systemctl daemon-reload

18

Juniper Networks, the Juniper Networks logo, Juniper, and Junos are registered trademarks of Juniper
Networks, Inc. in the United States and other countries. All other trademarks, service marks, registered
marks, or registered service marks are the property of their respecve owners. Juniper Networks assumes
no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change,
modify, transfer, or otherwise revise this publicaon without noce. Copyright © 2021 Juniper Networks,
Inc. All rights reserved.