1
Avaya Proactive Outreach Manager Event
SDK/Kafka High Availability
Release 3.1.3
Issue 1.0
January 2020
2
©
2019-2020, Avaya Inc. All
Rights Reserved.
Notice
While reasonable efforts have been made to ensure
that the information in this document is complete and
accurate at the time of printing, Avaya assumes no
liability for any errors. Avaya reserves the right to
make changes and corrections to the information in
this document without the obligation to notify any
person or organization of such changes.
Documentation disclaimer
“Documentation” means information published in
varying mediums which may include product
information, operating instructions and performance
specifications that are generally made available to users
of products. Documentation does not include marketing
materials.
Avaya shall not be responsible for any modifications,
additions, or deletions to the original published version
of Documentation unless such modifications, additions,
or deletions were performed by or on the express behalf
of Avaya. End User agrees to indemnify and hold
harmless Avaya, Avaya's agents, servants and
employees against all claims, lawsuits, demands and
judgments arising out of, or in connection with,
subsequent modifications, additions or deletions to this
documentation, to the extent made by End User.
Link disclaimer
Avaya is not responsible for the contents or reliability of
any linked websites referenced within this site or
Documentation provided by Avaya. Avaya is not
responsible for the accuracy of any information,
statement or content provided on these sites and does
not necessarily endorse the products, services, or
information described or offered within them. Avaya
does not guarantee that these links will work all the time
and has no control over the availability of the linked
pages.
Warranty
Avaya provides a limited warranty on Avaya hardware
and software. Refer to your sales agreement to establish
the terms of the limited warranty. In addition, Avaya’s
standard warranty language, as well as information
regarding support for this product while under warranty
is available to Avaya customers and other parties
through the Avaya Support website:
https://support.avaya.com/helpcenter/
getGenericDetails?detailId=C20091120112456651010
under the link “Warranty & Product Lifecycle” or such
successor site as designated by Avaya. Please note that
if You acquired the product(s) from an authorized Avaya
Channel Partner outside of the United States and
Canada, the warranty is provided to You by said Avaya
Channel Partner and not by Avaya.
“Hosted Service” means an Avaya hosted service
subscription that You acquire from either Avaya or an
authorized Avaya Channel Partner (as applicable) and
which is described further in Hosted SAS or other
service description documentation regarding the
applicable hosted service. If You purchase a Hosted
Service subscription, the foregoing limited warranty
may not apply but You may be entitled to support
services in connection with the Hosted Service as
described further in your service description documents
for the applicable Hosted Service. Contact Avaya or
Avaya Channel Partner (as applicable) for more
information.
Hosted Service
THE FOLLOWING APPLIES ONLY IF YOU
PURCHASE AN AVAYA HOSTED SERVICE
SUBSCRIPTION FROM AVAYA OR AN AVAYA
CHANNEL PARTNER (AS APPLICABLE), THE TERMS
OF USE FOR HOSTED SERVICES ARE AVAILABLE
ON THE AVAYA WEBSITE,
HTTPS://SUPPORT.AVAYA.COM/LICENSEINFO
UNDER
THE LINK “Avaya Terms of Use for Hosted Services”
OR SUCH SUCCESSOR SITE AS DESIGNATED BY
AVAYA, AND ARE APPLICABLE TO ANYONE WHO
ACCESSES OR USES THE HOSTED SERVICE. BY
ACCESSING OR USING THE HOSTED SERVICE, OR
AUTHORIZING OTHERS TO DO SO, YOU, ON
BEHALF OF YOURSELF AND THE ENTITY FOR
WHOM YOU ARE DOING SO (HEREINAFTER
REFERRED TO INTERCHANGEABLY AS “YOU” AND
“END USER”), AGREE TO THE TERMS OF USE. IF
YOU ARE ACCEPTING THE TERMS OF USE ON
BEHALF A COMPANY OR OTHER LEGAL ENTITY,
YOU REPRESENT THAT YOU HAVE THE
AUTHORITY TO BIND SUCH ENTITY TO THESE
TERMS OF USE. IF YOU DO NOT HAVE SUCH
AUTHORITY, OR IF YOU DO NOT WISH TO ACCEPT
THESE TERMS OF USE, YOU MUST NOT ACCESS
OR USE THE HOSTED SERVICE OR AUTHORIZE
ANYONE TO ACCESS OR USE THE HOSTED
SERVICE.
License types
Designated System(s) License (DS). End User may
install and use each copy or an Instance of the
Software only on a number of Designated
Processors up to the number indicated in the order.
Avaya may require the Designated Processor(s) to
be identified in the order by type, serial number,
feature key, Instance, location or other specific
designation, or to be provided by End User to Avaya
through electronic means established by Avaya
specifically for this purpose.
Concurrent User License (CU). End User may install
and use the Software on multiple Designated
Processors or one or more Servers, so long as only
the licensed number of Units are accessing and using
the Software at any given time. A “Unit” means the unit
on which Avaya, at its sole discretion, bases the
pricing of its licenses and can be, without limitation, an
agent, port or user, an e-mail or voice mail account in
the name of a person or corporate function (e.g.,
webmaster or helpdesk), or a directory entry in the
administrative database utilized by the Software that
permits one user to interface with the Software. Units
may be linked to a specific, identified Server or an
Instance of the Software.
Licenses
THE SOFTWARE LICENSE TERMS AVAILABLE ON
THE AVAYA WEBSITE,
HTTPS://SUPPORT.AVAYA.COM/LICENSEINFO,
UNDER THE LINK “AVAYA SOFTWARE LICENSE
TERMS (Avaya Products)” OR SUCH SUCCESSOR
SITE AS DESIGNATED BY AVAYA, ARE
APPLICABLE TO ANYONE WHO DOWNLOADS,
USES AND/OR INSTALLS AVAYA SOFTWARE,
PURCHASED FROM AVAYA INC., ANY AVAYA
AFFILIATE, OR AN AVAYA CHANNEL PARTNER (AS
APPLICABLE) UNDER A COMMERCIAL
AGREEMENT WITH AVAYA OR AN AVAYA CHANNEL
PARTNER. UNLESS OTHERWISE AGREED TO BY
AVAYA IN WRITING, AVAYA DOES NOT EXTEND
THIS LICENSE IF THE SOFTWARE WAS
3
OBTAINED FROM ANYONE OTHER THAN AVAYA,
AN AVAYA AFFILIATE OR AN AVAYA CHANNEL
PARTNER; AVAYA RESERVES THE RIGHT TO
TAKE LEGAL ACTION AGAINST YOU AND ANYONE
ELSE USING OR SELLING THE SOFTWARE
WITHOUT A LICENSE. BY INSTALLING,
DOWNLOADING OR USING THE SOFTWARE, OR
AUTHORIZING OTHERS TO DO SO, YOU, ON
BEHALF OF YOURSELF AND THE ENTITY FOR
WHOM YOU ARE INSTALLING, DOWNLOADING
OR USING THE SOFTWARE (HEREINAFTER
REFERRED TO INTERCHANGEABLY AS “YOU”
AND “END USER”), AGREE TO THESE TERMS AND
CONDITIONS AND CREATE A BINDING CONTRACT
BETWEEN YOU AND AVAYA INC. OR THE
APPLICABLE AVAYA AFFILIATE (“AVAYA”).
Avaya grants You a license within the scope of the
license types described below, with the exception of
Heritage Nortel Software, for which the scope of the
license is detailed below. Where the order
documentation does not expressly identify a license
type, the applicable license will be a Designated
System License as set forth below in the Designated
System(s) License (DS) section as applicable. The
applicable number of licenses and units of capacity
for which the license is granted will be one (1), unless
a different number of licenses or units of capacity is
specified in the documentation or other materials
available to You. “Software” means computer
programs in object code, provided by Avaya or an
Avaya Channel Partner, whether as stand-alone
products, pre-installed on hardware products, and any
upgrades, updates, patches, bug fixes, or modified
versions thereto. “Designated Processor” means a
single stand-alone computing device. “Server” means
a set of Designated Processors that hosts (physically
or virtually) a software application to be accessed by
multiple users. “Instance” means a single copy of the
Software executing at a particular time: (i) on one
physical machine; or (ii) on one deployed software
virtual machine (“VM”) or similar deployment.
Heritage Nortel Software
“Heritage Nortel Software” means the software that
was acquired by Avaya as part of its purchase of the
Nortel Enterprise Solutions Business in December
2009. The Heritage Nortel Software is the software
contained within the list of Heritage Nortel Products
located at https://support.avaya.com/LicenseInfo
under the link “Heritage Nortel Products” or such
successor site as designated by Avaya. For Heritage
Nortel Software, Avaya grants Customer a license to
use Heritage Nortel Software provided hereunder
solely to the extent of the authorized activation or
authorized usage level, solely for the purpose
specified in the Documentation, and solely as
embedded in, for execution on, or for communication
with Avaya equipment.
Charges for Heritage Nortel Software may be
based on extent of activation or use authorized as
specified in an order or invoice.
Copyright
Except where expressly stated otherwise, no use
should be made of materials on this site, the
Documentation, Software, Hosted Service, or hardware
provided by Avaya. All content on this site, the
documentation, Hosted Service, and the product
provided by Avaya including the selection, arrangement
and design of the content is owned either by Avaya or
its licensors and is protected by copyright and other
intellectual property laws including the sui generis
rights relating to the protection of databases. You may
not modify, copy, reproduce, republish, upload, post,
transmit or distribute in any way any content, in whole
or in part, including any code and software unless
expressly authorized by Avaya. Unauthorized
reproduction, transmission, dissemination, storage, and
or use without the express written consent of Avaya
can be a criminal, as well as a civil offense under the
applicable law.
Virtualization
The following applies if the product is deployed on a
virtual machine. Each product has its own ordering
code and license types. Unless otherwise stated, each
Instance of a product must be separately licensed and
ordered. For example, if the end user customer or
Avaya Channel Partner would like to install two
Instances of the same type of products, then two
products of that type must be ordered.
Third Party Components
“Third Party Components” mean certain software
programs or portions thereof included in the Software or
Hosted Service may contain software (including open
source software) distributed under third party
agreements (“Third Party Components”), which contain
terms regarding the rights to use certain portions of the
Software (“Third Party Terms”). As required, information
regarding distributed Linux OS source code (for those
products that have distributed Linux OS source code)
and identifying the copyright holders of the Third Party
Components and the Third Party Terms that apply is
available in the products, Documentation or on Avaya’s
website at: https:// support.avaya.com/Copyright or
such successor site as designated by Avaya. The open
source software license terms provided as Third Party
Terms are consistent with the license rights granted in
these Software License Terms, and may contain
additional rights benefiting You, such as modification
and distribution of the open source software. The Third
Party Terms shall take precedence over these Software
License Terms, solely with respect to the applicable
Third Party Components to the extent that these
Software License Terms impose greater restrictions on
You than the applicable Third Party Terms.
The following applies only if the H.264 (AVC) codec is
distributed with the product. THIS PRODUCT IS
LICENSED UNDER THE AVC PATENT PORTFOLIO
LICENSE FOR THE PERSONAL USE OF A
CONSUMER OR OTHER USES IN WHICH IT DOES
NOT RECEIVE REMUNERATION TO (i) ENCODE
VIDEO IN COMPLIANCE WITH THE AVC STANDARD
(“AVC VIDEO”) AND/OR (ii) DECODE AVC VIDEO
THAT WAS ENCODED BY A CONSUMER ENGAGED
IN A PERSONAL ACTIVITY AND/OR WAS OBTAINED
FROM A VIDEO PROVIDER LICENSED TO PROVIDE
AVC VIDEO. NO LICENSE IS GRANTED OR SHALL
BE IMPLIED FOR ANY OTHER USE. ADDITIONAL
INFORMATION MAY BE OBTAINED FROM MPEG LA,
L.L.C. SEE HTTP://WWW.MPEGLA.COM.
Service Provider
THE FOLLOWING APPLIES TO AVAYA CHANNEL
PARTNER’S HOSTING OF AVAYA PRODUCTS OR
SERVICES. THE PRODUCT OR HOSTED SERVICE
MAY USE THIRD PARTY COMPONENTS SUBJECT
TO THIRD PARTY TERMS AND REQUIRE A SERVICE
PROVIDER TO BE INDEPENDENTLY LICENSED
DIRECTLY FROM THE THIRD PARTY SUPPLIER. AN
AVAYA CHANNEL PARTNER’S HOSTING OF AVAYA
PRODUCTS MUST BE AUTHORIZED IN WRITING BY
AVAYA AND IF THOSE HOSTED PRODUCTS USE OR
4
EMBED CERTAIN THIRD PARTY SOFTWARE,
INCLUDING BUT NOT LIMITED TO MICROSOFT
SOFTWARE OR CODECS, THE AVAYA CHANNEL
PARTNER IS REQUIRED TO INDEPENDENTLY
OBTAIN ANY APPLICABLE LICENSE AGREEMENTS,
AT THE AVAYA CHANNEL PARTNER’S EXPENSE,
DIRECTLY FROM THE APPLICABLE THIRD PARTY
SUPPLIER.
WITH RESPECT TO CODECS, IF THE AVAYA
CHANNEL PARTNER IS HOSTING ANY
PRODUCTS THAT USE OR EMBED THE H.264
CODEC OR H.265 CODEC, THE AVAYA CHANNEL
PARTNER ACKNOWLEDGES AND AGREES THE
AVAYA CHANNEL PARTNER IS RESPONSIBLE
FOR ANY AND ALL RELATED FEES AND/OR
ROYALTIES. THE H.264 (AVC) CODEC IS
LICENSED UNDER THE AVC PATENT PORTFOLIO
LICENSE FOR THE PERSONAL USE OF A
CONSUMER OR OTHER USES IN WHICH IT DOES
NOT RECEIVE REMUNERATION TO: (I) ENCODE
VIDEO IN COMPLIANCE WITH THE AVC
STANDARD (“AVC VIDEO”) AND/OR (II) DECODE
AVC VIDEO THAT WAS ENCODED BY A
CONSUMER ENGAGED IN A PERSONAL ACTIVITY
AND/OR WAS OBTAINED FROM A VIDEO
PROVIDER LICENSED TO PROVIDE AVC VIDEO.
NO LICENSE IS GRANTED OR SHALL BE IMPLIED
FOR ANY OTHER USE. ADDITIONAL
INFORMATION FOR H.264 (AVC) AND H.265
(HEVC) CODECS MAY BE OBTAINED FROM MPEG
LA, L.L.C. SEE HTTP:// WWW.MPEGLA.COM.
Compliance with Laws
You acknowledge and agree that it is Your
responsibility for complying with any applicable laws
and regulations, including, but not limited to laws and
regulations related to call recording, data privacy,
intellectual property, trade secret, fraud, and music
performance rights, in the country or territory where
the Avaya product is used.
Preventing Toll Fraud
“Toll Fraud” is the unauthorized use of your
telecommunications system by an unauthorized party
(for example, a person who is not a corporate
employee, agent, subcontractor, or is not working on
your company's behalf). Be aware that there can be a
risk of Toll Fraud associated with your system and
that, if Toll Fraud occurs, it can result in substantial
additional charges for your telecommunications
services.
Avaya Toll Fraud intervention
If You suspect that You are being victimized by Toll
Fraud and You need technical assistance or support,
call Technical Service Center Toll Fraud Intervention
Hotline at +1-800-643-2353 for the United States
and Canada. For additional support telephone
numbers, see the Avaya Support website:
https://support.avaya.com or such successor site as
designated by Avaya.
Security Vulnerabilities
Information about Avaya’s security support policies
can be found in the Security Policies and Support
section of https:// support.avaya.com/security.
Suspected Avaya product security vulnerabilities are
handled per the Avaya Product Security Support Flow
(https://
support.avaya.com/css/P8/documents/100161515).
Downloading Documentation
For the most current versions of Documentation,
see the Avaya Support website:
https://support.avaya.com, or such successor site as
designated by Avaya.
Contact Avaya Support
See the Avaya Support website:
https://support.avaya.com for product or Hosted
Service notices and articles, or to report a problem
with your Avaya product or Hosted Service. For a list
of support telephone numbers and contact addresses,
go to the Avaya Support website:
https://support.avaya.com (or such successor site as
designated by Avaya), scroll to the bottom of the page,
and select Contact Avaya Support.
Trademarks
The trademarks, logos and service marks (“Marks”)
displayed in this site, the Documentation, Hosted
Service(s), and product(s) provided by Avaya are the
registered or unregistered Marks of Avaya, its
affiliates, its licensors, its suppliers, or other third
parties. Users are not permitted to use such Marks
without prior written consent from
Avaya or such third party which may own the Mark.
Nothing contained in this site, the Documentation,
Hosted Service(s) and product(s) should be construed
as granting, by implication, estoppel, or otherwise, any
license or right in and to the Marks without the express
written permission of Avaya or the applicable third party.
Avaya is a registered trademark of Avaya Inc.
All non-Avaya trademarks are the property of their
respective owners. Linux® is the registered trademark of
Linus Torvalds in the U.S. and other countries.
Java is a registered trademark of Oracle and/or its
affiliates.
5
Table of Contents
Preface............................................................................................................................................................ 6
Purpose ...................................................................................................................................................... 6
Audience .................................................................................................................................................... 6
Related documents .................................................................................................................................... 6
Working of Apache Kafka ........................................................................................................................... 6
Kafka HA ......................................................................................................................................................... 7
Zookeeper Ensemble ................................................................................................................................. 7
Kafka Broker .............................................................................................................................................. 8
Partitioning ............................................................................................................................................ 8
Producer Changes ..................................................................................................................................... 8
Producer configuration properties ......................................................................................................... 8
Enriched Attempt Event Aggregator ........................................................................................................... 8
Enable Kafka HA Configuration ...................................................................................................................... 9
Kafka and Zookeeper co-residing with POM .............................................................................................. 9
Steps to Enable Event SDK-Kafka HA ....................................................................................................... 9
External Kafka and Zookeeper installation and configuration ........................................................................10
ZooKeeper Setup ......................................................................................................................................10
ZooKeeper Configuration .....................................................................................................................10
Kafka Setup ..........................................................................................................................................11
Kafka Upgrade ..........................................................................................................................................13
Verify Kafka HA Configuration .......................................................................................................................14
Kafka Events Retention .................................................................................................................................16
Geo redundancy ............................................................................................................................................17
January 2020 Avaya POM Event SDK/Kafka High Availability 6
Preface
Purpose
The purpose of this guide is to provide detailed information about Avaya Proactive
Outreach Manager 3.1.3 EventSDK Kafka server HA configuration.
This document provides detailed information about how to implement a highly available
EventSDK-Kafka for Avaya Proactive Outreach Manager (POM) system in a single data
center.
Audience
This guide is for personnel who develop client applications for Avaya Proactive Outreach
Manager. Implementation engineers, field technicians, business partners, and customers
can use this guide to understand high availability for Event SDK-Kafka server.
Related documents
• Avaya Proactive Outreach Manager Overview and Specification Release 3.1.3
• Implementing Proactive Outreach Manager Release 3.1.3
• Using Proactive Outreach Manager Release 3.1.3
• Avaya Proactive Outreach Manager 3.1.3 Event SDK
Working of Apache Kafka
Apache Kafka® is a distributed messaging platform which allows users to subscribe to it
and publish data to any number of systems or real-time applications. It provides a unified,
high-throughput, low-latency platform for handling real-time data feeds.
Refer https://kafka.apache.org/10/documentation.html for further details.
POM creates one topic per event type per organization on Kafka server, for example
Default organization topic names would be:
• Default_POM_JOB
• Default_POM_JOB_STATISTICS
• Default_POM_AGENT
• Default_POM_AGENT_STATISTICS
• Default_POM_ENRICHED_ATTEMPT_RESULT
• Default_POM_ATTEMPT
January 2020 Avaya POM Event SDK/Kafka High Availability 7
Kafka HA
Zookeeper Ensemble
In case of POM HA deployment, Kafka HA is also supported.
Apache Kafka uses ZooKeeper to store cluster metadata.
For a ZooKeeper service to be active, it’s contract quorum must be fulfilled. The quorum
is the minimum number of zookeepers that must be up and running at a given time. As
per the contract, to form the quorum there must be a majority of non-failing machines
that can communicate with each other. To create a deployment that can tolerate the
failure of F machines, you should count on deploying 2xF+1 machine.
For example, if one zookeeper died, another zookeeper will jump in. This behavior also
applies to Kafka brokers, in this case the system is fault tolerant.
Thus, a deployment that consists of three machines can handle one failure, and a
deployment of five machines can handle two failures.
Note that a deployment of six machines can only handle two failures since three
machines is not a majority. For this reason, ZooKeeper deployments are usually made
up of an odd number of machines.
Diagram shows how ZooKeeper and Kafka deployment architecture on three POM
instances (without external Kafka-Zookeeper).
January 2020 Avaya POM Event SDK/Kafka High Availability 8
Kafka Broker
Partitioning
POM supports one partition per topic.
Replicas
The list of nodes that replicate the log for this partition regardless of whether they are the
leader or even if they are currently alive.
The minimum replication factor recommended for each topic is 3 to support HA. Hence,
we require a Kafka cluster with three nodes as depicted in the above diagram.
The replication factor must be less than or equal to the total broker count.
ISR
Set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and
caught-up to the leader.
Producer Changes
No impact if Primary producer going down and dormant becoming master.
The producer reads configuration properties from table pim_kafka_producer_config in
POM database.
Producer configuration properties
retries 2
value.serializer org.apache.kafka.common.serialization.StringSerializer
request.timeout.ms 120000
acks all
max.block.ms 50000
retry.backoff.ms 10000
key.serializer org.apache.kafka.common.serialization.StringSerializer
Enriched Attempt Event Aggregator
For Stream Processor or aggregator, it would be started on all configured POM machines
along with campaign director. And all will point to the leader.
However, due to same groupID and single partition, only one stream processor will
process the events.
January 2020 Avaya POM Event SDK/Kafka High Availability 9
Enable Kafka HA Configuration
Kafka and Zookeeper co-residing with POM
The Zookeeper and Kafka gets installed on each POM server as part of POM installation
under directory $POM_HOME/kafka_2.12-2.2.0
The post-install script enableKafkaHA.sh at $POM_HOME/bin location must be executed
to update Kafka and Zookeeper config files in case of Co-residing setup. The primary
POM server will sync configuration required for HA to auxiliary POM servers.
In case event data exists on the system, we recommend referring the Zookeeper/Kafka
documentation for detailed steps on backup of the Kafka config and event data. Follow
steps to backup the existing event data.
Steps to Enable Event SDK-Kafka HA
Follow these instructions to enable Event SDK-Kafka High Availability in Proactive
Outreach Manager.
1. Log in to a primary POM server using the command prompt. Navigate to
$POM_HOME/bin directory
2. Run the script ./enableKafkaHA.sh
System prompts you with the message:
Please enter number of brokers to handle HA[Recommended is
an odd number]:
3. Type 3 or above and press Enter.
POM prompts you with the message:
Is there an external server to be configured? (y/n)
4. Type y in case of external Kafka server and press Enter. Note that maximum one
external server is allowed.
5. Type n and POM prompts you with the message:
This script can modify properties files of Kafka.Would you
like to continue? (y/n)
6. Type y and press Enter.
POM prompts you with the message:
Please enter IP address of POM server 1
7. Type the IP address of POM server 1.
POM prompts you with the message:
Please enter zookeeper clientPort[DefaultPort = xxxx]
8. Type the port number of the zookeeper and press Enter.
9. Repeat steps 6 and 7 for POM server 2 and POM server 3.
10. Follow the instructions displayed on console after successful execution of the script.
11. Refer to section “Verify Kafka HA Configuration” to ensure successful Kafka-HA
configuration.
January 2020 Avaya POM Event SDK/Kafka High Availability 10
External Kafka and Zookeeper installation
and configuration
In case of external Kafka and zookeeper installations, the Zookeeper Ensemble and the
Kafka cluster needs to be configured manually. The following section contains details
about installing and configuring the nodes manually.
Note that maximum one external Kafka server is supported.
Before installing Kafka, Java and Zookeeper should be installed on the system. The
Zookeeper is primarily responsible for managing Kafka cluster. Please note that the Kafka
and Zookeeper version should be in synch with the Kafka and Zookeeper installed with
POM server.
ZooKeeper Setup
• From POM Primary - copy $KAFKA_HOME to external KAFKA Server
• Before executing further steps please ensure successful execution of
./enabledKafkaHA.sh script on primary POM server.
• Add entries for all POM server in /etc/hosts file on external Kafka machine.
Similarly add external server machine entry in /etc/hosts file on all POM servers.
ZooKeeper Configuration
• Go to the config directory. Open the file zookeeper.properties
• The zooKeeper deployment should have ensemble at least with 3 nodes to handle
one failure. Hence the Zookeeper config file on external server will need entry of
zookeeper residing on each POM machine.
• The default listen port is 2181. You can change this port by changing clientPort.
• The default data directory is /tmp/data. Change this, as you will not want
ZooKeeper's data to be deleted after some random timeframe. Create a directory
with the name data in the ZooKeeper directory and change the dataDir in zoo.cfg
dataDir=/opt/kafka-store/zookeeper
clientPort=2181
tickTime=2000
initLimit=5
syncLimit=2
server.1=<IP_Primary_POM>:2888:3888
server.2=<IP_AUX_POM>:2888:3888
server.3=<IP_EXTERNAL_KAFKA>:2888:3888
The property details can be found in ZooKeeper Administrator's Guide at
https://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html
• Create myid file under directory mentioned in dataDir parameter above. The myid
file should contain unique zookeeper id and it must match with x in server.x
mentioned for external kafka entry in above file.
January 2020 Avaya POM Event SDK/Kafka High Availability 11
For example in this case server.3 is mentioned, so 3 will become zookeeper ID for
myid file as well.
• Go to the bin directory.
• Start ZooKeeper by executing the command:
./zookeeper-server-start.sh config/zookeeper.properties
Note that if you want to skip manual changes or steps mentioned above, the
zookeeper.properties can be copied from POM server location
$KAFKA_HOME/config/zookeeper.properties
Kafka Setup
• Go to the config directory. Replace server.properties file with file
$KAFKA_HOME/config/server.properties from primary POM server.
• Change log.dirs to /opt/kafka-store/kafka/ , this is optional step.
• Modify broker.id to unique number across all servers.
For example broker.id on primary is 1 and aux server is 2 then broker.id on external
machine should be any valid positive number except 1 or 2.
• Update host name in listeners and advertised.listeners to external server machine
hostname.
• Copy “$POM_HOME/bin/pomCertificateGenerate.sh” to external KAFKA server’s
root directory.
• On External KAFKA server execute:
cd /root
./pomCertificateGenerate.sh
• Verify /tmp/pim.crt & pim.key generated.
• Combine your key and certificate in a PKCS#12 (P12) bundle using command:
openssl pkcs12 -export -name pomservercert -in /tmp/pom.crt -inkey /tmp/pom.key -out
/root/certificate.p12 -password pass:changeit
• Generate keystore with p12 from before steps:
keytool -importkeystore -srckeystore certificate.p12 -srcstoretype pkcs12 -destkeystore
pomKeyStore -deststoretype
• Provide the path of pomKeyStore generated in previous step in
"ssl.keystore.location" of server.properties
• Import certificate(pim.crt) generated in step above to the pomTrustStore of Primary
server using “POM Trusted Certficates” page.
• Take the modified $POM_HOME/config/pomTrustStore of primary and put it on
external Kafka server and configure the path in "ssl.truststore.location"
broker.id=3
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka-store/kafka
January 2020 Avaya POM Event SDK/Kafka High Availability 12
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=148.147.XX.XX:2181,148.147.XX.XX:2181,148.147.XX.X
X:2181
zookeeper.connection.timeout.ms=30000
group.initial.rebalance.delay.ms=0
listeners=SSL://kafkaexternal:9093
advertised.listeners=SSL://kafkaexternal:9093
ssl.keystore.location=/opt/config/pomKeyStore
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/opt/config/pomTrustStore
ssl.truststore.password=changeit
ssl.client.auth=required
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.enabled.protocols=TLSv1.2
ssl.cipher.suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_
RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_ECDH
_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA
256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_
CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC
_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_
SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CB
C_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_
CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_
AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_W
ITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH
E_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECD
SA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RS
A_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_EMPTY_RE
NEGOTIATION_INFO_SCSV
security.inter.broker.protocol=SSL
default.replication.factor=3
• Go to the bin directory and execute the command ./kafka-server-start.sh
config/server.properties
January 2020 Avaya POM Event SDK/Kafka High Availability 13
Kafka Upgrade
Refer the Zookeeper/Kafka upgrade documentation for detailed steps on backup and
restore the kafka config and event data. Note that back and restore of event data needs
to be done as per Kafka documentation.
In summary steps needs to be performed in following sequence:
• Note that the Kafka and Zookeeper version should be in synch with the Kafka and
Zookeeper installed with POM server.
• Back up all configuration files before upgrading. This includes, for
example, /kafka, /kafka-rest, and /etc/schema-registry.
• Event Data Backup: In case of POM server, the event data is stored at
$KAFKA_HOME/kafka-store/kafka location. Take a backup of the kafka-store
directory. The location may change as per configuration.
• Upgrade the software by following instructions from
https://kafka.apache.org/10/documentation/streams/upgrade-guide
• Update the server config to match with the older one.
• Restore the event data.
• Start the zookeeper and kafka server.
January 2020 Avaya POM Event SDK/Kafka High Availability 14
Verify Kafka HA Configuration
After executing $POM_HOME/bin/enableKafkaHA.sh follow the steps below to verify
whether Kafka HA Configuration is done successfully or not.
• Please check if zookeeper and Kafka servers are up and running on all POM
systems.
On external Kafka server verify using Java Virtual Machine Process Status Tool (jps)
• If all servers are up and running, verify number of brokers (Kafka servers) joined the
cluster with below command
$KAFKA_HOME/bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Check if highlighted broker id matches with number of kafka servers & their respective
ids.
• Check if Kafka topics are created successfully.
$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 –list
• If topics are created, verify that replicas of all topic partitions are distributed over all
the brokers.
$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 –describe
January 2020 Avaya POM Event SDK/Kafka High Availability 15
• Stop the broker (Kafka server) which is Leader for one of the topic partition & check
Leader is changing for that topic partition & same broker id is not getting listed in Isr
replicas for all the partitions.
• Start the broker again and check broker id is getting listed in Isr replicas for all the
partitions.
January 2020 Avaya POM Event SDK/Kafka High Availability 16
Kafka Events Retention
By default retention duration is 7 days/168 hours and events are purged automatically
after this number of days. Based on available disk space log.retention.hours can be
changed in server.properties file at $KAFKA_HOME/config/ location.
The sample performance run with below configuration:
• No. of Producers: 4 (CM, AM, CD, Event Aggregator)
• Campaign Jobs: 20
• Number of Agents: 1000
• Contacts/Attempts: 1707069
• Execution duration : 25 hours
• Disk size of kafka-store directory: 27 GB
• Number of Consumers(c): 5 (EventSDK sample client, Event Aggregator app)
• Number of topics: 6
• Replication factor(R): 3
• Retention Period in Days (RP): 7
Total
Attempts
Expected Dialing
Attempts Per Hour
No. of Hours divided
with attempts
Total Size in
GB
Per
hour
1707069
67000
25
27
1.08
Topic
MB/hour
Default_POM_AGENT
255
Default_POM_AGENT_STATISTICS
255
Default_POM_ATTEMPT
132
Default_POM_ENRICHED_ATTEMPT_RESULT
12
Default_POM_JOB
40
Default_POM_JOB_STATISTICS
13
Zookeeper directory size
204
859
Based on this, we can calculate our cluster-wide disk size according to retention period
requirement. Note that the MB/hour may vary based on callflow and dialing parameters,
agents for the campaigns.
January 2020 Avaya POM Event SDK/Kafka High Availability 17
Geo redundancy
For Geo redundancy deployment we recommend running separate event client or
consumers for each data center. Each client will connect to primary POM server running
on that datacenter. The event client connected to kafka server of active datacenter will
keep getting the events.
Once standby datacenter is active, the event client connected to that Kafka server will
start getting the events. Note that only real-time or latest events will be available for
consumer in case of POM geo redundancy deployments.
Loading...