Huawei Distributed Message Service for Kafka User Manual

Distributed Message Service for Kafka

User Guide

Issue	01
Date	2020-12-02

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise c fi in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every ff has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	Contents

Contents


1	Service Overview.....................................................................................................................			1
1.1		What is DMS for Kafka?........................................................................................................................................................		1
1.2		Product Advantages................................................................................................................................................................		1
1.3		Application Scenarios.............................................................................................................................................................		2
1.4		c fic	n ............................................................................................................................................................................	4
1.5		Comparing Kafka Instances and DMS Advanced Queues.........................................................................................		6
1.6		Comparing DMS for Kafka and Open-Source Kafka...................................................................................................		8
1.7		Notes and Constraints.........................................................................................................................................................		10
1.8		Related Services.....................................................................................................................................................................		11
1.9		Basic Concepts.......................................................................................................................................................................		11
2	Preparing Required Resources............................................................................................			14
3	Creating an Instance.............................................................................................................			16
4	Accessing a Kafka Premium Instance...............................................................................			20
4.1		Accessing a Kafka Premium Instance Without SASL................................................................................................		20
4.2		Accessing a Kafka Premium Instance with SASL.......................................................................................................		22
5	Managing Instances..............................................................................................................			26
5.1		Viewing an Instance.............................................................................................................................................................		26
5.2		Restarting an Instance........................................................................................................................................................		27
5.3		Deleting an Instance............................................................................................................................................................		28
5.4		Modifying the Information About an Instance...........................................................................................................		30
5.5		nfig	ng Public Access..................................................................................................................................................	31
5.6		Resetting Kafka Password..................................................................................................................................................		32
5.7		Viewing Background Tasks................................................................................................................................................		33
6	Managing Topics....................................................................................................................			35
6.1		Creating a Topic.....................................................................................................................................................................		35
6.2		Deleting a Topic.....................................................................................................................................................................		37
7	Auditing...................................................................................................................................			38
7.1		Operations That Can Be Recorded by CTS...................................................................................................................		38
7.2		Viewing Traces on the CTS Console................................................................................................................................		41
8	FAQs..........................................................................................................................................			42

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide							Contents
8.1 Instances..................................................................................................................................................................................								42
8.1.1 Why Can't I Select Two AZs?.........................................................................................................................................								42
8.1.2 Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?.......								42
8.1.3 How Do I Select Storage Space for a Kafka Instance?.........................................................................................								42
8.1.4 How Do I Choose Between High I/O and Ultra-high I/O?..................................................................................								43
8.1.5 Which Capacity Threshold Policy Should I Use?.....................................................................................................								43
8.1.6 Which Kafka Version Is Supported?............................................................................................................................								43
8.1.7 What Is the ZooKeeper Version of a Kafka Instance?..........................................................................................								43
8.1.8 Are Kafka Instances in Cluster Mode?.......................................................................................................................								43
8.1.9 Can I Modify the Connection Address for Accessing a Kafka Instance?........................................................								43
8.1.10 How Long Are Kafka SSL					fic	Valid for?..................................................................................................		44
8.1.11 How to Synchronize Data from One Kafka Instance to Another?.................................................................								44
8.1.12 How Do I Change the SASL_SSL Setting of a Kafka Instance?.......................................................................								44
8.1.13 Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on ff n VMs?...........................								44
8.1.14 What Cipher Suites Are Supported by Kafka?.......................................................................................................								44
8.1.15 Can I Change an Instance from Single-AZ Deployment to Multi-AZ Deployment?................................								44
8.1.16 Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I View the AZs nfig
for a Purchased Instance?.........................................................................................................................................................								44
8.1.17 Can I Change the VPC and Subnet After a Kafka Instance Is Created?.......................................................								45
8.1.18 Where Can I Find Kafka Streams Use Cases?........................................................................................................								45
8.2 Connections............................................................................................................................................................................								45
8.2.1 Troubleshooting Kafka Connection Exceptions.......................................................................................................								45
8.2.2 How Do I Select and			nfig		a Security Group?................................................................................................			47
8.2.3 Can I Access a Kafka Premium Instance Over a Public Network?...................................................................								48
8.2.4 How Many Connection Addresses Does a Kafka Instance Have by Default?..............................................								49
8.2.5 Do Kafka Instances Support Cross-Region Access?...............................................................................................								49
8.2.6 Can I Access a Kafka Instance Using DNAT?...........................................................................................................								49
8.2.7 Do Kafka Premium Instances Support Cross-Subnet Access?............................................................................								49
8.2.8 Why Do I Fail to Access Kafka Using SSL from a Go Client?.............................................................................								49
8.2.9 What If	fic	Fail to Be Loaded for SASL Connection?............................................................................						49
8.2.10 Does DMS for Kafka Support Password-Free Access?........................................................................................								49
8.2.11 Obtaining Kafka Clients................................................................................................................................................								50
8.2.12 Does DMS for Kafka Support Authentication on Clients by the Server?.....................................................								50
8.2.13 Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?........								50
8.2.14 What Are the		ff	nc	Between JKS and CRT			fic .......................................................................	50
8.2.15 Which TLS Version Does DMS for Kafka Support, 1.0, 1.1, or 1.2?...............................................................								50
8.2.16 Is There a Limit on the Number of Connections to a Kafka Instance?........................................................								50
8.2.17 How Many Connections from Each IP Address Are Allowed?.........................................................................								50
8.3 Topics and Partitions............................................................................................................................................................								50
8.3.1 Is There a Limit on the Number of Topics in a Kafka Instance?.......................................................................								51
8.3.2 Why Is Partition Quantity Limited?.............................................................................................................................								51
8.3.3 Can I Reduce the Partition Quantity?.........................................................................................................................								51
8.3.4 Why Do I Fail to Create Topics?...................................................................................................................................								51
Issue 01 (2020-12-02)				Copyright © Huawei Technologies Co., Ltd.				iii

Distributed Message Service for Kafka
User Guide		Contents
8.3.5 Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?	.................................	51
8.3.6 Why Do Deleted Topics Still Exist?..............................................................................................................................		51
8.3.7 How Do I Create a Topic?...............................................................................................................................................		51
8.3.8 Are Periods (.) Allowed in Topic Names?..................................................................................................................		52
8.3.9 What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?
............................................................................................................................................................................................................		52
8.3.10 What Can I Do If Disk Usage Is as High as 96%?...............................................................................................		52
8.3.11 Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is .............	fi	52
8.3.12 How Do I Disable Automatic Topic Creation?.......................................................................................................		52
8.3.13 Can I Delete Unnecessary Topics in a Consumer Group?.................................................................................		52
8.4 Consumer Groups.................................................................................................................................................................		52
8.4.1 Do I Need to Create Consumer Groups, Producers, and Consumers for Kafka Instances?.....................		52
8.4.2 How Do I Delete Consumer Groups?..........................................................................................................................		53
8.4.3 Do I Need to Unsubscribe from a Topic Before Deleting a Consumer Group?............................................		53
8.5 Messages..................................................................................................................................................................................		53
8.5.1 What Is the Maximum Size of a Message that Can be Created?.....................................................................		54
8.5.2 Why Do I Frequently Fail to Poll Due to Rebalancing?........................................................................................		54
8.5.3 Why Do Messages Still Exist After the Retention Period Elapses?...................................................................		54
8.5.4 Do Kafka Instances Support Delayed Message Delivery?...................................................................................		54
8.5.5 Does the Message Query Function on the Console Show the Replica Message Quantity?....................		54
A Change History......................................................................................................................		55

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

1Service Overview

1.1 What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming,

n n ffl n system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service that uses the open-source Apache Kafka. It provides Kafka premium instances with isolated computing, storage, and bandwidth resources. DMS for Kafka allows you to apply resources, c nfig topics, partitions, and replicas based on service requirements. It can be used out of the box and frees you from deployment and O&M so that you can focus on the agile development of your applications.

Readers' Guide

This documentation introduces DMS for Kafka and its ff nc	from Apache
Kafka. You will learn about the detailed information about the	c fic	n

console operations, API calling, and client access to instances of DMS for Kafka.

For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the ffi Apache Kafka website.

1.2 Product Advantages

DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.

●Rapid deployment

Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka premium instance will be automatically created and deployed.

● Service migration without m fic n

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka.

If your application services are developed based on open-source Kafka, you can easily migrate them to DMS for Kafka after specifying a few authentication c nfig n

NOTE

Kafka premium instances are compatible with Apache Kafka 2.3.0.

●Security

Operations on Kafka premium instances are recorded and can be audited. Messages can be encrypted before storage.

In addition to SASL, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.

●Data reliability

Kafka premium instances support data persistence and replication. Messages can be replicated synchronously or asynchronously between replicas.

●High availability

Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly.

Kafka premium instances can be deployed across AZs to further enhance service availability.

●Simple O&M

The public cloud provides a whole set of monitoring and alarm services,

eliminating the need for 24/7 attendance. A set of Kafka premium instance metrics are monitored and reported, including the number of partitions,

topics, and accumulated messages. You can c nfig				alarm rules and receive
SMS or email n		fic	n on how your services are running in real time.
● Flexible	c fic	n

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

1.3 Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between

ff n systems in the enterprise application, payment, telecommunications, e- commerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a n fic n email and SMS message after a user has registered with a website, providing fast responses throughout the registration process.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview
Figure 1-1 Serial registration and n fic	n

Figure 1-2 Asynchronous registration and n fic

n using message queues

ffi Control

In e-commerce systems or large-scale websites, there is a processing capability gap between upstream and downstream systems. ffic bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with lower processing capabilities. For example, online sales promotions

involve a huge amount of	ffic fl	ng into e-commerce systems. Kafka
provides a three-day b ff	by default for hundreds of millions of messages, such

as orders and other information. In this way, message consumption systems can


process the messages during ff		periods.
In addition, fl	sale ffic bursts originating from frontend systems can be
handled with Kafka, keeping the backend systems from crashing.
Figure 1-3	ffic burst handling using Kafka

Log Synchronization

In large-scale service systems, logs of ff n applications are collected for quick troubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or ffl n In addition, Kafka can collect key log information to monitor applications.

Log synchronization involves three major components: log collection clients, Kafka, and backend log processing applications.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

1.The log collection clients collect log data from a user application service and asynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has a minor impact on the service performance.

2.Kafka persists logs.

3.Log processing applications, such as Logstash, subscribe to messages in Kafka and retrieve log messages from Kafka. Then, the messages are searched for

by fi search services or delivered to big data applications such as Hadoop for storage and analysis.

Figure 1-4 Log synchronization process

NOTE

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big data analytics. They are all open-source tools.

1.4fi n

Kafka Premium Instance

Kafka premium instances are compatible with open-source Kafka 2.3.0. The instance c fic n are c fi based on bandwidth, namely, 100 MB/s, 300 MB/s, 600 MB/s, and 1200 MB/s.

NOTE

●The number of brokers varies according to the underlying resources, and the underlying resources vary from region to region. The following table lists the

n .

●In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB.

Table 1-1 TPS and maximum number of partitions supported by ff						n instance
c fic	n

Band	Brok	Underlying	I/O	TPS (High-	TPS		Maxim
width	ers	Resource	Type	Throughput	(Synchro		um
		Type		)	nous		Partitio
					Replicati		ns
					on)
100	3	c6_2 vCPUs \| 4	High I/O	100,000	60,000		300
MB/s		GB

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide						1 Service Overview

	Band	Brok	Underlying	I/O	TPS (High-	TPS	Maxim
	width	ers	Resource	Type	Throughput	(Synchro	um
			Type		)	nous	Partitio
						Replicati	ns
						on)
			c6_2 vCPUs \| 4	Ultra-	100,000	80,000	300
			GB	high I/O

	300	3	c6_4 vCPUs \| 8	High I/O	300,000	150,000	900
	MB/s		GB

			c6_4 vCPUs \| 8	Ultra-	300,000	200,000	900
			GB	high I/O

	600	4	c6_8 vCPUs \|	Ultra-	600,000	300,000	1800
	MB/s		16 GB	high I/O

	1200	8	c6_8 vCPUs \|	Ultra-	1.2 million	400,000	1800
	MB/s		16 GB	high I/O

Bandwidth Selection

The bandwidth of a Kafka instance refers to the maximum read or write bandwidth. You are advised to select a bandwidth 30% higher than what is required.

●100 MB/s

Recommended for up to 3000 client connections, 60 consumer groups, and 70 MB/s of service ffic

●300 MB/s

Recommended for up to 10,000 client connections, 300 consumer groups, and 210 MB/s of service ffic

●600 MB/s

Recommended for up to 20,000 client connections, 600 consumer groups, and 420 MB/s of service ffic

●1200 MB/s

Recommended for up to 20,000 client connections, 600 consumer groups, and 840 MB/s of service ffic

Storage Space Selection

Kafka premium instances support storage with 1 to 3 replicas. The storage space is space consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved).

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

Topic Quantity

There is no limit on the topic quantity, but there is an upper limit on the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Table 1-1) and the c fi number of partitions in each topic (see Figure 1-5).

Figure 1-5 Setting the number of partitions

For example, the maximum number of partitions for a 100 MB/s instance is 300.

If the number of partitions of each topic in the instance is 3, the number of topics is 300/3 = 100.

If the number of partitions of each topic in the instance is 1, the number of topics is 300/1 = 300.

1.5 Comparing Kafka Instances and DMS Advanced

Queues

Both Kafka premium instances and DMS advanced queues are compatible with Apache Kafka. However, they ff in the following aspects.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

Open-Source Compatibility

●DMS advanced queues: Kafka 0.10.2.1

●Kafka premium instances: Kafka 2.3.0

With each version upgrade, Apache Kafka introduces new features, improves APIs, and updates producer and consumer c nfig n fi To check whether your application features and APIs are compatible with your Kafka clients, see the upgrade notes on the ffi Apache Kafka website.

Creation

●DMS advanced queues:

An advanced queue (equivalent to a topic) is created on the DMS console. You do not need to c nfig the storage space or the bandwidth because these resources are allocated by the system.

●Kafka premium instances:

A Kafka premium instance is created on the DMS for Kafka console. Before creating a Kafka premium instance, determine the required bandwidth and storage space based on your service expectations for the next one or two years. You also need to prepare a VPC and security group for the instance.

After the instance has been created, you must create topics in the instance and c nfig the number of partitions and replicas for the topics.

Usage

●DMS advanced queues:

Advanced queues are compatible with Kafka APIs. DMS provides SDKs in Java, Python, Lua, C, and Go languages..

To use open-source Kafka clients, see "Using the Enhanced Java SDK" in Developer Guide. Add the enhanced Kafka Java SDK provided by DMS for Kafka to the directory of the open-source client package and then pass the security authentication.

●Kafka premium instances:

DMS for Kafka is fully compatible with open-source Kafka. You can access Kafka premium instances and topics using open-source Kafka clients. If SASL access is enabled, you must use the SSL c fic provided by DMS for Kafka.

Performance

●DMS advanced queues:

There are two queue modes: high-throughput and high-reliability. In the highthroughput mode, messages are fl to disk asynchronously, ensuring high concurrency.

●Kafka premium instances:

Compute, bandwidth, and storage resources are physically isolated for each instance. Determine the required bandwidth and storage space when creating an instance. For storage space, you can choose Ultra-high I/O, which indicates that messages are stored on SSDs.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

Other Dimensions

You can customize the number of partitions and replicas for a Kafka premium instance. Each topic can have 1 to 50 partitions and 1 to 3 replicas.

By default, an advanced queue has three partitions and three replicas.

NOTE

Divide a topic into a certain number of partitions so that messages can be evenly distributed to partitions, enabling load balancing and horizontal scalability. ff n consumers can retrieve messages from one or more partitions, improving message processing performance.

With more replicas come higher reliability. However, synchronizing messages between replicas consumes bandwidth and ff compute performance.

1.6 Comparing DMS for Kafka and Open-Source Kafka

DMS for Kafka is compatible with open-source Kafka and has customized and enhanced Kafka features. In addition to the advantages of open-source Kafka, DMS for Kafka provides more reliable and useful features.

Table 1-2	ff nc	between DMS for Kafka and open-source Kafka

Catego	Item	DMS for Kafka				Open-source Kafka
ry
Ease of	Readily	Instances can be created				Preparing server resources and
use	availab	intuitively within minutes				installing and c nfig ng the
	le	and used right out of the				software is time-consuming
		box with visualized				and prone to mistakes.
		operations and real-time
		monitoring.

	APIs	Instances can be managed				N/A
		easily by calling RESTful
		APIs.

Costs	On-	Multiple	c fic	n	are	Expenses are incurred for
	deman	available to suit		ff	n	setting up a message service
	d use	needs.				and occupying underlying
						resources.

	Fully	Services are readily				Users must prepare hardware
	manag	available without requiring				resources and set up the
	ed	additional hardware				service by themselves, and bear
		resources or expenses.				high usage and maintenance
						costs.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide				1 Service Overview

	Catego	Item	DMS for Kafka	Open-source Kafka
	ry
	Proven	Mature	DMS has been proven	Using open-source software
	success		successful in large e-	requires lengthy self-
			commerce events such as	development and	fic	n
			the Vmall 11.11 Shopping	and has had few successful
			Festival. It is also used in	cases.
			the clouds of carrier-grade
			customers across the
			world, and meets strict
			carrier-grade reliability
			standards. DMS closely
			follows up with
			community updates to
			continuously fix known
			open-source vulnerabilities
			and add support for new
			features.

		Feature	While maintaining 100%	Functionality is limited and
		-rich	open-source compatibility,	requires self-development.
			DMS further optimizes
			open-source code to
			improve performance and
			reliability, and provides
			message querying,
			dumping, tracing
			(available soon), and
			many other features.

	Reliabil	Highly	DMS supports cross-AZ	High availability requires self-
	ity	availab	deployment to improve	development or open-source
		le	reliability. In addition,	code implementation, which
			automatic fault detection	are costly and cannot
			and alarms ensure reliable	guarantee reliability.
			operations of key services.

		Simple	O&M is entirely	Users need to develop and
		O&M	transparent to tenants	optimize O&M functions,
			with a full set of	especially alarm n	fic	n
			monitoring and alarm	functions. Otherwise, manual
			functions. O&M personnel	attendance is required.
			will be informed of any
			exceptions, eliminating the
			need for 24/7 attending.

		Secure	DMS uses VPC isolation	Security must be hardened by
			and SSL channel	users themselves.
			encryption.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide	1 Service Overview

1.7 Notes and Constraints

DMS for Kafka has the following constraints, as listed in Table 1-3.

Table 1-3 Kafka usage restrictions

Item	Constraint	Description

Kafka Zookeeper	Not exposed externally	DMS ZooKeeper does
		not provide services
		externally. It is only used
		to serve Kafka instances.

Version	2.3.0	Clients later than version
		0.10 are supported. Use
		a version that is
		consistent with the
		service version.

Message size	10 MB	The message size cannot
		exceed 10 MB.
		Otherwise, the message
		creation will fail.

Logging in to the VM	No supported	N/A
where the Kafka brokers
reside

Partition quantity	Limited	Kafka manages
		messages by partition. If
		there are too many
		partitions, message
		creation, storage, and
		retrieval will be
		fragmented,	ff c ng
		the performance and
		stability. If the total
		number of partitions of
		topics reaches the upper
		limit, you cannot create
		more topics.

Automatic topic creation	Supported	nfig b	during
		instance creation.
		If it is enabled, a topic
		will be automatically
		created with 3 partitions
		and 3 replicas when a
		message is created in or
		retrieved from a topic
		that does not exist.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka
User Guide			1 Service Overview

	Item	Constraint	Description

	Creating consumer	Not required	Consumer groups,
	groups, consumers, and		consumers, and
	producers		producers are generated
			automatically when you
			use the instance.

	Decreasing partition	Not supported	The partition quantity
	quantity		cannot be decreased due
			to the limitations of
			Apache Kafka.

1.8Related Services

●CTS

Cloud Trace Service (CTS) generates traces to provide you with a history of operations performed on cloud service resources. The traces include operation requests sent using the management console or open APIs, as well as the operation results. You can view all generated traces to query, audit, and backtrack performed operations.

For details about the operations recorded by CTS, see section "Operations That Can Be Recorded by CTS".

●VPC

Kafka premium instances run in VPCs and use the IP addresses and bandwidth of VPC. Security groups of VPCs enhance the security of network access to the Kafka premium instances.

1.9Basic Concepts

DMS for Kafka of the public cloud uses Kafka as the message engine. This chapter presents explanations of basic concepts of Kafka.

Topic

A topic is a category for messages. Messages are created, retrieved, and managed in the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages into topics. One or more consumers subscribe to the messages in the topics. The producers and consumers are not directly linked to each other.

Producer

A producer publishes messages into topics. The messages are then delivered to other systems or modules for processing as agreed.

Issue 01 (2020-12-02)

Distributed Message Service for Kafka

User Guide 1 Service Overview

Consumer

A consumer subscribes to messages in topics and processes the messages. For example, a monitoring and alarm platform (a consumer) subscribing to log messages in certain topics can identify alarm logs and then send SMS or email alarm n fic n

Broker

A broker is a Kafka process in a Kafka cluster. Each process runs on a server, so a broker includes the storage, bandwidth, and other server resources.

Partition

Messages in a topic are distributed to multiple partitions to achieve scalability and fault tolerance.

Replica

A replica is a redundant copy of a partition in a topic. Each partition can have one or more replicas, enabling message reliability.

Messages in each partition are fully replicated and synchronized, preventing data loss if one replica fails.

Each partition has one replica as the leader which handles the creation and retrievals of all messages. The rest replicas are followers which replicate the leader.

Topics and partitions are logical concepts, while replicas and brokers are physical concepts. The following diagram shows the relationships between partitions, brokers, and topics in messages streaming.

Issue 01 (2020-12-02)

Huawei Distributed Message Service for Kafka User Manual

Distributed Message Service for Kafka
User Guide	1 Service Overview

Figure 1-6 Kafka message streaming

Issue 01 (2020-12-02)

+ 42 hidden pages