HUAWEI Kafka User Manual

Distributed Message Service for Kafka

Service Overview

Issue	01
Date	2021-04-07

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise c fi in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every ff has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview	Contents

Contents

1	What is DMS for Kafka?.........................................................................................................	1
2	Product Advantages................................................................................................................	2
3	Application Scenarios.............................................................................................................	4
4	............................................................................................................................	7
5	Comparing Kafka and RabbitMQ......................................................................................	10
6	Comparing DMS for Kafka and Open-Source Kafka.....................................................	13
7	Notes and Constraints..........................................................................................................	15
8	Related Services.....................................................................................................................	17
9	Basic Concepts........................................................................................................................	18
10 Permissions Management.................................................................................................		20
11 Billing.....................................................................................................................................		23

Issue 01 (2021-04-07)

Distributed Message Service for Kafka Service Overview

1 What is DMS for Kafka?

1What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, n n ffl n system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service based on Apache Kafka. This service provides Kafka premium instances. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user. You can apply for instances as required and customize partitions and replicas for the topics in the instances. The instances can be used right out of the box, taking ff the deployment and O&M pressure for you so that you can focus on developing your services.

Readers' Guide

This documentation introduces DMS for Kafka and its ff nc	from Apache
Kafka. You will learn about the detailed information about the	c fic	n

console operations, API calling, and client access to instances of HUAWEI CLOUD DMS for Kafka.

For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the Apache Kafka website.

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview	2 Product Advantages

2Product Advantages

HUAWEI CLOUD DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.

●Rapid deployment

Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka premium instance will be automatically created and deployed.

● Service migration without m fic n

DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka.

If your application services are developed based on open-source Kafka, you can easily migrate them to HUAWEI CLOUD DMS for Kafka after specifying a few authentication c nfi n

NOTE

Kafka premium instances are compatible with Apache Kafka 1.1.0 and 2.3.0. Keep the client and server versions the same.

●Security

Operations on Kafka premium instances are recorded and can be audited. Messages can be encrypted before storage.

In addition to SASL, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.

●Data reliability

Kafka premium instances support data persistence and replication. Messages can be replicated synchronously or asynchronously between replicas.

●High availability

Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly.

Kafka premium instances can be deployed across AZs to further enhance service availability.

●Simple O&M

HUAWEI CLOUD provides a whole set of monitoring and alarm services, eliminating the need for 24/7 attendance. A set of Kafka premium instance

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview		2 Product Advantages
metrics are monitored and reported, including the number of partitions,
topics, and accumulated messages. You can c nfi		alarm rules and receive
SMS or email n fic	n on how your services are running in real time.

●Massive accumulation and auto scaling

Kafka features high scalability because it runs in a distributed system, or cluster. You can c nfi up to 100 partitions for a topic. The storage space can be also expanded. This means that billions of messages can be accumulated, suitable for scenarios requiring high concurrency, high performance, and large-scale access.

● Flexible

c fic n

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview	3 Application Scenarios

3Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between

ff n systems in the enterprise application, payment, telecommunications, e- commerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a n fic n email and SMS message after a user has registered with a website, providing fast responses throughout the registration process.

Figure 3-1 Serial registration and n fic n

Figure 3-2 Asynchronous registration and n fic

n using message queues

Control

In e-commerce systems or large-scale websites, there is a processing capability gap between upstream and downstream systems. ffic bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with lower processing capabilities. For example, online sales promotions involve a huge amount of ffic fl n into e-commerce systems. Kafka

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview			3 Application Scenarios
provides a three-day b		ff by default for hundreds of millions of messages, such
as orders and other information. In this way, message consumption systems can
process the messages during ff			periods.
In addition, fl	sale	ffic bursts originating from frontend systems can be
handled with Kafka, keeping the backend systems from crashing.
Figure 3-3	ffic burst handling using Kafka

Log Synchronization

In large-scale service systems, logs of ff n applications are collected for quick troubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or ffl n In addition, Kafka can collect key log information to monitor applications.

Log synchronization involves three major components: log collection clients, Kafka, and backend log processing applications.

1.The log collection clients collect log data from a user application service and asynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has a minor impact on the service performance.

2.Kafka persists logs.

3.Log processing applications, such as Logstash, subscribe to messages in Kafka

and retrieve log messages from Kafka. Then, the messages are searched for by fi search services or delivered to big data applications such as Hadoop for storage and analysis.

Figure 3-4 Log synchronization process

Issue 01 (2021-04-07)

Distributed Message Service for Kafka
Service Overview	3 Application Scenarios

NOTE

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big data analytics. They are all open-source tools.

Issue 01 (2021-04-07)

+ 19 hidden pages

HUAWEI Kafka User Manual

Contents

1 What is DMS for Kafka?

2 Product Advantages

3 Application Scenarios