HUAWEI Kafka User Manual

Distributed Message Service for Kafka

Service Overview

Issue 01

Date 2021-04-07

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every eort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Distributed Message Service for Kafka Service Overview Contents

1 What is DMS for Kafka?.........................................................................................................1

2 Product Advantages................................................................................................................2

3 Application Scenarios............................................................................................................. 4

Specications............................................................................................................................7

5 Comparing Kafka and RabbitMQ...................................................................................... 10

6 Comparing DMS for Kafka and Open-Source Kafka.....................................................13

7 Notes and Constraints..........................................................................................................15

8 Related Services.....................................................................................................................17

9 Basic Concepts........................................................................................................................18

10 Permissions Management................................................................................................. 20

11 Billing..................................................................................................................................... 23

Distributed Message Service for Kafka Service Overview 1 What is DMS for Kafka?

1 What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/oine system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service based on Apache Kafka. This service provides Kafka premium instances. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user. You can apply for instances as required and customize partitions and replicas for the topics in the instances. The instances can be used right out of the box, taking on developing your services.

Readers' Guide

This documentation introduces DMS for Kafka and its Kafka. You will learn about the detailed information about the specications, console operations, API calling, and client access to instances of HUAWEI CLOUD DMS for Kafka.

For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the

website.

o the deployment and O&M pressure for you so that you can focus

dierences from Apache

ocial Apache Kafka

NO TE

Distributed Message Service for Kafka Service Overview 2 Product Advantages

2 Product Advantages

HUAWEI CLOUD DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.

● Rapid deployment Simply set instance information on the DMS for Kafka console, submit your

order, and a complete Kafka premium instance will be automatically created and deployed.

● Service migration without DMS for Kafka is compatible with open-source Kafka APIs and supports all

message processing functions of open-source Kafka. If your application services are developed based on open-source Kafka, you

can easily migrate them to HUAWEI CLOUD DMS for Kafka after specifying a few authentication

congurations.

modications

Kafka premium instances are compatible with Apache Kafka 1.1.0 and 2.3.0. Keep the client and server versions the same.

● Security Operations on Kafka premium instances are recorded and can be audited.

Messages can be encrypted before storage. In addition to SASL, Virtual Private Clouds (VPCs) and security groups also

provide security controls on network access.

● Data reliability Kafka premium instances support data persistence and replication. Messages

can be replicated synchronously or asynchronously between replicas.

● High availability Kafka runs in clusters, enabling failover and fault tolerance so that services

can run smoothly. Kafka premium instances can be deployed across AZs to further enhance

service availability.

● Simple O&M HUAWEI CLOUD provides a whole set of monitoring and alarm services,

eliminating the need for 24/7 attendance. A set of Kafka premium instance

Distributed Message Service for Kafka Service Overview 2 Product Advantages

metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can congure alarm rules and receive SMS or email notications on how your services are running in real time.

● Massive accumulation and auto scaling Kafka features high scalability because it runs in a distributed system, or

cluster. You can

congure up to 100 partitions for a topic. The storage space can be also expanded. This means that billions of messages can be accumulated, suitable for scenarios requiring high concurrency, high performance, and large-scale access.

● Flexible

specications

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

3 Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between dierent systems in the enterprise application, payment, telecommunications, ecommerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a after a user has registered with a website, providing fast responses throughout the registration process.

Figure 3-1 Serial registration and

Figure 3-2 Asynchronous registration and notication using message queues

notication email and SMS message

notication

Trac Control

In e-commerce systems or large-scale websites, there is a processing capability gap between upstream and downstream systems. Trac bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with lower processing capabilities. For example, online sales promotions involve a huge amount of

trac ooding into e-commerce systems. Kafka

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

provides a three-day buer by default for hundreds of millions of messages, such as orders and other information. In this way, message consumption systems can process the messages during o-peak periods.

In addition, ash sale trac bursts originating from frontend systems can be handled with Kafka, keeping the backend systems from crashing.

Figure 3-3

Log Synchronization

In large-scale service systems, logs of troubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or monitor applications.

Trac burst handling using Kafka

dierent applications are collected for quick

oine. In addition, Kafka can collect key log information to

Log synchronization involves three major components: log collection clients, Kafka, and backend log processing applications.

1. The log collection clients collect log data from a user application service and asynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has a minor impact on the service performance.

2. Kafka persists logs.

3. Log processing applications, such as Logstash, subscribe to messages in Kafka and retrieve log messages from Kafka. Then, the messages are searched for

le search services or delivered to big data applications such as Hadoop for

by storage and analysis.

Figure 3-4 Log synchronization process

NO TE

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big data analytics. They are all open-source tools.

+ 19 hidden pages