HUAWEI Kafka User Manual

Download

Distributed Message Service for Kafka

Service Overview

Issue 01

Date 2021-04-07

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every eort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Distributed Message Service for Kafka Service Overview Contents

1 What is DMS for Kafka?.........................................................................................................1

2 Product Advantages................................................................................................................2

3 Application Scenarios............................................................................................................. 4

Specications............................................................................................................................7

5 Comparing Kafka and RabbitMQ...................................................................................... 10

6 Comparing DMS for Kafka and Open-Source Kafka.....................................................13

7 Notes and Constraints..........................................................................................................15

8 Related Services.....................................................................................................................17

9 Basic Concepts........................................................................................................................18

10 Permissions Management................................................................................................. 20

11 Billing..................................................................................................................................... 23

Distributed Message Service for Kafka Service Overview 1 What is DMS for Kafka?

1 What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/oine system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service based on Apache Kafka. This service provides Kafka premium instances. The computing, storage, and bandwidth resources used by an instance are exclusively occupied by the user. You can apply for instances as required and customize partitions and replicas for the topics in the instances. The instances can be used right out of the box, taking on developing your services.

Readers' Guide

This documentation introduces DMS for Kafka and its Kafka. You will learn about the detailed information about the specications, console operations, API calling, and client access to instances of HUAWEI CLOUD DMS for Kafka.

For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the

website.

o the deployment and O&M pressure for you so that you can focus

dierences from Apache

ocial Apache Kafka

NO TE

Distributed Message Service for Kafka Service Overview 2 Product Advantages

2 Product Advantages

HUAWEI CLOUD DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.

● Rapid deployment Simply set instance information on the DMS for Kafka console, submit your

order, and a complete Kafka premium instance will be automatically created and deployed.

● Service migration without DMS for Kafka is compatible with open-source Kafka APIs and supports all

message processing functions of open-source Kafka. If your application services are developed based on open-source Kafka, you

can easily migrate them to HUAWEI CLOUD DMS for Kafka after specifying a few authentication

congurations.

modications

Kafka premium instances are compatible with Apache Kafka 1.1.0 and 2.3.0. Keep the client and server versions the same.

● Security Operations on Kafka premium instances are recorded and can be audited.

Messages can be encrypted before storage. In addition to SASL, Virtual Private Clouds (VPCs) and security groups also

provide security controls on network access.

● Data reliability Kafka premium instances support data persistence and replication. Messages

can be replicated synchronously or asynchronously between replicas.

● High availability Kafka runs in clusters, enabling failover and fault tolerance so that services

can run smoothly. Kafka premium instances can be deployed across AZs to further enhance

service availability.

● Simple O&M HUAWEI CLOUD provides a whole set of monitoring and alarm services,

eliminating the need for 24/7 attendance. A set of Kafka premium instance

Distributed Message Service for Kafka Service Overview 2 Product Advantages

metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can congure alarm rules and receive SMS or email notications on how your services are running in real time.

● Massive accumulation and auto scaling Kafka features high scalability because it runs in a distributed system, or

cluster. You can

congure up to 100 partitions for a topic. The storage space can be also expanded. This means that billions of messages can be accumulated, suitable for scenarios requiring high concurrency, high performance, and large-scale access.

● Flexible

specications

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

3 Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between dierent systems in the enterprise application, payment, telecommunications, ecommerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a after a user has registered with a website, providing fast responses throughout the registration process.

Figure 3-1 Serial registration and

Figure 3-2 Asynchronous registration and notication using message queues

notication email and SMS message

notication

Trac Control

In e-commerce systems or large-scale websites, there is a processing capability gap between upstream and downstream systems. Trac bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with lower processing capabilities. For example, online sales promotions involve a huge amount of

trac ooding into e-commerce systems. Kafka

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

provides a three-day buer by default for hundreds of millions of messages, such as orders and other information. In this way, message consumption systems can process the messages during o-peak periods.

In addition, ash sale trac bursts originating from frontend systems can be handled with Kafka, keeping the backend systems from crashing.

Figure 3-3

Log Synchronization

In large-scale service systems, logs of troubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or monitor applications.

Trac burst handling using Kafka

dierent applications are collected for quick

oine. In addition, Kafka can collect key log information to

Log synchronization involves three major components: log collection clients, Kafka, and backend log processing applications.

1. The log collection clients collect log data from a user application service and asynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has a minor impact on the service performance.

2. Kafka persists logs.

3. Log processing applications, such as Logstash, subscribe to messages in Kafka and retrieve log messages from Kafka. Then, the messages are searched for

le search services or delivered to big data applications such as Hadoop for

by storage and analysis.

Figure 3-4 Log synchronization process

NO TE

Distributed Message Service for Kafka Service Overview 3 Application Scenarios

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big data analytics. They are all open-source tools.

NO TE

Distributed Message Service for Kafka Service Overview 4 Specications

4 Specications

Kafka Premium Instance Specications

Kafka premium instances are compatible with open-source Kafka 1.1.0 and 2.3.0. The instance specications are classied based on bandwidth, namely, 100 MB/s, 300 MB/s, 600 MB/s, and 1200 MB/s.

● The number of brokers varies according to the underlying resources, and the

underlying resources vary from region to region. The following table lists the specications.

● In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB.

Table 4-1 TPS and maximum number of partitions supported by dierent instance specications

Band width

100 MB/s

300 MB/s

Brok ers

3 c6_2 vCPUs | 4GBHigh I/O 100,000 60,000 300

3 c6_4 vCPUs | 8GBHigh I/O 300,000 150,000 900

Underlying Resource Type

c6_2 vCPUs | 4GBUltra-

c6_4 vCPUs | 8GBUltra-

I/O Type

high I/O

TPS (HighThroughput )

100,000 80,000 300

300,000 200,000 900

TPS (Synchro nous Replicati on)

Maxim um Partitio ns

600 MB/s

1200 MB/s

4 c6_8 vCPUs |

16 GB

8 c6_8 vCPUs |

16 GB

Ultrahigh I/O

600,000 300,000 1800

1.2 million 400,000 1800

Distributed Message Service for Kafka Service Overview 4 Specications

Bandwidth Selection

The bandwidth of a Kafka instance refers to the maximum read or write bandwidth. You are advised to select a bandwidth 30% higher than what is required.

● 100 MB/s Recommended for up to 3000 client connections, 60 consumer groups, and 70

MB/s of service

● 300 MB/s Recommended for up to 10,000 client connections, 300 consumer groups, and

210 MB/s of service

● 600 MB/s Recommended for up to 20,000 client connections, 600 consumer groups, and

420 MB/s of service

● 1200 MB/s Recommended for up to 20,000 client connections, 600 consumer groups, and

840 MB/s of service trac.

trac.

Storage Space Selection

Kafka premium instances support storage with 1 to 3 replicas. The storage space is space consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved).

The storage space can be expanded as your service grows.

Topic Quantity

There is no limit on the topic quantity, but there is an upper limit on the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Table 4-1) and the

4-1).

specied number of partitions in each topic (see Figure

Distributed Message Service for Kafka Service Overview 4 Specications

Figure 4-1 Setting the number of partitions

For example, the maximum number of partitions for a 100 MB/s instance is 300.

If the number of partitions of each topic in the instance is 3, the number of topics is 300/3 = 100.

If the number of partitions of each topic in the instance is 1, the number of topics is 300/1 = 300.

Distributed Message Service for Kafka Service Overview 5 Comparing Kafka and RabbitMQ

5 Comparing Kafka and RabbitMQ

Kafka is pull-based and provides higher throughput. It is suitable for collecting and delivering large volumes of data, such as collecting and analyzing logs. RabbitMQ does not provide as high throughput as Kafka, but it functions.

oers more message queuing

The following is a comparison analysis on the performance, data reliability, service availability, and functions of Kafka and RabbitMQ.

Performance

The performance of message-oriented middleware is measured by throughput. While RabbitMQ provides tens of thousands of QPS, Kafka provides millions.

However, if idempotency and transactions are enabled for Kafka, its performance will be compromised.

Data Reliability

Both Kafka and RabbitMQ provide the replication mechanism to ensure high data reliability.

Service Availability

Kafka runs in clusters and has partitions and replicas. Therefore, single-node failure does not

RabbitMQ also supports clustered deployment with multiple node quantity options.

aect services and the capacity of Kafka can be linearly scaled up.

Functions

Both Kafka and RabbitMQ are popular open-source message-oriented middleware.

dier mainly in the functions, which are listed in the following table.

They

Distributed Message Service for Kafka Service Overview 5 Comparing Kafka and RabbitMQ

Table 5-1 Function dierences between Kafka and RabbitMQ

Function Kafka 1.1.0 or 2.3.0 RabbitMQ 3.7.17

Priority queue

Delayed

Not supported Supported. It is recommended that

the priority be set to 0–10.

Not supported Supported

queue

Dead

Not supported Supported letter queue

Retry Not supported Not supported

Retrieval

Pull-based Pull-based and push-based mode

Message

Supported Supported broadcasti ng

Message tracking

Supports oset and

timestamp tracking.

Not supported. Once a message retrieval has been acknowledged, RabbitMQ will be notied that the message can be deleted.

Message accumulat ion

Supports higher

accumulation performance

than RabbitMQ thanks to

Supported

high throughput.

PersistenceSupported Supported

Message tracing

Not supported Supported by the rehose feature or

the rabbitmq_tracing plugin. However, rabbitmq_tracing reduces performance and should be used only for troubleshooting.

Message

ltering

Multi-

Supported Not supported, but can be

encapsulated.

Not supported Supported tenancy

Multiprotocol

Multilanguage

Only Apache Kafka is

supported.

Kafka is written in Scala and

Java and supports clients in

multiple programming

RabbitMQ is based on AMQP and supports MQTT and STOMP.

RabbitMQ is written in Erlang and supports clients in multiple programming languages.

languages.

NO TE

Distributed Message Service for Kafka Service Overview 5 Comparing Kafka and RabbitMQ

Function Kafka 1.1.0 or 2.3.0 RabbitMQ 3.7.17

Throttling Supports throttling on

producer or consumer

clients.

Ordered message

Supports partition-level

FIFO. delivery

Supports credit-based throttling on producers, a mechanism that triggers protection from within.

Supports FIFO only for singlethreaded message queuing without advanced features such as delayed queues or priority queues.

Security Supports SSL and SASL

Similar to Kafka. authentication and read/ write permissions control.

Idempote ncy

Transactio

Supports idempotency for a

Not supported single producer session.

Supported Supported

n messages

The comparison is made between open-source Kafka and RabbitMQ.

DMS for Kafka and DMS for RabbitMQ maintain open-source compatibility while supporting or enhancing features in the open-source versions.

Distributed Message Service for Kafka Service Overview 6 Comparing DMS for Kafka and Open-Source Kafka

6 Comparing DMS for Kafka and Open-

Source Kafka

DMS for Kafka is compatible with open-source Kafka and has customized and enhanced Kafka features. In addition to the advantages of open-source Kafka, DMS for Kafka provides more reliable and useful features.

Table 6-1

Catego ry

Ease of use

Costs On-

Dierences between DMS for Kafka and open-source Kafka

Item DMS for Kafka Open-source Kafka

Readily availab le

APIs Instances can be managed

deman d use

Fully manag ed

Instances can be created intuitively within minutes and used right out of the box with visualized operations and real-time monitoring.

easily by calling RESTful APIs.

Multiple specications are available to suit dierent needs. The instance bandwidth and disk space can be expanded without downtime.

Services are readily available without requiring additional hardware resources or expenses.

Preparing server resources and installing and conguring the software is time-consuming and prone to mistakes.

N/A

Expenses are incurred for setting up a message service and occupying underlying resources.

Users must prepare hardware resources and set up the service by themselves, and bear high usage and maintenance costs.

Distributed Message Service for Kafka Service Overview 6 Comparing DMS for Kafka and Open-Source Kafka

CategoryItem DMS for Kafka Open-source Kafka

Proven success

Mature DMS has been deployed in

Huawei products and proven successful in large Huawei e-commerce events such as the Vmall

11.11 Shopping Festival. It is also used in the clouds of carrier-grade customers across the world, and meets strict carrier-grade reliability standards. DMS closely follows up with community updates to continuously

x known open-source vulnerabilities and add support for new features.

Feature

-rich

While maintaining 100% open-source compatibility, DMS further optimizes open-source code to improve performance and reliability, and provides message querying, dumping, tracing (available soon), and many other features.

Using open-source software requires lengthy selfdevelopment and verication and has had few successful cases.

Functionality is limited and requires self-development.

Reliabil ity

Highly availab le

DMS supports cross-AZ deployment to improve reliability. In addition, automatic fault detection and alarms ensure reliable operations of key services.

Simple O&M

O&M is entirely transparent to tenants with a full set of monitoring and alarm functions. O&M personnel will be informed of any exceptions, eliminating the need for 24/7 attending.

Secure DMS uses VPC isolation

and SSL channel encryption.

High availability requires selfdevelopment or open-source code implementation, which are costly and cannot guarantee reliability.

Users need to develop and optimize O&M functions, especially alarm notication functions. Otherwise, manual attendance is required.

Security must be hardened by users themselves.

Distributed Message Service for Kafka Service Overview 7 Notes and Constraints

7 Notes and Constraints

DMS for Kafka has the following constraints, as listed in Table 7-1.

Table 7-1 Kafka usage restrictions

Item Constraint Description

Kafka Zookeeper Not exposed externally DMS ZooKeeper does

not provide services externally. It is only used to serve Kafka instances.

Version 1.1.0 and 2.3.0 Clients later than version

0.10 are supported. Use a version that is consistent with the service version.

Message size 10 MB The message size cannot

exceed 10 MB. Otherwise, the message creation will fail.

Logging in to the VM where the Kafka brokers reside

No supported N/A

Distributed Message Service for Kafka Service Overview 7 Notes and Constraints

Item Constraint Description

Partition quantity Limited Kafka manages

messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented,

aecting

the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.

Automatic topic creation Supported Congurable during

instance creation. If it is enabled, a topic

will be automatically created with 3 partitions and 3 replicas when a message is created in or retrieved from a topic that does not exist.

Creating consumer groups, consumers, and producers

Decreasing partition quantity

Not required Consumer groups,

consumers, and producers are generated automatically when you use the instance.

Not supported The partition quantity

cannot be decreased due to the limitations of Apache Kafka.

NO TE

Distributed Message Service for Kafka Service Overview 8 Related Services

8 Related Services

● Cloud Trace Service (CTS) Cloud Trace Service (CTS) generates traces to provide you with a history of

operations performed on cloud service resources. The traces include operation requests sent using the management console or open APIs, as well as the operation results. You can view all generated traces to query, audit, and backtrack performed operations.

For details about the operations recorded by CTS, see Operations That Can

Be Recorded by CTS.

● VPC Kafka premium instances run in VPCs and use the IP addresses and bandwidth

of VPC. Security groups of VPCs enhance the security of network access to the Kafka premium instances.

● Cloud Eye Cloud Eye is an open platform that provides monitoring, alarm reporting, and

alarm

notication for your resources in real time.

The values of all Kafka instance metrics are reported to Cloud Eye every minute.

Distributed Message Service for Kafka Service Overview 9 Basic Concepts

9 Basic Concepts

DMS for Kafka of HUAWEI CLOUD uses Kafka as the message engine. This chapter presents explanations of basic concepts of Kafka.

Topic

Producer

Consumer

Broker

A topic is a category for messages. Messages are created, retrieved, and managed in the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages into topics. One or more consumers subscribe to the messages in the topics. The producers and consumers are not directly linked to each other.

A producer publishes messages into topics. The messages are then delivered to other systems or modules for processing as agreed.

A consumer subscribes to messages in topics and processes the messages. For example, a monitoring and alarm platform (a consumer) subscribing to log messages in certain topics can identify alarm logs and then send SMS or email alarm

A broker is a Kafka process in a Kafka cluster. Each process runs on a server, so a broker includes the storage, bandwidth, and other server resources.

notications.

Partition

Messages in a topic are distributed to multiple partitions to achieve scalability and fault tolerance.

Replica

A replica is a redundant copy of a partition in a topic. Each partition can have one or more replicas, enabling message reliability.

Distributed Message Service for Kafka Service Overview 9 Basic Concepts

Messages in each partition are fully replicated and synchronized, preventing data loss if one replica fails.

Each partition has one replica as the leader which handles the creation and retrievals of all messages. The rest replicas are followers which replicate the leader.

Topics and partitions are logical concepts, while replicas and brokers are physical concepts. The following diagram shows the relationships between partitions, brokers, and topics in messages streaming.

Figure 9-1 Kafka message streaming

NO TE

Distributed Message Service for Kafka Service Overview 10 Permissions Management

10 Permissions Management

If you need to assign dierent permissions to employees in your enterprise to access your DMS resources on HUAWEI CLOUD, Identity and Access Management (IAM) is a good choice for identity authentication, permissions management, and access control, helping you secure access to your HUAWEI CLOUD resources.

ne-grained permissions management. IAM provides

With IAM, you can use your HUAWEI CLOUD account to create IAM users for your employees, and assign permissions to the users to control their access to resource types. For example, some software developers in your enterprise need to use DMS resources but should not be allowed to delete the resources or perform any other high-risk operations. In this scenario, you can create IAM users for the software developers and grant them only the permissions required for using DMS resources.

If your HUAWEI CLOUD account does not require individual IAM users for permissions management, skip this section.

IAM is free of charge. You pay only for the resources you use. For more information about IAM, see the IAM Service Overview.

Kafka permissions policies are based on DMS. Therefore, when assigning permissions, select DMS permissions policies.

DMS Kafka Permissions

By default, new IAM users do not have permissions assigned. You need to add a user to one or more groups, and attach permissions policies or roles to these groups. Users inherit permissions from the groups to which they are added and can perform

specic

specied operations on cloud services based on the permissions.

DMS is a project-level service deployed and accessed in specic physical regions. To assign DMS permissions to a user group, specify the scope as projects and select projects (for example, cn-north-1 for CN North-Beijing1) for the permissions to take eect for the user group in all region-specic projects. When accessing DMS, the users need to switch to a region where they have been authorized to use this service.

eect. If All projects is selected, the permissions will take

region-specic

NO TE

Distributed Message Service for Kafka Service Overview 10 Permissions Management

You can grant permissions by using roles and policies.

● Roles: A type of coarse-grained authorization mechanism that denes permissions related to user responsibilities. This mechanism provides only a limited number of service-level roles for authorization. When using roles to grant permissions, you need to also assign other roles on which the permissions depend to take

eect. However, roles are not an ideal choice for

ne-grained authorization and secure access control.

● Policies: A type of ne-grained authorization mechanism that denes permissions required to perform operations on specic cloud resources under certain conditions. This mechanism allows for more exible policy-based authorization, meeting requirements for secure access control. For example, you can grant DMS users only the permissions for managing instances. Most policies

dene permissions based on APIs. For the API actions supported by

DMS, see Permissions Policies and Supported Actions.

Table 10-1 lists all the

system-dened roles and policies supported by DMS for

Kafka.

Table 10-1

Role/Policy

System-dened roles and policies supported by DMS

Description Type Dependency

Name

DMS FullAccess Administrator permissions

for DMS. Users granted these permissions can perform all operations on DMS.

DMS UserAccess

Common user permissions for DMS, excluding permissions for creating, modifying, deleting, dumping, and scaling up instances.

DMS ReadOnlyAcces s

Read-only permissions for DMS. Users granted these permissions can only view DMS data.

Systemdened

policy

Systemdened

policy

Systemdened

policy

None

● DMS FullAccess, DMS UserAccess, and DMS ReadOnlyAccess policies are used to control the operation permissions for DMS for Kafka and DMS for RabbitMQ.

● The DMS FullAccess, DMS UserAccess, and DMS ReadOnlyAccess policies contain OBS actions. Due to data caching, the policy will take eect ve minutes after it is attached to a user, user group, or project.

Table 2 lists the common operations supported by each system-dened policy or

role of DMS for Kafka. Select the policies or roles as required.

Distributed Message Service for Kafka Service Overview 10 Permissions Management

Table 10-2 Common operations supported by each system-dened policy or role of DMS

Operation DMS FullAccess DMS UserAccess DMS

ReadOnlyAccess

Creating instances

Modifying instances

Deleting instances

Modifying instance

specications

Enabling dumping

Creating dumping tasks

Restarting instances

Querying instance information

√ × ×

√ √ ×

√ √ √

Helpful Links

● IAM Service Overview

● Creating User Groups, Users, and Granting DMS permissions

● Permissions Policies and Supported Actions

Distributed Message Service for Kafka Service Overview 11 Billing

11 Billing

DMS for Kafka supports pay-per-use. For details, see Pricing Details.

Billing Items

DMS for Kafka is billed based on Kafka instance specications and storage space.

Table 11-1 DMS for Kafka billing

Billing Item

Instance ● Kafka instances are billed based on

Description

their bandwidth. To ensure stable service running, you are advised to choose a bandwidth 30% higher than the actual throughput expected for read or write whichever is higher. You should also consider other parameters described in Table 11-2. For example, if the maximum write trac is 70 MB/s and the maximum read trac is 200 MB/s, the bandwidth to be chosen should be at least 200 MB/s x (1 + 30%) = 260 MB/s.

● Kafka instances can be billed on a pay-per-use (hourly) basis.

trac,

Distributed Message Service for Kafka Service Overview 11 Billing

Billing Item Description

Storage space ● Queues are billed based on the

storage space. For each type of instance specication, you can choose the common I/O, high I/O, or ultra-high I/O disk type to meet your service requirements. You can specify the number of replicas. For example, if the required disk size to store message data is 500 GB and there are three replicas, the disk capacity should be at least: 500 GB x 3 = 1500 GB.

● Table 11-2 describes the storage space options in 100 GB increments.

● The storage space can be billed on a pay-per-use (hourly) basis.

Billing Modes

Table 11-2 Kafka instance

Specication

(Bandwidth)

100 MB/s 300 60 3000 600 GB–

300 MB/s 900 300 10,000 1,200 GB–

600 MB/s 1800 600 20,000 2,400 GB–

1200 MB/s 1800 600 20,000 4,800 GB–

Pay-per-use (hourly): This billing mode is exible, enabling you to start and stop services anytime. You pay only for the actual usage duration. The minimum time unit is one hour. Less than an hour is recorded as an hour.

Maximum Partitions

specications

Maximum Consumer Groups

Maximum Client Connections

Storage Space

90,000 GB

Changing

Congurations

● You can change the bandwidth for a Kafka instance. After the change, you will be billed based on the new

● You can also change the storage space of Kafka queues. You will be billed based on the new storage space after the storage space increase. Storage

specications.

Distributed Message Service for Kafka Service Overview 11 Billing

space can only be increased, and cannot be decreased. The minimum increase increment is 100 GB.

HUAWEI Kafka User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

1 What is DMS for Kafka?

2 Product Advantages

3 Application Scenarios

5 Comparing Kafka and RabbitMQ

Distributed Message Service for Kafka Service Overview 6 Comparing DMS for Kafka and Open-Source Kafka

7 Notes and Constraints

8 Related Services

9 Basic Concepts

10 Permissions Management

11 Billing