Huawei Distributed Message Service for Kafka User Manual

Distributed Message Service for Kafka

User Guide

Issue 01

Date 2020-12-02

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every eort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Distributed Message Service for Kafka User Guide Contents

1 Service Overview..................................................................................................................... 1

1.1 What is DMS for Kafka?........................................................................................................................................................1

1.2 Product Advantages................................................................................................................................................................1

1.3 Application Scenarios............................................................................................................................................................. 2

1.4 Specications............................................................................................................................................................................ 4

1.5 Comparing Kafka Instances and DMS Advanced Queues.........................................................................................6

1.6 Comparing DMS for Kafka and Open-Source Kafka...................................................................................................8

1.7 Notes and Constraints......................................................................................................................................................... 10

1.8 Related Services..................................................................................................................................................................... 11

1.9 Basic Concepts....................................................................................................................................................................... 11

2 Preparing Required Resources............................................................................................14

3 Creating an Instance.............................................................................................................16

4 Accessing a Kafka Premium Instance............................................................................... 20

4.1 Accessing a Kafka Premium Instance Without SASL................................................................................................ 20

4.2 Accessing a Kafka Premium Instance with SASL....................................................................................................... 22

5 Managing Instances..............................................................................................................26

5.1 Viewing an Instance............................................................................................................................................................. 26

5.2 Restarting an Instance........................................................................................................................................................ 27

5.3 Deleting an Instance............................................................................................................................................................ 28

5.4 Modifying the Information About an Instance...........................................................................................................30

5.5 Conguring Public Access.................................................................................................................................................. 31

5.6 Resetting Kafka Password..................................................................................................................................................32

5.7 Viewing Background Tasks................................................................................................................................................ 33

6 Managing Topics.................................................................................................................... 35

6.1 Creating a Topic..................................................................................................................................................................... 35

6.2 Deleting a Topic.....................................................................................................................................................................37

7 Auditing................................................................................................................................... 38

7.1 Operations That Can Be Recorded by CTS................................................................................................................... 38

7.2 Viewing Traces on the CTS Console................................................................................................................................41

8 FAQs..........................................................................................................................................42

Distributed Message Service for Kafka User Guide Contents

8.1 Instances.................................................................................................................................................................................. 42

8.1.1 Why Can't I Select Two AZs?......................................................................................................................................... 42

8.1.2 Why Can't I View the Subnet and Security Group Information When Creating a DMS Instance?.......42

8.1.3 How Do I Select Storage Space for a Kafka Instance?.........................................................................................42

8.1.4 How Do I Choose Between High I/O and Ultra-high I/O?..................................................................................43

8.1.5 Which Capacity Threshold Policy Should I Use?..................................................................................................... 43

8.1.6 Which Kafka Version Is Supported?............................................................................................................................ 43

8.1.7 What Is the ZooKeeper Version of a Kafka Instance?.......................................................................................... 43

8.1.8 Are Kafka Instances in Cluster Mode?....................................................................................................................... 43

8.1.9 Can I Modify the Connection Address for Accessing a Kafka Instance?........................................................ 43

8.1.10 How Long Are Kafka SSL Certicates Valid for?..................................................................................................44

8.1.11 How to Synchronize Data from One Kafka Instance to Another?.................................................................44

8.1.12 How Do I Change the SASL_SSL Setting of a Kafka Instance?....................................................................... 44

8.1.13 Are Kafka Brokers and ZooKeeper Deployed on the Same VM or on

8.1.14 What Cipher Suites Are Supported by Kafka?.......................................................................................................44

8.1.15 Can I Change an Instance from Single-AZ Deployment to Multi-AZ Deployment?................................44

8.1.16 Does DMS for Kafka Support Cross-AZ Disaster Recovery? Where Can I View the AZs

for a Purchased Instance?......................................................................................................................................................... 44

8.1.17 Can I Change the VPC and Subnet After a Kafka Instance Is Created?....................................................... 45

8.1.18 Where Can I Find Kafka Streams Use Cases?........................................................................................................45

8.2 Connections............................................................................................................................................................................ 45

8.2.1 Troubleshooting Kafka Connection Exceptions....................................................................................................... 45

8.2.2 How Do I Select and

8.2.3 Can I Access a Kafka Premium Instance Over a Public Network?................................................................... 48

8.2.4 How Many Connection Addresses Does a Kafka Instance Have by Default?.............................................. 49

8.2.5 Do Kafka Instances Support Cross-Region Access?............................................................................................... 49

8.2.6 Can I Access a Kafka Instance Using DNAT?........................................................................................................... 49

8.2.7 Do Kafka Premium Instances Support Cross-Subnet Access?............................................................................49

8.2.8 Why Do I Fail to Access Kafka Using SSL from a Go Client?............................................................................. 49

8.2.9 What If

8.2.10 Does DMS for Kafka Support Password-Free Access?........................................................................................49

8.2.11 Obtaining Kafka Clients................................................................................................................................................ 50

8.2.12 Does DMS for Kafka Support Authentication on Clients by the Server?.....................................................50

8.2.13 Can I Use PEM SSL Truststore When Connecting to a Kafka Instance with SASL_SSL Enabled?........50

8.2.14 What Are the

8.2.15 Which TLS Version Does DMS for Kafka Support, 1.0, 1.1, or 1.2?............................................................... 50

8.2.16 Is There a Limit on the Number of Connections to a Kafka Instance?........................................................ 50

8.2.17 How Many Connections from Each IP Address Are Allowed?.........................................................................50

8.3 Topics and Partitions............................................................................................................................................................50

8.3.1 Is There a Limit on the Number of Topics in a Kafka Instance?....................................................................... 51

8.3.2 Why Is Partition Quantity Limited?............................................................................................................................. 51

8.3.3 Can I Reduce the Partition Quantity?.........................................................................................................................51

8.3.4 Why Do I Fail to Create Topics?................................................................................................................................... 51

Certicates Fail to Be Loaded for SASL Connection?............................................................................49

Congure a Security Group?................................................................................................47

Dierences Between JKS and CRT Certicates?....................................................................... 50

Dierent VMs?........................... 44

Congured

Distributed Message Service for Kafka User Guide Contents

8.3.5 Do Kafka Instances Support Batch Importing Topics or Automatic Topic Creation?.................................51

8.3.6 Why Do Deleted Topics Still Exist?.............................................................................................................................. 51

8.3.7 How Do I Create a Topic?............................................................................................................................................... 51

8.3.8 Are Periods (.) Allowed in Topic Names?.................................................................................................................. 52

8.3.9 What Should I Do If Kafka Storage Space Is Used Up Because Retrieved Messages Are Not Deleted?

............................................................................................................................................................................................................ 52

8.3.10 What Can I Do If Disk Usage Is as High as 96%?............................................................................................... 52

8.3.11 Will a Kafka Instance Be Restarted After Its Automatic Topic Creation Setting Is Modied?............. 52

8.3.12 How Do I Disable Automatic Topic Creation?.......................................................................................................52

8.3.13 Can I Delete Unnecessary Topics in a Consumer Group?................................................................................. 52

8.4 Consumer Groups................................................................................................................................................................. 52

8.4.1 Do I Need to Create Consumer Groups, Producers, and Consumers for Kafka Instances?..................... 52

8.4.2 How Do I Delete Consumer Groups?..........................................................................................................................53

8.4.3 Do I Need to Unsubscribe from a Topic Before Deleting a Consumer Group?............................................53

8.5 Messages..................................................................................................................................................................................53

8.5.1 What Is the Maximum Size of a Message that Can be Created?.....................................................................54

8.5.2 Why Do I Frequently Fail to Poll Due to Rebalancing?........................................................................................54

8.5.3 Why Do Messages Still Exist After the Retention Period Elapses?...................................................................54

8.5.4 Do Kafka Instances Support Delayed Message Delivery?................................................................................... 54

8.5.5 Does the Message Query Function on the Console Show the Replica Message Quantity?....................54

A Change History...................................................................................................................... 55

Distributed Message Service for Kafka User Guide 1 Service Overview

1 Service Overview

1.1 What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/oine system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service that uses the open-source Apache Kafka. It provides Kafka premium instances with isolated computing, storage, and bandwidth resources. DMS for Kafka allows you to apply resources, requirements. It can be used out of the box and frees you from deployment and O&M so that you can focus on the agile development of your applications.

congure topics, partitions, and replicas based on service

Readers' Guide

This documentation introduces DMS for Kafka and its Kafka. You will learn about the detailed information about the specications, console operations, API calling, and client access to instances of DMS for Kafka.

For more information about the basic knowledge of Kafka or technical details about creating and retrieving messages, please go to the ocial Apache Kafka

website.

1.2 Product Advantages

DMS for Kafka provides easy-to-use message queuing based on Apache Kafka. Services can be quickly migrated to the cloud without any change, reducing maintenance and usage costs.

dierences from Apache

● Rapid deployment

Simply set instance information on the DMS for Kafka console, submit your order, and a complete Kafka premium instance will be automatically created and deployed.

● Service migration without

modications

NO TE

Distributed Message Service for Kafka User Guide 1 Service Overview

DMS for Kafka is compatible with open-source Kafka APIs and supports all message processing functions of open-source Kafka.

If your application services are developed based on open-source Kafka, you can easily migrate them to DMS for Kafka after specifying a few authentication

Kafka premium instances are compatible with Apache Kafka 2.3.0.

● Security

Operations on Kafka premium instances are recorded and can be audited. Messages can be encrypted before storage.

In addition to SASL, Virtual Private Clouds (VPCs) and security groups also provide security controls on network access.

● Data reliability

Kafka premium instances support data persistence and replication. Messages can be replicated synchronously or asynchronously between replicas.

● High availability

Kafka runs in clusters, enabling failover and fault tolerance so that services can run smoothly.

Kafka premium instances can be deployed across AZs to further enhance service availability.

● Simple O&M

The public cloud provides a whole set of monitoring and alarm services, eliminating the need for 24/7 attendance. A set of Kafka premium instance metrics are monitored and reported, including the number of partitions, topics, and accumulated messages. You can SMS or email

● Flexible specications

You can customize the bandwidth and storage space for the instance and the number of partitions and replicas for topics in the instance.

congurations.

congure alarm rules and receive

notications on how your services are running in real time.

1.3 Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable, asynchronous message delivery. It is widely used for transmitting data between dierent systems in the enterprise application, payment, telecommunications, ecommerce, social networking, instant messaging, video, Internet of Things, and Internet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receiving systems, so that the main service process is not kept waiting for the results of other systems, allowing for faster responses.

For example, Kafka can be used to send a after a user has registered with a website, providing fast responses throughout the registration process.

notication email and SMS message

Distributed Message Service for Kafka User Guide 1 Service Overview

Figure 1-1 Serial registration and notication

Figure 1-2 Asynchronous registration and notication using message queues

Trac Control

In e-commerce systems or large-scale websites, there is a processing capability gap between upstream and downstream systems. Trac bursts from upstream systems with high processing capabilities may have a large impact on downstream systems with lower processing capabilities. For example, online sales promotions involve a huge amount of provides a three-day as orders and other information. In this way, message consumption systems can process the messages during

trac ooding into e-commerce systems. Kafka

buer by default for hundreds of millions of messages, such

o-peak periods.

In addition, ash sale trac bursts originating from frontend systems can be handled with Kafka, keeping the backend systems from crashing.

Figure 1-3

Log Synchronization

In large-scale service systems, logs of dierent applications are collected for quick troubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send log messages to message queues over reliable transmission channels. Other components can read the log messages from message queues for further analysis, either in real time or monitor applications.

Trac burst handling using Kafka

oine. In addition, Kafka can collect key log information to

Log synchronization involves three major components: log collection clients, Kafka, and backend log processing applications.

NO TE

Distributed Message Service for Kafka User Guide 1 Service Overview

1. The log collection clients collect log data from a user application service and asynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has a minor impact on the service performance.

2. Kafka persists logs.

3. Log processing applications, such as Logstash, subscribe to messages in Kafka and retrieve log messages from Kafka. Then, the messages are searched for by le search services or delivered to big data applications such as Hadoop for storage and analysis.

Figure 1-4 Log synchronization process

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big data analytics. They are all open-source tools.

1.4 Specications

Kafka Premium Instance Specications

Kafka premium instances are compatible with open-source Kafka 2.3.0. The instance MB/s, 600 MB/s, and 1200 MB/s.

Table 1-1 TPS and maximum number of partitions supported by dierent instance specications

specications are classied based on bandwidth, namely, 100 MB/s, 300

● The number of brokers varies according to the underlying resources, and the underlying resources vary from region to region. The following table lists the specications.

● In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB.

Band width

Brok ers

Underlying Resource Type

I/O Type

TPS (HighThroughput )

TPS (Synchro nous Replicati

Maxim um Partitio ns

on)

100

3 c6_2 vCPUs | 4GBHigh I/O 100,000 60,000 300

MB/s

Distributed Message Service for Kafka User Guide 1 Service Overview

Band width

300 MB/s

600 MB/s

1200 MB/s

Bandwidth Selection

Brok ers

3 c6_4 vCPUs | 8GBHigh I/O 300,000 150,000 900

4 c6_8 vCPUs |

8 c6_8 vCPUs |

Underlying Resource Type

c6_2 vCPUs | 4GBUltra-

c6_4 vCPUs | 8GBUltra-

16 GB

I/O Type

high I/O

Ultrahigh I/O

TPS (HighThroughput )

100,000 80,000 300

300,000 200,000 900

600,000 300,000 1800

1.2 million 400,000 1800

TPS (Synchro nous Replicati on)

Maxim um Partitio ns

The bandwidth of a Kafka instance refers to the maximum read or write bandwidth. You are advised to select a bandwidth 30% higher than what is required.

● 100 MB/s

Recommended for up to 3000 client connections, 60 consumer groups, and 70 MB/s of service

● 300 MB/s

Recommended for up to 10,000 client connections, 300 consumer groups, and 210 MB/s of service

● 600 MB/s

Recommended for up to 20,000 client connections, 600 consumer groups, and 420 MB/s of service

● 1200 MB/s

Recommended for up to 20,000 client connections, 600 consumer groups, and 840 MB/s of service

Storage Space Selection

Kafka premium instances support storage with 1 to 3 replicas. The storage space is space consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.

trac.

For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x Number of replicas + 100 GB (reserved).

Distributed Message Service for Kafka User Guide 1 Service Overview

Topic Quantity

There is no limit on the topic quantity, but there is an upper limit on the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed (see Table 1-1) and the specied number of partitions in each topic (see Figure

1-5).

Figure 1-5 Setting the number of partitions

For example, the maximum number of partitions for a 100 MB/s instance is 300.

If the number of partitions of each topic in the instance is 3, the number of topics is 300/3 = 100.

If the number of partitions of each topic in the instance is 1, the number of topics is 300/1 = 300.

1.5 Comparing Kafka Instances and DMS Advanced Queues

Both Kafka premium instances and DMS advanced queues are compatible with Apache Kafka. However, they

dier in the following aspects.

Distributed Message Service for Kafka User Guide 1 Service Overview

Open-Source Compatibility

● DMS advanced queues:

Kafka 0.10.2.1

● Kafka premium instances:

Kafka 2.3.0

With each version upgrade, Apache Kafka introduces new features, improves APIs, and updates producer and consumer whether your application features and APIs are compatible with your Kafka clients, see the upgrade notes on the

conguration les. To check

ocial Apache Kafka website.

Creation

● DMS advanced queues:

An advanced queue (equivalent to a topic) is created on the DMS console. You do not need to these resources are allocated by the system.

● Kafka premium instances:

A Kafka premium instance is created on the DMS for Kafka console. Before creating a Kafka premium instance, determine the required bandwidth and storage space based on your service expectations for the next one or two years. You also need to prepare a VPC and security group for the instance.

After the instance has been created, you must create topics in the instance

congure the number of partitions and replicas for the topics.

and

congure the storage space or the bandwidth because

Usage

Performance

● DMS advanced queues:

Advanced queues are compatible with Kafka APIs. DMS provides SDKs in Java, Python, Lua, C, and Go languages..

To use open-source Kafka clients, see "Using the Enhanced Java SDK" in

Developer Guide

Kafka to the directory of the open-source client package and then pass the security authentication.

● Kafka premium instances:

DMS for Kafka is fully compatible with open-source Kafka. You can access Kafka premium instances and topics using open-source Kafka clients. If SASL access is enabled, you must use the SSL

● DMS advanced queues:

There are two queue modes: high-throughput and high-reliability. In the highthroughput mode, messages are concurrency.

● Kafka premium instances:

Compute, bandwidth, and storage resources are physically isolated for each instance. Determine the required bandwidth and storage space when creating an instance. For storage space, you can choose Ultra-high I/O, which indicates that messages are stored on SSDs.

. Add the enhanced Kafka Java SDK provided by DMS for

certicate provided by DMS for Kafka.

ushed to disk asynchronously, ensuring high

NO TE

Distributed Message Service for Kafka User Guide 1 Service Overview

Other Dimensions

You can customize the number of partitions and replicas for a Kafka premium instance. Each topic can have 1 to 50 partitions and 1 to 3 replicas.

By default, an advanced queue has three partitions and three replicas.

Divide a topic into a certain number of partitions so that messages can be evenly distributed to partitions, enabling load balancing and horizontal scalability. Dierent consumers can retrieve messages from one or more partitions, improving message processing performance.

With more replicas come higher reliability. However, synchronizing messages between replicas consumes bandwidth and osets compute performance.

1.6 Comparing DMS for Kafka and Open-Source Kafka

DMS for Kafka is compatible with open-source Kafka and has customized and enhanced Kafka features. In addition to the advantages of open-source Kafka, DMS for Kafka provides more reliable and useful features.

Table 1-2

Dierences between DMS for Kafka and open-source Kafka

Catego ry

Ease of use

Costs On-

Item DMS for Kafka Open-source Kafka

Readily availab le

Instances can be created intuitively within minutes and used right out of the box with visualized

Preparing server resources and installing and conguring the software is time-consuming

and prone to mistakes. operations and real-time monitoring.

APIs Instances can be managed

N/A easily by calling RESTful APIs.

deman d use

Multiple specications are available to suit dierent needs.

Expenses are incurred for

setting up a message service

and occupying underlying

resources.

Fully manag ed

Services are readily available without requiring additional hardware resources or expenses.

Users must prepare hardware

resources and set up the

service by themselves, and bear

high usage and maintenance

costs.

Distributed Message Service for Kafka User Guide 1 Service Overview

CategoryItem DMS for Kafka Open-source Kafka

Proven success

Mature DMS has been proven

successful in large ecommerce events such as the Vmall 11.11 Shopping Festival. It is also used in the clouds of carrier-grade customers across the world, and meets strict carrier-grade reliability standards. DMS closely follows up with community updates to continuously

x known open-source vulnerabilities and add support for new features.

Feature

-rich

While maintaining 100% open-source compatibility, DMS further optimizes open-source code to improve performance and reliability, and provides message querying, dumping, tracing (available soon), and many other features.

Using open-source software requires lengthy selfdevelopment and verication and has had few successful cases.

Functionality is limited and requires self-development.

Reliabil ity

Highly availab le

DMS supports cross-AZ deployment to improve reliability. In addition, automatic fault detection and alarms ensure reliable operations of key services.

Simple O&M

O&M is entirely transparent to tenants with a full set of monitoring and alarm functions. O&M personnel will be informed of any exceptions, eliminating the need for 24/7 attending.

Secure DMS uses VPC isolation

and SSL channel encryption.

High availability requires selfdevelopment or open-source code implementation, which are costly and cannot guarantee reliability.

Users need to develop and optimize O&M functions, especially alarm notication functions. Otherwise, manual attendance is required.

Security must be hardened by users themselves.

Distributed Message Service for Kafka User Guide 1 Service Overview

1.7 Notes and Constraints

DMS for Kafka has the following constraints, as listed in Table 1-3.

Table 1-3 Kafka usage restrictions

Item Constraint Description

Kafka Zookeeper Not exposed externally DMS ZooKeeper does

not provide services externally. It is only used to serve Kafka instances.

Version 2.3.0 Clients later than version

0.10 are supported. Use a version that is consistent with the service version.

Message size 10 MB The message size cannot

exceed 10 MB. Otherwise, the message creation will fail.

Logging in to the VM where the Kafka brokers reside

Partition quantity Limited Kafka manages

Automatic topic creation Supported Congurable during

No supported N/A

messages by partition. If there are too many partitions, message creation, storage, and retrieval will be fragmented, the performance and stability. If the total number of partitions of topics reaches the upper limit, you cannot create more topics.

instance creation.

If it is enabled, a topic will be automatically created with 3 partitions and 3 replicas when a message is created in or retrieved from a topic that does not exist.

aecting

Distributed Message Service for Kafka User Guide 1 Service Overview

Item Constraint Description

Creating consumer groups, consumers, and producers

Decreasing partition quantity

1.8 Related Services

● CTS

Cloud Trace Service (CTS) generates traces to provide you with a history of operations performed on cloud service resources. The traces include operation requests sent using the management console or open APIs, as well as the operation results. You can view all generated traces to query, audit, and backtrack performed operations.

For details about the operations recorded by CTS, see section "Operations That Can Be Recorded by CTS".

Not required Consumer groups,

consumers, and producers are generated automatically when you use the instance.

Not supported The partition quantity

cannot be decreased due to the limitations of Apache Kafka.

● VPC

Kafka premium instances run in VPCs and use the IP addresses and bandwidth of VPC. Security groups of VPCs enhance the security of network access to the Kafka premium instances.

1.9 Basic Concepts

DMS for Kafka of the public cloud uses Kafka as the message engine. This chapter presents explanations of basic concepts of Kafka.

Topic

A topic is a category for messages. Messages are created, retrieved, and managed in the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages into topics. One or more consumers subscribe to the messages in the topics. The producers and consumers are not directly linked to each other.

Producer

A producer publishes messages into topics. The messages are then delivered to other systems or modules for processing as agreed.

Distributed Message Service for Kafka User Guide 1 Service Overview

Consumer

A consumer subscribes to messages in topics and processes the messages. For example, a monitoring and alarm platform (a consumer) subscribing to log messages in certain topics can identify alarm logs and then send SMS or email alarm notications.

Broker

A broker is a Kafka process in a Kafka cluster. Each process runs on a server, so a broker includes the storage, bandwidth, and other server resources.

Partition

Messages in a topic are distributed to multiple partitions to achieve scalability and fault tolerance.

Replica

A replica is a redundant copy of a partition in a topic. Each partition can have one or more replicas, enabling message reliability.

Messages in each partition are fully replicated and synchronized, preventing data loss if one replica fails.

Each partition has one replica as the leader which handles the creation and retrievals of all messages. The rest replicas are followers which replicate the leader.

Topics and partitions are logical concepts, while replicas and brokers are physical concepts. The following diagram shows the relationships between partitions, brokers, and topics in messages streaming.

Distributed Message Service for Kafka User Guide 1 Service Overview

Figure 1-6 Kafka message streaming

+ 42 hidden pages