HUAWEI Object Storage Migration Service User Manual

Object Storage Migration Service

User Guide

Issue 06

Date 2021-03-24

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every eort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Object Storage Migration Service User Guide Contents

1 Permissions Management..................................................................................................... 1

1.1 Creating a User and Granting OMS Permissions......................................................................................................... 1

2 Migration Tasks........................................................................................................................ 3

2.1 Creating a Migration Task.................................................................................................................................................... 3

2.2 Viewing a Migration Task.................................................................................................................................................. 17

2.3 Managing a Migration Task.............................................................................................................................................. 18

3 Migration Task Groups......................................................................................................... 20

3.1 Creating a Migration Task Group.................................................................................................................................... 20

3.2 Viewing a Migration Task Group..................................................................................................................................... 31

3.3 Managing a Migration Task Group................................................................................................................................. 32

4 Pre-migration Evaluation.................................................................................................... 35

4.1 Evaluating Data in a Bucket..............................................................................................................................................35

4.2 Creating a Migration Task Based on an Evaluation.................................................................................................. 37

5 Synchronization Tasks.......................................................................................................... 40

5.1 Creating a Synchronization Task..................................................................................................................................... 40

Conguring a Synchronization Request on the Source Platform.........................................................................46

5.2

5.3 Monitoring a Synchronization Task................................................................................................................................ 53

6 Viewing CTS Traces............................................................................................................... 57

7 Change History...................................................................................................................... 58

Object Storage Migration Service User Guide 1 Permissions Management

1 Permissions Management

1.1 Creating a User and Granting OMS Permissions

This chapter describes how to use IAM for ne-grained permissions control for your OMS resources. With IAM, you can:

Prerequisites

● Create IAM users for employees based on your enterprise's organizational structure. Each IAM user will have their own security credentials for accessing OMS resources.

● Grant only the permissions required for users to perform a

● Entrust a HUAWEI CLOUD account or cloud service to perform professional

ecient O&M on OMS.

and

If your HUAWEI CLOUD account does not need individual IAM users, then you may skip over this chapter.

This section describes the procedure for granting permissions (see Figure 1-1).

Learn about the permissions (see Permission Management) supported by OMS and choose policies or roles according to your requirements. For the system policies of other services, see System Permissions.

specic task.

Object Storage Migration Service User Guide 1 Permissions Management

Process Flow

Figure 1-1 Process for granting OMS permissions

1. Create a user group and assign permissions to it.

Create a user group on the IAM console, and assign the OMS Administrator and OBS OperateAccess permissions to the group.

2. Create an IAM user.

Create a user on the IAM console and add the user to the group created in 1.

3. Log in and verify permissions.

– Choose Service List > Object Storage Migration Service. On the OMS

console, click Create Migration Task in the upper right corner. If a migration task can be created, the OMS Administrator policy has already

eect.

taken

– Choose any other service in Service List. If a message appears indicating

insucient permissions to access the service, the OMS Administrator policy has already taken eect.

Object Storage Migration Service User Guide 2 Migration Tasks

2 Migration Tasks

If there is less than 3 TB of data or less than 5 million objects in a source bucket, you can create a migration task to quickly migrate the source data.

SMS provides APIs for you to manage migration tasks. You can also customize APIs for performing desired operations, for example, creating synchronization tasks periodically.

2.1 Creating a Migration Task

Scenarios

This section describes how to create a task to migrate your source objects.

Background

The following migration sources are supported:

● HUAWEI CLOUD

● AWS

● Alibaba Cloud

● Microsoft Azure

● Tencent Cloud

● Qiniu Cloud

● QingCloud

● Baidu Cloud

● Kingsoft Cloud

● HTTP/HTTPS data source

OMS makes API calls to the object storage services on source platforms during migration. These API calls may generate fees on the source platforms and you will be responsible for those fees based on those source platform billing structures.

Object Storage Migration Service User Guide 2 Migration Tasks

Prerequisites

● You have obtained the AK/SK pairs for the source and destination platform accounts.

– The source platform account needs the following permissions:

▪ Listing all buckets ▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content

For details about how to obtain these permissions, see Source

Permissions.

– The destination platform account needs the following permissions:

▪ Listing all buckets

Constraints

Obtaining bucket location

▪

Listing objects

▪

Obtaining object metadata

▪

Obtaining object content

▪

Listing uploaded parts

▪

Restoring archive objects

▪

For details about how to obtain these permissions, see Destination

Permissions.

● You have created at least one bucket on OBS.

● You have created less than 500 migration tasks during the past 24 hours.

● You have less than 500 migration tasks in the Waiting to migrate state.

● You have registered an account with HUAWEI CLOUD and completed realname authentication.

OMS only supports the migration of the ContentType

elds of object metadata.

Procedure

Step 1 Log in to the management console.

Step 2 Click

Step 3 Click Service List. Under Migration, click Object Storage Migration Service.

The Object Storage Migration Service page is displayed.

in the upper left corner and select the desired region.

Object Storage Migration Service User Guide 2 Migration Tasks

Step 4 In the upper right corner, click Create Migration Task.

Read the privacy statement, select Agree to the above privacy agreements, and click OK.

Step 5 For step Select Source/Destination, you need to:

1. Select the source platform. For details about the parameters, see Table 2-1.

Table 2-1 Source platform parameters

Parameter Description

Data Source Species the source cloud service provider.

(Optional) appID Species the identier of the account for logging in

to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, appID must be

specied.

Access Key Species the AK of the source platform account.

Enter a maximum of 100 characters.

Secret Access Key Species the SK of the source platform account. The

SK matches with the AK. Enter a maximum of 100 characters.

Specify Bucket Name If you do not have the permissions required to list

buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects, enable this parameter to specify the path of the source bucket for the migration.

For details, see Table 2-2.

Table 2-2 Description of Specify Bucket Name

If You Need to...

Then...

Enable Specify Bucket Name 1. Select the region where the source

bucket resides from the Select Region drop-down list.

2. In the Bucket Name or Path text box, enter the source bucket name or the source bucket path with the bucket name.

3. Click Connect to Source Bucket.

NOTE

OMS will check whether the source platform information is correct. If the entered information is correct, a message is displayed indicating that the source bucket is connected.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Need to... Then...

Disable Specify Bucket Name (This feature is disabled by default.)

1. Click Connect to Source Bucket. OMS will check whether the source platform information is correct. If the entered information is correct and your account has the permissions needed to list bucket names, all buckets are listed in the Bucket drop-down list.

2. In the Bucket drop-down list, select the bucket where the data to be migrated resides.

2. Select the destination platform. For details about the parameters, see Table

2-3.

Table 2-3 Destination platform parameters

Parameter Description

Access Key Species the AK for accessing HUAWEI CLOUD.

Enter a maximum of 100 characters.

Secret Access Key Species the SK for accessing HUAWEI CLOUD. The

SK matches with the AK. Enter a maximum of 100 characters.

After entering the AK and SK, click Connect to Destination Bucket.

Bucket Species the destination OBS bucket for storing the

migrated data.

Step 6 Click Next to set task parameters.

Step 7 Select a migration method in the Select Migration Method area.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

File/Folder Click Select, and in the displayed

dialog box, select the les or folders to be migrated, and click OK. You can

click to delete a selected le or folder from the list.

NOTE

● If you select Qiniu Cloud for Data Source, you cannot select

folders.

● If you do not select folders, all objects in the source bucket will be migrated.

● For each migration task, you can select a maximum of 500 les or folders in a given directory.

● The name of any individual object on OBS cannot exceed 1,024 characters. If the name of a migrated object on OBS exceeds this limit, the migration task will fail.

les or

specic les or

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

Object List Generally, this parameter is used to

migrate incremental objects. You need to write the names of source objects to be migrated into an object list le and store the le in an OBS bucket on HUAWEI CLOUD. OMS migrates all objects

specied in the object list le.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

The buckets listed are the buckets obtained by entering the destination AK and SK in Step 5.2.

If the bucket used for storing object list les is not located in the same region as the destination bucket, you cannot select the object list

les.

2. Click Select next to Select Object List File, select the target object list les, and click OK.

NOTE

● Each object list MB.

● The object list and ContentType in the metadata of the le must be text/plain.

● The object list without BOM.

● Each line in the object list le can contain only one object name, and the object name must be URL encoded.

● Spaces are not allowed in each line in the object list migration failures because they may be mistakenly identied as object names.

● The length of each line in the object list le cannot exceed 65,535 characters, or the migration will fail.

● ContentEncoding cannot be set in the metadata of the object list le, or the migration will fail.

● Archive data can only be accessed and obtained by OMS after it is restored. If the object list manually restore them before migration.

le cannot exceed 100

le must be a .txt le,

le must be in UTF-8

le. Spaces may cause

les are archive,

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

Object Name Prex Enter the names or prexes of objects

to be migrated and click Add. The system will automatically list the

matched objects. You can click to delete the objects that do not need to be migrated from the list.

NOTE

If no object names or name prexes are specied, all objects in the source bucket

will be migrated.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

URL List Generally, this setting is used to

migrate HTTP/HTTPS source data. You need to write the URLs of the les to be migrated and their destination object names into a URL list le and store the

le in an OBS bucket on HUAWEI CLOUD. OMS migrates all les specied in the URL list le.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

– The buckets listed are the buckets

obtained by entering the destination AK and SK in Step 5.2.

– If the bucket used for storing object

les is not located in the same

list region as the destination bucket, you cannot select the object list

les.

– For security purposes, if the

migration source supports both HTTP and HTTPS, you are advised to use HTTPS during the migration. If HTTP is used, data may be eavesdropped on, tampered with, or vulnerable to man-in-the-middle or replay attacks.

– The URL list migration does not

support URL redirection.

2. Click Select next to the

Specied

URL File area, select the target URL list les, and click OK.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

NOTE

– Each URL list le cannot exceed 100

MB.

– The URL list le must be a .txt le,

and ContentType in the metadata of the le must be text/plain.

– The URL list le must be in UTF-8

without BOM.

– Each line in the URL list

contain only one URL and one destination object.

– The length of each line in the URL

le cannot exceed 65,535

list characters, or the migration will fail.

– ContentEncoding cannot be set in

the metadata of the URL list le, or the migration will fail.

– In the URL list

le, each line uses a tab character (\t) to separate the URL from the destination object name. The format is [

character][Destination object name

]. The destination object name is encoded using URL Encode. For example:

http://xxx.obs.myhuaweicloud.com/doc/

doc%2f%e6%96%87%e4%bb

le1.txt

%b61.txt

http://xxx.obs.myhuaweicloud.com/doc/ thele2.txt doc%2fthele2.txt http://xxx.obs.myhuaweicloud.com/the %20le.txt http://xxx.obs.myhuaweicloud.com/the %20le2.txt http://xxx.obs.myhuaweicloud.com/doc/ thele.txt

the%20le.txt

the+le2.txt

doc/thele.txt

Use a tab character (Tab key on the keyboard) to separate the URL from the destination object name. Do not use spaces.

In the preceding examples, after the les represented by the URLs are copied to the destination bucket, the objects are named doc/doc/

thele2.txt

, the le.txt, the

le2.txt, and doc/thele.txt.

– Spaces are not allowed in each line

in the URL list le. Spaces may cause migration failures because they may be mistakenly identied as object names.

le can

URL][Tab

Step 8 (Optional) Set Source Parameters.

Parameters described in Table 2-4 are only available if the source cloud service provider oers the related functions.

Object Storage Migration Service User Guide 2 Migration Tasks

Table 2-4 Source parameters

Parameter Description

Use Source Custom Domain Name

If the default domain name cannot meet your migration requirements, then if the source cloud service provider supports custom domain names, you can use a source custom domain name to download data to reduce downloading fees.

NOTE

If the source cloud service provider is Qiniu Cloud, this option is selected by default to ensure a smooth migration because Qiniu Cloud has restrictions on default domain names.

If you select this option, you must:

1. Select a transmission protocol for Transfer Protocol.

NOTE

HTTPS is more secure than HTTP and is recommended.

2. Enter a custom domain name of the bucket for Custom Bucket Domain Name.

3. If you use the CDN authentication provided by the source cloud service provider, select Enable CDN Authentication and

congure the authentication type

and keys.

NOTE

– You can use CDN

CDN by referring to documents of the source cloud service provider.

– The fees generated by using CDN on the source platform are

billed by the source cloud service provider.

– URI redirection is not supported.

4. Click Check to check whether the

authentication only after conguring

conguration is

correct.

– If the

conguration fails the check, click Details to

view the failure causes.

– If the

conguration passes the check, complete other

migration congurations.

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

Archive Data Restoration

Only restored archive data can be migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

● If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.

● If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.

NOTE

● Currently, the following cloud service providers can automatically restore archive data: HUAWEI CLOUD, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.

● It takes at least a minute to restore an archive object. The larger the archive object is, the longer it takes to restore it, but if an archive object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see related documents of the specic cloud service provider.

● Two types of fees are generated during archive data restoration: fees for API calls and fees for archive data retrieval. These fees are billed by the source cloud service provider.

● After restoration, archive objects will be in restored state for a certain period of time. During this period, objects can be downloaded or accessed. After this period expires, archive objects need to be restored again for migration. For details about how long they stay in the restored state, see related documents from the cloud service provider.

● After the object data is migrated from the source platform to HUAWEI CLOUD, the storage class of the migrated data must be the same as that of the destination bucket.

Step 9 Click Show next to Advanced Settings to expand the advanced parameters.

Step 10 (Optional) Set Advanced Settings parameters.

Object Storage Migration Service User Guide 2 Migration Tasks

Table 2-5 Advanced Settings parameters

Parameter Description

Data Encryption If you select KMS, migrated data is encrypted before

being transferred to the destination OBS buckets.

NOTE

Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

This option is only available if KMS encryption is supported in the current region.

Failed Object Record Species whether to record the objects that fail to be

migrated.

● If this parameter is set to No, the system will not record the objects that fail to be migrated. If the migration fails, all objects in the source bucket need to be transmitted again.

● If this parameter is set to Yes, the system will record the objects that fail to be migrated. If the migration fails, only the objects that fail to be migrated need to transmitted again.

NOTE

The objects that fail to be migrated are recorded in an object

le named by the task name and uploaded to the oms/

list failed_object_lists directory of the destination bucket.

This le contains information about the failed migration tasks, including the failure causes, occurrence time (local UTC), URLencoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs.

le is stored in the destination bucket after the objects in

This

le are successfully retransmitted. If you do not need the

the le, delete it manually.

Incremental Data Species whether to lter source objects to be

migrated by time.

● If this option is set to No, source objects are not ltered by time.

● If this option is set to Yes, only the source objects that are

are migrated. You need to click

modied or added after the specied time

to specify the

time.

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

SMN Notication Species whether to use SMN to notify you of

migration results.

● If this parameter is set to No, no SMN message is sent after the migration.

● If this parameter is set to Yes, you will receive an SMN message informing you of the migration results. Perform the following steps to set SMN parameters:

1. Click Add next to the Select SMN Topic area.

The Select SMN Topic dialog box is displayed.

2. Select a topic for receiving SMN messages and

click OK. If OMS is not authorized to publish messages to the selected SMN topic, a Warning dialog box will display after you click OK. If you need the authorization, click OK to authorize OMS. Otherwise, click Cancel to return to the Select SMN Topic page and select another SMN topic.

Congure Trigger Condition. You can select one

of the following options or both:

– Migration succeeded: An SMN message will be

sent after the migration task succeeds.

– Migration failed: An SMN message will be sent

after the migration task fails.

NOTE

If you experience trouble in using SMN, see What Can I

Do If the SMN Function Works Improperly?

NO TE

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

Trac Limit Set the maximum bandwidth for the migration task

during a specied period of time.

● If you do not need to limit migration trac, select No.

● If you need to limit migration trac, select Yes. Perform the following steps to set

trac limits:

1. Click Add Rule.

A trac limiting rule is displayed.

2. Specify Start Time, End Time, and Maximum

Bandwidth. The value range of the maximum bandwidth is 1 Mbyte/s to 200 Mbyte/s.

3. Repeat Step 10.a and Step 10.b to add another

trac limiting rule.

You can click

in the Operation column to

delete a rule.

NOTE

ve rules.

Description

You can create a maximum of

You can set trac limiting rules in the task details even after the task is created.

In the Description area, enter a maximum of 255 characters to briey describe the migration task.

Step 11 Click Next to enter the conrmation page.

Step 12 Conrm the settings of the migration task and start the migration.

1. Check whether the settings of migration parameters are correct.

If the settings are incorrect, click Previous to return to the previous page for

modication.

2. Click Migrate Now.

A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.

For details about migration task statuses, see Viewing a Migration Task.

After the task is created, a task ID is generated for you for quick identication. It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

----End

+ 43 hidden pages