HUAWEI Object Storage Migration Service User Manual

Download

Object Storage Migration Service

User Guide

Issue 06

Date 2021-03-24

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specied in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every eort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Object Storage Migration Service User Guide Contents

1 Permissions Management..................................................................................................... 1

1.1 Creating a User and Granting OMS Permissions......................................................................................................... 1

2 Migration Tasks........................................................................................................................ 3

2.1 Creating a Migration Task.................................................................................................................................................... 3

2.2 Viewing a Migration Task.................................................................................................................................................. 17

2.3 Managing a Migration Task.............................................................................................................................................. 18

3 Migration Task Groups......................................................................................................... 20

3.1 Creating a Migration Task Group.................................................................................................................................... 20

3.2 Viewing a Migration Task Group..................................................................................................................................... 31

3.3 Managing a Migration Task Group................................................................................................................................. 32

4 Pre-migration Evaluation.................................................................................................... 35

4.1 Evaluating Data in a Bucket..............................................................................................................................................35

4.2 Creating a Migration Task Based on an Evaluation.................................................................................................. 37

5 Synchronization Tasks.......................................................................................................... 40

5.1 Creating a Synchronization Task..................................................................................................................................... 40

Conguring a Synchronization Request on the Source Platform.........................................................................46

5.2

5.3 Monitoring a Synchronization Task................................................................................................................................ 53

6 Viewing CTS Traces............................................................................................................... 57

7 Change History...................................................................................................................... 58

Object Storage Migration Service User Guide 1 Permissions Management

1 Permissions Management

1.1 Creating a User and Granting OMS Permissions

This chapter describes how to use IAM for ne-grained permissions control for your OMS resources. With IAM, you can:

Prerequisites

● Create IAM users for employees based on your enterprise's organizational structure. Each IAM user will have their own security credentials for accessing OMS resources.

● Grant only the permissions required for users to perform a

● Entrust a HUAWEI CLOUD account or cloud service to perform professional

ecient O&M on OMS.

and

If your HUAWEI CLOUD account does not need individual IAM users, then you may skip over this chapter.

This section describes the procedure for granting permissions (see Figure 1-1).

Learn about the permissions (see Permission Management) supported by OMS and choose policies or roles according to your requirements. For the system policies of other services, see System Permissions.

specic task.

Object Storage Migration Service User Guide 1 Permissions Management

Process Flow

Figure 1-1 Process for granting OMS permissions

1. Create a user group and assign permissions to it.

Create a user group on the IAM console, and assign the OMS Administrator and OBS OperateAccess permissions to the group.

2. Create an IAM user.

Create a user on the IAM console and add the user to the group created in 1.

3. Log in and verify permissions.

– Choose Service List > Object Storage Migration Service. On the OMS

console, click Create Migration Task in the upper right corner. If a migration task can be created, the OMS Administrator policy has already

eect.

taken

– Choose any other service in Service List. If a message appears indicating

insucient permissions to access the service, the OMS Administrator policy has already taken eect.

Object Storage Migration Service User Guide 2 Migration Tasks

2 Migration Tasks

If there is less than 3 TB of data or less than 5 million objects in a source bucket, you can create a migration task to quickly migrate the source data.

SMS provides APIs for you to manage migration tasks. You can also customize APIs for performing desired operations, for example, creating synchronization tasks periodically.

2.1 Creating a Migration Task

Scenarios

This section describes how to create a task to migrate your source objects.

Background

The following migration sources are supported:

● HUAWEI CLOUD

● AWS

● Alibaba Cloud

● Microsoft Azure

● Tencent Cloud

● Qiniu Cloud

● QingCloud

● Baidu Cloud

● Kingsoft Cloud

● HTTP/HTTPS data source

OMS makes API calls to the object storage services on source platforms during migration. These API calls may generate fees on the source platforms and you will be responsible for those fees based on those source platform billing structures.

Object Storage Migration Service User Guide 2 Migration Tasks

Prerequisites

● You have obtained the AK/SK pairs for the source and destination platform accounts.

– The source platform account needs the following permissions:

▪ Listing all buckets ▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content

For details about how to obtain these permissions, see Source

Permissions.

– The destination platform account needs the following permissions:

▪ Listing all buckets

Constraints

Obtaining bucket location

▪

Listing objects

▪

Obtaining object metadata

▪

Obtaining object content

▪

Listing uploaded parts

▪

Restoring archive objects

▪

For details about how to obtain these permissions, see Destination

Permissions.

● You have created at least one bucket on OBS.

● You have created less than 500 migration tasks during the past 24 hours.

● You have less than 500 migration tasks in the Waiting to migrate state.

● You have registered an account with HUAWEI CLOUD and completed realname authentication.

OMS only supports the migration of the ContentType

elds of object metadata.

Procedure

Step 1 Log in to the management console.

Step 2 Click

Step 3 Click Service List. Under Migration, click Object Storage Migration Service.

The Object Storage Migration Service page is displayed.

in the upper left corner and select the desired region.

Object Storage Migration Service User Guide 2 Migration Tasks

Step 4 In the upper right corner, click Create Migration Task.

Read the privacy statement, select Agree to the above privacy agreements, and click OK.

Step 5 For step Select Source/Destination, you need to:

1. Select the source platform. For details about the parameters, see Table 2-1.

Table 2-1 Source platform parameters

Parameter Description

Data Source Species the source cloud service provider.

(Optional) appID Species the identier of the account for logging in

to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, appID must be

specied.

Access Key Species the AK of the source platform account.

Enter a maximum of 100 characters.

Secret Access Key Species the SK of the source platform account. The

SK matches with the AK. Enter a maximum of 100 characters.

Specify Bucket Name If you do not have the permissions required to list

buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects, enable this parameter to specify the path of the source bucket for the migration.

For details, see Table 2-2.

Table 2-2 Description of Specify Bucket Name

If You Need to...

Then...

Enable Specify Bucket Name 1. Select the region where the source

bucket resides from the Select Region drop-down list.

2. In the Bucket Name or Path text box, enter the source bucket name or the source bucket path with the bucket name.

3. Click Connect to Source Bucket.

NOTE

OMS will check whether the source platform information is correct. If the entered information is correct, a message is displayed indicating that the source bucket is connected.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Need to... Then...

Disable Specify Bucket Name (This feature is disabled by default.)

1. Click Connect to Source Bucket. OMS will check whether the source platform information is correct. If the entered information is correct and your account has the permissions needed to list bucket names, all buckets are listed in the Bucket drop-down list.

2. In the Bucket drop-down list, select the bucket where the data to be migrated resides.

2. Select the destination platform. For details about the parameters, see Table

2-3.

Table 2-3 Destination platform parameters

Parameter Description

Access Key Species the AK for accessing HUAWEI CLOUD.

Enter a maximum of 100 characters.

Secret Access Key Species the SK for accessing HUAWEI CLOUD. The

SK matches with the AK. Enter a maximum of 100 characters.

After entering the AK and SK, click Connect to Destination Bucket.

Bucket Species the destination OBS bucket for storing the

migrated data.

Step 6 Click Next to set task parameters.

Step 7 Select a migration method in the Select Migration Method area.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

File/Folder Click Select, and in the displayed

dialog box, select the les or folders to be migrated, and click OK. You can

click to delete a selected le or folder from the list.

NOTE

● If you select Qiniu Cloud for Data Source, you cannot select

folders.

● If you do not select folders, all objects in the source bucket will be migrated.

● For each migration task, you can select a maximum of 500 les or folders in a given directory.

● The name of any individual object on OBS cannot exceed 1,024 characters. If the name of a migrated object on OBS exceeds this limit, the migration task will fail.

les or

specic les or

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

Object List Generally, this parameter is used to

migrate incremental objects. You need to write the names of source objects to be migrated into an object list le and store the le in an OBS bucket on HUAWEI CLOUD. OMS migrates all objects

specied in the object list le.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

The buckets listed are the buckets obtained by entering the destination AK and SK in Step 5.2.

If the bucket used for storing object list les is not located in the same region as the destination bucket, you cannot select the object list

les.

2. Click Select next to Select Object List File, select the target object list les, and click OK.

NOTE

● Each object list MB.

● The object list and ContentType in the metadata of the le must be text/plain.

● The object list without BOM.

● Each line in the object list le can contain only one object name, and the object name must be URL encoded.

● Spaces are not allowed in each line in the object list migration failures because they may be mistakenly identied as object names.

● The length of each line in the object list le cannot exceed 65,535 characters, or the migration will fail.

● ContentEncoding cannot be set in the metadata of the object list le, or the migration will fail.

● Archive data can only be accessed and obtained by OMS after it is restored. If the object list manually restore them before migration.

le cannot exceed 100

le must be a .txt le,

le must be in UTF-8

le. Spaces may cause

les are archive,

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

Object Name Prex Enter the names or prexes of objects

to be migrated and click Add. The system will automatically list the

matched objects. You can click to delete the objects that do not need to be migrated from the list.

NOTE

If no object names or name prexes are specied, all objects in the source bucket

will be migrated.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

URL List Generally, this setting is used to

migrate HTTP/HTTPS source data. You need to write the URLs of the les to be migrated and their destination object names into a URL list le and store the

le in an OBS bucket on HUAWEI CLOUD. OMS migrates all les specied in the URL list le.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

– The buckets listed are the buckets

obtained by entering the destination AK and SK in Step 5.2.

– If the bucket used for storing object

les is not located in the same

list region as the destination bucket, you cannot select the object list

les.

– For security purposes, if the

migration source supports both HTTP and HTTPS, you are advised to use HTTPS during the migration. If HTTP is used, data may be eavesdropped on, tampered with, or vulnerable to man-in-the-middle or replay attacks.

– The URL list migration does not

support URL redirection.

2. Click Select next to the

Specied

URL File area, select the target URL list les, and click OK.

Object Storage Migration Service User Guide 2 Migration Tasks

If You Select... Then...

NOTE

– Each URL list le cannot exceed 100

MB.

– The URL list le must be a .txt le,

and ContentType in the metadata of the le must be text/plain.

– The URL list le must be in UTF-8

without BOM.

– Each line in the URL list

contain only one URL and one destination object.

– The length of each line in the URL

le cannot exceed 65,535

list characters, or the migration will fail.

– ContentEncoding cannot be set in

the metadata of the URL list le, or the migration will fail.

– In the URL list

le, each line uses a tab character (\t) to separate the URL from the destination object name. The format is [

character][Destination object name

]. The destination object name is encoded using URL Encode. For example:

http://xxx.obs.myhuaweicloud.com/doc/

doc%2f%e6%96%87%e4%bb

le1.txt

%b61.txt

http://xxx.obs.myhuaweicloud.com/doc/ thele2.txt doc%2fthele2.txt http://xxx.obs.myhuaweicloud.com/the %20le.txt http://xxx.obs.myhuaweicloud.com/the %20le2.txt http://xxx.obs.myhuaweicloud.com/doc/ thele.txt

the%20le.txt

the+le2.txt

doc/thele.txt

Use a tab character (Tab key on the keyboard) to separate the URL from the destination object name. Do not use spaces.

In the preceding examples, after the les represented by the URLs are copied to the destination bucket, the objects are named doc/doc/

thele2.txt

, the le.txt, the

le2.txt, and doc/thele.txt.

– Spaces are not allowed in each line

in the URL list le. Spaces may cause migration failures because they may be mistakenly identied as object names.

le can

URL][Tab

Step 8 (Optional) Set Source Parameters.

Parameters described in Table 2-4 are only available if the source cloud service provider oers the related functions.

Object Storage Migration Service User Guide 2 Migration Tasks

Table 2-4 Source parameters

Parameter Description

Use Source Custom Domain Name

If the default domain name cannot meet your migration requirements, then if the source cloud service provider supports custom domain names, you can use a source custom domain name to download data to reduce downloading fees.

NOTE

If the source cloud service provider is Qiniu Cloud, this option is selected by default to ensure a smooth migration because Qiniu Cloud has restrictions on default domain names.

If you select this option, you must:

1. Select a transmission protocol for Transfer Protocol.

NOTE

HTTPS is more secure than HTTP and is recommended.

2. Enter a custom domain name of the bucket for Custom Bucket Domain Name.

3. If you use the CDN authentication provided by the source cloud service provider, select Enable CDN Authentication and

congure the authentication type

and keys.

NOTE

– You can use CDN

CDN by referring to documents of the source cloud service provider.

– The fees generated by using CDN on the source platform are

billed by the source cloud service provider.

– URI redirection is not supported.

4. Click Check to check whether the

authentication only after conguring

conguration is

correct.

– If the

conguration fails the check, click Details to

view the failure causes.

– If the

conguration passes the check, complete other

migration congurations.

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

Archive Data Restoration

Only restored archive data can be migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

● If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.

NOTE

● Currently, the following cloud service providers can automatically restore archive data: HUAWEI CLOUD, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.

● It takes at least a minute to restore an archive object. The larger the archive object is, the longer it takes to restore it, but if an archive object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see related documents of the specic cloud service provider.

● Two types of fees are generated during archive data restoration: fees for API calls and fees for archive data retrieval. These fees are billed by the source cloud service provider.

● After restoration, archive objects will be in restored state for a certain period of time. During this period, objects can be downloaded or accessed. After this period expires, archive objects need to be restored again for migration. For details about how long they stay in the restored state, see related documents from the cloud service provider.

● After the object data is migrated from the source platform to HUAWEI CLOUD, the storage class of the migrated data must be the same as that of the destination bucket.

Step 9 Click Show next to Advanced Settings to expand the advanced parameters.

Step 10 (Optional) Set Advanced Settings parameters.

Object Storage Migration Service User Guide 2 Migration Tasks

Table 2-5 Advanced Settings parameters

Parameter Description

Data Encryption If you select KMS, migrated data is encrypted before

being transferred to the destination OBS buckets.

NOTE

Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

This option is only available if KMS encryption is supported in the current region.

Failed Object Record Species whether to record the objects that fail to be

migrated.

● If this parameter is set to No, the system will not record the objects that fail to be migrated. If the migration fails, all objects in the source bucket need to be transmitted again.

● If this parameter is set to Yes, the system will record the objects that fail to be migrated. If the migration fails, only the objects that fail to be migrated need to transmitted again.

NOTE

The objects that fail to be migrated are recorded in an object

le named by the task name and uploaded to the oms/

list failed_object_lists directory of the destination bucket.

This le contains information about the failed migration tasks, including the failure causes, occurrence time (local UTC), URLencoded names of failed objects, non-URL-encoded names of failed objects, and error codes returned by source SDKs.

le is stored in the destination bucket after the objects in

This

le are successfully retransmitted. If you do not need the

the le, delete it manually.

Incremental Data Species whether to lter source objects to be

migrated by time.

● If this option is set to No, source objects are not ltered by time.

● If this option is set to Yes, only the source objects that are

are migrated. You need to click

modied or added after the specied time

to specify the

time.

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

SMN Notication Species whether to use SMN to notify you of

migration results.

● If this parameter is set to No, no SMN message is sent after the migration.

● If this parameter is set to Yes, you will receive an SMN message informing you of the migration results. Perform the following steps to set SMN parameters:

1. Click Add next to the Select SMN Topic area.

The Select SMN Topic dialog box is displayed.

2. Select a topic for receiving SMN messages and

click OK. If OMS is not authorized to publish messages to the selected SMN topic, a Warning dialog box will display after you click OK. If you need the authorization, click OK to authorize OMS. Otherwise, click Cancel to return to the Select SMN Topic page and select another SMN topic.

Congure Trigger Condition. You can select one

of the following options or both:

– Migration succeeded: An SMN message will be

sent after the migration task succeeds.

– Migration failed: An SMN message will be sent

after the migration task fails.

NOTE

If you experience trouble in using SMN, see What Can I

Do If the SMN Function Works Improperly?

NO TE

Object Storage Migration Service User Guide 2 Migration Tasks

Parameter Description

Trac Limit Set the maximum bandwidth for the migration task

during a specied period of time.

● If you do not need to limit migration trac, select No.

● If you need to limit migration trac, select Yes. Perform the following steps to set

trac limits:

1. Click Add Rule.

A trac limiting rule is displayed.

2. Specify Start Time, End Time, and Maximum

Bandwidth. The value range of the maximum bandwidth is 1 Mbyte/s to 200 Mbyte/s.

3. Repeat Step 10.a and Step 10.b to add another

trac limiting rule.

You can click

in the Operation column to

delete a rule.

NOTE

ve rules.

Description

You can create a maximum of

You can set trac limiting rules in the task details even after the task is created.

In the Description area, enter a maximum of 255 characters to briey describe the migration task.

Step 11 Click Next to enter the conrmation page.

Step 12 Conrm the settings of the migration task and start the migration.

1. Check whether the settings of migration parameters are correct.

If the settings are incorrect, click Previous to return to the previous page for

modication.

2. Click Migrate Now.

A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.

For details about migration task statuses, see Viewing a Migration Task.

After the task is created, a task ID is generated for you for quick identication. It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

----End

Object Storage Migration Service User Guide 2 Migration Tasks

2.2 Viewing a Migration Task

Scenarios

After creating a migration task, you can view the migration progress and details at any time.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Step 2 In the upper right corner of the migration task list, set

migration tasks.

● Task group ID box: If a migration task is issued by creating a migration task group, you can enter the task group ID to task group.

Obtain the task group ID from the Migration Task Group page.

● All statuses box: See Table 2-6 to learn more about the statuses.

Table 2-6 Migration task statuses

Status

Waiting to migrate

Migrating The migration task is ongoing and object data is being

Migration paused

Migration succeeded

Description

A migration task has been created, and background initialization is ongoing or other migration tasks are ongoing, preventing this task from starting.

migrated from other cloud vendors to HUAWEI CLOUD.

The migration task is paused. You can resume the task.

The source servers in the migration task have been successfully migrated.

lter all the migration tasks in that

lter criteria to search for

Migration failed The migration task fails, and the migration is not

complete. You can restart a failed task.

Step 3 Click View Details in the lower right corner of a task to view details about that

task.

NO TE

Object Storage Migration Service User Guide 2 Migration Tasks

The migration task details include the number of migrated objects and parameters for creating a migration task.

You can click the name of the destination OBS bucket to view the object data that has been migrated to HUAWEI CLOUD.

trac limit is enabled for a task in the migrating status, you can click next to Trac

Limit to modify the trac limiting rule for the task.

----End

2.3 Managing a Migration Task

Scenarios

After creating a migration task, you can pause, resume, restart, or delete the task.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Step 2 Manage the target migration task.

Table 2-7 Managing migration tasks

Item

Pausing a migration task

Scenario Operations

You can pause a migration task if:

● The number of concurrent migration tasks exceeds what is supported. You can pause migration tasks less

1. Click .

2. Read the content in the dialog box and click OK. The task status changes to Migration paused.

important to you so that the important migration tasks can be executed preferentially.

● You need to add new objects to be migrated. You can pause an ongoing migration task, and resume the task after these objects are added to the migration task.

Object Storage Migration Service User Guide 2 Migration Tasks

Item Scenario Operations

Resuming a migration task

Restarting a migration task

You can resume a paused migration task.

If a migration task fails, you can restart the task to perform the migration again.

1. Click .

2. In the displayed Resume Task dialog box, enter the source and destination AKs and SKs and click OK.

NOTE

If CDN authentication is enabled, you need to enter the

verication key.

The task status changes to Migrating.

1. Click .

2. In the displayed Resume Task dialog box, enter the source and destination AKs and SKs and click OK.

NOTE

– If Failed Object Record is

enabled, you need to set Migration Range. You can select All to migrate all the objects in the failed migration task or select

Objects failed to be migrated to remigrate

only the failed objects.

– If CDN authentication is

enabled, you need to enter the

verication key.

The task status changes to Migrating.

Deleting a migration task

If you do not need to continue the migration or clear historical migration tasks, you can delete the tasks that are not in the migrating status.

NOTE

If the task is being executing, pause and delete it.

1. Click . The deleted migration tasks will not be displayed in the task list.

----End

NO TE

Object Storage Migration Service User Guide 3 Migration Task Groups

3 Migration Task Groups

If there is more than 3 TB of data or more than 5 million objects to migrate in a source bucket, you can create a migration task group to migrate the source data quickly. The migration task group will intelligently group the source objects into multiple sub-tasks for concurrent migration. These sub-tasks are managed in groups, including progress query, resumption, and restart.

Compared with a migration task, a migration task group provides a faster and more reliable migration, helping you easily identify failed objects and quickly migrate them again.

3.1 Creating a Migration Task Group

Scenarios

A migration task group manages multiple sub-tasks that are automatically split from a large migration task.

Constraints

OMS only supports the migration of the ContentType

Prerequisites

● You have obtained the AK/SK pairs for the source and destination platform accounts.

– The source platform account needs the following permissions:

▪ Listing all buckets

elds of object metadata.

▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content

Object Storage Migration Service User Guide 3 Migration Task Groups

For details about how to obtain these permissions, see Source

Permissions.

– The destination platform account needs the following permissions:

▪ Listing all buckets ▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content ▪ Listing uploaded parts ▪ Restoring archive objects

For details about how to obtain these permissions, see Destination

Permissions.

● You have created at least one bucket on OBS.

● You have created less than 500 migration tasks during the past 24 hours.

● You have less than 500 migration tasks in the Waiting to migrate state.

● You have registered an account with HUAWEI CLOUD and completed realname authentication.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Step 2 In the navigation pane on the left, choose Migration Task Group.

Step 3 In the upper right corner, click Create Migration Task Group.

Step 4 For step Select Source/Destination, you need to:

Service page.

The Migration Task Group page is displayed.

Read the privacy statement, select Agree to the above privacy agreements, and click OK.

The page for creating a migration task group is displayed.

1. Select the source platform. For details about the parameters, see Table 3-1.

Table 3-1 Source platform parameters

Parameter

Data Source Species the source cloud service provider.

(Optional) appID Species the identier of the account for logging in

Description

to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, appID must be

specied.

Object Storage Migration Service User Guide 3 Migration Task Groups

Parameter Description

Access Key Species the AK of the source platform account.

Enter a maximum of 100 characters.

Secret Access Key Species the SK of the source platform account. The

SK matches with the AK. Enter a maximum of 100 characters.

Specify Bucket Name If you do not have the permissions required to list

buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects, enable this parameter to specify the path of the source bucket for the migration.

For details, see Table 3-2.

Table 3-2 Description of Specify Bucket Name

If You Need to... Then...

Enable Specify Bucket Name 1. Select the region where the source

bucket resides from the Select Region drop-down list.

2. In the Bucket Name or Path text box, enter the source bucket name or the source bucket path with the bucket name.

3. Click Connect to Source Bucket.

NOTE

OMS will check whether the source platform information is correct. If the entered information is correct, a message is displayed indicating that the source bucket is connected.

Disable Specify Bucket Name (This feature is disabled by default.)

2. In the Bucket drop-down list, select the bucket where the data to be migrated resides.

2. Select the destination platform. For details about the parameters, see Table

3-3.

Object Storage Migration Service User Guide 3 Migration Task Groups

Table 3-3 Destination platform parameters

Parameter Description

Access Key Species the AK for accessing HUAWEI CLOUD.

Enter a maximum of 100 characters.

Secret Access Key Species the SK for accessing HUAWEI CLOUD. The

SK matches with the AK. Enter a maximum of 100 characters.

After entering the AK and SK, click Connect to Destination Bucket.

Bucket Species the destination OBS bucket for storing the

migrated data.

Step 5 Click Next to set task parameters.

Step 6 Select a migration method in the Select Migration Method area.

If You Select... Then...

Object Name Prex Enter the names or prexes of objects

to be migrated and click Add. The system will automatically list the

matched objects. You can click to delete the objects that do not need to be migrated from the list.

NOTE

If no object names or name prexes are specied, all objects in the source bucket

will be migrated.

The objects to be migrated are recorded in a .txt le, and each migration task has a list le. The storage path of the les is oms/taskgroup/ in the destination bucket. These list migration, and you can manually delete them.

les will be retained after the

Object Storage Migration Service User Guide 3 Migration Task Groups

If You Select... Then...

Object List Folder Generally, this parameter is used to

migrate incremental objects. You need to write the names of source objects to be migrated into an object list le and store the le in an OBS bucket on HUAWEI CLOUD. OMS migrates all objects

specied in the object list le.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

The buckets listed are the buckets obtained by entering the destination AK and SK in Step 4.2.

If the bucket used for storing object list les is not located in the same region as the destination bucket, you cannot select the object list

les.

2. Click Select next to Select Object List Folder, select the target object

les, and click OK.

list

NOTE

● Each object list MB.

● The object list and ContentType in the metadata of the le must be text/plain.

● The object list without BOM.

● Each line in the object list le can contain only one object name, and the object name must be URL encoded.

● Spaces are not allowed in each line in the object list migration failures because they may be mistakenly identied as object names.

● The length of each line in the object list le cannot exceed 65,535 characters, or the migration will fail.

● ContentEncoding cannot be set in the metadata of the object list le, or the migration will fail.

● Archive data can only be accessed and obtained by OMS after it is restored. If the object list manually restore them before migration.

le cannot exceed 1,024

le must be a .txt le,

le must be in UTF-8

le. Spaces may cause

les are archive,

Object Storage Migration Service User Guide 3 Migration Task Groups

If You Select... Then...

URL List Generally, this setting is used to

migrate HTTP/HTTPS source data. You need to write the URLs of the les to be migrated and their destination objects names into one or more .txt URL list

les and store the les in an OBS bucket on HUAWEI CLOUD. You can store up to 2,000 list les in a xed folder in the OBS bucket. Each list le cannot exceed 1 GB. OMS migrates all objects

specied in the

URL list les.

1. Select List File Bucket (HUAWEI

CLOUD).

NOTE

– The buckets listed are the buckets

obtained by entering the destination AK and SK in Step 4.2.

– If the bucket used for storing object

les is not located in the same

list region as the destination bucket, you cannot select the object list

les.

– For security purposes, if the

– The URL list migration does not

support URL redirection.

2. Click Select next to the Select URL

File Folder area, select the target URL list folder, and click OK.

Object Storage Migration Service User Guide 3 Migration Task Groups

If You Select... Then...

NOTE

– Each object list le cannot exceed

1,024 MB.

– The total number of les in the URL

list folder cannot exceed 2,000.

– The URL list

and ContentType in the metadata of the le must be text/plain.

– The URL list

without BOM.

– Each line in the URL list le can

contain only one URL and one destination object.

– The length of each line in the URL

le cannot exceed 65,535

list characters, or the migration will fail.

– ContentEncoding cannot be set in

the metadata of the URL list the migration will fail.

– In the URL list le, each line uses a

tab character (\t) to separate the URL from the destination object name. The format is [

character][Destination object name

is encoded using URL Encode. For example:

http://xxx.obs.myhuaweicloud.com/doc/ le1.txt doc%2f%e6%96%87%e4%bb

%b61.txt

Use a tab character (Tab key on the keyboard) to separate the URL from the destination object name. Do not use spaces.

In the preceding examples, after the les represented by the URLs are copied to the destination bucket, the objects are named

thele2.txt, the le.txt, the le2.txt, and doc/thele.txt.

– Spaces are not allowed in each line

in the URL list le. Spaces may cause migration failures because they may be mistakenly identied as object names.

le must be a .txt le,

le must be in UTF-8

le, or

URL][Tab

]. The destination object name

the%20le.txt

the+le2.txt

doc/thele.txt

doc/doc/

Step 7 (Optional) Set Source Parameters.

Object Storage Migration Service User Guide 3 Migration Task Groups

Parameters described in Table 3-4 are only available if the source cloud service provider oers the related functions.

Table 3-4 Source parameters

Parameter Description

Use Source Custom Domain Name

NOTE

If the source cloud service provider is Qiniu Cloud, this option is selected by default to ensure a smooth migration because Qiniu Cloud has restrictions on default domain names.

If you select this option, you must:

1. Select a transmission protocol for Transfer Protocol.

NOTE

HTTPS is more secure than HTTP and is recommended.

2. Enter a custom domain name of the bucket for Custom Bucket Domain Name.

3. If you use the CDN authentication provided by the source cloud service provider, select Enable CDN Authentication and

congure the authentication type

and keys.

NOTE

– You can use CDN

CDN by referring to documents of the source cloud service provider.

– The fees generated by using CDN on the source platform are

billed by the source cloud service provider.

– URI redirection is not supported.

4. Click Check to check whether the

authentication only after conguring

conguration is

correct.

– If the

conguration fails the check, click Details to

view the failure causes.

– If the

conguration passes the check, complete other

migration congurations.

Object Storage Migration Service User Guide 3 Migration Task Groups

Parameter Description

Archive Data Restoration

Only restored archive data can be migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

● If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.

NOTE

● Currently, the following cloud service providers can automatically restore archive data: HUAWEI CLOUD, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.

● Two types of fees are generated during archive data restoration: fees for API calls and fees for archive data retrieval. These fees are billed by the source cloud service provider.

● After the object data is migrated from the source platform to HUAWEI CLOUD, the storage class of the migrated data must be the same as that of the destination bucket.

Step 8 Click Show next to Advanced Settings to expand the advanced parameters.

Step 9 (Optional) Set Advanced Settings parameters.

Object Storage Migration Service User Guide 3 Migration Task Groups

Table 3-5 Advanced Settings parameters

Parameter Description

Data Encryption If you select KMS, migrated data is encrypted before

being transferred to the destination OBS buckets.

NOTE

Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

This option is only available if KMS encryption is supported in the current region.

Failed Object Record Species whether to record the objects that fail to be

migrated.

● If this parameter is set to No, the system will not record the objects that fail to be migrated. If the migration fails, all objects in the source bucket need to be transmitted again.

● If this parameter is set to Yes, the system will record the objects that fail to be migrated. If the migration fails, only the objects that fail to be migrated need to be transmitted again.

NOTE

The objects that fail to be migrated are recorded in an object

le named by the task name and uploaded to the oms/

list failed_object_lists directory of the destination bucket.

le is stored in the destination bucket after the objects in

This

le are successfully retransmitted. If you do not need the

the le, delete it manually.

Incremental Data Species whether to lter source objects to be

migrated by time.

● If this option is set to No, source objects are not ltered by time.

● If this option is set to Yes, only the source objects that are

modied or added after the specied time

are migrated. You need to click to specify the time.

Object Storage Migration Service User Guide 3 Migration Task Groups

Parameter Description

SMN Notication Species whether to use SMN to notify you of

migration results.

● If this parameter is set to No, no SMN message is sent after the migration.

● If this parameter is set to Yes, you will receive an SMN message informing you of the migration results. Perform the following steps to set

trac limits:

1. Click Add next to the Select SMN Topic area.

The Select SMN Topic dialog box is displayed.

2. Select a topic for receiving SMN messages and

Congure Trigger Condition. You can select one

of the following options or both:

– Migration succeeded: An SMN message will be

sent after the migration task succeeds.

– Migration failed: An SMN message will be sent

after the migration task fails.

NOTE

If you experience trouble in using SMN, see What Can I

Do If the SMN Function Works Improperly?

Object Storage Migration Service User Guide 3 Migration Task Groups

Parameter Description

Trac Limit Set the maximum bandwidth for the migration task

during a specied period of time.

● If you do not need to limit migration trac, select No.

● If you need to limit migration trac, select Yes. Perform the following steps to set

1. Click Add Rule.

A trac limiting rule is displayed.

2. Specify Start Time, End Time, and Maximum

Bandwidth. The value range of the maximum bandwidth is 1 Mbyte/s to 200 Mbyte/s.

3. Repeat Step 9.a and Step 9.b to add another

trac limiting rule.

trac limits:

You can click delete a rule.

NOTE

You can create a maximum of

You can set trac limiting rules in the task details even after the task is created.

Description

Step 10 Click Next to enter the conrmation page.

Step 11 Conrm the settings of the migration task and start the migration.

1. Check whether the settings of migration parameters are correct.

If the settings are incorrect, click Previous to return to the previous page for

modication.

2. Click Migrate Now.

A message is displayed, indicating that the migration task is successfully created. The task group is displayed in the migration task group list.

----End

In the Description area, enter a maximum of 255 characters to briey describe the migration task.

in the Operation column to

ve rules.

3.2 Viewing a Migration Task Group

Scenarios

After creating a migration task group, you can view the migration progress and details at any time.

NO TE

Object Storage Migration Service User Guide 3 Migration Task Groups

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Step 2 In the navigation pane on the left, choose Migration Task Group.

The Migration Task Group page is displayed.

Step 3 In the upper right corner of the migration task group list, choose a status to

task groups.

Table 3-6 describes the statuses.

Table 3-6 Migration task group statuses

Status Description

Waiting The migration task group is being initialized.

Creating Migration tasks in the task group are being created.

Executing Migration tasks in the task group are being executed.

Suspending creation

Creation suspended

Creation failed The migration task creation is failed.

Failed The migration fails. You can restart the task group to retry

The creation of migration tasks in the task group is being suspended.

The creation of migration tasks in the task group is suspended and can be resumed.

the migration.

lter

Succeeded The migration is complete.

Step 4 Click

----End

on the left of a group ID to view details about the task group.

The group details include the number of migration tasks, number of objects, and parameter information of the migration task.

You can click the name of the destination OBS bucket to view the object data that has been migrated to HUAWEI CLOUD.

3.3 Managing a Migration Task Group

After creating a migration task group, you can pause, resume, or restart the group.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Object Storage Migration Service User Guide 3 Migration Task Groups

Step 2 In the navigation pane on the left, choose Migration Task Group.

The Migration Task Group page is displayed.

Step 3 Manage a target migration task group.

Table 3-7 Managing a migration task group

Item Scenario Operations

Pausing a migration task group

Resuming a migration task group

Restarting a migration task group

A task group in the Creating status can be paused.

You can resume paused migration tasks in a group.

If migration tasks fail, you can restart the tasks to retry the migration.

1. Click Pause.

2. Read the content in the dialog box and click OK. The task status changes to Creation suspended.

1. Click Resume.

2. In the displayed Resume Task Group dialog box, enter the source and destination AKs and SKs and click OK.

NOTE

If CDN authentication is enabled, you need to enter the verication key.

The task status changes to Executing.

1. Click Restart.

2. In the displayed Restart Task Group dialog box, enter the source and destination AKs and SKs and click OK.

NOTE

– If Failed Object Record is

enabled, you need to set Migration Range. You can select All to migrate all the objects in the failed migration task or select

Objects failed to be migrated to remigrate

only the failed objects.

– If CDN authentication is

enabled, you need to enter the

verication key.

The task status changes to

Executing.

----End

Object Storage Migration Service User Guide 3 Migration Task Groups

NO TE

Object Storage Migration Service User Guide 4 Pre-migration Evaluation

4 Pre-migration Evaluation

A pre-migration evaluation collects statistics on object distribution in source buckets and estimates how much time will be needed for the migration

4.1 Evaluating Data in a Bucket

Scenarios

If there is more than 3 TB of data or more than 5 million objects in a source bucket, pre-migration evaluation is recommended. The pre-migration evaluation collects statistics on object distribution in the source bucket and estimates how much time will be needed for the migration, so the system can automatically divide and create migration tasks based on the evaluation. This greatly maximizes the concurrent performance of the service.

Currently, the pre-migration evaluation is supported by only the following cloud service providers: HUAWEI CLOUD, Alibaba Cloud, Microsoft Azure, Tencent Cloud, AWS, Qiniu Cloud, Kingsoft Cloud, and Baidu Cloud.

Background

● The basic information about objects in a bucket needs to be fully analyzed during pre-migration evaluation. If there are a large number of objects in the bucket, the evaluation will take a long time. For details, see Table 4-1.

Table 4-1 Object storage evaluation speed

Cloud Service Provider

HUAWEI CLOUD 27 million objects per hour

Alibaba Cloud 27 million objects per hour

Tencent Cloud 20 million objects per hour

Amazon Web Services 24 million objects per hour

Evaluation Speed

Object Storage Migration Service User Guide 4 Pre-migration Evaluation

Cloud Service Provider Evaluation Speed

Qiniu Cloud 20 million objects per hour

Kingsoft Cloud 20 million objects per hour

Baidu Cloud 21 million objects per hour

● Your objects will not be downloaded during evaluation. The evaluation obtains only the information about object attributes.

● The evaluation will bill you only for API calls. You will not be billed for any additional fees.

For example, 10,000 API invocations cost $0.01 USD, and each API invocation lists 1,000 objects. If you have 100 million objects to be evaluated, you will be billed $0.1 USD.

● A maximum of 500 buckets can be evaluated per day for a given tenant.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Step 2 In the navigation pane on the left, choose Pre-Migration Evaluation.

The Pre-Migration Evaluation page is displayed.

Step 3 Click Create Evaluation.

Read the privacy statement, select Agree to the above privacy agreements, and click OK. The page for creating an evaluation task is displayed.

Step 4 Select source parameters based on Table 4-2 and click List Bucket.

All the buckets on the source platform are displayed.

Table 4-2 Source parameter

Parameter

Cloud Service Provider Species the cloud service provider on whose

Access Key Species the access key ID of the source platform

Secret Access Key Species the secret access key of the source platform

conguration

Description

platform the object data to be evaluated is located.

account. Enter a maximum of 100 characters.

account. The SK matches the AK, and they group into a pair. Enter a maximum of 100 characters.

appID Species the identier of the account for logging in

to the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, the value of appID must be entered.

NO TE

Object Storage Migration Service User Guide 4 Pre-migration Evaluation

Step 5 Select the bucket you want to perform the evaluation for and click Start

Evaluation.

If you want to stop the evaluation, click Terminate in the Operation column.

Go to the Pre-Migration Evaluation page, locate the row where the destination bucket resides. If the evaluation succeeds, click View Details in the Evaluation Report column to learn evaluation details.

The evaluation details include the object size distribution, object prex distribution, and object distribution statistics over time.

For details about the evaluation status, see Table 4-3.

Table 4-3 Evaluation status description

Status Description

Waiting to evaluate The evaluation task is being initialized on the

background after being created.

Evaluating The system is evaluating the specied bucket.

Evaluation succeeded The evaluation is complete. You can create a

migration task based on this evaluation.

Evaluation failed The evaluation failed. You need to view the possible

causes.

Evaluation Terminating The system is terminating the evaluation.

Evaluation Terminated The evaluation is terminated. If you need to evaluate

the bucket again, create another task.

----End

4.2 Creating a Migration Task Based on an Evaluation

Scenarios

OMS allows you to automatically divides and creates migration tasks based on evaluation results.

Background

For details about parameter description, see Creating a Migration Task.

Prerequisites

The source bucket to be migrated has been evaluated successfully.

NO TE

Object Storage Migration Service User Guide 4 Pre-migration Evaluation

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration Service

page.

Step 2 On the Pre-Migration Evaluation page, select the bucket that has been

successfully evaluated and click Create Migration Task.

Read the privacy statement, select Agree to the above privacy agreements, and click OK. The page for creating a migration task is displayed.

You can select multiple buckets under the same cloud service provider.

Step 3 Complete the authentication conguration. For details, see Table 4-4.

Table 4-4 Authentication conguration

Type Parameter Description

Step 4

Source Authentication

Access Key Species the AK of the source cloud

platform account. Enter a maximum of 100 characters.

Secret Access Key Species the SK of the source cloud

platform account. The SK matches the AK. Enter a maximum of 100 characters.

Destination Authentication

Access Key Species the AK for accessing

HUAWEI CLOUD. Enter a maximum of 100 characters.

Secret Access Key Species the SK for accessing

HUAWEI CLOUD. The SK matches the AK. Enter a maximum of 100 characters.

Congure basic parameters for each migration task. For details, see Table 4-5.

Table 4-5 Migration task

Parameter

conguration

Description

Destination Bucket Species the HUAWEI CLOUD bucket to which objects

are migrated.

KMS Encryption If you enable KMS Encryption, migrated data is

encrypted before being stored to the destination OBS bucket.

Archive Data Restoration

You can select this option if your source cloud service provider supports automatic restoration of archive data.

Object Storage Migration Service User Guide 4 Pre-migration Evaluation

Parameter Description

Trac Limit Set the maximum trac bandwidth for the migration

task in specied time segments. Click to modify the trac limit rule.

Source Custom Domain Name

If the default domain name cannot meet your migration requirements, you can enable this function

and click to complete conguration.

Step 5

Congure advanced settings. For details, see Table 4-6.

Table 4-6 Advanced settings

Parameter Description

Failed Object Record Species whether to record the objects that failed to

be migrated.

Incremental Data Species whether to lter source objects to be

migrated by time.

Send Notication Species whether to use SMN to notify you of

migration task results.

Step 6 Click Create Task to submit the migration task.

The Request submitted successfully window pops up, indicating that the task has been submitted. Back to the migration task list, view the task creation progress in the upper left corner of the task list.

----End

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

5 Synchronization Tasks

A synchronization task is used to proactively synchronize changed objects in a source bucket. You can the source platform. When a source object is added or modied, the source object storage calls the synchronization API of OMS through a function workow to synchronize the object to the destination OBS bucket on HUAWEI CLOUD.

congure an event trigger in the object storage service on

5.1 Creating a Synchronization Task

Scenarios

Enter the AKs and SKs for accessing the source and destination buckets, and congure the synchronization task parameters for migrating source objects contained in the synchronization request and viewing the task details.

Currently, the synchronization task function is only available in CN North-Beijing4.

Prerequisites

● You have obtained the AK/SK pairs for the source and destination platform accounts.

– The source platform account needs the following permissions:

▪ Listing all buckets ▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content

For details about how to obtain these permissions, see Source

Permissions.

– The destination platform account needs the following permissions:

Object Storage Migration Service User Guide 5 Synchronization Tasks

▪ Listing all buckets ▪ Obtaining bucket location ▪ Listing objects ▪ Obtaining object metadata ▪ Obtaining object content ▪

Listing uploaded parts

▪ Restoring archive objects

For details about how to obtain the permissions, see Destination

Permissions.

● You have created at least one bucket on OBS.

● You have created less than 500 migration tasks during the past 24 hours.

● You have less than 500 migration tasks in the Waiting to migrate state.

● You have less than 5 synchronization tasks in the Synchronizing state.

● You have registered an account with HUAWEI CLOUD and completed realname authentication.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Step 2 In the navigation pane on the left, choose Synchronization Task.

Step 3 In the upper right corner, click Create Synchronization Task.

Step 4 Read the privacy statement, select Agree to the above privacy agreements, and

Step 5 For step Select Source/Destination, you need to:

Service page.

click OK.

1. Select the source platform. For details about the parameters, see Table 5-1.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-1 Source platform parameters

Parameter Description

Data Source Species the source cloud service provider. The

following options are available:

● HUAWEI CLOUD

● AWS

● Alibaba Cloud

● Microsoft Azure

● Tencent Cloud

● Qiniu Cloud

● QingCloud

● Baidu Cloud

● Kingsoft Cloud

● UCloud

appID Species the identier of the account for logging in to

the Tencent Cloud console. If the selected cloud service provider is Tencent Cloud, appID must be specied.

Access Key Species the AK of the source platform account. Enter

a maximum of 100 characters.

Secret Access Key Species the SK of the source platform account. The SK

matches with the AK. Enter a maximum of 100 characters.

Specify Bucket Name If you do not have the permissions required to list

buckets but have the permissions to list objects in buckets, obtain object metadata, and obtain objects, enable this parameter to specify the path of the source bucket for the migration.

For details, see Table 5-2.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-2 Description of Specify Bucket Name

If You Need to... Then...

Enable Specify Bucket Name. 1. Select the region where the source bucket

resides from the Select Region dropdown list.

2. In the Bucket Name or Path text box, enter the source bucket name or the source bucket path with the bucket name.

3. Click Connect to Source Bucket. OMS will check whether the source platform information is correct. If the entered information is correct, a message is displayed indicating that the source bucket is connected.

Disable Specify Bucket Name (This feature is disabled by default.)

2. In the Bucket drop-down list, select the bucket where the data to be migrated resides.

2. Select the destination platform. For details about the parameters, see Table 5-3.

Table 5-3 Destination platform parameters

Parameter

Description

Access Key Species the AK for accessing HUAWEI CLOUD. Enter a

maximum of 100 characters.

Secret Access Key Species the SK for accessing HUAWEI CLOUD. The SK

matches with the AK. Enter a maximum of 100 characters.

After entering the AK and SK, click Connect to Destination Bucket.

Bucket Species the destination OBS bucket for storing the

migrated data.

Step 6 Click Next to set task parameters. For details about the parameters, see Table 5-4.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-4 Task parameters

Parameter Description

Data Encryption If you select KMS, migrated data is encrypted before

being transferred to the destination OBS buckets.

NOTE

Using KMS to encrypt migrated data may decrease the migration speed by about 10%.

This option is only available if KMS encryption is supported in the current region.

Archive Data Restoration

Only restored archive data can be migrated. You can select this option if your source cloud service provider supports automatic restoration of archive data.

● If you select this option, the system automatically restores and migrates archive objects in the migration task. If an archive object fails to be restored, the system records it in the list of objects that failed to be migrated and continues to migrate other objects in the migration task. If you do not select this option, the system directly records archive objects in the list of objects that failed to be migrated and continues to migrate other objects in the migration task.

NOTE

– Currently, the following cloud service providers can

automatically restore archive data: HUAWEI CLOUD, Alibaba Cloud, Kingsoft Cloud, and Tencent Cloud.

– It takes at least a minute to restore an archive object.

The larger the archive object is, the longer it takes to restore it, but if an archive object cannot be restored within three minutes, the restoration fails. The time needed depends on the cloud service provider. For details, see related documents of the service provider.

– Two types of fees are generated during archive data

restoration: fees for API calls and fees for archive data retrieval. These fees are billed by the source cloud service provider.

– After restoration, archive objects will be in restored state

for a certain period of time. During this period, objects can be downloaded or accessed. After this period expires, archive objects need to be restored again for migration. For details about how long they stay in the restored state, see related documents from the cloud service provider.

– After the object data is migrated from the source

platform to HUAWEI CLOUD, the storage class of the migrated data must be the same as that of the destination bucket.

specic cloud

Step 7 Click Next to enter the

conrmation page.

Conrm the settings of the synchronization task and start the synchronization.

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

1. Check whether the settings of synchronization parameters are correct.

If the settings are incorrect, click Previous to return to the previous page for

modication.

2. Click Synchronize Now.

A message is displayed indicating that the task has been submitted. Switch back to the synchronization task list to view the task.

For details about task statuses, see Table 5-5.

Table 5-5 Synchronization task statuses

Status Description

Synchronizing Indicates that the objects in the source bucket are being

synchronized.

NOTE

After you create a synchronization task, the task status is Synchronizing, which indicates means that the synchronization request from the source bucket can be received and the source objects can be synchronized. You need to synchronization request on the source platform to migrate newly added or modied objects in the source bucket. For more details, see

Source Platform. After the conguration, OMS will synchronize

your source objects.

Conguring a Synchronization Request on the

congure a

Paused Indicates that the synchronization of source objects is

paused.

NOTE

If you want to pause a synchronization task, you are advised to stop writing data to the source bucket and wait for 15 minutes before clicking Pause on the synchronization task page. Otherwise, it may cause a migration failure for some objects covered by the received synchronization requests.

Step 8 Restart, pause, or delete the synchronization task.

● Restarting

● Only synchronization tasks in the Paused state can be restarted.

● For security purposes, the system does not save your personal data. When you restart the synchronization task, you need to enter the AKs and SKs for accessing the source and destination buckets again to complete identity authentication.

● Pausing

● Only synchronization tasks in the Synchronizing state can be paused.

● If you want to pause a synchronization task, you are advised to stop writing data to the source bucket and wait for 15 minutes before clicking Pause on the synchronization task page. Otherwise, it may cause a migration failure for some objects covered by the received synchronization requests.

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

● Deleting

● Only synchronization tasks in the Paused state can be deleted.

----End

5.2 Conguring a Synchronization Request on the Source Platform

Scenarios

After you create a synchronization task, you need to congure a synchronization request on the source platform. Then the source object storage will call the synchronization API of OMS through a message newly added or modied source objects.

notication or function to migrate

Desired Eects

● The names of newly added and modied objects in the source bucket can be obtained in real time.

● The synchronization API can be invoked to migrate the newly added and modied objects to the destination bucket.

Conguration Methods

● Using the object storage service and function service

On the source platform, you need to use a function service to create a function and congure event notication for synchronizing incremental source objects through OMS.

When objects are added to or modied in the source bucket, the source object storage service will call the congured function to trigger migration of these objects to the destination bucket.

● Using a self-built service system

When objects are added or modied in the source bucket, the self-build service system proactively captures the information about these objects through the messages received from the object storage service on the source platform or by analyzing the database changes to the service system. Then the system invokes the synchronization API of OMS to migrate the newly added and modied objects from the source bucket to the destination bucket.

Procedure

Following operations use HUAWEI CLOUD as the source platform to describe how to use FunctionGraph to

Step 1 Log in to the management console.

congure synchronization requests:

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

Step 2 Click Service List and choose Computing > FunctionGraph.

The FunctionGraph console is displayed.

Step 3 In the navigation pane on the left, choose Templates.

Step 4 In the search box in the upper right corner of the displayed page, enter OMS and

locate the OMS Data Synchronization template, as shown in Figure 5-1.

Figure 5-1 OMS data synchronization template

Click View Details to view the template details, such as template type, input parameters, output parameters, and precautions.

Step 5 Click Congure. The Create Function page is displayed, as shown in Figure 5-2.

Set function parameters. For details about the parameters, see Table 5-6.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Figure 5-2 Creating a function

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-6 Parameters required for creating a function

Parameter Description

Template sync-oms-data-python

Function Name Specify the function name as needed.

The name must meet the following requirements:

● Consists of 1 character to 60 characters, and only letters, digits, hyphens (-), and underscores (_) are allowed.

● Starts with a letter and ends with a letter or digit.

App Select default, the default app on

FunctionGraph.

Agency Create an agency for delegating

FunctionGraph to access the OBS and OMS services.

Enterprise Project Set this parameter as needed. The

default value is default.

(Optional) Description Describe the function as needed.

Runtime Python 3.6

Handler Enter index.handler.

Environment Variables ● sync_url: Enter the address for

receiving synchronization requests, which can be obtained from the synchronization task details.

● sync_task_id: Enter the synchronization task ID, which can be obtained from the synchronization task list page.

● log_bucket: Enter the name of the bucket used for storing the function invoking logs. The lists of requested or unrequested objects will also be stored into this bucket. The bucket must resides in the current region.

● obs_endpoint: Enter the OBS domain name for current region. For details about regions and endpoints, see Regions and

Endpoints.

NOTE

Click Add Environment Variable to add a customized environment variable.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Step 6 Click Create Now in the pane on the right.

The function details page is displayed.

Step 7 On the displayed Code page, click Select on the right of Dependencies. In the

displayed Select Dependency dialog box, select dependency obssdk-v3 and click

OK.

Step 8 Click the

Conguration tab. On the displayed page, set Execution Timeout (s) to 60 (recommended), and click Save in the upper right corner, as shown in Figure

5-3.

Figure 5-3 Setting the execution timeout duration

Step 9 Click the Triggers tab. In the upper right corner of the displayed page, click Create

Trigger. The Create Trigger dialog box is displayed, as shown in Figure 5-4. Set

trigger parameters. For details about the parameters, see Table 5-7.

Figure 5-4 Creating a trigger

NO TICE

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-7 Parameters required for creating a trigger

Parameter Description

Trigger Type Select Object Storage Service (OBS)

from the drop-down list.

Bucket Name Select the bucket where the data to be

synchronized resides from the dropdown list.

Events Set this parameter based on your

requirements.

(Optional) Event Notication Name An event name is automatically

generated. You can specify a name as needed.

(Optional) Prex Enter a prex to limit notications to

objects whose names start with the matching characters.

(Optional) Sux Enter a sux to limit notications to

objects whose names end with the matching characters.

Step 10 Click OK.

You need to congure a test event to verify the function.

Step 11 In the upper right corner of the function details page, select Congure Test Event

from the Select test event drop-down list. The

Congure Test Event dialog box

is displayed, as shown in Figure 5-5. Set the parameters listed in Table 5-8.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Figure 5-5 Conguring a test event

Table 5-8 Parameters required for conguring a test event

Parameter

Description

Congure Test Event Use the default option Create new

test event.

Event Template Select obs-event-template from the

drop-down list.

Event Name An event name is automatically

generated. You can specify a name as needed.

Step 12 Click Save.

Step 13 Click Test in the upper right corner of the function details page. The test result is

displayed in the upper part of the page.

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

Figure 5-6 Test result

After the test succeeds, upload an object to the source bucket. Wait for 10 minutes. Then the object will be synchronized to the destination bucket.

----End

Obtaining Object Lists

Object List

List of requested objects oms_source_record/success_object

List of unrequested objects oms_source_record/failed_object/

The objects in the lists are those to be synchronized.

Path

5.3 Monitoring a Synchronization Task

Scenarios

After a synchronization task is started, you can view the synchronization task progress and details at any time.

Procedure

Step 1 Log in to the management console and enter the Object Storage Migration

Service page.

Step 2 In the navigation pane on the left, choose Synchronization Task.

Step 3 Click on the left of the task ID and view the task progress. For details about the

parameters, see Table 5-9.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Figure 5-7 Monitoring a synchronization task

Table 5-9 Monitored synchronization task data

Parameter Description

Requested Objects (This Month)

Number of objects covered by the synchronization requests received from the source platform this month.

Objects (This Month) ● Successful

Number of objects that have been synchronized this month. You can view the synchronized objects in the synchronized object list stored in the destination bucket.

● Failed Number of objects that failed to be synchronized this month. You can view the failed objects in the failed object list stored in the destination bucket.

● Ignored Number of objects that have been ignored this month. You can view the ignored objects in the ignored object list stored in the destination bucket.

NOTE

If an object with the same name and size exists in the destination bucket, and its last later than that in the source bucket, the source object will be ignored during synchronization.

modication time is

Synchronized Object Size (This Month)

Total size of objects that have been synchronized this month.

Object Lists OMS records the successfully synchronized objects,

objects that failed to be synchronized, and ignored objects in three separate lists and stores these lists in the destination bucket. You can use these lists to verify the numbers and names of the synchronized, failed, and ignored objects. For details, see Table

5-10.

Synchronization Request Receiving Address

Species the address for receiving the synchronization requests from the source platform. These requests will trigger synchronization. For details about OMS API for synchronization requests, see Creating a Synchronization Event.

Object Storage Migration Service User Guide 5 Synchronization Tasks

Table 5-10 Object lists

Object Type Directory List Content

Synchronized objects

oms/sync_task/

synchronization task ID

success_object_lists

Failed objects ● oms/sync_task/

synchronization task ID

failed_object_lists For example, oms/ sync_task/

synchronization task ID

failed_object_lists/omstask-synctest-20210107111623197 _596

NOTE

This directory stores the list of objects that failed to be synchronized during the migration.

● oms/sync_task/

synchronization task ID

creation_failures For example, oms/ sync_task/

synchronization task ID

failed_object_lists/ creation_failures/ 1609833838031_541d6 e85e877149d44bb55_4.f ailed_list.txt

NOTE

This directory stores the list of objects that fail to be migrated due to task creation failures. If the account is in arrears, frozen, or does not have the required permissions, the task creation will fail.

Occurrence time (local UTC),

non-URL-encoded names of successfully synchronized objects, and object sizes (in bytes)

Failure causes, occurrence time

(local UTC), URL-encoded and non-URL-encoded names of failed objects, and error codes returned by source SDKs

Ignored objects oms/sync_task/

synchronization task ID

skip_object_lists

Occurrence time (local UTC), non-URL-encoded names of ignored objects, and object sizes (in bytes)

NO TE

Object Storage Migration Service User Guide 5 Synchronization Tasks

Object Type Directory List Content

Objects to be synchronized

oms/sync_task/

synchronization task ID

object_lists

Named of objects to be

synchronized

NOTE

OMS automatically records the names of objects to be synchronized covered by the received synchronization requests in a .txt object list the le in the oms/sync_task/

synchronization task ID

object_lists path of the destination bucket.

le, and stores

The statistics displayed on the synchronization task details page are delayed for about 1 hour and for reference only.

----End

NO TE

Object Storage Migration Service User Guide 6 Viewing CTS Traces

6 Viewing CTS Traces

With Cloud Trace Service (CTS), you can record operations associated with OMS. For details, see Table 6-1.

You can view these operations to evaluate their impacts on services. For details, see Querying Real-Time Traces.

On the Trace List page on the CTS console, click Filter. In the displayed pane, select OMS for Trace Source, specify other parameters as needed, and click Query to query operations related to OMS.

Table 6-1 Key OMS operations recorded by CTS

Operation

Starting migration tasks migrationTask startTask

Creating migration tasks migrationTask createTask

Pausing migration tasks migrationTask stopTask

Deleting migration tasks migrationTask deleteTask

Resource Type Trace Name

Object Storage Migration Service User Guide 7 Change History

7 Change History

Released On Description

2021-03-24 This issue is the sixth ocial release.

Optimized Conguring a Synchronization Request on

the Source Platform.

2021-03-18 This issue is the fth ocial release.

● Added Creating a Migration Task.

● Optimized Creating a Migration Task Group.

2021-03-10 This issue is the fourth ocial release.

● Added Synchronization Tasks.

● Added description of application scenarios in

Migration Tasks.

● Added description of application scenarios in

Migration Task Groups.

● Added description of application scenarios in Pre-

migration Evaluation.

2020-02-10 This issue is the third ocial release.

Added Creating a User and Granting OMS

Permissions.

2019-08-31 This issue is the second ocial release.

● Added the description of trac limiting on the task details page to Viewing a Migration Task.

● Added Creating a Migration Task Group to

Managing a Migration Task Group.

● In Evaluating Data in a Bucket, added descriptions about evaluation termination and evaluation statuses.

● Added conguration in Creating a Migration Task Based

on an Evaluation.

Trac Limit to the migration task

Object Storage Migration Service User Guide 7 Change History

Released On Description

2019-07-25 This issue is the rst ocial release.

HUAWEI Object Storage Migration Service User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Contents

1 Permissions Management

1.1 Creating a User and Granting OMS Permissions

2 Migration Tasks

2.1 Creating a Migration Task

2.2 Viewing a Migration Task

2.3 Managing a Migration Task

3 Migration Task Groups

3.1 Creating a Migration Task Group

3.2 Viewing a Migration Task Group

3.3 Managing a Migration Task Group

4 Pre-migration Evaluation

4.1 Evaluating Data in a Bucket

4.2 Creating a Migration Task Based on an Evaluation

5 Synchronization Tasks

5.1 Creating a Synchronization Task

5.3 Monitoring a Synchronization Task

6 Viewing CTS Traces

7 Change History