HUAWEI Object Storage Migration Service User Manual

Object Storage Migration Service

User Guide

Issue	06
Date	2021-03-24

HUAWEI TECHNOLOGIES CO., LTD.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise c fi in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every ff has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	Contents

Contents


1	Permissions Management.....................................................................................................		1
1.1		Creating a User and Granting OMS Permissions.........................................................................................................	1
2	Migration Tasks........................................................................................................................		3
2.1		Creating a Migration Task....................................................................................................................................................	3
2.2		Viewing a Migration Task..................................................................................................................................................	17
2.3		Managing a Migration Task..............................................................................................................................................	18
3	Migration Task Groups.........................................................................................................		20
3.1		Creating a Migration Task Group....................................................................................................................................	20
3.2		Viewing a Migration Task Group.....................................................................................................................................	31
3.3		Managing a Migration Task Group.................................................................................................................................	32
4	Pre-migration Evaluation....................................................................................................		35
4.1		Evaluating Data in a Bucket..............................................................................................................................................	35
4.2		Creating a Migration Task Based on an Evaluation..................................................................................................	37
5	Synchronization Tasks..........................................................................................................		40
5.1		Creating a Synchronization Task.....................................................................................................................................	40
5.2		nfig ng a Synchronization Request on the Source Platform.........................................................................	46
5.3		Monitoring a Synchronization Task................................................................................................................................	53
6	Viewing CTS Traces...............................................................................................................		57
7	Change History......................................................................................................................		58

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	1 Permissions Management

1Permissions Management

1.1 Creating a User and Granting OMS Permissions

This chapter describes how to use IAM for fin g n permissions control for your OMS resources. With IAM, you can:

●Create IAM users for employees based on your enterprise's organizational structure. Each IAM user will have their own security credentials for accessing OMS resources.

● Grant only the permissions required for users to perform a

c fic task.

●Entrust a HUAWEI CLOUD account or cloud service to perform professional

and ffic n O&M on OMS.

If your HUAWEI CLOUD account does not need individual IAM users, then you may skip over this chapter.

This section describes the procedure for granting permissions (see Figure 1-1).

Prerequisites

Learn about the permissions (see Permission Management) supported by OMS and choose policies or roles according to your requirements. For the system policies of other services, see System Permissions.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	1 Permissions Management

Process Flow

Figure 1-1 Process for granting OMS permissions

1.Create a user group and assign permissions to it.

Create a user group on the IAM console, and assign the OMS Administrator and OBS OperateAccess permissions to the group.

2.Create an IAM user.

Create a user on the IAM console and add the user to the group created in 1.

3.Log in and verify permissions.

–Choose Service List > Object Storage Migration Service. On the OMS console, click Create Migration Task in the upper right corner. If a migration task can be created, the OMS Administrator policy has already taken ff c

–Choose any other service in Service List. If a message appears indicating

n ffic n permissions to access the service, the OMS Administrator policy has already taken ff c

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

2Migration Tasks

If there is less than 3 TB of data or less than 5 million objects in a source bucket, you can create a migration task to quickly migrate the source data.

SMS provides APIs for you to manage migration tasks. You can also customize APIs for performing desired operations, for example, creating synchronization tasks periodically.

2.1 Creating a Migration Task

Scenarios

This section describes how to create a task to migrate your source objects.

Background

The following migration sources are supported:

●HUAWEI CLOUD

●AWS

●Alibaba Cloud

●Microsoft Azure

●Tencent Cloud

●Qiniu Cloud

●QingCloud

●Baidu Cloud

●Kingsoft Cloud

●HTTP/HTTPS data source

OMS makes API calls to the object storage services on source platforms during migration. These API calls may generate fees on the source platforms and you will be responsible for those fees based on those source platform billing structures.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

Prerequisites

●You have obtained the AK/SK pairs for the source and destination platform accounts.

–The source platform account needs the following permissions:

▪Listing all buckets

▪Obtaining bucket location

▪Listing objects

▪Obtaining object metadata

▪Obtaining object content

For details about how to obtain these permissions, see Source

Permissions.

–The destination platform account needs the following permissions:

▪Listing all buckets

▪Obtaining bucket location

▪Listing objects

▪Obtaining object metadata

▪Obtaining object content

▪Listing uploaded parts

▪Restoring archive objects

For details about how to obtain these permissions, see Destination

Permissions.

●You have created at least one bucket on OBS.

●You have created less than 500 migration tasks during the past 24 hours.

●You have less than 500 migration tasks in the Waiting to migrate state.

●You have registered an account with HUAWEI CLOUD and completed realname authentication.

Constraints

OMS only supports the migration of the ContentType fi

of object metadata.

Procedure

Step 1 Log in to the management console.

Step 2 Click in the upper left corner and select the desired region.

Step 3 Click Service List. Under Migration, click Object Storage Migration Service. The Object Storage Migration Service page is displayed.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

Step 4 In the upper right corner, click Create Migration Task.

Read the privacy statement, select Agree to the above privacy agreements, and click OK.

Step 5 For step Select Source/Destination, you need to:

1.Select the source platform. For details about the parameters, see Table 2-1.

Table 2-1 Source platform parameters

Parameter	Description

Data Source	c fi	the source cloud service provider.

(Optional) appID	c fi	the n fi of the account for logging in
	to the Tencent Cloud console. If the selected cloud
	service provider is Tencent Cloud, appID must be
	c fi

Access Key	c fi	the AK of the source platform account.
	Enter a maximum of 100 characters.

Secret Access Key	c fi	the SK of the source platform account. The
	SK matches with the AK. Enter a maximum of 100
	characters.

Specify Bucket Name	If you do not have the permissions required to list
	buckets but have the permissions to list objects in
	buckets, obtain object metadata, and obtain objects,
	enable this parameter to specify the path of the
	source bucket for the migration.
	For details, see Table 2-2.

Table 2-2 Description of Specify Bucket Name

If You Need to...		Then...

Enable Specify Bucket Name		1. Select the region where the source
		bucket resides from the Select Region
		drop-down list.
		2. In the Bucket Name or Path text box,
		enter the source bucket name or the
		source bucket path with the bucket
		name.
		3. Click Connect to Source Bucket.
		NOTE
		OMS will check whether the source platform
		information is correct. If the entered
		information is correct, a message is
		displayed indicating that the source bucket
		is connected.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide		2 Migration Tasks

	If You Need to...	Then...

	Disable Specify Bucket Name	1. Click Connect to Source Bucket.
	(This feature is disabled by	OMS will check whether the source
	default.)	platform information is correct. If the
		entered information is correct and your
		account has the permissions needed to
		list bucket names, all buckets are listed
		in the Bucket drop-down list.
		2. In the Bucket drop-down list, select the
		bucket where the data to be migrated
		resides.

2.Select the destination platform. For details about the parameters, see Table 2-3.

Table 2-3 Destination platform parameters

Parameter	Description

Access Key	c fi	the AK for accessing HUAWEI CLOUD.
	Enter a maximum of 100 characters.

Secret Access Key	c fi	the SK for accessing HUAWEI CLOUD. The
	SK matches with the AK. Enter a maximum of 100
	characters.
	After entering the AK and SK, click Connect to
	Destination Bucket.

Bucket	c fi	the destination OBS bucket for storing the
	migrated data.

Step 6 Click Next to set task parameters.

Step 7 Select a migration method in the Select Migration Method area.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide				2 Migration Tasks

	If You Select...	Then...

	File/Folder	Click Select, and in the displayed
		dialog box, select the fi		or folders to
		be migrated, and click OK. You can
		click	to delete a selected fi		or
		folder from the list.
		NOTE
		● If you select Qiniu Cloud for Data
			Source, you cannot select fi		or
			folders.
		● If you do not select		c fic fi	or
			folders, all objects in the source bucket
			will be migrated.
		● For each migration task, you can select
			a maximum of 500 fi	or folders in a
			given directory.
		● The name of any individual object on
			OBS cannot exceed 1,024 characters. If
			the name of a migrated object on OBS
			exceeds this limit, the migration task
			will fail.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide						2 Migration Tasks

	If You Select...	Then...

	Object List	Generally, this parameter is used to
		migrate incremental objects. You need
		to write the names of source objects to
		be migrated into an object list fi and
		store the fi		in an OBS bucket on
		HUAWEI CLOUD. OMS migrates all
		objects		c fi	in the object list fi
		1. Select List File Bucket (HUAWEI
		CLOUD).
		NOTE
		The buckets listed are the buckets
		obtained by entering the destination AK
		and SK in Step 5.2.
		If the bucket used for storing object list
		fi	is not located in the same region
		as the destination bucket, you cannot
		select the object list fi
		2. Click Select next to Select Object
		List File, select the target object list
		fi	and click OK.
		NOTE
		● Each object list fi				cannot exceed 100
		MB.
		● The object list fi				must be a .txt fi
		and ContentType in the metadata of
		the fi		must be text/plain.
		● The object list fi				must be in UTF-8
		without BOM.
		● Each line in the object list fi					can
		contain only one object name, and the
		object name must be URL encoded.
		● Spaces are not allowed in each line in
		the object list fi				Spaces may cause
		migration failures because they may be
		mistakenly			n fi	as object names.
		● The length of each line in the object list
		fi	cannot exceed 65,535 characters, or
		the migration will fail.
		● ContentEncoding cannot be set in the
		metadata of the object list fi					or the
		migration will fail.
		● Archive data can only be accessed and
		obtained by OMS after it is restored. If
		the object list fi				are archive,
		manually restore them before
		migration.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide				2 Migration Tasks

	If You Select...		Then...

	Object Name	fix	Enter the names or	fix	of objects
			to be migrated and click Add. The
			system will automatically list the
			matched objects. You can click			to
			delete the objects that do not need to
			be migrated from the list.
			NOTE
			If no object names or name		fix	are
			c fi all objects in the source bucket
			will be migrated.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide				2 Migration Tasks

	If You Select...	Then...

	URL List	Generally, this setting is used to
		migrate HTTP/HTTPS source data. You
		need to write the URLs of the fi			to
		be migrated and their destination
		object names into a URL list fi			and
		store the fi		in an OBS bucket on
		HUAWEI CLOUD. OMS migrates all
		fi	c fi	in the URL list fi
		1. Select List File Bucket (HUAWEI
		CLOUD).
		NOTE
		– The buckets listed are the buckets
		obtained by entering the destination
		AK and SK in Step 5.2.
		– If the bucket used for storing object
		list fi		is not located in the same
		region as the destination bucket,
		you cannot select the object list
		fi
		– For security purposes, if the
		migration source supports both
		HTTP and HTTPS, you are advised
		to use HTTPS during the migration.
		If HTTP is used, data may be
		eavesdropped on, tampered with, or
		vulnerable to man-in-the-middle or
		replay attacks.
		– The URL list migration does not
		support URL redirection.
		2. Click Select next to the
		URL File area, select the target URL
		list fi	and click OK.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

If You Select...	Then...

	NOTE
	– Each URL list fi					cannot exceed 100
		MB.
	– The URL list fi					must be a .txt fi
		and ContentType in the metadata
		of the fi			must be text/plain.
	– The URL list fi					must be in UTF-8
		without BOM.
	– Each line in the URL list fi							can
		contain only one URL and one
		destination object.
	– The length of each line in the URL
		list fi		cannot exceed 65,535
		characters, or the migration will fail.
	– ContentEncoding cannot be set in
		the metadata of the URL list fi							or
		the migration will fail.
	– In the URL list fi					each line uses a
		tab character (\t) to separate the
		URL from the destination object
		name. The format is [URL][Tab
		character][Destination object
		name]. The destination object name
		is encoded using URL Encode. For
		example:
				xxx	b my	w	c	c m	c
		fi	1 x		doc%2f%e6%96%87%e4%bb
		%b61.txt
				xxx	b my	w	c	c m	c
			fi	2 x	c	2f	fi 2	x
				xxx	b my	w	c	c m
			20fi	x		20fi	x
				xxx	b my	w	c	c m
			20fi	2	x	fi 2	x
				xxx	b my	w	c	c m	c
			fi	x	c	fi	x
		Use a tab character (Tab key on the
		keyboard) to separate the URL from
		the destination object name. Do not
		use spaces.
		In the preceding examples, after the
		fi	represented by the URLs are
		copied to the destination bucket,
		the objects are named
					, the		, the
				, and				.
	– Spaces are not allowed in each line
		in the URL list fi				Spaces may
		cause migration failures because
		they may be mistakenly						n fi
		as object names.

Step 8 (Optional) Set Source Parameters.

Parameters described in Table 2-4 are only available if the source cloud service provider ff the related functions.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

Table 2-4 Source parameters

Parameter	Description

Use Source	If the default domain name cannot meet your migration
Custom Domain	requirements, then if the source cloud service provider
Name	supports custom domain names, you can use a source
	custom domain name to download data to reduce
	downloading fees.
	NOTE
	If the source cloud service provider is Qiniu Cloud, this option is
	selected by default to ensure a smooth migration because Qiniu
	Cloud has restrictions on default domain names.
	If you select this option, you must:
	1. Select a transmission protocol for Transfer Protocol.
	NOTE
	HTTPS is more secure than HTTP and is recommended.

2.Enter a custom domain name of the bucket for Custom Bucket Domain Name.

3.If you use the CDN authentication provided by the source cloud service provider, select Enable CDN

Authentication and c	nfig	the authentication type
and keys.
NOTE
– You can use CDN	n fic	n only after c nfig ng

CDN by referring to documents of the source cloud service provider.

–The fees generated by using CDN on the source platform are billed by the source cloud service provider.

–URI redirection is not supported.

4. Click Check to check whether the c nfig			n is
correct.
–	If the c nfig	n fails the check, click Details to
	view the failure causes.
–	If the c nfig	n passes the check, complete other
	migration c nfig	n

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide		2 Migration Tasks

	Parameter	Description

	Archive Data	Only restored archive data can be migrated. You can select
	Restoration	this option if your source cloud service provider supports
		automatic restoration of archive data.
		● If you select this option, the system automatically
		restores and migrates archive objects in the migration
		task. If an archive object fails to be restored, the system
		records it in the list of objects that failed to be migrated
		and continues to migrate other objects in the migration
		task.
		● If you do not select this option, the system directly
		records archive objects in the list of objects that failed to
		be migrated and continues to migrate other objects in
		the migration task.
		NOTE
		● Currently, the following cloud service providers can
		automatically restore archive data: HUAWEI CLOUD, Alibaba
		Cloud, Kingsoft Cloud, and Tencent Cloud.
		● It takes at least a minute to restore an archive object. The
		larger the archive object is, the longer it takes to restore it, but
		if an archive object cannot be restored within three minutes,
		the restoration fails. The time needed depends on the cloud
		service provider. For details, see related documents of the
		c fic cloud service provider.
		● Two types of fees are generated during archive data restoration:
		fees for API calls and fees for archive data retrieval. These fees
		are billed by the source cloud service provider.
		● After restoration, archive objects will be in restored state for a
		certain period of time. During this period, objects can be
		downloaded or accessed. After this period expires, archive
		objects need to be restored again for migration. For details
		about how long they stay in the restored state, see related
		documents from the cloud service provider.
		● After the object data is migrated from the source platform to
		HUAWEI CLOUD, the storage class of the migrated data must
		be the same as that of the destination bucket.

Step 9 Click Show next to Advanced Settings to expand the advanced parameters. Step 10 (Optional) Set Advanced Settings parameters.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide	2 Migration Tasks

Table 2-5 Advanced Settings parameters

Parameter	Description

Data Encryption	If you select KMS, migrated data is encrypted before
	being transferred to the destination OBS buckets.
	NOTE
	Using KMS to encrypt migrated data may decrease the
	migration speed by about 10%.
	This option is only available if KMS encryption is supported in
	the current region.

Failed Object Record	c fi	whether to record the objects that fail to be
	migrated.
	● If this parameter is set to No, the system will not
	record the objects that fail to be migrated. If the
	migration fails, all objects in the source bucket need
	to be transmitted again.
	● If this parameter is set to Yes, the system will record
	the objects that fail to be migrated. If the migration
	fails, only the objects that fail to be migrated need
	to transmitted again.
	NOTE
	The objects that fail to be migrated are recorded in an object
	list fi	named by the task name and uploaded to the oms/
	failed_object_lists directory of the destination bucket.
	This fi	contains information about the failed migration tasks,
	including the failure causes, occurrence time (local UTC), URL-
	encoded names of failed objects, non-URL-encoded names of
	failed objects, and error codes returned by source SDKs.
	This fi	is stored in the destination bucket after the objects in
	the fi	are successfully retransmitted. If you do not need the
	fi delete it manually.

Incremental Data	c fi	whether to fi	source objects to be
	migrated by time.
	● If this option is set to No, source objects are not
	fi	by time.
	● If this option is set to Yes, only the source objects
	that are m fi or added after the				c fi time
	are migrated. You need to click			to specify the
	time.

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide						2 Migration Tasks

	Parameter			Description

	SMN	fic	n	c fi	whether to use SMN to notify you of
				migration results.
				● If this parameter is set to No, no SMN message is
				sent after the migration.
				● If this parameter is set to Yes, you will receive an
				SMN message informing you of the migration
				results.
				Perform the following steps to set SMN parameters:
				1. Click Add next to the Select SMN Topic area.
				The Select SMN Topic dialog box is displayed.
				2. Select a topic for receiving SMN messages and
				click OK.
				If OMS is not authorized to publish messages to
				the selected SMN topic, a Warning dialog box
				will display after you click OK. If you need the
				authorization, click OK to authorize OMS.
				Otherwise, click Cancel to return to the Select
				SMN Topic page and select another SMN topic.
				3.	nfig	Trigger Condition. You can select one
				of the following options or both:

–Migration succeeded: An SMN message will be sent after the migration task succeeds.

–Migration failed: An SMN message will be sent after the migration task fails.

NOTE

If you experience trouble in using SMN, see What Can I

Do If the SMN Function Works Improperly?

Issue 06 (2021-03-24)

Object Storage Migration Service
User Guide					2 Migration Tasks

	Parameter	Description

	ffic Limit	Set the maximum bandwidth for the migration task
		during a	c fi period of time.
		● If you do not need to limit migration			ffic select
		No.
		● If you need to limit migration ffic			select Yes.
		Perform the following steps to set			ffic limits:
		1. Click Add Rule.
		A ffic limiting rule is displayed.
		2. Specify Start Time, End Time, and Maximum
		Bandwidth.
		The value range of the maximum bandwidth is 1
		Mbyte/s to 200 Mbyte/s.
		3. Repeat Step 10.a and Step 10.b to add another
			ffic limiting rule.
		You can click		in the Operation column to
		delete a rule.
		NOTE
		You can create a maximum of fiv rules.
		You can set		ffic limiting rules in the task details even
		after the task is created.

	Description	In the Description area, enter a maximum of 255
		characters to b fly describe the migration task.

Step 11 Click Next to enter the c		nfi m	n page.

Step 12 nfi m the settings of the migration task and start the migration.

1.Check whether the settings of migration parameters are correct.

If the settings are incorrect, click Previous to return to the previous page for m fic n

2.Click Migrate Now.

A message is displayed, indicating that the migration task is successfully created. The migration task is displayed in the migration task list.

For details about migration task statuses, see Viewing a Migration Task.

NOTE

After the task is created, a task ID is generated for you for quick n fic n It is displayed in the upper left corner of the task list. It is composed of the source bucket name, destination bucket name, and time sequence ID.

----End

Issue 06 (2021-03-24)

+ 43 hidden pages

HUAWEI Object Storage Migration Service User Manual

Contents

1 Permissions Management

1.1 Creating a User and Granting OMS Permissions

2 Migration Tasks

2.1 Creating a Migration Task