This document supports the version of each product listed and
supports all subsequent versions until the document is
replaced by a new edition. To check for more recent editions
of this document, see http://www.vmware.com/support/pubs.
EN-001772-01
Site Recovery Manager Administration
You can find the most up-to-date technical documentation on the VMware Web site at:
http://www.vmware.com/support/
The VMware Web site also provides the latest product updates.
If you have comments about this documentation, submit your feedback to:
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com
2 VMware, Inc.
Contents
About VMware Site Recovery Manager Administration7
Updated Information9
Site Recovery Manager Privileges, Roles, and Permissions11
1
How Site Recovery Manager Handles Permissions 12
Site Recovery Manager and the vCenter Server Administrator Role 13
Site Recovery Manager and vSphere Replication Roles 13
Managing Permissions in a Shared Recovery Site Configuration 14
Assign Site Recovery Manager Roles and Permissions 15
Site Recovery Manager Roles Reference 17
Replicating Virtual Machines23
2
Using Array-Based Replication with Site Recovery Manager 23
Configure Array-Based Replication 24
Using vSphere Replication with Site Recovery Manager 28
Replicating a Virtual Machine and Enabling Multiple Point in Time Instances 29
Using Array-Based Replication and vSphere Replication with Site Recovery Manager 30
Configuring Mappings33
3
Inventory Mappings for Array-Based Replication Protection Groups and vSphere Replication
Protection Groups 34
Inventory Mappings for Storage Policy Protection Groups 34
Configure Temporary Placeholder Mappings 35
Users Gain Access to Virtual Machines After Configuring Temporary Placeholder Mappings 37
Configure Inventory Mappings 37
About Storage Policy Mappings 39
Select Storage Policy Mappings 39
VMware, Inc.
About Placeholder Virtual Machines41
4
What Happens to Placeholder Virtual Machines During Recovery 42
Select a Placeholder Datastore 43
Creating and Managing Protection Groups45
5
About Array-Based Replication Protection Groups and Datastore Groups 46
How Site Recovery Manager Computes Datastore Groups 46
About vSphere Replication Protection Groups 47
About Storage Policy Protection Groups 48
Prerequisites for Storage Policy Protection Groups 49
Limitations of Storage Policy Protection Groups 50
Storage Policy Protection Groups and Periodic Polling 52
3
Site Recovery Manager Administration
Storage Policy Protection Groups and Nonprotected Virtual Machines 52
Create Protection Groups 53
Organize Protection Groups in Folders 55
Add or Remove Datastore Groups or Virtual Machines to or from a Protection Group 56
Apply Inventory Mappings to All Members of a Protection Group 57
Configure Inventory Mappings for an Individual Virtual Machine in a Protection Group 58
Modifying the Settings of a Protected Virtual Machine 59
Remove Protection from a Virtual Machine 60
Protection Group Status Reference 61
Virtual Machine Protection Status Reference 62
Creating, Testing, and Running Site Recovery Manager Recovery Plans65
6
Testing a Recovery Plan 66
Test Networks and Datacenter Networks 67
Performing a Planned Migration or Disaster Recovery By Running a Recovery Plan 67
Running a Recovery with Forced Recovery 68
Differences Between Testing and Running a Recovery Plan 69
Performing Test Recovery of Virtual Machines Across Multiple Hosts on the Recovery Site 70
Create, Test, and Run a Recovery Plan 70
Create a Recovery Plan 71
Organize Recovery Plans in Folders 71
Edit a Recovery Plan 72
Test a Recovery Plan 72
Clean Up After Testing a Recovery Plan 73
Run a Recovery Plan 74
Recover a Point-in-Time Snapshot of a Virtual Machine 75
Cancel a Test or Recovery 76
Export Recovery Plan Steps 76
View and Export a Recovery Plan History 77
Delete a Recovery Plan 77
Recovery Plan Status Reference 77
Configuring a Recovery Plan81
7
Recovery Plan Steps 81
Creating Custom Recovery Steps 82
Types of Custom Recovery Steps 83
How Site Recovery Manager Handles Custom Recovery Step Failures 84
Create Top-Level Message Prompts or Command Steps 84
Create Message Prompts or Command Steps for Individual Virtual Machines 85
Guidelines for Writing Command Steps 86
Environment Variables for Command Steps 86
Suspend Virtual Machines When a Recovery Plan Runs 87
Specify the Recovery Priority of a Virtual Machine 88
Configure Virtual Machine Dependencies 88
Enable vSphere vMotion for Planned Migration 89
Configure Virtual Machine Startup and Shutdown Options 90
Limitations to Protection and Recovery of Virtual Machines 91
4 VMware, Inc.
Customizing IP Properties for Virtual Machines93
8
Manually Customize IP Properties For an Individual Virtual Machine 94
Customizing IP Properties for Multiple Virtual Machines 95
Customizing IP Properties for Multiple Virtual Machines By Using the DR IP Customizer Tool 95
Customize IP Properties for Multiple Virtual Machines by Defining IP Customization Rules 109
Contents
Reprotecting Virtual Machines After a Recovery111
9
How Site Recovery Manager Reprotects Virtual Machines with Array Based Replication 112
How Site Recovery Manager Reprotects Virtual Machines with vSphere Replication 113
How Site Recovery Manager Reprotects Virtual Machines with Storage Policy Protection 113
Preconditions for Performing Reprotect 114
Reprotect Virtual Machines 115
Reprotect States 115
Restoring the Pre-Recovery Site Configuration By Performing Failback117
10
Perform a Failback 118
Interoperability of Site Recovery Manager with Other Software121
11
Site Recovery Manager and vCenter Server 122
Using Site Recovery Manager with VMware Virtual SAN Storage and vSphere Replication 122
How Site Recovery Manager Interacts with DPM and DRS During Recovery 123
How Site Recovery Manager Interacts with Storage DRS or Storage vMotion 123
Using Site Recovery Manager with Array-Based Replication on Sites with Storage DRS or
Storage vMotion 123
Using Site Recovery Manager with vSphere Replication on Sites with Storage DRS or Storage
vMotion 124
How Site Recovery Manager Interacts with vSphere High Availability 125
How Site Recovery Manager Interacts with Stretched Storage 125
Using Site Recovery Manager with VMware NSX 127
Site Recovery Manager and vSphere PowerCLI 127
Site Recovery Manager and vRealize Orchestrator 128
Automated Operations That the vRealize Orchestrator Plug-In for Site Recovery Manager
Provides 128
Protecting Microsoft Cluster Server and Fault Tolerant Virtual Machines 130
Using Site Recovery Manager with SIOC Datastores 131
Using Site Recovery Manager with Admission Control Clusters 131
Site Recovery Manager and Virtual Machines Attached to RDM Disk Devices 132
Site Recovery Manager and Active Directory Domain Controllers 132
Advanced Site Recovery Manager Configuration133
12
Reconfigure Site Recovery Manager Settings 133
Change Connections Settings 133
Change Site Recovery Manager History Report Collection Setting 134
Change Local Site Settings 134
Change Logging Settings 135
Change Recovery Settings 137
Change Remote Manager Settings 140
Change Remote Site Settings 141
VMware, Inc. 5
Site Recovery Manager Administration
Change Replication Settings 141
Change SSO Setting 142
Change Storage Settings 142
Change ABR Storage Policy Setting 143
Change Storage Provider Settings 144
Change vSphere Replication Settings 146
Modify Settings to Run Large Site Recovery Manager Environments 147
Settings for Large Site Recovery Manager Environments 148
Site Recovery Manager Events and Alarms151
13
How Site Recovery Manager Monitors Connections Between Sites 151
Configure Site Recovery Manager Alarms 152
Site Recovery Manager Events Reference 153
Collecting Site Recovery Manager Log Files163
14
Collect Site Recovery Manager Log Files By Using the Site Recovery Manager Interface 163
Collect Site Recovery Manager Log Files Manually 164
Change Size and Number of Site Recovery Manager Server Log Files 164
Configure Site Recovery Manager Core Dumps 166
Troubleshooting Site Recovery Manager169
15
Site Recovery Manager Doubles the Number of Backslashes in the Command Line When Running
Callouts 170
Powering on Many Virtual Machines Simultaneously on the Recovery Site Can Lead to Errors 171
LVM.enableResignature=1 Remains Set After a Site Recovery Manager Test Recovery 171
Adding Virtual Machines to a Protection Group Fails with an Unresolved Devices Error 172
Configuring Protection fails with Placeholder Creation Error 172
Rapid Deletion and Recreation of Placeholders Fails 173
Planned Migration Fails Because Host is in an Incorrect State 173
Recovery Fails with a Timeout Error During Network Customization for Some Virtual Machines 174
Recovery Fails with Unavailable Host and Datastore Error 174
Reprotect Fails with a vSphere Replication Timeout Error 175
Recovery Plan Times Out While Waiting for VMware Tools 175
Synchronization Fails for vSphere Replication Protection Groups 175
Rescanning Datastores Fails Because Storage Devices are Not Ready 176
Recovery Sticks at 36% During Planned Migration 177
Operations Fail with Error About a Nonreplicated Configuration File 177
Index179
6 VMware, Inc.
About VMware Site Recovery Manager
Administration
VMware Site Recovery Manager is an extension to VMware vCenter Server that delivers a business
continuity and disaster recovery solution that helps you plan, test, and run the recovery of vCenter Server
virtual machines. Site Recovery Manager can discover and manage replicated datastores, and automate
migration of inventory from one vCenter Server instance to another.
Intended Audience
This book is intended for Site Recovery Manager administrators who are familiar with vSphere and its
replication technologies, such as host-based replication and replicated datastores. This solution serves the
needs of administrators who want to configure protection for their vSphere inventory. It might also be
appropriate for users who need to add virtual machines to a protected inventory or to verify that an existing
inventory is properly configured for use with Site Recovery Manager.
VMware, Inc.
7
Site Recovery Manager Administration
8 VMware, Inc.
Updated Information
Site Recovery Manager Administration is updated with each release of the product or when necessary.
This table provides the update history of Site Recovery Manager Administration.
RevisionDescription
EN-001772-01
EN-001772-00 Initial release.
Revised and expanded the information in “Inventory Mappings for Storage Policy Protection
n
Groups,” on page 34.
Revised and expanded the information about storage policy protection groups throughout Chapter 5,
n
“Creating and Managing Protection Groups,” on page 45.
Added stretched storage information to “Recovery Plan Steps,” on page 81.
n
Added “Enable vSphere vMotion for Planned Migration,” on page 89.
n
Added “How Site Recovery Manager Reprotects Virtual Machines with Storage Policy Protection,”
n
on page 113.
Corrected the supported version of VMware Virtual SAN in “Using Site Recovery Manager with
n
VMware Virtual SAN Storage and vSphere Replication,” on page 122.
Added “How Site Recovery Manager Interacts with Stretched Storage,” on page 125.
n
Added “Using Site Recovery Manager with VMware NSX,” on page 127.
n
Revised and expanded “Recovery Plan Steps,” on page 81, adding subtopics about adding recovery
n
settings to VMs in a recovery plan and in a protection group.
Added “Operations Fail with Error About a Nonreplicated Configuration File,” on page 177.
n
VMware, Inc. 9
Site Recovery Manager Administration
10 VMware, Inc.
Site Recovery Manager Privileges,
Roles, and Permissions1
Site Recovery Manager provides disaster recovery by performing operations for users. These operations
involve managing objects, such as recovery plans or protection groups, and performing operations, such as
replicating or powering off virtual machines. Site Recovery Manager uses roles and permissions so that only
users with the correct roles and permissions can perform operations.
Site Recovery Manager adds several roles to vCenter Server, each of which includes privileges to complete
Site Recovery Manager and vCenter Server tasks. You assign roles to users to permit them to complete tasks
in Site Recovery Manager.
Privilege
Role
Permission
For information about the roles that Site Recovery Manager adds to vCenter Server and the privileges that
users require to complete tasks, see “Site Recovery Manager Roles Reference,” on page 17.
How Site Recovery Manager Handles Permissions on page 12
n
Site Recovery Manager determines whether a user has permission to perform an operation, such as
configuring protection or running the individual steps in a recovery plan. This permission check
ensures the correct authentication of the user, but it does not represent the security context in which
the operation is performed.
Site Recovery Manager and the vCenter Server Administrator Role on page 13
n
If a user or user group has the vCenter Server administrator role on a vCenter Server instance when
you install Site Recovery Manager, that user or user group obtains all Site Recovery Manager
privileges.
The right to perform an action, for example to create a recovery plan or to
modify a protection group.
A collection of privileges. Default roles provide the privileges that certain
users require to perform a set of Site Recovery Manager tasks, for example
users who manage protection groups or perform recoveries. A user can have
at most one role on an object, but roles can be combined if the user belongs to
multiple groups that all have roles on the object.
A role granted to a particular user or user group on a specific object. A user
or user group is also known as a principal. A permission is a combination of
a role, an object, and a principal. For example, a permission is the privilege to
modify a specific protection group.
VMware, Inc.
Site Recovery Manager and vSphere Replication Roles on page 13
n
When you install vSphere Replication with Site Recovery Manager, the vCenter Server administrator
role inherits all of the Site Recovery Manager and vSphere Replication privileges.
11
Site Recovery Manager Administration
Managing Permissions in a Shared Recovery Site Configuration on page 14
n
You can configure permissions on Site Recovery Manager to use a shared recovery site. The
vCenter Server administrator on the shared recovery site must manage permissions so that each user
has sufficient privileges to configure and use Site Recovery Manager, but no user has access to
resources that belong to another user.
Assign Site Recovery Manager Roles and Permissions on page 15
n
During installation of Site Recovery Manager, users with the vCenter Server administrator role are
granted the administrator role on Site Recovery Manager. At this time, only vCenter Server
administrators can log in to Site Recovery Manager, unless they explicitly grant access to other users.
Site Recovery Manager Roles Reference on page 17
n
Site Recovery Manager includes a set of roles. Each role includes a set of privileges, which allow users
with those roles to complete different actions.
How Site Recovery Manager Handles Permissions
Site Recovery Manager determines whether a user has permission to perform an operation, such as
configuring protection or running the individual steps in a recovery plan. This permission check ensures the
correct authentication of the user, but it does not represent the security context in which the operation is
performed.
Site Recovery Manager performs operations in the security context of the user ID that is used to connect the
sites, or in the context of the ID under which the Site Recovery Manager service is running, for example, the
local system ID.
After Site Recovery Manager verifies that a user has the appropriate permissions on the target vSphere
resources, Site Recovery Manager performs operations on behalf of users by using the vSphere
administrator role.
For operations that configure protection on virtual machines, Site Recovery Manager validates the user
permissions when the user requests the operation. Operations require two phases of validation.
1During configuration, Site Recovery Manager verifies that the user configuring the system has the
correct permissions to complete the configuration on the vCenter Server object. For example, a user
must have permission to protect a virtual machine and use resources on the secondary vCenter Server
instance that the recovered virtual machine uses.
2The user performing the configuration must have the correct permissions to complete the task that they
are configuring. For example, a user must have permissions to run a recovery plan.
Site Recovery Manager then completes the task on behalf of the user as a vCenter Server administrator.
As a result, a user who completes a particular task, such as a recovery, does not necessarily require
permissions to act on vSphere resources. The user only requires the permission to run a recovery in
Site Recovery Manager. The role authorizes the action, but the action is performed by
Site Recovery Manager acting as an administrator. Site Recovery Manager performs the operations by using
the administrator credentials that you provide when you connect the protected and recovery sites.
Site Recovery Manager maintains a database of permissions for internal Site Recovery Manager objects that
uses a model similar to the one the vCenter Server uses. Site Recovery Manager verifies its own
Site Recovery Manager privileges even on vCenter Server objects. For example, Site Recovery Manager
checks for the Resource.Recovery Use permission on the target datastore rather than checking multiple lowlevel permissions, such as Allocate space. Site Recovery Manager also verifies the permissions on the
remote vCenter Server instance.
To use Site Recovery Manager with vSphere Replication, you must assign vSphere Replication roles to users
as well as Site Recovery Manager roles. For information about vSphere Replication roles, see
vSphere Replication Administration.
12 VMware, Inc.
Chapter 1 Site Recovery Manager Privileges, Roles, and Permissions
Site Recovery Manager and the vCenter Server Administrator Role
If a user or user group has the vCenter Server administrator role on a vCenter Server instance when you
install Site Recovery Manager, that user or user group obtains all Site Recovery Manager privileges.
If you assign the vCenter Server administrator role to users or user groups after you install
Site Recovery Manager, you must manually assign the Site Recovery Manager roles to those users on
Site Recovery Manager objects.
You can assign Site Recovery Manager roles to users or user groups that do not have the vCenter Server
administrator role. In this case, those users have permission to perform Site Recovery Manager operations,
but they do not have permission to perform all vCenter Server operations.
Site Recovery Manager and vSphere Replication Roles
When you install vSphere Replication with Site Recovery Manager, the vCenter Server administrator role
inherits all of the Site Recovery Manager and vSphere Replication privileges.
If you manually assign a Site Recovery Manager role to a user or user group, or if you assign a
Site Recovery Manager role to a user or user group that is not a vCenter Server administrator, these users do
not obtain vSphere Replication privileges. The Site Recovery Manager roles do not include the privileges of
the vSphere Replication roles. For example, the Site Recovery Manager Recovery Administrator role
includes the privilege to run recovery plans, including recovery plans that contain vSphere Replication
protection groups, but it does not include the privilege to configure vSphere Replication on a virtual
machine. The separation of the Site Recovery Manager and vSphere Replication roles allows you to
distribute responsibilities between different users. For example, one user with the VRM administrator role is
responsible for configuring vSphere Replication on virtual machines, and another user with the
Site Recovery Manager Recovery Administrator role is responsible for running recoveries.
In some cases, a user who is not vCenter Server administrator might require the privileges to perform both
Site Recovery Manager and vSphere Replication operations. To assign a combination of
Site Recovery Manager and vSphere Replication roles to a single user, you can add the user to two user
groups.
Example: Assign Site Recovery Manager and vSphere Replication Roles to a
User
By creating two user groups, you can grant to a user the privileges of both a Site Recovery Manager role and
a vSphere Replication role, without that user being a vCenter Server administrator.
1Create two user groups.
2Assign a Site Recovery Manager role to one user group, for example Site Recovery Manager
administrator.
3Assign a vSphere Replication role to the other user group, for example VRM administrator.
4Add the user to both user groups.
The user has all the privileges of the Site Recovery Manager administrator role and of the VRM
administrator role.
VMware, Inc. 13
Site Recovery Manager Administration
Managing Permissions in a Shared Recovery Site Configuration
You can configure permissions on Site Recovery Manager to use a shared recovery site. The vCenter Server
administrator on the shared recovery site must manage permissions so that each user has sufficient
privileges to configure and use Site Recovery Manager, but no user has access to resources that belong to
another user.
In the context of a shared recovery site, a user is the owner of a pair of Site Recovery Manager Server
instances. Users with adequate permissions must be able to access the shared recovery site to create, test,
and run the recovery plans for their own protected site. The vCenter Server administrator at the shared
recovery site must create a separate user group for each user. No user's user accounts can be a member of
the vCenter Server Administrators group. The only supported configuration for a shared recovery site is for
one organization to manage all of the protected sites and the recovery site.
CAUTION Certain Site Recovery Manager roles allow users to run commands on
Site Recovery Manager Server, so you should assign these roles to trusted administrator-level users only.
See “Site Recovery Manager Roles Reference,” on page 17 for the list of Site Recovery Manager roles that
run commands on Site Recovery Manager Server.
On a shared recovery site, multiple customers share a single vCenter Server instance. In some cases,
multiple customers can share a single ESXi host on the recovery site. You can map the resources on the
protected sites to shared resources on the shared recovery site. You might share resources on the recovery
site if you do not need to keep all of the customers' virtual machines separate, for example if all of the
customers belong to the same organization.
You can also create isolated resources on the shared recovery site and map the resources on the protected
sites to their own dedicated resources on the shared recovery site. You might use this configuration if you
must keep all of the customers' virtual machines separate from each other, for example if all of the
customers belong to different organizations.
Guidelines for Sharing User Resources
Follow these guidelines when you configure permissions for sharing user resources on the shared recovery
site:
All users must have read access to all folders of the vCenter Server on the shared recovery site.
n
Do not give a user the permission to rename, move, or delete the datacenter or host.
n
Do not give a user the permission to create virtual machines outside of the user’s dedicated folders and
n
resource pools.
Do not allow a user to change roles or assign permissions for objects that are not dedicated to the user’s
n
own use.
To prevent unwanted propagation of permissions across different organizations’ resources, do not
n
propagate permissions on the root folder, datacenters, and hosts of the vCenter Server on the shared
recovery site.
Guidelines for Isolating User Resources
Follow these guidelines when you configure permissions for isolating user resources on the shared recovery
site:
Assign to each user a separate virtual machine folder in the vCenter Server inventory.
n
Set permissions on this folder to prevent any other user from placing their virtual machines in it.
n
For example, set the Administrator role and activate the propagate option for a user on that user's
folder. This configuration prevents duplicate name errors that might otherwise occur if multiple
users protect virtual machines that have identical names.
14 VMware, Inc.
Chapter 1 Site Recovery Manager Privileges, Roles, and Permissions
Place all of the user's placeholder virtual machines in this folder, so that they can inherit its
n
permissions.
Do not assign permissions to access this folder to other users.
n
Assign dedicated resource pools, datastores, and networks to each user, and configure the permissions
n
in the same way as for folders.
CAUTION A deployment in which you isolate user resources still assumes trust between the vSphere sites.
Even though you can isolate user resources, you cannot isolate the users themselves. This is not a suitable
deployment if you must keep all users completely separate.
Viewing Tasks and Events in a Shared Recovery Site Configuration
In the Recent Tasks panel of the vSphere Client, users who have permissions to view an object can see tasks
that other users start on that object. All users can see all of the tasks that other users perform on a shared
resource. For example, all users can see the tasks that run on a shared host, datacenter, or the vCenter Server
root folder.
Events that all of the instances of Site Recovery Manager Server generate on a shared recovery site have
identical permissions. All users who can see events from one instance of Site Recovery Manager Server can
see events from all Site Recovery Manager Server instances that are running on the shared recovery site.
Assign Site Recovery Manager Roles and Permissions
During installation of Site Recovery Manager, users with the vCenter Server administrator role are granted
the administrator role on Site Recovery Manager. At this time, only vCenter Server administrators can log in
to Site Recovery Manager, unless they explicitly grant access to other users.
To allow other users to access Site Recovery Manager, vCenter Server administrators must grant them
permissions in the Site Recovery Manager interface in the vSphere Web Client. You assign site-wide
permission assignments on a per-site basis. You must add corresponding permissions on both sites.
Site Recovery Manager requires permissions on vCenter Server objects as well as on Site Recovery Manager
objects. To configure permissions on the remote vCenter Server installation, start another instance of the
vSphere Web Client. You can change Site Recovery Manager permissions from the same
vSphere Web Client instance on both sites after you connect the protected and recovery sites.
Site Recovery Manager augments vCenter Server roles and permissions with additional permissions that
allow detailed control over Site Recovery Manager specific tasks and operations. For information about the
permissions that each Site Recovery Manager role includes, see “Site Recovery Manager Roles Reference,”
on page 17.
You can assign more granular permissions to users by assigning them permissions on specific
Site Recovery Manager objects, including individual array managers, protection groups, and recovery plans.
You can also allow users to access specific groups of protections groups, recovery plans, and array managers
by assigning permissions to protection group and recovery plan folders, and to all of the array managers for
a site.
Procedure
1In the vSphere Web Client, select the objects on which to assign permissions.
OptionDescription
Assign site-wide permissions
Assign permissions to an individual
protection group
VMware, Inc. 15
Click Site Recovery > Sites, and select a site.
Click Site Recovery, expand Inventories, click Protection Groups, and
select a protection group.
Site Recovery Manager Administration
OptionDescription
Assign permissions to a protection
group folder
Assign permissions to an individual
recovery plan
Assign permissions to a recovery
plan folder
Assign permissions to an individual
array manager
Assign permissions to all array
managers for a site
2In the Manage tab, click Permissions, then click the Add Permission icon.
3Identify a user or group for the role.
aClick Add in the Users and Groups column.
bFrom the Domain drop-down menu, select the domain that contains the user or group.
cEnter a user or user group name in the Search text box or select a name from the User/Group list.
Click Site Recovery, expand Inventory Trees, click Protection Groups and
select a protection group folder. You can assign permissions to the root
folder or to a subfolder.
Click Site Recovery, expand Inventories, click Recovery Plans, and select
a recovery plan.
Click Site Recovery, expand Inventory Trees, click Recovery Plans and
select a recovery plan folder. You can assign permissions to the root folder
or to a subfolder.
Click Site Recovery > Array Based Replication, and select an array
manager.
Click Site Recovery, expand Inventory Trees, click Array BasedReplication and select a site folder.
dClick Add and click OK.
4Select a role from the Assigned Role drop-down menu to assign to the user or user group that you
selected in Step 3.
The Assigned Role drop-down menu includes all of the roles that vCenter Server and its plug-ins make
available. Site Recovery Manager adds several roles to vCenter Server.
OptionAction
Allow a user or user group to
perform all Site Recovery Manager
configuration and administration
operations.
Allow a user or user group to
manage and modify protection
groups and to configure protection
on virtual machines.
Allow a user or user group to
perform recoveries and test
recoveries.
Allow a user or user group to
create, modify, and test recovery
plans.
Allow a user or user group to test
recovery plans.
Assign the SRM Administrator role.
Assign the SRM Protection Groups Administrator role.
Assign the SRM Recovery Administrator role.
Assign the SRM Recovery Plans Administrator role.
Assign the SRM Recovery Test Administrator role.
When you select a role, the hierarchical list displays the privileges that the role includes. Click a
privilege in the hierarchical list to see a description of that privilege. You cannot modify the list of
privileges that each role includes.
16 VMware, Inc.
Chapter 1 Site Recovery Manager Privileges, Roles, and Permissions
5Select Propagate to Children to apply the selected role to all of the child objects of the inventory objects
that this role can affect.
For example, if a role contains privileges to modify folders, selecting this option extends the privileges
to all the virtual machines in a folder. You might deselect this option to create a more complex
hierarchy of permissions. For example, deselect this option to override the permissions that are
propagated from the root of a certain node from the hierarchy tree, but without overriding the
permissions of the child objects of that node.
6Click OK to assign the role and its associated privileges to the user or user group.
7Repeat Step 2 through Step 6 to assign roles and privileges to the users or user groups on the other
Site Recovery Manager site.
You assigned a given Site Recovery Manager role to a user or user group. This user or user group has
privileges to perform the actions that the role defines on the objects on the Site Recovery Manager site that
you configured.
Example: Combining Site Recovery Manager Roles
You can assign only one role to a user or user group. If a user who is not a vCenter Server administrator
requires the privileges of more than one Site Recovery Manager role, you can create multiple user groups.
For example, a user might require the privileges to manage recovery plans and to run recovery plans.
1Create two user groups.
2Assign the SRM Recovery Plans Administrator role to one group.
3Assign the SRM Recovery Administrator role to the other group.
4Add the user to both user groups.
By being a member of groups that have both the SRM Recovery Plans Administrator and the SRMRecovery Administrator roles, the user can manage recovery plans and run recoveries.
Site Recovery Manager Roles Reference
Site Recovery Manager includes a set of roles. Each role includes a set of privileges, which allow users with
those roles to complete different actions.
Roles can have overlapping sets of privileges and actions. For example, the Site Recovery Manager
Administrator role and the Site Recovery Manager Protection Groups Administrator have the Create
privilege for protection groups. With this privilege, the user can complete one aspect of the set of tasks that
make up the management of protection groups.
Assign roles to users on Site Recovery Manager objects consistently on both sites, so that protected and
recovery objects have identical permissions.
All users must have at least the System.Read privilege on the root folders of vCenter Server and the
Site Recovery Manager root nodes on both sites.
NOTE If you uninstall Site Recovery Manager Server, Site Recovery Manager removes the default
Site Recovery Manager roles but the Site Recovery Manager privileges remain. You can still see and assign
Site Recovery Manager privileges on other roles after uninstalling Site Recovery Manager. This is standard
vCenter Server behavior. Privileges are not removed when you unregister an extension from vCenter Server.
VMware, Inc. 17
Site Recovery Manager Administration
Table 1‑1. Site Recovery Manager Roles
Role
Site Recovery
Manager
Administrator
Site Recovery
Manager Protection
Groups
Administrator
Actions that this
Role PermitsPrivileges that this Role Includes
The
Site Recovery
Manager
Administrator grants
permission to perform
all
Site Recovery
Manager
configuration and
administration
operations.
Configure
n
advanced settings.
Configure
n
connections.
Configure
n
inventory
preferences.
Configure
n
placeholder
datastores.
Configure array
n
managers.
Manage
n
protection groups.
Manage recovery
n
plans.
Perform reprotect
n
operations.
Configure
n
protection on
virtual machines.
Edit protection
n
Site Recovery Manager.Advanced Settings.Modify
Site Recovery Manager.Array Manager.Configure
Site Recovery Manager.Diagnostics.Export
Site Recovery Manager.Inventory Preferences.Modify
Site Recovery Manager.Placeholder
Datastores.Configure
Site Recovery Manager.Protection Group.Assign to
Plan
Site Recovery Manager.Protection Group.Create
Site Recovery Manager.Protection Group.Modify
Site Recovery Manager.Protection Group.Remove
Site Recovery Manager.Protection Group.Remove from
Plan
Site Recovery Manager.Recovery History .View
Deleted Plans
Site Recovery Manager.Recovery Plan.Configure
Site Recovery Manager.Recovery Plan.Create
Site Recovery Manager.Recovery Plan.Modify
Site Recovery Manager.Recovery Plan.Remove
Site Recovery Manager.Recovery Plan.Reprotect
Site Recovery Manager.Recovery Plan.Test
Site Recovery Manager.Remote Site.Modify
Datastore.Replication.Protect
Datastore.Replication.Unprotect.Stop
Resource.Recovery Use
Virtual Machine. SRM Protection.Protect
Virtual Machine. SRM Protection.Stop
Site Recovery Manager.Profile-driven storage.Profiledriven storage view
groups.
Remove
n
protection groups.
View storage
n
policy objects.
Users with this role
cannot run recoveries.
Only users with the
Site Recovery
Manager Recovery
Administrator role can
perform recoveries.
The
Site Recovery
Manager Protection
Groups Administrator
role allows users to
manage protection
groups.
Site Recovery Manager.Protection Group.Create
Site Recovery Manager.Protection Group.Modify
Site Recovery Manager.Protection Group.Remove
Datastore.Replication.Protect
Datastore.Replication.Unprotect.Stop
Resource.Recovery Use
Objects in
vCenter
Server
Inventory
that this
Role Can
Access
Virtual
n
machin
es
Datasto
n
res
vCenter
n
Server
folders
Resourc
n
e pools
Site
n
Recover
y
Manage
r
service
instance
s
Networ
n
ks
Site
n
Recover
y
Manage
r
folders
Protecti
n
on
groups
Recover
n
y plans
Array
n
manage
rs
Site
n
Recover
y
Manage
r
folders
18 VMware, Inc.
Table 1‑1. Site Recovery Manager Roles (Continued)
Actions that this
Role
Site Recovery
Manager Recovery
Administrator
Role PermitsPrivileges that this Role Includes
Create protection
n
groups.
Modify protection
n
Virtual Machine. SRM Protection.Protect
Virtual Machine. SRM Protection.Stop
groups.
Add virtual
n
machines to
protection groups.
Delete protection
n
groups.
Configure
n
protection on
virtual machines.
Remove
n
protection from
virtual machines.
Users with this role
cannot perform or test
recoveries or create or
modify recovery
plans.
The
Site Recovery
Manager Recovery
Administrator role
allows users to
perform recoveries
and reprotect
operations.
Remove
n
protection groups
from recovery
Site Recovery Manager.Protection Group.Remove from
plan
Site Recovery Manager.Recovery Plan.Modify
Site Recovery Manager.Recovery Plan.Test
Site Recovery Manager.Recovery Plan.Recovery
Site Recovery Manager.Recovery Plan.Reprotect
Site Recovery Manager.Recovery
Plan.Configure.Configure commands
Site Recovery Manager.Recovery History.View deleted
plans
plans.
Test recovery
n
plans.
Run recovery
n
plans.
Run reprotect
n
operations.
Configure custom
n
command steps
on virtual
machines.
View deleted
n
recovery plans.
Edit virtual
n
machine recovery
properties.
Users with this role
cannot configure
protection on virtual
machines, or create or
modify recovery
plans.
Chapter 1 Site Recovery Manager Privileges, Roles, and Permissions
Objects in
vCenter
Server
Inventory
that this
Role Can
Access
Protecti
n
on
groups
Protecti
n
on
groups
Recover
n
y plans
Site
n
Recover
y
Manage
r
service
instance
s
VMware, Inc. 19
Site Recovery Manager Administration
Table 1‑1. Site Recovery Manager Roles (Continued)
Role
Site Recovery
Manager Recovery
Plans Administrator
Site Recovery
Manager Test
Administrator
Actions that this
Role PermitsPrivileges that this Role Includes
The
Site Recovery
Manager Recovery
Plans Administrator
role allows users to
create and test
recovery plans.
Add protection
n
groups to
recovery plans.
Remove
n
protection groups
Site Recovery Manager.Protection Group.Assign to
plan
Site Recovery Manager.Protection Group.Remove from
plan
Site Recovery Manager.Recovery Plan.Configure
Commands
Site Recovery Manager.Recovery Plan.Create
Site Recovery Manager.Recovery Plan.Modify
Site Recovery Manager.Recovery Plan.Remove
Site Recovery Manager.Recovery Plan.Test
Resource.Recovery Use
from recovery
plans.
Configure custom
n
command steps
on virtual
machines.
Create recovery
n
plans.
Test recovery
n
plans.
Cancel recovery
n
plan tests.
Edit virtual
n
machine recovery
properties.
Users with this role
cannot configure
protection on virtual
machines, or perform
recoveries or reprotect
operations.
The
Site Recovery
Site Recovery Manager.Recovery Plan.Modify
Site Recovery Manager.Recovery Plan.Test
Manager Test
Administrator role
only allows users to
test recovery plans.
Test recovery
n
plans.
Cancel recovery
n
plan tests.
Edit virtual
n
machine recovery
properties.
Users with this role
cannot configure
protection on virtual
machines, create
Objects in
vCenter
Server
Inventory
that this
Role Can
Access
Protecti
n
on
groups
Recover
n
y plans
vCenter
n
Server
folders
Datasto
n
res
Resourc
n
e pools
Networ
n
ks
Recovery
plans
20 VMware, Inc.
Table 1‑1. Site Recovery Manager Roles (Continued)
Actions that this
Role
Role PermitsPrivileges that this Role Includes
protection groups or
recovery plans, or
perform recoveries or
reprotect operations.
Chapter 1 Site Recovery Manager Privileges, Roles, and Permissions
Objects in
vCenter
Server
Inventory
that this
Role Can
Access
VMware, Inc. 21
Site Recovery Manager Administration
22 VMware, Inc.
Replicating Virtual Machines2
Before you create protection groups, you must configure replication on the virtual machines to protect.
You can replicate virtual machines by using either array-based replication, vSphere Replication, or a
combination of both.
This chapter includes the following topics:
“Using Array-Based Replication with Site Recovery Manager,” on page 23
n
“Using vSphere Replication with Site Recovery Manager,” on page 28
n
“Using Array-Based Replication and vSphere Replication with Site Recovery Manager,” on page 30
n
Using Array-Based Replication with Site Recovery Manager
When you use array-based replication, one or more storage arrays at the protected site replicate data to peer
arrays at the recovery site. With storage replication adapters (SRAs), you can integrate
Site Recovery Manager with a wide variety of arrays.
To use array-based replication with Site Recovery Manager, you must configure replication first before you
can configure Site Recovery Manager to use it.
If your storage array supports consistency groups, Site Recovery Manager is compatible with vSphere
Storage DRS and vSphere Storage vMotion. You can use Storage DRS and Storage vMotion to move virtual
machine files within a consistency group that Site Recovery Manager protects. If your storage array does not
support consistency groups, you cannot use Storage DRS and Storage vMotion in combination with
Site Recovery Manager.
You can protect virtual machines that contain disks that use VMware vSphere Flash Read Cache storage.
Since the host to which a virtual machine recovers might not be configured for Flash Read Cache,
Site Recovery Manager disables Flash Read Cache on disks when it starts the virtual machines on the
recovery site. Site Recovery Manager sets the reservation to zero. Before performing a recovery on a virtual
machine that is configured to use vSphere Flash Read Cache, take a note of virtual machine's cache
reservation from the vSphere Web Client. After the recovery, you can migrate the virtual machine to a host
with Flash Read Cache storage and manually restore the original Flash Read Cache setting on the virtual
machine.
If you protect virtual machines by using storage policy protection groups, you must replicate those virtual
machines by using array-based replication.
Storage Replication Adapters
Storage replication adapters are not part of a Site Recovery Manager release. Your array vendor develops
and supports them. You must install an SRA specific to each array that you use with Site Recovery Manager
on the Site Recovery Manager Server host. Site Recovery Manager supports the use of multiple SRAs.
VMware, Inc.
23
SRM plug-in
vSphere Web Client
Protected Site
Recovery Site
ESXi
Server
ESXi
Server
vCenter Server
VMFS
VMFS
Storage
VMFS
VMFS
Storage
ESXi
Server
SRM ServerSRM Server
SRA
ESXi
Server
ESXi
Server
vCenter Server
SRA
Array based replication
SRM plug-in
vSphere Web Client
Site Recovery Manager Administration
Figure 2‑1. Site Recovery Manager Architecture with Array-Based Replication
Configure Array-Based Replication
To protect virtual machines that you replicate by using array-based replication, including virtual machines
that you protect by using storage policy protection groups, you must configure storage replication adapters
(SRAs) at each site.
Install Storage Replication Adapters
If you protect virtual machines by using array-based replication or by using array-based replication with
storage policy protection, you must install a Storage Replication Adapter (SRA) specific to each storage
array that you use with Site Recovery Manager. An SRA is a program that an array vendor provides that
enables Site Recovery Manager to work with a specific kind of array.
You must install an appropriate SRA on the Site Recovery Manager Server hosts at the protected and
recovery sites. If you use more than one type of storage array, you must install the SRA for each type of
array on both of the Site Recovery Manager Server hosts.
NOTE You can configure Site Recovery Manager to use more than one type of storage array, but you cannot
store the virtual machine disks for a single virtual machine on multiple arrays from different vendors. You
must store all of the disks for a virtual machine on the same array.
Storage replication adapters come with their own installation instructions. You must install the version of an
SRA that corresponds to a specific Site Recovery Manager version. Install the same version of the SRA at
both sites. Do not mix SRA versions.
If you are using vSphere Replication, you do not require an SRA.
Prerequisites
Check the availability of an SRA for your type of storage by consulting the VMware Compatibility Guide
Read the documentation provided with your SRA. SRAs do not support all features that storage arrays
n
support. The documentation that your SRA provides details what the SRA supports and requires. For
example, HP and EMC have detailed physical requirements which must be met for the SRA to perform
as expected.
Install Site Recovery Manager Server before you install the SRAs.
n
Your SRA might require the installation of other vendor-provided components. You might need to
n
install some of these components on the Site Recovery Manager Server host. Other components might
require only network access by the Site Recovery Manager Server. For the latest information on such
requirements, review the release notes and readme files for the SRAs you are installing.
Enable the storage array's capability to create snapshot copies of the replicated devices. See your SRA
n
documentation.
Procedure
1Install the SRA on each Site Recovery Manager Server host.
The installer installs the SRA in C:\Program Files\VMware\VMware vCenter Site Recovery
Manager\storage\sra.
2In the vSphere Web Client, go to Site Recovery > Sites, and select a site.
3In the Monitor tab, click SRAs, and click the Rescan SRAs button.
This action refreshes SRA information, allowing Site Recovery Manager to discover the SRAs.
Configure Array Managers
After you pair the protected site and recovery site, configure their respective array managers so that
Site Recovery Manager can discover replicated devices, compute datastore groups, and initiate storage
operations.
You typically configure array managers only once after you connect the sites. You do not need to
reconfigure them unless array manager connection information or credentials change, or you want to use a
different set of arrays.
Prerequisites
Connect the sites as described in Connect the Protected and Recovery Sites in Site Recovery Manager
n
Installation and Configuration.
Install SRAs at both sites as described in “Install Storage Replication Adapters,” on page 24.
n
Procedure
1In the vSphere Web Client, click Site Recovery > Array Based Replication.
2In the Objects tab, click the icon to add an array manager.
3Select from two options:
Add a pair of array managers
n
Add a single array manager
n
VMware, Inc. 25
Site Recovery Manager Administration
4Select a site or pair of sites for the array manager and click Next.
5Select the array manager type that you want Site Recovery Manager to use from the SRA Type table
and click Next.
If no manager type appears, rescan for SRAs or check that you have installed an SRA on the
Site Recovery Manager Server host.
6Enter a name for the array in the Display Name text box and click Next..
Use a descriptive name that makes it easy for you to identify the storage associated with this array
manager.
7Provide the required information for the type of SRA you selected.
For more information about how to fill in these text boxes, see the documentation that your SRA vendor
provides. Text boxes vary between SRAs, but common text boxes include IP address, protocol
information, mapping between array names and IP addresses, and user name and password.
8If you chose to add a pair of array managers, on the Enable array pairs page, select the the array pair to
enable, then click Next.
You can also configure array pairs in the single option mode if the array manager on the peer site is
already created.
9Review the configuration and click Finish.
10 Repeat steps to configure an array manager for the recovery site, if necessary.
Rescan Arrays to Detect Configuration Changes
By default, Site Recovery Manager checks arrays for changes to device configurations by rescanning arrays
every 24 hours. However, you can force an array rescan at any time.
You can reconfgure the frequency with which Site Recovery Manager performs regular array scans by
changing the storage.minDsGroupComputationInterval option in Advanced Settings. See Change Storage
Settings.
Configuring array managers causes Site Recovery Manager to compute datastore groups based on the set of
replicated storage devices that it discovers. If you change the configuration of the array at either site to add
or remove devices, Site Recovery Manager must rescan the arrays and recompute the datastore groups.
Procedure
1In the vSphere Web Client, click Site Recovery > Array Based Replication.
2Select an array.
3In the Manage tab, select Array Pairs.
The Array Pairs tab provides information about all the storage devices in the array, including the local
device name, the device it is paired with, the direction of replication, the protection group to which the
device belongs, whether the datastore is local or remote, and the consistency group ID for each SRA
device.
4Right-click an array pair and select Discover Devices to rescan the arrays and recompute the datastore
groups.
26 VMware, Inc.
Chapter 2 Replicating Virtual Machines
Edit Array Managers
Use the Edit Array Manager wizard to modify an array manager's name or other settings, such as the IP
address or user name and password.
For more information about how to fill in the adapter fields, see the documentation that your SRA vendor
provides. While fields vary among SRAs, common fields include IP address, protocol information, mapping
between array names and IP addresses, and user names and passwords.
Procedure
1In the vSphere Web Client, click Site Recovery > Array Based Replication.
2Right-click an array and select Edit Array Manager.
3Modify the name for the array in the Display Name field.
Use a descriptive name that makes it easy for you to identify the storage associated with this array
manager. You cannot modify the array manager type.
4Modify the adapter information.
These fields are created by the SRA.
5Enable the array pair and click Next.
6Click Finish to complete the modification of the array manager.
Specify an Unreplicated Datastore for Swap Files
Every virtual machine requires a swap file. By default, vCenter Server creates swap files in the same
datastore as the other virtual machine files. To prevent Site Recovery Manager from replicating swap files,
you can configure virtual machines to create them in an unreplicated datastore.
Under normal circumstances, you should keep the swap files in the same datastore as other virtual machine
files. However, you might need to prevent replication of swap files to avoid excessive consumption of
network bandwidth. Some storage vendors recommend that you do not replicate swap files. Only prevent
replication of swap files if it is absolutely necessary.
NOTE If you are using an unreplicated datastore for swap files, you must create an unreplicated datastore
for all protected hosts and clusters at both the protected and recovery sites. The unreplicated datastore must
be visible to all hosts in a cluster, otherwise vMotion will not work.
Procedure
1In the vSphere Web Client, select Hosts and Clusters, select a host, and click Manage > Settings.
2Under Virtual Machines, select Swap file location, and click Edit.
3Select Use a specific datastore, and select an unreplicated datastore.
4Click OK.
5Power off and power on all of the virtual machines on the host.
Resetting the guest operating system is not sufficient. The change of swapfile location takes effect after
you power off then power on the virtual machines.
6Browse the datastore that you selected for swapfiles and verify that VSWP files are present for the
virtual machines.
VMware, Inc. 27
Site Recovery Manager Administration
Isolating Devices For Stretched Storage During Disaster Recovery
In a disaster recovery with stretched storage, the failover command must isolate devices at the recovery site.
If some hosts at the protected site are still operational and continue running virtual machines when you
initiate a disaster recovery, Site Recovery Manager cannot power on the corresponding virtual machines at
the recovery site due to file locks. If the storage array isolates the devices at the recovery site, the ESX hosts
at the recovery site can break the necessary locks and power on the virtual machines.
Site Recovery Manager must use isolation="true" in the failover SRA command for the stretched devices
that were not deactivated at the protected site.
Implementation details of isolation for stretched storage is specific to array vendors. Some array vendors
might make the devices inaccessible at the protected site after running the failover SRA command with
isolation. Some array vendors might break the communication between source and target site for that
particular device.
Using vSphere Replication with Site Recovery Manager
Site Recovery Manager can use vSphere Replication to replicate data to servers at the recovery site.
You deploy the vSphere Replication appliance and configure vSphere Replication on virtual machines
independently of Site Recovery Manager. See the vSphere Replication documentation at
https://www.vmware.com/support/pubs/vsphere-replication-pubs.html for information about deploying
and configuring vSphere Replication.
vSphere Replication does not require storage arrays. The vSphere Replication storage replication source and
target can be any storage device, including, but not limited to, storage arrays.
You can configure vSphere Replication to regularly create and retain snapshots of protected virtual
machines on the recovery site. Taking multiple point-in-time (PIT) snapshots of virtual machines allows you
to retain more than one replica of a virtual machine on the recovery site. Each snapshot reflects the state of
the virtual machine at a certain point in time. You can select which snapshot to recover when you use
vSphere Replication to perform a recovery.
28 VMware, Inc.
Figure 2‑2. Site Recovery Manager Architecture with vSphere Replication
Protected Site
SRM plug-in
vSphere Web Client
SRM plug-in
vSphere Web Client
Recovery Site
ESXi
Server
VR Agent
ESXi
Server
VR Agent
vCenter Server
VMFS
VMFS
Storage
VMFS
VMFS
Storage
ESXi
Server
VR Agent
SRM Server
VR Appliance
Network
File Copy
ESXi
SRM Server
Additional
VR Server
Network
File Copy
ESXi
VR Appliance
vSphere replication
vCenter Server
Chapter 2 Replicating Virtual Machines
Replicating a Virtual Machine and Enabling Multiple Point in Time Instances
You can recover virtual machines at specific points in time (PIT) such as the last known consistent state.
When you configure replication of a virtual machine, you can enable multiple point in time (PIT) instances
in the recovery settings in the Configure Replication wizard. vSphere Replication retains a number of
snapshot instances of the virtual machine on the target site based on the retention policy that you specify.
vSphere Replication supports a maximum of 24 snapshot instances. After you recover a virtual machine,
you can revert it to a specific snapshot.
During replication, vSphere Replication replicates all aspects of the virtual machine to the target site,
including any potential viruses and corrupted applications. If a virtual machine suffers from a virus or
corruption and you have configured vSphere Replication to keep PIT snapshots, you can recover the virtual
machine and then revert it to a snapshot of the virtual machine in its uncorrupted state.
You can also use the PIT instances to recover the last known good state of a database.
NOTE vSphere Replication does not replicate virtual machine snapshots.
VMware, Inc. 29
vSphere Web Client
VR Appliance
t1
VM
VMVMVM
t3t2
vSphere Web Client
VR Appliance
VM
Replication
Source Site
Target Site
t0
Site Recovery Manager Administration
Figure 2‑3. Recovering a Virtual Machine at Points in Time (PIT)
Using Array-Based Replication and vSphere Replication with
Site Recovery Manager
You can use a combination of array-based replication and vSphere Replication in your
Site Recovery Manager deployment.
To create a mixed Site Recovery Manager deployment that uses array-based replication and
vSphere Replication, you must configure the protected and recovery sites for both types of replication.
Set up and connect the storage arrays and install the appropriate storage replication adapters (SRA) on
n
both sites.
Deploy vSphere Replication appliances on both sites and configure the connection between the
n
appliances.
Configure virtual machines for replication using either array-based replication or vSphere Replication,
n
as appropriate.
NOTE Do not attempt to configure vSphere Replication on a virtual machine that resides on a datastore
that you replicate by using array-based replication.
You create array-based protection groups for virtual machines that you configure with array-based
replication, and vSphere Replication protection groups for virtual machines that you configure with
vSphere Replication. You cannot mix replication types in a protection group. You can mix array-based
protection groups and vSphere Replication protection groups in the same recovery plan.
30 VMware, Inc.
Loading...
+ 154 hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.