xiiImplementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Notices
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”
WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in
certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright
and trademark information” at http://www.ibm.com/legal/copytrade.shtml
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
IBM FlashSystem®
IBM SmartCloud®
IBM Spectrum™
IBM Spectrum Control™
IBM Spectrum Protect™
IBM Spectrum Scale™
IBM Spectrum Virtualize™
PowerHA®
Real-time Compression™
Redbooks®
Redbooks (logo)®
Storwize®
System Storage®
Tivoli®
The following terms are trademarks of other companies:
SoftLayer, and The Planet are trademarks or registered trademarks of SoftLayer, Inc., an IBM Company.
Celeron, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
xivImplementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Preface
Organizations of all sizes face the challenge of managing massive volumes of increasingly
valuable data. But storing this data can be costly, and extracting value from the data is
becoming more difficult. IT organizations have limited resources but must stay responsive to
dynamic environments and act quickly to consolidate, simplify, and optimize their IT
infrastructures. The IBM® Storwize® V5000 Gen2 system provides a smarter solution that is
affordable, easy to use, and self-optimizing, which enables organizations to overcome these
storage challenges.
The Storwize V5000 Gen2 delivers efficient, entry-level configurations that are designed to
meet the needs of small and midsize businesses. Designed to provide organizations with the
ability to consolidate and share data at an affordable price, the Storwize V5000 Gen2 offers
advanced software capabilities that are found in more expensive systems.
This IBM Redbooks® publication is intended for pre-sales and post-sales technical support
professionals and storage administrators.
It applies to the Storwize V5030, V5020, and V5010, and to IBM Spectrum Virtualize™ V8.1.
Authors
This book was produced by a team of specialists from around the world working at the
International Technical Support Organization, San Jose Center.
Jon Tate is a Project Manager for IBM System Storage® SAN
Solutions at the International Technical Support Organization
(ITSO), San Jose Center. Before Jon joined the ITSO in 1999,
he worked in the IBM Technical Support Center, providing
Level 2 support for IBM storage products. Jon has 32 years of
experience in storage software and management, services,
and support. He is both an IBM Certified IT Specialist and an
IBM SAN Certified Specialist. He is also the UK Chairman of
the Storage Networking Industry Association.
Dharmesh Kamdar has been working in IBM Systems group
for over 15 years as a Senior Software Engineer. He works in
the Open Systems Lab (OSL), where he focuses on
interoperability testing of a range of IBM storage products with
various vendor products, including operating systems,
clustering solutions, virtualization platforms, volume managers,
and file systems.
Hartmut Lonzer is an OEM Alliance Manager for IBM Storage.
Before this position, he was a Client Technical Specialist for
IBM Germany. He works in the IBM Germany headquarters in
Ehningen. His main focus is on the IBM SAN Volume
Controller, IBM Storwize Family, and IBM VersaStack. His
experience with the IBM SAN Volume Controller and Storwize
products goes back to the beginning of these products.
Hartmut has been with IBM in various technical roles for 40
years.
Gustavo Tinelli Martins is a Storage Technical Leader who
works for IBM Global Technology Services® in Brazil. He is
also an IBM Certified IT Specialist, member of IBM’s IT
Specialist Advisory Board, responsible for evaluating
candidates who wish to acquire the title of IBM IT specialist.
Gustavo has eight years of professional experience, of which
two of those years were dedicated to Customer’s Service
Center and the other six years were dedicated to the Storage
Service Line. Gustavo is certified in multiple IBM storage
technologies and also in other vendor storage products.
Thanks to the following people for their contributions to this project:
Thanks to the following authors of the previous edition of this book:
Catarina Castro
Uwe Dubberke
Justin Heather
Andrew Hickey
Imran Imtiaz
Nancy Kinney
Hartmut Lonzer
Adam Lyon-Jones
Saiprasad Prabhakar Parkar
Edward Seager
Lee Sirett
Chris Tapsell
Paulo Tomiyoshi Takeda
Dieter Utesch
Thomas Vogel
Mikhail Zakharov
xviImplementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Now you can become a published author, too
Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time. Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us.
We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
Use the online Contact us review Redbooks form:
ibm.com/redbooks
Send your comments in an email:
redbooks@us.ibm.com
Mail your comments:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
Stay connected to IBM Redbooks
Find us on Facebook:
http://www.facebook.com/IBMRedbooks
Follow us on Twitter:
http://twitter.com/ibmredbooks
Look for us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html
Preface xvii
xviiiImplementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Summary of changes
This section describes the technical changes made in this edition of the book and in previous
editions. This edition might also include minor corrections and editorial changes that are not
identified.
Summary of Changes
for SG24-8162-03
for Implementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
as created or updated on March 20, 2018.
March 2018, Fourth Edition
This revision includes the following substantial new and changed information.
xxImplementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Chapter 1.Overview of the IBM Storwize
1
V5000 Gen2 system
This chapter provides an overview of the IBM Storwize V5000 Gen2 architecture and includes
a brief explanation of storage virtualization.
Specifically, this chapter provides information about the following topics:
Overview
Te r mi n ol o gy
Models
IBM Storwize V5000 Gen1 and Gen2 compatibility
Hardware
Te r ms
Features
Problem management and support
More information resources
The IBM Storwize V5000 Gen2 solution is a modular entry-level and midrange storage
solution. The IBM Storwize V5000 Gen2 includes the capability to virtualize its own internal
Redundant Array of Independent Disk (RAID) storage and existing external storage area
network (SAN)-attached storage (the Storwize V5030 only).
The three IBM Storwize V5000 Gen2 models (Storwize V5010, Storwize V5020, and Storwize
V5030) offer a range of performance scalability and functional capabilities. Table 1-1 shows a
summary of the features of these models.
Table 1-1 IBM Storwize V5000 Gen2 models
Storwize V5010Storwize V5020Storwize V5030
CPU cores226
Cache16 GBUp to 32 GBUp to 64 GB
Supported expansion
enclosures
External storage
virtualization
CompressionNoNoYes
EncryptionNoYesYes
101020
NoNoYes
For a more detailed comparison, see Table 1-3 on page 6.
IBM Storwize V5000 Gen2 features the following benefits:
Enterprise technology available to entry and midrange storage
Expert administrators are not required
Easy client setup and service
Simple integration into the server environment
Ability to grow the system incrementally as storage capacity and performance needs
change
The IBM Storwize V5000 Gen2 addresses the block storage requirements of small and
midsize organizations. The IBM Storwize V5000 Gen2 consists of one 2U control enclosure
and, optionally, up to ten 2U expansion enclosures on the Storwize V5010 and Storwize
V5020 systems and up to twenty 2U expansion enclosures on the Storwize V5030 systems.
The Storwize V5030 systems are connected by serial-attached Small Computer Systems
Interface (SCSI) (SAS) cables that make up one system that is called an
I/O group.
With the Storwize V5030 systems, two I/O groups can be connected to form a cluster,
providing a maximum of two control enclosures and 40 expansion enclosures. With the High
Density expansion drawers, you are able to attach up to 16 expansion enclosures to a cluster.
The control and expansion enclosures are available in the following form factors, and they can
be intermixed within an I/O group:
12 x 3.5-inch (8.89-centimeter) drives in a 2U unit
24 x 2.5-inch (6.35-centimeter) drives in a 2U unit
92 x 2.5-inch in carriers or 3.5-inch drives in a 5U unit
2Implementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Two canisters are in each enclosure. Control enclosures contain two node canisters, and
expansion enclosures contain two expansion canisters.
The IBM Storwize V5000 Gen2 supports up to 1,520 x 2.5 inch drives or 3.5 inch drives or a
combination of both drive form factors for the internal storage in a two I/O group Storwize
V5030 cluster.
SAS, Nearline (NL)-SAS, and flash drive types are supported.
The IBM Storwize V5000 Gen2 is designed to accommodate the most common storage
network technologies to enable easy implementation and management. It can be attached to
hosts through a Fibre Channel (FC) SAN fabric, an Internet Small Computer System Interface
(iSCSI) infrastructure, or SAS. Hosts can be attached directly or through a network.
Important: For more information about supported environments, configurations, and
restrictions, see the IBM System Storage Interoperation Center (SSIC):
The IBM Storwize V5000 Gen2 is a virtualized storage solution that groups its internal drives
into RAID arrays, which are called
the Storwize V5030 systems by importing logical unit numbers (LUNs) from external FC
SAN-attached storage. These MDisks are then grouped into
created from these storage pools and provisioned out to hosts.
managed disks (MDisks). MDisks can also be created on
storage pools. Volumes are
Storage pools are normally created with MDisks of the same drive type and drive capacity.
Vo l um e s can be moved non-disruptively between storage pools with differing performance
characteristics. For example, a volume can be moved between a storage pool that is made up
of NL-SAS drives to a storage pool that is made up of SAS drives to improve performance.
The IBM Storwize V5000 Gen2 system also provides several configuration options to simplify
the implementation process. It also provides configuration presets and automated wizards
that are called
occur.
Included with an IBM Storwize V5000 Gen2 system is a simple and easy to use graphical
user interface (GUI) to enable storage to be deployed quickly and efficiently. The GUI runs on
any supported browser. The management GUI contains a series of preestablished
configuration options that are called
configure objects on the system. Presets are available for creating volumes and IBM
FlashCopy® mappings and for setting up a RAID configuration.
You can also use the command-line interface (CLI) to set up or control the system.
Directed Maintenance Procedures (DMP) to help resolve any events that might
presets that use commonly used settings to quickly
Chapter 1. Overview of the IBM Storwize V5000 Gen2 system 3
1.2 IBM Storwize V5000 Gen2 terminology
The IBM Storwize V5000 Gen2 system uses terminology that is consistent with the entire IBM
Storwize family and the IBM SAN Volume Controller. The terms are defined in Table 1-2. More
terms can be found in Appendix B, “Terminology” on page 819.
Table 1-2 IBM Storwize V5000 Gen2 terminology
IBM Storwize V5000 Gen2 termDefinition
BatteryEach control enclosure node canister in an IBM Storwize V5000
Gen2 contains a battery.
ChainEach control enclosure has either one or two chains, which are
used to connect expansion enclosures to provide redundant
connections to the inside drives.
CloneA copy of a volume on a server at a particular point. The
contents of the copy can be customized and the contents of the
original volume are preserved.
Control enclosureA hardware unit that includes a chassis, node canisters, drives,
and power sources.
Data migrationIBM Storwize V5000 Gen2 can migrate data from existing
external storage to its internal volumes.
Distributed RAID (DRAID)No dedicated spare drives are in an array. The spare capacity is
distributed across the array, which allows faster rebuild of the
failed disk.
DriveIBM Storwize V5000 Gen2 supports a range of hard disk drives
(HDDs) and Flash Drives.
EventAn occurrence that is significant to a task or system. Events can
include the completion or failure of an operation, a user action,
or the change in the state of a process.
Expansion canisterA hardware unit that includes the SAS interface hardware that
enables the control enclosure hardware to use the drives of the
expansion enclosure. Each expansion enclosure has two
expansion canisters.
Expansion enclosureA hardware unit that includes expansion canisters, drives, and
power supply units.
External storageMDisks that are SCSI logical units (LUs) that are presented by
storage systems that are attached to and managed by the
clustered system.
Fibre Channel portFibre Channel ports are connections for the hosts to get access
to the IBM Storwize V5000 Gen2.
Host mappingThe process of controlling which hosts can access specific
volumes within an IBM Storwize V5000 Gen2.
Internal storageArray MDisks and drives that are held in enclosures that are part
of the IBM Storwize V5000 Gen2.
iSCSI (Internet Small Computer
System Interface)
Internet Protocol (IP)-based storage networking standard for
linking data storage facilities.
4Implementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
IBM Storwize V5000 Gen2 termDefinition
Managed disk (MDisk)A component of a storage pool that is managed by a clustered
system. An MDisk is part of a RAID array of internal storage or a
SCSI LU for external storage. An MDisk is not visible to a host
system on the SAN.
Node canisterA hardware unit that includes the node hardware, fabric, and
service interfaces, SAS expansion ports, and battery. Each
control enclosure contains two node canisters.
PHYA single SAS lane. Four PHYs are in each SAS cable.
Power Supply UnitEach enclosure has two power supply units (PSU).
Quorum diskA disk that contains a reserved area that is used exclusively for
cluster management. The quorum disk is accessed when it is
necessary to determine which half of the cluster continues to
read and write data.
Serial-Attached SCSI (SAS) portsSAS ports are connections for expansion enclosures and direct
attachment of hosts to access the IBM Storwize V5000 Gen2.
SnapshotAn image backup type that consists of a point-in-time view of a
volume.
Storage poolAn amount of storage capacity that provides the capacity
requirements for a volume.
StrandThe SAS connectivity of a set of drives within multiple
enclosures. The enclosures can be control enclosures or
expansion enclosures.
Thin provisioning or thin
provisioned
Traditional RAID (TRAID)Traditional RAID uses the standard RAID levels.
VolumeA discrete unit of storage on disk, tape, or other data recording
Worldwide port namesEach Fibre Channel port and SAS port is identified by its
The ability to define a storage unit (full system, storage pool, or
volume) with a logical capacity size that is larger than the
physical capacity that is assigned to that storage unit.
medium that supports a form of identifier and parameter list,
such as a volume label or input/output control.
physical port number and worldwide port name (WWPN).
Chapter 1. Overview of the IBM Storwize V5000 Gen2 system 5
1.3 IBM Storwize V5000 Gen2 models
The IBM Storwize V5000 Gen2 platform consists of different models. Each model type
supports a different set of features, as shown in Table 1-3.
Table 1-3 IBM Storwize V5000 feature comparison
FeatureV5000 Gen1V5010 V5020 V5030
Cache16 GB16 GB16 GB or 32 GB32 GB or 64 GB
CPU4 - core
Ivy Bridge Xeon
CPU
2 GHz
CompressionNoneNoneNoneLicensed (with 64
D R A I DYesYe sYe sYe s
SAS HW
Encryption
External
Virtualization
IBM Easy Tier®LicensedLicensedLicensedLicensed
FlashCopyLicensedLicensedLicensedLicensed
HyperswapYesNoNoYes
Remote CopyLicensedLicensedLicensedLicensed
Thin ProvisioningYesYesYesYes
Traditional RAIDYesYesYesYes
Volume MirroringYesYesYesYes
NoneNoneLicensedLicensed
LicensedData Migration
2- core
Broadwell-DE
Celeron CPU
1.2 GHz
Only
2- core
Broadwell-DE
Xeon CPU
2.2 GHz
Hyper-threading
Data Migration
Only
6 - core
Broadwell-DE
Xeon CPU
1.9 GHz
Hyper-threading
GB cache only)
Licensed
VMware Virtual
Volumes (VVols)
Ye sYe sYesYes
More information: For more information about the features, benefits, and specifications of
IBM Storwize V5000 Gen2 models, see the following website:
https://ibm.biz/BdrRjb
The information in this book is accurate at the time of writing. However, as the IBM
Storwize V5000 Gen2 matures, expect to see new features and enhanced specifications.
6Implementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
The IBM Storwize V5000 Gen2 models are described in Table 1-4. All control enclosures
have two node canisters. F models are expansion enclosures.
Table 1-4 IBM Storwize V5000 Gen2 models
ModelDescriptionCacheDrive Slots
One-year warranty
2077-112 IBM Storwize V5010
large form factor (LFF)
Control Enclosure
2077-124 IBM Storwize V5010
small form factor (SFF)
Control Enclosure
2077-212 IBM Storwize V5020
LFF Control Enclosure
2077-224 IBM Storwize V5020
SFF Control Enclosure
2077-312 IBM Storwize V5030
LFF Control Enclosure
2077-324 IBM Storwize V5030
SFF Control Enclosure
2077-AF3IBM Storwize V5030F
All-Flash Array Control
Enclosure
2077-12F IBM Storwize V5000
LFF Expansion
Enclosure
2077-24F IBM Storwize V5000
SFF Expansion
Enclosure
16 GB12 x 3.5-inch
16 GB24 x 2.5-inch
16 GB or 32 GB12 x 3.5-inch
16 GB or 32 GB24 x 2.5-inch
32 GB or 64 GB12 x 3.5-inch
32 GB or 64 GB24 x 2.5-inch
64GB24 x 2.5-inch
N/A12 x 3.5-inch
N/A24 x 2.5-inch
2077-AFFIBM Storwize V5030F
SFF Expansion
Enclosure
2077-A9FIBM Storwize V5030F
High Density LFF
Expansion Enclosure
Three-year warranty
2078-112 IBM Storwize V5010
LFF Control Enclosure
2078-124 IBM Storwize V5010
SFF Control Enclosure
2078-212 IBM Storwize V5020
LFF Control Enclosure
2078-224 IBM Storwize V5020
SFF Control Enclosure
Chapter 1. Overview of the IBM Storwize V5000 Gen2 system 7
N/A24 x 2.5-inch
N/A92 x 3.5-inch
16 GB12 x 3.5-inch
16 GB24 x 2.5-inch
16 GB or 32 GB12 x 3.5-inch
16 GB or 32 GB24 x 2.5-inch
ModelDescriptionCacheDrive Slots
2078-312 IBM Storwize V5030
LFF Control Enclosure
2078-324 IBM Storwize V5030
SFF Control Enclosure
2078-AF3IBM Storwize V5030F
All-Flash Array Control
Enclosure
2078-12F IBM Storwize V5000
LFF Expansion
Enclosure
2078-24F IBM Storwize V5000
SFF Expansion
Enclosure
2078-AFFIBM Storwize V5030F
SFF Expansion
Enclosure
2078-A9FIBM Storwize V5030F
High Density LFF
Expansion Enclosure
32 GB or 64 GB12 x 3.5-inch
32 GB or 64 GB24 x 2.5-inch
64GB24 x 2.5-inch
N/A12 x 3.5-inch
N/A24 x 2.5-inch
N/A24 x 2.5-inch
N/A92 x 3.5-inch
Storwize V5030F control enclosures support only the attachment of Storwize V5030F
expansion enclosures (Models AFF and A9F). Storwize V5000 expansion enclosures (Models
12E, 24E, 12F, 24F, and 92F) are not supported with Storwize V5030F control enclosures.
Storwize V5030F expansion enclosures are only supported for attachment to Storwize
V5030F control enclosures. Storwize V5000 control enclosures (Models 12C, 24C, 112, 124,
212, 224, 312, and 324) do not support the attachment of Storwize V5030F expansion
enclosures.
Table 1-5 shows the 2U expansion enclosures and 5U expansion enclosure mix rules. Shown
are the maximum # of Drive Slots per SAS expansion string without disks in the controller
itself.
The Storwize V5030 systems can be added to an existing IBM Storwize V5000 Gen1 cluster
to form a two-I/O group configuration. This configuration can be used as a migration
mechanism to upgrade from the IBM Storwize V5000 Gen1 to the IBM Storwize V5000 Gen2.
8Implementing the IBM Storwize V5000 Gen2 with IBM Spectrum Virtualize V8.1
Loading...
+ 834 hidden pages
You need points to download manuals.
1 point = 1 manual.
You can buy points or you can get point for every manual you upload.