Spectra Logic Spectra nTier500, Spectra nTier700, Spectra nTier File Migrator User Manual

Spectra nTier File Migrator

User Guide

PN 90990032 Revision A

Notices

Notices

Except as expressly stated herein, Spectra Logic Corporation makes its products and associated documentation on an “AS IS”
BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, BOTH OF WHICH ARE EXPRESSLY DISCLAIMED.
In no event shall Spectra Logic be liable for any loss of profits, loss of business, loss of use or data, interruption of business, or
for indirect, special, incidental or consequential damages of any kind, even if Spectra Logic has been advised of the possibility of
such damages arising from any defect or error.

Information furnished in this manual is believed to be accurate and reliable. However, no responsibility is assumed by
Spectra Logic for its use. Due to continuing research and development, Spectra Logic may revise this publication from time
to time without notice, and reserves the right to change any product specification at any time without notice.

If you do not agree to the above, do not use the Spectra nTier700; instead, promptly contact Spectra Logic for instructions
on how to return the Spectra nTier700 for a refund.

BlueScale, Spectra, SpectraGuard, TeraPack, and the Spectra Logic logo are registered trademarks and RXT is a trademark
of Spectra Logic Corporation. All rights reserved worldwide. All other trademarks and registered trademarks are the
property of their respective owners.

Spectra nTier File Migrator User Guide
Revision A, July 2008

Copyright © 2008 Spectra Logic Corporation. All rights reserved.

2

Contacting Spectra Logic

To obtain general information

Spectra Logic web Site: www.spectralogic.com

United States Headquarters European Office

Contacting Spectra Logic

Spectra Logic Corporation
1700 North 55th Street
Boulder, CO 80301
USA

Phone: [phone] or [phone]
International: 00 (1) 303 449 6400
Fax: [phone]

Spectra Logic Europe Ltd.
Magdalen Centre
Robert Robinson Avenue
Oxford Science Park
OXFORD
OX4 4GA
United Kingdom

Phone: [phone]
Fax: [phone]

SpectraGuard Technical Support

web Site: www.spectralogic.com/support/index.cfm/

Knowledge Base: www.spectralogic.com/support/kbase/index.cfm/

United States and Canada
Phone: [phone]

E-mail: [e-mail]

Mexico, Central and South America, Asia, Australia, and New Zealand
Phone: [phone]

E-mail: [e-mail]

Europe, Middle East, Africa Phone
Phone: [phone] (UK only)

49 (0) 6028 9796 507

E-mail: [e-mail]

Spectra Logic Sales

web Site: www.SpectraLogic.com

United States and Canada
Phone: [phone] or [phone]
Fax: [phone]

E-mail: [e-mail]

Media Sales
E-mail: [e-mail]

European Office
Phone: [phone]
Fax: [phone]
E-mail: [e-mail]

3

Contacting Spectra Logic

Notes

4

Contents

About This Guide 9

Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Conventions Used in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Chapter 1 – Overview 11

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Software Feature Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Installations Using a Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Installations using Stand-alone Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Windows Offline Attribute . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Determining Disk Space Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Chapter 2 – Concepts 23

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Constructing the File Name, Path Pattern, or Exclude Pattern . . . . . . . . . . . . . . 24

Examples of Allocating Files to File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Tapes: Volumes, Volume Sets, and Automatic Replication . . . . . . . . . . . . . . . . . . . 28

Understanding Volume Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

About the Blank Media Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

About the Quarantined Media Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Partial File Restore and Cartridge Spanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Off-line File Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

File Version Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

File Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5

Contents

Chapter 3 – Installing nTier File Migrator 35

Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Access the nTier Windows Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Prepare the nTier Windows Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Configure the Archive Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Disable Microsoft Removable Storage Service . . . . . . . . . . . . . . . . . . . . . . . . . 38

Remove Microsoft Services for NFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Enable Internet Information Services (ISS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Change the Default System Name (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Install nTier File Migrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Install and Activate the Tape Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Install Device Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Enable Stand-Alone Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Activate the nTier File Migrator License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Activate the Archive Drive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Start the License Administration Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Complete the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Chapter 4 – Configuring nTier File Migrator 47

Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Start the Archive Management Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Configure Volume Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Create a New Volume Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Configure the Volume Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Configure Replication for the Volume Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Allocate Tape Cartridges to the Volume Set . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Allocate Files to File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Create One or More New File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Configure the File Group Listing Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Allocate Files to a File Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Set the Policies for the File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6

Contents

Chapter 5 – Using nTier File Migrator 65

Accessing Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Access the Current Version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Access Old or Deleted Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Working with Tape Cartridges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Importing Tapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Reformatting Tapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Setting the Administrator-Defined Information for a Tape . . . . . . . . . . . . . . . . 72

Displaying Information About a Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Verifying the Data on a Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Finalizing a Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Write-Protecting Tapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Exporting a Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Managing Volume Sets and File Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Deleting a Volume Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Renaming a File Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Retensioning Tapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

Chapter 6 – System Recovery 79

System Rebuild from Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

About File System Rebuild From Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Rebuilding a File System from Tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Disabling and Re-Enabling a Tape Drive or Library . . . . . . . . . . . . . . . . . . . . . . . . 80

Disabling a Tape Drive or Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Re-enabling a Tape Drive or Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Options In Case of Tape Library or Stand-Alone Drive Failure . . . . . . . . . . . . . 82

7

Contents

Chapter 7 – Diagnostics & Maintenance 83

Event Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Opening the Event Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Viewing More Detail about an Error or Warning . . . . . . . . . . . . . . . . . . . . . . . 84

Diagnostic Trace Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Opening the Trace Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

Changing the Level of Detail in the Trace Log . . . . . . . . . . . . . . . . . . . . . . . . . 85

Automatic Generation of Trace Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Tape Drive and Library Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Tape Drive Cleaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

Tape Cartridge Error States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

Index 89

8

About This Guide

This guide describes how to configure and use the Spectra® nTier™ File Migrator
software.

Intended Audience

This guide is intended for data center administrators and operators who maintain and
operate backup systems. It also assumes a knowledge of technical tasks such as
configuring operating systems and installing drivers.

Related Publications

The following documents are related to the Spectra nTier Intelligent Storage appliances
used with the nTier File Migrator software. For these documents and additional
information about the Spectra nTier Intelligent Storage appliances and libraries, check
the Spectra Logic web site at www.spectralogic.com/documentation.

Spectra nTier Intelligent Secondary Storage Appliances

 The Spectra nTier500 Intelligent Secondary Storage Appliance User Guide provides

detailed information about installing, configuring, and using the Spectra nTier500.

 The Spectra nTier700 Intelligent Secondary Storage Appliance User Guide provides

detailed information about installing, configuring, and using the Spectra nTier700.

Microsoft Windows Operating System and Software

The integrated Windows server in the Spectra nTier Intelligent Storage appliances runs
the Windows Storage Server 2003 R2 operating system. You can find information about
this operating system on the Microsoft web site at www.microsoft.com/

windowsserver2003.

9

About This Guide

Conventions Used in This Guide

This manual uses the following conventions to highlight important information:

Note: Provides additional information or suggestions about the current

topic.

Caution: Provides information you must know to avoid damage to the

library or tape drives or avoid losing data.

Warning: Provides information you must know to avoid personal injury.

10

1 Overview

Introduction

The Spectra® nTier™ File Migrator software manages data tapes and magnetic disk on
a Windows file server (a Spectra nTier Intelligent Secondary Storage appliance) to
create a highly scalable digital archive solution. The solution can be configured with a
wide range of tape libraries, stand-alone tape drives and magnetic storage. It is ideal
for archiving professional digital video, medical images, scientific and engineering
data and any application which requires a high performance data repository ranging
from a few terabytes to multiple petabytes.

The nTier File Migrator software provides the following key functionality:

 Standard Windows File System Interface—The magnetic disk storage and

managed tapes appear as a single standard Windows logical drive which means
one or more software applications can write to and read from the archive as
though it were a standard disk-based logical drive letter.

 Strong Data Protection—The system combines continuous data protection and

tape cartridge replication.

 Hierarchical Storage Management—The solution supports online, nearline and

offline files. A file is online when it is available from magnetic disk. Nearline refers
to a file that is not on magnetic disk but is available from one or more tape
cartridges held in the library or within stand-alone tape drives. Offline refers to a
file that is not on magnetic disk but is written to one or more tape cartridges all of
which have been exported from the library and/or tape drives.

 Supports WORM Tape—The solution supports both standard rewritable tape and

unalterable WORM tape. WORM tape is particularly useful for compliance
applications.

Chapter 1 – Overview

nTier700 RAID running nTier File Migrator

Spectra T50e library

LAN

Clients

Fibre Channel or SCSI connection

The nTier File Migrator software is hosted on an Windows server (for example, the
Spectra nTier700) running Microsoft Storage Server 2003 R2. The Windows server is
connected to a tape library or stand-alone tape drives and is always configured with a
dedicated logical drive that is managed by the nTier File Migrator software. Figure 1-1
shows a typical system configuration with a Spectra nTier700 and a single Spectra T50e
tape library.

Figure 1-1 An nTier700 running File Migrator and connected to a single T50 tape library.

When writing to and reading from tape, the system always uses the dedicated logical
drive for caching purposes. On writing to tape, a file is first written to magnetic disk
and after it has been successfully written to disk, the file goes into a queue for writing
to tape. On reading a file from tape, it is streamed onto disk cache and simultaneously
transferred over the network to fulfill the read request.

The system administrator defines policies that determine where data files are
physically stored. A single server may have many different policies, tailored to the
needs of the different file types that are being archived.

The file management policies are defined via the nTier File Migrator Archive
Management Console, which is a Microsoft Management Console (MMC) snap-in. Using
this snap-in provides a familiar administration interface which can be easily integrated
with other MMC snap-ins. The policies determine whether a file is written to tape and,
if so, allocates a specific group of tapes for the file. The policies also define whether
the tapes will be replicated and the schedule for replica updates.

12

Introduction

The online retention period is set by policy and can range from permanent retention
on magnetic disk to flushing from disk as soon as the last tape replica has been
successfully written. A typical policy is illustrated in Figure 1-2.

Figure 1-2 A file storage policy example.

For example, in the file management policy illustrated in Figure 1-2, a file is written to
both magnetic disk and a specified group of tape cartridges. This policy immediately
provides two copies of the file for high data integrity. The system administrator defines
the replication schedule and the number of tape copies for replication. After all copies
of the file have been produced, the file can be flushed (deleted) from magnetic disk
according to the administrator-defined policy. When a file is read, it is always retrieved
from the fastest available physical storage device. If the file is stored on both magnetic
disk and tape, the file will be read from disk. If the file is available only on tape, it will
be retrieved from tape.

The software supports both rewritable and Write-Once-Read-Many (WORM) tape
cartridges. WORM tape provides an unalterable record of the data written to it and
provides best practice when authenticity of the digital content is important. The nTier
File Migrator software supports the simultaneous use of WORM and rewritable media.

13

Chapter 1 – Overview

The software presents the magnetic disk storage and tape library as a standard
rewritable logical drive, even when unalterable WORM tape is being used within the
library. The ability to delete and overwrite files is required by many standard
applications. Consequently, the rewritable nature of the archive allows standard
applications to work with the nTier File Migrator storage solution without need to
write any special interface software. When a file is deleted, it is no longer available
using the standard file system interface. Similarly, when a new version of a file is
written, only the latest version is available using the standard interface. The nTier File
Migrator software also includes a browser-based utility, History Explorer, which makes
it easy to identify, access and restore all old file versions and deleted files from tape.

Software Feature Overview

 Standard File System—A magnetic disk logical drive and all managed tapes are

virtualized, making these physical storage devices appear as a file system within a
single logical drive letter. The solution uses the standard Windows offline file
attribute to identify when a file is no longer online.

Benefit: Seamlessly integrates with standard applications and existing network
infrastructure without modification.

 Combining HSM and Data Protection—Archive Series software provides policy-

based Hierarchical Storage Management based on file deletion rather than classic
file migration.

Benefit: This approach lets the administrator match the life-cycle of a file to the
physical storage media and provides built-in data protection, removing the need for
conventional backup.

 Offline Tape Cartridge Management—The system retains meta-data for offline

tape cartridges.

Benefit: The system supports an unlimited number of tapes on the shelf.

 Partial Read of Large Files—Often, only a portion of a very large file needs to be

read. For example, this frequently occurs with multi-gigabyte video files when a
short clip is requested. The nTier File Migrator software supports partial reading of
large files.

Benefit: Enhanced performance when dealing with large files

 Automated Tape Cartridge Replication—Tape cartridges are replicated

automatically based on the policies defined by the Administrator. When a tape
cartridge replica is reintroduced into the library after being retained off-site, it is
automatically brought up-to-date by the system.

Benefit: It is easy to rotate tape cartridges off-site for data protection purposes.

14

Software Feature Overview

 Multiple Tape Set Support—The software allows file groups to be allocated to

specified groups of tapes.

Benefit: The Administrator can group related files together on the same set of
tapes.

 Cartridge Spanning—The system supports storage of large files across multiple

tape cartridges.

Benefit: Results in higher use of tape capacity and also supports storage of very
large files with a size greater than that of an individual tape cartridge.

 Supports both WORM and Rewritable Tape—The software simultaneously

manages both WORM and rewritable tape cartridges in the same tape library.
Policies that are set by the Administrator define which files are written to which
type of tape cartridge.

Benefit: Allows the Administrator to match tape cartridge type to the data storage
needs of the organization.

 Comprehensive File Version Management—The nTier File Migrator software

provides a full audit trail of all file changes and deletions. History Explorer is a
browser-based utility that lets authorized users access all versions of a file and all
deleted files.

Benefit: Provides legal compliance and best business practice, including meeting
SEC17a-4 requirements.

 Open Standard Tape Format—Open standard TAR file format is used on the tape,

allowing the tape cartridges to be read using third party utilities.

Benefit: The use of open standards on industry standard hardware ensures the
long-term availability of data.

 Dynamic Expansion of Tape Sets—The system dynamically expands tape sets to

meet capacity demands.

Benefit: This minimizes system administration.

 Familiar Administration—File policy and tape cartridge management is performed

by the administrator using the Archive Management Console which is a Microsoft
Management Console Snap-In.

Benefit: The Administrator uses a familiar tool for system management.

 Microsoft Security—File Migrator software is fully integrated with the Microsoft

Windows security model, based on Active Directory.

Benefit: Effortlessly integrates with existing security, minimizing system
administration.

 Supports High Performance Tape Formats—The software supports a number of

leading tape cartridge formats including LTO.

15

Chapter 1 – Overview

nTier700 RAID running nTier File Migrator

Spectra T50 library

Fibre Channel or SCSI connections

LAN

Clients

Tape drive

Benefit: These popular industry formats provide high capacities, high data transfer
rates and estimated data lifetimes up to 30 years without need to rewrite data or
rewind tapes.

 Highly Scalable—The nTier File Migrator software supports unlimited RAID

capacity and tape libraries with capacities from less than 1 TB to over 1PB.

Benefit: Works easily with tape libraries, independent of their capacities.

 API Available—An API is available.

Benefit: This can be used by developers to obtain tight integration with their

applications.

Installations Using a Tape Library

The nTier File Migrator software can be configured and licensed to manage a tape
library, optionally combined with one or more data tape drives.

Figure 1-3 shows a Windows server (the nTier700) connected to tape library and a
stand-alone data tape drive. A configuration combining a tape library and a stand-alone
drive is useful when files are retained both nearline and offline. The stand-alone tape
drive is convenient for quickly bringing individual offline tapes back to nearline.

Figure 1-3 An nTier700 running File Migrator, connected to a T50 tape library and a stand­alone tape drive.

With the configuration shown in Figure 1-3, all archived files appear within one logical
drive on the Windows server (the nTier700), independent of file location—whether
those files are on a tape within the library, on a tape in the tape drive or are offline.
This single logical drive may be shared over the network, making the archive
accessible to network clients.

16

Installations Using a Tape Library

When configuring a system using a tape library, consider the following guidelines to
ensure that the library includes the appropriate number of tape drives.

 With only one tape drive within the library, it is important to be careful in setting

the file management policies to prevent the single drive from becoming a
bottleneck and in turn degrading performance.

When only one drive is present, we recommend that frequently accessed files are
retained online by employing a large RAID for the dedicated logical magnetic drive
and appropriate RAID retention rules. Make sure that only one set of tapes (Volume
Set) is configured and that tape replication is scheduled to occur at a time when
there is no other writing or reading activity.

 With two tape drives within the library, the concern over tape drive bottleneck is

much less severe than for a single drive system. However, we recommend that only
one Volume Set should be configured.

 Three or more tape drives within one library are recommended for large or

frequently accessed archives. The nTier File Migrator software intelligently manages
many drives, allowing simultaneous writing and multiple file accesses.

All installations that include a tape library, even if it has only one drive within it,
support tape cartridge replication.

17

Chapter 1 – Overview

Fibre Channel or SCSI connection

LAN

Clients

Offline tapes

Tape drives

nTier500 RAID running nTier File Migrator

Installations using Stand-alone Drives

The nTier File Migrator software can be configured and licensed to manage one or
more stand-alone tape drives connected to the Windows server. Figure 1-4 shows a
Windows server (the nTier500) connected to two tape drives. Even though there are
two tape drives in this system, all archived files appear on one logical drive on the
Windows server, independent of whether those files are on a cartridge loaded in one of
the tape drives or on a cartridge stored offline. This single logical drive can be shared
over the network, making the archive accessible to network clients.

Figure 1-4 The nTier500 running File Migrator connected to two stand-alone tape drives.

When a file is written to tape, it is first written to magnetic disk, then the file is
immediately written to tape. The system does not permit a file to be written to disk
unless there is a suitable tape with sufficient available capacity loaded into one of the
tape drives. The following sequence of events occurs when writing a file to tape:

1. The file is first written to disk cache and appears in the file system.

2. The file is immediately written to tape.

3. After the file is on tape, it is then eligible to be flushed (i.e., deleted) from the disk

cache. The disk retention period is determined by administrator settings.

The following sequence of events occurs when reading a file from a tape cartridge
loaded in one of the tape drives:

18

Windows Offline Attribute

1. The tape is moved to the beginning of the file (or a specified portion of the file).

Note: When reading a file that is only available on an offline tape, the

system generates a Windows event log entry that identifies the
tape cartridge on which the file is store. The cartridge must be
loaded into a tape drive before the read request can be fulfilled.

2. The file (or portion of file) is streamed to the disk cache making it available for read

request.

3. When the file is closed, it is eligible to be flushed from the disk cache. The disk

retention period is determined by administrator settings.

A system with two drives can be configured to generate two copies of each tape using
the automatic replication capabilities of the nTier File Migrator software. This can be
useful if one set of tape cartridge replicas is to be held off-site for data protection
purposes. This option is not available in a system with only one stand-alone tape drive.

Windows Offline Attribute

The nTier File Migrator software uses the standard Windows offline file attribute to
identify files that are no longer online on magnetic disk storage. After a file has been
successfully written to tape and is no longer on magnetic disk, the offline attribute is
set. This Microsoft file attribute changes network timeout periods to longer access
times. This allows retrieval of files from tape. This attribute also changes the
appearance of a file within Windows Explorer—a small clock is added to the bottom
left of the file icon, as shown in Figure 1-5.

Figure 1-5 Offline file identification, as indicated by clock icon.

19

Chapter 1 – Overview

The state of the offline file attribute for the three tiers of HSM is presented in the
following table.

HSM Tier Description Offline Attribute

Online One instance of the file is on magnetic disk and, in addition,

there may be one or more instances on tape.

Nearline At least one instance of the file is on tape within a tape library or

tape drive and no instance is on magnetic disk.

Offline The file is not stored on magnetic disk and the instances on tape,

all of which have been exported from the tape libraries or ejected
from stand-alone drives.

Not set

Set

Set

Determining Disk Space Requirements

The Windows server (for example, the nTier700) must be configured with at least two
logical drives on magnetic disk—a boot drive (usually C:) and another logical drive
dedicated for use by the nTier File Migrator archive. The nTier File Migrator software is
installed on the boot drive but it requires minimal available capacity. The dedicated
logical drive stores file system metadata archive. This archive is used for read and write
caching of files held on tape and for storing all files that are retained online. This
dedicated logical drive must be formatted with NTFS and must be configured as a
dynamic disk.

Capacity requirements for the dedicated logical drive vary considerably depending on
how the system is configured. The total capacity requirement is the sum of the
following.

 File system metadata—Each file in the archive file system requires two NTFS

clusters. If the dedicated logical drive is formatted using the default cluster size of
4096 bytes, a file system with one million files will require 8 GB for the metadata.

 Read and write caching—The dedicated logical drive provides caching for each

file that is being written and all files that are open due to being read by a program.
Using a capacity allowance of twenty times the largest file size is adequate for most
environments.

 Online Files—Any group of files may be retained online. The administrator uses

the Archive Management Console to define policies for online retention. If tape
cartridge replication is not employed, the system can be configured to delete files
immediately after writing and reading, in which case the additional capacity
requirements for online files is zero. At the other extreme, the system can be
configured to keep all current files online, in which case the additional capacity
requirements for online files is the sum of all the file sizes within the file system.

20

Determining Disk Space Requirements

Example calculation for an installation with only stand-alone tape drives:

 Up to a 250,000 files: 2 GB required for metadata

 Maximum file size of 1 GB: 20 GB allowed for caching

 Immediate file deletion and no replication: zero allowance is required for online

retention

A total dedicated logical drive capacity of 22 GB or higher is estimated for this
installation.

Example calculation for a 200 TB digital video archive using a large tape library:

 100,000 files with high resolution content and 100,000 low resolution proxy files:

1.6 GB required for metadata

 Average high resolution file size of 2 GB: 40 GB allowed for caching

 Average low resolution proxy file is 100 MB and all must be retained online: 10 TB

required for online retention.

A total dedicated logical drive capacity of just over 10 TB is estimated for this
installation.

21

Chapter 1 – Overview

Notes

22

2 Concepts

Introduction

Using the nTier File Migrator software, the system administrator defines policies that
allow:

 Groups of files (termed File Groups) to be defined based on the file name and

folder (that is, location in the directory structure)

 Groups of files to be stored on defined sets of tape cartridges (termed Volume Sets)

 Groups of files to be retained online on magnetic disk for predefined periods

 Automatic generation of tape replicas for off-site retention

 Fast partial restore of very large files and tape cartridge spanning

 Notification of requests for files that are stored offline

 File version management

 Configuration of file security to control file writing and access

File Groups and Volume Sets are configured by the administrator using the Archive
Management Console.

File Groups

A File Group is a collection of files that have the same file management policy and
consequently are all treated in the same way by the system. Every file that is handled
by the system belongs to exactly one File Group. Files are assigned to a File Group on
the basis of name and path. This assignment can be based on the name of the folder
that contains a file, the name of the file or both. For example, the system may be
configured with the following three File Groups.

 File Group 1 contains all files with names of the form *.tmp, regardless of folder.

These are saved to magnetic disk but are not saved to tape.

 File Group 2 contains all other files saved to the folder \project1\. These are

permanently retained online on magnetic disk and are also saved to tape to a
specific Volume Set called “Tapes01.”

23

Chapter 2 – Concepts

 File Group 3 contains all other files saved to the folder \project2\. These are

retained online on magnetic disk for 24 hours after they were written or last read.
They are also saved to tape to another Volume Set called “Tapes02.”

A file cannot be written to the system if a File Group is not present that matches its file
name and path. In the example above, files can only be stored if written to the folders
\project1\ or \project2\ of if they have a name of the form *.tmp. Attempts to write
other files is not permitted. If appropriate, the system can be configured with an
additional “catch all” File Group that sets a policy for all other files.

File Group characteristics are defined by the administrator using the Archive
Management Console. The following parameters must be defined for each File Group:

 File name or path pattern for the File Group. This, together with the relative

position of the File Group in the Archive Management Console display, defines the
files allocated to this File Group.

 The position of the File Group relative to other File Groups in the Archive

Management Console display, which defines the order in which file name or path
pattern selection is applied. If a file name matches the selection criteria for more
than one File Group, it will be assigned to the first matching File Group.

 Enable or disable Save all files to tape.

 Selection of a Volume Set, if the File Group is saved to tape.

 Enable or disable file fragmentation. If enabled, set the fragment size. File

fragmentation must be enabled to provide partial file restore and cartridge
spanning.

 Retention periods on magnetic disk, if the File Group is saved to tape.

The policies defined by the administrator determine how files are stored on disk and
tape. The policies do not change folders or files within the Windows file system. For
example, folders cannot be created by using File Group rules; new folders are created
by using standard utilities like Windows Explorer.

Constructing the File Name, Path Pattern, or Exclude Pattern

Whenever a program creates a file, the nTier File Migrator software needs to know
how to treat it. The way the file should be handled is defined by File Group rules, so
the first thing the system does when a file is opened or created is to allocate it to a File
Group.

24

File Groups

The system maintains a list of File Groups, each of which has a File name or path
pattern. To establish which File Group to use for a particular file, the system starts at
the top of the list of File Groups and tries to match the file name to the pattern for the
File Group. If the file name matches the pattern for the first File Group then the file is
allocated to the first File Group. If the file name does not match the first path pattern,
the system tries the next File Group in the list, and so on down the list until it finds a
match or reaches the end of the list. If it reaches the end of the list, the system blocks
opening or creation of the file, returning instead an error to the application that tried
to open the file.

Files are allocated to File Groups based on their folder name, file name, file name
extension in any combination. Standard file name and wildcard conventions (such as *
and ?) can be used during the pattern match. As an extension to normal pattern
matching syntax, the special directory wild card ... can be used to match any number of
intermediate sub-folders. The system supports multiple patterns per File Group,
separated by semicolons.

Character Definition

* (that is, an asterisk) any number and type of character; often used before a file

extension, such as *.png

? (that is, a question
mark)

... (that is, three period

characters)

a single alphabetic or numeric character in the file name, such as
image09?.png, that represents image090 - image099

subdirectories \ subfolders, such as \financials\...\*.xls represents
all subfolders under the financials directory and file names that
end in xls (such as
\financials\july\payroll.xls

\financials\july\vacation.xls
\financials\august\payroll.xls
\financials\august\vacation.xls

Some examples file name or path patterns are:

example results

*.tif Selects files with the extension .tif for the File Group.

abc???.tif Selects files that start with abc, have the extension .tif and have

six characters before the extension.

\Images\* Selects files that are in the folder \Images.

\Images\...\*

\Images\...\*.tif

Selects files that are in the folder \Images or any of its sub-folders.

Selects files with the extension .tif that are in the folder \Images
or any of its sub-folders.

25

Chapter 2 – Concepts

Examples of Allocating Files to File Groups

The easiest way to illustrate how files are allocated to File Groups is by way of
examples, shown below. In each case, files are allocated to three different File Groups.

Example 1 One File Group contains all files with names ending in .tif; a second File

Group is for all files with names ending in .txt; and a third File Group contains all
other files.

File Group File name or path pattern Exclude pattern

File Group 1 *.tif

File Group 2 *.txt

File Group 3 *

In this example, no path has been specified and consequently the file name rules apply
to all files written to the logical drive managed by Archive Series software, no matter to
which folder they are written. Note that the Exclude pattern fields are blank in this
example. Note also that we used * instead of *.* in File Group 3 to ensure that all files
are included in the File Group including those without a name extension. This set
includes any files that do not have the extensions tif and txt.

Example 2 One File Group contains all files written to a folder at the root called

\project01\; another contains all files written to a folder at the root called \project02\;
and a third File Group contains all other files.

File Group File name or path pattern Exclude pattern

File Group 1 \project01\*

File Group 2 \project02\*

File Group 3 *

Note that Exclude pattern field is left blank in this example. Note also that we used *
instead of *.* to ensure that all files are included in the file group including those
without a name extension.

26

File Groups

Example 3 This is similar to example 2, but includes all sub-folders for the first two

File Groups. One File Group contains all files written to a folder at the root called
\project01\ and its sub-folders; another contains all files written to a folder at the root
called \project02\ and its sub-folders; and a third File Group contains all other files.

File Group File name or path pattern Exclude pattern

File Group 1 \project01\...\*

File Group 2 \project02\...\*

File Group 3 *

In this example, the use of …\ denotes all folders below the specified path.

Example 4 This is similar to example 3, but all temporary files are excluded from the

first two File Groups by using the Exclude pattern field. Consequently, all file names
ending in .tmp are allocated to the third File Group.

File Group File name or path pattern Exclude pattern

File Group 1 \project01\...\* *.tmp

File Group 2 \project02\...\* *.tmp

File Group 3 *

Example 5 This is similar to example 4, where all temporary files are excluded from

the first two File Groups by using the Exclude pattern field. As for example 4, all file
names ending in .tmp are allocated to the third File Group. However, the administrator
has not set up a catch-all File Group rule at the bottom of the File Group list. In this
case, the system does not write files unless they are written to the folders proect01\,
project02\ or their sub-folders or the files end with *.tmp.

File Group File name or path pattern Exclude pattern

File Group 1 \project01\...\* *.tmp

File Group 2 \project02\...\* *.tmp

File Group 3 *.tmp

27

Chapter 2 – Concepts

Example 6 This example illustrates the importance of File Group order.

File Group File name or path pattern Exclude pattern

File Group 1 \project01\*

File Group 2 *.txt

File Group 3 *

In this example, files ending in .txt in folder project01\ are allocated to the same File
Group as the other files in this folder. If the order of the first two rules was changed,
files ending in .txt would be allocated to the same File Group as the.txt files in the
other folders.

Tapes: Volumes, Volume Sets, and Automatic
Replication

The nTier File Migrator software can automatically create multiple tape cartridge
replicas. For this software, the term “Volume” refers a complete set of replica tapes
which, when up-to-date, all contain the same data. If replication is not enabled a
Volume refers to an individual tape cartridge.

A Volume Set stores files from designated File Groups and consists of one or more
Volumes. As more data is written to a Volume Set, the initial Volume eventually
becomes full. At a preset threshold, defined by the administrator, the system
automatically adds another Volume by taking the appropriate number of tapes from the
blank media set and extends the Volume Set.

The data on replicated tapes in a Volume Set are kept synchronized whenever the tapes
are available to the system. If one or more tapes in a Volume Set are removed from the
library, the system maintains a record of which files need to be written to those tapes.
When tapes are reintroduced into the library, the data on them is automatically brought
up to date.

With the exception of the blank media set and quarantined media set, all media within
a Volume Set must be either WORM tape or rewritable. Tape replicas must all be of the
same capacity.

One unique Volume Set, termed the blank media set, contains all the tapes that are
recognized by the system but are not formatted for storing data. These may be new
(unused) tapes or rewritable tapes that have been reformatted by the system
administrator.

28