Quantum DATAMGR 3.5 User Manual

DataMgr



Overview

DataMgr Version 3.5

January, 2000

Document Number 600920

Trademark Notice

AMASS, EMASS, DataMgr, FileServ, and VolServ are either trademarks or registered
trademarks of ADIC, Advanced Digital Information Corporation. DAS is a trademark of
Grau, an ADIC subsidiary. All other product names and identifications are trademarks or
registered trademarks of their respective manufacturers.

Copyright Notice

Copyright © 1996-2000 by ADIC. All rights reserved. This document is the property of
ADIC. No part of this document may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language or computer language in any form or by
any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise,
without the express written permission of:

ADIC
10949 East Peakview Ave.
Englewood, CO 80111 USA
Phone: [phone]
FAX: [phone]

U.S. Government Rights Restricted

Use, duplication, or disclosure of either the software or documentation is subject to
restrictions set forth by the U.S. Government in FAR 52.227-19(c)(2) and subparagraph
(c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 52.227­7013 and/or in similar or following clauses in the FAR, DoD, or NASA FAR Supplement.

Technical Assistance

ADIC Technical Assistance Center:

• In the USA and Canada, call 1-800-827-3822.

• Outside the USA and Canada, call 303-874-0188 or toll-free 00800-9999-3822.

• Send e-mail to: [e-mail].

Documentation

Although the material contained herein has been carefully reviewed, ADIC does not
warrant it to be free of errors or omissions. We reserve the right to make corrections,
updates, revisions, or changes to the information contained herein.

• Send e-mail to: [e-mail]

READER COMMENT FORM

ADIC includes this Form in an effort to provide the best possible documentation to our
customers. Please take a few moments to mail or FAX your response to:

ADIC
Software Documentation
10949 East Peakview Ave.
Englewood, CO 80111
FAX: [phone]
Email: [e-mail]

Question Circle One

Information was complete. Agree Disagree
Information was easy to find. Agree Disagree
Information was easy to follow. Agree Disagree

Is there anything you especially like or dislike about the organization, presentation,
or writing in this manual?_______________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

Book Title Document Number
Customer Name Telephone
E-mail Address
Company Name
Address
City, State, Zip

NOTES

Contents

Purpose of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3

Who Should Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3

How This Book is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-4

Managing
Client Files with DataMgr 1

Using DataMgr to Manage Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3

Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4

BFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5

SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5

SLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5

DMFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-6

BFS Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-7

BFS Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7

Bitfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-8

Removing Bitfile IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-12

Managed File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-13

Resident File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-13

Nonresident File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-14

Shadowed File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-16

i

DataMgr Overview

Storage Policies 2

Migration Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-3

Files Not Migrated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3

Reload Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-4

Storage Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4

Using Two Instances of AMASS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-8

Automated Migration Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9

File Import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10

Local Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-11

Technical Support 3

Phone Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3

Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3

Solutions Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3

Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4

Online Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-4

Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-5

Contact Publications Department . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-5

Secured Web Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5

Who’s On First? A

Migration Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3

ii Contents

Purpose of This Book . . . . . . . . . . . . . . . . . . . . . . .P-3

Who Should Read This Book. . . . . . . . . . . . . . . . P-3

How This Book is Organized . . . . . . . . . . . . . . . P-3

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-4

Preface

P

Preface

DataMgr Overview

NOTES

P-2 Preface

DataMgr Overview

Preface

Purpose of

This Book

Who Should

Read This

Book

This book provides an introduction or high-level summary of
DataMgr, an hierarchical storage management (HSM)
application.

HSM is a data management strategy where data is migrated to
storage in either a layered or serial method based on a set of
policies. A paradigm that often controls this migration is
frequency of access. For example, the most frequently accessed
files are first migrated onto expensive quick-access optical
platters. However, as data is less frequently accessed, the files
are stored onto cheaper magnetic tape. The goal for
implementing an HSM strategy is to provide clients with a
seemingly infinite storage capacity and to decrease the overall
cost of storage.

This book is written for prospective customers as well as the
system administrator and clients who will be using DataMgr.

How This

Book is

Organized

This book contains the following chapters:
Chapter 1: Managing Client Files with DataMgr— DataMgr

components and states of files in a file system.
Chapter 2: Storage Policies —Migration principles, storage

policies, and importing files from a foreign file system.
Chapter 3: Technical Support — What technical support is

available to you?

Preface P-3

DataMgr Overview

Appendix A: Who’s On First? — How does DataMgr selects
what files to automatically migrate?

Glossary — Defines terms.

Conventions

The conventions used throughout the DataMgr technical books
are listed below:

Convention Example

The word “library” is a generic way to
reference a storage device.

Screen text, file names, program names, and
commands are in Courier font.

The root prompt is shown as a number
symbol.

What you should type in is shown in Courier
bold font.

Site-specific variables are in a Times italics
font.

A backward slash ( \ ) denotes the input is
continued onto the next line; the printed page
is just not wide enough to accommodate the
line.

If using HP SunSpot libraries, install patch

1234.

Files/Dirs created for MFS
/mrktcol:
/mrktcol/Migration
/mrktcol/Migration/locklist

# su root

# cd /etc/dmfs/usr/utils

# dmfscntl -p /mfspath

# rsh nodename -n dd if=/dev \

/tapedevicename/bs=20b | tar \
xvfb - 20

Type the entire command without the
backward slashes.

Pressing <Return> after each command is
assumed.

A menu name with an arrow refers to a
sequence of menus or options.

P-4 Preface

Main Menu -—> Edit —>Add-—> Select
Policy

NOTES

Preface

DataMgr Overview

Preface P-5

DataMgr Overview

NOTES

P-6 Preface

Using DataMgr to Manage Files . . . . . . . . . . . . . 1-3

Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4

BFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5

1

SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5

SLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5

DMFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-6

BFS Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-7

BFS Database . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7

Bitfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-8

Security for Bitfile IDs . . . . . . . . . . . . . . . .1-9

Modified Bitfiles . . . . . . . . . . . . . . . . . . . .1-10

Removing Bitfile IDs . . . . . . . . . . . . . . . . . . .1-12

Managed File Systems . . . . . . . . . . . . . . . . . . . . .1-13

Resident File . . . . . . . . . . . . . . . . . . . . . . . . . .1-13

Nonresident File . . . . . . . . . . . . . . . . . . . . . . .1-14

Stub Files . . . . . . . . . . . . . . . . . . . . . . . . . .1-14

Shadowed File . . . . . . . . . . . . . . . . . . . . . . . .1-16

Reloaded . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16

Prestaged . . . . . . . . . . . . . . . . . . . . . . . . . .1-17

Managing

Client Files

with DataMgr

ADIC

Manage Client

Files with DataMgr

DataMgr Overview

Roadmap

Topic

Describes:

• DataMgr components.

• States of files in a managed file system.
Describes:

• Migration principles.

• Storage policies.

• Importing files from a foreign file system.
Available technical assistance:

• Phone support.

• Training.

• Publications.

• Solutions group.
Understand how DataMgr selects what files to

automatically migrate.
Glossary

Refer To

Chapter

1

2

3

A

1-2 Managing Client Files with DataMgr

DataMgr Overview

Using

DataMgr to

Manage Files

DataMgr, in conjunction with AMASS, provides a method for
migrating client files — based on frequency of access and file
size — from fast, expensive media to slower, more economical
media.

DataMgr migrates files either:

• Automatically—initiated by a predefined storage policy in
response to “watermarks” on a client’s file system.
Watermarks define a percentage of a client’s file system that
should remain free.

• Manually—initiated by a client.

File migration and retrieval are transparent to clients. When a
client does an
migrated files are still listed because DataMgr leaves stub files
behind. Stub files contain all the information necessary to
access the migrated file.

ls on a directory in the managed file system,

Manage Client

Files with DataMgr

Managing Client Files with DataMgr 1-3

DataMgr Overview

Components

The DataMgr components — BFS, SSD, SLD, and DMFS —
communicate with each other through a variety of remote
procedure calls (RPCs) and data sockets.

Although there are a variety of combinations you can use to
install the DataMgr components, the figure belowillustrates just
one possibility.

DataMgr
Client

DataMgr and
AMASS installed

on a UNIX Server

BFS

SSD

SLD

AMASS

DMFS

1-4 Managing Client Files with DataMgr

Library

DataMgr Overview

BFS

SSD

Bitfile Server (BFS) maintains storage policies, assigns bitfile
IDs, and manages the licensing of DMFS clients. Install BFS on
a server that has network access to the DataMgr clients (DMFS)
and to the DataMgr daemons (SLD and SSD). A minimum of
one BFS component is required.

All client file systems do not have to be managed by the same
BFS. For example, if there are two BFSs on the network, the
client’s file system
while the file system

/techpubs can be managed by BFS_A,

/cdbooks is managed by BFS_B.

Multiple AMASS servers can be used as well. For additional
information, see “Using Two Instances of AMASS” on

page 2-8.

Storage Server Daemon (SSD) is the interface to AMASS from
the DMFS client so the client can migrate and reload files. The
SSD log — located in

/etc/dm/log/ssd/transfile —

contains all SSD-related file movement in the AMASS file
system. This includes file creates, deletes, and renames. Install
SSD on the same server where AMASS is installed.

Manage Client

Files with DataMgr

SLD

Service Locator Daemon (SLD) is a naming service that maps
the name of a DataMgr service (BFS, SSD) to the service
daemon’s network location. Install SLD on any machine with
network access to BFS, DMFS clients, and SSD. Only one SLD
component is required.

Managing Client Files with DataMgr 1-5

DataMgr Overview

DMFS

DataMgr File System (DMFS) provides file management
capability for each client machine it is installed on. Install this
component on all client machines with file systems that
DataMgr will manage. Each client machine must have network
access to the BFS, SLD, and SSD.

Note

The number of machines where DMFS can run is limited by
the number of licenses your company has purchased.

The DMFS uses the client’s Managed File System Database,
located under

/etc/dm/raima/dmfs, for storing information

used in generating a list of files to migrate.

1-6 Managing Client Files with DataMgr

DataMgr Overview

BFS Tasks

BFS Database

The Bitfile Server is responsible for the following tasks:

• Allocates bitfile IDs (BFID).

• Maintains mapping between BFID and file location on
AMASS by using the BFS Database.

• Manages retention time and contains the functionality to
remove stale bitfiles (the trashcan feature) when the
retention time expires. For additional information about the
trashcan, refer to System Administrator’s Guide to Using
DataMgr.

• Manage licenses and keeps track of how many clients are
authorized to access DataMgr.As a client requests a service,
it assigns a license to that specific client.

• Manages configuration files that define storage policies.

• Manages error recovery.

The BFS Database is located under /etc/dm/raima/bfs.
This Database contains the bitfile IDs for migrated files. A
bitfile ID points to a specific RID (record ID) on AMASS. This
is how DataMgr keeps track of where client files have been
migrated.

Manage Client

Files with DataMgr

Managing Client Files with DataMgr 1-7

DataMgr Overview

Bitfiles

DMFS Client

Stub

BFS & SLD SSD & AMASS

RID &
Bitfile
ID for

File A

BFS Database

Bitfile

for
File A

Library

File A

Bitfile is the term used to identify the migrated file on AMASS.
The BFS assigns each bitfile a bitfile ID that points to a specific

record ID (RID) on AMASS. The bitfile ID is never changed
and is never reused. Bitfile IDs are assigned from a very large
address space and are virtually impossible to guess.

If a client reloads and modifies a migrated file, a new bitfile
(that identified the new modified file) — as well as a new bitfile
ID — is created when the file is subsequently prestaged or
migrated to AMASS.

1-8 Managing Client Files with DataMgr

The figure below illustrates the bitfile concept.

Client’s File System

DataMgr Overview

MaryWork.txt

2. Stub file for
MarkWork.txt is left
behind after DataMgr
migrates file.

MaryWork.txt Bitfile ID

3. This bitfile ID points
toa specific RID (record
ID) on AMASS.

Security for Bitfile
IDs

DataMgr

file

1. MaryWork.txt file is
migrated by DataMgr.

AMASS

Record ID
(RID)

4. This RID points to the
bitfile, which is the
migrated file under the
mount point /archive.

MaryWork.txt bitfile

5. This bitfile stays on the
AMASS file system until
the retention time has
expired and the Trashcan
is dumped with this file in it.

Security for bitfile IDs is provided in the following ways:

• Guessing bitfile IDs is difficult because of the sparseness of
the bitfile ID space. There are many more possible bitfile
IDs than actual ones, and the actual ones are distributed
more or less evenly through the space of possible ones. It is
practically impossible for a user to generate a valid stub file
that points to someone else’s bitfile. (There is of course,
nothing wrong with someone knowing the bitfile ID of a
read-accessible file.)

Manage Client

Files with DataMgr

• No part of DataMgr, other than the routine that generates
bitfile IDs, is aware of, or interprets the fields of a bitfile ID.

Managing Client Files with DataMgr 1-9

DataMgr Overview

Modified Bitfiles DataMgr places the original file (the original file as opposed to

the modified file, which has been reloaded, modified, and then
newly remigrated to media in a library) into an

/archive/FMSclients/clientname/

managedfs

/.versions

directory on AMASS. Bitfiles under these directories have a
sequence number appended to the file name in the following
format: “.versions/filename@n.”

For example, a bitfile name for the original

workdata file that

has been reloaded, modified, and remigrated would look like
the following:

.versions/workdata@0

workdata bitfile for the modified file is under the

The

/archive/FMSclients/clientname/

workdata@0 bitfile for the original file is under the

the

versions directory.

.

managedfs

directory; while

1-10 Managing Client Files with DataMgr

First time file is migrated
from client’s file system.

This concept is illustrated below:

/archive/FMSclients/maui/techpubs

workdata.fm

workdata.fm

DataMgr Overview

Manage Client

Files with DataMgr

File is reloaded and
modified.

Again, file is reloaded

and modified.

/archive/FMSclients/maui/techpubs

workdata.fm

workdata.fm

/archive/FMSclients/maui/techpubs

workdata.fm

workdata.fm

Second time file is migrated
from client’s file system,

original file moves under

.versions directory and

gets @o extension.

/archive/FMSclients/maui/techpubs/.versions/

workdata.fm@0

Third time file is migrated
from client’s file system,

original bitfile name remains @0
and first modified file moves under

.versions directory and gets a @1 extension.

/archive/FMSclients/maui/techpubs/.versions/

workdata.fm@0

workdata.fm@1

workdata.fm@o

workdata.fm@o

workdata.fm@1

Caution

DataMgr Administrators: Do not remove any files under

/archive/FMSclients on the AMASS server. Deleting or
modifying these files will corrupt the BFS Database.

Managing Client Files with DataMgr 1-11