Quantum DATAMGR 3.5 User Manual
DataMgr
Overview
DataMgr Version 3.5
January, 2000
Document Number 600920
Trademark Notice
AMASS, EMASS, DataMgr, FileServ, and VolServ are either trademarks or registered
trademarks of ADIC, Advanced Digital Information Corporation. DAS is a trademark of
Grau, an ADIC subsidiary. All other product names and identifications are trademarks or
registered trademarks of their respective manufacturers.
Copyright Notice
Copyright © 1996-2000 by ADIC. All rights reserved. This document is the property of
ADIC. No part of this document may be reproduced, transmitted, transcribed, stored in a
retrieval system, or translated into any language or computer language in any form or by
any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise,
without the express written permission of:
ADIC
10949 East Peakview Ave.
Englewood, CO 80111 USA
Phone: 303-792-9700
FAX: 303-792-2465
U.S. Government Rights Restricted
Use, duplication, or disclosure of either the software or documentation is subject to
restrictions set forth by the U.S. Government in FAR 52.227-19(c)(2) and subparagraph
(c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 52.2277013 and/or in similar or following clauses in the FAR, DoD, or NASA FAR Supplement.
Technical Assistance
ADIC Technical Assistance Center:
• In the USA and Canada, call 1-800-827-3822.
• Outside the USA and Canada, call 303-874-0188 or toll-free 00800-9999-3822.
• Send e-mail to: techsup@adic.com.
Documentation
Although the material contained herein has been carefully reviewed, ADIC does not
warrant it to be free of errors or omissions. We reserve the right to make corrections,
updates, revisions, or changes to the information contained herein.
• Send e-mail to: swpubs@adic.com
READER COMMENT FORM
ADIC includes this Form in an effort to provide the best possible documentation to our
customers. Please take a few moments to mail or FAX your response to:
ADIC
Software Documentation
10949 East Peakview Ave.
Englewood, CO 80111
FAX: 303-792-2465
Email: swpubs@adic.com
Question Circle One
Information was complete. Agree Disagree
Information was easy to find. Agree Disagree
Information was easy to follow. Agree Disagree
Is there anything you especially like or dislike about the organization, presentation,
or writing in this manual?_______________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
Book Title Document Number
Customer Name Telephone
E-mail Address
Company Name
Address
City, State, Zip
NOTES
Contents
Purpose of This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3
Who Should Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3
How This Book is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-3
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-4
Managing
Client Files with DataMgr 1
Using DataMgr to Manage Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
BFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5
SLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5
DMFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-6
BFS Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-7
BFS Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Bitfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-8
Removing Bitfile IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-12
Managed File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-13
Resident File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-13
Nonresident File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-14
Shadowed File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-16
i
DataMgr Overview
Storage Policies 2
Migration Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-3
Files Not Migrated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-3
Reload Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-4
Storage Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Using Two Instances of AMASS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-8
Automated Migration Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
File Import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10
Local Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2-11
Technical Support 3
Phone Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3
Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3
Solutions Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-3
Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Online Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-4
Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-5
Contact Publications Department . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3-5
Secured Web Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Who’s On First? A
Migration Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3
ii Contents
Purpose of This Book . . . . . . . . . . . . . . . . . . . . . . .P-3
Who Should Read This Book. . . . . . . . . . . . . . . . P-3
How This Book is Organized . . . . . . . . . . . . . . . P-3
Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .P-4
Preface
P
Preface
DataMgr Overview
NOTES
P-2 Preface
DataMgr Overview
Preface
Purpose of
This Book
Who Should
Read This
Book
This book provides an introduction or high-level summary of
DataMgr, an hierarchical storage management (HSM)
application.
HSM is a data management strategy where data is migrated to
storage in either a layered or serial method based on a set of
policies. A paradigm that often controls this migration is
frequency of access. For example, the most frequently accessed
files are first migrated onto expensive quick-access optical
platters. However, as data is less frequently accessed, the files
are stored onto cheaper magnetic tape. The goal for
implementing an HSM strategy is to provide clients with a
seemingly infinite storage capacity and to decrease the overall
cost of storage.
This book is written for prospective customers as well as the
system administrator and clients who will be using DataMgr.
How This
Book is
Organized
This book contains the following chapters:
Chapter 1: Managing Client Files with DataMgr— DataMgr
components and states of files in a file system.
Chapter 2: Storage Policies —Migration principles, storage
policies, and importing files from a foreign file system.
Chapter 3: Technical Support — What technical support is
available to you?
Preface P-3
DataMgr Overview
Appendix A: Who’s On First? — How does DataMgr selects
what files to automatically migrate?
Glossary — Defines terms.
Conventions
The conventions used throughout the DataMgr technical books
are listed below:
Convention Example
The word “library” is a generic way to
reference a storage device.
Screen text, file names, program names, and
commands are in Courier font.
The root prompt is shown as a number
symbol.
What you should type in is shown in Courier
bold font.
Site-specific variables are in a Times italics
font.
A backward slash ( \ ) denotes the input is
continued onto the next line; the printed page
is just not wide enough to accommodate the
line.
If using HP SunSpot libraries, install patch
1234.
Files/Dirs created for MFS
/mrktcol:
/mrktcol/Migration
/mrktcol/Migration/locklist
# su root
# cd /etc/dmfs/usr/utils
# dmfscntl -p /mfspath
# rsh nodename -n dd if=/dev \
/tapedevicename/bs=20b | tar \
xvfb - 20
Type the entire command without the
backward slashes.
Pressing <Return> after each command is
assumed.
A menu name with an arrow refers to a
sequence of menus or options.
P-4 Preface
Main Menu -—> Edit —>Add-—> Select
Policy
NOTES
Preface
DataMgr Overview
Preface P-5
DataMgr Overview
NOTES
P-6 Preface
Using DataMgr to Manage Files . . . . . . . . . . . . . 1-3
Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
BFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
1
SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5
SLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-5
DMFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-6
BFS Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-7
BFS Database . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-7
Bitfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1-8
Security for Bitfile IDs . . . . . . . . . . . . . . . .1-9
Modified Bitfiles . . . . . . . . . . . . . . . . . . . .1-10
Removing Bitfile IDs . . . . . . . . . . . . . . . . . . .1-12
Managed File Systems . . . . . . . . . . . . . . . . . . . . .1-13
Resident File . . . . . . . . . . . . . . . . . . . . . . . . . .1-13
Nonresident File . . . . . . . . . . . . . . . . . . . . . . .1-14
Stub Files . . . . . . . . . . . . . . . . . . . . . . . . . .1-14
Shadowed File . . . . . . . . . . . . . . . . . . . . . . . .1-16
Reloaded . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16
Prestaged . . . . . . . . . . . . . . . . . . . . . . . . . .1-17
Managing
Client Files
with DataMgr
ADIC
Manage Client
Files with DataMgr
DataMgr Overview
Roadmap
Topic
Describes:
• DataMgr components.
• States of files in a managed file system.
Describes:
• Migration principles.
• Storage policies.
• Importing files from a foreign file system.
Available technical assistance:
• Phone support.
• Training.
• Publications.
• Solutions group.
Understand how DataMgr selects what files to
automatically migrate.
Glossary
Refer To
Chapter
1
2
3
A
1-2 Managing Client Files with DataMgr
DataMgr Overview
Using
DataMgr to
Manage Files
DataMgr, in conjunction with AMASS, provides a method for
migrating client files — based on frequency of access and file
size — from fast, expensive media to slower, more economical
media.
DataMgr migrates files either:
• Automatically—initiated by a predefined storage policy in
response to “watermarks” on a client’s file system.
Watermarks define a percentage of a client’s file system that
should remain free.
• Manually—initiated by a client.
File migration and retrieval are transparent to clients. When a
client does an
migrated files are still listed because DataMgr leaves stub files
behind. Stub files contain all the information necessary to
access the migrated file.
ls on a directory in the managed file system,
Manage Client
Files with DataMgr
Managing Client Files with DataMgr 1-3
DataMgr Overview
Components
The DataMgr components — BFS, SSD, SLD, and DMFS —
communicate with each other through a variety of remote
procedure calls (RPCs) and data sockets.
Although there are a variety of combinations you can use to
install the DataMgr components, the figure belowillustrates just
one possibility.
DataMgr
Client
DataMgr and
AMASS installed
on a UNIX Server
BFS
SSD
SLD
AMASS
DMFS
1-4 Managing Client Files with DataMgr
Library
DataMgr Overview
BFS
SSD
Bitfile Server (BFS) maintains storage policies, assigns bitfile
IDs, and manages the licensing of DMFS clients. Install BFS on
a server that has network access to the DataMgr clients (DMFS)
and to the DataMgr daemons (SLD and SSD). A minimum of
one BFS component is required.
All client file systems do not have to be managed by the same
BFS. For example, if there are two BFSs on the network, the
client’s file system
while the file system
/techpubs can be managed by BFS_A,
/cdbooks is managed by BFS_B.
Multiple AMASS servers can be used as well. For additional
information, see “Using Two Instances of AMASS” on
page 2-8.
Storage Server Daemon (SSD) is the interface to AMASS from
the DMFS client so the client can migrate and reload files. The
SSD log — located in
/etc/dm/log/ssd/transfile —
contains all SSD-related file movement in the AMASS file
system. This includes file creates, deletes, and renames. Install
SSD on the same server where AMASS is installed.
Manage Client
Files with DataMgr
SLD
Service Locator Daemon (SLD) is a naming service that maps
the name of a DataMgr service (BFS, SSD) to the service
daemon’s network location. Install SLD on any machine with
network access to BFS, DMFS clients, and SSD. Only one SLD
component is required.
Managing Client Files with DataMgr 1-5
DataMgr Overview
DMFS
DataMgr File System (DMFS) provides file management
capability for each client machine it is installed on. Install this
component on all client machines with file systems that
DataMgr will manage. Each client machine must have network
access to the BFS, SLD, and SSD.
Note
The number of machines where DMFS can run is limited by
the number of licenses your company has purchased.
The DMFS uses the client’s Managed File System Database,
located under
/etc/dm/raima/dmfs, for storing information
used in generating a list of files to migrate.
1-6 Managing Client Files with DataMgr
DataMgr Overview
BFS Tasks
BFS Database
The Bitfile Server is responsible for the following tasks:
• Allocates bitfile IDs (BFID).
• Maintains mapping between BFID and file location on
AMASS by using the BFS Database.
• Manages retention time and contains the functionality to
remove stale bitfiles (the trashcan feature) when the
retention time expires. For additional information about the
trashcan, refer to System Administrator’s Guide to Using
DataMgr.
• Manage licenses and keeps track of how many clients are
authorized to access DataMgr.As a client requests a service,
it assigns a license to that specific client.
• Manages configuration files that define storage policies.
• Manages error recovery.
The BFS Database is located under /etc/dm/raima/bfs.
This Database contains the bitfile IDs for migrated files. A
bitfile ID points to a specific RID (record ID) on AMASS. This
is how DataMgr keeps track of where client files have been
migrated.
Manage Client
Files with DataMgr
Managing Client Files with DataMgr 1-7
DataMgr Overview
Bitfiles
DMFS Client
Stub
BFS & SLD SSD & AMASS
RID &
Bitfile
ID for
File A
BFS Database
Bitfile
for
File A
Library
File A
Bitfile is the term used to identify the migrated file on AMASS.
The BFS assigns each bitfile a bitfile ID that points to a specific
record ID (RID) on AMASS. The bitfile ID is never changed
and is never reused. Bitfile IDs are assigned from a very large
address space and are virtually impossible to guess.
If a client reloads and modifies a migrated file, a new bitfile
(that identified the new modified file) — as well as a new bitfile
ID — is created when the file is subsequently prestaged or
migrated to AMASS.
1-8 Managing Client Files with DataMgr
The figure below illustrates the bitfile concept.
Client’s File System
DataMgr Overview
MaryWork.txt
2. Stub file for
MarkWork.txt is left
behind after DataMgr
migrates file.
MaryWork.txt Bitfile ID
3. This bitfile ID points
toa specific RID (record
ID) on AMASS.
Security for Bitfile
IDs
DataMgr
file
1. MaryWork.txt file is
migrated by DataMgr.
AMASS
Record ID
(RID)
4. This RID points to the
bitfile, which is the
migrated file under the
mount point /archive.
MaryWork.txt bitfile
5. This bitfile stays on the
AMASS file system until
the retention time has
expired and the Trashcan
is dumped with this file in it.
Security for bitfile IDs is provided in the following ways:
• Guessing bitfile IDs is difficult because of the sparseness of
the bitfile ID space. There are many more possible bitfile
IDs than actual ones, and the actual ones are distributed
more or less evenly through the space of possible ones. It is
practically impossible for a user to generate a valid stub file
that points to someone else’s bitfile. (There is of course,
nothing wrong with someone knowing the bitfile ID of a
read-accessible file.)
Manage Client
Files with DataMgr
• No part of DataMgr, other than the routine that generates
bitfile IDs, is aware of, or interprets the fields of a bitfile ID.
Managing Client Files with DataMgr 1-9
DataMgr Overview
Modified Bitfiles DataMgr places the original file (the original file as opposed to
the modified file, which has been reloaded, modified, and then
newly remigrated to media in a library) into an
/archive/FMSclients/clientname/
managedfs
/.versions
directory on AMASS. Bitfiles under these directories have a
sequence number appended to the file name in the following
format: “.versions/filename@n.”
For example, a bitfile name for the original
workdata file that
has been reloaded, modified, and remigrated would look like
the following:
.versions/workdata@0
workdata bitfile for the modified file is under the
The
/archive/FMSclients/clientname/
workdata@0 bitfile for the original file is under the
the
versions directory.
.
managedfs
directory; while
1-10 Managing Client Files with DataMgr
First time file is migrated
from client’s file system.
This concept is illustrated below:
/archive/FMSclients/maui/techpubs
workdata.fm
workdata.fm
DataMgr Overview
Manage Client
Files with DataMgr
File is reloaded and
modified.
Again, file is reloaded
and modified.
/archive/FMSclients/maui/techpubs
workdata.fm
workdata.fm
/archive/FMSclients/maui/techpubs
workdata.fm
workdata.fm
Second time file is migrated
from client’s file system,
original file moves under
.versions directory and
gets @o extension.
/archive/FMSclients/maui/techpubs/.versions/
workdata.fm@0
Third time file is migrated
from client’s file system,
original bitfile name remains @0
and first modified file moves under
.versions directory and gets a @1 extension.
/archive/FMSclients/maui/techpubs/.versions/
workdata.fm@0
workdata.fm@1
workdata.fm@o
workdata.fm@o
workdata.fm@1
Caution
DataMgr Administrators: Do not remove any files under
/archive/FMSclients on the AMASS server. Deleting or
modifying these files will corrupt the BFS Database.
Managing Client Files with DataMgr 1-11
Loading...