Quantum TIMECODE-BASED PARTIAL FILE RETRIEVAL User Manual
User’s Guide
StorNext Timecode-Based
Partial File Retrieval 1.0.1
6-66957-02 Rev A
Document Title, 6-66957-02 Rev A, January 2011, Product of USA.
Quantum Corporation provides this publication “as is” without warranty of any kind, either express or
implied, including but not limited to the implied warranties of merchantability or fitness for a particular
purpose. Quantum Corporation may revise this publication from time to time without notice.
COPYRIGHT STATEMENT
© 2011 Quantum Corporation. All rights reserved.
Your right to copy this manual is limited by copyright law. Making copies or adaptations without prior
written authorization of Quantum Corporation is prohibited by law and constitutes a punishable
violation of the law.
TRADEMARK STATEMENT
Quantum, the Quantum logo, DLT, DLTtape, the DLTtape logo, Scalar, and StorNext are registered
trademarks of Quantum Corporation, registered in the U.S. and other countries.
Backup. Recovery. Archive. It’s What We Do., the DLT logo, DLTSage, DXi, DXi-Series, Dynamic
Powerdown, FastSense, FlexLink, GoVault, MediaShield, Optyon, Pocket-sized. Well-armored, SDLT,
SiteCare, SmartVerify, StorageCare, Super DLTtape, SuperLoader, and Vision are trademarks of Quantum.
LTO and Ultrium are trademarks of HP, IBM, and Quantum in the U.S. and other countries. All other
trademarks are the property of their respective companies.
Specifications are subject to change without notice.
ii Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Contents
Chapter 1 Introduction to Timecode-Based Partial File Retrieval 1
Chapter 2 How Timecode-Based Partial File Retrieval Works 3
Partial File Retrieval Indexer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Partial File Retrieval Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Folder Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Issuing a Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Monitoring Requests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
PFR Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Chapter 3 Installation 11
Installation Components and Install Options. . . . . . . . . . . . . . . . . . . . 11
Upgrading from an Earlier Version . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Uninstalling a Previous Version . . . . . . . . . . . . . . . . . . . . . . . . 12
Installing v1.0.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide iii
Contents
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Supported Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
StorNext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
VMWare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Bonjour. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
User Accounts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
.NET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Sharing the StorNext File System . . . . . . . . . . . . . . . . . . . . . . 18
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 4 Configuration 21
SNAPI / PFR Linux Component Configuration . . . . . . . . . . . . . . . . . . . 21
Ensuring that the VM is running. . . . . . . . . . . . . . . . . . . . . . . 21
Optional Step: Advanced Network Configuration . . . . . . . . . . 24
Showing the VM and its GUI. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Setting the Location of StorNext Storage Manager. . . . . . . . . 28
Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Primary Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Archive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Job Results Folder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Work Folders. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Transfer Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Indexer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Indexer Configuration and Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 40
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Licence Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Chapter 5 Using the Status GUI Application 45
Anatomy of Status GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Manipulating the Retrievals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
iv Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Contents
Chapter 6 Request EDL Format 47
Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Chapter 7 Performance and Scalability 49
Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Indexer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Note on Existing Archives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Partial File Retrieval via Index from Tape. . . . . . . . . . . . . . . . . 50
Very Short Partial Retrievals . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Longer Partial Retrievals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
File and Folder Structure Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Quantum PFR Indexer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Quantum PFR Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Distributing the Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Dedicated Tape Drives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Suggested Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide v
Contents
vi Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Figures
Figure 1 PFR Steps 1 - 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Figure 2 PFR Steps 3 - 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Figure 3 PFR Steps 6 - 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Figure 4 PFR Steps 8 - 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Figure 5 Server Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Figure 6 Add Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Figure 7 Install .NET Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 8 PFR Installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Figure 9 VMWare Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Figure 10 Virtual Machine Settings. . . . . . . . . . . . . . . . . . . . . . . . . . 23
Figure 11 Virtual Machine Inventory . . . . . . . . . . . . . . . . . . . . . . . . 24
Figure 12 Summary Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Figure 13 Network Adapter Configuration . . . . . . . . . . . . . . . . . . . . 26
Figure 14 Console Tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Figure 15 PFR GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 16 Edit SNAPI Configuration File . . . . . . . . . . . . . . . . . . . . . . 29
Figure 17 Network Settings Screen. . . . . . . . . . . . . . . . . . . . . . . . . . 30
Figure 18 Edit Network Settings Address Tab. . . . . . . . . . . . . . . . . . 31
Figure 19 Edit Network Settings Overview Tab. . . . . . . . . . . . . . . . . 32
Figure 20 Welcome Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Figure 21 Archive Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Figure 22 Job Results Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide vii
Figures
Figure 23 Work Folders Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Figure 24 Transfer Engines Screen . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Figure 25 PFR Indexer Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Figure 26 Summary Screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Figure 27 Complete Screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Figure 28 PFR Index Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Figure 29 Indexer Log Window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Figure 30 Status GUI Screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
viii Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Chapter 1
Introduction to Timecode-
Based Partial File Retrieval
Quantum Timecode-Based Partial File Retrieval (PFR) allows you to
quickly retrieve and utilize segments of large media files—rather than
the entire file— based on timecode parameters.
Segments of large media files that have been archived to tape using
Quantum StorNext Storage Manager can be requested using a simple
XML request file. Quantum Partial File Retrieval instructs StorNext to
retrieve only the necessary portion from tape, and then presents that
segment as a complete MXF or QuickTime file.
If the file has not yet been truncated by StorNext Storage Manager, the
result will still be the same. However, in this case the Partial File will be
generated directly from the complete media file on disk.
The Quantum Partial File Retrieval feature requires a dedicated
Windows-based server on which to run. For larger scale installations, the
Quantum Partial File Retrieval functionality can be distributed across
multiple Windows server machines.
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 1
Chapter 1: Introduction to Timecode-Based Partial File Retrieval
Note: Partial File Retrieval is designed to work exclusively with
QuickTime and MXF video files, which makes PFR suitable for
applications which reference video by timecode.
The QuickTime or MXF wrapper for a retrieved file will be reconstructed during the Partial File Retrieval process, and as
such retrieved files will not be byte-for-byte replicas of the
original. However, the video frames and audio samples in the
retrieved files are not transcoded and therefore there is no
generation loss from the retrieval process. The only exception
to this is for retrieving MPEG Long GOP video (for example
Sony XDCAM HD 50), where a small number of frames are
transcoded at the start and end of the retrieval in order to
reconstruct a coherent GOP structure.
2 Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
How Timecode-Based Partial
Quantum Timecode-Based Partial File Retrieval consists of two primary
components: an indexer which indexes media files before they are
archived to tape and truncated, and a Partial File Retrieval service that
performs the requested retrievals.
Partial File Retrieval Indexer
Chapter 2
File Retrieval Works
The Partial File Retrieval Indexer looks for new media to appear in
StorNext Storage Manager. When it finds a new media file it generates
an index for it, in effect mapping timecode values in the original media
file to byte offsets. This allows subsequent timecode- based requests to
retrieve only the necessary portions of a large media file from tape.
The index files, which are written as QuickTime reference files, are
generated in a parallel folder structure on the managed storage. The
two file locations below represent a MXF based media file and its
associated index file.
//StorNext/snfs1/Media/Project/Movie.mxf
//StorNext/snfs1/PFR‐INDEX/Media/Project/Movie.mov
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 3
Chapter 2: How Timecode-Based Partial File Retrieval Works
Partial File Retrieval Service
Note: The index files do not contain any content themselves, and are
therefore very small in comparison with the files they index.
It is important that the indexed files themselves are never
truncated.
The Indexer does not attempt to index files that it thinks might
be currently in-record. Therefore, the first time the indexer
encounters the file, there may be some delay before a
particular file is indexed while the indexer is trying to establish
if the file is currently being written to.
Partial File Retrieval Service
Folder Structure Within the Quantum Partial File Retrieval Configuration Tool, it is
possible to set up multiple sets of folders for different request types. For
example, a set of folders for news requests might look like this:
//StorNext/snfs1/Requests/News
//StorNext/snfs1/Requests/News/PFR‐REQUEST
//StorNext/snfs1/Requests/News/success
//StorNext/snfs1/Requests/News/fail
The configuration tool creates these folders for you and allows you to
set an output format: QuickTime Movie or MXF.
The four folders shown in the example are used in the following
manner:
1 The parent folder (in this case
extract will be placed once retrieved and created;
2 The
3 Once the request has been completed, the XML request file will be
4 Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
PFR-REQUEST
request file;
moved to the
folder is the location to place the XML-formatted
success
folder;
News
) is where the resultant file
Chapter 2: How Timecode-Based Partial File Retrieval Works
Partial File Retrieval Service
4 If an error occurred in retrieving the partial file, the XML request file
will be moved to the
fail
folder.
Issuing a Request A Partial File Retrieval request is issued in the form of a small XML file,
the format of which can be seen in the example later in this User Guide.
The XML file simply describes a movie (via a full UNC path) with an in
and out timecode value.
Quantum PFR will handle a number of the requests simultaneously with
the overall retrieval performance dependant upon the quantity of tape
drives available.
The number of tape drives limits how many requests can be processed
simultaneously. Once all tape drives are busy, additional requests will be
queued until a tape drive becomes available.
Monitoring Requests Requests will be processed automatically once the request file is
detected. The Partial File Retrieval service automatically detects the XML
request file, appends the processed request to the queue of pending
requests, and processes them in the order received. XML request files
are moved to the
status of the request.
success
or
fail
folder as appropriate based on the
Use the PFR status GUI to monitor requests, cancel pending or active
requests or change the priority of the pending requests.
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 5
Chapter 2: How Timecode-Based Partial File Retrieval Works
Index
XX
PFR-Request
1
2
PFR Workflow
PFR Workflow
This section illustrates the steps involved in the retrieval process. Circled
numbers in the illustrations correspond to the step number.
Figure 1 PFR Steps 1 - 2
6 Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Step 1: An MXF (or QuickTime) file is written into the managed file
system.
Step 2: Frames are read from this new file and converted into a
QuickTime reference index file.
Figure 2 PFR Steps 3 - 5
Index
XX
PFR-Request
3
4
5
<xml><xml><xml>
<xml><xml><xml>
Chapter 2: How Timecode-Based Partial File Retrieval Works
PFR Workflow
Step 3: Some time later, according to the policies that have been
configured, the Storage Manager copies the file to the tape archive and
truncates the file on disk.
Step 4: A third-party media asset management (MAM) system
generates an XML request for a partial file and writes it into the request
folder.
Step 5: PFR picks up the request XML and starts to parse it.
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 7
Chapter 2: How Timecode-Based Partial File Retrieval Works
Index
XX
PFR-Request
<xml><xml><xml>
01101011010
6
7
PFR Workflow
Figure 3 PFR Steps 6 - 7
Step 6: PFR locates the required frames in the index file and converts
these to byte offset data for Storage Manager.
Step 7: Storage Manager performs a partial restore from tape, and
writes a temporary file to the destination directory.
8 Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Figure 4 PFR Steps 8 - 9
Index
XX
PFR-Request
<xml><xml><xml>
8
9
<xml><xml><xml>
Chapter 2: How Timecode-Based Partial File Retrieval Works
PFR Workflow
Step 8: PFR then immediately rewraps this temporary file back into a
valid MXF or QuickTime file.
Step 9: The request XML is dropped into the success folder to indicate
that a successful restore has occurred.
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 9
Chapter 2: How Timecode-Based Partial File Retrieval Works
PFR Workflow
10 Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide
Installation
Installation Components and Install Options
Quantum Partial File Retrieval consists of the following software
components:
• PFR Control Server and Configuration GUI: This is the core of the
system and the tool that configures it
• PFR Transfer Engine: This is the engine that retrieves and creates
the partial file
• PFR Status GUI: This is the user application which allows
monitoring and modifying the queue of pending and in-progress
retrievals
Chapter 3
• Index Engine and Configuration / Status GUI: This is the service
that indexes media files before they are truncated
• Virtual Machine (VM) with SNAPI and PFR Linux Component:
This is the interface to the StorNext API (SNAPI)
A typical installation of Quantum Partial File Retrieval installs all of the
above components on a single Windows-based server machine. This is
fine for most installations, but for larger installations the components
can be distributed as follows:
• PFR Server: Includes "PFR Control Server and Configuration GUI" and
"VM w/ SNAPI and PFR Linux Component"
Quantum Timecode-Based Partial File Retrieval 1.0.1 User’s Guide 11
Loading...