Apple Qmaster User Manual
Apple Qmaster
Distributed Processing Setup Guide
Copyright © 2009 Apple Inc. All rights reserved.
Your rights to the software are governed by the
accompanying software license agreement. The owner or
authorized user of a valid copy of Final Cut Studio or
Logic Studio software may reproduce this publication for
the purpose of learning to use such software. No part of
this publication may be reproduced or transmitted for
commercial purposes, such as selling copies of this
publication or for providing paid for support services.
The Apple logo is a trademark of Apple Inc., registered in
the U.S. and other countries. Use of the “keyboard” Apple
logo (Shift-Option-K) for commercial purposes without
the prior written consent of Apple may constitute
trademark infringement and unfair competition in violation
of federal and state laws.
Every effort hasbeen made to ensure thatthe information
in this manual is accurate. Apple is not responsible for
printing or clerical errors.
Note: Because Apple frequently releases new versions
and updates to its system software, applications, and
Internet sites,images shownin this manualmay beslightly
different from what you see on your screen.
Apple
1 Infinite Loop
Cupertino, CA 95014
408-996-1010
www.apple.com
Apple, the Apple logo, Bonjour, Final Cut, Final Cut Pro,
Final Cut Studio, FireWire, Logic, Logic Studio, Mac, Mac OS,
QuickTime, Shake, Xsan, and Xserve are trademarks of
Apple Inc., registered in the U.S. and other countries.
Apple Remote Desktop, Finder, and NetInfo are trademarks
of Apple Inc.
AppleCare is a service mark of Apple Inc., registered in the
U.S. and other countries.
Adobe, the Adobe logo, Acrobat, the Acrobat logo,
Distiller, PostScript, andthe PostScriptlogo are trademarks
or registered trademarks of Adobe Systems Incorporated
in the U.S. and/or other countries.
“Dolby,” “Pro Logic,” and the double-D symbol are
trademarks of Dolby Laboratories. Confidential
Unpublished Works,© 1992–1997Dolby Laboratories, Inc.
All rights reserved.
UNIX is a registered trademark of The Open Group.
Other company and product names mentioned herein
are trademarks of their respective companies. Mention of
third-party products is for informational purposes only
and constitutes neither an endorsement nor a
recommendation. Apple assumes no responsibility with
regard to the performance or use of these products.
Production stills from the film “Koffee House Mayhem”
provided courtesy of Jean-Paul Bonjour. “Koffee House
Mayhem” © 2004 Jean-Paul Bonjour. All rights reserved.
http://www.jeanpaulbonjour.com
Contents
Welcome to Distributed Processing5Preface
About the Apple Qmaster Distributed Processing System5
About the Distributed Processing Documentation6
Additional Resources6
Distributed Processing Basics9Chapter 1
Using Distributed Processing to Increase Speed and Efficiency9
Basic Components of the Apple Qmaster Distributed Processing System10
QuickClusters13
Compressor AutoClusters13
How the Apple Qmaster System Distributes Batches13
Getting Started Quickly15Chapter 2
Quick and Easy Distributed Processing15
About the Compressor Service Sharing Pane19
Preparing a Network for Distributed Processing21Chapter 3
The Minimum You Need to Know21
Example of a Minimal Distributed Processing Network22
Using Compressor AutoClusters23
Other Possible Components of a Distributed Processing Network24
Example of an Expanded Distributed Processing Network26
Sample Setup for Part-Time Processing on Desktop Computers26
Configuring Access for Part-Time Distributed Processing27
Setting Up for Part-Time Distributed Processing with Shake29
Additional Steps for Distributed Processing with Shake30
The Interfaces33Chapter 4
The Interfaces in the Apple Qmaster Distributed Processing System33
Apple Qmaster Pane of System Preferences35
Apple Qadministrator41
Compressor and Apple Qmaster as Client Interfaces43
Batch Monitor45
3
Creating Clusters47Chapter 5
An Overview of Configuring a Cluster47
Configuring Service Nodes and Cluster Controllers48
Creating Clusters with Apple Qadministrator55
About QuickClusters58
Administering Clusters61Chapter 6
Modifying and Deleting Clusters with Apple Qadministrator61
Monitoring Cluster Activity63
Accessing Activity Logs63
Setting Preferences63
Setting Passwords67
Scheduling Service Availability68
Using Cluster Storage70
Defining Ports for Service Advertisements73
Recovery and Failure Notification Features73
Using the Command Line77Chapter 7
Installing Apple Qmaster from the Command Line77
Using the Command Line for Distributed Processing78
Using Scripts to Run Apple Qmaster, Compressor, and Batch Monitor85
Solving Problems87Appendix
Resources for Solving Problems87
Solutions to Common Problems87
Contacting AppleCare Support92
4 Contents
Welcome to Distributed Processing
Transcoding or processing a series of large files on one desktop computer is processor
intensive and time-consuming. You can increase speed and productivity by distributing
processing across multiple computers.
This preface covers the following:
• About the Apple Qmaster Distributed Processing System (p. 5)
• About the Distributed Processing Documentation (p. 6)
• Additional Resources (p. 6)
About the Apple Qmaster Distributed Processing System
High-volume processing is sometimes addressed by carefully managing multiple
computers; technicians set up batches of processing tasks for each computer and then
monitor their progress, collect and route the processed files, and start over again with
new batches. Although this is an improvement over the single-computer method, the
resource and process management can be laborious and slow.
Preface
The Apple Qmaster distributed processing system provides a more efficient solution,
handling all the work distribution and processing for you, behind the scenes.
Apple Qmaster and the Apple Qmaster features of Compressor manage the processing
across designated computers. They subdivide the work for speed, route the work to the
computers with the most available computing power, and direct the processing.
The Apple Qmaster system uses the processing capacity of your network computers for
a wide range of tasks, including transcoding and rendering for digital visual effects software
packages such as Apple Shake, Adobe After Effects, and Autodesk Maya.
5
About the Distributed Processing Documentation
The Apple Qmaster distributed processing system is included with every copy of
Final Cut Studio and Shake. The system comes with various documents that help you get
started as well as provide detailed information about each of the applications. (To access
onscreen help for an application, open the application and choose the application’s help
from the Help menu.)
• Distributed Processing Setup Guide: This manual is the primary documentation for the
Apple Qmaster distributed processing system. The manual explains how to set up and
maintain a distributed processing network for use with Compressor or digital visual
effects software packages. It includes information on preparing a distributed processing
network, creating and administering clusters of computers, monitoring batches, and
using the system from the command line.
Although all of the chapters should be useful to administrators, client users (users who
are not administrators) may need to refer only to the Compressor User Manual or the
Apple Qmaster User Manual, and possibly to the introductory information in this
document, to understand how to use the Apple Qmaster distributed processing system
for their purposes.
The Distributed Processing Setup Guide is available in Compressor Help, Apple Qmaster
Help, and Apple Qadministrator Help.
• Compressor User Manual: This is a comprehensive document that describes the
Compressor interface, commands, and menus and gives step-by-step instructions for
using Compressor and accomplishing specific tasks. It is written for users of all levels
of experience. The Compressor User Manual is available in Compressor Help.
• Batch Monitor User Manual: This brief document describes how to use the Batch Monitor
application to monitor the transcoding progress of batches and jobs. The Batch Monitor
User Manual is available in Batch Monitor Help, Compressor Help, and Apple Qmaster
Help.
• Apple Qmaster User Manual: This document describes the Apple Qmaster client
application only. This is a specialized application for submitting jobs with Shake,
Autodesk Maya, and other digital visual effects software packages. The Apple Qmaster
User Manual is available in Apple Qmaster Help.
Additional Resources
Along with the documentation that comes with the Apple Qmaster software, there are
a variety of other resources you can use to find out more about the Apple Qmaster
distributed processing system.
6 Preface Welcome to Distributed Processing
Final Cut Studio Website
For general information and updates, as well as the latest news on Final Cut Studio, go
to:
• http://www.apple.com/finalcutstudio
Compressor Website
For general information and updates, as well as the latest news on Compressor, go to:
• http://www.apple.com/finalcutstudio/compressor
Apple Service and Support Websites
For software updates and answers to the most frequently asked questions for all Apple
products, go to the general Apple Support webpage. You’ll also have access to product
specifications, reference documentation, and Apple and third-party product technical
articles.
• http://www.apple.com/support
For software updates, documentation, discussion forums, and answers to the most
frequently asked questions for Compressor, go to:
• http://www.apple.com/support/compressor
For discussion forums for all Apple products from around the world, where you can search
for an answer, post your question, or answer other users’ questions, go to:
• http://discussions.apple.com
7Preface Welcome to Distributed Processing
Distributed Processing Basics
1
This section introduces most of the basic concepts and terms related to using the
Apple Qmaster distributed processing system.
This chapter covers the following:
• Using Distributed Processing to Increase Speed and Efficiency (p. 9)
• Basic Components of the Apple Qmaster Distributed Processing System (p. 10)
• QuickClusters (p. 13)
• Compressor AutoClusters (p. 13)
• How the Apple Qmaster System Distributes Batches (p. 13)
Using Distributed Processing to Increase Speed and Efficiency
Distributed processing accelerates processing by distributing the work to multiple
computers that have been chosen to provide more processing power. You can submit
batches of processing jobs to the Apple Qmaster distributed processing system, which
allocates those jobs to other computers in the most efficient way (described in more
detail in How the Apple Qmaster System Distributes Batches).
9
Computers that submit batches to the Apple Qmaster distributed processing system are
Batch of
processing jobs
Client computer
Jobs are submitted.
Destination folder
Files are placed at
specified destination.
Processed
files
Apple Qmaster cluster
Processing is
performed by cluster.
Network
called clients. A job is a processing task such as a Compressor preset-source pair, or a
Shake file, or other files or commands that use UNIX commands to specify settings such
as rendering instructions and file locations and destinations.
A batch is one or more jobs submitted for processing at one time. The procedure is
analogous to printing multipage documents from a word processing program; the files
are spooled and processed in the background. Although a batch can include just one
job, you will typically want to submit several jobs at once for processing. Similarly, several
people can use the same Apple Qmaster system at the same time, with several client
computers sending batches in the same time frame. Batches are managed and distributed
by the computer that is designated as the Apple Qmaster cluster controller, which is
described in the next section.
10 Chapter 1 Distributed Processing Basics
Basic Components of the Apple Qmaster Distributed Processing System
Although the Apple Qmaster software includes a few different applications (see The
Interfaces), as a whole it is part of a networked system that includes the following basic
components.
• Client(s): The computer or computers that use Compressor or Apple Qmaster to submit
jobs for distributed processing. Applications that can use Apple Qmaster services for
processing include Compressor, Shake, Autodesk Maya, and many UNIX command-line
programs.
• An Apple Qmaster cluster: An Apple Qmaster cluster contains:
• Service nodes: The computers that perform the processing of batches submitted via
Compressor or Apple Qmaster. A batch can include one or more jobs.
• Cluster controller: The software, enabled on a computer by means of the
Cluster controller manages the
distribution of client’s jobs across the cluster
Service node
processes jobs
Cluster
(can contain multiple service nodes, but only one cluster controller)
Client computer from
which users send jobs to the cluster
Service node
processes jobs
Service node
processes jobs
Service node
processes jobs
Apple Qmaster pane of System Preferences, that divides up batches, determines
which service nodes to send work to, and generally tracks and directs the processes.
The client computer, the service nodes, and the cluster controller are often on separate
(but network-connected) computers, for the most rapid processing potential. However,
the cluster controller could be on a client computer or a service node. See Using One
Computer to Serve Two Distributed Processing Roles for more information on this scenario.
Following is a closer look at the part each component plays in the Apple Qmaster system.
Clients
Batches are submitted for distributed processing from the client computers. A client
computer can be any computer that has Compressor or Apple Qmaster installed and is
on the same network (subnet) as the cluster controller. Multiple client computers can be
on the same subnet, using the same cluster to do the processing for various applications.
You use Compressor or the Apple Qmaster application to submit batches to be processed
for a client. See the Compressor User Manual and the Apple Qmaster User Manual for details
on using these applications.
Note: On any given Final Cut Studio system or network, the Compressor and
Apple Qmaster versions (numbers) and the version of QuickTime must all match. For
example, Compressor 3.0.5 will work only with the 3.0.5 version of Apple Qmaster, and
no other version. In this example, the correct version of QuickTime is QuickTime 7.5.5 or
later. Non-matching configurations are not supported.
11Chapter 1 Distributed Processing Basics
Clusters
Cluster
controller
Service
node
Service
node
Service
node
Example of a cluster
When a client sends batches to the Apple Qmaster distributed processing system, all the
processing and subsequent moving of any output files is performed by a group of
Apple Qmaster–configured computers called a cluster. You can create one ormore clusters
of service nodes, with one cluster controller included in each cluster. Each computer in
the cluster is connected to the other computers in the cluster through a network
connection.
Note: This illustration provides only one simple example of a cluster. Other possibilities
are described in Preparing a Network for Distributed Processing.
Service Nodes
The service nodes are where the processing work is done. When you assign a group of
service nodes to a cluster, they function as one very powerful computer because all their
resources are shared. If one service node is overloaded or otherwise inaccessible, another
service node is used.
12 Chapter 1 Distributed Processing Basics
You make a computer available as a service node by configuring it in the Apple Qmaster
pane of System Preferences. The simple steps involved in using System Preferences to
configure a service node are described in Administering Clusters.
Note: The terms processing and rendering will come up frequently as you read this
document. The term processing is used here in a general way to cover both rendering
(for Shake and other frame-based rendering applications) and encoding (or transcoding
or compression) for Compressor. For more information, see the Shake User Manual and
the Compressor User Manual.
Cluster Controllers
The cluster controller software acts as the manager of a cluster. The cluster controller
directs the distribution of batches within the cluster. It has the ability to determine the
best use of the cluster resources based on work and availability variables. (See How the
Apple Qmaster System Distributes Batches for more details.) The cluster controller is
responsible for accepting batch submissions,maintaining and managing thebatch queue,
and doling out the work to the appropriate service node. It also tracks the status of all
outstanding batches.
You make a computer available as a cluster controller by turning on the cluster–controlling
service in the Apple Qmaster pane of System Preferences.
QuickClusters
The QuickCluster feature of the Apple Qmaster distributed processing system is a simple
and automated alternative to creating and configuring clusters manually. For more
information about QuickClusters, see Creating QuickClusters.
Compressor AutoClusters
The AutoCluster feature in Compressor gives you an easy way to take advantage of the
distributed processing capabilities offered by Apple Qmaster without requiring a lot of
knowledge about how clusters are configured, setting up file sharing, and so on. For more
information about AutoClusters, see Using Compressor AutoClusters.
How the Apple Qmaster System Distributes Batches
The Apple Qmaster cluster controller determines the most efficient use of the cluster
resources. It makes this determination based on the availability of each service node and
the number of separable parts (described next) of the batch.
Because Apple Qmaster subdivides individual batches across different service nodes, the
work is shared and completed more quickly. And, because this method uses all the service
nodes as much as possible, you avoid under-utilizing your resources.
Batches can be distributed to a cluster by the cluster controller in one or both of the
following ways. (Apple Qmaster determines which way is the most efficient for specific
batches, depending on the circumstances.)
• The batch is subdivided into data segments: For example, for a render batch, the cluster
controller could divide the frames into groups (segments). Each segment would be
processed in parallel on the service nodes in the cluster.
• The batch is subdivided into tasks: For example, for a render batch, the cluster controller
could subdivide the rendering work into different processing tasks. Different tasks
would be run on different service nodes.
13Chapter 1 Distributed Processing Basics
Rather than actually moving segments, Apple Qmaster tells the service nodes which
Batch submitted by
Compressor or
Apple Qmaster (job request
for frames 1–30)
Cluster controller
divides and distributes
job to available
service nodes
Service node 1
Service node 2
Service node 3
Instructions specifying
locations of source files
and frames 11–20
Instructions specifying
locations of source files
and frames 21–30
Instructions specifying
locations of source files
and frames 01–10
Processed file (frames 1-30)
placed in specified
destination
segments to read via the network, where to find them, and what to do with them. Below
is an example of how one batch could be processed in an Apple Qmaster system.
In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate
services in a cluster on the same IP subnet and to dynamically share and receive
information. Because the computers can continually transmit their current processing
availability status, Apple Qmaster can distribute (load-balance) the workload evenly across
14 Chapter 1 Distributed Processing Basics
the cluster.
Getting Started Quickly
2
The Apple Qmaster distributed processing system has default settings that allow you to
use distributed processing immediately.
This chapter covers the following:
• Quick and Easy Distributed Processing (p. 15)
• About the Compressor Service Sharing Pane (p. 19)
Quick and Easy Distributed Processing
The steps below describe the simplest and quickest way to start using distributed
processing.
• Stage 1: Installing the Software
• Stage 2: Configuring a QuickCluster
• Stage 3: Adding Service Nodes to a Cluster
• Stage 4: Creating and Submitting a Batch
Stage 1: Installing the Software
Install the appropriate software on the computers you want to include in your distributed
processing network. Each computer in the network requires Apple Qmaster and/or
Compressor software.
To install the software
1 Make sure the client software is on at least one computer in your network.
In order to submit jobs and batches to the distributed processing system, you need to
use client (submission) software (either Compressor or Apple Qmaster). If you are reading
this, you have probably already installed one or both of these. For further information,
see the installation booklet that came with either of these applications.
2 Install Apple Qmaster software on each computer you want to use for distributed
processing. (All computers must be on the same subnet.) By default, the Final Cut Studio
installer installs the Apple Qmaster software necessary for distributed processing.
15
However, in most standard distributed processing networks, you need to install
Apple Qmaster software on every computer in the network. You can use the standalone
Apple Qmaster Node Installer to install the Apple Qmaster software on additional
computers on your network.
a Locate the folder containing the Apple Qmaster software.
• If you are using Compressor: Insert the Final Cut Studio installation disc and locate
the Apple Qmaster Node Installer package (AppleQmasterNode.mpkg).
• If youare usingShake: Insert the Shake installation disc and locate the Apple Qmaster
Node Installer package (AppleQmasterNode.mpkg).
b Do one of the following:
• Double-click the AppleQmasterNode.mpkg installer package, and follow the onscreen
instructions. By default, this installer installs all the necessary Apple Qmaster software.
• Install Apple Qmaster software from the command line. See Installing Apple Qmaster
from the Command Line for more information.
Both the Final Cut Studio Installer and the Apple Qmaster Node Installer include the
Compressor Service Sharing pane, in which you can quickly set up the computer as an
unmanaged service node. See About the Compressor Service Sharing Pane for more
information.
Note: On any given Final Cut Studio system or network, the Compressor and
Apple Qmaster versions (numbers) and the version of QuickTime must all match. For
example, Compressor 3.0.5 will work only with the 3.0.5 version of Apple Qmaster, and
no other version. In this example, the correct version of QuickTime is QuickTime 7.5.5 or
later. Non-matching configurations are not supported.
Keep the following useful tips in mind as you prepare your Apple Qmaster distributed
processing system:
• If you are using the Apple Qmaster distributed processing system with Compressor or
Shake, all nodes (computers) in a cluster must have the same version of QuickTime
installed. To download the most recent version of QuickTime, go to
http://www.apple.com/quicktime/download/mac.html.
• In order to use the Apple Qmaster distributed processing system to process Dolby
Digital Professional audio, each node (computer) in your distributed processing network
must have Final Cut Studio installed.
• Shake users can create distributed processing clusters containing computers that do
not have any Apple Qmaster software installed. See the Apple Qmaster User Manual for
more information.
16 Chapter 2 Getting Started Quickly
Stage 2: Configuring a QuickCluster
Use the Apple Qmaster pane of System Preferences to configure the cluster controller
and service node computers.
To configure the cluster controller and service node computers
1 Open System Preferences.
2 Click the Apple Qmaster button, located in the Other section.
The Apple Qmaster pane appears.
3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter
the administrator name and password.
4 Click Start Sharing.
This creates a QuickCluster with this computer as its controller, and an instance of
processing services for each processor on the computer. For information about multiple
service instances, see Using Virtual Clusters to Make the Most of Multiprocessor Computers.
Stage 3: Adding Service Nodes to a Cluster
Do the following on each computer that you would like to make a service node on your
cluster.
To add a service node to a cluster
1 Open System Preferences.
17Chapter 2 Getting Started Quickly
2 Click the Apple Qmaster button, located in the Other section.
The Apple Qmaster pane appears.
3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter
the administrator name and password.
4 Select “Services only.”
5 Click Start Sharing.
This creates a service node that will automatically process jobs submitted to the
QuickCluster you set up in Stage 2: Configuring a QuickCluster.
Stage 4: Creating and Submitting a Batch
In the Compressor Batch window or in the Apple Qmaster window, create a batch with
one or more jobs. See the Compressor User Manual or the Apple Qmaster User Manual for
details.
To submit the batch
1 Do one of the following:
• If you are using Compressor: In the Compressor Batch window, click Submit, and in the
resulting dialog, do one of the following:
• Use the Cluster pop-up menu to choose the cluster you created in stage 2.
• Leave the Cluster menu set to the default This Computer setting, and select the
“Include unmanaged services on other computers” checkbox. For more information
on this option, see Using Compressor AutoClusters.
18 Chapter 2 Getting Started Quickly
• If you are using Apple Qmaster: In the Apple Qmaster window, choose the cluster you
created in stage 2 from the Submit To pop-up menu.
2 Click Submit.
The distributed processing system processes the batch.
For more advanced information on creating and controlling clusters and services, see
any of the following:
• Basic Components of the Apple Qmaster Distributed Processing System
• Sample Setup for Part-Time Processing on Desktop Computers
• The Interfaces in the Apple Qmaster Distributed Processing System
• Apple Qmaster Pane of System Preferences
• Apple Qadministrator
• An Overview of Configuring a Cluster
• Configuring Service Nodes and Cluster Controllers
• Creating Clusters with Apple Qadministrator
• About QuickClusters
• Modifying and Deleting Clusters with Apple Qadministrator
About the Compressor Service Sharing Pane
Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the
Compressor Service Sharing pane, in which you can quickly set up the computer as an
unmanaged service node. Selecting the Yes option in this pane is the first of two steps
in setting up AutoClusters in Compressor. An AutoCluster is a temporary QuickCluster
that you create automatically when you submit a batch. For more information about
AutoClusters, see Using Compressor AutoClusters.
Note: You can change any of these settings at a later time in the Apple Qmaster pane of
System Preferences.
If you are unsure which role each computer will play in your network, just install the
Apple Qmaster software on each computer in your network. You can sort out the details
later. See Basic Components of the Apple Qmaster Distributed Processing System for
additional information on what roles individual computers can play in the distributed
processing system.
19Chapter 2 Getting Started Quickly
Preparing a Network for Distributed Processing
A distributed processing network can consist of as few as one or two computers, whereas
a high-volume network may include many computers, an Xserve system and Xserve
cluster nodes in a rack, and high-speed networking infrastructures.
You can scale up a distributed processing system as your workload demands by adding
features and devices to the network that supports it.
This chapter covers the following:
• The Minimum You Need to Know (p. 21)
• Example of a Minimal Distributed Processing Network (p. 22)
• Using Compressor AutoClusters (p. 23)
• Other Possible Components of a Distributed Processing Network (p. 24)
• Example of an Expanded Distributed Processing Network (p. 26)
• Sample Setup for Part-Time Processing on Desktop Computers (p. 26)
• Configuring Access for Part-Time Distributed Processing (p. 27)
• Setting Up for Part-Time Distributed Processing with Shake (p. 29)
• Additional Steps for Distributed Processing with Shake (p. 30)
3
The Minimum You Need to Know
The following are the basic rules for setting up a distributed processing network:
• A cluster must contain one (and only one) computer acting as the cluster controller,
and at least one computer acting as the service node. (These two can be the same
computer, as shown in Example of a Minimal Distributed Processing Network.)
• The client computers and the computers in any cluster that supports them must be
on the same network.
• The network must support the Apple networking technology built in to Mac OS X.
• All the computers in a cluster need read-and-write access to any computers (or storage
devices) that will be specified as output destinations for files.
21
For information about a very basic setup, see Example of a Minimal Distributed Processing
Client
Service node with
cluster controller enabled
Minimum setup for distributed processing
Network.
Example of a Minimal Distributed Processing Network
A very small distributed processing setup could include as few as two computers:
• One computer connected to the client and configured to act as both the service node
and the cluster controller
• One client computer
Though simple, this setup is useful in a small-scale environment because it allows the
client computer to offload a lot of processing work.
See Example of an Expanded Distributed Processing Network for an illustration of a more
powerful setup.
22 Chapter 3 Preparing a Network for Distributed Processing
Using One Computer to Serve Two Distributed Processing Roles
To maximize your resources, you may want to consider using some computers for more
than one distributed processing function.
• Service node andcluster controller: In a small setup, one of the service nodes in a cluster
can also act as the cluster controller so that it performs both functions. However, in
a clusterof many service nodes, theprocessing load required for the cluster controller
could be so high that it would not be efficient to use one computer as both a service
node and a cluster controller.
• Client computer and cluster controller or service node: You could also set up a client
computer to act as a cluster controller or service node in a cluster, but again, keep in
mind that the more available processing power a computer has, the faster it can
manage or process jobs.
Using Compressor AutoClusters
The AutoCluster feature in Compressor gives you an easy way to take advantage of the
distributed processing capabilities offered by Apple Qmaster without requiring a lot of
knowledge about how clusters are configured, how to set up file sharing, and so on.
Using AutoCluster is a two-step process:
• Automatically creating Apple Qmaster service nodes as you install Final Cut Studio or
Apple Qmaster
• Selecting “Include unmanaged services on other computers” when you submit a
Compressor batch for processing
These two steps let you harness the processing power of any number of computers on
your network without any additional effort or knowledge on your part.
• Stage 1: Creating Service Nodes During Installation
• Stage 2: Submitting Compressor Batches for Processing
Stage 1: Creating Service Nodes During Installation
Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the
Compressor Service Sharing pane, in which you can quickly set up the computer as an
unmanaged Apple Qmaster service node, which is essentially a free agent “processing
slave,” available on the network for any computer that requests its services.
If you select Yes in this pane, the Apple Qmaster pane of System Preferences is configured
as follows:
• Share this computer as: Services only
• Compressor service: Selected, unmanaged
• Start Sharing: Started
23Chapter 3 Preparing a Network for Distributed Processing
You can change any of these settings at a later time in the Apple Qmaster pane of System
Preferences. For more information, see Apple Qmaster Pane of System Preferences and
Administering Clusters.
Important: Although the Final Cut Studio installer requires a separate serial number for
each computer, you can use the Apple Qmaster Node Installer (available in the Extras
folder) to install the necessary Apple Qmaster software to make any computer on your
network into an Apple Qmaster service node.
Stage 2: Submitting Compressor Batches for Processing
When you submit a Compressor batch for processing, a dialog appears that allows you
to name the batch and select the computers to process the batch. At this point, you can
harness the processing power of all the computers you set up as service nodes in stage
1 to complete the batch.
To process the batch using AutoCluster
1 Leave the Cluster pop-up menu set to the default selection (This Computer).
2 Select the “Include unmanaged services on other computers” checkbox.
Compressor and Apple Qmaster coordinate the distribution of the processing tasks
between the available computers and deposit the resulting output files at the location(s)
you designated in Compressor.
Other Possible Components of a Distributed Processing Network
There are many ways to expand the capacity of a distributed processing network. You
could include any of the following.
• High-speed switch and cables: A 100Base-T or Gigabit Ethernet switch and compatible
cables to allow your data to move over the LAN at maximum speed.
• Multiple clients: Multiple client computers can use the services of the same cluster. And,
you can have multiple client applications on the same client computer, using the same
cluster.
24 Chapter 3 Preparing a Network for Distributed Processing
• Multiple clusters: Depending on how extensive your network is and how many clients
it needs to serve, you may want to divide up available computers and create more than
one cluster to serve various clients. (Users select the cluster they want to send a batch
to when they submit the batch.)
• Multiple service nodes: In general, more service nodes mean more processing power.
In deciding how many service nodes to have in a cluster, consider the ratio of data
movement time to computing time. If the processing demand is greater than the
network demand required to move job segments throughout the cluster, as is the case
with rendering, more service nodes are a good idea. If the computing load, per job, is
closer to the network load, having a smaller number of service nodes per cluster may
be more efficient. If you are using the Apple Qmaster distributed processing system
with applications other than Shake or Compressor, consult the application’s user manual
on how to optimize the number of service nodes.
• Storage device: A storage device, such as a remote disk or group of disk arrays, can be
used as cluster scratch storage, which is a place for short-term storage of temporary
data generated by the cluster controller, clients, and service nodes. (You set the scratch
storage location in the Apple Qmaster pane of System Preferences. See Using Cluster
Storage.) Alternatively, a storage device can be used as a final destination for the files
after they are processed.
For more information, see Example of an Expanded Distributed Processing Network.
25Chapter 3 Preparing a Network for Distributed Processing
Example of an Expanded Distributed Processing Network
Client Client Client Client
Example of a network setup for distributed rendering
High-speed switches (chained together)
Rack of servers containing
cluster controller and service nodes,
plus rack-mounted shared storage device
LAN
For rendering, a network might include a number of client computerson a LAN, connected
to a cluster using a high-speed switch. A rack of servers plus a shared storage device,
acting as the cluster, would be an extremely strong rendering engine. The service nodes
would each have a local copy of the relevant client application software so that they
could process the rendering jobs.
Sample Setup for Part-Time Processing on Desktop Computers
This section takes you through the basic steps involved in a sample setup for “part-time”
distributed processing. You can use it to get an idea of the kind of distributed processing
environment you want to create, and as a guide in setting up that environment.
26 Chapter 3 Preparing a Network for Distributed Processing
This setup is for an environment that uses desktop computers. It is called “part-time”
Network
FireWire
drive
FireWire
drive
FireWire
drive
FireWire
drive
FireWire
drive
Each computer acts as both
a client that submits jobs for processing and
a service node that performs the processing.
All source and output files are stored on the FireWire drives.
processing because each computer acts as someone’s workstation, but at the same time
is also part of the distributed processing cluster. The bulk of the processing jobs can be
submitted with Compressor or Apple Qmaster at the end of the day, so that the computers
are busy processing a large queue of distributed processing batches after everyone has
gone home. (See Scheduling Service Availability in the Work Schedule Dialog for
information on scheduling service node availability.)
In the sample setup shown above, five computers act as both the clients (user workstations
from which users submit jobs for distributed processing) and cluster computers (which
do the processing). Each computer has an additional volume, such as a FireWire drive,
that is used for media before and after it is rendered, and for the associated files.
For more information on how to configure this desktop distributed processing
environment, see Configuring Access for Part-Time Distributed Processing.
Configuring Access for Part-Time Distributed Processing
Follow the instructions below to set up a “part-time” distributed processing system.
Before you get started, keep these essentials in mind:
• The cluster computers (cluster controller and service nodes) and the client computers
(user workstations) need to be on the same local network (subnet).
27Chapter 3 Preparing a Network for Distributed Processing
• All the computers in the setup need read-and-write access to any volumes that will be
specified as the source location or output destination for files, including Shake scripts.
(An appropriate way to configure this access is included in the sample configuration
procedure below.)
Stage 1: Installing Compressor and/or Apple Qmaster
The necessary components of Compressor or Apple Qmaster need to be installed on each
computer. See Stage 1: Installing the Software for more information.
On any given Final Cut Studio system or network, the Compressor and Apple Qmaster
versions (numbers) and the version of QuickTime must all match. For example,
Compressor 3.0.5 will work only with the 3.0.5 version of Apple Qmaster, and no other
version. In this example, the correct version of QuickTime is QuickTime 7.5.5 or later.
Non-matching configurations are not supported.
Important: The Compressor distributed processing feature for Dolby Digital Professional
(AC-3) and Final Cut Pro is limited to computers that have Final Cut Studio installed.
Stage 2: Setting Up Media Storage
Make sure each computer has the necessary available storage, preferably a dedicated
media volume. For example, give each computer a second volume, such as a FireWire
drive, that is used for media. Each computer can use this volume for all source and
destination files associated with distributed processing.
If you are a Compressor user, you can skip to one of the following chapters to complete
the setup of your part-time distributed processing system:
• Getting Started Quickly
• Administering Clusters
If you are a Shake user and you cannot consolidate all of the necessary source files (Shake
scripts, media files, and so on) on a single-cluster storage volume, skip to AdditionalSteps
for Distributed Processing with Shake before going on to stage 3.
Stage 3: Creating a Cluster
First, use the Apple Qmaster pane of System Preferences to enable cluster controlling on
one of the computers and enable the processing services on all the computers (making
them service nodes). Then, you can assemble these computers as a cluster.
Apple Qadministrator may not be necessary. See Administering Clusters for detailed
instructions.
28 Chapter 3 Preparing a Network for Distributed Processing
Setting Up for Part-Time Distributed Processing with Shake
If you cannot consolidate all of the necessary source files (Shake scripts, media files, and
so on) on a single-cluster storage volume, follow the instructions below.
• Stage 1: Turning Off the Shake UNC Setting
• Stage 2: Turning On Personal File Sharing
• Stage 3: Mounting the Media Storage Volumes
Stage 1: Turning Off the Shake UNC Setting
To make sharing and volume mounting work smoothly in this setup, you need to turn
off the Shake UNC setting on each computer. The UNC setting uses the entire file
pathname, with the network address, in a convention that starts with
//ComputerName/DriveName/path. You don’t want Shake to use this filenaming convention
because it conflicts with the file sharing and volume mounting used in this setup.
Note: All the media volumes created in stage 2 of Configuring Access for Part-Time
Distributed Processing should have the same name.
In the three steps below, you make this change in a Shake startup .h file. As described in
the Shake documentation, the startup .h files, located in the startup directory, are used
to customize Shake settings (similar to setting preferences).
To turn off the UNC setting, do the following on each of the computers
1 Log in as the user who will use Shake on the computer.
2 Double-click the Terminal icon in /Applications/Utilities/ to open a Terminal window.
3 Enter these two command lines in the Terminal window, pressing Return after each
command line:
mkdir -p ~/nreal/include/startup/
echo 'script.uncFileNames = 0;' > ~/nreal/include/startup/UNC_off.h
Stage 2: Turning On Personal File Sharing
On each computer, turn on Personal File Sharing. This allows the computers to share the
media volumes.
To turn on Personal File Sharing
1 Open System Preferences.
2 Click Sharing.
3 Select the File Sharing checkbox.
Stage 3: Mounting the Media Storage Volumes
Follow the instructions below so that all the computers in the cluster are mounting all
the media volumes in the cluster.
29Chapter 3 Preparing a Network for Distributed Processing
Loading...