Apple COMPRESSOR 3, QMASTER 3 Distributed Processing Setup

Apple Qmaster 3 and Compressor 3
Distributed Processing Setup
K
Apple Inc.
Copyright © 2007 Apple Inc. All rights reserved.
Your rights to the software are governed by the accompanying software license agreement. The owner or authorized user of a valid copy of Final Cut Studio software may reproduce this publication for the purpose of learning to use such software. No part of this publication may be reproduced or transmitted for commercial purposes, such as selling copies of this publication or for providing paid for support services.
The Apple logo is a trademark of Apple Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Shift-Option-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.
Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.
Note:
Because Apple frequently releases new versions and updates to its system software, applications, and Internet sites, images shown in this book may be slightly different from what you see on your screen.
Apple Inc. 1 Infinite Loop Cupertino, CA 95014–2084 408-996-1010 www.apple.com
Apple, the Apple logo, Bonjour, Final Cut, Final Cut Pro, Final Cut Studio, FireWire, Mac, Mac OS, QuickTime, Shake, Xsan, and Xserve are trademarks of Apple Inc., registered in the U.S. and other countries.
Apple Remote Desktop and Finder are trademarks of Apple Inc.
AppleCare and Apple Store are service marks of Apple Inc., registered in the U.S. and other countries.
Dolby Laboratories: Manufactured under license from Dolby Laboratories. “Dolby,” “Pro Logic,” and the double­D symbol are trademarks of Dolby Laboratories. Confidential Unpublished Works, © 1992–1997 Dolby Laboratories, Inc. All rights reserved.
UNIX is a registered trademark of The Open Group.
Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.
1

Contents

Preface 5 Introduction to Distributed Processing
5
Using Distributed Processing to Increase Speed and Efficiency
6
Basic Components of the Apple Qmaster Distributed Processing System
9
QuickClusters
9
Compressor AutoClusters
9
How the Apple Qmaster System Distributes Batches
11
About the Distributed Processing Setup Guide
12
Apple Websites
Chapter 1 15 Getting Started Quickly
15
Quick and Easy Distributed Processing
Chapter 2 21 Preparing a Network for Distributed Processing
21
The Minimum You Need to Know
23
Using Compressor AutoClusters
25
Other Possible Components of a Distributed Processing Network
27
Sample Setup for Part-Time Processing on Desktop Computers
29
Additional Steps For Part-Time Distributed Processing With Shake
Chapter 3 31 The Interfaces
31
The Interfaces in the Apple Qmaster Distributed Processing System
33
Apple Qmaster Pane in System Preferences
35
Apple Qadministrator
37
Client Interfaces: Compressor and Apple Qmaster
39
Batch Monitor
3
Chapter 4 41 Creating and Administering Clusters
42
An Overview of Configuring a Cluster
42
Configuring Service Nodes and Cluster Controllers
50
Creating Clusters with Apple Qadministrator
53
About QuickClusters
54
Advanced Settings in the Apple Qmaster Preferences Pane
58
Modifying and Deleting Clusters With Apple Qadministrator
59
Monitoring Cluster Activity
59
Accessing Activity Logs
60
Setting Cluster Preferences
61
Setting Passwords and Scratch Storage
65
Recovery and Failure Notification Features
Appendix A 67 Command-Line Usage
Appendix B 77 Troubleshooting
Index 83
4
Contents
Introduction to Distributed Processing
Rendering a series of large files on one desktop computer is processing intensive and time consuming. You can increase speed and productivity by distributing processing across multiple computers.
High-volume processing is sometimes addressed by carefully managing multiple computers; technicians set up batches of processing tasks for each computer, and then monitor their progress, collect and route the processed files, and start over again with new batches. While this is an improvement over the single-computer method, the resource and process management can be laborious and slow.
The Apple Qmaster distributed processing engine provides a more efficient solution, handling all the work distribution and processing for you, behind the scenes. Apple Qmaster and the Apple Qmaster features of Compressor manage the processing across designated computers. They subdivide the work for speed, route the work to the computers with the most available computing power, and direct the processing.
Preface

Using Distributed Processing to Increase Speed and Efficiency

Distributed processing
computers that have been chosen to provide more processing power. You can submit batches of processing jobs to Apple Qmaster, which allocates those jobs to other computers in the most efficient way (described in more detail in “How the
Apple Qmaster System Distributes Batches” on page 9).
accelerates processing by distributing the work to multiple
5
.
Computers that submit batches to the Apple Qmaster distributed processing system are called
clients.
A
job
is a processing task such as a Compressor preset-source pair, or a Shake file, or other file or commands, that use UNIX commands to specify settings such as rendering instructions and file locations and destinations.
Network
Batch of
processing jobs
Client computer
Jobs are submitted.
A
batch
is one or more jobs submitted for processing at one time. The procedure is
Apple Qmaster cluster
Processing is
performed by cluster.
Processed
files
Destination folder
Files are placed at
specified destination
analogous to printing multi-page documents from a word processing program; the files are spooled and processed in the background. Although a batch can include just one job, you will typically want to submit several jobs at once for processing. Similarly, several people can use the same Apple Qmaster system at the same time, with several client computers sending batches in the same time frame. Batches are managed and distributed by the computer that is designated as the Apple Qmaster
cluster controller,
which is described in the next section.
Basic Components of the Apple Qmaster Distributed Processing System
While the Apple Qmaster software includes a few different applications (see Chapter 3, “The Interfaces,” on page 31), as a whole it is part of a networked system that includes the following basic components:
Â
Client(s):
submit jobs for distributed processing. Applications that can use Apple Qmaster services for processing include Compressor, Shake, Alias Maya, and many UNIX command-line programs.
The computer or computers that use Compressor or Apple Qmaster to
6 Preface
Introduction to Distributed Processing
Â
Apple Qmaster cluster,
Â
Service nodes:
which contains:
The computers that perform the processing of batches submitted
via Compressor or Apple Qmaster. A batch can include one or more jobs.
Â
Cluster controller:
The software, enabled on a computer by means of the Apple Qmaster pane in System Preferences, that divides up batches, determines which service nodes to send work to, and generally tracks and directs the processes.
Client computer from
which users send jobs to the cluster
(can contain multiple service nodes, but only one cluster controller)
Cluster controller manages the
distribution of client’s jobs across the cluster
Service node
processes jobs
Service node
processes jobs
Cluster
Service node
processes jobs
Service node
processes jobs
The client computer, the service nodes, and the cluster controller are often on separate (but network-connected) computers, for the most rapid processing potential. However, the cluster controller could be on a client computer or a service node. See “Using One
Computer to Serve Two Distributed Processing Roles” on page 22 for more information
on this scenario.
Following is a closer look at the part each component plays in the Apple Qmaster system.

Clients

Batches are submitted for distributed processing from the computer can be any computer that has Compressor or Apple Qmaster installed and is on the same network (subnet) as the cluster controller. Multiple client computers can be on the same subnet, using the same cluster to do the processing for various applications.
You use Compressor or the Apple Qmaster application to submit batches to be processed for a client. See the
User Manual
for details on using these applications.
Compressor User Manual
client computers
and the
Apple Qmaster
. A client
Preface
Introduction to Distributed Processing
7

Clusters

When a client sends batches to the Apple Qmaster distributed processing system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster–configured computers called a clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection.
Example of a cluster
cluster.
You can create one or more
Cluster
controller
Note:
Service
node
Service
node
Service
node
This illustration provides only one simple example of a cluster. Other possibilities
are described in Chapter 2, “Preparing a Network for Distributed Processing.”
Service Nodes
The service nodes are where the processing work is done. When you assign a group of service nodes to a cluster, they function as one very powerful computer because all their resources are shared. If one service node is overloaded or otherwise inaccessible, another service node is used.
You make a computer available as a service node by configuring it in the Apple Qmaster pane in System Preferences. The simple steps involved in using System Preferences to configure a service node are described in Chapter 4, “Creating
and Administering Clusters,” on page 41.
Note:
The terms
processing
and
rendering
will come up frequently as you read this document. The term processing is used here generically to cover both rendering (for Shake, and other frame-based rendering applications) and encoding (or transcoding or compression) for Compressor. For more information see the
Compressor User Manual
.
Shake User Manual
and the
8 Preface
Introduction to Distributed Processing
Cluster Controllers
The cluster controller software acts as the manager of a cluster. The cluster controller directs the distribution of batches within the cluster. It has the ability to determine the best use of the cluster resources based on work and availability variables. (See “How
the Apple Qmaster System Distributes Batches,” next, for more details.) The cluster
controller is responsible for accepting batch submissions, maintaining and managing the batch queue, and doling out the work to the appropriate service node. It also tracks the status of all outstanding batches.
You make a computer available as a cluster controller by turning on the cluster­controlling service in the Apple Qmaster pane in System Preferences.

QuickClusters

The QuickCluster feature of the Apple Qmaster distributed processing system is a simple and automated alternative to creating and configuring clusters manually. For more information about QuickClusters, see “Creating QuickClusters” on page 53.

Compressor AutoClusters

The AutoCluster feature in Compressor gives you an easy way to take advantage of the distributed processing capabilities offered by Apple Qmaster without requiring a lot of knowledge about how clusters are configured, setting up file sharing, and so on. For more information about AutoClusters, see “Using Compressor AutoClusters” on page 23.
How the Apple Qmaster System Distributes Batches
The Apple Qmaster cluster controller determines the most efficient use of the cluster resources. It makes this determination based on the availability of each service node and the number of separable parts (described next) of the batch.
Because Apple Qmaster subdivides individual batches across different service nodes, the work is shared and completed more quickly. And, because this method uses all the service nodes as much as possible, you avoid under-utilizing your resources.
Preface
Introduction to Distributed Processing
9
Batches can be distributed to a cluster by the cluster controller in one or both of the following ways. (Apple Qmaster determines which way is the most efficient for specific batches, depending on the circumstances.)
Â
The batch is subdivided into data segments:
cluster controller could divide the frames into groups (
For example, for a render batch, the
segments
). Each segment
would be processed in parallel on the service nodes in the cluster.
Â
The batch is subdivided into tasks:
For example, for a render batch, the cluster controller could subdivide the rendering work into different processing tasks. Different tasks would be run on different service nodes.
Rather than actually moving segments, Apple Qmaster tells the service nodes which segments to read via the network, where to find them, and what to do with them. Below is an example of how one batch could be processed in an Apple Qmaster system.
Batch submitted by
Compressor or
Apple Qmaster (job request
for frames 1–30)
Cluster controller
divides and distributes
job to available
service nodes
Instructions specifying
locations of source files
and frames 01–10
In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate services in a cluster on the same IP subnet and to dynamically share and receive information. Because the computers can continually transmit their current processing availability status, Apple Qmaster can distribute (load-balance) the workload evenly across the cluster.
10 Preface
Instructions specifying locations of source files
and frames 11–20
Service node 1
Introduction to Distributed Processing
Service node 2
Processed file (frames 1-30)
placed in specified
destination
Instructions specifying locations of source files
and frames 21–30
Service node 3

About the Distributed Processing Setup Guide

The background information that introduces most of the basic concepts and terms related to using the Apple Qmaster distributed processing system is contained in this preface: “Introduction to Distributed Processing.”
Â
Chapter 1, “Getting Started Quickly,” on page 15, gets you going with a few very simple steps.
Â
Chapter 2, “Preparing a Network for Distributed Processing,”on page 21, will help you make sure that the network part of your Apple Qmaster system is ready.
Â
Chapter 3, “The Interfaces,” on page 31, introduces each of the applications and utilities included with the Apple Qmaster distributed processing software, and describes when you need to use each of them.
Â
Chapter 4, “Creating and Administering Clusters,” on page 41, tells you how to configure and administer the clusters, and how to submit and monitor the batches that you want to have processed by the clusters.
Â
The Appendix, “Command-Line Usage,” on page 67, explains how you can use the distributed processing system entirely from the command line.
While all of the chapters should be useful to administrators, client users may only need to refer to the to the introductory information in this preface, to understand how to use the Apple Qmaster distributed processing system for their purposes.
Compressor User Manual
or the
Apple Qmaster User Manual,
and possibly
To access the Distributed Processing Setup guide:
m
Choose Help > Distributed Processing Setup.
You can also choose either Help > Apple Qmaster User Manual or Help > Compressor User Manual.
Each user manual contains a homepage that provides quick access to various features, including Late-Breaking News.
Additionally, a comprehensive bookmark list allows you to quickly choose what you want to see and takes you there as soon as you click the link.
In addition to these navigational tools, the Help documentation gives you other means to locate information quickly:
Â
All cross-references in the text are linked. You can click any cross-reference and jump immediately to that location. Then, you can use the navigation Back button to return to where you were before you clicked the cross-reference.
Â
The table of contents and index are also linked. If you click an entry in either of these sections, you jump directly to that section of Help.
Â
You can also use the Find dialog to search the index or text for specific words or phrases.
Preface
Introduction to Distributed Processing
11

Apple Websites

There are a variety of Apple websites that you can visit to find additional information.
Apple Qmaster, Compressor, and Shake Websites
To access the Apple Qmaster support page, go to:
Â
http://www.apple.com/support/appleqmaster
To access the Compressor website, go to:
Â
http://www.apple.com/finalcutpro/compressor.html
To access the Compressor support page, go to:
Â
http://www.apple.com/support/compressor
To access the Compressor discussion page, go to:
Â
http://discussions.info.apple.com
To access the Shake website, go to:
Â
http://www.apple.com/shake
To access the Shake support page, go to:
Â
http://www.apple.com/support/shake

AppleCare Service and Support Website

For software updates and answers to the most frequently asked questions for all Apple products, go to:
Â
http://www.apple.com/support
You’ll also have access to product specifications, reference documentation, and Apple and third-party product technical articles.

Other Apple Websites

Start at the Apple homepage to find the latest information about Apple products:
Â
http://www.apple.com
QuickTime is industry-standard technology for handling video, sound, animation, graphics, text, music, and 360-degree virtual reality (VR) scenes. QuickTime provides a high level of performance, compatibility, and quality for delivering digital video. Go to the QuickTime website for information on the types of media supported, a tour of the QuickTime interface, and specifications:
Â
http://www.apple.com/quicktime
12 Preface
Introduction to Distributed Processing
For information about seminars, events, and third-party tools used in web publishing, design and print, music and audio, desktop movies, digital imaging, and the media arts go to:
 http://www.apple.com/pro
For resources, stories, and information about projects developed by users in education using Apple software, including Compressor, go to:
 http://www.apple.com/education
Go to the Apple Store to buy software, hardware, and accessories direct from Apple. You can also find special promotions and deals that include third-party hardware and software products:
 http://www.apple.com/store
Preface Introduction to Distributed Processing 13

1 Getting Started Quickly

1
The Apple Qmaster distributed processing system has default settings that allow you to use distributed processing immediately.
Whether you are using the distributed processing system for Compressor or Apple Qmaster, you can get up and running quickly with the following steps.
Note: The Compressor distributed processing feature for Dolby Digital Professional (AC-3) and Final Cut Pro is limited to computers that have Final Cut Studio installed.

Quick and Easy Distributed Processing

The steps below describe the simplest and quickest way to start using distributed processing.
Step 1: Install the software
Install the appropriate software on the computers you want to include in your distributed processing network. Each computer in the network will require Apple Qmaster and/or Compressor software.
To install the software:
1 Make sure the client software is on at least one computer in your network.
In order to submit jobs and batches to the distributed processing system, you will need to run client (submission) software (either Compressor or Apple Qmaster). If you are reading this, you have probably already installed one or both of these. For further information, see the installation booklet that came with either of these applications.
2 Install Apple Qmaster software on each computer you want to use for distributed
processing. (All computers must be on the same subnet.) By default, the Final Cut Studio installer will install Apple Qmaster software necessary for distributed processing.
15
However, in most standard distributed processing networks, you will need to install Apple Qmaster software on every computer in the network. You can use the standalone Apple Qmaster Node Installer to install the Apple Qmaster software on additional computers on your network.
a Locate the folder containing the Apple Qmaster software:
 If you are using Compressor, insert the Final Cut Studio installation disc and locate
the “Apple Qmaster Node Installer” package (AppleQmasterNode.mpkg).
 If you are using Shake, insert the Shake 4 installation disc and locate the
“Apple Qmaster Node Installer” package (AppleQmasterNode.mpkg).
b Do one of the following:
 Double-click the “AppleQmasterNode.mpkg” installer package, and follow the
onscreen instructions. By default, this installer will install all the necessary Apple Qmaster software.
 Install Apple Qmaster software from the command line. See “Installing
Apple Qmaster from the Command Line” on page 67 for more information.
About the Compressor Service Sharing Pane
Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the Compressor Service Sharing pane, in which you can quickly set up the computer as an unmanaged service node. Selecting the Yes option in this pane is the first of two steps in setting up AutoClusters in Compressor. An AutoCluster is a temporary QuickCluster which you create automatically when you submit a batch. For more information about AutoClusters, see “Using Compressor AutoClusters” on page 23.
Note: You can change any of these settings at a later time in the Apple Qmaster pane in System Preferences.
If you are unsure which role each computer will play in your network, just install the Apple Qmaster software on each computer in your network. You can sort out the details later. See “Basic Components of the Apple Qmaster Distributed Processing
System” on page 6 for additional information on what roles individual computers can
play in the distributed processing system.
16 Chapter 1 Getting Started Quickly
Additional Installation Tips
m In order to use the Apple Qmaster distributed processing system to process
Dolby Digital Professional audio, each node (computer) in your distributed processing network must have Final Cut Studio installed.
m Shake users can create distributed processing clusters containing computers that do
not have any Apple Qmaster software installed. See the Apple Qmaster User Manual for more information.
m If you are using the Apple Qmaster distributed processing system with Compressor
or Shake, all nodes (computers) in a cluster must have the same version of QuickTime installed.
To download the most recent version of QuickTime, go to:
 http://www.apple.com/quicktime/download/mac.html
Step 2: Configure a QuickCluster
Use the Apple Qmaster pane in System Preferences to configure the cluster controller and service node computers.
1 Open System Preferences.
2 Click the Apple Qmaster button, located in the Other section.
The Apple Qmaster pane appears.
Chapter 1 Getting Started Quickly 17
3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter
the administrator name and password.
4 Click Start Sharing.
This creates a QuickCluster with this computer as its controller, and an instance of processing services for each processor on the computer.
Step 3: Add service nodes to the cluster
On each computer that you would like to make a service node on your cluster, do the following:
1 Open System Preferences.
2 Click the Apple Qmaster button, located in the Other section.
The Apple Qmaster pane appears.
3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter
the administrator name and password.
4 Click the Services Only radio button.
5 Click Start Sharing.
This creates a service node that will automatically process jobs submitted to the QuickCluster you set up in “Configure a QuickCluster” on page 17.
Step 4: Create a batch
In the Compressor Batch window, or in the Apple Qmaster window, create a batch with one or more jobs. See the Compressor User Manual or the Apple Qmaster User Manual for details.
18 Chapter 1 Getting Started Quickly
Step 5: Submit the batch
1 Depending on whether you are using Compressor or Apple Qmaster, do one of
the following:
a In the Compressor Batch window, click Submit, and in the resulting dialog, do one of
the following:
 Use the Cluster pop-up menu to choose the cluster you created in Step 2.
 Leave the Cluster menu set to the default This Computer setting, and click the
“Include unmanaged services on other computers” checkbox. For more information on this option, see “Using Compressor AutoClusters” on page 23.
b In the Apple Qmaster window, click the Submit To pop-up menu and choose the
cluster you created in Step 2.
2 Click Submit.
The distributed processing system processes the batch.
Chapter 1 Getting Started Quickly 19
For more advanced information on creating and controlling clusters and services see any of the following:
 Basic Components of the Apple Qmaster Distributed Processing System” on page 6
 Sample Setup for Part-Time Processing on Desktop Computers” on page 27
 The Interfaces in the Apple Qmaster Distributed ProcessingSystem” on page 31
 Apple Qmaster Pane in System Preferences” on page 33
 Apple Qadministrator” on page 35
 An Overview of Configuring a Cluster” on page 42
 Configuring Service Nodes and Cluster Controllers” on page 42
 Creating Clusters with Apple Qadministrator” on page 50
 About QuickClusters” on page 53
 Modifying and Deleting Clusters With Apple Qadministrator” on page 58
20 Chapter 1 Getting Started Quickly
2 Preparing a Network
for Distributed Processing
2
A distributed processing network can consist of as few as one or two computers, while a high-volume network may include many computers, an Xserve and Xserve cluster nodes in a rack, and high-speed networking infrastructures.
You can scale up a distributed processing system as your workload demands by adding features and devices to the network that supports it. This chapter helps you prepare your network by describing the following:
 The Minimum You Need to Know (p. 21)
 Using Compressor AutoClusters (p. 23)
 Sample Setup for Part-Time Processing on Desktop Computers (p. 27)

The Minimum You Need to Know

The following are the basic rules for setting up a distributed processing network:
 A cluster must contain one (and only one) computer acting as the cluster controller,
and at least one computer acting as the service node. (These two can be the same computer, as shown next in “Example of a Minimal Distributed Processing Network.”)
 The client computers and the computers in any cluster that supports them must be
on the same network.
 The network must support the Apple networking technology built in to Mac OS X.
 All the computers in a cluster need Read and Write access to any computers (or
storage devices) that will be specified as output destinations for files.
21

Example of a Minimal Distributed Processing Network

A very small distributed processing setup could include as few as two computers:
 One computer connected to the client and configured to act as both the service
node and the cluster controller
 One client computer
Minimum setup for distributed processing
Client
Service node with
cluster controller enabled
Though simple, this setup is useful in a small-scale environment because it allows the client computer to off-load a lot of processing work.
See “Example of an Expanded Distributed Processing Network” on page 26 for an illustration of a more powerful setup.

Using One Computer to Serve Two Distributed Processing Roles

To maximize your resources, you may want to consider using some computers for more than one distributed processing function.
 Service node and cluster controller: In a small setup, one of the service nodes in a
cluster can also act as the cluster controller so that it performs both functions. However, in a cluster of many service nodes, the processing load required for the cluster controller could be so high that it would not be efficient to use one computer as both a service node and a cluster controller.
 Client computer and cluster controller or service node: You could also set up a client
computer to act as a cluster controller or service node in a cluster, but again, keep in mind that the more available processing power a computer has, the faster it can manage or process jobs.
22 Chapter 2 Preparing a Network for Distributed Processing

Using Compressor AutoClusters

The AutoCluster feature in Compressor gives you an easy way to take advantage of the distributed processing capabilities offered by Apple Qmaster without requiring a lot of knowledge about how clusters are configured, setting up file sharing, and so on.
Using AutoCluster is a two-step process:
 Automatically creating Apple Qmaster service nodes as you install Final Cut Studio
or Apple Qmaster
 Selecting the “Include unmanaged services on other computers” when you submit
a Compressor batch for processing
These two steps let you harness the processing power of any number of computers on your network without any additional effort or knowledge on your part.
Step 1: Creating Service Nodes During Installation
Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the Compressor Service Sharing pane, in which you can quickly set up the computer as an unmanaged Apple Qmaster service node, which is essentially a free agent “processing slave,” available on the network for any computer which requests its services.
Chapter 2 Preparing a Network for Distributed Processing 23
If you select Yes in this pane, the Apple Qmaster pane in System Preferences will be configured as follows:
 Share this computer as: Services only
 Compressor service: Selected, unmanaged
 Start Sharing: Started
You can change any of these settings at a later time in the Apple Qmaster pane in System Preferences. For more information, see “Apple Qmaster Pane in System Preferences” on page 33 and Chapter 4, “Creating and Administering Clusters,” on page 41.
Important: While the Final Cut Studio installer requires a separate serial number for
each computer, you can use the Apple Qmaster Node Installer (available in the Extras folder) to install the necessary Apple Qmaster software to make any computer on your network into an Apple Qmaster service node.
Step 2: Submitting Compressor Batches for Processing
When you submit a Compressor batch for processing, a dialog appears that allows you to name the batch and select the computers to process the batch. At this point, you can harness the processing power of all the computers you set up as service nodes in Step 1 to complete the batch.
To process the batch using AutoCluster
1 Leave the Cluster pop-up menu set to the default selection (“This Computer”).
2 Select the “Include unmanaged services on other computers” checkbox.
Compressor and Apple Qmaster will coordinate the distribution of the processing tasks between the available computers and deposit the resulting output files at the location(s) you designated in Compressor.
24 Chapter 2 Preparing a Network for Distributed Processing
Other Possible Components of a Distributed Processing Network
There are many ways to expand the capacity of a distributed processing network. You could include any of the following:
 High-speed switch and cables: A 100BaseT or Gigabit Ethernet switch and compatible
cables to allow your data to move over the LAN at maximum speed.
 Multiple clients: Multiple client computers can use the services of the same cluster.
And, you can have multiple client applications on the same client computer, using the same cluster.
 Multiple clusters: Depending on how extensive your network is and how many clients
it needs to serve, you may want to divide up available computers and create more than one cluster to serve various clients. (Users select the cluster they want to send a batch to when they submit the batch.)
 Multiple service nodes: In general, more service nodes means more processing power.
In deciding how many service nodes to have in a cluster, consider the ratio of data movement time to computing time. If the processing demand is greater than the network demand required to move job segments throughout the cluster, as is the case with rendering, more service nodes are a good idea. If the computing load, per job, is closer to the network load, having a smaller number of service nodes per cluster may be more efficient. If you are using the Apple Qmaster distributed processing system with applications other than Shake or Compressor, consult the application’s user manual on how to optimize the number of service nodes.
 Storage device: A storage device, such as a remote disk or group of disk arrays, can be
used as cluster scratch storage, which is a place for short-term storage of temporary data generated by the cluster controller, clients, and service nodes. (You set the scratch storage location in the Apple Qmaster pane in System Preferences. See “Cluster Storage: Setting a Scratch Storage Location” on page 62.) Alternatively, a storage device can be used as a final destination for the files after they are processed.
Many of these items are incorporated in the example that follows.
Chapter 2 Preparing a Network for Distributed Processing 25

Example of an Expanded Distributed Processing Network

For rendering, a network might include a number of client computers on a LAN, connected to a cluster using a high-speed switch. A rack of servers plus a shared storage device, acting as the cluster, would be an extremely strong rendering engine. The service nodes would each have a local copy of the relevant client application software so that they could process the rendering jobs.
Example of a network setup for distributed rendering
Client Client Client Client
LAN
High-speed switches (chained together)
Rack of servers containing
cluster controller and service nodes,
plus rack-mounted shared storage device
26 Chapter 2 Preparing a Network for Distributed Processing
Loading...
+ 60 hidden pages