Apple COMPRESSOR 3, QMASTER 3 Distributed Processing Setup

Download

Page 1

Apple Qmaster 3 and Compressor 3

Distributed Processing Setup

Page 2

Apple Inc.

Your rights to the software are governed by the accompanying software license agreement. The owner or authorized user of a valid copy of Final Cut Studio software may reproduce this publication for the purpose of learning to use such software. No part of this publication may be reproduced or transmitted for commercial purposes, such as selling copies of this publication or for providing paid for support services.

The Apple logo is a trademark of Apple Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Shift-Option-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.

Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.

Note:

Because Apple frequently releases new versions and updates to its system software, applications, and Internet sites, images shown in this book may be slightly different from what you see on your screen.

Apple Inc. 1 Infinite Loop Cupertino, CA 95014–2084 408-996-1010 www.apple.com

Apple, the Apple logo, Bonjour, Final Cut, Final Cut Pro, Final Cut Studio, FireWire, Mac, Mac OS, QuickTime, Shake, Xsan, and Xserve are trademarks of Apple Inc., registered in the U.S. and other countries.

Apple Remote Desktop and Finder are trademarks of Apple Inc.

AppleCare and Apple Store are service marks of Apple Inc., registered in the U.S. and other countries.

Dolby Laboratories: Manufactured under license from Dolby Laboratories. “Dolby,” “Pro Logic,” and the doubleD symbol are trademarks of Dolby Laboratories. Confidential Unpublished Works, © 1992–1997 Dolby Laboratories, Inc. All rights reserved.

UNIX is a registered trademark of The Open Group.

Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.

Page 3

Preface 5 Introduction to Distributed Processing

Using Distributed Processing to Increase Speed and Efficiency

Basic Components of the Apple Qmaster Distributed Processing System

QuickClusters

Compressor AutoClusters

How the Apple Qmaster System Distributes Batches

About the Distributed Processing Setup Guide

Apple Websites

Chapter 1 15 Getting Started Quickly

Quick and Easy Distributed Processing

Chapter 2 21 Preparing a Network for Distributed Processing

The Minimum You Need to Know

Using Compressor AutoClusters

Other Possible Components of a Distributed Processing Network

Sample Setup for Part-Time Processing on Desktop Computers

Additional Steps For Part-Time Distributed Processing With Shake

Chapter 3 31 The Interfaces

The Interfaces in the Apple Qmaster Distributed Processing System

Apple Qmaster Pane in System Preferences

Apple Qadministrator

Client Interfaces: Compressor and Apple Qmaster

Batch Monitor

Page 4

Chapter 4 41 Creating and Administering Clusters

An Overview of Configuring a Cluster

Configuring Service Nodes and Cluster Controllers

Creating Clusters with Apple Qadministrator

About QuickClusters

Advanced Settings in the Apple Qmaster Preferences Pane

Modifying and Deleting Clusters With Apple Qadministrator

Monitoring Cluster Activity

Accessing Activity Logs

Setting Cluster Preferences

Setting Passwords and Scratch Storage

Recovery and Failure Notification Features

Appendix A 67 Command-Line Usage

Appendix B 77 Troubleshooting

Index 83

Contents

Page 5

Introduction to Distributed Processing

Rendering a series of large files on one desktop computer is processing intensive and time consuming. You can increase speed and productivity by distributing processing across multiple computers.

High-volume processing is sometimes addressed by carefully managing multiple computers; technicians set up batches of processing tasks for each computer, and then monitor their progress, collect and route the processed files, and start over again with new batches. While this is an improvement over the single-computer method, the resource and process management can be laborious and slow.

The Apple Qmaster distributed processing engine provides a more efficient solution, handling all the work distribution and processing for you, behind the scenes. Apple Qmaster and the Apple Qmaster features of Compressor manage the processing across designated computers. They subdivide the work for speed, route the work to the computers with the most available computing power, and direct the processing.

Preface

Using Distributed Processing to Increase Speed and Efficiency

Distributed processing

computers that have been chosen to provide more processing power. You can submit batches of processing jobs to Apple Qmaster, which allocates those jobs to other computers in the most efficient way (described in more detail in “How the

Apple Qmaster System Distributes Batches” on page 9).

accelerates processing by distributing the work to multiple

Page 6

Computers that submit batches to the Apple Qmaster distributed processing system are called

clients.

job

is a processing task such as a Compressor preset-source pair, or a Shake file, or other file or commands, that use UNIX commands to specify settings such as rendering instructions and file locations and destinations.

Network

Batch of

processing jobs

Client computer

Jobs are submitted.

batch

is one or more jobs submitted for processing at one time. The procedure is

Apple Qmaster cluster

Processing is

performed by cluster.

Processed

files

Destination folder

Files are placed at

specified destination

analogous to printing multi-page documents from a word processing program; the files are spooled and processed in the background. Although a batch can include just one job, you will typically want to submit several jobs at once for processing. Similarly, several people can use the same Apple Qmaster system at the same time, with several client computers sending batches in the same time frame. Batches are managed and distributed by the computer that is designated as the Apple Qmaster

cluster controller,

which is described in the next section.

Basic Components of the Apple Qmaster Distributed Processing System

While the Apple Qmaster software includes a few different applications (see Chapter 3, “The Interfaces,” on page 31), as a whole it is part of a networked system that includes the following basic components:

Client(s):

submit jobs for distributed processing. Applications that can use Apple Qmaster services for processing include Compressor, Shake, Alias Maya, and many UNIX command-line programs.

The computer or computers that use Compressor or Apple Qmaster to

6 Preface

Introduction to Distributed Processing

Page 7

Apple Qmaster cluster,

Service nodes:

which contains:

The computers that perform the processing of batches submitted

via Compressor or Apple Qmaster. A batch can include one or more jobs.

Cluster controller:

The software, enabled on a computer by means of the Apple Qmaster pane in System Preferences, that divides up batches, determines which service nodes to send work to, and generally tracks and directs the processes.

Client computer from

which users send jobs to the cluster

(can contain multiple service nodes, but only one cluster controller)

Cluster controller manages the

distribution of client’s jobs across the cluster

Service node

processes jobs

Service node

processes jobs

Cluster

Service node

processes jobs

Service node

processes jobs

The client computer, the service nodes, and the cluster controller are often on separate (but network-connected) computers, for the most rapid processing potential. However, the cluster controller could be on a client computer or a service node. See “Using One

Computer to Serve Two Distributed Processing Roles” on page 22 for more information

on this scenario.

Following is a closer look at the part each component plays in the Apple Qmaster system.

Clients

Batches are submitted for distributed processing from the computer can be any computer that has Compressor or Apple Qmaster installed and is on the same network (subnet) as the cluster controller. Multiple client computers can be on the same subnet, using the same cluster to do the processing for various applications.

You use Compressor or the Apple Qmaster application to submit batches to be processed for a client. See the

User Manual

for details on using these applications.

Compressor User Manual

client computers

and the

Apple Qmaster

. A client

Preface

Introduction to Distributed Processing

Page 8

Clusters

When a client sends batches to the Apple Qmaster distributed processing system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster–configured computers called a clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection.

Example of a cluster

cluster.

You can create one or more

Cluster

controller

Note:

Service

node

Service

node

Service

node

This illustration provides only one simple example of a cluster. Other possibilities

are described in Chapter 2, “Preparing a Network for Distributed Processing.”

Service Nodes

The service nodes are where the processing work is done. When you assign a group of service nodes to a cluster, they function as one very powerful computer because all their resources are shared. If one service node is overloaded or otherwise inaccessible, another service node is used.

You make a computer available as a service node by configuring it in the Apple Qmaster pane in System Preferences. The simple steps involved in using System Preferences to configure a service node are described in Chapter 4, “Creating

and Administering Clusters,” on page 41.

Note:

The terms

processing

and

rendering

will come up frequently as you read this document. The term processing is used here generically to cover both rendering (for Shake, and other frame-based rendering applications) and encoding (or transcoding or compression) for Compressor. For more information see the

Compressor User Manual

Shake User Manual

and the

8 Preface

Introduction to Distributed Processing

Page 9

Cluster Controllers

The cluster controller software acts as the manager of a cluster. The cluster controller directs the distribution of batches within the cluster. It has the ability to determine the best use of the cluster resources based on work and availability variables. (See “How

the Apple Qmaster System Distributes Batches,” next, for more details.) The cluster

controller is responsible for accepting batch submissions, maintaining and managing the batch queue, and doling out the work to the appropriate service node. It also tracks the status of all outstanding batches.

You make a computer available as a cluster controller by turning on the clustercontrolling service in the Apple Qmaster pane in System Preferences.

QuickClusters

The QuickCluster feature of the Apple Qmaster distributed processing system is a simple and automated alternative to creating and configuring clusters manually. For more information about QuickClusters, see “Creating QuickClusters” on page 53.

Compressor AutoClusters

The AutoCluster feature in Compressor gives you an easy way to take advantage of the distributed processing capabilities offered by Apple Qmaster without requiring a lot of knowledge about how clusters are configured, setting up file sharing, and so on. For more information about AutoClusters, see “Using Compressor AutoClusters” on page 23.

How the Apple Qmaster System Distributes Batches

The Apple Qmaster cluster controller determines the most efficient use of the cluster resources. It makes this determination based on the availability of each service node and the number of separable parts (described next) of the batch.

Because Apple Qmaster subdivides individual batches across different service nodes, the work is shared and completed more quickly. And, because this method uses all the service nodes as much as possible, you avoid under-utilizing your resources.

Preface

Introduction to Distributed Processing

Page 10

Batches can be distributed to a cluster by the cluster controller in one or both of the following ways. (Apple Qmaster determines which way is the most efficient for specific batches, depending on the circumstances.)

The batch is subdivided into data segments:

cluster controller could divide the frames into groups (

For example, for a render batch, the

segments

). Each segment

would be processed in parallel on the service nodes in the cluster.

The batch is subdivided into tasks:

For example, for a render batch, the cluster controller could subdivide the rendering work into different processing tasks. Different tasks would be run on different service nodes.

Rather than actually moving segments, Apple Qmaster tells the service nodes which segments to read via the network, where to find them, and what to do with them. Below is an example of how one batch could be processed in an Apple Qmaster system.

Batch submitted by

Compressor or

Apple Qmaster (job request

for frames 1–30)

Cluster controller

divides and distributes

job to available

service nodes

Instructions specifying

locations of source files

and frames 01–10

In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate services in a cluster on the same IP subnet and to dynamically share and receive information. Because the computers can continually transmit their current processing availability status, Apple Qmaster can distribute (load-balance) the workload evenly across the cluster.

10 Preface

Instructions specifying locations of source files

and frames 11–20

Service node 1

Introduction to Distributed Processing

Service node 2

Processed file (frames 1-30)

placed in specified

destination

Instructions specifying locations of source files

and frames 21–30

Service node 3

Page 11

About the Distributed Processing Setup Guide

The background information that introduces most of the basic concepts and terms related to using the Apple Qmaster distributed processing system is contained in this preface: “Introduction to Distributed Processing.”

Chapter 1, “Getting Started Quickly,” on page 15, gets you going with a few very simple steps.

Chapter 2, “Preparing a Network for Distributed Processing,”on page 21, will help you make sure that the network part of your Apple Qmaster system is ready.

Chapter 3, “The Interfaces,” on page 31, introduces each of the applications and utilities included with the Apple Qmaster distributed processing software, and describes when you need to use each of them.

Chapter 4, “Creating and Administering Clusters,” on page 41, tells you how to configure and administer the clusters, and how to submit and monitor the batches that you want to have processed by the clusters.

The Appendix, “Command-Line Usage,” on page 67, explains how you can use the distributed processing system entirely from the command line.

While all of the chapters should be useful to administrators, client users may only need to refer to the to the introductory information in this preface, to understand how to use the Apple Qmaster distributed processing system for their purposes.

Compressor User Manual

or the

Apple Qmaster User Manual,

and possibly

To access the Distributed Processing Setup guide:

Choose Help > Distributed Processing Setup.

You can also choose either Help > Apple Qmaster User Manual or Help > Compressor User Manual.

Each user manual contains a homepage that provides quick access to various features, including Late-Breaking News.

Additionally, a comprehensive bookmark list allows you to quickly choose what you want to see and takes you there as soon as you click the link.

In addition to these navigational tools, the Help documentation gives you other means to locate information quickly:

All cross-references in the text are linked. You can click any cross-reference and jump immediately to that location. Then, you can use the navigation Back button to return to where you were before you clicked the cross-reference.

The table of contents and index are also linked. If you click an entry in either of these sections, you jump directly to that section of Help.

You can also use the Find dialog to search the index or text for specific words or phrases.

Preface

Introduction to Distributed Processing

Page 12

Apple Websites

There are a variety of Apple websites that you can visit to find additional information.

Apple Qmaster, Compressor, and Shake Websites

To access the Apple Qmaster support page, go to:

http://www.apple.com/support/appleqmaster

To access the Compressor website, go to:

http://www.apple.com/finalcutpro/compressor.html

To access the Compressor support page, go to:

http://www.apple.com/support/compressor

To access the Compressor discussion page, go to:

http://discussions.info.apple.com

To access the Shake website, go to:

http://www.apple.com/shake

To access the Shake support page, go to:

http://www.apple.com/support/shake

AppleCare Service and Support Website

For software updates and answers to the most frequently asked questions for all Apple products, go to:

http://www.apple.com/support

You’ll also have access to product specifications, reference documentation, and Apple and third-party product technical articles.

Other Apple Websites

Start at the Apple homepage to find the latest information about Apple products:

http://www.apple.com

QuickTime is industry-standard technology for handling video, sound, animation, graphics, text, music, and 360-degree virtual reality (VR) scenes. QuickTime provides a high level of performance, compatibility, and quality for delivering digital video. Go to the QuickTime website for information on the types of media supported, a tour of the QuickTime interface, and specifications:

http://www.apple.com/quicktime

12 Preface

Introduction to Distributed Processing

Page 13

For information about seminars, events, and third-party tools used in web publishing, design and print, music and audio, desktop movies, digital imaging, and the media arts go to:

Â http://www.apple.com/pro

For resources, stories, and information about projects developed by users in education using Apple software, including Compressor, go to:

Â http://www.apple.com/education

Go to the Apple Store to buy software, hardware, and accessories direct from Apple. You can also find special promotions and deals that include third-party hardware and software products:

Â http://www.apple.com/store

Preface Introduction to Distributed Processing 13

Page 14

Page 15

1 Getting Started Quickly

The Apple Qmaster distributed processing system has default settings that allow you to use distributed processing immediately.

Whether you are using the distributed processing system for Compressor or Apple Qmaster, you can get up and running quickly with the following steps.

Note: The Compressor distributed processing feature for Dolby Digital Professional (AC-3) and Final Cut Pro is limited to computers that have Final Cut Studio installed.

Quick and Easy Distributed Processing

The steps below describe the simplest and quickest way to start using distributed processing.

Step 1: Install the software

Install the appropriate software on the computers you want to include in your distributed processing network. Each computer in the network will require Apple Qmaster and/or Compressor software.

To install the software:

1 Make sure the client software is on at least one computer in your network.

In order to submit jobs and batches to the distributed processing system, you will need to run client (submission) software (either Compressor or Apple Qmaster). If you are reading this, you have probably already installed one or both of these. For further information, see the installation booklet that came with either of these applications.

2 Install Apple Qmaster software on each computer you want to use for distributed

processing. (All computers must be on the same subnet.) By default, the Final Cut Studio installer will install Apple Qmaster software necessary for distributed processing.

Page 16

However, in most standard distributed processing networks, you will need to install Apple Qmaster software on every computer in the network. You can use the standalone Apple Qmaster Node Installer to install the Apple Qmaster software on additional computers on your network.

a Locate the folder containing the Apple Qmaster software:

Â If you are using Compressor, insert the Final Cut Studio installation disc and locate

the “Apple Qmaster Node Installer” package (AppleQmasterNode.mpkg).

Â If you are using Shake, insert the Shake 4 installation disc and locate the

“Apple Qmaster Node Installer” package (AppleQmasterNode.mpkg).

b Do one of the following:

Â Double-click the “AppleQmasterNode.mpkg” installer package, and follow the

onscreen instructions. By default, this installer will install all the necessary Apple Qmaster software.

Â Install Apple Qmaster software from the command line. See “Installing

Apple Qmaster from the Command Line” on page 67 for more information.

About the Compressor Service Sharing Pane

Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the Compressor Service Sharing pane, in which you can quickly set up the computer as an unmanaged service node. Selecting the Yes option in this pane is the first of two steps in setting up AutoClusters in Compressor. An AutoCluster is a temporary QuickCluster which you create automatically when you submit a batch. For more information about AutoClusters, see “Using Compressor AutoClusters” on page 23.

Note: You can change any of these settings at a later time in the Apple Qmaster pane in System Preferences.

If you are unsure which role each computer will play in your network, just install the Apple Qmaster software on each computer in your network. You can sort out the details later. See “Basic Components of the Apple Qmaster Distributed Processing

System” on page 6 for additional information on what roles individual computers can

play in the distributed processing system.

16 Chapter 1 Getting Started Quickly

Page 17

Additional Installation Tips

m In order to use the Apple Qmaster distributed processing system to process

Dolby Digital Professional audio, each node (computer) in your distributed processing network must have Final Cut Studio installed.

m Shake users can create distributed processing clusters containing computers that do

not have any Apple Qmaster software installed. See the Apple Qmaster User Manual for more information.

m If you are using the Apple Qmaster distributed processing system with Compressor

or Shake, all nodes (computers) in a cluster must have the same version of QuickTime installed.

To download the most recent version of QuickTime, go to:

Â http://www.apple.com/quicktime/download/mac.html

Step 2: Configure a QuickCluster

Use the Apple Qmaster pane in System Preferences to configure the cluster controller and service node computers.

1 Open System Preferences.

2 Click the Apple Qmaster button, located in the Other section.

The Apple Qmaster pane appears.

Chapter 1 Getting Started Quickly 17

Page 18

3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter

the administrator name and password.

4 Click Start Sharing.

This creates a QuickCluster with this computer as its controller, and an instance of processing services for each processor on the computer.

Step 3: Add service nodes to the cluster

On each computer that you would like to make a service node on your cluster, do the following:

1 Open System Preferences.

2 Click the Apple Qmaster button, located in the Other section.

The Apple Qmaster pane appears.

3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter

the administrator name and password.

4 Click the Services Only radio button.

5 Click Start Sharing.

This creates a service node that will automatically process jobs submitted to the QuickCluster you set up in “Configure a QuickCluster” on page 17.

Step 4: Create a batch

In the Compressor Batch window, or in the Apple Qmaster window, create a batch with one or more jobs. See the Compressor User Manual or the Apple Qmaster User Manual for details.

18 Chapter 1 Getting Started Quickly

Page 19

Step 5: Submit the batch

1 Depending on whether you are using Compressor or Apple Qmaster, do one of

the following:

a In the Compressor Batch window, click Submit, and in the resulting dialog, do one of

the following:

Â Use the Cluster pop-up menu to choose the cluster you created in Step 2.

Â Leave the Cluster menu set to the default This Computer setting, and click the

“Include unmanaged services on other computers” checkbox. For more information on this option, see “Using Compressor AutoClusters” on page 23.

b In the Apple Qmaster window, click the Submit To pop-up menu and choose the

cluster you created in Step 2.

2 Click Submit.

The distributed processing system processes the batch.

Chapter 1 Getting Started Quickly 19

Page 20

For more advanced information on creating and controlling clusters and services see any of the following:

Â “Basic Components of the Apple Qmaster Distributed Processing System” on page 6

Â “Sample Setup for Part-Time Processing on Desktop Computers” on page 27

Â “The Interfaces in the Apple Qmaster Distributed ProcessingSystem” on page 31

Â “Apple Qmaster Pane in System Preferences” on page 33

Â “Apple Qadministrator” on page 35

Â “An Overview of Configuring a Cluster” on page 42

Â “Configuring Service Nodes and Cluster Controllers” on page 42

Â “Creating Clusters with Apple Qadministrator” on page 50

Â “About QuickClusters” on page 53

Â “Modifying and Deleting Clusters With Apple Qadministrator” on page 58

20 Chapter 1 Getting Started Quickly

Page 21

2 Preparing a Network

for Distributed Processing

A distributed processing network can consist of as few as one or two computers, while a high-volume network may include many computers, an Xserve and Xserve cluster nodes in a rack, and high-speed networking infrastructures.

You can scale up a distributed processing system as your workload demands by adding features and devices to the network that supports it. This chapter helps you prepare your network by describing the following:

Â The Minimum You Need to Know (p. 21)

Â Using Compressor AutoClusters (p. 23)

Â Sample Setup for Part-Time Processing on Desktop Computers (p. 27)

The Minimum You Need to Know

The following are the basic rules for setting up a distributed processing network:

Â A cluster must contain one (and only one) computer acting as the cluster controller,

and at least one computer acting as the service node. (These two can be the same computer, as shown next in “Example of a Minimal Distributed Processing Network.”)

Â The client computers and the computers in any cluster that supports them must be

on the same network.

Â The network must support the Apple networking technology built in to Mac OS X.

Â All the computers in a cluster need Read and Write access to any computers (or

storage devices) that will be specified as output destinations for files.

Page 22

Example of a Minimal Distributed Processing Network

A very small distributed processing setup could include as few as two computers:

Â One computer connected to the client and configured to act as both the service

node and the cluster controller

Â One client computer

Minimum setup for distributed processing

Client

Service node with

cluster controller enabled

Though simple, this setup is useful in a small-scale environment because it allows the client computer to off-load a lot of processing work.

See “Example of an Expanded Distributed Processing Network” on page 26 for an illustration of a more powerful setup.

Using One Computer to Serve Two Distributed Processing Roles

To maximize your resources, you may want to consider using some computers for more than one distributed processing function.

Â Service node and cluster controller: In a small setup, one of the service nodes in a

cluster can also act as the cluster controller so that it performs both functions. However, in a cluster of many service nodes, the processing load required for the cluster controller could be so high that it would not be efficient to use one computer as both a service node and a cluster controller.

Â Client computer and cluster controller or service node: You could also set up a client

computer to act as a cluster controller or service node in a cluster, but again, keep in mind that the more available processing power a computer has, the faster it can manage or process jobs.

22 Chapter 2 Preparing a Network for Distributed Processing

Page 23

Using Compressor AutoClusters

Using AutoCluster is a two-step process:

Â Automatically creating Apple Qmaster service nodes as you install Final Cut Studio

or Apple Qmaster

Â Selecting the “Include unmanaged services on other computers” when you submit

a Compressor batch for processing

These two steps let you harness the processing power of any number of computers on your network without any additional effort or knowledge on your part.

Step 1: Creating Service Nodes During Installation

Both the Final Cut Studio installer and the Apple Qmaster Node Installer include the Compressor Service Sharing pane, in which you can quickly set up the computer as an unmanaged Apple Qmaster service node, which is essentially a free agent “processing slave,” available on the network for any computer which requests its services.

Chapter 2 Preparing a Network for Distributed Processing 23

Page 24

If you select Yes in this pane, the Apple Qmaster pane in System Preferences will be configured as follows:

Â Share this computer as: Services only

Â Compressor service: Selected, unmanaged

Â Start Sharing: Started

You can change any of these settings at a later time in the Apple Qmaster pane in System Preferences. For more information, see “Apple Qmaster Pane in System Preferences” on page 33 and Chapter 4, “Creating and Administering Clusters,” on page 41.

Important: While the Final Cut Studio installer requires a separate serial number for

each computer, you can use the Apple Qmaster Node Installer (available in the Extras folder) to install the necessary Apple Qmaster software to make any computer on your network into an Apple Qmaster service node.

Step 2: Submitting Compressor Batches for Processing

When you submit a Compressor batch for processing, a dialog appears that allows you to name the batch and select the computers to process the batch. At this point, you can harness the processing power of all the computers you set up as service nodes in Step 1 to complete the batch.

To process the batch using AutoCluster

1 Leave the Cluster pop-up menu set to the default selection (“This Computer”).

2 Select the “Include unmanaged services on other computers” checkbox.

Compressor and Apple Qmaster will coordinate the distribution of the processing tasks between the available computers and deposit the resulting output files at the location(s) you designated in Compressor.

24 Chapter 2 Preparing a Network for Distributed Processing

Page 25

Other Possible Components of a Distributed Processing Network

There are many ways to expand the capacity of a distributed processing network. You could include any of the following:

Â High-speed switch and cables: A 100BaseT or Gigabit Ethernet switch and compatible

cables to allow your data to move over the LAN at maximum speed.

Â Multiple clients: Multiple client computers can use the services of the same cluster.

And, you can have multiple client applications on the same client computer, using the same cluster.

Â Multiple clusters: Depending on how extensive your network is and how many clients

it needs to serve, you may want to divide up available computers and create more than one cluster to serve various clients. (Users select the cluster they want to send a batch to when they submit the batch.)

Â Multiple service nodes: In general, more service nodes means more processing power.

In deciding how many service nodes to have in a cluster, consider the ratio of data movement time to computing time. If the processing demand is greater than the network demand required to move job segments throughout the cluster, as is the case with rendering, more service nodes are a good idea. If the computing load, per job, is closer to the network load, having a smaller number of service nodes per cluster may be more efficient. If you are using the Apple Qmaster distributed processing system with applications other than Shake or Compressor, consult the application’s user manual on how to optimize the number of service nodes.

Â Storage device: A storage device, such as a remote disk or group of disk arrays, can be

used as cluster scratch storage, which is a place for short-term storage of temporary data generated by the cluster controller, clients, and service nodes. (You set the scratch storage location in the Apple Qmaster pane in System Preferences. See “Cluster Storage: Setting a Scratch Storage Location” on page 62.) Alternatively, a storage device can be used as a final destination for the files after they are processed.

Many of these items are incorporated in the example that follows.

Chapter 2 Preparing a Network for Distributed Processing 25

Page 26

Example of an Expanded Distributed Processing Network

For rendering, a network might include a number of client computers on a LAN, connected to a cluster using a high-speed switch. A rack of servers plus a shared storage device, acting as the cluster, would be an extremely strong rendering engine. The service nodes would each have a local copy of the relevant client application software so that they could process the rendering jobs.

Example of a network setup for distributed rendering

Client Client Client Client

LAN

High-speed switches (chained together)

Rack of servers containing

cluster controller and service nodes,

plus rack-mounted shared storage device

26 Chapter 2 Preparing a Network for Distributed Processing

Page 27

Sample Setup for Part-Time Processing on Desktop Computers

This section takes you through the basic steps involved in a sample setup for “part-time” distributed processing. You can use it to get an idea of the kind of distributed processing environment you want to create, and as a guide in setting up that environment.

This setup is for an environment that uses desktop computers. It is called “part-time” processing because each computer acts as someone’s workstation, but at the same time is also part of the distributed processing cluster. The bulk of the processing jobs can be submitted with Compressor or Apple Qmaster at the end of the day, so that the computers are busy processing a large queue of distributed processing batches after everyone has gone home. (See “Advanced Settings in the Apple Qmaster Preferences

Pane” on page 54 for information on scheduling service node availability.)

FireWire

drive

FireWire

drive

Network

FireWire

drive

Each computer acts as:

A client that submits jobs for processing

•

A service node that performs the processing

•

All source and output files are stored on the FireWire drives.

FireWire

drive

FireWire

drive

In this sample setup, as shown above, five computers act as both the clients (user workstations from which users submit jobs for distributed processing) and cluster computers (which do the processing). Each computer has an additional volume, such as a FireWire drive, that is used for media before and after it is rendered, and for the associated files.

Chapter 2 Preparing a Network for Distributed Processing 27

Page 28

The steps that follow describe how to configure this desktop distributed processing environment. Before you get started, keep these essentials in mind:

Â The cluster computers (cluster controller and service nodes) and the client

computers (user workstations) need to be on the same local network (subnet).

Â All the computers in the setup need Read and Write access to any volumes that will

be specified as the source location or output destination for files, including Shake scripts. (An appropriate way to configure this access is included in the sample steps below.)

Configuring Access for Part-Time Distributed Processing

Step 1: Install Compressor 3 and/or Apple Qmaster

The necessary components of Compressor 3 or Apple Qmaster need to be installed on each computer. See “Install the software” on page 15 for more information.

Important: The Compressor distributed processing feature for Dolby Digital Professional

(AC-3) and Final Cut Pro is limited to computers that have Final Cut Studio installed.

Step 2: Make sure each computer has a volume dedicated to media

Make sure each computer has the necessary available storage, preferably a dedicated media volume. For example, give each computer a second volume, such as a FireWire drive, that is used for media. Each computer can use this volume for all source and destination files associated with distributed processing.

If you are a Compressor user, you can skip to one of the following chapters to complete the setup of your part time distributed processing system:

Â Chapter 1, “Getting Started Quickly,” on page 15

Â Chapter 4, “Creating and Administering Clusters,” on page 41

If you are a Shake user and you cannot consolidate all of the necessary source files (Shake scripts, media files, etc.) on a single-cluster storage volume, skip to “Additional

Steps For Part-Time Distributed Processing With Shake” on page 29 before going on

to Step 3.

Step 3: Create a cluster

First, use the Apple Qmaster pane in System Preferences to enable cluster controlling on one of the computers and enable the processing services on all the computers (making them service nodes). Then, you can assemble these computers as a cluster. Apple Qadministrator may not be necessary. See Chapter 4, “Creating and

Administering Clusters,” on page 41, for detailed instructions.

28 Chapter 2 Preparing a Network for Distributed Processing

Page 29

Additional Steps For Part-Time Distributed Processing With Shake

The following additional steps may be necessary for Shake users who cannot consolidate all of the necessary source files (Shake scripts, media files, etc.) on a single cluster storage volume.

Step 1: Turn off the UNC (Universal Naming Convention) setting for Shake

To make sharing and volume mounting work smoothly in this setup, you need to turn off the Shake UNC setting on each computer. The UNC setting uses the entire file pathname, with the network address, in a convention that starts with //ComputerName/DriveName/path. You don’t want Shake to use this filenaming convention because it conflicts with the file sharing and volume mounting used in this setup.

Note: All the media volumes created in Step 2 of “Configuring Access for Part-Time

Distributed Processing,” above, should have the same name.

The Shake startup .h file

In the three steps below, you make this change in a Shake startup .h file. As described in the Shake documentation, the startup .h files, located in the startup directory, are used to customize Shake settings (similar to setting preferences).

To turn off the UNC setting, do the following on each of the computers:

1 Log in as the user who will use Shake on the computer.

2 Double-click the Terminal icon in /Applications/Utilities to open a Terminal window.

3 Enter these two command lines in the Terminal window, pressing Return after each

command line:

mkdir -p ~/nreal/include/startup/

echo 'script.uncFileNames = 0;' > ~/nreal/include/startup/UNC_off.h

Step 2: Turn Personal File Sharing on

On each computer, open System Preferences, click Sharing, and turn on Personal File Sharing. This allows the computers to share the media volumes.

Step 3: Mount all the media storage volumes

On each computer, log in as the administrator. (The first user account you create when you set up Mac OS X is an administrator account.) Then, on each computer in the group, use the Connect to Server command in the Finder’s Go menu to mount each media volume.

Chapter 2 Preparing a Network for Distributed Processing 29

Page 30

On each computer, you need to:

Â Enter another computer’s name in the Connect to Server dialog.

Â Choose the associated media volume (FireWire drive) as the volume you want to

mount.

Do this until all the computers are mounting all the media volumes in the cluster.

Submitting Processing Jobs in the Sample Part-Time Distributed Processing Setup

After you finish the final step above, each one of these computers can be used to submit jobs for distributed processing.

Important: Because of the way access has been configured in this setup, all file

pathnames are conveniently consistent and simple for the purposes of specifying them in Compressor, in Shake scripts, and in Apple Qmaster, assuming that:

Â Users place the source media on a mounted media volume (one of the FireWire drives).

Â Users place the Shake scripts on a mounted media volume.

Â All folders and files on the shared media volumes have Read and Write access

enabled for everyone (for Owner, Group, and Others). You can make this access setting by selecting the folder or file and choosing File > Get Info.

The above three assumptions are important because they ensure that all the computers have Read and Write access to all the source files and output destinations.

Specifying Media File and Script Locations

The following additional configuration guidelines apply to anyone using Shake (or any other UNIX-based rendering applications).

Specifying the Media File Locations in Shake Scripts

In the above setup, all the Shake render scripts should specify their source media (File In) locations and output (File Out) destinations as: /Volumes/MediaDiskName.

For example: /Volumes/Media3.

Specifying Shake Script Locations in Apple Qmaster

In the above setup, all the Shake script locations should be specified in Apple Qmaster as: /Volumes/MediaDiskName/ScriptFilename.

For example: /Volumes/Media3/Script.shk.

30 Chapter 2 Preparing a Network for Distributed Processing

Page 31

The Interfaces

Instead of one individual interface, the Apple Qmaster distributed processing system includes up to four different applications and utilities for configuring, monitoring, and managing services.

This chapter introduces the following elements of the Apple Qmaster distributed processing system:

The Interfaces in the Apple Qmaster Distributed Processing System (p. 31)

Apple Qmaster Pane in System Preferences (p. 33)

Apple Qadministrator (p. 35)

Client Interfaces: Compressor and Apple Qmaster (p. 37)

Batch Monitor (p. 39)

The Interfaces in the Apple Qmaster Distributed Processing System

The Apple Qmaster system is a suite of applications that work together to provide maximum power and flexibility for distributed processing. The elements of the system can be combined in a variety of different ways to suit your needs.

Page 32

In general, you use the Apple Qmaster pane in System Preferences to configure service nodes and cluster controllers, and to create simple clusters. System administrators use Apple Qadministrator for advanced cluster creation and control. Next, client users use Compressor or Apple Qmaster to submit batches of jobs for processing. Then, the Batch Monitor can be used by both administrators and client users to monitor batches.

System Preferences

Users: Administrators

Use to: Create service nodes and

cluster controllers

Batch Monitor

Apple Qadministrator

Users: Administrators

Use to: Assemble clusters of service nodes

and cluster controllers

QuickCluster

Compressor or Apple Qmaster

Users: Administrators and client users

Use to: Monitor batches that have

been sent to clusters

Note: It is possible to create a simple (personal) distributed processing system and skip Apple Qadministrator altogether. See “Apple Qmaster Pane in System Preferences” (next) and “About QuickClusters” on page 53 for more information.

32 Chapter 3 The Interfaces

Use to: Submit jobs to clusters for processing

Users: Client users

Page 33

Apple Qmaster Pane in System Preferences

Use the Apple Qmaster pane in System Preferences to activate, create, or make changes to Apple Qmaster cluster-controlling and processing services (including passwords and scratch storage locations).

Use Apple Qmaster System Preferences to configure service nodes and cluster controllers:

Service

node

Cluster

controller

For details about using the Apple Qmaster pane in System Preferences, see:

Â “Configuring Service Nodes and Cluster Controllers” on page 42

Â “Using Virtual Clusters to Make the Most of Multiprocessor Computers” on page 49

Â “Setting a Service Password for Including a Computer in a Cluster” on page 62

Â “Cluster Storage: Setting a Scratch Storage Location” on page 62

Also see the Apple Qmaster User Manual for more information, including creating an extended node cluster which uses nodes without Apple Qmaster installed.

Chapter 3 The Interfaces 33

Page 34

To open the Apple Qmaster pane in System Preferences:

1 Open System Preferences.

2 Click the Apple Qmaster button, located in the Other section.

The Apple Qmaster pane appears.

34 Chapter 3 The Interfaces

Page 35

Apple Qadministrator

Use the Apple Qadministrator application to create and modify Apple Qmaster clusters. Apple Qadministrator can be used on any computer that is on the same network as the cluster you want to administer. With the administrative password (if one was created), you can also use Apple Qadministrator to see and modify existing clusters on the network.

Use Apple Qadministrator to assemble clusters:

Drag available service nodes to assign them to a cluster.

For details about using Apple Qadministrator, see:

Â “Creating Clusters with Apple Qadministrator” on page 50

Â “Modifying and Deleting Clusters With Apple Qadministrator” on page 58

Â “Monitoring Cluster Activity” on page 59

Â “Setting Cluster Preferences” on page 60

Â “Setting Cluster Administrator and User Passwords” on page 61

Chapter 3 The Interfaces 35

Page 36

To open Apple Qadministrator:

m Double-click the Apple Qadministrator icon in the Applications folder.

The Apple Qadministrator window appears. If a password was created for the currently selected cluster, you will not be able to see or modify the cluster until you click the Lock button and then enter the password in the dialog that appears.

36 Chapter 3 The Interfaces

Page 37

Client Interfaces: Compressor and Apple Qmaster

Client computer users use either Compressor, or the interface called Apple Qmaster, to submit batches for processing.

Batch Batch

Use Compressor to submit a batch to a cluster:

Batch Batch

Cluster A

Network

ster B

Use Apple Qmaster to submit

a batch to a cluster:

Chapter 3 The Interfaces 37

Page 38

Using Compressor

Use the Cluster pop-up menu in the Compressor Batch window to choose a cluster for any given batch. For more information on submitting batches with Compressor, see the

Compressor User Manual.

To open Compressor:

m Double-click the Compressor icon in the Applications folder.

The Compressor default window layout appears.

Using Apple Qmaster

Â For Shake processing batches, you can drag Shake files into the Apple Qmaster

window. A default script for submitting the jobs is automatically created. In Apple Qmaster, you can then specify certain details, such as which cluster to use, and make adjustments to certain settings.

Â For Maya batches, there is also a special interface within Apple Qmaster for

submitting and customizing Maya jobs.

Â You can use the Generic Render command in Apple Qmaster for the distributed

processing of projects from other frame-based rendering applications (such as After Effects and LightWave).

For complete information about the Apple Qmaster application, see the Apple Qmaster User Manual.

38 Chapter 3 The Interfaces

Page 39

To open Apple Qmaster:

m Double-click the Apple Qmaster icon in the Applications folder.

The Apple Qmaster window appears.

Batch Monitor

As an administrator, you can use the Batch Monitor to track the progress of all the batch activity for all the clusters on your network. You can see how close to completion each job is, along with other details, and you can stop, resume, or delete batches as well. If you are a client user, you can use the Batch Monitor to view and manage your own batches.

Use the Batch Monitor to see information about batches that have been sent to specified clusters:

Status of

batches

Cluster Cluster

Status of

batches

Network

For complete information about the Batch Monitor, see the Batch Monitor User Manual.

Chapter 3 The Interfaces 39

Page 40

To open the Batch Monitor, do one of the following:

m Use the Batch Monitor that automatically opens after you submit a batch with

Compressor or Apple Qmaster.

m Double-click the Batch Monitor icon in the Utilities folder in the Applications folder.

m Click the Batch Monitor button in the Apple Qmaster window or in the Compressor

batch window.

m In Apple Qadministrator, choose Cluster > Show Batch Monitor.

The Batch Monitor appears.

40 Chapter 3 The Interfaces

Page 41

4 Creating and

Administering Clusters

Once your network is set up and you have installed the necessary components, you are ready to create distributed processing clusters.

As the administrator of your distributed processing network, you can set a number of cluster options and security controls. You may also want to know about failure notification and recovery features.

This chapter covers the following:

Â An Overview of Configuring a Cluster (p. 42)

Â Configuring Service Nodes and Cluster Controllers (p. 42)

Â Creating Clusters with Apple Qadministrator (p. 50)

Â About QuickClusters (p. 53)

Â Advanced Settings in the Apple Qmaster Preferences Pane (p. 54)

Â Modifying and Deleting Clusters With Apple Qadministrator (p. 58)

Â Monitoring Cluster Activity (p. 59)

Â Setting Cluster Preferences (p. 60)

Â Setting Passwords and Scratch Storage (p. 61)

Â Recovery and Failure Notification Features (p. 65)

If you have questions about any concepts and terms used here, refer to the preface, “Introduction to Distributed Processing” on page 5.

Page 42

An Overview of Configuring a Cluster

Assuming Apple Qmaster software is installed on all the computers that you plan to use as part of the cluster, there are three basic steps involved in configuring a cluster.

Note: If you are an Apple Qmaster user and you want to create a cluster that includes computers that do not have Apple Qmaster installed, see the Apple Qmaster User Manual for instructions.

Step 1: Configure service nodes

Configuring a service node to perform distributed processing is a matter of turning on processing services in the Apple Qmaster pane in System Preferences. Optionally, you can also set passwords at this time. See “Configuring Service Nodes and Cluster

Controllers,” below, for details.

Step 2: Configure a cluster controller

To configure a computer to control the cluster, turn on the cluster control services in the Apple Qmaster pane in System Preferences. See “Turning Cluster Controller Services

On or Off” on page 48 for more information.

Step 3: Create a cluster

Important: You can create a simple “QuickCluster” in the Apple Qmaster pane in System

Preferences, or you can create a “managed cluster” from the service nodes and cluster controller using Apple Qadministrator. After a cluster is created, client applications on the same network can start sending batches to the cluster. You can use Apple Qadministrator from any computer (with Apple Qadministrator installed) that is on the same network as an Apple Qmaster cluster. See any of the following for more details:

Â “Getting Started Quickly” on page 15

Â “Creating QuickClusters” on page 53

Â “Creating Clusters with Apple Qadministrator” on page 50

Configuring Service Nodes and Cluster Controllers

Once service processing or cluster controlling is enabled on a computer, the computer is advertised on the network as available to be used in a cluster.

There can only be one cluster controller in a cluster. However, a computer can be designated as both a cluster controller and a service node (see “Using One Computer

to Serve Two Distributed Processing Roles” on page 22).

42 Chapter 4 Creating and Administering Clusters

Page 43

Configuring Service Node Processing

Use the Apple Qmaster pane in System Preferences or Apple Qadministrator to set processing services on a computer.

To turn on processing services:

1 Open the Apple Qmaster pane in System Preferences.

2 Optionally, you can configure a number of settings before you turn on the processing

services. (See “Options in the Apple Qmaster Pane in System Preferences” on page 45.)

Note: It’s easiest to do this now because you can’t make these settings when processing services are enabled. To make these settings after services have been enabled, you need to turn off the services, make the settings, and then turn the services on again.

3 In the “Share this computer as” section, select one of the following buttons:

Â “QuickCluster with services”

Â “Services and cluster controller”

Â “Services only”

4 In the Services section, do one of the following:

Â In the Share column, select the checkbox for Compressor Processing (for Compressor

services).

Â In the Share column, select the checkbox for Rendering (for Apple Qmaster services).

5 Click Start Sharing.

The processing service is enabled, making this computer a service node that can process batches.

Chapter 4 Creating and Administering Clusters 43

Page 44

Service Node Status Indicator in the Menu Bar

By default, once a service node is enabled, an Apple Qmaster icon appears in the computer’s menu bar. The icon now changes color based on the service node’s current status:

Â Gray = Idle

Â Green = Processing

Â Red = Connection failure

Click the icon to reveal additional information, including the service node’s IP address, port number, and capture status, as well as a link to the Apple Qmaster pane in System Preferences.

For information on enabling and disabling this display, see “Show Qmaster service

status in menu bar” on page 57.

44 Chapter 4 Creating and Administering Clusters

Page 45

Options in the Apple Qmaster Pane in System Preferences

You can configure any of the following settings before you turn on processing services. (In order for you to change any of the following settings, processing services must be turned off.)

Setting the Name

By default, a computer is identified on the network by its computer name (as it is entered in the Sharing pane in System Preferences). You can change this name to something more meaningful if you like, since it is the name used to identify this computer in the Apple Qmaster distributed processing system. If you are setting up a QuickCluster, this is the name that will appear in the Compressor Cluster pop-up menu or the Apple Qmaster Submit To pop-up menu. If you are setting up a managed cluster controller, this is the name that will appear in the Apple Qadministrator Controller pop-up menu.

Â If you are setting up a QuickCluster, enter the new name for the cluster in the

“Identify this QuickCluster as” field.

Â If you are setting up a controller to use with Apple Qadministrator, do the

following:

a Click Advanced to open the Advanced pane.

b Enter the new name in the “Identify this computer to Apple Qadministrator as” field.

Unmanaged Services

You can enable unmanaged services for a QuickCluster. For more information, see “Managed Vs. Unmanaged Services” on page 46.

Setting the Password

To add a password requirement, click the “Require password” checkbox.

Â If you are setting up a QuickCluster, other users will be required to enter this

password before being allowed to submit requests to this computer.

Â If you are setting up a cluster to use with Apple Qadministrator, an administrator

will be required to enter this password before being allowed to add this computer to a cluster.

For more information, see “Setting Passwords and Scratch Storage” on page 61. (If you configure the computer as both a cluster controller and a service node, this password is used for both.)

Setting Cluster Storage

You can change the default scratch storage for processing that occurs on this computer in the Advanced pane. For more information, see “Cluster Storage: Setting a

Scratch Storage Location” on page 62.

Chapter 4 Creating and Administering Clusters 45

Page 46

Managed Vs. Unmanaged Services

You have flexibility in how you build clusters for distributed processing with Compressor or Apple Qmaster. When you turn on processing services (See “Turning

Cluster Controller Services On or Off” on page 48), you can choose to make them either

managed services or unmanaged services (the default).

Managed Services

Managed services can be assigned to serve one particular cluster controller. Once assigned, managed services remain exclusively dedicated to that cluster until they are removed with the Apple Qadministrator application. QuickClusters cannot use managed services from other nodes, except in the case of extended node clusters. For more information, see “Modifying and Deleting Clusters With Apple Qadministrator” on page 58. See the Apple Qmaster User Manual for more information on extended node clusters.

Unmanaged Services

Unmanaged services will automatically assign themselves to the first available QuickCluster with enabled unmanaged services support. QuickClusters listen for unmanaged service advertisements and may mark or remember any of them for later use. A QuickCluster can use any available unmanaged service on the same local network (subnet). An unmanaged service will remain dedicated to its QuickCluster only long enough to finish the current job. Once the current job is complete, an unmanaged service is once again a “free agent,” and will advertise its availability to all QuickClusters.

Note: Managed clusters (those created with Apple Qadministrator) can also use unmanaged services. When unmanaged services support is enabled on a “managed” cluster, the cluster will automatically add any available unmanaged services in addition to its managed services (that were explicitly added using Apple Qadministrator).

46 Chapter 4 Creating and Administering Clusters

Page 47

Enabling Unmanaged Services on QuickClusters

Follow these steps to enable unmanaged services on QuickClusters.

1 Open the Apple Qmaster pane of System Preferences.

2 Select the “QuickCluster with services” button to create a QuickCluster.

3 Click “Include unmanaged services.”

4 Click Start Sharing.

For more information on creating QuickClusters, see “Getting Started Quickly” on page 15 and “About QuickClusters” on page 53.

Enabling Unmanaged Services on Managed Clusters

Follow these steps to enable unmanaged services on Managed Clusters.

1 Select a cluster in the Cluster list, or click the Add (+) button to add a new cluster.

2 Click “Allow use of unmanaged services.”

For more information on creating managed clusters, see “Creating Clusters with

Apple Qadministrator” on page 50.

To set the type of processing service:

In the Apple Qmaster pane in System Preferences, do one of the following with the checkbox in the Managed column:

m For managed services, select the checkbox.

m For unmanaged services, deselect the checkbox.

Note: If processing services are enabled, you must turn them off before you can adjust the processing service type.

Chapter 4 Creating and Administering Clusters 47

Page 48

To turn off processing services

1 In the Apple Qmaster pane in System Preferences, click Stop Sharing.

2 In the dialog that appears, enter the number of minutes you want processing services

to continue before shutting down, then click OK.

The default is 0 minutes, but you can enter a different number in the field. If you enter any number greater than 0, a countdown appears next to the Cancel button. The shutdown delay is provided because some computers may be in the middle of processing batches, and these could be damaged by a premature shutdown.

3 In the Services section, do one of the following:

Â Deselect the On checkbox for Compressor Processing (to turn off Compressor services).

Â Deselect the On checkbox for Rendering (to turn off Apple Qmaster services).

Turning Cluster Controller Services On or Off

Use the Services pane in the Apple Qmaster pane in System Preferences to turn the cluster controller on or off on a specific computer.

To turn on cluster controller services:

1 Open the Apple Qmaster pane in System Preferences.

2 Optionally, you can configure a number of settings before you turn on the processing

services. (See “Options in the Apple Qmaster Pane in System Preferences” on page 45.)

3 In the “Share this computer as” section, select one of the following buttons:

Â QuickCluster with services: Choose this option to create an “instant” cluster with

unmanaged services.

Â Services and cluster controller: Choose this option to build a cluster in

Apple Qadministrator. (See “Creating Clusters with Apple Qadministrator” on page 50 for more information.)

Also see “Managed Vs. Unmanaged Services” on page 46 for more information.

4 In the Share column, select the checkbox.

5 Click Start Sharing.

The cluster is enabled, making this computer a cluster controller.

48 Chapter 4 Creating and Administering Clusters

Page 49

To turn off cluster controller services:

1 Open the Apple Qmaster pane in System Preferences.

2 Click Stop Sharing.

3 In the dialog that appears, enter the number of minutes you want controller services to

continue before shutting down, then click OK.

Note: Optionally, you can turn off the controller and just make the node a service node. To do this, select “Services only” in the “Share this computer as” section.

Using Virtual Clusters to Make the Most of Multiprocessor Computers

If any of the service node computers in your network have multiple processors, you can adjust the number of instances of processing service, essentially creating virtual clusters on individual computers. For process-intensive work, having many instances may increase speed and efficiency, depending on the processing application.

Note: By default, the Apple Qmaster system creates one Rendering service instance for each processor. Rendering services are for Shake (with Apple Qmaster), Alias Maya, and other UNIX command-line programs, as opposed to Compressor services, which are solely for Compressor distributed processing. Consult the documentation that came with the application to see if using each processor individually is ideal.

Chapter 4 Creating and Administering Clusters 49

Page 50

Choose the number of

instances from the

pop-up menu.

To change the number of instances of processing services on a computer:

1 Open the Apple Qmaster pane in System Preferences.

2 Select the Compressor or Rendering service in the Services section.

3 Click the “Options for selected service...” button.

4 In the dialog that appears, choose the number of instances from the pop-up menu.

5 Click OK.

6 In the Services section, select the Share checkbox for the service.

Note: The Service Options dialog is also used to add extended nodes to a cluster, as described in the Apple Qmaster User Manual.

Creating Clusters with Apple Qadministrator

Once you configure managed service nodes and/or cluster controllers, they are visible in Apple Qadministrator, which you use to create and modify Apple Qmaster clusters.

There are two basic steps to creating a managed cluster with Apple Qadministrator. First, you create a new cluster and choose the cluster controller. Then, you add service nodes to the cluster.

50 Chapter 4 Creating and Administering Clusters

Page 51

Click the Add button,

then rename the

Untitled Cluster.

Step 1: Create a new cluster

1 Open Apple Qadministrator, and then do the following:

a Click the Add ( + ) button.

b Select Untitled Cluster and rename it. (The cluster name you create will also appear

in the cluster pop-up menus in the Batch Monitor and Apple Qmaster.)

2 From the Controller pop-up menu, choose a cluster controller from those available on

the network.

Use this pop-up menu to choose a cluster controller.

Note: If a password was created for the cluster controller in System Preferences, a password authentication dialog appears.

3 Optionally, create cluster passwords by clicking the Security tab and selecting and

entering the passwords you want.

Â Administrator Password: If you create this password, administrators will need to

know it in order to modify this cluster and to view this cluster’s batches in the Batch Monitor.

Â User Password: If you create this password, users will need to know it in order to

submit batches to this cluster and to view those batches in the Batch Monitor.

Chapter 4 Creating and Administering Clusters 51

Page 52

Click this disclosure

available nodes.

Step 2: Assign service nodes to the cluster

1 If the Qmaster Service Browser is not already displayed, click the disclosure triangle

to see it.

triangle to see

2 Add service nodes to the new cluster by dragging them from the Qmaster Service

Browser list at the bottom of the window, up to the cluster’s service nodes list.

Â If there is a closed lock icon next to a computer name, click the lock and enter the

password that was assigned to it in the Apple Qmaster pane in System Preferences. Otherwise, you won’t be able to drag that service node into the cluster.

Â Service nodes that are already assigned to another cluster are not shown.

Drag nodes to this list from the Service Browser list.

Clicking this disclosure

triangle displays each

instance of the services

set in System Preferences

for this node.

Note: The computer names you see at the top level of the Name columns may appear in one of three formats, depending on your configuration: the computer name (for example, Lemur node), the Apple networking name (for example, Lemur-node.local), or the network address for the computer (for example, 02030b-dhcp45.company.com).

3 When you have finished adding service nodes, click Apply.

Your cluster is now ready to process batches.

Note: Although Apple Qadministrator currently allows you to create a cluster with unmanaged services, valid clusters require at least one managed service for the cluster to be viewable in Apple Qmaster, Compressor, and Batch Monitor.

52 Chapter 4 Creating and Administering Clusters

Page 53

About QuickClusters

QuickClusters offer a simple and automated way to create and configure clusters, and an alternative to creating and configuring clusters manually with Apple Qadministrator. QuickClusters with enabled unmanaged support will auto-configure themselves and use any available unmanaged services on the same local network (subnet). QuickClusters listen for unmanaged service advertisements and may mark or remember any of them for later use.

Creating QuickClusters

You can create and modify QuickClusters in the Apple Qmaster pane in System Preferences.

1 Open the Apple Qmaster pane in System Preferences.

2 Under “Share this computer as,” click “QuickCluster with services.”

3 Optionally, you can configure a number of settings before you turn on the processing

services. (See “Options in the Apple Qmaster Pane in System Preferences” on page 45.)

4 Click Start Sharing.

This creates a QuickCluster with this computer as its controller.

Chapter 4 Creating and Administering Clusters 53

Page 54

Note: With an active QuickCluster, Apple Qmaster users can create extended node clusters, which contain one or more computers that do not have Apple Qmaster installed. See the Apple Qmaster User Manual for more information.

About AutoClusters

The AutoCluster feature in Compressor gives you an easy way to take advantage of the distributed processing capabilities offered by Apple Qmaster without requiring a lot of knowledge about how clusters are configured, setting up file sharing, and so on. The AutoCluster feature in Compressor allows you to do the following steps, on the fly, with just one click, at the moment of batch submission:

Â Turn “This computer” into a temporary QuickCluster.

Â Use unmanaged services from other computers.

For more details on AutoClusters, see “Using Compressor AutoClusters” on page 23.

Advanced Settings in the Apple Qmaster Preferences Pane

You can use the Advanced section in the Apple Qmaster Preferences Pane to further configure your distributed processing system.

54 Chapter 4 Creating and Administering Clusters

Page 55

Advanced Service Settings

Use these features to schedule service restarts and service availability.

Restart all services every 24 hours

The “Restart all services every 24 hours” checkbox ensures a robust distributed processing system. Refreshing the services periodically prevents increased virtual memory sizes and memory leaks in third-party software.

Set schedule for unmanaged services

If you enabled unmanaged services, you can open a calendar interface and schedule the availability of these services to the distributed processing system.

To schedule service availability:

1 Click Set.

The work schedule interface appears.

By default, the services are available 24 hours a day, 7 days a week.

2 To constrain the availability of the services, click the pop-up menu next to a particular

day of the week, and choose one of the following:

Â Off makes the service unavailable on that day of the week.

Â On between allows you to enter the period of time the service will be available.

Â Off between allows you to enter the period of time the service will not be available.

3 Enter any constraining time periods in the appropriate time fields.

4 Click OK to save the settings.

Chapter 4 Creating and Administering Clusters 55

Page 56

Note: You must enter valid days and times. The time cannot overlap into the next day in one entry. There must be two entries when the range ends after 12 A.M. For example, to set the schedule availability from 6 P.M. on Sunday to 8 A.M. on Monday, enter the following two entries:

Â Sunday ON BETWEEN 06:00 P.M. and 12:00 A.M.

Â Monday ON BETWEEN 12:00 A.M. and 8:00 A.M.

Shared Cluster Storage

Use these features to configure scratch storage for this computer’s cluster controller. For more information on cluster storage, see “Cluster Storage: Setting a Scratch Storage

Location” on page 62.

Delete Files Older Than __ Days

Enter the number of days temporary process files may remain on the cluster’s scratch location before they are automatically deleted. If you anticipate a transcoding session that will last up to seven days or longer, you must adjust this value.

Cluster Storage

Enter a local folder directory to change the scratch location for the cluster’s temporary process files. See “Cluster Storage: Setting a Scratch Storage Location” on page 62 for more information.

Network

Use these features to configure network settings.

Allow wide area Bonjour in domain

Select this checkbox to have Apple Qmaster use computers beyond the local subnet, and enter the DNS Domain in the text field. This feature requires Mac OS X v10.4 or later.

For more information about using WAN browsing with your distributed processing network, go to:

Â http://developer.apple.com/networking/bonjour/index.html

Â http://www.dns-sd.org

Â http://www.dns-sd.org/ServerSetup.html

56 Chapter 4 Creating and Administering Clusters

Page 57

Use Network Interface

Restrict distributed processing activity to a particular network interface card by choosing it from this pop-up menu. If you do this on a service node computer, use a different computer to submit Compressor jobs and batches.

Enable Port Range

With Apple Qmaster 3, you can define which ports Apple Qmaster uses for service advertisements with the Enable Port Range checkbox and text fields in the Network section of the Apple Qmaster pane in System Preferences.

To define ports for Apple Qmaster service advertisements:

1 Open the Apple Qmaster pane in System Preferences.

2 Click Advanced to open the Advanced pane.

3 Select Enable Port Range.

4 In the From field, enter any integer value between 50,000 and 65,535 to set the start

of the range, such as 50,000.

5 In the “Number of ports” field, enter the size of the range, such as 1,000.

Extras

Use these features to configure additional settings.

Log service activity to file

If selected, an activity log is created and updated regularly with information about the Apple Qmaster actions on this computer. Logs are stored in /Library/Logs/Qmaster/. To turn this feature off, deselect the checkbox.

Note: You can also access this log information using the Log button in Apple Qadministrator or the Log tab in Batch Monitor.

Show Qmaster service status in menu bar

If selected, an Apple Qmaster icon appears in the computer’s menu bar. The menu bar item provides Apple Qmaster status and activity information. For more information, see “Service Node Status Indicator in the Menu Bar” on page 44.

Identify this computer to Apple Qadministrator as

By default, a computer is identified on the network by its computer name (as it is entered in the Sharing pane in System Preferences). You can change this name to something more meaningful if you like, since it is the name used to identify this computer in the Apple Qadministrator application. If you are setting up a managed cluster controller, this is the name that will appear in the Apple Qadministrator Controller pop-up menu.

Chapter 4 Creating and Administering Clusters 57

Page 58

Modifying and Deleting Clusters With Apple Qadministrator

Using Apple Qadministrator, you can change and delete clusters. Once a cluster is configured, you can use Apple Qadministrator to deactivate and reactivate the processing services on a computer in the cluster, to add a service node to the cluster, or to remove a service node from the cluster.

Note: Unless Apple Qadministrator and the QuickCluster are on the same computer, QuickClusters are not visible in Apple Qadministrator. Only “managed” clusters can be modified and deleted in Apple Qadministrator. Managed clusters are clusters that were created in Apple Qadministrator. QuickClusters must be modified in the Apple Qmaster pane in System Preferences.

Note: If you want to change the cluster controller in a cluster, you need to delete the cluster and then re-create it with a new cluster controller.

To modify a cluster:

1 Open Apple Qadministrator.

2 In the Cluster list (on the left side of the window), select the cluster you want to change.

If the cluster’s Service Nodes list isn’t already showing individual services, click the disclosure triangle in the Name column to reveal them.

Make any of the following changes:

Â To temporarily turn off the processing services on a computer in the cluster, deselect the

Active checkbox for that service node.

Â To remove a service node from the cluster, select the computer and drag it back to the

Qmaster Service Browser at the bottom of the Apple Qadministrator window.

Â To add a service node to the cluster, drag it from the service browser list at the bottom

of the window to the service nodes list.

58 Chapter 4 Creating and Administering Clusters

To remove nodes from a cluster, drag them to the Qmaster Service Browser.

Page 59

3 Click Apply Now.

Note: To turn off the service node or cluster controlling services on any computer within a cluster, see “Configuring Service Node Processing” on page 43 and “Turning

Cluster Controller Services On or Off” on page 48.

To change a cluster’s name in Apple Qadministrator:

1 In the Cluster list, double-click the cluster name.

2 Type a new name, then press Return.

3 Click Apply Changes.

To delete a cluster in Apple Qadministrator:

1 In the Cluster list, select the cluster you want to delete.

2 Click the Remove ( – ) button.

Monitoring Cluster Activity

You can use Apple Qadministrator to find out what is happening within a cluster by examining details (such as CPU usage, which batch is being processed, disk space usage, and data activity) about each node in the cluster.

To monitor cluster activity in Apple Qadministrator:

1 In the Cluster list, select the cluster you want to examine.

2 Select a node in the cluster service nodes list.

3 Click the Info ( I ) button near the bottom of the Cluster list.

4 In the resulting window, click the CPU, Memory, Volume Info, or I/O Activity tabs to see

a variety of details about the selected node.

Accessing Activity Logs

Apple Qadministrator provides log information for a selected service or cluster. This information is useful for troubleshooting distributed processing issues, and it can be saved to a file and processed with XML tools and UNIX scripts.

To view or copy log information in Apple Qadministrator:

1 In Apple Qadministrator, select the service or cluster whose log you wish to view.

2 Click the Log button near the bottom of the Cluster list.

A new window appears displaying the log information for the selected service or cluster. The log file is deleted when you close the window.

Chapter 4 Creating and Administering Clusters 59

Page 60

Setting Cluster Preferences

You can use Apple Qadministrator Preferences to configure several aspects of Apple Qadministrator.

To set cluster preferences:

1 Open Apple Qadministrator.

2 In the Cluster list, select the cluster for which you want to set preferences.

3 Click Preferences.

Click Preferences to display the Preferences pane.

4 Set any of the following preferences:

Queue

Â Maximum number of jobs in the queue: Enter the maximum number of batches that

can be queued up at one time for this cluster. If the maximum number is reached, the cluster does not accept new batches until there is an opening in the queue.

Â Keep job history for: Set how long batches are listed in the History table of the

Batch Monitor.

Â Email notification for service down after: Set how much time should pass, after a

service becomes inaccessible, before the cluster controller sends an alert message to the administrator. (See “Email Notification,” below.)

Â Status Interval: Set how often status information about this cluster should be

generated and sent to the Batch Monitor.

Email Notification

To have the cluster controller send service failure alerts to an administrator, enter the relevant information in the fields provided. See “Recovery and Failure Notification

Features” on page 65 for more information.

Â Admin Email: Enter the administrator’s email address.

Â Send with mail server: Enter the administrator’s mail server.

Â Domain: Enter the cluster controller’s domain.

Note: The Apple Qmaster distributed processing system does not currently support SMTP servers that require authentication.

60 Chapter 4 Creating and Administering Clusters

Page 61

Setting Passwords and Scratch Storage

You can create several different types of passwords for the Apple Qmaster distributed processing system. All these passwords are optional; you can use the system without creating them.

Â Cluster administrator password: A password required for modifying a cluster in

Apple Qadministrator, and for modifying the status of the cluster’s batches in the Batch Monitor. See “Setting Cluster Administrator and User Passwords,” next.

Â Cluster user password: A password that client users will need in order to submit

batches to a cluster and to modify the status of those batches in the Batch Monitor. See “Setting Cluster Administrator and User Passwords,” next.

Â Service password: A password required for an administrator to add a specific service

node or cluster controller to a cluster. See “Setting a Service Password for Including a

Computer in a Cluster” on page 62.

You can also change the default scratch storage location for a cluster, or for each computer in a cluster. See “Cluster Storage: Setting a Scratch Storage Location” on page 62.

Setting Cluster Administrator and User Passwords

You can create cluster passwords while creating a new cluster, as described in “Creating

Clusters with Apple Qadministrator” on page 50. However, once the cluster is created,

you can still add or change passwords, using the same settings in Apple Qadministrator.

To create or change cluster passwords:

1 In Apple Qadministrator, select the cluster from the Cluster list.

2 Click Security.

3 Select and enter or change the passwords you want.

4 Click Apply Changes.

Note: Cluster administrator and cluster user passwords can be stored in a user’s keychain.

Chapter 4 Creating and Administering Clusters 61

Page 62

Setting a Service Password for Including a Computer in a Cluster

If you want to control who is able to include a specific service node or cluster controller in a cluster, you can create a password called a service password for the computer.

Note: A service password can be stored in a user’s keychain.

To set a service password:

1 On the computer designated as the service node or cluster controller, open the

Apple Qmaster pane in System Preferences.

2 If any Apple Qmaster services are enabled on this computer, temporarily turn them off

by clicking Stop Sharing.

3 Click Require Password.

The password sheet opens.

4 Enter and verify a password, and click OK.

5 Click the relevant checkbox or the Start Sharing button to restart the services you need

on this computer.

Cluster Storage: Setting a Scratch Storage Location

By default, the Apple Qmaster distributed processing system saves temporary process files in the /var/spool/qmaster directory on the cluster controller. You can also choose any other location on a local disk. Computers in the cluster will access this location as needed.

Selecting a Cluster Storage Location

Follow the steps below to change the cluster storage location.

To select a new storage location for a cluster:

1 On the cluster controller, open the Apple Qmaster pane in System Preferences.

2 If any Apple Qmaster services are enabled on this computer, click Stop Sharing to

temporarily turn them off.

62 Chapter 4 Creating and Administering Clusters

Page 63

3 Click Advanced to open the Advanced pane.

Click Set to open a dialog for choosing a new storage folder.

4 Click the Set button next to the Cluster Storage field.

5 Navigate to the folder in the dialog, select it, and then click Choose.

Note: If you are using the default This Computer setting in the Cluster pop-up menu in the Compressor Batch window, and you choose Cluster Storage as the destination, the output file will be copied to the Source location.

Cluster Storage Capacity

If you are processing large source media files that exceed the available storage space on the startup disk, you may run out of storage space on that disk. There are a number of things you can do to address this.

If cluster storage capacity is low, do any of the following:

m Change the cluster storage location to a disk with more free space. For more information,

see below.

m Configure cluster storage settings to delete files more frequently. For more information,

see “Adjusting Cluster Storage Settings,” below.

m Compressor users can set Cluster Options preferences (Compressor > Preferences) to

“Never copy source to cluster.” For more information, see the Compressor User Manual.

Chapter 4 Creating and Administering Clusters 63

Page 64

Enter a new number to

change how often cluster

storage files are deleted.

Adjusting Cluster Storage Settings

Follow the steps below to change scratch storage settings for a cluster.

To change cluster storage settings:

1 On the cluster controller, open the Apple Qmaster pane in System Preferences.

2 If any Apple Qmaster services are enabled on this computer, click Stop Sharing to

temporarily turn them off.

Note: Do not attempt to change the cluster storage settings while the cluster is turned on.

3 Click Advanced to open the Advanced pane.

Click Set to open a dialog for choosing a new storage folder.

Click Stop Sharing to turn off the cluster before changing any settings.

4 Do any of the following:

Â To change the cluster storage location:

Â Click the Set button next to the Cluster Storage field.

Â Navigate to the folder in the dialog, select it, and then click Choose.

Â To change how often cluster storage files are deleted, enter a new number in the

“Delete Files Older Than _ Days” field.

5 Click the Start Sharing button to restart the cluster.

64 Chapter 4 Creating and Administering Clusters

Page 65

Cleaning Up Cluster Storage

If you are using cluster storage, and an error occurs, partial files may be left on the designated cluster storage location. Check the designated cluster storage location to make sure no partial media files are left there. If you find partial media files, delete them and submit the job again.

Cluster Storage and QuickTime Reference Movies

Strictly speaking, only actual QuickTime movies (not QuickTime reference movies) are supported for distributed processing. If you submit a reference movie for distributed processing, make sure media files specified in the reference movie are available to each node of the Apple Qmaster cluster. In other words, put the media on the shared (cluster storage) volume.

Recovery and Failure Notification Features

The Apple Qmaster distributed processing system has a number of built-in features designed to attempt recovery if there is a problem, and to notify you when it attempts a recovery.

Recovery Features

The recovery actions described next occur automatically if failures occur in the Apple Qmaster distributed processing system. There is no need for you, as the administrator, to enable or configure these features.

If a service stops unexpectedly

If either the cluster controller service or the processing enabled on a service node stops unexpectedly, the Apple Qmaster distributed processing system restarts the service. To avoid the risk of endless stopping and restarting, the system restarts the failed service a maximum of four times. The first two times, it restarts the service right away. If the service stops abruptly a third or fourth time, the system restarts it only if it had been running for at least 10 seconds before the service stopped.

If a batch is interrupted

When a service stops suddenly while in the middle of processing an Apple Qmaster batch, the cluster controller resubmits the interrupted batch in a way that prevents the reprocessing of any batch segments that were complete before the service stopped. The cluster controller delays resuming the batch for about a minute from the time it loses contact with the service.

Chapter 4 Creating and Administering Clusters 65

Page 66

If a batch fails

When the service is running, but one batch fails to process, a

service exception occurs.

When this happens, the cluster controller resubmits the batch immediately. It resubmits the batch a maximum of two times. If the job fails on the third submission, the distributed processing system stops resubmitting the job. In the Batch Monitor, the job is moved to the History table, where the status column indicates that a failure occurred.

Failure Notification

There are two different ways that the Apple Qmaster distributed processing system can provide information about a problem.

Email Notification

When a processing service stops unexpectedly, Apple Qmaster sends a notification email to the address that was entered in the Apple Qadministrator Cluster Preferences dialog for that cluster. If no address was entered there, the email is sent to the address in the Internet settings of the computer on which the cluster controller is enabled.

Note:

Apple Qmaster does not currently support SMTP servers that require authentication.

Log files for individual jobs or batches

If a particular job or batch fails, a log file is generated that describes this failure. You can find the name and location of this log file through the Batch Monitor. Select the batch or job in the History table of the Batch Monitor window, and click the Info icon. If any log files were generated because of failures in the processing of the item, the names and locations of those logs are shown.

Notification and Log Labels

The following table lists the service labels used in the email notifications and logs:

Processing service type Notification label

Local Compressor service servicecontroller:com.apple.stomp.transcoder

Distributed Compressor service servicecontroller:com.apple.stomp.transcoderx

Distributed Apple Qmaster service

66 Chapter 4

servicecontroller:com.apple.qmaster.executor

Creating and Administering Clusters

Page 67

Command-Line Usage

If you are accustomed to doing your work from Terminal shells, and need or prefer to run the distributed processing system from the command line with minimal use of application interfaces, this appendix is for you.

The appendix includes the following topics:

Installing Apple Qmaster from the Command Line (p. 67)

Shell Commands for Configuring Service Nodes and Cluster Controllers (p. 68)

Shell Commands for Submitting Compressor Jobs (p. 69)

Shell Commands for Submitting Apple Qmaster Jobs (p. 71)

Shell Commands for Monitoring Batches (p. 73)

Note:

While it is possible to use the command line to run an Apple Qmaster distributed processing network, each Compressor service node (each computer providing Compressor distributed processing services) must be logged in (with a Mac OS X user name and password) for full functionality.

Appendix

Installing Apple Qmaster from the Command Line

Follow these steps to install Apple Qmaster software on each computer you want to use for distributed processing. (All computers must be on the same subnet.)

Copy the standalone installer package (AppleQmasterNode.mpkg) to the remote system.

Note:

Do not alter the file hierarchy of the packages.

Log into the remote system.

Enter the following command:

sudo installer -pkg AppleQmasterNode.mpkg -target /

Apple Qmaster processes start automatically after installation.

Page 68

4 Optionally, do one of the following to start the Apple Qmaster processes:

Â Enter this command:

sudo SystemStarter start Qmaster

Â Restart the remote computer.

5 Repeat these steps for each computer in your distributed processing network.

You can also install Apple Qmaster using Apple Remote Desktop. Just select a node, click Install Package, and select the standalone installer package (AppleQmasterNode.mpkg). The copy and installation process is done automatically.

Shell Commands for Configuring Service Nodes and Cluster Controllers

As an alternative to using the Apple Qmaster pane in System Preferences, you can use the command qmasterprefs, with command-line options for enabling and disabling service node and cluster-controlling services.

In the command-line descriptions below, angle brackets < > indicate a mandatory argument in a command and brackets [ ] indicate an optional argument.

Note: While it is possible to use the command line to run an Apple Qmaster distributed processing network, each Compressor service node (each computer providing Compressor distributed processing services) must be logged in (with a Mac OS X user name and password) for full functionality.

Synopsis

Below is a synopsis of the command for enabling and disabling cluster-controlling services on a computer. The

qmasterprefs -cluster <on | off> [-timeout <minutes>] [-servername <name>]

[-list] [-help]

qmasterprefs command is located in /usr/sbin.

Below is a synopsis of the command for enabling and disabling service-node processing on a computer.

qmasterprefs -service “Unix Processing” <on | off> [-instances <n>]

[-timeout <minutes>] [-servername <name>] [-list] [-help]

68 Appendix A Command-Line Usage

Page 69

Command options

This table provides information about each of the enabling and disabling service-node and cluster-controlling services on a computer.

Preference command option Description

[-cluster] <on | off> Turns cluster-controlling services on or off.

[timeout min] [servername name] [quickcluster on | off ] [unmanagedservices on | off ] [unmanagedmulticapturethreshold sec] [networkinterface allinterfaces | <bsd name>]

[-service Rendering] <on | off>

[-list] Lists the current qmasterprefs settings.

[-help] Displays information about supported options for qmasterprefs.

[-restart] Restarts Apple Qmaster services.

[-service name options] Sets service options.

Turns UNIX Rendering services on or off. [timeout min] [instances n] [autorestart on | off]

[unmanaged on | off]

Shell Commands for Submitting Compressor Jobs

You can run the Compressor application from the command line using the Compressor command, with a number of command-line options for submitting jobs.

In the command-line descriptions below, angle brackets < > indicate a mandatory argument in a command and brackets [ ] indicate an optional argument.

Synopsis

Below is a synopsis of the command for submitting a job to a cluster. The Compressor command is located in /Applications/Compressor.app/Contents/MacOS.

Compressor [-clustername <name>]

[-clusterid <username:password@ipaddress:portNumber>] [-batchfilepath <path>] [-batchname <batchname>] [-priority <value>] [-jobpath <path>] [-settingpath <path>] [-destinationpath <path>] [-info <xml>] [-timeout <seconds>] [-help] [-show]

In this example, -jobpath, -settingpath, and -destinationpath can be repeated as many times as the number of jobs you want to submit.

Note: Not all the options are necessary. For example, you can specify the cluster either by its -clustername or by its -clusterid. You do not need to specify both. If both are specified, only -clusterid is used.

Additionally, if you specify -batchfilepath, then -jobpath, -settingpath, and

-destinationpath

are not necessary because the previously saved batch file already

contains information about the job, settings, and destination.

Appendix A Command-Line Usage 69

Page 70

Example of -batchfilepath:

Compressor -clustername "This Computer" -batchfilepath "/Volumes/Hermione

/SavedCompressorBatches/FreeChampagne.compressor"

Once the job is submitted successfully, this command displays the batch ID (identifier) and job ID (identifier) in the shell, and you can monitor the progress of a batch in the Batch Monitor.

Command options

This table provides information about each of the command options for submitting jobs.

Submission command option Description

[-clustername <name>] Use to specify the name of the cluster to which you want to send

the job. Using the cluster name, Compressor looks for the cluster on the network in order to use it.

[-password <value>] User password for the cluster specified by ID or name.

[-batchname <name>] Use to specify a name for the batch so that you can easily

recognize it in the Batch Monitor.

[-clusterid <user name:password@IP address:port number>]

[-priority <value>] Specifies the priority level for a job.

[-jobpath <url>] Specifies the location of the source file.

[-settingpath <url>] Specifies the location of the settings for the job.

[-destinationpath <url>] Specifies the destination file URL for the job.

[-info <xml>] Gives detailed information for a batch or a job.

[-timeout <seconds>] Use to specify the number of seconds before Compressor can quit

[-show] Shows the ID information for the cluster specified with

[-help] Displays information regarding the required parameters for the

Optionally, you can use -clusterid to enter the cluster ID and port number instead of using

-clustername. (When you enter the

cluster ID and port, less time is required to find the cluster on the network.)

Or, if you used

-clusterid to specify the user name and password. (You

use

-clustername and the cluster requires a password,

need to include the IP address: port number as well whenever you

-clusterid.)

use Tip: Use

Compressor -show to see a cluster’s IP address and port

number.

when looking for a cluster. The default value is 0, which puts no limit on the timeout and allows Compressor to browse the network for as long as it needs to find the cluster.

-clustername or -clusterid, or for all clusters if no cluster is

specified.

Compressor command.

70 Appendix A Command-Line Usage

Page 71

Example of Compressor Command XML

The code below is an example of XML code for submitting a Compressor command. Notice that because it needs to be entered as one command line, every character after

-options that isn’t alphanumeric must be preceded with a backslash (\). ./Compressor -clusterid tcp://127.0.0.1:51737 -batchname myBatch -jobpath

/Volumes/Source/ShortClips/NTSC24p.mov -settingpath /Users/stomper10 /Library/Application\ Support/Compressor/PhotoJPEG.setting destinationpath /Users/machinename/Movies/myDestinationFilename.mov.

This command has the following elements:

Â Cluster address is tcp://127.0.0.1:51737.

Â Batchname is myBatch.

Â Job path is /Volumes/Source/ShortClips/NTSC24p.mov.

Â Setting path is /Users/stomper10/Library/Application Support/Compressor/

PhotoJPEG.setting.

Destination path is /Users/machinename/Movies

Shell Commands for Submitting Apple Qmaster Jobs

You can use the Apple Qmaster command, Apple Qmaster, with a number of commandline options for submitting jobs.

In the command-line descriptions below, angle brackets < > indicate a mandatory argument in a command and brackets [ ] indicate an optional argument.

Synopsis

Below is a synopsis of the command for submitting a job to a cluster. The Apple Qmaster command is located in /Applications/Apple Qmaster.app/Contents/MacOS.

Apple Qmaster[-clustername <name>] [-clusterid <user name:password@IP

address:port number>][-command <command type> -options <XML command> [-wd <working directory>] [-timeout <seconds>] [-show] [-batchname <name>] [-help]

Once the job is submitted successfully, this command displays the batch ID (identifier) and job ID (identifier) in the shell.

Appendix A Command-Line Usage 71

Page 72

Command options

This table provides information about each of the command options for submitting jobs.

Submission command option Description

[-clustername <name>] Use to specify the name of the cluster to which you want to send

the job. Using the cluster name, Apple Qmaster looks for the cluster on the network in order to use it.

[-batchname <name>] Use to specify a name for the batch so that you can easily

recognize it in the Batch Monitor.

[-clusterid <user name:password@IP address:port number>]

[-command <command type>]

[-options <XML command>] Specifies the command with XML code. Enter the XML code after

[-wd <working directory>]

[-timeout <seconds>] Use to specify the number of seconds before Apple Qmaster can

[-show] Shows the ID information for the cluster specified with -clustername

[-help] Displays information about supported options for Apple Qmaster.

Optionally, you can use -clusterid to enter the cluster ID and port number instead of using -clustername. (When you enter the cluster ID and port, less time is required to find the cluster on the network.)

Or, if you used -clustername and the cluster requires a password, use -clusterid to specify the user name and password. (You need to include the IP address: port number as well whenever you use

-clusterid.) Tip: Use Apple\ Qmaster -show to see a cluster’s IP address and

port number.

Specifies the kind of command you are entering: Shell, Shake, Maya, or other command, depending on the application you want to use for distributed processing.

-options, with the necessary qualifiers for entering it in a shell. See “Example of Shake Command XML,” next, for an example.

If no -option is entered, the values from the application’s preferences file, in ~/Library/Preferences, are used (which are the values of the most recent job submitted).

Use to specify the working directory path (from which the command should be executed). The default working directory is /Applications/Shake.

quit when looking for a cluster. The default value is 0, which puts no limit on the timeout and allows Apple Qmaster to browse the network for as long as it needs to find the cluster.

or -clusterid, or for all clusters if no cluster is specified.

72 Appendix A Command-Line Usage

Page 73

Example of Shake Command XML

The code below is an example of XML code for submitting a Shake command. Notice that because it needs to be entered as one command line, every character after

-options that isn’t alphanumeric must be preceded with a backslash (\).

/Applications/Apple\ Qmaster.app/Contents/MacOS/Apple\ Qmaster -clustername

elvis -command "Shake" -options \<command\ executable\=\"\/Applications\/ Shake\/shake.app\/Contents\/MacOS\/shake\"\ script\=\"\/Volumes\/Jaguar\/ scripts\/applestyle.shk\"\ start\=\"1\"\ end\=\"1000\"\ stepsOf\=\"1\"\ minCount\=\"10\"\ otherOptions\=\"\"\ previewNode\=\"\"\ previewWidth\=\"0\"\ shutterOn\=\"yes\"\ motion\=\"yes\"\ proxyFlags\=\"0\"\ proxyScale\=\"1.000000\"\ proxyRatio\=\"1000.000000\"\ shutter\=\"1.000000\"\>\<\/command\>

Note: Apple Qmaster stores the XML code for the last command you entered in ~Library/Preferences/com.apple.AppleQmaster.plist. You can copy the command in XML form there, and customize it to use for a new job submission.

Shell Commands for Monitoring Batches

You can use the Batch Monitor command, Batch Monitor, with a number of command-line options for monitoring jobs.

In the command-line descriptions below, angle brackets < > indicate a mandatory argument in a command and brackets [ ] indicate an optional argument.

Synopsis

Below is a synopsis of the command for monitoring batches. The Batch Monitor command is located in /Applications/Utilities/Batch Monitor.app/Contents/MacOS.

Batch Monitor [-clustername <name>] [-clusterid <user name:password@IP

address:port number>] [-jobid <identifier> -batchid <identifier>] [-timeout <seconds>] [-query <seconds>] [-help]

To cancel a job or batch:

Batch Monitor [-clustername <name>] [-clusterid <IP address> <port number>

<user name> <password>] -kill -jobid <identifier> -batchid <identifier>

Appendix A Command-Line Usage 73

Page 74

Command options

This table provides information about each of the command options for monitoring batches.

Monitoring command option Description

[-clustername <name>] Use to specify the name of the cluster to which the job was sent.

[-password <value>] User password for the cluster specified by ID or name.

[-clusterid <user name:password@IP address:port number>]

[-jobid <identifier> batchid <identifier>]

[-timeout <seconds>] Use to specify the number of seconds before Batch Monitor can

[-query <seconds>] Use to specify how frequently, in seconds, the job status should

[-kill -jobid <identifier> -batchid <identifier>]

[-help] Displays information about supported options for Batch Monitor.

Optionally, you can use -clusterid to enter the cluster ID and port number instead of using

Or, if you used

-clusterid to specify the user name and password. (You need

use

-clustername and the cluster requires a password,

-clustername.

to include the IP address: port number as well whenever you use

-clusterid.)

Tip: Use

Apple Qmaster -show or Compressor -show to see a

cluster’s IP address and port number.

Use to specify the job you want to monitor. When you use the

-batchid, in the form of the name that was given to the batch

-jobid option, you must also specify the

when it was submitted. (The batchid and jobid are displayed after a batch is submitted.)

If you do not use the

-jobid option, all the jobs submitted to the

specified cluster are listed.

quit when looking for a cluster. The default value is 0, which puts no limit on the timeout and allows Batch Monitor to browse the network for as long as it needs to find the cluster.

be updated.

Cancels the specified job or batch.

74 Appendix A Command-Line Usage

Page 75

Using Scripts to Run Apple Qmaster, Compressor, and Batch Monitor

To simplify your use of the command line, you can add command-line-friendly scripts to /usr/bin. Each script sets an ENV variable for location, then executes the binary. Using the following scripts would be useful if you frequently use the command line for submissions or automated submissions. You do not need to specify the full path and you can add frequently used ENV variables and options to speed up the submission process. To run Apple Qmaster, Compressor, or Batch Monitor, you can simply open up a terminal, and type qmaster, compressor, or batchmonitor.

Apple Qmaster Script Example

#!/bin/csh -f # # set env var, QMASTER_LOCATION if not set # if ${? QMASTER_LOCATION} == 0 then pushd `dirname $0` >& /dev/null setenv QMASTER_LOCATION /Applications/Apple\ Qmaster popd >& /dev/null endif # # launch qmaster # exec ${QMASTER_LOCATION}/Apple\ Qmaster.app/Contents/MacOS/Apple\ Qmaster

$argv:q

Compressor Script Example

#!/bin/csh -f

# # set env var, COMPRESSOR_LOCATION if not set # if ${?COMPRESSOR_LOCATION} == 0 then pushd `dirname $0` >& /dev/null setenv COMPRESSOR_LOCATION /Applications/Compressor.app/Contents/MacOS popd >& /dev/null endif

# # launch Compressor # exec ${COMPRESSOR_LOCATION}/Compressor $argv:q

Appendix A Command-Line Usage 75

Page 76

Batch Monitor Script Example

#!/bin/csh -f

# set env var, BATCHMONITOR_LOCATION if not set

if ${?BATCHMONITOR_LOCATION} == 0 then

pushd `dirname $0` >& /dev/null

setenv BATCHMONITOR_LOCATION "/Applications/Utilities/Batch Monitor.app/

Contents/MacOS"

popd >& /dev/null

endif

# launch Batch Monitor

exec "${BATCHMONITOR_LOCATION}/Batch Monitor" $argv:q

76 Appendix A Command-Line Usage

Page 77

Troubleshooting

If you are having trouble with your Apple Qmaster distributed processing system, look here for answers to your questions.

The appendix includes the following topics:

Exporting from Final Cut Pro and Distributed Processing (p. 77)

QuickTime Reference Movies (p. 80)

Cluster Settings for Extended Transcoding Sessions (p. 80)

Cleaning Up Cluster Storage (p. 80)

Using Apple Qmaster 2 with an NFS Server (p. 80)

Apple Qmaster Distributed Processing and Xsan (p. 80)

Compressor Command-Line Usage Requires Login (p. 81)

Exporting from Final Cut Pro and Distributed Processing

You can use Compressor to transcode sequences or clips from within other applications, such as Final Cut Pro. This saves time and hard disk space by eliminating the need to export self-contained media files before processing them.

Appendix

There are two methods you can use to avoid exporting self-contained media files for Compressor distributed processing:

Export a sequence directly from Final Cut Pro to Compressor and then submit that job to a cluster in an Apple Qmaster distributed processing network. In this workflow, Final Cut Pro opens on each processing node in that cluster to complete the job.

Export the Final Cut Pro sequence as a QuickTime reference movie and then submit the resulting movie to the cluster for processing. Although this method does not require that Final Cut Pro be installed on each computer in the distributed processing cluster, the media files specified in the reference movie must be available to each computer in the cluster.

Page 78

To export a Final Cut Pro sequence directly to Compressor for distributed processing:

1 Make sure Final Cut Pro is installed on each computer in the cluster that you intend to

use for distributed processing.

Each installation of Final Cut Pro requires a product serial number. For information about volume licenses, go to http://www.apple.com/finalcutstudio.

2 Make sure all of the source media files and render files for your Final Cut Pro project are

on a hard disk that can be shared (mounted) by all the computers in the cluster that will process the job. (The following instructions use an example of a shared hard disk named Media1.)

If you copy all the media files to another hard disk for this purpose, you may need to reconnect the media files in Final Cut Pro before proceeding.

Important: The hard disk you use to store the media files may not be a startup disk for

any computer in the cluster.

3 In Final Cut Pro, set the scratch disk to the same hard disk used for storing media files

in step 2:

a Choose Final Cut Pro > System Settings, then click the Scratch Disks tab.

b Click Set.

c In the dialog that appears, locate and select the disk you want to use (Media1

in this example).

d Click Choose. The specified disk (Media1) is listed next to the Set button, along with

the amount of available disk space.

For more information about Final Cut Pro scratch disks, see the Final Cut Pro User Manual.

4 Enable file sharing on the computer where the scratch disk (Media1) is located (in the

Sharing pane of System Preferences, click Services and then select Personal File Sharing).

5 On each computer in the intended cluster, mount the scratch disk you specified in

step 3 (in the Finder sidebar, click Network, navigate to the computer that contains the Media1 disk, click Connect, and select Media1).

6 In Compressor Preferences, specify cluster options settings:

a Choose Compressor > Preferences, or press Command-comma ( , ). The Preferences

window appears.

b Choose “Copy Source to Cluster as Needed” from the Cluster Options pop-up menu.

7 In Final Cut Pro, choose File > Export > Using Compressor.

Compressor opens with the selected media file (the Final Cut Pro sequence) in the Batch window.

8 In Compressor, double-click the selected file and play it in the Preview window to verify

the integrity of the clip.

9 In the Batch window, assign settings and destinations to the selected file as necessary.

78 Appendix B Troubleshooting

Page 79

10 Choose the intended cluster from the Cluster pop-up menu in the lower-left corner of

the Batch window.

11 Click Submit.

To export a Final Cut Pro sequence as a QuickTime reference movie and submit it to Compressor for distributed processing:

1 Make sure all of the source media files and render files for your Final Cut Pro project are

on a hard disk that can be shared (mounted) by all the computers in the cluster that will process the job. (The following instructions use an example of a shared hard disk named Media1.)

If you copy all the media files to another hard disk for this purpose, you may need to reconnect the media files in Final Cut Pro before proceeding.

Important: The hard disk you use to store the media files may not be a startup disk for

any computer in the cluster.

2 In Final Cut Pro, choose File > Export > QuickTime Movie.

3 In the Save dialog, make sure that the Make Movie Self-Contained checkbox is

not selected.

4 Save the QuickTime reference movie to the same hard disk (Media1) used to store

media files in step 1.

5 Enable file sharing on the computer where the scratch disk (Media1) is located (in the

Sharing pane of System Preferences, click Services and then select Personal File Sharing).

6 On each computer in the intended cluster, mount the hard disk (Media1) where the

media files and QuickTime reference movie are located (in the Finder sidebar, click Network, navigate to the computer that contains the Media1 disk, click Connect, and select Media1).

7 In Compressor Preferences, specify cluster options settings:

a Choose Compressor > Preferences, or press Command-comma ( , ). The Preferences

window appears.

b Choose “Never Copy Source to Cluster” from the Cluster Options pop-up menu.

8 Import the QuickTime reference movie into Compressor. (Drag it from the desktop to

the Compressor Batch window.)

9 Double-click the file in the Batch window and play it in the Preview window to verify

the integrity of the clip.

10 In the Batch window, assign settings and destinations to the selected file as necessary.

11 Choose the intended cluster from the Cluster pop-up menu in the lower-left corner of

the Batch window.

12 Click Submit.

Appendix B Troubleshooting 79

Page 80

QuickTime Reference Movies

Cluster Settings for Extended Transcoding Sessions

If you are using Compressor 2 or later for distributed processing, and you anticipate a transcoding session that will last up to seven days or longer, you must make an adjustment in the Advanced section of the Apple Qmaster Preferences Pane. By default, temporary process files may remain on a cluster’s scratch location for seven days before they are automatically deleted. You can increase this value (the number of days) in the Apple Qmaster pane in System Preferences.

Cleaning Up Cluster Storage

Using Apple Qmaster 2 with an NFS Server

By default, Apple Qmaster uses /etc/exports to define its Cluster Storage export. This can cause a conflict if you defined an NFS export in your local Netinfo database. When you enable a controller using Apple Qmaster 2 or later, Apple Qmaster will use /etc/exports, not entries defined in your Netinfo database. To work around this issue, either move the exports to /etc/exports, or move the controller to a computer that doesn’t export anything.

Apple Qmaster Distributed Processing and Xsan

Here are some tips on using the Apple Qmaster distributed processing system with Xsan.

Restarting Apple Qmaster and Xsan

Using previous versions of Apple Qmaster distributed processing on an Xsan may cause mounting problems when restarting an Apple Qmaster distributed processing cluster controller.

80 Appendix B Troubleshooting

Page 81

Restart Apple Qmaster and Xsan computers in the following order:

1 Turn off the cluster controller by clicking Stop Sharing in the Apple Qmaster pane in

System Preferences.

2 Restart the Apple Qmaster cluster controller computer.

3 Wait for the Xsan volume to mount on the desktop.

4 Click Start Sharing in the Apple Qmaster pane in System Preferences to restart

the controller.

Xsan Compatibility

Xsan 1.3 is not compatible with Compressor’s Apple Qmaster distributed processing system when Xsan media drives are used for cluster (scratch) storage. To use Xsan with Apple Qmaster 2.3, update to Xsan 1.4.

To download and install Xsan 1.4, go to:

Â http://www.apple.com/support/xsan

Compressor Command-Line Usage Requires Login

Appendix B Troubleshooting 81

Page 82

Page 83

Index

activity logs 66 administrator email address 60 administrator mode, Batch Monitor 40 administrator password

Apple Qadministrator 35 cluster administrator 51, 61

Apple Qadministrator

creating clusters 50–52 deleting clusters 58, 59 failure notification 66 modifying clusters 58–59 monitoring cluster activity 59 notification emails to 66 opening 36 overview 35–36 passwords 35 QuickClusters 58 setting cluster preferences 60 setting passwords 61–62

Apple Qmaster

alternatives to 68 failure notification 66 installing 28 interfaces 31–40 introduction to 5–11 opening 38–39 preferences. See Apple Qmaster pane in System Preferences recovery features 65–66 setting scratch location 62, 64 specifying Shake script locations 30 submitting batches for processing 7 turning cluster controller on/off 48 turning off temporarily 58 turning processing services on/off 43–48 using shell commands 71–73

Apple Qmaster networks 21–30

basic rules for 21 described 21 desktop processing environment 27–30 examples of 22, 26 expanded 26

expanding capacity of 25–26 simple 22

subnet 7 Apple Qmaster pane 68 Apple Qmaster pane in System Preferences 17, 33– 34

advanced settings 54–57

opening 17, 34

Services pane 48

settings overview 45 Apple websites 12

batches