Apple QMASTER 2 Distributed Processing Setup Manual

Apple Qmaster 2 and Compressor 2

Distributed Processing Setup

 Apple Computer, Inc.

Under the copyright laws, this manual may not be copied, in whole or in part, without the written consent of Apple. Your rights to the software are governed by the accompanying software license agreement.

The Apple logo is a trademark of Apple Computer, Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Option-Shift-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.

Every effort has been made to ensure that the information in this manual is accurate. Apple Computer, Inc., is not responsible for printing or clerical errors.

Apple Computer, Inc. 1 Infinite Loop Cupertino, CA 95014-2084 408-996-1010 www.apple.com

Apple, the Apple logo, AppleScript, Apple Store, DVD Studio Pro, Final Cut, Final Cut Pro, FireWire, Mac, Mac OS, QuickTime, Shake, and Xserve are trademarks of Apple Computer, Inc., registered in the U.S. and other countries.

Finder is a trademark of Apple Computer, Inc. AppleCare is a service mark of Apple Computer, Inc., registered in the U.S. and other countries.

Dolby is a registered trademark of Dolby Laboratories.

Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.

Preface 5 Introduction to Distributed Processing

Using Distributed Processing to Increase Speed and Efficiency

Basic Components of the Apple Qmaster Distributed Processing System

Clients

Clusters

QuickClusters

How the Apple Qmaster System Distributes Batches

About the Distributed Processing Setup Guide

Apple Websites

12 12 12

Compressor and Shake Websites AppleCare Service and Support Website Other Apple Websites

Chapter 1 15 Getting Started Quickly

Quick and Easy Distributed Processing

Chapter 2 21 Preparing a Network for Distributed Processing

The Minimum You Need to Know

22 22 23 24 25 26 27

Example of a Minimal Distributed Processing Network Using One Computer to Serve Two Distributed Processing Roles

Other Possible Components of a Distributed Processing Network

Example of an Expanded Distributed Processing Network

Sample Setup for Part-Time Processing on Desktop Computers

Configuring Access for Part-Time Distributed Processing Additional Steps For Part-Time Distributed Processing With Shake

Chapter 3 29 The Interfaces

The Interfaces in the Apple Qmaster Distributed Processing System

Apple Qmaster Pane in System Preferences

Apple Qadministrator

Client Interfaces: Compressor and Apple Qmaster

36 36 37

Using Compressor Using Apple Qmaster

Batch Monitor

Chapter 4 39 Creating and Administering Clusters

An Overview of Configuring a Cluster

Configuring Service Nodes and Cluster Controllers

41 42 43 45 46 48 50 50

51 54 55 56 57 57 58 59 60 60

Configuring Service Node Processing Options in the Apple Qmaster Pane in System Preferences Managed Vs. Unmanaged Services Turning Cluster Controller Services On or Off

Making the Most of Multi-processor Computers Creating Clusters With Apple Qadministrator About QuickClusters

Creating QuickClusters Advanced Settings in the Apple Qmaster Preferences Pane Modifying and Deleting Clusters With Apple Qadministrator Monitoring Cluster Activity Setting Cluster Preferences Setting Passwords and Scratch Storage

Setting Cluster Administrator and User Passwords

Setting a Service Password for Including a Computer in a Cluster

Cluster Storage: Setting a Scratch Storage Location Recovery and Failure Notification Features

Recovery Features

Failure Notification

Appendix 63 Command-Line Usage

63 64 64 66 68

Installing Apple Qmaster from the Command Line

Shell Commands for Configuring Service Nodes and Cluster Controllers

Shell Commands for Submitting Compressor Jobs

Shell Commands for Submitting Apple Qmaster Jobs

Shell Commands for Monitoring Batches

Index 71

Contents

Introduction to Distributed Processing

Rendering a series of large files on one desktop computer is processing intensive and time consuming. You can increase speed and productivity by distributing processing across multiple computers.

High-volume processing is sometimes addressed by carefully managing multiple computers; technicians set up batches of processing tasks for each computer, and then monitor their progress, collect and route the processed files, and start over again with new batches. While this is an improvement over the single-computer method, the resource and process management can be laborious and slow.

The Apple Qmaster distributed processing engine provides a more efficient solution, handling all the work distribution and processing for you, behind the scenes. Apple Qmaster and the Apple Qmaster features of Compressor 2 manage the processing across designated computers. They subdivide the work for speed, route the work to the computers with the most available computing power, and direct the processing.

Preface

Using Distributed Processing to Increase Speed and Efficiency

Distributed processing

computers that have been chosen to provide more processing power. You can submit batches of processing jobs to Apple Qmaster, which allocates those jobs to other computers in the most efficient way (described in more detail in “How the

Apple Qmaster System Distributes Batches” on page 9).

Computers that submit batches to the Apple Qmaster distributed processing system are called

clients.

a Shake file, or other file or commands, that use UNIX commands to specify settings such as rendering instructions and file locations and destinations.

accelerates processing by distributing the work to multiple

job

is a processing task such as a Compressor preset-source pair, or

Batch of

processing jobs

Client computer

Jobs are submitted.

batch

is one or more jobs submitted for processing at one time. The procedure is

Apple Qmaster cluster

Processing is

performed by cluster.

Processing

files

Destination folder

Files are placed at

specified destination.

analogous to printing multi-page documents from a word processing program; the files are spooled and processed in the background. Although a batch can include just one job, you will typically want to submit several jobs at once for processing. Similarly, several people can use the same Apple Qmaster system at the same time, with several client computers sending batches in the same time frame. Batches are managed and distributed by the computer that is designated as the Apple Qmaster

cluster controller,

which is described in the next section.

6 Preface

Introduction to Distributed Processing

Basic Components of the Apple Qmaster Distributed Processing System

While the Apple Qmaster software includes a few different applications (see Chapter 3, “The Interfaces,” on page 29), as a whole it is part of a networked system that includes the following basic components:

Client(s):

submit jobs for distributed processing. Applications that can use Apple Qmaster services for processing include Compressor 2, Shake, Alias Maya, and many UNIX command-line programs.

Apple Qmaster cluster,

The computer or computers that use Compressor or Apple Qmaster to

which contains:

Service nodes:

The computers that perform the processing of batches submitted

via Compressor or Apple Qmaster. A batch can include one or more jobs.

Cluster controller:

The software, enabled on a computer by means of the Apple Qmaster pane in System Preferences, that divides up batches, determines which service nodes to send work to, and generally tracks and directs the processes.

Client computer

from which users work and

send jobs to the cluster

(can contain multiple service nodes, but only one cluster controller)

Cluster controller

manages the

distribution of

client’s jobs

across the cluster

Cluster

Service node

processes jobs

Service node

processes jobs

Service node

processes jobs

Service node

processes jobs

The client computer, the service nodes, and the cluster controller are often on separate (but network-connected) computers, for the most rapid processing potential. However, the cluster controller could be on a client computer or a service node. See “Using One

Computer to Serve Two Distributed Processing Roles” on page 22 for more information

on this scenario.

Following is a closer look at the part each component plays in the Apple Qmaster system.

Preface

Introduction to Distributed Processing

Clients

Batches are submitted for distributed processing from the computer can be any computer that has Compressor 2 or Apple Qmaster installed and is on the same network (subnet) as the cluster controller. Multiple client computers can be on the same subnet, using the same cluster to do the processing for various applications.

You use Compressor 2 or the Apple Qmaster application to submit batches to be processed for a client. See the

User Manual

for details on using these applications.

Compressor 2 User Manual

client computers.

and the

Apple Qmaster 2

A client

Clusters

When a client sends batches to the Apple Qmaster distributed processing system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster–configured computers called a clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection.

Example of a cluster

cluster.

You can create one or more

Cluster

controller

Note:

are described in Chapter 2, “Preparing a Network for Distributed Processing.”

8 Preface

Service

node

Service

node

Service

node

This illustration provides only one simple example of a cluster. Other possibilities

Introduction to Distributed Processing

Service Nodes

The service nodes are where the processing work is done. When you assign a group of service nodes to a cluster, they function as one very powerful computer because all their resources are shared. If one service node is overloaded or otherwise inaccessible, another service node is used.

You make a computer available as a service node by configuring it in the Apple Qmaster pane in System Preferences. The simple steps involved in using System Preferences to configure a service node are described in Chapter 4, “Creating and

Administering Clusters,” on page 39.

Note:

The terms document. The term Shake, and other frame-based rendering applications) and encoding (or transcoding or compression) for Compressor. For more information see the the

Compressor 2 User Manual.

Cluster Controllers

The cluster controller software acts as the manager of a cluster. The cluster controller directs the distribution of batches within the cluster. It has the ability to determine the best use of the cluster resources based on work and availability variables. (See “How

the Apple Qmaster System Distributes Batches,” next, for more details.) The cluster

controller is responsible for accepting batch submissions, maintaining and managing the batch queue, and doling out the work to the appropriate service node. It also tracks the status of all outstanding batches.

processing

and

rendering

is used here generically to cover both rendering (for

will come up frequently as you read this

Shake 4 User Manual

and

You make a computer available as a cluster controller by turning on the clustercontrolling service in the Apple Qmaster pane in System Preferences.

QuickClusters

The QuickCluster feature of the Apple Qmaster distributed processing system is a simple and automated alternative to creating and configuring clusters manually. For more information about QuickClusters, see “Creating QuickClusters” on page 50.

How the Apple Qmaster System Distributes Batches

The Apple Qmaster cluster controller determines the most efficient use of the cluster resources. It makes this determination based on the availability of each service node and the number of separable parts (described next) of the batch.

Because Apple Qmaster subdivides individual batches across different service nodes, the work is shared and completed more quickly. And, because this method uses all the service nodes as much as possible, you avoid under-utilizing your resources.

Preface

Introduction to Distributed Processing

Batches can be distributed to a cluster by the cluster controller in one or both of the following ways. (Apple Qmaster determines which way is the most efficient for specific batches, depending on the circumstances.)

The batch is subdivided into data segments:

cluster controller could divide the frames into groups (

For example, for a render batch, the

segments

). Each segment

would be processed in parallel on the service nodes in the cluster.

The batch is subdivided into tasks:

For example, for a render batch, the cluster controller could subdivide the rendering work into different processing tasks. Different tasks would be run on different service nodes.

Rather than actually moving segments, Apple Qmaster tells the service nodes which segments to read via the network, where to find them, and what to do with them. Below is an example of how one batch could be processed in an Apple Qmaster system.

Batch submitted

by Compressor or

Apple Qmaster

Job request for frames 1–30

Cluster controller

divides and distributes

job to available

service nodes

Instructions specifying

locations of source files

and frames 01–10

In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate services in a cluster on the same IP subnet and to dynamically share and receive information. Because the computers can continually transmit their current processing availability status, Apple Qmaster can distribute (load-balance) the workload evenly across the cluster.

10 Preface

Instructions specifying locations of source files

and frames 11–20

Service node 1

Introduction to Distributed Processing

Service node 2

Processed files

placed in specified

destination

Instructions specifying

locations of source files

and frames 21–30

Service node 3

About the Distributed Processing Setup Guide

The background information that introduces most of the basic concepts and terms related to using the Apple Qmaster distributed processing system is contained in this preface: “Introduction to Distributed Processing.”

Â Chapter 1, “Getting Started Quickly,” on page 15, gets you going with a few very

simple steps.

Â Chapter 2, “Preparing a Network for Distributed Processing,”on page 21, will help you

make sure that the network part of your Apple Qmaster system is ready.

Â Chapter 3, “The Interfaces,” on page 29, introduces each of the applications and

utilities included with the Apple Qmaster distributed processing software, and describes when you need to use each of them.

Â Chapter 4, “Creating and Administering Clusters,” on page 39, tells you how to

configure and administer the clusters, and how to submit and monitor the batches that you want to have processed by the clusters.

Â The Appendix, “Command-Line Usage,” on page 63, explains how you can use the

distributed processing system entirely from the command line.

While all of the chapters should be useful to administrators, client users may only need to refer to the Compressor 2 User Manual or the Apple Qmaster 2 User Manual, and possibly to the introductory information in this preface, to understand how to use the Apple Qmaster distributed processing system for their purposes.

To access the Distributed Processing Setup guide:

m Choose Help > Distributed Processing Setup.

You can also choose either Help > Apple Qmaster User Manual or Help > Compressor User Manual.

Each user manual contains a homepage that provides quick access to various features, including Late-Breaking News.

Additionally, a comprehensive bookmark list allows you to quickly choose what you want to see and takes you there as soon as you click the link.

In addition to these navigational tools, the Help documentation gives you other means to locate information quickly:

Â All cross-references in the text are linked. You can click any cross-reference and jump

immediately to that location. Then, you can use the navigation Back button to return to where you were before you clicked the cross-reference.

Â The table of contents and index are also linked. If you click an entry in either of these

sections, you jump directly to that section of Help.

Â You can also use the Find dialog to search the index or text for specific words

or phrases.

Preface

Introduction to Distributed Processing

Apple Websites

There are a variety of Apple websites that you can visit to find additional information.

Apple Qmaster, Compressor, and Shake Websites

To access the Apple Qmaster support page, go to:

http://www.apple.com/support/appleqmaster

To access the Compressor website, go to:

http://www.apple.com/finalcutpro/compressor.html

To access the Compressor support page, go to:

http://www.apple.com/support/compressor

To access the Compressor discussion page, go to:

http://discussions.info.apple.com/compressor

To access the Shake website, go to:

http://www.apple.com/shake

To access the Shake support page, go to:

http://www.apple.com/support/shake

AppleCare Service and Support Website

For software updates and answers to the most frequently asked questions for all Apple products, go to:

http://www.apple.com/support

You’ll also have access to product specifications, reference documentation, and Apple and third-party product technical articles.

Other Apple Websites

Start at the Apple homepage to find the latest information about Apple products:

http://www.apple.com

QuickTime is industry-standard technology for handling video, sound, animation, graphics, text, music, and 360-degree virtual reality (VR) scenes. QuickTime provides a high level of performance, compatibility, and quality for delivering digital video. Go to the QuickTime website for information on the types of media supported, a tour of the QuickTime interface, and specifications:

http://www.apple.com/quicktime

12 Preface

Introduction to Distributed Processing

For information about seminars, events, and third-party tools used in web publishing, design and print, music and audio, desktop movies, digital imaging, and the media arts go to:

Â http://www.apple.com/pro

For resources, stories, and information about projects developed by users in education using Apple software, including Compressor, go to:

Â http://www.apple.com/education

Go to the Apple Store to buy software, hardware, and accessories direct from Apple. You can also find special promotions and deals that include third-party hardware and software products:

Â http://www.apple.com/store

Preface Introduction to Distributed Processing 13

1 Getting Started Quickly

The Apple Qmaster distributed processing system has default settings that allow you to use distributed processing immediately.

Whether you are using the distributed processing system for Compressor or Apple Qmaster, you can get up and running quickly with the following steps.

Note: The Compressor 2 distributed processing feature is limited to computers that have either Final Cut Studio or DVD Studio Pro 4 installed.

Quick and Easy Distributed Processing

The steps below describe the simplest and quickest way to start using distributed processing.

Step 1: Install the software

Install the appropriate software on the computers you want to include in your distributed processing network. Each computer in the network will require Apple Qmaster 2 and/or Compressor 2 software.

To install the software:

1 Make sure the client software is on at least one computer in your network.

In order to submit jobs and batches to the distributed processing system, you will need to run client (submission) software (either Compressor or Apple Qmaster). If you are reading this, you have probably already installed one or both of these. For further information, see the installation booklet that came with either of these applications.

2 Install Apple Qmaster software on each computer you want to use for distributed

processing. (All computers must be on the same subnet.) In most standard distributed processing networks, you will need to install

Apple Qmaster software on every computer in the network.

a Locate the folder containing the Apple Qmaster software:

Â If you are using Compressor, insert the DVD Studio Pro 4 or the Final Cut Studio

installation disc and locate the “AppleQmasterNode.mpkg” installer package.

Â If you are using Shake, insert the Shake 4 installation disc and locate the

“AppleQmasterNode.mpkg” installer package.

b Do one of the following:

Â Double-click the “AppleQmasterNode.mpkg” installer package, and follow the

onscreen instructions. By default, this installer will install all the necessary Apple Qmaster software. You may choose to customize the process by removing either of the following from the installation: Apple Qmaster Applications (which installs the Apple Qmaster application, Apple Qadministrator, and Batch Monitor) or Apple Qmaster Services (which installs the Apple Qmaster pane in System Preferences, necessary to create QuickClusters and/or service nodes).

Â Install Apple Qmaster software from the command line. See “Installing

Apple Qmaster from the Command Line” on page 63 for more information.

Note: In order to use the Apple Qmaster distributed processing system to process Dolby Digital Professional audio, each node (computer) in your distributed processing network must have either Final Cut Studio or DVD Studio Pro 4 installed.

If you are unsure which role each computer will play in your network, just install the Apple Qmaster software on each computer in your network. You can sort out the details later. See “Basic Components of the Apple Qmaster Distributed Processing

System” on page 7 for additional information on what roles individual computers can

play in the distributed processing system.

Note: Shake users can create distributed processing clusters containing computers that do not have any Apple Qmaster software installed. See the Apple Qmaster 2 User Manual for more information.

16 Chapter 1 Getting Started Quickly

Step 2: Configure a QuickCluster

Use the Apple Qmaster pane in System Preferences to configure the cluster controller and service node computers.

1 Open System Preferences. 2 Click the Apple Qmaster button, located in the Other section.

The Apple Qmaster pane appears.

3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter

the administrator name and password.

4 Click Start Sharing.

This creates a QuickCluster with this computer as its controller, and an instance of processing services for each processor on the computer.

Chapter 1 Getting Started Quickly 17

Step 3: Add service nodes to the cluster

On each computer that you would like to make a service node on your cluster, do the following:

1 Open System Preferences. 2 Click the Apple Qmaster button, located in the Other section.

The Apple Qmaster pane appears.

3 If the pane is locked, unlock it by clicking the padlock in the lower-left corner to enter

the administrator name and password.

4 Click the Services Only radio button.

5 Click Start Sharing.

This creates a service node that will automatically process jobs submitted to the QuickCluster you set up in “Configure a QuickCluster” on page 17.

Step 4: Create a batch

In the Compressor Batch window, or in the Apple Qmaster window, create a batch with one or more jobs. See the Compressor 2 User Manual or the Apple Qmaster 2 User Manual for details.

18 Chapter 1 Getting Started Quickly

Step 5: Submit the batch

1 Depending on whether you are using Compressor or Apple Qmaster, do one of

the following: Â In the Compressor Batch window, click the Cluster pop-up menu and choose the

cluster you created in Step 2.

Â In the Apple Qmaster window, click the Submit To pop-up menu and choose the

cluster you created in Step 2.

2 Click Submit.

The distributed processing system will now process the batch using the cluster you created.

For more advanced information on creating and controlling clusters and services see any of the following:

Â “Basic Components of the Apple Qmaster Distributed Processing System” on page 7 Â “Sample Setup for Part-Time Processing on Desktop Computers” on page 25 Â “The Interfaces in the Apple Qmaster Distributed Processing System” on page 30 Â “Apple Qmaster Pane in System Preferences” on page 31 Â “Apple Qadministrator” on page 33 Â “An Overview of Configuring a Cluster” on page 40 Â “Configuring Service Nodes and Cluster Controllers” on page 40 Â “Creating Clusters With Apple Qadministrator” on page 48 Â “About QuickClusters” on page 50 Â “Modifying and Deleting Clusters With Apple Qadministrator” on page 54

Chapter 1 Getting Started Quickly 19

2 Preparing a Network

for Distributed Processing

A distributed processing network can consist of as few as one or two computers, while a high-volume network may include many computers, an Xserve and Xserve cluster nodes in a rack, and high-speed networking infrastructures.

You can scale up a distributed processing system as your workload demands by adding features and devices to the network that supports it. This chapter helps you prepare your network by describing the following:

Â The Minimum You Need to Know (p. 21) Â Other Possible Components of a Distributed Processing Network (p. 23) Â Sample Setup for Part-Time Processing on Desktop Computers (p. 25)

The Minimum You Need to Know

The following are the basic rules for setting up a distributed processing network: Â A cluster must contain one (and only one) computer acting as the cluster controller,

and at least one computer acting as the service node. (These two can be the same computer, as shown next in “Example of a Minimal Distributed Processing Network.”)

Â The client computers and the computers in any cluster that supports them must be

on the same network.

Â The network must support the Apple networking technology built in to Mac OS X. Â All the computers in a cluster need Read and Write access to any computers (or

storage devices) that will be specified as output destinations for files.

Example of a Minimal Distributed Processing Network

A very small distributed processing setup could include as few as two computers: Â One computer connected to the client and configured to act as both the service

node and the cluster controller

Â One client computer

Minimum setup for distributed processing

Client

Service node with

cluster controller enabled

Though simple, this setup is useful in a small-scale environment because it allows the client computer to off-load a lot of processing work.

See “Example of an Expanded Distributed Processing Network” on page 24 for an illustration of a more powerful setup.

Using One Computer to Serve Two Distributed Processing Roles

To maximize your resources, you may want to consider using some computers for more than one distributed processing function.

Â Service node and cluster controller: In a small setup, one of the service nodes in a

cluster can also act as the cluster controller so that it performs both functions. However, in a cluster of many service nodes, the processing load required for the cluster controller could be so high that it would not be efficient to use one computer as both a service node and a cluster controller.

Â Client computer and cluster controller or service node: You could also set up a client

computer to act as a cluster controller or service node in a cluster, but again, keep in mind that the more available processing power a computer has, the faster it can manage or process jobs.

22 Chapter 2 Preparing a Network for Distributed Processing

Other Possible Components of a Distributed Processing Network

There are many ways to expand the capacity of a distributed processing network. You could include any of the following:

Â High-speed switch and cables: A 100BaseT or Gigabit Ethernet switch and compatible

cables to allow your data to move over the LAN at maximum speed.

Â Multiple clients: Multiple client computers can use the services of the same cluster.

And, you can have multiple client applications on the same client computer, using the same cluster.

Â Multiple clusters: Depending on how extensive your network is and how many clients

it needs to serve, you may want to divide up available computers and create more than one cluster to serve various clients. (Users select the cluster they want to send a batch to when they submit the batch.)

Â Multiple service nodes: In general, more service nodes means more processing power.

In deciding how many service nodes to have in a cluster, consider the ratio of data movement time to computing time. If the processing demand is greater than the network demand required to move job segments throughout the cluster, as is the case with rendering, more service nodes are a good idea. If the computing load, per job, is closer to the network load, having a smaller number of service nodes per cluster may be more efficient. If you are using the Apple Qmaster distributed processing system with applications other than Shake or Compressor, consult the application’s user manual on how to optimize the number of service nodes.

Â Storage device: A storage device, such as a remote disk or group of disk arrays, can be

used as cluster scratch storage, which is a place for short-term storage of temporary data generated by the cluster controller, clients, and service nodes. (You set the scratch storage location in the Apple Qmaster pane in System Preferences. See “Cluster Storage: Setting a Scratch Storage Location” on page 59.) Alternatively, a storage device can be used as a final destination for the files after they are processed.

Many of these items are incorporated in the example that follows.

Chapter 2 Preparing a Network for Distributed Processing 23

+ 51 hidden pages