Apple QMASTER 3 User Manual

Apple Qmaster 3

User Manual

Apple Inc.

Your rights to the software are governed by the accompanying software license agreement. The owner or authorized user of a valid copy of Final Cut Studio software may reproduce this publication for the purpose of learning to use such software. No part of this publication may be reproduced or transmitted for commercial purposes, such as selling copies of this publication or for providing paid for support services.

The Apple logo is a trademark of Apple Inc., registered in the U.S. and other countries. Use of the “keyboard” Apple logo (Shift-Option-K) for commercial purposes without the prior written consent of Apple may constitute trademark infringement and unfair competition in violation of federal and state laws.

Every effort has been made to ensure that the information in this manual is accurate. Apple is not responsible for printing or clerical errors.

Note:

Because Apple frequently releases new versions and updates to its system software, applications, and Internet sites, images shown in this book may be slightly different from what you see on your screen.

Apple Inc. 1 Infinite Loop Cupertino, CA 95014–2084 408-996-1010 www.apple.com

Apple, the Apple logo, Final Cut, Final Cut Pro, QuickTime, and Shake are trademarks of Apple Inc., registered in the U.S. and other countries.

UNIX is a registered trademark of The Open Group.

Other company and product names mentioned herein are trademarks of their respective companies. Mention of third-party products is for informational purposes only and constitutes neither an endorsement nor a recommendation. Apple assumes no responsibility with regard to the performance or use of these products.

Using Apple Qmaster 3

The Apple Qmaster system provides automated work distribution and processing for high-volume projects created with Shake and other digital visual effects software packages.

This document describes the Apple Qmaster application, one of several applications in the Apple Qmaster distributed processing system. The complete system includes controls for configuring clusters of processing computers and to monitor, pause, resume, or cancel work that has been submitted.

The Apple Qmaster application described in this document is the “client interface” that you use to submit jobs for distributed processing. Apple Qmaster can accept jobs from Shake, Autodesk Maya, and any UNIX command-line program. For information on how to create a complete network for Apple Qmaster processing, and on how the Apple Qmaster application works with the rest of the Apple Qmaster system, see the

Distributed Processing Setup

guide.

This document describes:

The Apple Qmaster Distributed Processing System (p. 4)

The Apple Qmaster Interface (p. 6)

Submitting Batches for Apple Qmaster Processing (p. 11)

Creating an Extended Node Cluster (Using Nodes Without Apple Qmaster Installed) (p. 23)

Setting Environment Variables in Apple Qmaster (p. 26)

Command-Line Usage in Apple Qmaster (p. 28)

The Apple Qmaster Distributed Processing System

Computers that submit batches to Apple Qmaster are called

job

is a processing task in the form of a Shake file, or other file or commands, that uses UNIX commands to specify settings such as rendering instructions and file locations and destinations.

Network

clients.

Apple Qmaster

Batch of

processing jobs

Client computer

Jobs are submitted.

Apple Qmaster cluster

Processing is

performed by cluster.

Processed

files

Destination folder

Files are placed at

specified destination

Although a batch can include just one job, you will typically want to submit several jobs at once for processing. Similarly, several people can use the same Apple Qmaster system at the same time, with several client computers sending batches in the same time frame. Batches are managed and distributed by the computer that is designated as the Apple Qmaster

Distributed Processing Setup

cluster controller.

guide.

For more information, see the

Clients

Batches are submitted for processing from the be any computer that has Apple Qmaster installed and is on the same network as the cluster controller. Multiple client computers can be on the same network, using the same cluster to do the processing for various applications. See “Submitting Batches

with The Apple Qmaster Application” on page 11 for details.

client computers.

A client computer can

Clusters

When a client sends batches to the Apple Qmaster system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster– configured computers called a Apple Qadministrator application to create one or more clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection. See

Distributed Processing Setup

“Preparing a Network for Distributed Processing.”

Note:

See the Shake Support web site (http://www.apple.com/support/shake) for an online guide to setting up a full-time render farm, incorporating an Xserve computer and cluster nodes.

cluster.

To set up Apple Qmaster services, you use the

for more information on cluster setup. In particular, see

Service Nodes

When you combine multiple nodes into a cluster, they function as one very powerful computer because all their resources are shared. You make a computer available as a service node by configuring it in the Apple Qmaster pane in System Preferences. The steps involved in using System Preferences to configure a service node are described in “Creating and Administering Clusters” in the

Distributed Processing Setup

guide.

Cluster Controllers

The cluster controller software acts as the manager of a cluster. The cluster controller directs the distribution of batches within the cluster. It has the ability to determine the best use of the cluster resources based on work and availability variables. (See the

Distributed Processing Setup

cluster controller by turning on the cluster controlling service in the Apple Qmaster pane in System Preferences. (See “Configuring Service Nodes and Cluster Controllers,” in the

Distributed Processing Setup

guide for more details.) You make a computer available as a

manual.)

The Apple Qmaster Interface

The Apple Qmaster application described in this document is one part of the larger distributed processing system described above. The Apple Qmaster application is the client software you use to submit jobs and batches to the system. Both this application and the larger system are known by the same name (Apple Qmaster).

The Apple Qmaster window contains a toolbar; a Batch Name field; a Submit To field; the Batch table; buttons to add, remove, and submit jobs (dimmed until at least one job is in the Batch table); and pop-up menus to choose job types and batch priority.

Toolbar Buttons

The Apple Qmaster toolbar contains the following buttons:

History:

submitted from your computer, check the current status of all processing batches, and resubmit any batches listed in the log.

Set Environment:

preflight scripts.

Notification:

the status of a Batch completion or failure can be sent. Apple Qmaster does not currently support SMTP servers that require authentication. The following table lists the service labels used in the email notifications and logs:

Opens the History drawer, where you can view a full log of all batches

Opens a sheet where you can add environment variables and

Opens a sheet where you can set (or reset) an email address to which

Processing service type Notification label

Local Compressor service servicecontroller:com.apple.stomp.transcoder

Distributed Compressor service servicecontroller:com.apple.stomp.transcoderx

Distributed Apple Qmaster service

Batch Monitor:

Opens the Batch Monitor, which allows you to view the status of all

servicecontroller:com.apple.qmaster.executor

batches being processed.

Apple Qmaster Text Fields and Pop-up Menus

The Apple Qmaster window has other important interface elements for creating and submitting jobs and batches.

Batch name:

This is where you can enter the name of a particular batch. (This is the

name that appears in the Batch Monitor.)

Submit To:

Use this pop-up menu to choose an available cluster to process the

current batch.

Batch table:

This is the middle area of the Apple Qmaster window which lists the

individual jobs in a batch. It includes the following columns:

Type:

One of four job types: Shake, Maya, Generic Render, Shell (UNIX).

Command:

Working Directory:

The actual command for the job.

If relevant, the directory from which you want the command to

be executed.

Command menu:

Use this pop-up menu to create commands. You can choose one of

the following types of jobs:

Generic Render Command

Maya Command

Shake Command

Shell Command

Use the Add ( + ) button to enter a job (with the selected job type) into the Batch table. Use the Remove ( – ) button to delete the selected job from the Batch table.

Use the Duplicate button to duplicate the selected job in the Batch table.

Duplicate button

Serialize Jobs:

Select this checkbox to execute jobs in exactly the order in which they appear in the batch list. For example, if you have a Shake script that requires the output of a Maya script, you can ensure that the Shake script does not execute until the Maya script is done.

Priority:

Use this pop-up menu to set the relative priority (urgency) for a batch. This pop-up menu allows you set the priority level (High, Medium, or Low) of a batch, relative to other batches. It defaults to Medium. High-priority batches get processed before medium-priority and low-priority batches.

Delay:

Use these text fields to set a delay in hours or minutes for any job in the batch. Click on either text field (left field = hours, right field = minutes) to enter the corresponding value. Click the stepper to step through minutes in increments of 5 at a time, or, if the Hour text field is selected, in increments of 1 hour at a time.

Submit:

Click the Submit button to process the batch using the parameters set in the

Apple Qmaster window.

Installing Apple Qmaster, Shake, and Other Applications

Each rendering application has a slightly different installation procedure. See the

Distributed Processing Setup

of applications.

To use Apple Qmaster with Shake, or other frame-based rendering applications, you must follow these installation and licensing guidelines:

To submit batches for Shake, a copy of Shake must be installed on every node in the cluster. For command-line rendering on Mac OS X, a license is not needed.

To submit batches for Maya, install and license a copy of Maya on every node in the cluster.

To submit batches for other frame-based rendering applications, install a copy of the rendering application on every node in the cluster. For example, you must install a copy of After Effects 6.5 on every node in the cluster. It’s important that you install aerender, the command-line renderer for After Effects. After Effects requires a product key and serial number before installation.

Note:

In all three of the above cases, you will need to install any third-party plug-ins,

fonts, and so on, that your projects require on each node in the cluster.

guide for information on installing the Apple Qmaster suite

Setting Up Rendering Services and Shared Storage

Here are some additional tips on setting up an Apple Qmaster “render farm”:

Changing the Number of Rendering Service Instances

By default, Apple Qmaster enables one Rendering service per CPU. On a dual-processor system, you’ll have two rendering services enabled by default. So that means out of the box, two copies of Shake will be running at the same time on a dual-processor system. It may be necessary to disable a rendering service. See the guide for information on changing the number of rendering services.

Note:

If you are using Shake, you can always change the number of CPUs Shake uses at

submission time by using

-cpus x

where x is the number of threads to use.

Distributed Processing Setup

Shared Volumes

All the nodes in the cluster must have a common volume to work from; otherwise they will not know where to find assets needed for any given batch. There are many ways to set up file sharing. The following is a quick tutorial on NFS export and mounting. NFS is not required, as you can also use AFP or SMB. If you already have an NFS server, such as your cluster controller, simply add another export in

-HUP (-1)

mountd

Enter the following if you are exporting from your cluster controller or an existing NFS server:

sudo echo “<SOME DIRECTORY>” >> /etc/exports sudo kill -1 `cat /var/run/

mountd.pid`

/etc/exports

or netinfo and send

This creates an NFS server exporting < following command:

showmount -e

SOME DIRECTORY

>. You can verify this by the

Dedicated NFS Servers

You can set up a dedicated NFS server.

If you are setting up a dedicated NFS server, enter the following:

sudo echo “<SOME DIRECTORY>” >> /etc/exports sudo rm /var/run/NFS.StartupItem sudo SystemStarter start NFS

The computer does not need to run OS X server; if you prefer to use a graphical user interface, you can use Mac OS X Server Admin tools and Netinfo Manager.

To mount the export, enter the following on all the nodes in the cluster:

mkdir -p <MOUNT POINT> mount_nfs <nfs server>:<SOME DIRECTORY> <MOUNT POINT>

If your NFS server is a node, it is important that you also mount the export, even though you are sharing it. All the paths on each node must appear to be the same. If you don’t mount it, this computer will produce File Not Found errors. Also, if you have any client computers that will submit to the cluster but not actually participate in it, you need to mount the export.

Shared Volume Media Management with Shake

It maybe necessary to relink your assets once you move your project to the shared volume. This varies depending on the types of files, plug-ins, fonts, and environmental requirements. Something that commonly needs to be changed is the FileIn and FileOut paths, and env variables such as NR_INCLUDE_PATH and NR_FONT_PATH. UNC should be disabled, as these paths are typically not resolvable to nodes in the cluster when arbitrary host names are used.

To disable UNC and enable Apple Qmaster from within the Shake application, do the following:

1 Go to ~/nreal/include/startup

2 Create a file called qmaster.h

3 Add the following:

script.uncFileNames = 0;

sys.useRenderQueue = "Qmaster";

4 Press Return several times after the last line.

5 Save your work.

Submitting Batches for Apple Qmaster Processing

Jobs are submitted in the form of a batch, and a batch can include one or more jobs. Batches can be submitted from any computer that:

Â Has Apple Qmaster Applications installed

Â Is on the same network as the cluster to which you are submitting the batch

Note: Certain bundled versions of Compressor 2 can also submit batches to the Apple Qmaster distributed processing system. See the Distributed Processing Setup guide or the Compressor 2 User Manual for more information.

Submitting Batches with The Apple Qmaster Application

Apple Qmaster includes a plug-in for Shake, which automates and customizes the process by which Shake batches are submitted with Apple Qmaster. Because of this, the process for submitting Shake batches is more automated than is the process for submitting batches for other applications that don’t have such plug-ins. A plug-in is included for Autodesk Maya, as well as a generic Render Command plug-in for other frame-based rendering applications, and a Shell Command plug-in that works with any application that can be executed from the UNIX command line.

+ 23 hidden pages

Apple QMASTER 3 User Manual

Specifications and Main Features

Frequently Asked Questions

User Manual

Clients

Clusters

Service Nodes

Cluster Controllers

Toolbar Buttons

Setting Up Rendering Services and Shared Storage

Changing the Number of Rendering Service Instances

Shared Volumes

Dedicated NFS Servers

Shared Volume Media Management with Shake