Mathworks PARALLEL COMPUTING TOOLBOX 4 user guide

Parallel Computin

User’s Guide

g Toolbox™ 4

How to Contact The MathWorks

www.mathworks.
comp.soft-sys.matlab Newsgroup
www.mathworks.com/contact_TS.html Technical Support

[e-mail] Product enhancement suggestions

bugs@mathwo
[e-mail] Documentation error reports
[e-mail] Order status, license renewals, passcodes

info@mathwo

com

rks.com

rks.com

Web

Bug reports

Sales, prici

ng, and general information

508-647-7000 (Phone)

508-647-7001 (Fax)

The MathWorks, Inc.
3 Apple Hill Drive
Natick, MA 01760-2098

For contact information about worldwide offices, see the MathWorks Web site.

Parallel Computing Toolbox™ User’s Guide

© COPYRIGHT 2004–20 10 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement. The software may be used
or copied only under the terms of the license agreement. No part of this manual may be photocopied or
reproduced in any form without prior written consent from The MathW orks, Inc.

FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation
by, for, or through the federal government of the United States. By accepting delivery of the Program
or Documentation, the government hereby agrees that this software or documentation qualifies as
commercial computer software or commercial computer software documentation as such terms are used
or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern
theuse,modification,reproduction,release,performance,display,anddisclosureoftheProgramand
Documentation by the federal government (or other entity acquiring for or through the federal government)
and shall supersede any conflicting contractual terms or conditions. If this License fails to meet the
government’s needs or is inconsistent in any respect with federal procurement law, the government agrees
to return the Program and Docu mentation, unused, to The MathWorks, Inc.

Trademarks

MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See

www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand

names may be trademarks or registered trademarks of their respective holders.

Patents

The MathWorks products are protected by one or more U.S. patents. Please see

www.mathworks.com/patents for more information.

Revision History

November 2004 Online only New for Version 1.0 (Release 14SP1+)
March 2005 Online only Revised for Version 1.0.1 (Release 14SP2)
September 2005 Online only Revised for Version 1.0.2 (Release 14SP3)
November 2005 Online only Revised for Version 2.0 (Release 14SP3+)
March 2006 Online only Revised for Version 2.0.1 (Release 2006a)
September 2006 Online only Revised for Version 3.0 (Release 2006b)
March 2007 Online only Revised for Version 3.1 (Release 2007a)
September 2007 Online only Revised for Version 3.2 (Release 2007b)
March 2008 Online only Revised for Version 3.3 (Release 2008a)
October 2008 Online only Revised for Version 4.0 (Release 2008b)
March 2009 Online only Revised for Version 4.1 (Release 2009a)
September 2009 Online only Revised for Version 4.2 (Release 2009b)
March 2010 Online only Revised for Version 4.3 (Release 2010a)

Getting Started

1

Product Overview ................................. 1-2

Contents

Typical Use Cases

Parallel for-Loops (parfor)
Batch Jobs
Large Data Sets

Introduction to Parallel Solutions

Interactively Running a Loop in Parallel
Running a Batch Job
Running a Batch Parallel Loop
Using Distributed Arrays, spmd, and Composites

Determining Product Installation and Versions

....................................... 1-4

................................. 1-3

.......................... 1-3

................................... 1-4

................... 1-5

.............. 1-5

............................... 1-7

...................... 1-8

....... 1-10

...... 1-12

Parallel for-Loops (parfor)

2

Getting Started with parfor ........................ 2-2

Introduction
When to Use parfor
Setting up MATLAB Resources Using matlabpool
Creating a parfor-Loop
Differences Be twee n for-Loops and parfor-Loops
Reduction Assignments
Displaying Output

...................................... 2-2

................................ 2-3

....... 2-3

............................. 2-4

........ 2-5

............................ 2-6

................................. 2-7

Programming Considerations

MATLAB Path
Error Handling

.................................... 2-8

................................... 2-8

....................... 2-8

v

Limitations ...................................... 2-9

Using Objects in parfor Loops
Performance Considerations
Compatibility with Earlier Versions of MATLAB

Software

....................................... 2-13

....................... 2-11

......................... 2-12

Advanced Topics

About Programming Notes
Classification of Variables
Improving Performance

.................................. 2-14

.......................... 2-14

.......................... 2-14

............................ 2-28

Single Program Multiple Data (spmd)

3

Using spmd Constructs ............................. 3-2

Introduction
When to Use spmd
Setting Up MATLAB Resources Using matlabpool
Defining an spmd Statement
Displaying Output

Accessing Data w ith Composites

Introduction
Creating Composites in spmd Statements
Variable Persistence and Sequences of spmd
Creating Com posites Outside spmd Statements

...................................... 3-2

................................ 3-2

....... 3-3

........................ 3-4

................................. 3-6

.................... 3-7

...................................... 3-7

.............. 3-7

........... 3-9

......... 3-10

vi Contents

Distributing Arrays

Distributed Versus Codistributed Arrays
Creating Distributed Arrays
Creating Codistributed Arrays

Programming Considerations

MATLAB Path
Error Handling
Limitations

...................................... 3-15

................................ 3-12

........................ 3-12

....................... 3-13

....................... 3-15

.................................... 3-15

................................... 3-15

.............. 3-12

Interactive Parallel Computation with pmode

4

Introduction ...................................... 4-2

Getting Started with pmode

Parallel Command Window

Running pmode on a Cluster

Plotting in pmode

Limitations and Unexpected Results

Using Graphics in pmode

Troubleshooting

Connectivity Testing
Hostname Re solution
Socket Connections

................................. 4-16

................................... 4-19

............................... 4-19

.............................. 4-19

................................ 4-19

Math with Codistributed Arrays

5

........................ 4-3

......................... 4-10

....................... 4-15

................ 4-18

........................... 4-18

Array Types ....................................... 5-2

Introduction
Nondistributed Arrays
Codistributed Arrays

Working with Codistributed Arrays

How MATLAB Software Distributes Arrays
Creating a Codistributed Array
Local Arrays
Obtaining information About the Array
Changing the Dimension of Distribution
Restoring the Full Array

...................................... 5-2

............................. 5-2

.............................. 5-4

................. 5-5

............ 5-5

...................... 5-7

..................................... 5-11

............... 5-12

............... 5-13

............................ 5-14

vii

Indexing into a Codistributed Array .................. 5-15

2-Dimensional Distribution

Using a for-Loop Over a Distributed Range

(for-drange)

Parallelizing a for-Loop
Codistributed Arrays in a for-drange Loop

..................................... 5-21

......................... 5-17

............................. 5-21

............. 5-22

Using MATLAB Functions on Codistributed Arrays

Programming Overview

6

Product Introduction .............................. 6-2

Overview
Toolbox and Server Components

Using Parallel Computing Toolbox Software

Example: Evaluating a Basic Function
Example: Programming a Basic Job with a Local

Scheduler

Getting Help

Program Development Guidelines

Life Cycle of a Job

........................................ 6-2

..................... 6-3

......... 6-8

................ 6-8

...................................... 6-8

..................................... 6-10

................... 6-12

................................. 6-14

... 5-25

viii Contents

Programming with User Configurations

Defining Configurations
Exporting and Importing Configurations
Validating Configurations
Applying Configurations in Client Code

Programming Tips and Notes

Saving or Sending Objects
Current Working Directory of a MATLAB Worker
Using clear functions

............................ 6-16

.............. 6-23

.......................... 6-24

............... 6-27

....................... 6-29

.......................... 6-29

.............................. 6-30

............. 6-16

....... 6-29

Running Tasks That Call Simulink Software ........... 6-30

Using the pause Function
Transmitting Large Amounts of Data
Interrupting a Job
Speeding Up a Job

................................. 6-30

................................. 6-31

........................... 6-30

................. 6-30

Using the Parallel Profiler

Introduction
Collecting Parallel Profile Data
Viewing Parallel Profile Data
Parallel Profiler Demos

Benchmarking Performance

Demos
HPC Challenge Benchmarks

Troubleshooting and Debugging

Object Data Size Limitations
File Access and Permissions
No Results or Failed Job
Connection Problems Between the Client and Job

Manager

...................................... 6-32

.......................................... 6-44

....................................... 6-48

Evaluating Functions in a Cluster

7

.......................... 6-32

...................... 6-32

........................ 6-33

............................. 6-42

........................ 6-44

........................ 6-44

.................... 6-45

........................ 6-45

......................... 6-45

............................ 6-47

Evaluating Functions Synchronously ................ 7-2

Scope of dfeval
Arguments of dfeval
Example — Using dfeval

Evaluating Functions Asynchronously

.................................... 7-2

............................... 7-3

........................... 7-4

.............. 7-8

ix

Programming Distributed Jobs

8

Using a Local Scheduler ............................ 8-2

Creating and Running Jobs with a Local Scheduler
Local Scheduler Behavior

........................... 8-7

...... 8-2

Using a Job Manager

Creating and Running Jobs with a Job Manager
Sharing Code
Managing Objects in the Job Manager

Using a Fully Supported Third-Party Scheduler

Creating and Running Jobs
Sharing Code
Managing Objects

Using the Generic Scheduler Interface

Overview
MATLABClientSubmitFunction
Example — Writing the Submit Function
MATLAB Worker Decode Function
Example — Writing the Decode Function
Example — Programming and Running a Job in the

Client
Supplied Submit and Decode Functions
Managing Jobs
Summary

..................................... 8-13

..................................... 8-28

........................................ 8-33

......................................... 8-42

........................................ 8-51

............................... 8-8

................ 8-16

......................... 8-20

................................. 8-30

.............. 8-33

.................... 8-34

.............. 8-38

................... 8-39

.............. 8-41

............... 8-47

................................... 8-48

........ 8-8

...... 8-20

x Contents

Programming Parallel Jobs

9

Introduction ...................................... 9-2

Using a Supported S cheduler

Schedulers and Conditions
Coding the Task Function
Coding in the Client

............................... 9-5

....................... 9-4

.......................... 9-4

.......................... 9-4

Using the Generic Scheduler Interface .............. 9-8

Introduction
Coding in the Client

...................................... 9-8

............................... 9-8

10

Further Notes on Parallel Jobs

Number of Tasks in a Parallel Job
Avoiding Deadlock and Other Dependency Errors

..................... 9-11

.................... 9-11

....... 9-11

Object Reference

Data Objects ...................................... 10-2

Scheduler Objects

Job Objects

Task Objects

Worker Objects

....................................... 10-3

................................. 10-2

...................................... 10-3

.................................... 10-3

11

12

Objects — Alphabetical List

Function Reference

Parallel Code Execution ............................ 12-2

Parallel Code on a MATLAB Pool
Configuration, Input, and Output
Interactive Functions

.............................. 12-3

.................... 12-2

.................... 12-2

xi

Distributed and Codistributed Arrays ............... 12-3

Toolbox Functions
Overloaded MATLAB Functions

................................. 12-3

..................... 12-4

13

14

Job and Task Programming

Job Creation
Job Management
Task Execution Information
Object Control

Interlab Communication Within a Parallel Job

Utilities

........................................... 12-10

..................................... 12-6

.................................. 12-7

.................................... 12-9

........................ 12-6

......................... 12-8

....... 12-9

Functions — Alphabetical List

Property Reference

Job Manager Properties ............................ 14-2

xii Contents

Scheduler Properties

Job Properties

Task Properties

Worker Properties

.................................... 14-5

................................... 14-6

.............................. 14-3

................................. 14-8

15

Properties — Alphabetical List

Glossary

Index

xiii

xiv Contents

Getting Started

• “Product Overview” on page 1-2

• “Typical Use Cases” on page 1-3

• “Introduction to Parallel Solutions” on page 1-5

• “Determining Product Installation and Versions” on page 1-12

1

1 Getting Started

Product Overview

Parallel Computing Toolbox™ software allows you to offlo ad work from one
MATLAB
You can use multiple workers to take advantage of parallel processing. You
can use a local worker to keep your MATLAB client session free for interactive
work, or with MATLAB
advantage of another computer’s speed.

Parallel Computing Toolbo x softwareallowsyoutorunasmanyaseight
MATLAB workers on your local machine in addition to your MATLAB client
session. M A TLAB Distributed Computing Server software allows you to
run as many MATLAB workers on a remote cluster of computers as your
licensing allows.

®

session (the client) to other MATLAB sessions, called workers.

®

Distributed Computing Server™ you can take

1-2

Typical Use Cases

In this section...

“Parallel for-Loops (parfor)” on page 1-3

“Batch Jobs” on page 1-4

“Large Data Sets” on page 1-4

Parallel for-Loops (parfor)

Many applications involve multiple segments of code, some of which are
repetitive. Often you can use for-loops to solve these cases. The ability to
execute code in parallel, on one computer or on a cluster of computers, can
significantly improve performance for many use cases:

• Parameter sweep applications

- Many iterations — A sweep might take a long time becaus e it comprises

many iterations. Each iteration by itself might not take long to execute,
but to complete thousands or millions of iterations in serial could take
a long time.

Typical Use Cases

- Long iterations — A sweep might not have a lot of iterations, but each

iteration could take a long time to run.

Typically, the only difference between iterations is defined by different
input data. In these cases, the ability to run separate sweep iterations
simultaneously can improve performance. Evaluating such iterations in
parallel is an ideal way to sweep through large or multiple data sets. The
only restriction on parallel loops is that no iterations be allowed to depend
on any other iterations.

• Test suites with independent segments — For applications that run a

series of unrelated tasks, you can run these tasks simultaneously on
separate resources. You might not have used a
this comprising distinctly different tasks, but a
appropriate solution.

Parallel Computing Toolbox software improves the performance of such loop
execution by allowing several MATLAB workers to execute individual loop
iterations simultaneously. For example, a loop of 100 iterations could run on

for-loop for a case such as

parfor-loop could offer an

1-3

1 Getting Started

a cluster of 20 MATLAB workers, so that simultaneously, the workers each
execute only five iterations of the loop. You might not get quite 20 times
improvement in speed because of communications overhe ad and network
traffic, but the speedup should be significant. Even running local workers all
on the same machine as the client, you might see significant performance
improvement on a multicore/multiprocessor machine. So whether your loop
takes a long time to run because it has many iterations or because each
iteration takes a long time, you c an improve your loop speed by distributing
iterations to MATLAB workers.

Batch Jobs

When working interactively in a MATLAB session, you can offload work to
a MATLAB worker session to run as a batch job. The command to perform
this job is asynchronous, which means that your client MATLAB session is
not blocked, and you can continue your own interactive session while the
MATLAB worker is busy evaluating your code. The MATLAB worker can run
either on the same machine as the client, or if using MATLAB Distributed
Computing Server, on a remote cluster machine.

1-4

Large Data Sets

If you have an array that is too large for your computer’s memory, it cannot
be easily handled in a single MATLAB session. Parallel Computing Toolbox
software allows you to distribute that array among multiple MATLAB
workers, so that each worker contains only a part of the array. Yet you can
operate on the entire array as a single entity. Each worker operates only
on its part of the array, and workers automatically transfer data between
themselves when necessary, as, for example, in matrix multiplication. A
large number of matrix operations and functions have been enhanced to work
directly with these arrays without further modification; see “Using MATLAB
Functions on Codistributed Arrays” on page 5-25 and “Using MATLAB
Constructor Functions” on page 5-10.

Introduction to Parallel Solutions

In this section...

“Interactively Running a Loop in Parallel” on page 1-5

“Running a Batch Job” on page 1-7

“Running a Batch Parallel Loop” on page 1-8

“Using Distributed Arrays, spmd, and Composite s” on page 1-10

Interactively Running a Loop in Parallel

This section shows how to modify a simple for-loop so that it runs in parallel.
This loop does not have a lot of iterations, and it does not take long to execute,
butyoucanapplytheprinciplestolargerloops. Forthesesimpleexamples,
you might not notice an increase in execution speed.

1 Suppose your code includes a loop to create a sine wave and plot the

waveform:

Introduction to Parallel Solutions

for i=1:1024

A(i) = sin(i*2*pi/1024);
end
plot(A)

2 To interactively run code that contains a parallel loop, you first open a

MATLAB pool. This reserves a collection of MATLAB worker sessions
to run your loop iterations. The MATLAB pool can consist of MATLAB
sessions running on your local machine or on a remote cluster:

matlabpool open local 3

3 With the MATLAB pool reserved, you can modify your code to run your loop

in parallel by using a

parfor i=1:1024

A(i) = sin(i*2*pi/1024);
end
plot(A)

parfor statement:

1-5

1 Getting Started

The only difference in this loop is the keyword parfor instead of for.
After the loop runs, the results look the same as those generated from
the previous

MATLAB
client

®

for-loop.

parfor

MATLAB
workers

®

Because the iterations run in parallel in other MATLAB sessions, each
iteration must be com pletely independent of all other iterations. The
worker calculating the value for
calculating
be calculated before
some problems with

A(500). There is no guarantee of sequence, so A(900) might

A(400). (The MATLAB Editor can help identify

parfor code that might not contain independent

A(100) might not be the same worker

iterations.) The only place where the values of all the elements of the array

A are available is in the MATLAB client, after the data returns from the

MATLAB workers and the loop completes.

1-6

4 When you are finished with your code, close the MATLAB pool and release

the workers:

matlabpool close

For more information on parfor-loops, see Chapte r 2, “Parallel for-Loops
(parfor)”.

Theexamplesinthissectionrunonthreelocalworkers. Withparallel
configurations, you can control how many workers run your loops, and
whether the workers are local or remote. For more information on parallel
configurations, see “Programming with User Configurations” on page 6-16.

Introduction to Parallel Solutions

Running a Batch J

To offload work f
the

batch comman

inside a script

1 To create the s

rom your MATLAB session to another session, you can use

d. This example uses th e

.

cript, type:

ob

for-loop from the last se ction

edit mywave

2 In the MATLAB Editor, enter the text of the for-loop:

for i=1:1024

A(i) = sin(i*2*pi/1024);
end

3 SavethefileandclosetheEditor.

4 Use the batch command in the MATLAB Command Window to run your

script on a separate MATLAB worker:

job = batch('mywave')

MATLAB
client

®

batch

MATLAB
worker

®

5 The batch commanddoesnotblockMATLAB,soyoumustwaitforthejob

to finish before you can retrieve and view its re su lts:

wait(job)

6 The load command transfers variables from the workspace of the worker to

the w orkspace of the client, where you can view the results:

load(job, 'A')
plot(A)

7 When the job is complete, permanently remove its data:

destroy(job)

1-7

1 Getting Started

Running a Batch P

You can combine t
previous two exa
you submitted a
the two to creat

1 Open your scri

edit mywave

2 Modify the sc

parfor i=1:1024

A(i) = sin(i*2*pi/1024);
end

3 Save the fi

4 Run the script in MATLAB with the batch command as before, but indicate

he abilities to offload a job and run a parallel loop. In the

mples, you modified a

script with a

e a batch

pt in the MATLAB Editor:

ript so that the

le and close the Editor.

arallel Loop

for-looptomakeaparfor-loop, and

for-loop as a batch job. This example combines

parfor-loop.

for statement is a parfor statement:

that the script should use a MATLAB pool for the parallel loop:

job = batch('mywave', 'matlabpool', 3)

This command specifies that three workers (in addition to the one running
the batch script) are to evaluate the loop iterations. Therefore, this example
uses a total of four local workers, including the one worker running the
batch script.

1-8

Introduction to Parallel Solutions

®

MATLAB
client

batch

5 To view the results:

wait(job
load(job

)
, 'A')

parfor

MATLAB
workers

®

plot(A)

The results look the same as before, however, there are tw o important
differences in execution:

• The work of defining the

offloaded to another MATLAB session (

parfor-loop and accumulating its results are

batch).

• The loop iterations are distributed from one MATLAB worker to another

set of workers running simultaneously (

matlabpool and parfor), so the

loop might run faster than having only one worker execute it.

6 When

the job is complete, permanently remove its data:

destroy(job)

1-9

1 Getting Started

Using Distribut

ed Arrays, spmd, and Composites

Distributed Arrays

The workers in a
distribute an
all the labs ar

First, open th

matlabpool open % Use default parallel configuration

Use the distributed function to dist

M = magic(4) % a 4-by-4 magic square in the client workspace
MM = distributed(M)

Now MM is a distributed array, equivalent to M,andyoucanma
access its elements in the same way as any other array.

M2 = 2*MM; % M2 is also distributed, calculation performed on workers
x = M2(1,1) % x on the client is set to first element of M2

When you are finished and have no further need of data from the labs, you can
close t he MATLAB pool. Data on the labs does not persist from one instance
of a M ATLA B pool to another.

MATLAB pool communicate with each other, so you can

array among the labs. Each lab contains part of the array, and

e aw are of which portion of the array each lab has.

e MATLAB pool:

ribute an array among the labs:

nipulate or

1-10

matlabpool close

Single Program Multiple Data

The single program multiple data (spmd) construct lets you define a block of
code that runs in parallel on all the labs (w orkers ) in the MATLAB pool. The

spmd block can run on some or all the labs in the pool.

matlabpool % Use default parallel configuration
spmd % By default uses all labs in the pool

R = rand(4);

end

This code creates an individual 4-by-4 matrix, R, of random numbers on
each lab in the pool.

Introduction to Parallel Solutions

Composites

Following an spmd statement, in the client context, the values from the
block a re accessible, even though the data is actually stored on the labs. On
the client, these variables are called Composite objects. Each element of a
composite is a symbol referencing the value(data)onalabinthepool.Note
that because a variable might not be define d on every lab, a Composite might
have undefined elements.

Continuing with the example from above, on the client, the Composite

R has

one element for each lab:

X = R{3}; % Set X to the value of R from lab 3.

The line above retrieves the data from lab 3 to assign the value of X.The
following code sends data to lab 3:

X=X+2;
R{3} = X; % Send the value of X from the client to lab 3.

If the MATLAB pool remains open between spmd statements and the same
labs are used, the data on each lab persists from one

spmd statement to

another.

spmd

R = R + labindex % Use values of R from previous spmd.

end

A typical use case for sp md is to run the same code on a number of labs, each
of which accesses a different set of data. For example:

spmd

INP = load(['somedatafile' num2str(labindex) '.mat']);
RES = somefun(INP)

end

Then the values of RES on the labs are accessible from the client as RES{1}
from lab 1, RES{2} from lab 2, etc.

There are two forms of indexing a Composite, comparable to indexing a cell
array:

1-11

1 Getting Started

• AA{n} returns the values of AA from lab n.

AA(n) returns a cell array of the content of AA from lab n.

•

When you are finished with all
data from the labs, y ou can close the MATLAB pool.

matlabpool close

Although data persists on the labs from one spmd block to another as long as
the MATLAB pool remains open, data does not persist from one instance of
a MATLAB pool to another.

For more information about using distributed arrays,
see Chapter 3, “Single Program Multiple Data (spmd)”.

spmd execution and have no further need of

spmd, and Composites,

Determining Product Installation and Versions

To determine if Parallel Computing Toolbox software is installed on your
system, type this command at the MATLAB prompt.

ver

When you enter this command, MATLAB displays information about the
version of MATLAB you are running, including a list of all toolboxes installed
on your system and their version numbers.

1-12

If you want to run your applications on a cluster, see your system
administrator to verify that the version of Parallel Computing Toolbox you
are using is the same as the version of MATLAB Distributed Computing
Server installed on your cluster.

Parallel for-Loops (parfor)

• “Getting Started with parfor” on page 2-2

• “Programming Considerations” on page 2-8

• “Advanced Topics” on page 2-14

2

2 Parallel for-Loops (parfor)

Getting Started with parfor

In this section...

“Introduction” on page 2-2

“WhentoUseparfor”onpage2-3

“Setting up MATLAB Resources Using matlabpool” on page 2-3

“Creating a parfor-Loop” on page 2-4

“Differences Between for-Loops and parfor-Loops” on page 2-5

“Reduction Assignments” on page 2-6

“Displaying Output” on page 2-7

Introduction

The basic concept of a parfor-loop in MATLAB software is the same as the
standard MATLAB
loop body) over a range of values. Part of the
MATLAB client (where the
on M ATLAB workers. The necessary data on which
from the client to workers, where most of the computation happens, and the
results are sent back to the client and pieced together.

for-loop: MATLAB executes a series of statements (the

parfor body is executed on the

parfor is issued) and part is executed in parallel

parfor operates is sent

2-2

Because several MATLAB workers can be computing concurrently on the
same loop, a
its analogous

Each execution of the body of a
workers evaluate iterations in no particular order, and independently of each
other. Because each iteration is independent, there is no guarantee that the
iterations are synchronized in any way, nor is there any need for this. If the
number of workers is equal to the number of loop iterations, each worker
performs one iteration of the loop. If there are m ore iterations than workers,
some workers perform more than one loop iteration; in this case, a worker
might receive multiple iterations at once to reduce communication time.

parfor-loop can provide significantly better performance than

for-loop.

parfor-loop is an iteration. M ATLA B

Getting Started with parfor

When to Use parfor

A parfor-loop is useful in situations where you need many loop iterations of
a simple calculation, such as a Monte Carlo simulation.
loop iterations into groups so that e ach worker executes some portion of the
total number of iterations.
iterations that take a long time to execute, because the workers can execute
iterations simultaneously.

parfor-loopsarealsousefulwhenyouhaveloop

parfor divides the

You cannot use a
results of other iterations. Each iteration must be independent of all others.
Since there is a communications cost involved in a
be no advantage to using one when you have only a small number of simple
calculations. The example of this section are only to illustrate the behavior
of

parfor-loops, not necessarily to demonstrate the applications best suited

to them.

parfor-loopwhenaniterationinyourloopdependsonthe

parfor-loop, there might

Setting up MATLAB Resources Using matlabpool

You use the function matlabpool to reserve a number of MATLAB w orkers
for executing a subsequent
workers might be running remotely on a cluster, or they might run locally
on your MATLAB client machine. You identify a scheduler and cluster by
selecting a parallel configuration. For a description of how to manage and use
configurations, see “Programming with User Configurations” on page 6-16.

To begin the examples of this section, allocate local MATLAB w orkers for
the evaluation of your loop iterations:

matlabpool

This command starts the number of MATLAB worker sessions defined by
the default parallel configuration. If the local configuration is your default
and does not specify the number of workers, this starts one worker per core
(maximum of eight) on your local MATLAB client machine.

parfor-loop. Depending on your scheduler, the

Note If matlabpool is not running, a parfor-loop runs serially on the client
without regard for iteration sequence.

2-3

2 Parallel for-Loops (parfor)

Creating a par for-Loop

The safest assumption about a parfor-loop is that each iteration of the
loop is evaluated by a different MATLAB worker. If you have a
which all iterations are completely independent of each other, this loop is a
good candidate for a

parfor-loop. Basically, if one iteration depends on the

results of another iteration, these iterations are not independent and cannot
be evaluated in parallel, so the loop does not lend itself easily to conversion
to a

parfor-loop.

for-loop in

The following examples produce equivalent results, with a
left, and a

parfor-loop on the right. Try typing each in your MATLAB

for-loop on the

Command Window:

clear A
fori=1:8

A(i) = i;
end
A

clear A
parfor i = 1:8

A(i) = i;
end
A

Notice that each element of A is equal to its index. The parfor-loop works
because each element depends only upon its iteration of the loop, and upon
no other iterations.
ideally suited candidates for

for-loops that merely repeat such independent tasks are

parfor-loops.

2-4

Getting Started with parfor

Differences Between for-Loops and parfor-Loops

Because parfor-loops are not quite the same as for-loops, there are special
behaviors to be aware of. As seen from the preceding example, when y ou
assign to an array variable (such as
indexing with the loop variable, the elements of that array are available to
you after the loop, much the same as with a

However, suppose you use a nonindexed variable inside the loop, or a variable
whose indexing does not depend on the loop variable
and notice the values of

d and i afterward:

A in that example) inside the loop by

for-loop.

i.Trytheseexamples

clear A
d=0;i=0;
fori=1:4

d = i*2;

A(i) = d;
end
A
d
i

clear A
d=0;i=0;
parfor i = 1:4

d = i*2;

A(i) = d;
end
A
d
i

Although the elements of A come out the same in both of these examples, the
value of
in sequence, so afterward
loop. In the
sequence, so it would be impossible to assign
of the loop. This also applies to the loop variable,
behavior is defined so that it does not affect the values

d does not. In the for-loop above on the left, the iterations execute

d has the value it held in the last iteration of the

parfor-loop on the right, the iterations execute in parallel, not in

d a definitive value at the end

i. Therefore, parfor-loop

d and i outside the

loop at all, and their values remain the same before and after the loop.
So, a

parfor-loop requires that each iteration be independent of the other

iterations, and that all code that follows the

parfor-loop not depend on the

loop iteration sequence.

2-5

2 Parallel for-Loops (parfor)

Reduction Assig

The next two exam
reduction is an a
left uses

x to ac

on the right gen
the execution
the workers ca
assembles the

x=0;
parfor i = 1:10

x=x+i;
end
x

ccumulation across iterations of a loop. The example on the

cumulate a sum across 10 iterations of the loop. The example

erates a concatenated array,

order of the iterations on the workers does not matter: w hile

lculate individual results, the client properly accumulates or
final loop re sult.

nments

ples show

parfor-loops using reduction assignments. A

1:10. In both of these examples,

x2 = [];
n=10;
parfor i = 1:n

x2 = [x2, i];
end
x2

If the loop iterations operate in random sequence, you might expect the
concatenation sequence in the example on the right to be nonconsecutive.
However, MATLAB recognizes the concatenation operation and yields
deterministic results.

The next example, which attempts to compute Fibonacci numbers, is not
avalid
depends on the values of other elements of

parfor-loop because the value of a n element of f in one iteration

f calculated in other iterations.

2-6

f = zeros(1,50);
f(1) = 1;
f(2) = 2;
parfor n = 3:50

f(n) = f(n-1) + f(n-2);

end

When
clos

you are finished with your loop examples, clear your workspace and

e or rele as e your pool of w orkers :

ar

cle

labpool close

mat

The following sections provide further information regarding programming
considerations and limitations for

parfor-loops.

Getting Started with parfor

Displaying Outp

When running a pa
from the workers
variable assig
displays, any g
does not displ

rfor

displays in the client Command Window, except output from

nments. Because the workers are MATLAB sessions without

raphical output (for example, figure windows) from the pool

ay at all.

ut

-loop on a MATLAB pool, all command-line output

2-7

2 Parallel for-Loops (parfor)

Programming Considerations

In this section...

“MATLAB Path” on page 2-8

“Error Handling” on page 2-8

“Limitations” on page 2-9

“Using Objects in parfor Loops” on page 2-11

“Performance Considerations” on page 2-12

“Compatibility with Earlier Versions of MATLAB Software” on page 2-13

MATLAB Path

All workers e xecuting a parfor-loop must have the same M ATLAB path
configuration as the client, so that they can execute any functions called in the
body of the loop. Therefore, whenever you use
client, it also executes on all the workers, if possible. For more information,
see the
different platform than the client, use the function
settheMATLABpathonallworkers.

matlabpool reference page. When the workers are running on a

cd, addpath,orrmpath on the

pctRunOnAll to properly

2-8

Error Handling

When an error occurs during the execution of a parfor-loop, all iterations
that are in progress are terminated, new ones are not initiated, and the loop
terminates.

Errors and warnings produced on workers are annotated with the worker ID
and displayed in the client’s Command Window in the order in which they
are received by the client M ATLA B.

The behavior of
within the loop body.

lastwarn is unspecified at the end of the parfor if used

Programming Considerations

Limitations

Unambiguous Variable Names

If you use a name that MATLAB cannot unambiguously distinguish as a
variable inside a
referencing a function. Then at run-time, if the function cannot be found,
MATLAB generates an error. (See “Naming Variables” in the MATLAB
documentation.) For example, in the following code
to the fifth element of an array named
argument of
looks for the function

parfor i=1:n

...
a = f(5);
...

end

parfor-loop, at parse time MATLAB assumes you are

f(5) could refer either

f, or to a function named f with an

5.Iff is not clearly defined as a variable in the code, MATLAB
f on the path when the code runs.

Transpar

ency

The body of a parfor-loop must be transparent, meaning that all references to
variables must be “visible” (i.e., they occur in the text of the program).

In the following example, because

parfor body (only the string 'X' is passed to eval), it does not get transferred

X is not visible as an input variable in the

to the workers. As a result, MATLAB issues an error at run time:

X=5;

rii=1:4

parfo

'X');

eval(

end

Similarly, you cannot clear variables from a worker’s workspace by executing

clear inside a parfor statement:

or ii= 1:4

parf

atements...>

<st

ar('X') % cannot clear: transparency violation

cle

atements...>

<st

d

en

2-9

2 Parallel for-Loops (parfor)

As a workaround, you can free up most of the memory used by a variable by
setting its value to empty, presumably when it is no longer needed in your

parfor statement:

parfor ii= 1:4

<statements...>
X=[];
<statements...>

end

Examples of some other functions that violate transparency are evalc,

evalin,andassignin with the workspace argument specified as 'caller';
save and load, unless the output of load is assigned.

MATLAB does successfully execute
functions called from the

parfor body.

eval and evalc statements that appear in

Sliced Variables Referencing Function H andles

Because of the way sliced input variables are segmented and distributed to
the workers in the pool, you cannot use a sliced input variable to reference a
function handle. If you need to call a function handle with the
variable as an argument, use

For example, supp ose you had a

B = @sin;
for ii = 1:100

A(ii) = B(ii);

end

feval.

for-loop that performs:

A corresponding parfor-loop does not allow B to reference a function handle.
So you can work around the problem with

B = @sin;
parfor ii = 1:100

A(ii) = feval(B, ii);

end

feval:

parfor index

2-10

Programming Considerations

Nondistributable Functions

If you use a function that is not strictly computational in nature (e.g., input,

plot, keyboard)inaparfor-loop or in any function called by a parfor-loop,

the behavior of that function occurs on the worker. The results might include
hanging the worker process or having no visible effect at all.

Nested Functions

The body of a parfor-loop cannot make reference to a nested function.
However, it can call a nested function by means of a function handle.

Nested parfor-Loops

The body of a parfor-loop cannot contain another parfor-loop. However, it
can call a function that contains another

parfor-loop.

Nested spmd Statements

The body of a parfor-loop cannot contain an spmd statement, and an spmd
statementcannotcontainaparfor-loop.

Break and Return Statements

The body of a parfor-loop cannot contain break or return statements.

Global and Persistent Variables

The body of a parfor-loop cannot contain global or persistent variable
declarations.

P-Code Scripts

You can call P-code script files from within a parfor-loop, but P-code script
cannot contain a

parfor-loop.

Using Objects in parfor Loops

If you are passing objects into or out of a parfor-loop, the objects must
properly facilitate being saved and loaded. For more information, see “Saving
and Loading Objects”.

2-11

2 Parallel for-Loops (parfor)

Performance Considerations

Slicing Arrays

If a variable is initialized before a parfor-loo p, then used inside the

parfor-loop, it has to be passed to each MATLAB worker evaluating the loop

iterations. Only those variables used inside the loop are passed from the
client w orkspace. However, if all occurrences of the variable are indexed by
the loop variable, each worker receives only the part of the array it needs. For
moreinformation,see“WheretoCreateArrays”onpage2-28.

Local vs. Cluster Workers

Running your code on local workers might offer the convenience of testing
your application without requiring the use of cluster resources. However,
there are certain drawbacks or limitations with using local workers. Because
the transfer of data does not occur over the network, transfer behavior on local
workers might not be indicative of how it will typically occur over a network.
For more details, see “Optimizing on Local vs. Cluster Workers” on page 2-29.

2-12

Programming Considerations

Compatibility with Earlier Versions of MATLAB
Software

In versions of MATLAB prior to 7.5 (R2007b), the keyword parfor designated
a more limited style of
and later. This old style w as intended for use with codistributed arrays (such
as inside an

for-loop that uses drange to define its range; see “Using a for-Loop Over a

spmd statement or a parallel job), and has been replaced by a

Distributed Range (for-drange)” on page 5-21.

parfor-loop than what is available in MATLAB 7.5

The past and current functionality of the
following table:

Functionality

Syntax Prior to
MATLAB 7.5

Parallel loop for
codistributed
arrays

Parallel loop

parfor i = range

loop body

.
.

end

Not Implemented
for implicit
distribution of
work

parfor keyword is outlined in the

Current Syntax

for i = drange(range)

loop body

.
.

end

parfor i = range

loop body

.
.

end

2-13

2 Parallel for-Loops (parfor)

Advanced Topics

In this section...

“About Programming Notes” on page 2-14

“Classification of Variables” on page 2-14

“Improving Performance” on page 2-28

About Programming Notes

This section presents guidelines and restrictions in shaded boxes like the one
shown below. Those labeled as Required result in an error if your
code does not adhere to them. MATLAB software catches some of these errors
at the time it reads the code, and others when it executes the code. These are
referred to here as static and dynamic errors, respectively, and are labeled as
Required (static) or Required (dynamic). Guidelines that do not cause
errors are labeled as Recommended. You can use M-Lint to help make your

parfor-loops comply w i th these guidelines.

parfor

2-14

Required (static): Description of the guideline or restriction

Classification of Variables

• “Overview” on page 2-14

• “Loop Variable” on page 2-15

• “Sliced Variables” on page 2-16

• “Broadcast Variables” o n page 2-20

• “Reduction Variables” on page 2-20

• “Temporary Variables” on page 2-26

Overview

When a name in a parfor-loop is recognized as referring to a variable, it is
classified into one of the following categories. A

parfor-loop generates an

Advanced Topics

error if it contains any variables that cannot be uniquely categorized or if any
variables violate their category restrictions.

Classification

Loop

Sliced An array whose segments are operated on by different

Broadcast

Reduction

Temporary

Each of these variable classifications appears in this code fragment:

temporary variable

reduction variable

Description

Serves as a loop index for arrays

iterations of the loop

A variable defined be fore the loop whose value is used
inside the loop, but never assigned inside the loop

Accumulates a value across iterations of the loop,
regardless of iteration order

Variable created inside the loop, but unlike sliced or
reduction variables, not available outside the loop

loop variable

sliced input variable

sliced output variable

broadcast variable

Loop Variable

The following restriction is required, because changing i in the parfor body
invalidates the assumptions M ATLA B makes about communication between
the client and workers.

2-15

2 Parallel for-Loops (parfor)

Required (static): Assignments to the loop variable are not allowed.

This example attempts to modify the value of the loop variable i in the body
of the loop, and thus is invalid:

parfor i = 1:n

i=i+1;
a(i) = i;

end

Sliced Variables

A sliced variable is one whose value can be broken up into segments, or slices,
which are then operated on separately by workers and by the MATLAB client.
Each iteration of the loop works on a different slice of the array. Using sliced
variables is important because this type of variable can reduce communication
between the client and workers. Only those slices needed by a worker are sent
to it, and only when it starts working on a particular range of indices.

2-16

Inthenextexample,asliceof

parfor i = 1:length(A)

B(i) = f(A(i));

end

A consists of a single element of that array:

Characteristics of a Sliced Variable. A variable in a parfor-loopisslicedif
it has all of the following chara cteri stic s. A description of each characteristic
follows the list:

• Type of First-Level Indexing — The first level of indexing is either

parentheses,

(), or braces, {}.

• Fixed Index Listing — Within the first-level parenthesis or braces, the list

ofindicesisthesameforalloccurrences of a given variable.

• Form of Indexing — Within the list of indices for the variable, exactly one

index involves the loop variable.

• ShapeofArray—Inassigningtoasliced variable, the right-hand side

of the assignme nt is not

[] or '' (these operators indicate deletion of

elements).

Advanced Topics

Type of First-Level Indexing. For a sliced variable, the first level of indexing is
enclosed in either parentheses,

(), or braces, {}.

This table lists the forms for the first level of indexing for arrays sliced and
not sliced.

Reference for Variable Not

Reference for Sliced Variable

Sliced

A.x A(...)

A.(...) A{...}

After the first level, you can use any type of valid MATLAB indexing in the
second and further levels.

The variable

A shown here on the left is not sliced; that shown on the right

is sliced:

A.q{i,12} A{i,12}.q

Fixed Index Listing. Within the first-level parentheses or braces of a sliced
variable’s indexing, the list of indices is the same for all occurrences of a
given variable.

The variable

A shown here on the left is not sliced because A is indexed by i

and i+ 1 in different places; that shown on the right is sliced:

parfor i = 1:k

B(:) = h(A(i), A(i+1));

end

parfor i = 1:k

B(:) = f(A(i));
C(:) = g(A{i});

end

The example above on the right show s some occurrences of a sliced variable
with first-level parenthesis indexing and with first-level brace indexing in the
sameloop. Thisisacceptable.

Form of Indexing. Within the list of indices for a sliced variable, one of these
indices is of the form

i, i+k, i-k, k+i,ork-i,wherei is the loop variable and

2-17

2 Parallel for-Loops (parfor)

k is a constant or a simple (nonindexed) variable; and every other index is a

constant, a simple variable, colon, or

With

i as the loop variable, the A variables shown here on the left are not

end.

sliced; those on the right are s liced:

A(i+f(k),j,:,3)
A(i,20:30,end)
A(i,:,s.field1)

A(i+k,j,:,3)
A(i,:,end)
A(i,:,k)

When you use other variables along with the loop variable to index an array,
you cannot set these variables inside the loop. In effect, such variables are
constant over the execution of the entire

parfor statement. You cannot

combine the loop variable with itself to form an index expre ssion.

Shape of Array. A sliced variable must maintain a constant shape. The
variable

A shown here on either line is not sliced:

A(i,:) = [];
A(end + 1) = i;

The reason A is not sliced in either case is because changing the shape of a
sliced array would violate assumptions governing communication between
the client and workers.

Sliced Input and Output Variables. All sliced variables have the
characteristics of being input or output. A sliced variable can sometimes have
both characteristics. MATLAB transmits sliced input variables from the client
to the workers, and sliced output variables from workers back to the client. If
a variable is both input and output, it is transmitted in both directions.

2-18

Advanced Topics

In this parfor-loop, r is a sliced input variable and b is a sliced output
variable:

a=0;
z=0;
r = rand(1,10);
parfor ii = 1:10

a = ii;
z=z+ii;
b(ii) = r(ii);

end

However, if it is clear that in every iteration, every reference to an array
element is set before it is used, the variable is not a sliced input variable. In
this example, all the elements of

A are set, and then only those fixed values

are used:

parfor ii = 1:n

if someCondition

A(ii) = 32;

else

A(ii) = 17;
end
loop code that uses A(ii)

end

Even if a sliced variable is not explicitly referenced as a n input, implicit
usage might make it so. In the following example, not all elements of
necessarily set inside the
are received, held, and then returned from the loop, making

parfor-loop, so the original values of the array

A both a sliced

A are

input and output variable.

A = 1:10;
parfor ii = 1:10

if rand < 0.5

A(ii) = 0;

end

end

2-19

2 Parallel for-Loops (parfor)

Broadcast Variables

A broadcast variable is any variable other than the loop variable or a sliced
variable that is not affected by an assignment inside the loop. At the start of
a

parfor-loop, the values of any broadcast variables are sent to all work ers.

Although this type of variable can be useful or even essential, broadcast
variables that are large can cause a lot of communication between client and
workers. In some cases it m ight be more efficient to use temporary variables
for this purpose, creating and assigning them inside the loop.

Reduction Variables

MATLAB supports an important exception, called reductions, to the rule that
loop iterations must be independent. A reduction variable accumulates a
value that depends on all the iterations together, but is independent of the
iteration order. MATLAB allows reduction variables in

Reduction variables appear on both side of an assignment statement, such as
any of the following, where

expr is a MATLAB expression.

parfor-loops.

2-20

X = X + expr X = expr + X

X = X - expr

See Associativity in Reduction
Assignments in “Further
Considerations with Reduction
Variables” on page 2-22

X=X.*expr X=expr.*X

X = X * expr X = expr * X

X = X & expr X = expr & X

X = X | expr X = expr | X

X = [X, expr] X = [expr, X]

X = [X; expr] X = [expr; X]

X = {X, expr} X = {expr, X}

X = {X; expr} X = {expr; X}

X = min(X, expr) X = min(expr, X)

X = max(X, expr) X = max(expr, X)

Advanced Topics

X = union(X, expr) X = union(expr, X)

X = intersect(X, expr) X = intersect(expr, X)

Each of the allowed statements listed in this table is referred to as a reduction
assignment,and,bydefinition,areduction variable can appear only in

assignments of this type.

The follow ing example show s a typical usage of a reduction variable

X = ...; % Do some init ialization of X
parfor i = 1:n

X = X + d(i);

end

X:

This loop is equivalent to the following, where each d(i) is calculated by
a different iteration:

X = X + d(1) + ... + d(n)

Iftheloopwerearegularfor-loop, the variable X in each iteration would get
its value either before entering the loop or from the previous iteration of the
loop. However, this concept does not apply to

In a

parfor-loop, the value of X is never transmitted from client to workers or

from worker to worker. Rather, additions of
with

i ranging over the subset of 1:n being performed on that worker. The

parfor-loops:

d(i) aredoneineachworker,

results are then transmitted back to the client, which adds the workers’
partial sums into

X. Thus, workers do some of the additions, and the client

does the rest.

Basic Rules for Reduction Variables. The following requirements further
define the reduction assignments associated with a given variable.

Required (static): For any reduction variable, the same reduction function
or operation must be used in all reduction assignments for that variable.

The parfor-loop on the left is not valid because the reduction assignment uses

+ in one instance, and [,] in another. The parfor-loop on the right is valid:

2-21

2 Parallel for-Loops (parfor)

parfor i = 1:n

if A > 5*k

A=A+i;

else

A = [A, 4+i];
end
% loop body continued

end

parfor i = 1:n

if A > 5*k

A=A+i;

else

A=A+i+5*k;
end
% loop body continued

end

Required (static): If the reduction assignment uses * or [,],thenin
every reduction assignment for

X, X must be consistently specified as the

first argument or consistently specified as the second.

The parfo r-loo p on the left below is not valid because the order of items in
the concatenation is not consistent throughout the loop. The

parfor-loop

on the right is valid:

parfor i = 1:n

if A > 5*k

A = [A, 4+i];

else

A = [r(i), A];

% loop body continued

end

parfor i = 1:n

if A > 5*k

A = [A, 4+i];
else

A = [A, r(i)];
% loop body continued

end

2-22

Further Considerations with Reduction Variables. This section provide
more detail about reduction assignments, associativity, commutativity, and
overloading of reduction functions.

Reduction Assignments. In addition to the specific forms of reduction
assignment listed in the table in “Reduction Variables” on page 2-20, the only
other (and more general) form of a reduction assignment is

X = f(X, expr) X = f(expr, X)

Advanced Topics

Required (static): f can be a function or a variable. If it is a variable, it
must not be affected by the

parfor body (in other words, it is a broadcast

variable).

If f is a variable, then for all practical purposes its value at run tim e is
a function handle. However, this is not strictly required; as long as the
right-hand side can b e evaluated, the resulting value is stored in

The

parfor-loop below on the left will not execute correctly because the

statement

f = @times causes f to be classified as a temporary variable and

therefore is cleared at the beg inning of each iteration. The
right is correct, because it does not assign to

f inside the loop:

X.

parfor on the

f = @(x,k)x * k;
parfor i = 1:n

a = f(a,i);
% loop body continued
f = @times; % Affects f

end

f = @(x,k)x * k;
parfor i = 1:n

a = f(a,i);
% loop body continued

end

Note that the op erators && and || are not listed in the table in “Reduction
Variables” on page 2-20. Except for
MATLAB have a corresponding function
to

f(u,v).For&& and ||, such a function cannot be written because u&&v

&& and ||, all the matrix operations of

f,suchthatuopvis equivalent

and u||v might or might not evaluate v,butf(u,v) always evaluates v
before calling f. This is why && and || areexcludedfromthetableofallowed
reduction assignments for a

Every reduction assignment has an associated function

f that ensure deterministic behavior of a parfor statement are discussed in

parfor-loop.

f. The properties of

the following sections.

Associativity in Reduction Assignments. Concerning the function

f as used in

the definition of a reduction variable, the following practice is recommended,
but does not generate an error if not adhered to. Therefore, it is up to you to
ensure that your code meets this recom mendation.

2-23

2 Parallel for-Loops (parfor)

Recommended: To get deterministic behavior of parfor-loops, the
reduction function

f must be associative.

To be associative, the function f must satisfy the following for all a, b,andc:

f(a,f(b,c)) = f(f(a,b),c)

The classification rules for variables, including reduction variables, are purely
syntactic. They cannot determine whether the

f you have supplied is truly

associative or not. If it is not, different executions of the loop might result in
different answers. In other w ords, although

parfor gives you the ability to

declare that a function is associative, MATLAB does not detect misuse of
that ability.

Note While the addition of mathematical real numbers is associative,
addition of floating-point numbers is only approximately asso ciativ e, and
different executions of this

parfor statement might produce values of X with

different round-off errors. This is an unavoidable cost of parallelism.

2-24

For e xample, the statement on the left yields 1, while the statement on the
right returns 1 +

(1 + eps/2) + eps/2 1 + (eps/2 + eps/2)

eps:

With the exception of the minus operator (-), all the special cases listed in the
table in “Reduction Variables” on page 2-20 have a corresponding (perhaps
approximately) associative function. MATLAB calculates the assignment

X=X-exprby using X = X + (-expr). (So, technically, the function for

calculating this reduction assignme nt is
assignment

X = expr - X cannot be written using an associative function,

plus,notminus.) However, the

which explains its exclusion from the table.

Commutativity in Reduction Assignments. Some associative functions,
including
That is, they satisfy the following for all

f(a,b) = f(b,a)

+, .*, min,andmax, intersect,andunion, are als o commutative.

a and b:

Advanced Topics

Examples of noncommutative functions are * (because matrix multiplication is
not commutative for matrices in which both dimensions have size greater than
one),

[,], [;], {,},and{;}. Noncommutativity is the reason that consistency

in the order of arguments to these functions is required. As a practical matter,
a more efficient algorithm is possible when a function is commutative as well
as associative, and

parfor is optimized to exploit commutativity.

Recommended: Except in the cas es of *, [,], [;], {,},and{;},the
function

f of a reduction assignment should be commutative. If f is not

commutative, different executions of the loop might result in different
answers.

Unless f is a known noncommutative built-in, it is assumed to be
commutative. There is currently no way to specify a user-defined,
noncommutative function in

parfor.

Overloading in Reduction Assignments. Most associative functions
identity eleme nt

f(e,a) = a = f(a,e)

e,sothatforanya, the following holds true:

f have an

Examples of identity elements for som efunctionsarelistedinthistable.

Function

+0

* and .*

min Inf

max -Inf

[,], [;],anduni on

Identity Element

1

[]

MATLAB uses the identity elements of reduction functions when it knows
them. So, in addition to associativity and commutativity, you should also keep
identity eleme nts in mind when overloading these functions.

2-25

2 Parallel for-Loops (parfor)

Recommended: An overload of +, * , .*, min, max, union, [,],or[;]
should be associative if it is usedinareductionassignmentinaparfor.
The overload must treat the respective identity element given above (all
with class

double) as an identity element.

Recommended: An overload of +, .*, min, max, union,orintersect
should be commutative.

There is no w ay to specify the identity element for a function. In these cases,
the b ehavior of

parfor is a little less efficient than it is for functions with a

known identity element, but the results are correct.

Similarly, because of the special treatment of

X=X-expr, the following

is recommended.

Recommended: An overload of the minus operator (-) should obey the
mathematical law that

X-(y + z) is equivalent to (X - y)-z.

Temporary Variables

A temporary variable is any variable that is the target of a direct, nonindexed
assignment, but is not a reduction variable. In the following
and d are temporary variables:

a=0;
z=0;
r = rand(1,10);
parfor i = 1:10

a = i; % Variable a is temporary
z=z+i;
if i <= 5

d = 2*a; % Variable d is temporary

end

end

In contrast to the behavior of a for-loop, MATLAB effectively clears any
temporary variables before each iteration of a

parfor-loop. To help ensure

the independence of iterations, the v alues of temporary variables cannot
be passed from one iteration of the loop to another. Therefore, temporary

parfor-loop, a

2-26

Advanced Topics

variables must be set inside the body of a parfor-loop, so that their values are
defined separately for each iteration.

MATLAB does not send temporary variables back to the client. A temporary
variable in the context of the

parfor statement has no effect on a variable

with the same name that exists outside the loop, again in contrast to ordinary

for-loops.

Uninitialized Temporaries. Because temporary variables are cleared at
the beginning of every iteration, MATLAB can detect certain cases in which
any iteration through the loop uses the temporary variable before it is set
in that iteration. In this case, MATLAB issues a static error rather than a
run-time error, because there is little point in allowing execution to proceed
if a run-time error is guaranteed to occur. T his kind of error often arises
because of confusion between

for and parfor, especially regarding the rules

of classification of variables. For example, suppose you write

b = true;
parfor i = 1:n

if b && some_condition(i )

do_something(i);

b = false;
end
...

end

This loop is acceptable as an ordinary for-loop, b ut as a parfor-loop, b is a
temporary variable because it occurs directly as the target of an assignment
inside the loop. Therefore it is cleared at the start of each iteration, so its use
in the condition of the

parfor to for,thevalueofb assumes sequential execution of the loop, so that
do_something(i) is executed for only the lower values of i until b is set
false.)

if is guaranteed to be uninitialized. (If you change

Temporary Variables Intended as Reduction Variables. Another
common cause of uninitialized temporaries can arise when you have a
variable that you intended to be a reduction variable, but you use it elsewhere
in the loop, causing it technically to be classified as a temporary variable.
For example:

2-27

2 Parallel for-Loops (parfor)

s=0;
parfor i = 1:n

s = s + f(i);
...
if (s > whatever)

...

end

end

If the only occurrences of s werethetwointhefirststatementofthebody,it
would be class if ied as a reductionvariable. Butinthisexample,

s is not a

reduction variable because it has a use outside of reduction assignments in
the line

s > whatever.Becauses isthetargetofanassignment(inthefirst

statement), it is a temporary, so MATLAB issues an error about this fact, but
points out the possible connection with reduction.

Note that if you change

parfor to for,theuseofs outside the reduction

assignment relies on the iterations being performed in a particular order. The
pointhereisthatina

parfor-loop, it matters that the loop “does not care”

about the value of a reduction variable as it goes along. It is only after the
loop that the reduction value becomes usable.

Improving Performance

Where to Create Arrays

With a parfor-loop, it might be faster to have each MATLAB worker create
its own arrays or portions of them in parallel, rather than to create a large
array in the client before the loop and send it out to all the wo rke rs separately.
Having each worker create its own copy of these arrays inside the loop saves
the time of transferring the data from client to workers, because all the
workers can be creating it at the same time. This might challenge your usual
practice to do as much variable initialization before a
that you do not needle ssly repeat it inside the loop.

Whether to create arrays before the

parfor-loop or inside the parfor-loop

depends on the size of the arrays, the time needed to create them, whether
the workers need all or part of the arrays, the number of loop iterations
that each worker performs, and other factors. While many

for-loop as possible, so

for-loops can be

2-28

Advanced Topics

directly converted to parfor-loops, even in these cases there might be other
issues involved in optimizing your code.

Optimizing on Local vs. Cluster Workers

With local workers, because all the MATLAB worker sessions are running
on the same machine, you might not see any performance improvement from
a

parfor-loop regarding execution time. This can depend on many factors,

including how many processors and cores your machine has. You might
experiment to see if it is faster to create the arrays before the loop (as shown
on the left below), rather than have each worker create its own arrays inside
theloop(asshownontheright).

Try the following examples running a matlabpool locally, and notice the
difference in time execution for each loop. First open a local matlabpool:

matlabpool

Then enter the following examples. (If you are viewing this documentation in
the MATLA B help brow ser, highlight each segment of code below, right-click,
and select Evaluate Selection in the context menu to execute the block in
MATLAB. That way the time measurement w ill not include the time required
to paste or type.)

tic;

n = 200;

M = magic(n);

R = rand(n);

parfori=1:n

A(i) = sum(M(i,:).*R(n+1-i,:));

end

toc

tic;

n = 200;

parfori=1:n

M = magic(n);

R = rand(n);

A(i) = sum(M(i,:).*R(n+1-i,:));

end

toc

Running on a remote cluster, you might find different behavior as workers
can simultaneously create their arrays, saving transfer time. Therefore, code
that is optimized for local workers might not be optimized for cluster workers,
and vice versa.

2-29

2 Parallel for-Loops (parfor)

2-30

Single Program Multiple
Data (spmd)

• “Using spmd Constructs” on page 3-2

• “Accessing Data with Composites” on page 3-7

• “Distributing Arrays” on page 3-12

• “Programming Considerations” on page 3-15

3

3 Single Program Multiple Data (spmd)

Using spmd Constructs

In this section...

“Introduction” on page 3-2

“WhentoUsespmd”onpage3-2

“Setting Up MATLAB Resources Using matlabpool” on page 3-3

“Defining an spmd Statement” on page 3-4

“Displaying Output” on page 3-6

Introduction

The single program multiple data (spmd) language construct allows seamless
interleaving of serial and parallel programming. The
you define a block of code to run simultaneously on multiple labs. Variab l es
assigned inside the
values from the client by reference via Composite objects.

spmd statement lets

spmd statement on the labs allow direct access to their

3-2

This chapter explains some of the characteristics of

Composite objects.

spmd statements and

When to Use spmd

The “single program” aspect of spmd means that the identical code runs on
multiple labs. You run one program in the MATLAB client, and those parts of
it labeled as
your program continues running in the client.

The “multiple data” aspect means that even though the
identical code on all labs, each lab can have different, unique data for that
code. So multiple data sets can be accommodated by multiple labs.

Typical applications appropriate for
simultaneous execution of a program on multiple data sets, when
communication or synchronization is required between the labs. Some
common cases are:

spmd blocks run on the labs. When the spmd block is complete,

spmd statement runs

spmd are those that require running

Using spmd Constructs

• Programs that take a long time to execute — spmd lets several labs compute

solutions simultaneously.

• Programs operating on large data sets —

spmd letsthedatabedistributed

to multiple labs.

Setting Up MATLAB Resources Using matlabpool

You use the function matlabpool to reserve a number of MATLAB labs
(workers) for executing a subsequent
Depending on your scheduler, the labs might be running remotely on a
cluster, or they might run locally on your MATLAB client machine. You
identify a scheduler and cluster by selecting a parallel con fi gu ration. For a
description of how to manage and use configurations, see “Programming with
User Configurations” on page 6-16.

To begin the examples of this section, allocate local MATLAB labs for the
evaluation of your

matlabpool

spmd statement:

This command starts the number of MATLAB worker sessions defined by
the default parallel configuration. If the local configuration is your default
and does not specify the number of workers, this starts one worker per core
(maximum of eight) on your local MATLAB client machine.

If you do not want to use default settings, you can specify in the
statement which configuration or how many labs to use. For example, to use
only three labs with y our default configuration, type:

spmd statement or parfor-loop.

matlabpool

matlabpool 3

To use a different configuration, type:

matlabpool MyConfigName

To inquire whether you currently have a MATLAB pool open, type:

matlabpool size

This command returns a value indicating the number of labs in the current
pool. If the command returns 0, there is currently no pool open.

3-3

3 Single Program Multiple Data (spmd)

Note If there is no MATLAB pool open, an spmd statement runs locally in the

MATLAB client without any parallel execution, provided you have Parallel
Computing Toolbox software installed. In other words, it runs in your client
session as though it were a sing le lab.

When you are finished using a MATLAB pool, close it with the comm and:

matlabpool close

Defining an spmd Statement

The general form of an spmd statement is:

spmd

<statements>

end

The block of code represented by <statements> executes in parallel
simultaneously on all lab s in the MATLAB pool. If you want to limit the
execution to only a portion of these labs, specify exactly how many labs to
run on:

3-4

spmd (n)

<statements>

end

This statement requires that n labs run the spmd code. n must be less than
or equal to the number of labs in the open MATLAB pool. If the pool is large
enough, but
are available. If

n labs are not available, the statement waits until enough labs

n is 0, the spmd statement uses no labs, and runs locally on

the client, the same as if there were not a pool currently open.

You can specify a range for the number of labs:

spmd (m, n)

<statements>

end

In this case, the spmd statement requires a minimum of m labs, and it uses
amaximumof

n labs.

Using spmd Constructs

If it is important to control the number of labs that execute your spmd
statement, set the exact number in the configuration or with the spmd
statement, rather than using a range.

For example, create a random matrix on three labs:

matlabpool
spmd (3)

R = rand(4,4);
end
matlabpool close

Note All subsequent examples in this chapter assume that a MATLAB pool is
open and remains open between sequences of

spmd statements.

Unlike a parfor-loop, the labs used for an spmd statement each have a unique
value for

labindex. This l ets you specify code to be run on only certain labs,

or to customize execution, usually for the purpose of accessing unique data.

For example, create d ifferent sized arrays depending on

spmd (3)

if labindex==1

R = rand(9,9);

else

R = rand(4,4);

end
end

labindex:

Load unique data on each lab according to labindex, and use the same
function on each lab to compute a result from the data:

spmd (3)

labdata = load(['datafile_' num2str(labindex) '.ascii'])

result = MyFunction(labdata)
end

The labs executing an spmd statement operate simultaneously and are
aware of each other. As with a parallel job, you are allowed to directly

3-5

3 Single Program Multiple Data (spmd)

control communications between the labs, transfer data between them, and
use codistributed arrays among them. For a list of toolbox functions that
facilitate these capabilities, see the Function Reference sections “Interlab
Communication Within a Parallel Job” on page 12-9 and “Distributed and
Codistributed Arrays” on page 12-3.

For example, use a codistributed array in an

spmd (3)

RR = rand(30, codistribut or());
end

Each lab has a 30-by-10 segment of the codistributed array RR.For
more information about codistributedarrays,seeChapter5,“Mathwith
Codistributed Arrays”.

spmd statement:

Displaying Output

When running an spmd statement on a MATLAB pool, all command-line
output from the workers displays in the client Command Window. Because
the workers are MATLAB sessions without displays, any graphical output (for
example, figure windows) from the pool does not display at all.

3-6

Accessing Data with Composites

In this section...

“Introduction” on page 3-7

“Creating Composites in spmd Statem ents” on page 3-7

“Variable Persistence and Sequences of spmd” on page 3-9

“Creating Composites Outside spmd Statements” on page 3-10

Introduction

Composite o bjects in the MATLAB client session let you directly acces s data
values on the labs. Most often you assigned these variables within
statements. In their display and usage, Composites resemble cell arrays.
There are two ways to create Composites:

Accessing Data with Composites

spmd

• Using the

Composite elements are stored on the labs.

• Defining variables on labs inside an

statement, the stored values are accessible on the client as Composites.

Composite function on the client. Values assigned to the

spmd statement. After the spmd

Creating Composites in spmd Statements

When you define or assign values to variables inside an spmd statement, the
data values are stored on the labs.

After the
Composites. Composite objects resem ble cell arrays, and behave similarly.
On the client, a Composite has one element per lab. For example, suppose
you open a MATLAB pool of three local w orkers and run an
on that pool:

matlabpool open local 3

spmd % Uses all 3 workers

end
MM{1} % In the client, MM is a Composite with one eleme nt per lab

spmd statement, those data values are accessible on the client as

spmd statement

MM = magic(labindex+2); % MM is a variable on each lab

816

3-7

3 Single Program Multiple Data (spmd)

MM{2}

162313

A variable might not be defined on every lab. For the labs on which a variable
is not defined, the corresponding Composite element has no value. Trying to
read that element throws an error.

spmd

if labindex > 1

end
end
HH

357
492

51110 8
97612
41415 1

HH = rand(4);

Lab1:Nodata
Lab 2: class = double, size = [4 4]
Lab 3: class = double, size = [4 4]

3-8

You can also set values of Composite elements from the client. This causes a
transfer of data, storing the value on the appropriate lab even though it is not
executed within an

MM{3} = eye(4);

spmd statement:

In this case, MM must already exist as a Composite, otherwise MATLAB
interprets it as a cell array.

Now when you do enter an

spmd statement, the value of the variable MM on

lab3isasset:

spmd

if labindex == 3, MM, end
end
Lab 3:

MM =

1000

Accessing Data with Composites

0100
0010
0001

Data transfers from lab to client when you explicitly assign a variable in the
client workspace using a Composite element:

M = MM{1} % Transfer data from lab 1 to variable M on the client

816
357
492

Assigning an entire Composite to another Composite does not cause a data
transfer. Instead, the client merely duplicates the Composite as a reference to
the appropriate data stored on the labs:

NN = MM % Set entire Composite equal to another, without transfer

However, accessing a Composite’s elements to assign values to other
Composites does result in a transfer of data from the labs to the client, even
if the assignment then goes to the same lab. In this case,

NN must already

exist as a Composite:

NN{1} = MM{1} % Transfer data to the client and then to lab

When finished, you can close the pool:

matlabpool close

Variable Persistence and Sequences of spmd

The values stored on the labs are retained between spmd statements. This
allows you to use multiple
thesamevariablesdefinedinprevious

The values are retained on the labs until the corresponding Composites are
cleared on the client, or until the MATLAB pool is closed. The following
example illustrates data value lifespan with
workers:

spmd statements in sequence, and continue to use

spmd blocks.

spmd blocks, using a pool of four

3-9

3 Single Program Multiple Data (spmd)

matlabpool open local 4

spmd

AA = labindex; % Initial setting
end
AA(:) % Composite

[1]

[2]

[3]

[4]
spmd

AA = AA * 2; % Multiply existing value
end
AA(:) % Composite

[2]

[4]

[6]

[8]
clear AA % Clearing in client also clears on labs

3-10

spmd; AA = AA * 2; end % Generates error

matlabpool close

Creating Composites Outside spmd Statements

The Composite function creates Composite objects without using an spmd
statement. This might be useful to prepopulate values of variables on labs
before an
pool is already open:

PP = Composite()

By default, this creates a Composite w ith an element for each lab in the
MATLAB pool. You can also create Composites on only a subset of the labs in
thepool.SeetheCompositereferencepageformoredetails. Theelementsof
the Composite can now be set as usual on the client, or as variables inside
an

spmd statement. When you set an element of a Composite, the data is

immediately transferred to the appropriate lab:

for ii = 1:numel(PP)

spmd statement begins executing on those labs. Assume a MATLAB

PP{ii} = ii;
end

Accessing Data with Composites

3-11

3 Single Program Multiple Data (spmd)

Distributing Arrays

In this section...

“Distributed Versus Codistributed Arrays” on page 3-12

“Creating Distributed Arrays” on page 3-12

“Creating Codistributed Arrays” on page 3-13

Distributed Versus Codistributed Arrays

You can create a distributed array in the MATLAB client, and its data
is stored on the labs of the open MATLAB pool. A distributed array is
distributed in one dimension, along the last nonsingleton dimension, and as
evenly as possible along that dimension among the labs. You cannot control
the details of distribution when creating a distributed array.

You can create a codistributed array by executing on the labs themselves,
either inside an
creating a codistributed array, you can control all as pects of distribution,
including dimensions and partitions.

spmd statement, in pmode, or inside a parallel job. When

3-12

The relationship between distributed and codistributed arrays is one of
perspective. Codistributed arrays are partitioned among the labs from which
you execute code to create or manipulate them. D istributed arrays are
partitioned among labs from the c l ie n t with the open MATLAB pool. When
you create a distributed array in the client, you can access it as a codistributed
array inside an
an

spmd statement, you can access is as a distributed array in the client.

Only

spmd statements let you access the same array data from two different

perspectives.

spmd statement. When you create a codistributed array in

Creating Distributed Arrays

You can create a distributed array in any of several ways:

• Use the

client workspace to the labs of an open MATLAB pool.

distributed function to distribute an existing array from the

Distributing Arrays

• Use any of the overloaded distributed object methods to directly construct a

distributed array on the labs. This technique does not require that the array
already exists in the client, thereby reducing client workspace memory
requirements. These overloaded functions include

distributed.rand, etc. For a full list, see the distributed object

distributed.eye,

reference page.

• Create a codistributed array inside an

distributed array outside the

spmd statement. This lets you use distribution

spmd statement, then access it as a

schemes other than the default.

The first two of these techniques do not involve
but you can see how

spmd might be used to manipulate arrays created this

spmd in creating the array,

way. For example:

Create an array in the client workspace, then make it a distributed array:

mablabpool open local 2
W = ones(6,6);
W = distributed(W); % Dist rib ute to the labs
spmd

T = W*2; % Calculation performed on labs, in parallel.

% T and W are both codistributed arrays here.
end
T % V iew results in client.

% T and W are both distributed arrays here.

matlabpool close

Creating Codistributed Arrays

You can create a codistributed array in any of several ways:

• Use the

codistributed function inside an spmd statement, a parallel job,

or pmode to codistribute data already existing on the labs running that job.

• Useanyoftheoverloadedcodistributedobject methods to directly construct

a codistributed array on the labs. This technique does not require that
the array already exists in the labs. These overloaded functions include

codistributed.eye, codistributed.rand, etc. For a full list, see the
codistributed object reference page.

3-13

3 Single Program Multiple Data (spmd)

• Create a distributed array outside an spmd statement, then access it as

a codistributed array inside the
MATLAB pool.

spmd statement running on the same

In this example, you create a codistributed array inside an

spmd statement,

using a nondefault distribution scheme. First, define 1-D distribution along
the third dimension, with 4 parts on lab 1, and 12 parts on lab 2. Then create
a3-by-3-by-16arrayofzeros.

matlabpool open local 2
spmd

codist = codistributor1d(3, [4, 12]);
Z = codistributed.zeros(3, 3, 16, co dist );

Z = Z + labindex;
end
Z % V iew results in client.

% Z is a distributed array here.

matlabpool close

For more details on codistributed arrays, see Chapter 5, “Math with
Codistributed Arrays”, and Chap ter 4, “Interactive Parallel Computation
with pmode”.

3-14

Programming Considerations

In this section...

“MATLAB Path” on page 3-15

“Error Handling” on page 3-15

“Limitations” on p age 3-15

MATLAB Path

All labs executing an spmd statement must have the same MATLAB path
configuration as the client, so that they can execute any functions called in
their common blo ck of code. Therefore, whenever you use

rmpath on the client, it also executes on all the labs, if possible. For more

information, see the
on a different platform than the client, use the function
properly set the MATLAB path on all labs.

matlabpool reference page. When the labs are running

Programming Considerations

cd, addpath,or

pctRunOnAll to

Error Handling

When an error occurs on a lab during the execution of an spmd statement, the
error is reported to the client. The client tries to interrupt execution on all
labs, and throws an error to the user.

Errors and warnings produced on labs are annotated with the lab ID and
displayed in the client’s Command Window in the order in which they are
received by the MATLAB client.

The behavior of
its body.

lastwarn is unspecified at the end of an spmd if used within

Limitations

Transparency

The body of an spmd statement must be transparent,meaningthatall
references to variables must be “visible” (i.e., they occur in the text of the
program).

3-15

3 Single Program Multiple Data (spmd)

In the following example, because X is not visible as an input variable in the

spmd body (only the string 'X' is passed to eval), it does not get transferred to

the l abs. As a result, MATLAB issues an error at run time:

X=5;
spmd

eval('X');
end

Similarly, you cannot clear variables from a worker’s workspace by executing

clear inside an spmd statement:

spmd; clear('X'); end

To clear a specific variable from a worker, clear its Composite from the client
workspace. Alternatively , you can free up most of the memory used by a
variable by setting its value to empty, presumably when it is no longer needed
in your

spmd statement:

spmd

<statements....>

X=[];
end

3-16

Examples of some other functions that violate transparency are evalc,

evalin,andassignin with the workspace argument specified as 'caller';
save and load, unless the output of load is assigned.

MATLAB does successfully execute
functions called from the

spmd body.

eval and evalc statements that appear in

Nested Functions

Inside a function, the body o f an spmd statement cannot make any direct
reference to a nested function. However, it can call a nested function by
means of a variable defined as a function handle to the nested function.

Because the
nested functions called inside an spmd statement do not get updated in the
workspace of the outer function.

spmd body executes on workers, variables that are updated by

Programming Considerations

Anonymous Functions

The body of an spmd statement cannot define an anonymous function.
However, it can reference an anonymous function by means of a function
handle.

Nested spmd Statements

The body of an spmd statement cannot contain another spmd. However, it
can call a function that contains another
MATLAB pool has enough labs to accommodate such expansion.

spmd statement. Be sure that your

Nested parfor-Loops

The body of a parfor-loop cannot contain an spmd statement, and an spmd
statementcannotcontainaparfor-loop.

Break and Return Statements

The body of an spmd statement cannot contain break or return statements.

Global and Persistent Variables

The body of an spmd statement cannot con tain global or persistent variable
declarations.

3-17

3 Single Program Multiple Data (spmd)

3-18

Interactive Parallel
Computation with pmode

This chapter describes interactive pmode in the following sections:

• “Introduction” on page 4-2

• “Getting Started with pmode” on page 4-3

• “Parallel Command W indow” on page 4-10

• “Running pmode on a Cluster” on page 4-15

4

• “Plotting in pmode” on page 4-16

• “Limitations and Unexpected Results” on page 4-18

• “Troubleshooting” on page 4-19

4 Interactive Parallel Computation with pmode

Introduction

pmode lets you work interactively with a parallel job running simultaneously
on several lab s. Commands you type at the pmode prompt in th e Parallel
Command Window are executed on all labs at the same time. Each lab
executes the commands in its own workspace on its own variables.

The way the labs remain synchronized is that each lab becomes idle when it
completes a command or statement, waiting until all the labs working on this
job have completed the same statement. Only when all the labs are idle, do
they then proceed together to the next pmode command.

In contrast to
running the job, where you can enter commands, see results, access each lab’s
workspace, etc. What pmode does not let you do is to freely interleave serial
and parallel work, like
job is effectively destroyed, and all information and data on the labs is lost.
Starting another pmode session always begins from a clean state.

spmd, pmode provides a desktop with a display for each lab

spmd does. When you exit your pmode session, its

4-2

Getting Started with pmode

This exam ple uses a local scheduler and runs the labs on your local MATLAB
client machine. It does not require an external cluster or scheduler. The
steps include the pmode prompt (
Parallel Command Window.

1 Start the pmode with the pmode command.

pmode start local 4

This starts four local labs, creates a parallel job to run on those labs, and
opens the Parallel Command Window.

Getting Started with pmode

P>>) for commands that you type in the

You can control where the command history appears. F or this exercise , the
position is set by clicking Window > History Position > Above Prompt,
but you can set it according to your own preference.

2 To illustrate that commands at the pmode prompt are executed on all labs,

ask for help on a function.

P>> help magic

4-3

4 Interactive Parallel Computation with pmode

3 Setavariableatthepmodeprompt. Noticethatthevalueissetonall

the labs.

P>>x=pi

4-4

4 A variable does not necessarily have the same value on every lab. The

labindex function returns the ID particular to each lab working on this

parallel job. In this example, the variable

x exists with a different value in

the workspace of each lab.

P>> x = labindex

5 Return the total number of labs working on the current parallel job with

the

numlabs function.

P>> all = numlabs

6 Create a replicate d array on all the labs.

P>> segment = [1 2; 3 4; 5 6 ]

Getting Started with pmode

4-5

4 Interactive Parallel Computation with pmode

7 Assign a unique value to the array on each lab, dependent on the lab

number. With a different value on each lab, this is a variant array.

P>> segment = segment + 10*labindex

4-6

8 Until this point in the example, the variant arrays are independent, other

than having the same name. Use the

codistributed.build function to

aggregate the array segments into a coherent array, distributed among
the labs.

P>> codist = codistributo r1d(2, [2 2 2 2],
P>> whole = codistributed. build(segme

This co
array.
along i
labs.

whole

9 Now, when you operate on the codistributed array whole, each lab handles

mbines four separate 3-by-2 arrays into one 3-by-8 codistributed

The

codistributor1d object indicates that the array is distributed

tsseconddimension(columns),with2columnsoneachofthefour

On each lab,

segment provided the data for the local portion of the

array.

nt, codist)

[3 8])

the calculations on only its portion, or segment, of the array, not the whole
array.

P>> whole = whole + 1000

Getting Started with pmode

10 Although the codistributed array allows for operations on its entirety, you

can use the

getLocalPart function to access the portion of a codistributed

array on a particular lab.

P>> section = getLocalPar t(whole)

Thus, section is now a variant array because it is different on e ach lab.

11 If you n

Notic
all th
arra

12 Because the labs ordinarily do not have displays, if you want to perform

eed the entire array in one workspace, use the

P>> combined = gather(who le)

e, ho we ver, that this gathers the entire array into the workspaces of
elabs. Seethe

gather referencepageforthesyntaxtogatherthe

y into the workspace of only one lab.

gather function.

any graphical tasks involving your data, such as plotting, you must do this
from the client workspace. Copy the array to the client workspace by typing
the following commands in the MATLAB (client) Command Window.

pmode lab2client combined 1

4-7

4 Interactive Parallel Computation with pmode

Notice that combined is now a 3-by-8 array in t he client workspace.

whos combined

To see the array, type its name.

combined

13 Many matrix functions that might be familiar can o perate on codistributed

arrays. For example, the
can create a codistributed identity matrix with the following commands
in the Parallel Command Window.

P>> distobj = codistribut or1d();
P>> I = eye(6, distobj)
P>> getLocalPart(I)

Calling the codistributor1d function without arguments specifies the
default distribution, which is by columns in this case, distributed as evenly
as possible.

eye function creates an identity matrix. Now you

4-8

Getting Started with pmode

14 If you require distribution along a different dimension , you can use

the

redistribute function. In this example, the argument 1 to

codistributor1d specifies distribution of the array along the first

dimension (rows).

P>> distobj = codistribut or1d(1);
P>> I = redistribute(I, distobj)
P>> getLocalPart(I)

15 Exit pmode and return to the regular MATLAB desktop.

P>> pmode exit

4-9

4 Interactive Parallel Computation with pmode

Parallel Command Window

When you start pmode on your local client machine with the command

pmode start local 4

four labs start on your local machine and a parallel job is created to run on
them. The first time you run pmo de with this conf iguration, you get a tiled
display of the four labs.

Clear all output
windows

Show commands
in lab output

Lab outputs
in tiled
arrangement

4-10

Command
history

Command
line

Parallel Command Window

The Parallel Command Window offers much of the same functionality as the
MATLAB desktop, including command line, output, and command history.

When you select one or more lines in the command history and right-click,
you see the following context menu.

You have several options for how to arrange the tiles showing your lab
outputs. Usually, you will choose an arrangement that depends on the format
of your data. For example, the data displayed until this point in this section,
as in the previous figure, is distributed by columns. It might be convenient to
arrange the tiles side by side.

Click tiling icon

Select layout

4-11

4 Interactive Parallel Computation with pmode

This arrangement results in the following figure, which might be more
convenient for viewing data distributed by columns.

Alternatively, if the data is distributed by rows, you might want to stack the
lab tiles vertically. For the following figure, the data is reformatted with
the command

4-12

P>> distobj = codistribut or('1d',1);
P>> I = redistribute(I, distobj)

When you rearrange the tiles, you see the following.

Select vertical
arrangement

Drag to adjust
tile sizes

Parallel Command Window

You can control the relative positions of the command window and the lab
output. The following figure shows how to set the output to display beside the
input, rather than above it.

You can choose to view the lab outputs by tabs.

1. Select tabbed
display

3. Select labs
shown in
this tab

2. Select tab

4-13

4 Interactive Parallel Computation with pmode

You can have multiple labs send their output to the same tile or tab. This
allows you to have fewer tiles or tabs than labs.

In this case, the window provides shading to help distinguish the ou tpu ts
from the various labs.

Click tabbed output

Select only two tabs

4-14

Multiple labs
in same tab

Running pmode on a Cluster

When you run pmode on a cluster of labs, you are running a job that is
much like any other parallel job, except it is interactive. The cluster
can be heterogeneous, but with certain limitations described at

http://www.mathworks.com/products/parallel-computing/requirements.html;

carefully locate your scheduler on that page and note that pmode
sessions run as jobs described as “parallel applications that use
inter-worker communication.”

Many of the job’s properties are determined by a configuration. For more
details about creating and using configurations, see “Programming with User
Configurations” on page 6-16.

The general form of the command to start a pmode session is

pmode start <config-name> <num-labs>

where <config-name> is the name of the configuration you want to use,
and

<num-labs> is the number of labs you want to run the pmode job

on. If
configuration. Coordinate with your system administrator when creating or
using a configuration.

<num-labs> is omitted, the number of labs is determined by the

Running pmode on a Cluster

If you omit

defaultParallelConfig reference page).

<config-name>, pmode uses the default configuration (see the

For details on all the command options, see the

pmode reference page.

4-15

4 Interactive Parallel Computation with pmode

Plotting in pmode

Because the labs running a job in pmode are MATLAB sessions without
displays, they cannot create plots or other graphic outputs on your desktop.

When working in pmode with codistributed arrays, one way to plot a
codistributed array is to follow these basic steps:

1 Use the gather function to collect the entire array into the workspace of

one lab.

2 Transfer the whole array from any lab to the MATLAB client with pmode

lab2client

.

3 Plot the dat

The follow

Create a 1
1-by-25 s

P>> D = zeros(1,100,codis tributor1d())

Lab 1: This lab stores D(1:25).
Lab 2: This lab stores D(26:50).
Lab 3: This lab stores D(51:75).
Lab 4: This lab stores D(76:100).

Use a f
conta

Gath
lab

or

ins a sine wave. Each lab does one-fourth of the array.

P>> for i = drange(1:100)
D(i) = sin(i*2*pi/100);
end;

erthearraysothatthewholearrayiscontainedintheworkspaceof

1.

P>> P = gather(D, 1);

a from the client workspace.

ing example illustrates this technique.

-by-100 codistributed array o f 0s. With four labs, each lab has a

egment of the whole array.

-loopoverthedistributedrangeto populate the array so that it

4-16

Plotting in pmode

Transfer the array from the workspace of lab 1 to the M ATLA B client
workspace, then plot the array from the client. Note that both commands are
entered in the MATLAB (client) Command Window.

pmode lab2client P 1
plot(P)

This is not the only way to plot codistributed data. One alternative method,
especially useful when running noninteractive parallel jobs, is to plot the data
to a file, then view it from a later MATLAB session.

4-17

4 Interactive Parallel Computation with pmode

Limitations and Unexpected Results

Using Graphics in pmode

Displaying a GUI

The labs that run the tasks of a parallel job are MATLAB sessions without
displays. As a result, these labs cannot display graphical tools and so you
cannot do things like plotting from within pmode. The general approach to
accomplish something graphical is to transfer the data into the workspace
of the MATLAB client using

pmode lab2client var lab

Then use the graphical tool on the MATLAB client.

Using Simulink Software

Because the labs running a pmode job do not have displays, you cannot use
Simulink
from within pmode. If you type
Library Browser opens in the background on the labs and is not visible.

®

software to edit diagrams or to perform interactive simulation

simulink at the p m ode pro mpt, th e Simulink

4-18

You can use the
parallel. If you edit your model in the MATLAB client outside of pmode, you
must save the model before accessing it in the labs via pmode; also, if the
labs had accessed the model previously, they must close and open the model
again to see the latest saved changes.

sim command to perform noninteractive simulations in

Troubleshooting

In this section...

“Connectivity Testing” on page 4-19

“Hostname Resolution” on page 4-19

“Socket Connections” on page 4-19

Connectivity Testing

For testing connectivity between the client machine and the machines of
your compute cluster, you can use Admin Center. For more information
about Admin Center, including how to start it and how to test connectivity,
see “Admin Center” in the MATLAB Distributed Computing Server
documentation.

Hostname Resolution

If a lab cannot resolve the hostname of the computer running the MATLAB
client, use
advertises itself.

Troubleshooting

pctconfig to change the hostname by which the client machine

Socket Connections

If a lab cannot open a socket connection to the MATLAB client, try the
following:

• Use

• Make sure that firewalls are not preventing communication between the

• Use

pctconfig to change the hostname by which the client machine

advertises itself.

lab and client machines.

pctconfig to change the client’s pmodeport property. This determines

the port that the labs will use to contact the client in the next pmode
session.

4-19

4 Interactive Parallel Computation with pmode

4-20

Math with Codistributed
Arrays

This chapter describes the distribution or partition of data across several labs,
and the functionality provided for operations on that data in
parallel jobs, and pmode. The sections are as follo ws.

• “Array Types” on page 5-2

• “Working with Codistributed Arrays” on p age 5-5

spmd statements,

5

• “Using a for-Loop Over a Distributed Range (for-drange)” on page 5-21

• “Using MATLAB Functions on Codistributed Arrays” on page 5-25

5 Math with Codistributed Arrays

Array Types

In this section...

“Introduction” on page 5-2

“Nondistributed Arrays” on page 5-2

“Codistributed Arrays” on page 5-4

Introduction

All built-in data types and data structures supported by MATLAB software
are also supported in the MATLAB parallel computing environment. This
includes arrays of any number of dimensions containing numeric, character,
logical values, cells, or structures; but not function handles or user-defined
objects. In addition to these basic building blocks, the MATLAB parallel
computing environment also offers different types of arrays.

Nondistributed Arrays

When you create a nondistributed array, MATLAB constructs a separate array
in the workspace of each lab and assigns a common variable to them. Any
operation performed on that variable affects all individual arrays assigned
to it. If you display from lab 1 the value assigned to this variable, all labs
respond by showing the array of that name that resides in their workspace.

5-2

The state of a nondistributed array depends on the value of that array in
the workspace of each lab:

• “Replicated Arrays” on page 5-2

• “Variant Arrays” on page 5-3

• “Private Arrays” on page 5-4

Replicated Arrays

A replicated array resides in the workspaces of all labs, and its size and
content are identical on all labs. When you create the array, MATLAB assigns
ittothesamevariableonalllabs. Ifyoudisplayinspmdthevalueassigned
to this variable, all labs respond by showing the same array.

spmd, A = magic(3), end

LAB 1 LAB 2 LAB 3 LAB 4

|||
816|816|816|816
357|357|357|357
492|492|492|492

Variant Arrays

A variant array also resides in the workspaces of all labs, but its content
differs on one or more labs. When you create the array, MATLAB assigns
a different value to the same variable on all labs. If you display the value
assigned to this variable, all labs respond by showing their version of the
array.

spmd, A = magic(3) + labindex - 1, end

LAB 1 LAB 2 LAB 3 LAB 4

|||
816|927|1038|1149
3 5 7|4 6 9|5 7 9|6 810
4 9 2|510 3|611 4|712 5

Array Types

A replicated array can become a variant array when its value becomes unique
on each lab.

spmd

B = magic(3); %replicated on all labs
B = B + labindex; %now a variant array, different on each lab

end

5-3

5 Math with Codistributed Arrays

Private Arrays

A private array is defined on one or more, but not all labs. You could create
this array by using the lab index in a conditional statement, as shown here:

spmd

end

undefined | undefined | 5 7 9 | 6 8 10

Codistributed Arrays

With replicated and variant arrays, the full content of the array is stored
in the workspace of each lab. Codistributed arrays,ontheotherhand,are
partitioned into segments, with each segment residing in the workspace of a
different lab. Each lab has its own array segment to work w ith. Reducing the
size of the array that each lab has to store and process means a more efficient
use of memory and faster processing, especially for large data sets.

if labindex >= 3, A = magic(3) + labindex - 1, end

LAB 1 LAB 2 LAB 3 LAB 4

|||

Ais|Ais|1038|1149

|6114|7125

5-4

This example distributes a 3-by-10 re plicated array A over four labs. The
resulting array D is also 3-by-10 in size, but only a segment of the full array
resides on each lab.

spmd

A = [11:20; 21:30; 31:40];
D = codistributed(A);
getLocalPart(D)

end

LAB 1 LAB 2 LAB 3 LAB 4

|||
111213|141516|1718|1920
212223|242526|2728|2930
313233|343536|3738|3940

For more details on using codistributed arrays, see “Working with
Codistributed Arrays” on page 5-5.

Working with Codistributed Array s

In this section...

“How MATLAB Software Distributes Arrays” on page 5-5

“Creating a Codistributed Array” on page 5-7

“Local Arrays” on page 5-11

“Obtaining information About the Array” on page 5-12

“Changing the Dimension of Distribution” on page 5-13

“Restoring the Full Array” on page 5-14

“Indexing into a Codistributed Array” on page 5-15

“2-Dimensional Distribution” on page 5-17

How MATLAB Software Distributes Arrays

Whenyoudistributeanarraytoanumberoflabs,MATLABsoftware
partitions the array into segments and assigns one segment of the array to
each lab. You can partition a two-dimensional array horizontally, assigning
columns of the original array to the different labs, or vertically, by assigning
rows. A n array with N dimensions can be partitioned along any of its N
dimensions. You choose w hich dimension of the array is to be partitioned by
specifying it in the array constructor command.

Working with Codistributed Arrays

For example, to distribute an 80-by-1000 array to four labs, you can partition
it either by columns, giving each lab an 80-by-250 segment, or by rows, with
each lab getting a 20-by-1000 segment. If the array dimension does not divide
evenly over the number of labs, MATLAB partitions it as evenly as possible.

The following example creates an 80-by-1000 replicated array and assigns
it to variable
workspace and assigns it to variable
command distributes
four labs. lab 1 stores columns 1 through 250, lab 2 stores columns 251
through 500, and so on. The default distribution is by the last nonsingleton
dimension, thus, columns in this case of a 2-dimensional array.

A. In doing so, each lab creates an identical array in its own

A,whereA is local to that lab. T he second

A, creating a single 80-by-1000 array D that spans all

5-5

5 Math with Codistributed Arrays

spmd

end

Each lab has access to all segments of the array. Access to the local segment
is faster than to a remote segment, because the latter requires sending and
receiving data between labs and thus takes more time.

How MATLAB Displays a Codistributed Array

For each lab, the MATLAB Parallel Command Window displays information
about the codistributed array, the local portion, and the codistributor. For
example, an 8-by-8 identity matrix codistributed among four labs, with two
columns on each lab, displays like this:

A = zeros(80, 1000);
D = codistributed(A);

Lab 1: This lab stores D(:,1:250).
Lab 2: This lab stores D(:,251:500).
Lab 3: This lab stores D(:,501:750).
Lab 4: This lab stores D(:,751:1000).

5-6

>> spmd
II = codistributed.eye(8)
end
Lab 1:

This lab stores II(:,1:2).

LocalPart: [8x2 double]

Codistributor: [1x1 codistributor1d]

Lab 2:

This lab stores II(:,3:4).

LocalPart: [8x2 double]

Codistributor: [1x1 codistributor1d]

Lab 3:

This lab stores II(:,5:6).

LocalPart: [8x2 double]

Codistributor: [1x1 codistributor1d]

Lab 4:

This lab stores II(:,7:8).

LocalPart: [8x2 double]

Codistributor: [1x1 codistributor1d]