CUDALink on Multiple Devices

Introduction	Memory
CUDALink Functions	Bandwidth
CUDALink Programming

The functional and list-oriented characteristics of the core Wolfram Language allow CUDALink to provide immediate built-in data parallelism, automatically distributing computations across available GPU cards.

Introduction

First, load the CUDALink application.

In[6]:=6

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-nldpuu

This launches as many worker kernels as there are devices.

In[2]:=2

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-i7imm

Out[2]=2

$CUDADeviceCount is the number of the devices on the system.

$CUDADeviceCount

number of CUDA devices on system

$CUDADeviceCount gets the number of CUDA GPUs on the system.

This loads CUDALink on all worker kernels.

In[3]:=3

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-b0ffin

CUDALink relies on existing Wolfram Language parallel computing capabilities to run on multiple GPUs. Throughout this section the following functions will be used.

ParallelNeeds	load a package into all parallel subkernels
DistributeDefinitions	distribute definitions needed for parallel computations
ParallelEvaluate	evaluate the input expression on all available parallel kernels and return the list of results obtained

CUDALink relies on the Wolfram Language's parallel computing capabilities to use multiple GPUs.

This sets the $CUDADevice variable on all kernels.

In[4]:=4

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-cykvw0

Out[4]=4

CUDALink Functions

High-level CUDALink functions like the image processing, linear algebra, and fast Fourier transforms can be used on different kernels like any other Wolfram Language function. The only difference is that the $CUDADevice variable is set to the device on which computation is performed.

Here you set image names to be taken from the TestImages dataset for ExampleData.

In[5]:=5

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-cfb2ma

Distribute the variable imgNames with the worker kernels.

In[6]:=6

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-ehiijj

Out[6]=6

Perform CUDAErosion on images taken from ExampleData.

In[7]:=7

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-e8zffx

Out[7]=7

Notice the 2x speed improvement. Since these images are small and data must be transferred, you do not get the 4x performance speedup.

In[8]:=8

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-k8viod

Out[8]=8

In other cases, the amount of time spent transferring the data is not as significant as the amount of time spent in calculation. Here, you allocate 2000 random integer vectors.

In[9]:=9

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-lu78hw

Map CUDAFold on each device.

In[10]:=10

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-4j5u1

Out[10]=10

Notice that there is now a better speedup.

In[11]:=11

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-qemk7

Out[11]=11

CUDALink Programming

Since a CUDAFunction is optimized and local to one GPU, it cannot be shared with worker kernels using DistributeDefinitions. This section describes an alternative way of programming the GPU.

Add Two

This loads a basic CUDA code that adds 2 to a vector.

In[12]:=12

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-tk2q1q

This loads the CUDAFunction. Notice the use of SetDelayed in the assignment. This allows DistributeDefinitions to distribute all dependent variables in the CUDAFunctionLoad call.

In[13]:=13

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-o49o6j

This sets the input parameters.

In[14]:=14

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-3q5m6

This distributes the definitions between worker kernels.

In[17]:=17

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-ghszwv

Out[17]=17

This runs the CUDAFunction on each worker kernel using different CUDA devices.

In[18]:=18

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-maik9c

Out[18]=18

This gathers the result showing the first 20 elements.

In[19]:=19

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-jjmntx

Out[19]=19

Mandelbrot Set

This is the same CUDA code defined in other sections of the CUDALink documentation.

In[7]:=7

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-d3qll1

Here, you load the CUDAFunction.

In[8]:=8

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-dzfw9v

In[9]:=9

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-b1ynen

This shares the variables with worker kernels.

In[10]:=10

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-dafk38

Out[10]=10

This launches the kernel, each with a different zoom level, returning the "Bit" image.

In[24]:=24

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-eu0fsr

Out[24]=24

Random Number Generators

The Mersenne Twister is implemented in the following file.

In[25]:=25

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-d84o29

Out[25]=25

This loads the function into the Wolfram Language.

In[26]:=26

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-u3efc

This sets the input variables for the CUDAFunction.

In[27]:=27

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-dscs93

This distributes the mersenneTwister function and input parameters.

In[32]:=32

✖

https://wolfram.com/xid/0k1j0pobq27zyibwarns7oq-hda44b

Out[32]=32

This allocates the seed values—note that seed evaluation needs to be performed on each worker kernel so that the random numbers are not correlated. The output memory is also allocated, computation is performed, and the result is visualized.

In[33]:=33