The Graphics Card

Thousands of Small Cores

A graphics card is a machine of its own beside the processor: thousands of small cores that run one routine for every element of a large range at once, with memory of their own or, on many laptops, the processor's memory shared. What it runs well is exactly what the compiler already recognises in a for (Loops on All Cores): rounds that read what they are given, compute, and put one result into a slot of their own.

So a program does not say "run this on the graphics card". It writes the loop. Here is the program of A Plain for once more, with one change: it computes in f32 and i32, the numbers every card has, where the original computed in double and int.

function row_count(y: i32, n: i32) returns i32
{
    let ci (2.0 * (y as f32) / (n as f32)) - 1.0
    var count: i32 0
    var x: i32 0
    while x < n
    {
        let cr (2.0 * (x as f32) / (n as f32)) - 1.5
        var zr: f32 0.0
        var zi: f32 0.0
        var i: i32 0
        var inside true
        while i < 50 and inside
        {
            let tr zr * zr - zi * zi + cr
            zi := 2.0 * zr * zi + ci
            zr := tr
            if zr * zr + zi * zi > 4.0
                inside := false
            i := i + 1
        }
        if inside
            count := count + 1
        x := x + 1
    }
    count
}

let n: i32 2000
var counts Vector[i32](n as size_t)
for y in n
    counts.put(y as size_t, row_count(y, n))

var total 0
for c in counts
    total := total + c
print "mandelbrot `n`: `total` points inside"

$ scaly run mandelbrot.scaly
mandelbrot 2000: 1588044 points inside
$ scalyc --gpu-plan mandelbrot.scaly
gpu-plan: 2 for-loops, 2 parallel, 1 kernels
gpu-plan: mandelbrot.scaly:31:1: kernel (metal): out counts; in n; calls row_count; not webgpu: floating point, and the loop is not relaxed
gpu-plan: mandelbrot.scaly:35:1: no kernel: iterates a collection
$ scaly build --gpu mandelbrot.scaly -o mandelbrot
$ ./mandelbrot
mandelbrot 2000: 1588044 points inside
$ SCALY_GPU=on ./mandelbrot
mandelbrot 2000: 1588044 points inside
$ SCALY_GPU=off ./mandelbrot
mandelbrot 2000: 1588044 points inside

--gpu-plan says for every parallel for whether a card could run it: the loop over the rows is a kernel — a routine a card runs once per round —, with the vector it writes, the values it reads from around it, and the functions it calls. A program built with --gpu carries each of its kernels along, and decides while it runs, loop by loop, whether the card or the processor's cores compute it (Where a Loop Runs). SCALY_GPU takes the decision away from it: on sends every kernel to the card, off none.

All three runs print the same count, and that is a promise, not luck: a loop goes to the card only if the program cannot tell where it ran (The Same Numbers). The count is not the 1588053 of the chapter on loops — an f32 rounds earlier than a double, and on the edge of the set that moves nine points to the other side —, but it is the count of this program wherever it runs.

Built with scaly build --release --gpu and n at 8000, the program took 4.0 seconds on one core, 0.52 on all ten, and 0.06 where the card counted the rows — 0.02 of them for the rows themselves.

The card is reached through Metal, on a Mac, and — for the loops that allow it — through WebGPU (The Numbers a Card Computes). On a system without either, --gpu builds the program without its kernels, and it runs on the processor's cores as it always did.

What a Card Can Run

A card has no pages, no strings and no stack to speak of. It has numbers, arrays of numbers, and one routine. A parallel for is a kernel when its rounds get by with that:

• they compute with bool, i32, u32 and f32 — and with 64-bit integers on a card that has them (The Numbers a Card Computes);

• they read numbers bound outside the loop, and elements of a Vector, Slice or Array of such numbers, at any index;

• they write with put, at exactly the loop's counter, into a Vector of such numbers, add to a sum, a product or a largest of such numbers (Sums, Products and the Largest), and write nothing else;

• the functions they call take and answer such numbers, do not call themselves, and are made of the same stuff;

• nothing in them builds a record or a string, and nothing leaves the loop early.

Inside a round almost everything else is allowed: bindings, if, while and for, a break out of an inner loop. A loop that is no kernel is not an error. It runs on the cores, and --gpu-plan names what kept it from the card:

function scaled(v: i32, k: i32) returns i32
    v * k + 1

function caption(i: int) returns String
    "row `i`"

let n 1000
let k: i32 3
var src Vector[i32](n as size_t)
for i in n
    src.put(i as size_t, (i as i32) % 17)

var out Vector[i32](n as size_t)
for i in n
    out.put(i as size_t, scaled(src[i as size_t], k))

var wide Vector[int](n as size_t)
for i in n
    wide.put(i as size_t, i * i)

var halves Vector[double](n as size_t)
for i in n
    halves.put(i as size_t, (i as double) * 0.5)

var names Vector[String](n as size_t)
for i in n
    names.put(i as size_t, caption(i))

var sum: i32 0
for i in n
    sum := sum + out[i as size_t]
print "`out[999]` `wide[999]` `halves[999]` `names[999]` `sum`"

$ scaly run kernels.scaly
40 998001 499.5 row 999 24937
$ scalyc --gpu-plan kernels.scaly
gpu-plan: 6 for-loops, 6 parallel, 4 kernels
gpu-plan: kernels.scaly:10:1: kernel (metal, webgpu): out src
gpu-plan: kernels.scaly:14:1: kernel (metal, webgpu): out out; in k, src; calls scaled
gpu-plan: kernels.scaly:18:1: kernel (metal): out wide; not webgpu: the counter i, used outside a cast, is a 64-bit integer
gpu-plan: kernels.scaly:22:1: no kernel: halves is a Vector[double]
gpu-plan: kernels.scaly:26:1: no kernel: names is a Vector[String]
gpu-plan: kernels.scaly:30:1: kernel (metal, webgpu): reduces sum; in out

All six loops run on all cores; four of them could run on a card. The first two compute in 32 bits — the counter i is an int, but it is cast or used as an index on the spot, and the number of rounds fits into 32 bits. The third squares the counter itself, in 64 bits. The fourth wants double, which no card of today's backends has; the fifth builds a string in every round. The last is a sum, and a kernel all the same: the card computes what each round adds, and the adding is done where the order of it can be kept (The Same Numbers).

The Same Numbers

A loop that prints one result on the processor and another on the card would make every run a question of where it happened to run. So the rule is the one that already holds between one core and ten: the program cannot tell. For whole numbers every card keeps it by itself. Floating point takes care on both sides, and the compiler takes it: it writes the kernel one operation at a time, in the order the processor computes them, and tells the card to round every one of them as the processor does — no multiplication fused with the addition after it, no sum reordered, no faster and coarser division or square root. What comes back is equal to the processor's result bit for bit, not merely close to it.

The other half of the promise is what happens when a round goes wrong. On the processor an index outside its vector stops the program with exit code 15, and a division by zero stops it too. A card cannot stop a program. Its round raises a flag instead and computes nothing; when the card is done and the flag is up, all of its results are thrown away, and the loop runs on the processor — which stops in the round it always stopped in, with the exit code it always had. Nothing a card computed is ever kept from a loop the processor would not have finished.

A sum is where a card would most naturally break the promise. Thousands of small cores add up thousands of numbers as a tree, in whatever order their work falls, and a floating-point sum depends on its order: the last digits of a tree are not those of a loop. So a card never adds up a loop's sum. It computes, for each round, what that round contributes, and hands back one number per round; the processor then folds those numbers into the variable one by one, in the rounds' order — which is what it does with the rounds it runs itself (Sums, Products and the Largest). The folding is a pass over memory and costs next to nothing beside the rounds. A product and a largest go the same way, and so do the sums of whole numbers, which no order could change.

The card works on copies. It gets the vectors a loop reads as they are when the loop begins, and the vectors it writes come back whole when it is done. For a loop the compiler calls parallel that is no different from what its rounds see on the cores: no round reads what another one writes.

Where a Loop Runs

A card is fast and far away. Handing it a loop means copying the loop's vectors over, waking the card, waiting for it, and copying the results back: a few tenths of a millisecond on the laptop of these measurements, whether the loop has eight rounds or eight million. And the first time a loop goes there its kernel has to be translated for the card, which takes some hundredths of a second. For the 8000 rows above that is nothing. For a small loop it is everything.

So the program decides, and it decides the way it already decides between one core and all of them (When the Cores Are Worth It): it starts every loop on one core, measures the first rounds, and only a loop with enough work left is handed on. At that moment it knows what a round costs. It compares what the cores would need for the rounds that are left with what the card needs for the whole loop — as measured the last time this loop ran there, or, the first time, the price of a translation and one call — and the card gets the loop if it wins by a clear margin. A loop that ran on the card is asked at its very first round the next time. And a loop that is too small to be worth a translation, but comes back again and again, is tried on the card once the work it has brought adds up to one: from then on it is measured like any other, and goes where it is faster.

SCALY_GPU_REPORT makes a program say at its end where each loop went. The program of this chapter with its 2000 rows, and then with 8000:

$ SCALY_GPU_REPORT=1 ./mandelbrot
gpu: mandelbrot.scaly:31: 0 on the card, 1 on the processor
mandelbrot 2000: 1588044 points inside
$ SCALY_GPU_REPORT=1 ./mandelbrot8000
gpu: mandelbrot.scaly:31: 1 on the card, 0 on the processor; the card 21.983 ms for 8000 rounds (0.030 in, 21.946 running, 0.007 out), its kernel compiled in 1.148 ms
mandelbrot 8000: 25402925 points inside

Two thousand rows are three hundredths of a second on ten cores: less than a translation costs, so they stay where they are. Eight thousand rows are half a second, and the card takes them. The line of a loop that went to the card says what it spent there — copying in, running, copying out — and what the translation of its kernel took; here the system had kept it from an earlier run. The times in such a line differ from run to run and from machine to machine, which is why they stand on standard error and not in what the program prints. A loop that never had enough work to be handed on is not in the report at all: nobody asked about it.

How much there is to lose shows a loop of eight rounds that a program runs three thousand times. Left alone, it never leaves its core. Forced to the card with SCALY_GPU=on it took 0.57 seconds for what the core does in a blink — three thousand journeys of two tenths of a millisecond each. on is for finding out what the card does with a loop, off for finding out what it costs to do without; a program that is to be fast sets neither.

The Numbers a Card Computes

Which loops a card can take depends on which numbers it has, and cards differ. Two rules cover them:

• A card computes in 32 bits. No card reached today has 64-bit floating-point numbers, and the cards a browser reaches (WebGPU) have no 64-bit integers either. A loop that computes with double stays on the processor, where its cores and lanes still scale it; one that computes with int, which is 64 bits, goes only to cards that have them. A double is never quietly computed as an f32: the digits would change with the place. A program that wants the card everywhere computes in f32 and i32; the loop's own counter does not count, as long as the range it counts fits into 32 bits.

• Floating point is exact or relaxed. Unless the program says otherwise, a loop with floating-point numbers may only go where every operation is rounded exactly as on the processor. Metal can be told to; WebGPU cannot — its specification lets a card reorder a sum, fuse a multiplication with an addition, and be off by a few last bits in a division. A loop written relaxed may go to such a card as well: its results stay within the tolerances the card's specification promises, which is what most graphics and machine learning want.

function half(x: f32) returns f32
    x * 0.5

let n 1000
var exact Vector[f32](n as size_t)
var loose Vector[f32](n as size_t)
for i in n
    exact.put(i as size_t, half(i as f32) / 3.0)
relaxed for i in n
    loose.put(i as size_t, half(i as f32) / 3.0)
print "`exact[999] as f64` `loose[999] as f64`"

$ scaly run relaxed.scaly
166.5 166.5
$ scalyc --gpu-plan relaxed.scaly
gpu-plan: 2 for-loops, 2 parallel, 2 kernels
gpu-plan: relaxed.scaly:7:1: kernel (metal): out exact; calls half; not webgpu: floating point, and the loop is not relaxed
gpu-plan: relaxed.scaly:9:9: kernel (metal, webgpu): out loose; calls half

relaxed stands before a loop or before a block, and holds for everything the loop's rounds compute, including the functions they call. It allows, it does not require: on the processor a relaxed loop computes exactly as before, and so it does under Metal, which rounds as the processor does whether it has to or not. Where a card of the looser kind computed it, a point on the edge of the Mandelbrot set may come out the other way — one whose iteration ends within a rounding error of 4.0. That is what the last words of the report at the head of this chapter say: the rows are a kernel for Metal as they stand, and would be one for WebGPU if the loop were relaxed. Counted there, the 2000 rows came to 1588048 points, four more than everywhere else.

A program built with --gpu carries a kernel in both languages where both may run it, Metal's and WebGPU's, and uses Metal unless told otherwise. SCALY_GPU_BACKEND=webgpu sends the loops that have a kernel for WebGPU there instead — through wgpu-native, the library that also sits under browsers, which the program opens when it first needs it and does without where it is not installed — and leaves the others on the processor. Whole numbers come back from that card exactly as from any other; a call takes two to three times as long as one through Metal. --gpu-msl and --gpu-wgsl print the kernels as the compiler writes them.

A browser is the other place with such a card. Built with scaly build --target wasm32-wasip1 --gpu, a program carries its kernels for WebGPU alone and asks the page it runs in for the card; a page that has none to give — or the command line — answers no, and every loop stays on the processor. The standard library brings the page's side along (web/gpu-host.mjs beside the WASI host): the program runs in a worker, the card belongs to a second one, and the loop waits for its rounds exactly as it does natively. The decision is the same too, taken where the processor has measured a round. A loop of whole numbers that keeps the one core a page's program has busy for 0.44 seconds came back from the card in 3 milliseconds, with the same numbers.