The Scaly Programming Language

Chapter 7

Many Connections

Waiting Is Not Working

The programs of the previous chapters keep every core busy with arithmetic. A server is busy differently: most of the time its connections wait — for the next request of a client, for a database, for a file. Ten thousand open connections may need the processor for a few hundred of them at any moment. What scales a server is how little a waiting connection costs.

A thread per connection costs a stack of megabytes and a place in the operating system's scheduler, and ten thousand of them are too many. The usual way out is to write the server as callbacks or with async and await, and every function that may wait has to be written that way too: the waiting becomes part of each function's type, and the program splits into two kinds of code.

Scaly does neither. A connection is a task: a small stack of its own (128 KB, of which only what is touched costs memory) and a region of its own, run by a scheduler that switches between tasks in user space. When a task reads from a socket that has nothing to read, it is parked, and the scheduler runs another; when data arrives, the task continues where it stopped. The read is an ordinary call — the same call that, in a program without a scheduler, simply blocks.

A Server

The http package of the standard library is a server built that way. A handler takes the request and the response, a router maps paths to handlers, and serve runs it:

package http 0.2.0

use http.Request.Request
use http.Response.Response
use http.Router.Router

procedure hello(req: ref[Request], mutable res: ref[Response])
{
    let name req.param("name")
    res.send_text("hello, `name`")
}

var app Router.create(Page.allocate_page())
app.get("/hello/{name}", &hello)
serve(app, 8080)

Run, it answers curl localhost:8080/hello/book with hello, book, and goes on serving until it is stopped. The program the book checks is one that stops by itself: a server that answers each request after a pause, and a hundred clients in the same program that ask at once. The handler sleeps with Scheduler.sleep, which parks its task, not a thread; each client is a task too, on a scheduler of the program's own, and counts how many clients are waiting for an answer at the same moment:

package http 0.2.0

use http.Request.Request
use http.Response.Response
use http.Router.Router

; waits ms milliseconds without holding a thread, then answers ms
procedure delay(req: ref[Request], mutable res: ref[Response])
{
    let ms parse_int_prefix(req.param("ms"))
    Scheduler.sleep(ms)
    res.send_int(ms)
}

mutable port: int 0
mutable waiting: int 0
mutable most: int 0
mutable answered: int 0

; one client: a request, then the answer up to the end of the connection
procedure client() mutable waiting, most, answered
{
    var s TcpStream.connect("127.0.0.1", port)
    s.write("GET /delay/100 HTTP/1.1\r\nHost: here\r\nConnection: close\r\n\r\n")
    waiting := waiting + 1
    if waiting > most
        most := waiting
    var buf u8[256]
    let view Slice[u8](256, &buf[0])
    var got: size_t 0
    repeat
    {
        let n s.read(view.slice_from(got))
        if n <= 0
            break
        got := got + (n as size_t)
    }
    waiting := waiting - 1
    s.close()
    if got > 12 and buf[9] = 50 and buf[10] = 48 and buf[11] = 48
        answered := answered + 1
}

var app Router.create(Page.allocate_page())
app.get("/delay/{ms}", &delay)
var l TcpListener.listen(0)
port := l.port()
let server start(app, l)

let clients Scheduler.create(Page.allocate_page())
var k 0
while k < 100
{
    Scheduler.spawn(&client)
    k := k + 1
}
Scheduler.run()

server.shutdown()
server.wait()
print "`answered` of 100 answered, at most `most` waiting at once"
$ scaly run waits.scaly
100 of 100 answered, at most 100 waiting at once
$ SCALY_WORKERS=1 scaly run waits.scaly
100 of 100 answered, at most 100 waiting at once

TcpListener.listen(0) takes a free port on the local machine, and start serves it on threads of its own and returns at once, where serve would not return; shutdown and wait end it. The router and the clients' scheduler each get a page to live on. &delay and &client hand the procedures themselves to the router and the scheduler.

All hundred clients waited at the same time, with all cores and with one: a hundred requests that take a tenth of a second each were answered, built with scaly build --release, after 0.11 seconds. A thousand clients took 0.15 seconds: the server held a thousand parked tasks, not a thousand threads.

The Same Calls Everywhere

Look at client again: it connects, writes and reads with the calls a program without any scheduler would use, and nothing in its declaration says that it may wait. Whether a call blocks or parks is decided when it runs, not written in the program:

  • outside a scheduler, read blocks the thread, as in any program;

  • inside a task, it parks the task and lets the scheduler run the others until data arrives;

  • a call the operating system can only do blocking — connect, which resolves a name and shakes hands, and the reads of a file — goes to a pool of helper threads, and the task parks until it is done.

So there is one kind of function. A procedure that reads a configuration file is called from a handler, from a test and from main alike, and code written before anyone thought of a server runs inside one unchanged. One limit is named: a socket is read and written by one task at a time, not by a reader and a writer in parallel.

One Scheduler per Core

A scheduler runs on one thread, and one thread uses one core. serve and start therefore run one scheduler per core — as many as SCALY_WORKERS allows — and the connections are spread over them. On Linux each scheduler has a listener of its own on the same port, and the kernel hands each new connection to one of them; on macOS and Windows one scheduler accepts and passes the connections round. A connection stays on the scheduler that took it, so its task never moves between cores, and its memory stays with one core's caches.

Each scheduler waits for its sockets with what the operating system offers for many at once — io_uring or epoll on Linux, kqueue on macOS, I/O completion ports on Windows — and none of that shows in the program. The checked program above ran its server on ten schedulers and on one, and printed the same.

A Region per Connection

Every task allocates from its own region (Memory That Is Not Shared). For a server that settles what else would be its hardest question: a request builds its strings, its parsed headers and its answer in the region of the handler's call, and all of it is released at once when the handler returns — no collector walks the heap, no other connection notices. A connection that ends gives back its stack and its region, and the records of a finished task are reused for the next connection, so a server does not grow with the number of connections it has served.

Neither is shared between the schedulers, so they never take turns at an allocator or a lock for memory. A connection costs what it uses, its stack, and nothing per request that outlives the request.