Chapter 9

Modules and Packages

Overview

A Scaly program can be split across several source files, and it can depend on other packages. There are no build files, manifests, or configuration files — three statements inside the source itself organize everything:

module loads another source file into the current program or package.
package declares a dependency on another package.
use makes types defined in another module visible in the current file.

Modules

Every source file is a module. The statement module shapes loads the file shapes.scaly from the directory of the file that declares it and makes it part of the program:

; shapes.scaly
define Box(width: int, height: int)
{
    function area(this) returns int
        width * height
}

; myprog.scaly
module shapes
use myprog.shapes.Box

let b Box(3, 4)
if b.area() = 12
    print "area is 12"

A module may declare child modules of its own. Those live in a directory named after the parent module: if compiler.scaly declares module lexer, the child is loaded from compiler/lexer.scaly. This mirrors how the standard library and the compiler themselves are laid out.

Top-level statements of a loaded module are ignored — only its definitions become part of the program. Programs have top-level statements; libraries consist of definitions only.

Packages

A package is a source distribution. Its version is encoded in its filesystem path — packages/scaly/0.2.1/ — and that path is the single source of truth; there is no separate version manifest. The statement

package scaly 0.2.0

declares a dependency on that package. A package is looked for among the packages of the installation, $SCALY_HOME/packages/ (the installed programs set SCALY_HOME themselves), and then among the project's own, ./packages/ under the directory the compiler runs in. So a project brings the packages the installation does not have by putting them beside its sources, and it cannot put a package of its own in front of an installed one of the same name. Without SCALY_HOME only the current directory is searched. A declared package that is nowhere fails the compile with an error naming the places.

The version in a declaration is a minimum. One compilation takes one version of a package, and where a program reaches the same package through several declarations — its own and those of the packages it uses — it takes the highest version any of them names: a program that asks for json 0.1.5 and uses a package that asks for json 0.1.2 is built, with 0.1.5 for both. That holds among versions that go together — within one major version, and below 1.0 within one second number (Versions, and what publishing means says why). Such versions form a line. Two lines of one package — a package that asks for json 0.1 beside one that asks for json 0.2 — are two packages to the build: both are compiled and linked, each asker gets its own, and a name of json written in a file means the line that file's package declares. What does not go: a value of one line handed to a routine of the other (they are different types), one file that declares both lines, and two lines of a package that brings C or assembly files. Each line has its own state — a cache or a counter kept in a package's global exists once per line — and what builds this way is not raised, so scaly publish says when a newer line of a dependency exists. Nothing else decides: there is no lock file and no solver, the versions follow from the declarations alone, and adding a dependency can raise a version but never lower one. A package's root file declares the packages it uses itself, so a program names only the packages it calls directly; circular package dependencies are forbidden.

The standard library package scaly is the implicit prelude: every program depends on it automatically, and its types are visible everywhere without any use statement. Pass --no-prelude to compile without it, for bare-metal or embedded targets. See Standard Library and Packages for what the standard library provides.

Packages from a git repository

A declaration may name where its package comes from, as a string after the version:

package json 0.1.3 "github.com/someone/json"

The source is a git repository that holds the package at packages/json/0.1.3/ — the layout a project's own packages have. One repository serves every version it has published (Versions, and what publishing means). "github.com/someone/json" stands for https://github.com/someone/json.git; an address with a scheme, a user@host:path address and a local path are taken as they are, and whatever git needs to reach a private repository is git's business. A version is required where a source is given.

scaly run, scaly test and scaly build fetch what a program declares before they compile it, and what the fetched packages declare in their turn; scalyc never reaches the network and says which commands do. This needs git on the PATH. A fetched package is kept under ~/.scaly/packages/ (SCALY_PACKAGES names another place), by source, name and version, with its files read-only; nothing is written into the project. It is the third place a package is looked for, after the installation and the project's ./packages/ — so copying a package there is how a project keeps its own copy, and editing that copy is how it patches a dependency.

A version the package's record names is taken from the commit named there, any other from the head of the default branch. A published version does not change (Versions, and what publishing means). The first fetch remembers the commit and the content of the version's directory beside it; a version that is fetched again and is no longer what it was is refused, with both commits named. One package comes from one source: a program that reaches it through two different sources is refused, while two spellings of one address are one source.

A package may bring C or assembly files, which are compiled on the machine that builds it; the tool says so when it fetches one.

What a package defines is the package's own: two packages may each define a function, a record with its methods or a mutable global of one name, and each keeps its own. A record of one package is another type than a same-named record of another, and a name written in a package means that package's own before anything a use brings in. A program may hold values of both, call their methods and hand each back to its package; handing one package's record to the other is refused like any wrong type. The standard library's names and a program's own are not marked.

A package's constants and mutable globals are its own as well. A mutable global is one cell wherever it is read: a program that names a package's counter reads the counter the package's routines write. A bare name means the global of the package it is written in before another package's of the same name; written where two packages in reach both declare it and neither is the writer's own, it is refused, since nothing says which is meant.

A use makes a name writable — as a type, in a construction, before a static call. A value brings its type along: its methods and fields are reached without naming the type, so let a make_alpha(1) and a.grown() need no use of what make_alpha returns.

The cache

A package is compiled once. The first program that uses it has it compiled into the cache, ~/.scaly/cache (SCALY_CACHE names another place); every later build links what lies there. Beside the compiled package lies its interface — the package as a caller sees it, every declaration and no body — and that is what the compiler reads from then on instead of the sources. Nothing of this is written into a project or a package, and nothing of it is ever committed.

Each entry is kept under a key over everything that went into it: the compiler, every file of the package, and the versions of the packages it declares. So an entry is never wrong, only no longer asked for — a changed package or a new compiler leaves the files of the state before. scaly clean empties the cache; what is needed is made again by the next build. The packages that were fetched (~/.scaly/packages) and the programs that were installed (~/.scaly/bin) are not the cache and stay.

Versions, and what publishing means

The rules of a package's versions are few, and the tool checks them.

A version is a directory. packages/json/0.1.3/ is version 0.1.3 of json, and the next version is another directory beside it. There are no tags, no registry and no manifest.

Published is what the package's record names. Beside its version directories a package keeps the file packages/json/published, a line for each published version:

0.1.2 8c1f0e…  5d2a97…
0.1.3 b41c77…  0e93ab…

The version, the commit its directory lay at when it was published, and the git tree of that directory. scaly publish writes the lines; nobody else needs to. A version directory without a line is a version being written: it may change from commit to commit, on the default branch too, and a program that declares it gets it as it stands — and is told when it has changed since.

A published version does not change. Its line stays as it is, and its directory, while it is in the tree, stays the tree the line names. A correction, however small, is a new version: copy the directory, change the copy.

A published version may leave the tree. The line names the commit that holds it, and a build that declares the version fetches that commit. So a repository need not carry every version it ever published side by side: the directories being worked on are in the tree, the others in its history. Removing one is an ordinary commit that deletes the directory and leaves the record alone.

The number says what changed. A new version is compared with the one before it — the highest lower version, in the tree or in the record — by what the two declare: every function, procedure, operator and initializer with its parameters and its result, every record with its fields, every union with its variants, every constant, global and alias. Bodies are not part of it, a generic routine's body neither. A package it declares counts too, by its line: a version that moves from json 0.1 to json 0.2 has changed what it declares, because a program that uses both packages must follow. Adding or dropping a dependency changes nothing a user had.

against the version before	from 1.0 on	below 1.0
a declaration is gone or reads differently	the first number rises	the second number rises
declarations were added, none is gone	the second number rises	the third number rises
bodies only	the third number rises	the third number rises

A version below 1.0 has promised nothing yet, so there a breaking change costs the second number; 1.0.0 is the statement that what the package declares is meant to stay. A build follows the same reading: the highest version asked for stands in for the lower ones where the table says nothing was taken away — within one first number from 1.0 on, within one second number below it. 0.2.0 does not stand in for 0.1.5, and a program that reaches both is told so.

The comparison is deliberately strict. Declarations are compared as they are written, comments and line breaks apart: a renamed parameter, a constant's new value, a field added to a record all count as a declaration that reads differently. And nothing in a package is private, so every declaration counts.

scaly publish --check
scaly publish

Both run at the root of the package's repository and speak with the remote origin (another remote, an address or a path may follow). scaly publish --check changes nothing: it says whether every published version is still what its line names, whether the remote's record is still all here, and whether every new version's number says what it changed. A version that understates its change is refused with the declarations that are gone and the number it takes at least; a new version that does not compile is refused too. A new version of another line (0.2.0 after 0.1.x, 2.0.0 after 1.x) may change anything, so the version before is not compared — it may be written in a language the compiler of today no longer reads. Where a new version declares a package of which a newer line has been published meanwhile, it says so in a note — the declaration is not wrong, but no build will take the newer line until its author raises it. It answers 0 when all is well, 1 when a rule is broken and 2 when it could not ask — no repository, or the remote out of reach. It belongs in the package's own test run, so a change to a published version is noticed before it is pushed and not by the first program that fetches it.

scaly publish checks the same and then publishes what is new: each new version gets its line — the commit is the current one, so the version must be committed —, the record is committed, and both commits are pushed. From then on the version is fixed.

Making the next version of a package is therefore: give packages/json/0.1.3/ the number the change takes, work and commit there as long as it takes, and scaly publish when it is done. The directory may be copied, and then both versions lie side by side; it may as well be renamed, because the published one lives on at its commit, and then the tree holds one directory per package. scaly next json 0.1.4, at the root of the repository, does that: it renames the directory with git and rewrites every tracked file that names it (a script, a test). It wants a tree without uncommitted changes and a number higher than the one that is there, and it leaves a published directory of another package as it was published. A package that is released together with a program — a compiler with its parser, say — publishes a version with each release and works on the next one in between.

A declared version is a minimum. package json 0.1.3 asks for 0.1.3 or what stands in for it. Among the declarations of one build that is the highest version asked for; and where a place a package is looked for holds no directory of the version itself, it is the highest version of that line there that is not lower. So the declarations need not follow when a package they name begins its next version: a program that says package json 0.1.3 builds against a tree or an installation that has packages/json/0.1.4/ alone. A version of another line never stands in, and neither does a lower one.

Starting a package

scaly new shapes --lib

creates shapes/packages/shapes/0.1.0/shapes.scaly: a package with one function and its test. Its root file declares the packages it uses — the standard library too, which only a program gets without saying so:

package scaly 0.2.0

define shapes
{
    function greeting(name: String) returns String
        "Hello, `name`!"

    function test_greeting() returns int
    {
        if greeting("Scaly") <> "Hello, Scaly!"
            return 1
        0
    }
}

scaly test packages/shapes/0.1.0/shapes.scaly runs its tests. A program in the directory that holds packages/ uses it with package shapes 0.1.0; once the directory is a git repository somewhere, any program does, by naming that repository as the source.

The programs of a package

A package that is a library first may still bring programs — a parser its validator and its normalizer, say. The programs a package offers are the files of the directory programs/ beside its root file, each a program named after its file:

packages/shapes/0.1.0/
    shapes.scaly              the library
    shapes/                   its modules
    programs/
        shapes.scaly          the program "shapes"
        shapes_check.scaly    the program "shapes_check"

A program there is an ordinary program that declares the package it belongs to, without a source — its package and the others of the same repository lie beside it:

package shapes 0.1.0

print(shapes.greeting("world"))

One of the programs may be called like the package. Other program files a package keeps — drivers for its tests, experiments — stay outside programs/ and are not offered.

scaly install github.com/someone/shapes shapes 0.1.0
scaly install github.com/someone/shapes shapes 0.1.0 shapes_check

fetches the package from that repository, builds its programs — all of them, or the one named — optimised as scaly build --release does, and puts them beside scaly itself, in ~/.scaly/bin (SCALY_BIN names another place). A package without programs/ is a library, and scaly install says so.

scaly install without arguments lists what it has installed — program, package, version and source —, and scaly uninstall shapes_check removes one of those. It goes by its own record and removes nothing else of that directory.

Imports and Visibility

Type definitions are private to their module by default. A type defined in another module — even one loaded into the same program with module — is hidden until the current file imports it with a use statement. This keeps every file's dependencies explicit: the list of use statements at the top of a file is a complete inventory of the outside types the file relies on.

What Is Visible Without an Import

Three kinds of types need no use statement:

Types defined in the same file.
Types of the prelude — the standard library (String, Vector, Option, …) is always visible.
The namespace of a declared child module. When a file declares module geometry and geometry.scaly defines a namespace of the same name, that namespace is visible in the declaring file, so its members can be reached qualified:
```
; geometry.scaly
define geometry
{
    function unit_area() returns int
        1
}

; myprog.scaly
module geometry

let a geometry.unit_area()   ; qualified access, no use needed
```

Visibility applies to named type definitions. Free functions are resolved through their namespaces and are not gated separately.

Importing a Single Type

A use statement names a type by its full dotted path. The path starts with the program or package name, followed by the module path, ending in the type name:

use myprog.shapes.Box            ; Box from module shapes of program myprog
use scalyc.compiler.Modeler      ; Modeler from package scalyc

A module that is named after the single type it defines — the file Modeler.scaly defining Modeler — is addressed once, as the path spells it; the module and the type do not appear as two separate path segments.

Importing Everything from a Module

A trailing * imports every type a module defines:

module shapes
use myprog.shapes.*

let b Box(3, 4)              ; Box is visible through the glob
if b.area() = 12
    print "glob works"

Globs are convenient for a module whose types are used pervasively. Prefer single-type imports when a file uses only one or two names from a module — they document the dependency more precisely.

When a Type Is Not Visible

Referencing a type that exists in the program but has not been imported fails the compile. The error names the module that defines the type and states the exact use line that would import it:

myprog.scaly:3:7: error: function not found: Box - defined in
    myprog.shapes but not visible here; add: use myprog.shapes.Box

Two compiler flags support working with visibility: --no-use-vis disables enforcement entirely (every type in the program is visible everywhere, the pre-visibility behavior), and --use-vis prints an advisory report of every reference that strict visibility would reject — useful when migrating an existing codebase, to collect the needed use lines before turning enforcement on.