Build your project

Jun 19, 2020 - Reading time: 4 minutes. Tags: rust

What if you could build a Rust project without Cargo? It's not an actual problem I encountered. It's still a problem I looked into.

Cargo is the Rust package manager. Cargo downloads your Rust package’s dependencies, compiles your packages, makes distributable packages, and uploads them to crates.io, the Rust community’s package registry.

(from the Cargo Book )

That's a lot of work this tool does and it's very good at doing it.

I wanted to understand just the bit where it takes your Rust code and builds that into a binary that you can execute and run on your machine. I didn't really want to read all of Cargo, so instead I turned to what Cargo itself tells me. So I took the output of cargo build --verbose and turned that into simple build instructions.

What does it look like?

$ cargo build --verbose
[...]
   Compiling smallvec v1.4.0
   Compiling matches v0.1.8
   Compiling libc v0.2.71
     Running `rustc --crate-name smallvec --edition=2018 /Users/jer/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.4.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 -C metadata=d85954c5ff803feb -C extra-filename=-d85954c5ff803feb --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name libc /Users/jer/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.71/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=bbaef975a82234c8 -C extra-filename=-bbaef975a82234c8 --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
[...]
     Running `rustc --crate-name bygge --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=c1c12e33a54c5b24 -C extra-filename=-c1c12e33a54c5b24 --out-dir /Users/jer/projects/rust/bygge/target/debug/deps -C incremental=/Users/jer/projects/rust/bygge/target/debug/incremental -L dependency=/Users/jer/projects/rust/bygge/target/debug/deps --extern cargo_lock=/Users/jer/projects/rust/bygge/target/debug/deps/libcargo_lock-43ce509885ad1aff.rlib --extern cargo_toml=/Users/jer/projects/rust/bygge/target/debug/deps/libcargo_toml-5b6fff58e6c9a546.rlib --extern dirs=/Users/jer/projects/rust/bygge/target/debug/deps/libdirs-4eb870f2ccf6ed43.rlib --extern petgraph=/Users/jer/projects/rust/bygge/target/debug/deps/libpetgraph-7cb995dcfae4c1f7.rlib --extern pico_args=/Users/jer/projects/rust/bygge/target/debug/deps/libpico_args-03badf8072a773d9.rlib`
    Finished dev [unoptimized + debuginfo] target(s) in 30.82s

I removed a bunch of the output, but it all boils down to the same. You can re-run these commands and you should end up with the same binary as the one Cargo puts into target/debug/ .

You could put them all in a Makefile and use that to re-run them, but you would have gained nothing. Changing your own code or adding, removing or updating dependencies won't be picked up unless you carefully specify all these dependencies in the Makefile and update them when something changes.

We can automate that part.

Lately (again) I've been also reading about other build systems, such as Ninja and was intrigued to rebuild that. I haven't done that yet, but I decided to generate Ninja configuration to see how it behaves.

Let's combine these two things.

The basic manual version

Let's work from a basic binary crate, with no dependencies and a single file.

$ cargo new --bin hello-world
     Created binary (application) `hello-world` package
$ cat
fn main() {
    println!("Hello, world!");
}

Building just that is easy:

$ rustc src/main.rs -o hello-world
$ ./hello-world
Hello, world!

If we rerun rustc it will re-build the code and generate a new file hello-world (that should be the same as the one from the first invocation). That's a bit of extra work; if nothing changed we shouldn't need to rebuild. Let's create a Ninja configuration file build.ninja :

# The build rule to invoke rustc.
# $in and $out are variables provided by ninja.
rule rustc
  command = rustc $in -o $out
  description = RUSTC $out

# Specifying dependencies:
# For `hello-world` to be build we invoke the above `rustc` rule with `src/main.rs` as $in
build hello-world: rustc src/main.rs

With Ninja installed we can build and run this:

$ ninja
[1/1] RUSTC hello-world
$ ./hello-world
Hello, world!

Re-running Ninja won't build it again:

$ ninja
ninja: no work to do.

Ninja knows that nothing changed and thus skips extra work. If we modify the source it rebuilds:

$ echo >> src/main.rs
$ ninja
[1/1] RUSTC hello-world
$ ./hello-world
Hello, world!

We don't gain anything if we keep continuing writing a Ninja build configuration from hand. Especially not once we add more files and some external crates.

Bygge - build your project

Instead of writing out the whole Ninja file, or coming up with some rules for Make that expand to the right files, we can generate the build configuration once and then not touch it again until something changes. That's exactly what Ninja was designed for:

it is designed to have its input files generated by a higher-level build system, [...]

(from the Ninja Website )

After some more exploration with Cargo and the command lines it produces and the files it creates I managed to more-or-less auto-generate a Ninja configuration for a crate.

I've published this experiment as bygge , the crate to build your crates (and itself).

What it does

bygge create generates a Ninja build configuration (in build.ninja by default), listing all the targets a binary crate depends on, including all crate dependencies. ninja can then take this configuration and assemble the final binary. The result should be about the same as an invocation of cargo build .

What it doesn't

bygge is and never will be an alternative to Cargo. Cargo is a full-fledged build system, aware of different build targets, allowing to enable features per dependency, easily cross-compile to different targets and run the built programs as well as tests and generate documentation.

bygge ... builds.

Features

build.rs
cargo fetch

Building bygge

Bygge is able to create a Ninja build configuration to build itself. But first you need a compiled bygge . It comes with a pre-generated configuration for that, that only works on my machine unless you change the paths to the local checkouts of the dependencies.

$ ninja -f manual.ninja
[29/29] RUSTC build/bygge

It builds a build/bygge file that is able to create the build configuration and run Ninja:

$ build/bygge create
==> Creating build file: build.ninja
==> Package: bygge
$ build/bygge build
[29/29] RUSTC build/bygge
$ build/bygge -V
bygge v0.1.0

You can also build it using cargo build and use cargo run create (or target/debug/bygge create ) to create the build.ninja file.

And there you have it: a 300-line Ninja build configuration to build bygge.