Myrddin: mbld: How To Use It

Mbld: A Simple Build System for Myrddin

Mbld is a build system put together to avoid the inadequacies of traditional make, which, although it's great, does not have a way of scanning for dependencies ahead of time, and would require additional tooling to generate makefiles.

Mbld knows enough about typical project structures and input files to know how to build, install, uninstall, clean, and test a project. It is aware of paths within a project, and while the project looks like it is recursively structured, it can refer to files in sibling directories painlessly.

The build files are split into two different kinds: Project files, which are named bld.proj, and subfiles, named bld.sub, which have identical contents, but affect the paths referenced in the build files differently.

This will be explained in more depth later.

An Trivial Example

To start off, we will show a trivial example for mbld:

bin hello-world =
    hello-world.myr
;;

This produces a binary target called 'hello-world', using the input file 'hello-world.myr'. You may notice that no libraries are listed. This is becasue 'mbld' will find installed libraries, and add them to the dependency list for the binaries. Local libraries will still need to be added manually.

To build, install, and test this binary, the following commands can be run:

mbld                # builds everything
mbld install        # installs
mbld test           # runs tests
mbld clean          # removes generated files

A More Complete Example

This example covers most of the useful features that are available in a single directory mbld project:

# This is a comment
bin demo =
        main.myr
        demosrc.myr
        lib convenience
;;

# Attributes go in '{' '}'
lib convenience {noinst} =
        convenience.myr
        asmstuff+x64.s
        portglue+linux-x64.myr
        portglue+plan9.myr
        portglue.myr
;;

test extratests =
        test.myr
;;

Starting off at the top, we have the 'demo' program. This program is composed of two source files: main.myr and demosrc.myr. It also pulls in the library convenience, or whatever your platform names it. This is similar to the trivial example above.

Next, there is the definition for the library convenience. This is a library which is not installed, hence the noinst attribute. It contains a Myrddin source file, an assembly file for x86-64, and a bit of portability glue. This showcases a few interesting features of the build system, which were inspired a bit by Plan 9 and Go: System tags. Files twhich have an identical stem, but end with a "+tag-list" will be filtered by the tags, and the one most appropriate for the system will be selected.

File Layout

Mbld files, as mentioned before, come in two flavors: There is bld.proj, which defines a top level project, and bld.sub, which defines a subproject. We will call the most recent directory in the heirarchy containing a 'bld.proj' file the project root.

When mbld is run, it will walk up the directory tree to the first bld.proj file that it can find, and treat that as the root of the build. This means that any references to local libraries listed in the bldfile may not escape the project, and the absolute paths (ie, those which start with @/) are all relative to the project root.

Mbld will accept projects inside projects, but references to files may not escape a single project. So, for example:

bld.proj
foo/bld.sub
bar/bld.proj
bar/baz/bld.sub

The root bld.proj may refer to any files in this heirarchy. So, for example, it could have a target:

bin b =
        main.myr
        lib @/foo:foo
        lib @/bar:bar
        lib bar/baz:baz
;;

Similarly, foo.sub may refer to other libraries:

bin foo =
        foo.myr
        @/bar:bar
        ../bar/baz:baz
;;

However, any attempts to use members of foo/ as part of the build from within bar/ will raise an error, as you are attempting to reference a build from outside of the project.

Commands

mbld supports a number of commands to run builds. The comprehensive list is below:

all

This is the default target, although it can be specified explicitly. It runs every command needed to generate a compiled build. It will run the gen and cmd code, but does not run tests or install the code.

gen

This will run all 'gen' rules in the entire project. It will not run any other commands.

clean

This will remove all files that were automatically created by mbld in the build process, including any files generated by 'gen' commands that do not have the 'durable' attribute. Generated files with the durable attribute are preserved.

install

This copies the files of interest to their final homes. If the installed files are not yet built, it will run 'mbld all' to compile them. If the DESTDIR environment is set

uninstall

This is the converse of install, removing all installed binaries from their final homes. It does not currently remove empty directories that were created.

test

This command runs unit tests. By default, if the input list contain a file named `foo.myr`, and there exists a file `test/foo.myr`, then the latter will be assumed to be a unit test for the former. In addition, all of the explicit test targets will be run.

targ/path:target

This is an explicit build target. It's equivalent to what 'all' does, only it does it for one specific target. I'm not sure it's useful.

Target Types

mbld supports a small number of top level targets. All of these targets may be tagged with any number of 'tag=' attributes to restrict them to a single system with those tags. Multiple targets of the same name may be defined with different tags, and the same selection algorithm as is applied to files will be used to determine which one to build.

bin

The bin target represents a single binary. By default, on Unix-like systems, this binary is installed to $prefix/bin. For input files, it will accept Myrddin or assembly sources. The binary name does not include any system specific prefixes or suffixes. For example, building with a custom runtime:

bin foo {rt=custom_runtime.o} = input.myr list.s ;;

The bin target can include libraries, including libraries from siblings. It may not reach outside of the current project, though. In other words, all elements that are not installed must be part of the same project.

In addition to the common attributes, the bin supports the attributes enumerated below:

ldscript script

When linking a binary, use the custom linker script `script` to link the binary.

runtime rt

When linking a binary, use the custom runtime version `rt` instead of the default one shipped with Myrddin.

inc dir

Append `dir` to the default package search path.

inst

Marks that this file should be installed. For a 'bin' target, this is the default.

noinst

Marks that this file should not be installed.

test

Marks that this file should be run as a test.

notest

Marks that this file should not be run as a test. This is the default for 'bin' targets.

lib

The lib target is similar to the bin target, accepting the exact same inputs. By default, on Unix-like systems, this library is installed to $prefix/lib/myr. This prefix is chosen both to make listing and removing Myrddin code easier, and to prevent conflicts with other system libraries.

The library name used does not include the system specific prefixes or suffixes. For example, to produce libfoo.a:

lib foo = input.myr list.s ;;

Lib targets accept the same set of attributes as bin targets, with the caveat that the ldscript and runtime options do nothing.

test

The test target is an explicit test. For the most part, this target should only be needed rarely: implicit tests should cover most needs.

It's equivalent to a bin target, only it is not installed. Instead, it is run from mbld as a unit test. A successful exit is interpreted as a successful test run. For example:

test foo = testfoo.myr

Test targets accept the same attributes as bin targets. In fact, they are identical, with the exception that their attributes default to noinst=true, test=true, inc=".".

data

The data target lists a number of blobs to install. The default install location defaults to '$PREFIX/$SHAREPATH/blobname', unless it is overridden with the 'path' attribute. On unix-like systems, $SHAREPATH defaults to 'share', while on Plan 9, it defaults to 'lib'.

For example, the rule below will install the inputs to $PREFIX/share/prog-icons/{foo.png, bar.png}

data prog-icons = foo.png bar.png ;;

If these blobs should be installed to, eg, "$PREFIX/foo/bar" instead, then the following rule can be used:

data prog-icons {path=foo/bar} = foo.png bar.png ;;

The data installed may be generated with a 'gen' target.

man

The man target is a list of man pages to install. Eventually it should probably be deprecated, and replaced with a 'doc' target that can handle generation from a number of sources. It takes a list of manpages, which are named with the section that they go into. For example:

man = apiref.3 cmdref.1 ;;

gen

The gen target generates a file or list of files from a command. It will run the command every time gen is run, or if the output files do not exist.

If there is a dep attribute set, it will also rerun the gen command whenever at least one of outputs is older than any of the inputs. The dep attribute is necessary for this behavior because the gen command doesn't know anything about the structure of the command that it runs.

A notable thing about the command is that it is not run in a shell. This is done to maximize portability and avoid accidental environmental dependencies. Commands are instead run using std.execvp, and are therefore resolved using $PATH, or $path on Plan 9. For example, if you had a configure script which takes a --redo option to rerun while preserving variables, you may run it like this:

gen config.myr {durable,dep=configure} = ./configure --redo ;;

The following additional attributes are supported by gen targets:

durable

Do not remove the output files when running 'mbld clean'

dep

Regenerate the generated outputs if one of these files changes.

sub

The sub target includes the subdirectories listed in the file to

System Tags

Mbld supports system specific file versions. These are selected via the tags in the file name: Given a set of files with the same stem, and 0 or more tags, then the file with the best match for the system will be selected for the build, and all others ignored.

Tags in the file name follow the first '+', and are a '-' separated group of words, which mbld will match against. So, for example, if I am building on x86-64 linux, and I have the following set of files listed in my build:

foo+osx-x64.myr
foo.myr
foo+posixy.myr
foo+linux.myr
foo+linux-x64.myr

the match will proceed as follows:

Since there is one file with 2 matches, there is a clear winner, and foo+linux-x64.myr is selected for the build.

If the build did not list foo+linux-x64.myr, then there would be an ambiguity, and the build would fail, as both foo+posixy.myr and foo+linux.myr have one match. There is no weighting for tags.

And finally, if there were no matching files -- for example, a linux build with only foo+osx-x64.myr listed -- then the build would fail due to a lack of matching inputs.

Attributes

Various targets support different sets of attributes in mbld; This is a list of all attribtues and which build targets they apply to:

dep=file

Explicitly adds a dependency for the target. When the file listed in the dependency attribute changes, the rule will be rerun. This attribute may be repeated to list multiple files as dependencies. Valid on gen and cmd targets.

durable

Makes the outputs of this rule exempt from cleaning on an 'mbld clean'. This is useful for outputs that are expensive to generate, or created on the output of user commands, such as 'configure' scripts. Valid on gen and cmd targets.

inc=path

Adds an entry to the include path when running 6m. This can be useful for nonstandard include, although a lib directive should already include the use path for the library. This should only be rarely needed. Valid on bin, test, and lib targets.

ldscript=script

Uses a linker script when linking binaries, instead of going for the default linker behavior. This is useful when linking binaries with strange linker requirements -- for example, when linking a kernel. Valid on bin and test targets.

noinst

Tells mbld that a file should not be installed. Valid on bin, test, and lib targets.

runtime=rtpath

Tells mbld to link with a custom runtime; This is the code that handles the startup of the binary, various low level abortions, and emulation of certain unsupported operations. If you need to build a kernel, you will probably want to write your own custom runtime. Valid on bin and test targets.

sys

This adds system tagging to targets, allowing them to be built or not on various platforms. Allowed on gen, cmd, bin, lib and test targets.

Semiformal Syntax

The full syntax is below, in pseudo-yacc:

A bldfile is a list of targets:

bldfile: target 
       | bldfile target

A target is a target type, followed by the target information.

target: "bin" myrtarget
      | "lib" myrtarget
      | "test" myrtarget
      | "gen" cmdtarget
      | "cmd" cmdtarget
      | "man" anontarget
      | "sub" anontarget

The target information generally either takes the form name {attrs} = inputs ;; or = inputs ;;. Command targets take a list of output files.

myrtarget: name attrs "=" inputlist ";;"
cmdtarget: wordlist attrs "=" cmd ";;"
subtarget: attrs "=" wordlist ";;"
inputlist: input
         | inputlist input

The inputs are generally a list of names. Myrddin targets can also include dependency tags, in the form of lib libraryname.

input: name
     | "lib" name
namelist: name
     | namelist name

Rules also accept a set of attributes, in the form of a brace-enclosed list of key-value pairs, for example, "{noinst,inc=local-path}"

kvplist: name
       | name "=" val
attrs : "{" kvplist "}"

All names are fairly liberal in what they accept. Any alphanumeric character and a large amount of punctuation is acceptable. A quoted string is also a word, in case the usual word characters are too restrictive.

name: wordchar*
    | "(\"|[^"])*"
wordchar: any unicode alphanumeric character
        | "." |"_" |"$" |"-" | "/" | "@" | "!" 
        | "~" | "+" |"%" |"&" |"(" | ")" | "["
        | "]" | ":"

Options

To see the most current list of options for mbld, run it with the '-h' option. For the sake of completeness, here's a dump of the options that are accepted.

-?  print this help message
-h  print this help message
-t  list all available targets
-S  generate assembly when building
-d  dump debugging information for mbld
-I inc  add 'inc' to your include path
-R root install into 'root'
-b bin  compile binary named 'bin' from inputs
-l lib  compile lib named 'lib' from inputs
-r rt   link against runtime 'rt' instead of default
-C mc   compile with 'mc' instead of the default compiler
-M mu   merge uses with 'mu' instead of the default muse