Felix Programming Language

I'd like to introduce yet another build system, this one called Fbuild. It's a different way to build things. As opposed to pretty much every other build system, Fbuild is designed as just a caching library for Python 3.0. It takes advantage of Python's evaluation scheme to implicitly describe the build dependency tree. In order to evaluate the build, we simply evaluate python functions and cache their results. This turns out to be an elegant and easy way to describe and program build systems.

On top of this alternative design, Fbuild comes out of the box with some pretty advanced features:

• Linux and Apple support (Windows support will come soon)
• C, C++, OCaml, Bison, and Felix builders
• Multilevel namespaces for builders
• Simple creation of new builders
• Extensive configuration system for c90, c99, most of posix, and other libraries
• On-demand configuration
• Simultaneous building
• Detects file changes via digests verses timestamps
• Pretty output
• Very speedy (though it could be much faster)

Here's a quick example, in a file called fbuildroot.py:

import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    static.build_exe('exe', ['a.c', 'b.c', 'c.c'])

When run with as fbuild, assuming all those files exist, produces:

determining platform     : {'bsd', 'darwin', 'macosx', 'posix'}
looking for program gcc  : ok /usr/bin/gcc
checking /usr/bin/gcc    : ok
checking /usr/bin/gcc with -g : ok
checking /usr/bin/gcc with -O2 : ok
checking /usr/bin/gcc with -c  : ok
looking for program ar         : ok /usr/bin/ar
looking for program ranlib     : ok /usr/bin/ranlib
checking if /usr/bin/gcc -c can make objects: ok
checking if /usr/bin/gcc -c can make libraries: ok
checking if /usr/bin/gcc -c can make exes: ok
checking if /usr/bin/gcc -c can link lib to exe: ok
 * /usr/bin/gcc -MM                     : c.c
 * /usr/bin/gcc -MM                     : b.c
 * /usr/bin/gcc -MM                     : a.c
 * /usr/bin/gcc                         : c.c -> build/c.o
 * /usr/bin/gcc                         : b.c -> build/b.o
 * /usr/bin/gcc                         : a.c -> build/a.o
 * /usr/bin/gcc                         : build/a.o build/b.o build/c.o -> build/exe

You may notice a couple interesting things about the output:

1. Fbuild configures the c static builder for you and tests that it works.
2. Much of the output is hidden. Fbuild does this because the output can get overwhelming for large projects. It can be very easy to miss warnings and messages. Fbuild hides this in a log file, normally called build/fbuild.log.
3. The generated files are stored in a build directory. This is a user-customizable directory that can be used to support different build targets, such as a debug and a release build.
4. Fbuild automatically discovers and tracks and sorts the dependencies between each c file.

To develop the last idea further, Fbuild was designed to work with globbed paths if you don't want to bother writing out each filename. I'll show this by rewriting the last example using the fbuild.path.Path class. This is a convenience class that implements many useful path functions, as well as supports replacing the Unix-style '/' separator with the native path separator. Lets combine that with showing off what happens if we tweak what we want built:

import fbuild.builders.c
import fbuild.path

def build():
    static = fbuild.builders.c.guess_static()
    lib = static.build_lib('lib', fbuild.path.Path.glob('*.c'))
    static.build_exe('exe', [], libs=[lib])}}}

This will produce:

 * /usr/bin/ar           : build/a.o build/b.o build/c.o -> build/liblib.a
 * /usr/bin/ranlib       : build/liblib.a
 * /usr/bin/gcc          :  -> build/exe

As you can see, most of the work was skipped. Fbuild detected that none of the source files changed, so it didn't need to recompile them. Since the glob passed all the source files to static.build_lib they were all linked together in a library. Finally, since the arguments changed for static.build_exe, build/exe was recompiled.

Now that you've got a taste for Fbuild, lets go through some of the reasoning behind its design. As I mentioned previously, every build system I've found is based off of a multi-phase declarative tree evaluation. Here's the list of the system I looked at:

• Ant
• Automake
• Boost Build
• Cmake
• Make
• Ocamlbuild
• Omake
• Qmake
• Rake
• SCons
• Waf

By "declarative tree evaluation", I mean that the order of evaluation is explicitly encoded in a dependency tree. Each node has a function that takes one or more inputs and returns one or more outputs. By "multi-phased", I consider building the tree as one phase, walking the dependencies as another, and evaluating the nodes as yet another. This contrasts with Fbuild, which I consider to be a single-phase procedural build system. As python evaluates each cached function, the dependency implicitly gets evaluated. This means that it's possible to dynamically modify the build on demand:

Here's a stupid example of something that's difficult with Make:

import random
import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    static.build('exe1', ['exe1.c'])

    # Exit the build early if we roll a '6'
    if random.randint(1, 6) == 6: return

    static.build('exe2', ['exe2.c'])

Because we don't have to build up the tree first before we evaluate it, we don't have to shoehorn weird nodes into the graph to perform an action between one node and another.

For a more realistic example of how this is useful, consider a common problem with compilers. The compiler is normally written in one language, then the compiler is used to compile the standard library. This essentially requires dynamically creating new nodes for the tree, and can result in some pretty scary build scripts. Fbuild makes it easy though:

import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    exe = static.build_exe('exe', [...])
    my_static = fbuild.builders.c.guess_static(exe)
    my_static.build_lib('lib', [...])

First, we configure the system C builder. Next, we reconfigure another C builder with the one we just created. Finally, we use that builder to compile our standard library. As far as I can tell, most other build systems require some painful contortions to support this, and I suspect most projects cheat by ignoring the dependency between the standard library and the compiler. Fbuild inherently supports this style since we don't have to work around the graph.

Some systems allow for dynamically specifying dependency nodes in their script, such as in SCons. As far as I can tell, this is the simplest node you can create:

def action(target, source, env):
    pass
env = Environment(BUILDERS={'A': Builder(action=Action(action))})
env.A('foo.out', 'foo.in')

First, you define a simple function that converts a source file into a target file. Next, you create an environment where you update the builders with your custom builder. Finally, you use the environment to perform your action. It's not actually that bad. Here's the equivalent in Fbuild:

import fbuild.db

@fbuild.db.caches
def action(target:fbuild.db.DST, source:fbuild.dst.SRC):
    pass

def build():
    action('foo.out', 'foo.in')

It's actually a little longer. Part of that is that SCons imports all of the common code into the global namespace. This can save some lines, but it also forces a lot of SCons into the same namespace. This forces each node to be uniquely named, and is more likely to run into conflicts. Anyway, for Fbuild we need to do a couple extra things to get the to cache. First, we mark that the function is cached using the fbuild.db.caches decorator. There's also fbuild.db.cachemethod for decorating methods, and fbuild.db.PersistentObject parent class for caching object creation. We also need to tell Fbuild if the arguments are source or destination files. We do this using python 3.0's annotation support. This tells the database to check to see if source files were modified or if the destination file was removed. If either of these occurred, the function is rerun.

As the builders get nontrivial and more complex, Fbuild becomes a much simpler. For instance, consider SCons's Yacc builder. There are three main things it needs to do. First, SCons has to set up the environment variables for yacc. Next, they register yacc with the C builder. Finally, they have a function that uses the values from the environment and actually evaluates yacc.

Fbuild's current bison builder is much more straight forward:

import fbuild
import fbuild.builders
import fbuild.db
from fbuild.path import Path

class Bison(fbuild.db.PersistentObject):
    def __init__(self, exe=None, flags=[], *, suffix='.c'):
        self.exe = fbuild.builders.find_program([exe or 'bison'])
        self.flags = flags
        self.suffix = suffix

    def __call__(self, src:fbuild.db.SRC, dst=None, *,
            suffix=None,
            verbose=False,
            name_prefix=None,
            defines=False,
            flags=[],
            buildroot=None) -> fbuild.db.DST:
        buildroot = buildroot or fbuild.buildroot
        suffix = suffix or self.suffix
        dst = Path.addroot(dst or src, buildroot).replaceext(suffix)
        dst.parent.makedirs()

        cmd = [self.exe]

        if verbose:
            cmd.append('-v')

        if name_prefix is not None:
            cmd.extend(('-p', name_prefix))

        if defines:
            cmd.append('-d')

        cmd.extend(self.flags)
        cmd.extend(flags)
        cmd.extend(('-o', dst))
        cmd.append(src)

        fbuild.execute(cmd, self.exe, '%s -> %s' % (src, dst), color='yellow')

        return dst

We do some simple configuration to set up the executable and flags. Then we define a function to evaluate the executable. Since we're able to wrap a callable object, we don't have to be aware of anything else happening in the tree. We also don't need to pollute the environment with extra values, when they're only needed in this builder.

It's easy to extend this paradigm to configuration. Say you want to check if a header exists. It's simple to write this:

import fbuild.builders.c
import fbuild.db

@fbuild.db.caches
def check_stdlib_h(builder):
    return builder.header_exists('stdlib_h')

def build():
    static = fbuild.builders.c.guess_static()

    if check_stdlib_h(builder):
        ...
    else:
        ...

It's even easier using fbuild.config though. That package replicates much of the functionality in autoconf to provide a convenient lazy configuration system:

import fbuild.builders.c
import fbuild.config.c.c90

def build():
    static = fbuild.buildesr.c.guess_static()
    stdlib_h = fbuild.config.c.c90.stdlib_h(static)
    if stdlib_h.header:
        ...
    else:
        ...

    if stdlib_h.atof.return_type == 'float':
        ...
    else:
        ...

There are already hundreds of tests defined in fbuild.config. It's a new system, but it's already proven to be pretty useful.

So, that's Fbuild. Hopefully you'll find it interesting. It's still a very young project, and the interfaces aren't stable. But if you still want to check it out, you can find the source here. If you want to try it out, you'll need python 3.0 and git installed. You can check it out with:

git clone http://git.felix-lang.org/r/fbuild.git

There are a handful of examples in the examples directory. These can be run either with examples/examples.py, or run directly as in:

cd examples/c
../../fbuild-light

Where fbuild-light is just a simple wrapper around the fbuild executable. It's used so that you can run fbuild without it being installed.

If you have any questions, feel free to ask on the felix mailing list, or reply to the comments.