Felix Programming Language

I'm pleased to announce FBuild version 0.2. It's been way too long since the last FBuild release, but a lot of features have been added over the past year. Overall, the core concept of FBuild is still the same, but there have been some structural changes in response to some suggestions I've had.

The most obvious change is that there is now a context object that you must pass around in order to interact with cacheable functions. Now, to write a simple builder, you must do:

import fbuild.builders.c

def build(ctx):
    shared = fbuild.builders.c.guess_shared(ctx)
    shared.build_exe('foo', ['foo.c', 'bar.c'])

Next is the support for command line targets. This allows you to easily call a specifically marked function, as in:

import fbuild.builders.ocaml
def build(ctx):
    ocamlc = fbuild.builders.ocaml.Ocamlc(ctx)
    ocamlc.build_exe('foo', ['foo.ml'])

import fbuild.target
@fbuild.target.register(name='run-test', help='run the foo test')
def run_test(ctx):
    # make sure 'foo' is built
    build(ctx)
    ctx.execute([ctx.buildroot / 'foo', 'world'])
    ctx.execute([ctx.buildroot / 'foo', 'fbuild'])

Which can be run by:

% fbuild run-test
looking for program ocamlc.opt : ok /opt/local/bin/ocamlc.opt
looking for program ocamldep.opt : ok /opt/local/bin/ocamldep.opt
determining platform             : {'bsd', 'darwin', 'macosx', 'posix'}
checking if ocamlc.opt can make objects : ok
checking if ocamlc.opt can make libraries: ok
checking if ocamlc.opt can make exes    : ok
checking if ocamlc.opt can link lib to exe: ok
 * ocamldep.opt                         : foo.ml
 * ocamlc.opt                           : build/foo.ml -> build/foo.cmo
 * ocamlc.opt                           : build/foo.cmo -> build/foo
Hello world
Hello fbuild!

One nice feature of the targets is that they are automatically integrated into the "--help" option:

% fbuild -h
Usage: fbuild-light [options] target1 [target2 target3 ...]

...

Targets:
  build     
  run-test  run the foo test

Next up is another removal of magic, this time in fbuild.config.c. Now, if you want to define a config test that requires a header, you must explicitly define it yourself, as in:

import fbuild.config.c as c
class assert_h(c.Test):
    header = c.header_test('assert.h')

Finally, the last major addition was support for Linux's LD_LIBRARY_PATH and OS X's DYLD_LIBRARY_PATH for shared libraries. Now it's much easier to execute test code that depends on a shared library. To use it, just do:

ctx.execute(cmd, runtime_libpaths=['lib1', 'lib2'])

Beyond that, we added builders for ghc, avr-gcc, gcc/iOS, O'Caml Findlib, and O'Caml Batteries. For configuration support, we now can check for SDL, LLVM, GNU Readline, GNU GMP, Google Test Framework, and OpenSSL. And, of course, lots of miscellaneous bug fixes. For a more detailed breakdown, please look at our git history.

Oh and one last thing. FBuild-0.2's build/ directory is incompatible with FBuild-0.1's build/ directory, so you'll have to remove the old directory before you can compile.

Please let us know if you run into any problems or need any help. You can:

• File a bug at http://github.com/erickt/fbuild/issues
• Contact our mailing list at http://groups.google.com/group/fbuild
• Or find us on IRC on freenode.net in #felix

I just wanted to let everyone know that I'm transitioning the hosting of felix and fbuild over to github. You can find the following repositories over there:

• felix: http://github.com/erickt/felix
• felix2 (my unannounced prototype new felix frontend): http://github.com/erickt/felix2
• fbuild: http://github.com/erickt/fbuild
• dypgen mirror: http://github.com/erickt/dypgen
• ocs mirror: http://github.com/erickt/ocs

I'll still be maintaining the http://git.felix-lang.org mirrors, but I'm considering github to be the primary location to get the code, as well as for bug tickets and etc.

Hi all, long time no see! I just wanted to say that we now officially own #felix for all your felix and fbuild related stuff. So hop on by if you want to chat :)

Holger on the fbuild mailing list was trying to use fbuild to create python c extensions, so I mocked up a simple builder for him. There's a little work looking up the include directory, and I'm hardcoding the gcc option "-undefined dynamic_lookup", but once done it's pretty simple:

import fbuild.builders.c
import fbuild.db
import fbuild.path

@fbuild.db.caches
def python_c_builder(ctx, python='python'):
    """Creates and returns a python C extension builder."""

    # If we didn't explicitly chose which version of python to use, search the
    # environment for python.
    python = fbuild.builders.find_program(ctx, [python])

    # Create a helper program that calls out to python to get the include and
    # lib directories.
    stdout, stderr = ctx.execute([python, '-c',
        'import distutils.sysconfig; '
        'print(distutils.sysconfig.get_python_inc())'],
        quieter=1)
    includepath = stdout.decode().strip()

    # Return a fully specified python c extension builder.
    return fbuild.builders.c.guess_shared(ctx,
        includes=[includepath],
        flags=['-undefined', 'dynamic_lookup'],
        lib_prefix='',
        lib_suffix='.so')

def build(ctx):
    for python in ('python', 'python2.5', 'python2.6', 'python3.1'):
        # Configure the python.
        shared = python_c_builder(ctx, python)

        # Copy the src file so we don't have collisions.
        fbuild.path.Path('build/%s' % python).makedirs()
        fbuild.path.Path('spam.c').copy('build/%s' % python)

        # Build the extension module.
        shared.build_lib('%s/spam' % python, ['build/%s/spam.c' % python])

        # Test if we can actually import the module.
        ctx.execute([python, '-c', 'import spam; spam.system("echo hello world!")'],
            cwd='build/%s' % python)

Which results in:

looking for program python : ok /opt/local/bin/python
determining platform       : {'bsd', 'darwin', 'macosx', 'posix'}
looking for program gcc    : ok /usr/bin/gcc
checking gcc               : ok
checking gcc with -g       : ok
checking gcc with -O2      : ok
checking gcc with -undefined dynamic_lookup -fPIC: ok
checking gcc with -undefined dynamic_lookup -dynamiclib: ok
checking gcc with -undefined dynamic_lookup: ok
checking if gcc -undefined dynamic_lookup -fPIC can make objects: ok
checking if gcc -undefined dynamic_lookup -fPIC can make libraries: ok
checking if gcc -undefined dynamic_lookup -fPIC can make exes: ok
checking if gcc -undefined dynamic_lookup -fPIC can link lib to exe: ok
 * gcc -undefined dynamic_lookup -fPIC  : build/python/spam.c -> build/python/spam.os
 * gcc -undefined dynamic_lookup -dynamiclib: build/python/spam.os -> build/python/spam.so
hello world!
looking for program python2.5           : ok /opt/local/bin/python2.5
checking gcc                            : ok
checking gcc with -g                    : ok
checking gcc with -O2                   : ok
checking gcc with -undefined dynamic_lookup -fPIC: ok
checking gcc with -undefined dynamic_lookup -dynamiclib: ok
checking gcc with -undefined dynamic_lookup: ok
checking if gcc -undefined dynamic_lookup -fPIC can make objects: ok
checking if gcc -undefined dynamic_lookup -fPIC can make libraries: ok
checking if gcc -undefined dynamic_lookup -fPIC can make exes: ok
checking if gcc -undefined dynamic_lookup -fPIC can link lib to exe: ok
 * gcc -undefined dynamic_lookup -fPIC  : build/python2.5/spam.c -> build/python2.5/spam.os
 * gcc -undefined dynamic_lookup -dynamiclib: build/python2.5/spam.os -> build/python2.5/spam.so
hello world!
looking for program python2.6           : ok /opt/local/bin/python2.6
 * gcc -undefined dynamic_lookup -fPIC  : build/python2.6/spam.c -> build/python2.6/spam.os
 * gcc -undefined dynamic_lookup -dynamiclib: build/python2.6/spam.os -> build/python2.6/spam.so
hello world!
looking for program python3.1           : ok /opt/local/bin/python3.1
checking gcc                            : ok
checking gcc with -g                    : ok
checking gcc with -O2                   : ok
checking gcc with -undefined dynamic_lookup -fPIC: ok
checking gcc with -undefined dynamic_lookup -dynamiclib: ok
checking gcc with -undefined dynamic_lookup: ok
checking if gcc -undefined dynamic_lookup -fPIC can make objects: ok
checking if gcc -undefined dynamic_lookup -fPIC can make libraries: ok
checking if gcc -undefined dynamic_lookup -fPIC can make exes: ok
checking if gcc -undefined dynamic_lookup -fPIC can link lib to exe: ok
 * gcc -undefined dynamic_lookup -fPIC  : build/python3.1/spam.c -> build/python3.1/spam.os
 * gcc -undefined dynamic_lookup -dynamiclib: build/python3.1/spam.os -> build/python3.1/spam.so
hello world!

I should be able to get a prototype builder out soon.

bohan on reddit pointed out that my text.m4_substitute doesn't actually have anything to do with m4. So I replaced it with two autoconf inspired functions: text.autoconfig_config_file and text.autoconfig_config_header. Consider this source:

const char* foo = @foo@;
#undef HAVE_FOO
#undef HAVE_BAR
#undef HAVE_BAZ

If you use this function:

text.autoconf_config_file(ctx, 'foo.py', 'foo.h.in', {
    'foo': '"FOO"',
})

which replicates the functionality of AC_CONFIG_FILES, you'll get:

const char* foo = "FOO";
#undef HAVE_FOO
#undef HAVE_BAR
#undef HAVE_BAZ

And if you process it with this:

text.autoconf_config_header(ctx, 'baz.h', 'baz.h.in', {
    'foo': '"FOO"',
    'HAVE_FOO': 1,
    'HAVE_BAR': 0,
    'HAVE_BAZ': '"BAZ"',
})

which replicates AC_CONFIG_HEADERS, you'll get:

const char* foo = "FOO";
#define HAVE_FOO 1
/* #undef HAVE_BAR */
#define HAVE_BAZ "BAZ"

Well that wasn't that terrible. I've removed all the global state from fbuild. Now a context object is passed around that contains this information. Now you must write your build system like:

import fbuild.db
import fbuild.builders.c

@fbuild.db.caches
def foo(ctx, value):
    ctx.logger.log('got: %s' % value)
    return value

def build(ctx);
    foo(ctx, 1)

    static = fbuild.builders.c.guess_static(ctx)
    static.build_exe('foo', ['a.c', 'b.c'])

As you can see, you now must pass the context around as the first argument to any cached function or object.

I'm pleased to announce the first release 0.1 of FBuild, which you can download here. FBuild is a new way to build software. It's designed around the concept of caching functions, instead of declarative tree building. For instance, this is a simple C fbuildroot.py:

import fbuild
import fbuild.builders.c

def build():
    # Create a C builder
    static = fbuild.builders.c.guess_static()

    # Build some libraries
    lib1 = static.build_lib('static1', ['lib1.c'])
    lib2 = static.build_lib('static2', ['lib2.c'], libs=[lib1])

    # ... an executable that uses those libraries
    exe = static.build_exe('static', ['exe.c'], libs=[lib2])

    # ... finally run it
    fbuild.logger.log(' * running %s:' % exe)
    fbuild.execute([exe])

When you run it, you'll get:

determining platform     : {'bsd', 'darwin', 'macosx', 'posix'}
looking for program gcc   : ok /usr/bin/gcc
checking gcc              : ok
checking gcc with -g      : ok
checking gcc with -O2     : ok
looking for program ar    : ok /usr/bin/ar
looking for program ranlib : ok /usr/bin/ranlib
checking if gcc can make objects : ok
checking if gcc can make libraries : ok
checking if gcc can make exes      : ok
checking if gcc can link lib to exe : ok
 * gcc                              : lib1.c -> build/lib1.o
 * ar -rc                           : build/lib1.o -> build/libstatic1.a
 * ranlib                           : build/libstatic1.a
 * gcc                              : lib2.c -> build/lib2.o
 * ar -rc                           : build/lib2.o -> build/libstatic2.a
 * ranlib                           : build/libstatic2.a
 * gcc                              : exe.c -> build/exe.o
 * gcc                              : build/exe.o build/libstatic2.a build/libstatic1.a -> build/static
 * running build/static:
5 6

This is a pretty simple example, but FBuild really shines when you need to do something more complex. Here's some of the details of the current release:

• Linux, Apple and Windows support
• C, C++, OCaml, Java, Scala, Bison, and Felix builders
• Multilevel namespaces for builders
• Simple creation of new builders
• Extensive configuration system for c90, c99, most of posix, and other libraries
• On-demand configuration
• Simultaneous building
• Detects file changes via file digests
• Pretty output
• Very speedy

For examples on how to use it, you can read the documentation here.

You'll need either Python 3.0 or Python 3.1, though I'd recommend Python 3.1 since FBuild is much faster with it.

If you need more help, you can ask on our mailing list, or in our irc channel.

I'd like to introduce yet another build system, this one called Fbuild. It's a different way to build things. As opposed to pretty much every other build system, Fbuild is designed as just a caching library for Python 3.0. It takes advantage of Python's evaluation scheme to implicitly describe the build dependency tree. In order to evaluate the build, we simply evaluate python functions and cache their results. This turns out to be an elegant and easy way to describe and program build systems.

On top of this alternative design, Fbuild comes out of the box with some pretty advanced features:

• Linux and Apple support (Windows support will come soon)
• C, C++, OCaml, Bison, and Felix builders
• Multilevel namespaces for builders
• Simple creation of new builders
• Extensive configuration system for c90, c99, most of posix, and other libraries
• On-demand configuration
• Simultaneous building
• Detects file changes via digests verses timestamps
• Pretty output
• Very speedy (though it could be much faster)

Here's a quick example, in a file called fbuildroot.py:

import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    static.build_exe('exe', ['a.c', 'b.c', 'c.c'])

When run with as fbuild, assuming all those files exist, produces:

determining platform     : {'bsd', 'darwin', 'macosx', 'posix'}
looking for program gcc  : ok /usr/bin/gcc
checking /usr/bin/gcc    : ok
checking /usr/bin/gcc with -g : ok
checking /usr/bin/gcc with -O2 : ok
checking /usr/bin/gcc with -c  : ok
looking for program ar         : ok /usr/bin/ar
looking for program ranlib     : ok /usr/bin/ranlib
checking if /usr/bin/gcc -c can make objects: ok
checking if /usr/bin/gcc -c can make libraries: ok
checking if /usr/bin/gcc -c can make exes: ok
checking if /usr/bin/gcc -c can link lib to exe: ok
 * /usr/bin/gcc -MM                     : c.c
 * /usr/bin/gcc -MM                     : b.c
 * /usr/bin/gcc -MM                     : a.c
 * /usr/bin/gcc                         : c.c -> build/c.o
 * /usr/bin/gcc                         : b.c -> build/b.o
 * /usr/bin/gcc                         : a.c -> build/a.o
 * /usr/bin/gcc                         : build/a.o build/b.o build/c.o -> build/exe

You may notice a couple interesting things about the output:

1. Fbuild configures the c static builder for you and tests that it works.
2. Much of the output is hidden. Fbuild does this because the output can get overwhelming for large projects. It can be very easy to miss warnings and messages. Fbuild hides this in a log file, normally called build/fbuild.log.
3. The generated files are stored in a build directory. This is a user-customizable directory that can be used to support different build targets, such as a debug and a release build.
4. Fbuild automatically discovers and tracks and sorts the dependencies between each c file.

To develop the last idea further, Fbuild was designed to work with globbed paths if you don't want to bother writing out each filename. I'll show this by rewriting the last example using the fbuild.path.Path class. This is a convenience class that implements many useful path functions, as well as supports replacing the Unix-style '/' separator with the native path separator. Lets combine that with showing off what happens if we tweak what we want built:

import fbuild.builders.c
import fbuild.path

def build():
    static = fbuild.builders.c.guess_static()
    lib = static.build_lib('lib', fbuild.path.Path.glob('*.c'))
    static.build_exe('exe', [], libs=[lib])}}}

This will produce:

 * /usr/bin/ar           : build/a.o build/b.o build/c.o -> build/liblib.a
 * /usr/bin/ranlib       : build/liblib.a
 * /usr/bin/gcc          :  -> build/exe

As you can see, most of the work was skipped. Fbuild detected that none of the source files changed, so it didn't need to recompile them. Since the glob passed all the source files to static.build_lib they were all linked together in a library. Finally, since the arguments changed for static.build_exe, build/exe was recompiled.

Now that you've got a taste for Fbuild, lets go through some of the reasoning behind its design. As I mentioned previously, every build system I've found is based off of a multi-phase declarative tree evaluation. Here's the list of the system I looked at:

• Ant
• Automake
• Boost Build
• Cmake
• Make
• Ocamlbuild
• Omake
• Qmake
• Rake
• SCons
• Waf

By "declarative tree evaluation", I mean that the order of evaluation is explicitly encoded in a dependency tree. Each node has a function that takes one or more inputs and returns one or more outputs. By "multi-phased", I consider building the tree as one phase, walking the dependencies as another, and evaluating the nodes as yet another. This contrasts with Fbuild, which I consider to be a single-phase procedural build system. As python evaluates each cached function, the dependency implicitly gets evaluated. This means that it's possible to dynamically modify the build on demand:

Here's a stupid example of something that's difficult with Make:

import random
import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    static.build('exe1', ['exe1.c'])

    # Exit the build early if we roll a '6'
    if random.randint(1, 6) == 6: return

    static.build('exe2', ['exe2.c'])

Because we don't have to build up the tree first before we evaluate it, we don't have to shoehorn weird nodes into the graph to perform an action between one node and another.

For a more realistic example of how this is useful, consider a common problem with compilers. The compiler is normally written in one language, then the compiler is used to compile the standard library. This essentially requires dynamically creating new nodes for the tree, and can result in some pretty scary build scripts. Fbuild makes it easy though:

import fbuild.builders.c

def build():
    static = fbuild.builders.c.guess_static()
    exe = static.build_exe('exe', [...])
    my_static = fbuild.builders.c.guess_static(exe)
    my_static.build_lib('lib', [...])

First, we configure the system C builder. Next, we reconfigure another C builder with the one we just created. Finally, we use that builder to compile our standard library. As far as I can tell, most other build systems require some painful contortions to support this, and I suspect most projects cheat by ignoring the dependency between the standard library and the compiler. Fbuild inherently supports this style since we don't have to work around the graph.

Some systems allow for dynamically specifying dependency nodes in their script, such as in SCons. As far as I can tell, this is the simplest node you can create:

def action(target, source, env):
    pass
env = Environment(BUILDERS={'A': Builder(action=Action(action))})
env.A('foo.out', 'foo.in')

First, you define a simple function that converts a source file into a target file. Next, you create an environment where you update the builders with your custom builder. Finally, you use the environment to perform your action. It's not actually that bad. Here's the equivalent in Fbuild:

import fbuild.db

@fbuild.db.caches
def action(target:fbuild.db.DST, source:fbuild.dst.SRC):
    pass

def build():
    action('foo.out', 'foo.in')

It's actually a little longer. Part of that is that SCons imports all of the common code into the global namespace. This can save some lines, but it also forces a lot of SCons into the same namespace. This forces each node to be uniquely named, and is more likely to run into conflicts. Anyway, for Fbuild we need to do a couple extra things to get the to cache. First, we mark that the function is cached using the fbuild.db.caches decorator. There's also fbuild.db.cachemethod for decorating methods, and fbuild.db.PersistentObject parent class for caching object creation. We also need to tell Fbuild if the arguments are source or destination files. We do this using python 3.0's annotation support. This tells the database to check to see if source files were modified or if the destination file was removed. If either of these occurred, the function is rerun.

As the builders get nontrivial and more complex, Fbuild becomes a much simpler. For instance, consider SCons's Yacc builder. There are three main things it needs to do. First, SCons has to set up the environment variables for yacc. Next, they register yacc with the C builder. Finally, they have a function that uses the values from the environment and actually evaluates yacc.

Fbuild's current bison builder is much more straight forward:

import fbuild
import fbuild.builders
import fbuild.db
from fbuild.path import Path

class Bison(fbuild.db.PersistentObject):
    def __init__(self, exe=None, flags=[], *, suffix='.c'):
        self.exe = fbuild.builders.find_program([exe or 'bison'])
        self.flags = flags
        self.suffix = suffix

    def __call__(self, src:fbuild.db.SRC, dst=None, *,
            suffix=None,
            verbose=False,
            name_prefix=None,
            defines=False,
            flags=[],
            buildroot=None) -> fbuild.db.DST:
        buildroot = buildroot or fbuild.buildroot
        suffix = suffix or self.suffix
        dst = Path.addroot(dst or src, buildroot).replaceext(suffix)
        dst.parent.makedirs()

        cmd = [self.exe]

        if verbose:
            cmd.append('-v')

        if name_prefix is not None:
            cmd.extend(('-p', name_prefix))

        if defines:
            cmd.append('-d')

        cmd.extend(self.flags)
        cmd.extend(flags)
        cmd.extend(('-o', dst))
        cmd.append(src)

        fbuild.execute(cmd, self.exe, '%s -> %s' % (src, dst), color='yellow')

        return dst

We do some simple configuration to set up the executable and flags. Then we define a function to evaluate the executable. Since we're able to wrap a callable object, we don't have to be aware of anything else happening in the tree. We also don't need to pollute the environment with extra values, when they're only needed in this builder.

It's easy to extend this paradigm to configuration. Say you want to check if a header exists. It's simple to write this:

import fbuild.builders.c
import fbuild.db

@fbuild.db.caches
def check_stdlib_h(builder):
    return builder.header_exists('stdlib_h')

def build():
    static = fbuild.builders.c.guess_static()

    if check_stdlib_h(builder):
        ...
    else:
        ...

It's even easier using fbuild.config though. That package replicates much of the functionality in autoconf to provide a convenient lazy configuration system:

import fbuild.builders.c
import fbuild.config.c.c90

def build():
    static = fbuild.buildesr.c.guess_static()
    stdlib_h = fbuild.config.c.c90.stdlib_h(static)
    if stdlib_h.header:
        ...
    else:
        ...

    if stdlib_h.atof.return_type == 'float':
        ...
    else:
        ...

There are already hundreds of tests defined in fbuild.config. It's a new system, but it's already proven to be pretty useful.

So, that's Fbuild. Hopefully you'll find it interesting. It's still a very young project, and the interfaces aren't stable. But if you still want to check it out, you can find the source here. If you want to try it out, you'll need python 3.0 and git installed. You can check it out with:

git clone http://git.felix-lang.org/r/fbuild.git

There are a handful of examples in the examples directory. These can be run either with examples/examples.py, or run directly as in:

cd examples/c
../../fbuild-light

Where fbuild-light is just a simple wrapper around the fbuild executable. It's used so that you can run fbuild without it being installed.

If you have any questions, feel free to ask on the felix mailing list, or reply to the comments.