Confix

• Introduction
• An Example
• How Confix Works
• The Coders Role in the Build Process
• The Users Role in the Build Process
• Options and Parameters
• Interfaces for Coder and Maintainer
• The Configuration File
• Frequently Asked Questions
• The Sample Utilities Package
• Concept Index
• Function Index
• Variable Index

1 Introduction

Confix is a build tool for source code packages. It doesn't require the package maintainer to write any build instructions by hand at all. Instead, it examines the source code, guesses what's to be done and writes the build instructions for the maintainer¹. Another important feature — perhaps the most important — is the ability of Confix to recognize and compute dependencies. That is, it relieves the package maintainer of the task of manually keeping track of the order in which libraries appear on the link line of an executable, for example, or of the build order inside the package.

Confix is built on top of the standard open source tool Automake, http://www.gnu.org/software/automake/². That is, the build instructions generated by Confix are in fact Automake input files. This way one uses the whole infrastructure supplied by Automake, involving such things as the well-known configure; make; make install sequence, or automatically building source distributions by simply saying make dist, just to mention a few.

• Prerequisites
• How To Get It
• Installation
• A Short Autotools Introduction
• What Confix Adds to the Autotools

1.1 Prerequisites

• Python, at least version 2.3.
• Automake, at least version 1.7.
• Autoconf, at least version 2.58.
• Libtool. Support for Libtool is optional in Confix, so you may leave it out entirely. Best to use version 1.5 because older versions had buggy C++ support. If you are building C libraries, older versions may or may not work.

1.2 How To Get It

The project's homepage it hosted on Sourceforge, http://confix.sourceforge.net. See there for releases, announcements et cetera.

Throughout this document we will occasionally refer to an example source code package which is also available from this site. You may dowload it as a tarball or check it out from source³

1.3 Installation

Confix packaging and installation is done using the Python Distutils package. So far I managed to do a source distribution, both in .tar.gz and .zip, a RPM package, and a “dumb binary distribution”. The easiest to use however is the source distribution, mainly because Confix cannot use the RedHat 7.3 default Python which is 1.5 (can anybody use it?).

• Source Distribution
• RPM distribution
• Dumb binary distribution

1.3.1 Source Distribution

Unpack the file, point the PYTHONPATH environment variable into the root directory of the Confix package, and your PATH environment variable into the scripts subdirectory thereof. For example,

     $ pwd
     /home/jfasch
     $ tar zxplf Confix-1.0.1.tar.gz

Then make a call to Distutils to build and install the source.

     $ cd Confix-1.0.1
     $ pwd
     /home/jfasch/Confix-1.0.1
     $ python setup.py install --prefix=${HOME}/sandbox

Then, provided the prefix you gave wasn't a standard one like /usr or /usr/local, you should update your environment accordingly. For example, you could add the following lines to your ~/.bashrc,

     PREFIX=${HOME}/sandbox
     export PYTHONPATH=${PYTHONPATH}:${PREFIX}/lib/pythonA.B/site-packages
     export PATH=${PATH}:${PREFIX}/bin

Make sure A.B corresponds to the version of Python that you use, such as 2.1 or 2.2.

1.3.2 RPM distribution

Ideally, one would just say

     rpm -Uvh Confix-<version>.noarch.rpm

However, Confix requires at least Python 2.1.3, and I built the RPM using the Python binary /usr/local/Python-2.1.3/bin/python (which I compiled myself). Unfortunately, the RPM package generated by Distutils is not relocatable, so the user is required to have Python installed in /usr/local/Python-2.1.3.

I could have built the RPM with the python2 interpreter which comes with RedHat 7.3, but, then again, this wouldn't help because then I'd have to write #!/usr/bin/env python2 in the first line of the confix.py script. This would break Confix for configurations which don't have python2.

What I want to say: since Confix requires Python 2.1.3 or higher, it is not easy to generate RPM for RedHat 7.3 whose default Python is 1.5⁴. Better to use the “dumb binary distribution” instead.

1.3.3 Dumb binary distribution

A “dumb binary distribution” is simply a .tar.gz or .zip file containing the tree fit for installation. Unpack it, chdir to the implicitly contained prefix directory (for example ./usr/local/Python-2.1.3, note the leading ., which means the current directory), and copy everthing contained therein into your favourite Python root.

1.4 A Short Autotools Introduction

If you ever compiled a GNU source package, you probably have enjoyed the convenience of saying configure; make; make install, without having to manually adjust Makefiles and source code. The tools which make this possible are automake and autoconf. They implement one part of the GNU Coding Standards, namely the part which specifies which make targets must be available to the user who compiles the package. Among these targets are install, clean, info (to generate emacs info files from Texinfo documentation), check (to perform tests), dist (to automatically generate a source distribution in a .tar.gz file), and many others.

configure is a Bourne shell script which performs checks specified by the maintainer. It is not a hand-written script - as you can see from looking at it. Rather, it is "compiled" from several input files written by the maintainer, mostly in the macro language M4. For example, if the package contains C code, configure will search for available C compilers on the user's machine (the host machine), and pick one which is then used to compile the package. Also, it checks for the availability of certain functions or libraries on demand of the maintainer. Based on the results of these checks, configure creates Makefiles in the package subdirectories which have to be built.

configure supports a great wealth of commandline options which parameterize the package build process. One of the most widely used options is --prefix=<prefix>. You use this option when you don't want the package installed at the default location /usr/local (e.g. if you don't have write permissions there), but rather want it installed at $HOME/my-private-packages. (--prefix is the only standard option we use in this manual, so we don't mention the others here.)

make install installs the package into subdirectories of $(prefix). If you haven't passed any other options than --prefix to configure, the header files of the package will be installed under $(prefix)/include, the executables under $(prefix)/bin, and the libraries under $(prefix)/lib. You can explicitly specify locations for each of these by passing specific options to configure; see the autoconf manual for more information.

1.5 What Confix Adds to the Autotools

As you have noticed, a package maintainer is responsible for at least two things. On one side, the maintainer hacks the code, a job which can be quite creative and is fun most of the time. On the other side, there is the task of crafting the build process. This job is annoying, it involves such things as keeping track of include paths, library names, checking for external packages, cross platform portability issues, etc. The GNU tools Automake and Autoconf largely automate the build process, but there is still a non-trivial amount of work left.

Confix's goal is to make build maintenance part of source code maintenance. The maintainer is not required to write lengthy configure.in and Makefile.am like he had to when using plain Automake. Rather, from the content of the package's directories and files, Confix guesses what has to be built and how.

Clearly Confix imposes a few restrictions (features) on the maintainer in order for it to be able to correctly guess. This manual goes to great lengths at explaining what these restrictions are, and how to live with them. The Coders Role in the Build Process has been written solely for this purpose.

2 An Example

Let's start out with a concrete example; it's easier to follow the explanations this way. If you want to immediately try out what's written in this chapter, I recommend you download the companion wx-utils package from the Confix web site (see How To Get It), unpack it, and warm up your fingers. The Sample Utilities Package contains a brief description of what's in that package. Throughout this manual, we will refer to source files of the package as examples.

What we will be doing in this chapter is to play through the different stages of a package,

• Raw source code: pure source code, as written by the developer.
• Instrumented: instrumented package; files generated by Confix suitable for input to Automake and Autoconf.
• Bootstrapped: prepared by Automake and Autoconf, ready for the user to call configure and make.
• Built: configured and compiled by the user.
• Installed: made public by calling make install.

Note that this chapter is about calling Confix to manage your build process. It is not about crafting the source code itself. For this we have a separate chapter, The Coders Role in the Build Process. There we describe in detail what you have to bear in mind when writing code that you intend to manage with Confix. I suggest you read that chapter before you actually start out with your own package.

• The Raw Source
• Instrumenting the Package
• Bootstrapping the Package
• The Configuration File (Short Introduction)
• Building the Package
• Installing the Package
• Distributing the Package
• Summary

2.1 The Raw Source

Here's the structure of the package as the maintainer hand-crafted it, with a few directories and files omitted to make it fit on a page⁵.

     wx-utils
     |-- Makefile.py
     |-- bin
     |   |-- Makefile.py
     |   `-- crypt.cc
     |-- check
     |   |-- Makefile.py
     |   `-- _check_debug_parse_levels.cc
     |-- code
     |   |-- Makefile.py
     |   |-- debug.cc
     |   |-- debug.h
     |   `-- linkassert.h
     |-- crypt
     |   |-- Makefile.py
     |   |-- crypt.cc
     |   `-- crypt.h
     |-- error
     |   |-- Makefile.py
     |   |-- error.cc
     |   `-- error.h
     `-- ext-crypt
         `-- Makefile.py

What strikes are the occurrences of Makefile.py in every directory. The meaning of this file is twofold. First, all directories in the package that do not have a Makefile.py are ignored by Confix. In other words, you have to place a Makefile.py in every directory that you want Confix to instrument.

Second, that file may contain explicit instructions for Confix on how to instrument that particular directory. But this is more advanced stuff; suffice it for now to say that this file is usually empty and only serves as a reminder for Confix to not overlook that directory. If you do feel the need to write something there - such as CVS keyword substitutions in a comment - be sure to use Python syntax for this. See The Module Makefile.py Interface for information about what can be written in Makefile.py. If you are curious, I suggest you take a look at Makefile.py in ext-crypt. This directory does not contain any real code; rather, its Makefile.py contains information on how to use the standard UNIX crypt library. You'll find a explanation in A Simple External Module.

A few words on terminology. Throughout this manual we will use certain terms over and over again. Now's the right time to formally introduce two of them.

• Package: a package is an entity which is subject to be instrumented by Confix. In our example, the tree beginning at the wx-utils directory is a package, and wx-utils is the package root.
• Module: a module is a subdirectory of the package root. A package is said to contain modules. A module normally contains buildable things such as C and C++ files and the like (more on that later). The outcome of a module, after the package has been built, is something like a C or C++ library, an executable, or whatnot.

2.2 Instrumenting the Package

The first step on the way to a successful build (apart from writing correct code and spreading Makefile.py files all over the package tree) is to have Confix examine the package and generate input for the Autotools.

     
     $ pwd
     /home/jfasch/work/wx-utils
     $ confix.py --prefix=/home/jfasch/installed-packages --output
     scanning package (wx-utils) ...
     done with wx-utils
     resolving dependencies ...
     done resolving

Now see what has happened. (See below what the --prefix option is for.)

     
     wx-utils
     |-- configure.in
     |-- Makefile.am
     |-- Makefile.py
     |-- depcomp
     |-- bin
     |   |-- Makefile.am
     |   |-- Makefile.py
     |   |-- crypt.cc
     |   `-- wx-utils_bin.py
     |-- check
     |   |-- Makefile.am
     |   |-- Makefile.py
     |   |-- _check_debug_parse_levels.cc
     |   `-- wx-utils_check.py
     |-- code
     |   |-- Makefile.am
     |   |-- Makefile.py
     |   |-- debug.cc
     |   |-- debug.h
     |   |-- linkassert.h
     |   `-- wx-utils_code.py
     |-- crypt
     |   |-- Makefile.am
     |   |-- Makefile.py
     |   |-- crypt.cc
     |   |-- crypt.h
     |   `-- wx-utils_crypt.py
     |-- error
     |   |-- Makefile.am
     |   |-- Makefile.py
     |   |-- error.cc
     |   |-- error.h
     |   `-- wx-utils_error.py
     `-- ext-crypt
         |-- Makefile.am
         |-- Makefile.py
         `-- wx-utils_ext-crypt.py

Confix created a couple of files in the tree. configure.in is the input for the Autoconf tool. It contains M4 code which will be expanded to bourne shell code by Autoconf in the next step. The various Makefile.am files, in the package root directory and its module directories are input for Automake and will be expanded to Autoconf input in the next step. It does not matter if you do not understand what is going on here. You do not normally need to. If you do want to understand it, we refer you to the Autoconf and Automake documentation. Also, there is a good book about both of them, written by some of the maintainers. Suffice it to say that Confix just did what is otherwise the package maintainer's job: it wrote input files for the Autotools. This is the state of a package as you would check it out from anonymous CVS if it were a regular open source project on the Internet.

There are a few files I did not mention: wx-utils_bin.py, wx-utils_check.py, wx-utils_code.py, wx-utils_crypt.py, wx-utils_error.py, and wx-utils_ext-crypt.py. They contain pickled⁶ Python objects whose purpose is to describe the properties of the respective module. When the package is installed, these description files are installed along with the other content of the package and made available for other packages that use this package. When Confix instruments that other package, it uses these description files to calculate inter-package dependencies (see Module Repository — Inter-Package Knowledge Transfer).

Inter-package dependencies are the reason why one must use the --prefix option when instrumenting a package. Strictly speaking, it is not necessary to specify --prefix when instrumenting the wx-utils package since it does not depend on any other package. Nevertheless, we use it for clarity because our example package could just as well depend on another package.

2.3 Bootstrapping the Package

In Autotools talk, the term bootstrapping refers to the process of converting the input files, configure.in and Makefile.am, to the files the end user sees, primarily configure and Makefile.in. The Autotools are maintainer tools and require the presence of other tools such as Perl and M4 which the user may not have installed at his site. configure.in, for example, is a bunch of M4 macro invocations which, when expanded by Autoconf, turn into an even greater (and even more unreadable) bunch of Bourne shell code, the configure script. Likewise, Makefile.am is a bunch of bells and whistles, subject to be parsed and interpreted by the Perl program Automake. The outcome of a Makefile.am is a file Makefile.in which looks like an ordinary Makefile, only with some placeholders to be substituted by the user, by calling configure (but more on that later).

You see what the whole story adds up to: bootstrapping is using the Autotools to convert the non-portable input files into an equivalent portable form. The only two things that the user is required to have are a Bourne shell, some associated programs like sed, and a reasonable incantation of the make program. All these are available on every Unix or Unix-like system since about 20 years now.

Having all that said, here's how we bootstrap.

     
     $ pwd
     /home/jfasch/work/wx-utils
     $ aclocal -I /home/jfasch/confix/ac-m4 \
               -I /home/jfasch/confix/ac-m4/autoconf-archive/C++_Support
     $ autoheader
     $ automake --foreign --add-missing --copy

An alternative (and more comfortable) way of bootstrapping a package is integrated into Confix. The following command does exactly the steps we have performed so far - instrumenting and bootstrapping.

     $ confix.py --prefix=/home/jfasch/installed-packages --bootstrap

By the way, the Python getopt module recognizes option prefixes, as long as the prefix is unique. boo is a unique prefix of bootstrap, so you can save keystrokes and just yell

     $ confix.py --prefix=/home/jfasch/installed-packages --boo

Here's how the package looks like after bootstrapping, with some irrelevant parts omitted (Autoconf maintains some cache structures in the hierachy, but these add nothing to the big picture). As explained above, what's new is the shell script configure and the Makefile.in files in the package root and the module subdirectories. Apart from these primary files, Autoconf created a few other files which are used during the build process in the next section.

     wx-utils
     |-- Makefile.am
     |-- Makefile.in
     |-- Makefile.py
     |-- aclocal.m4
     |-- config.guess
     |-- config.h.in
     |-- config.sub
     |-- configure
     |-- configure.in
     |-- depcomp
     |-- install-sh
     |-- ltmain.sh
     |-- missing
     |-- mkinstalldirs
     |-- stamp-h.in
     |-- bin
     |   |-- Makefile.am
     |   |-- Makefile.in
     |   |-- Makefile.py
     |   |-- crypt.cc
     |   `-- wx-utils_bin.py
     |-- check
     |   |-- Makefile.am
     |   |-- Makefile.in
     |   |-- Makefile.py
     |   |-- _check_debug_parse_levels.cc
     |   `-- wx-utils_check.py
     |-- code
     |   |-- Makefile.am
     |   |-- Makefile.in
     |   |-- Makefile.py
     |   |-- debug.cc
     |   |-- debug.h
     |   |-- linkassert.h
     |   `-- wx-utils_code.py
     |-- crypt
     |   |-- Makefile.am
     |   |-- Makefile.in
     |   |-- Makefile.py
     |   |-- crypt.cc
     |   |-- crypt.h
     |   `-- wx-utils_crypt.py
     |-- error
     |   |-- Makefile.am
     |   |-- Makefile.in
     |   |-- Makefile.py
     |   |-- error.cc
     |   |-- error.h
     |   `-- wx-utils_error.py
     `-- ext-crypt
         |-- Makefile.am
         |-- Makefile.in
         |-- Makefile.py
         `-- wx-utils_ext-crypt.py

2.4 The Configuration File (Short Introduction)

We noticed (at least) one annoying thing. Everytime we called Confix we had to specify that --prefix option. You will be using Confix more than once during package maintenance, and I bet you will want to have a way to get around this. This is the time to give a short introduction to the configuration file. (See The Configuration File for a more thorough explanation of the configuration file and a list of all the settings available.)

At startup, Confix reads a file called .confix in your home directory⁷. For our purposes the following simple configuration file suffices.

     
     myconfig = {
         'PREFIX': '/home/jfasch/installed-packages',
         'BUILDROOT': '/home/jfasch/tmp/build'
     }
     
     PROFILES = {
         'default': myconfig,
         'myownconfig': myconfig
     }

This file is nothing but a Python script, it's just that it is not executed by the Python interpreter directly, but rather by Confix itself. You can put any code you like in it, but most of the time it will be enough to define one or more profiles. A profile is a set of parameters and settings which are valid for one Confix session. Profiles are made visible to Confix through the global PROFILES dictionary. The user selects a particular profile through the --profile option, for example

     confix.py --profile=myownconfig --boo

or just

     confix.py --boo

because 'default' is the name of the default profile, and that entry in the PROFILES dictionary is set up to point to the same profile as 'myownconfig'.

That's enough explanation for the time being. The only settings we are interested in so far are 'PREFIX' and 'BUILDROOT'. (We haven't got around to using 'BUILDROOT' yet, but we will so soon.)

Now that we have got rid of the --prefix parameter, let's continue with our package.

2.5 Building the Package

So far, until now, we have performed the job of the package maintainer. We used Confix to write Autotools input for us, and used the Autotools to create a configure script for the user. Now let's switch our mode and become the user of our own package. You probably have compiled an open source package before and have already performed the configure; make; make install dance. That's what we will be doing in this section.

Before we begin, let us state one important issue: we never build in the source directory. Rather, the build directory is a distinct directory. Through this policy we can place the sources in a directory which is subject to nightly backup, or on a RAID system, or both, for example, whereas the build directory can be placed in a location which is much cheaper. The build directory contains only data which are easily reconstructed from the source, so it is unnecessary to have it backed up.

Now here we go. We create the build directory, chdir to that directory, call configure (in the package root), and then make.

     
     $ mkdir -p ~/tmp/build/wx-utils
     $ cd ~/tmp/build/wx-utils
     $ /home/jfasch/work/wx-utils/configure \
              --prefix=/home/jfasch/installed-packages
     $ make

(configure and make generate lots of messages to standard error, we omit them for clarity.)

Notice the usage of the --prefix option here. We passed that option to Confix when we were instrumenting the package; its purpose there was to point Confix to module description file of other packages that the wx-utils packages eventually depends on. The meaning of the --prefix option with configure is a bit different, albeit related. The prefix is the location the package will get installed to (we will cover installation in the next section). The relation to the module description files is that installing a package involves installing the module description files into a location called the module description repository. (Remember how we generated the module description files in Instrumenting the Package.) Finally, installed module description files are used by Confix when it instruments a package.

As always, there is a simpler way to do this. Confix can start configure and make for you. From the package root, call

     $ pwd
     /home/jfasch/work/wx-utils
     $ confix.py --builddir=/home/jfasch/tmp/build/wx-utils --configure

or alternatively, from anywhere you want,

     $ confix.py --packageroot=/home/jfasch/work/wx-utils \
              --builddir=/home/jfasch/tmp/build/wx-utils \
              --configure

I admit that this is not much simpler than calling configure by hand. But recall that we have set the 'BUILDROOT' variable in the configuration file to /home/jfasch/tmp/build. The effect of this setting is that Confix assumes the package's build directory is a subdirectory wx-utils of /home/jfasch/tmp/build (wx-utils is the basename of the package root, that's how Confix determines the name).

With this you can say,

     $ confix.py --configure

It's exactly the same with make, you just say

     $ confix.py --make

or eventually, the equivalent of calling make -k,

     $ confix.py --make --targets=-k

Put realistically, after a while of using Confix you will just end up using the following power user composite command,

     $ confix.py --boo --configu --make --targ=-k

which instruments the package, bootstraps it, configures and builds it in one big swoop. You will even set your Emacs compile-command variable to something like

     $ confix.py --packageroot=/home/jfasch/work/wx-utils \
                 --boo --configu --make --targ=-k

(Note that it is not necessary to re-bootstrap and re-configure a package if you only want to compile it. Emacs gives you a chance to edit the command before it is executed, and remembers the last billion or so of commands you executed. So what I do at this point is to remove --boo and --configu if I only want to compile.)

2.6 Installing the Package

Package installation is simple,

     $ pwd
     /home/jfasch/tmp/build/wx-utils
     $ make install

or, alternatively, with the current working directory anywhere you want,

     $ confix.py --packageroot=/home/jfasch/work/wx-utils \
                 --make --targets=install

This copies all files which should be visible to other code to the installation directory under $(prefix) (as given by configure --prefix=<prefix>). Which files are visible after installation is determined by Confix according to several rules which are explained in Installation — Going Public. For example, if a particular module is a C or C++ library, then that library is be installed into $(prefix)/lib. Typically, a library is accompanied by one or more header files; these are installed into $(prefix)/include, or subdirectories thereof.

One point to note here, again, is that installation covers module description files as well. These files are installed in a subdirectory repo of the installation directory, the module description repository. (It is important that Confix users be aware of that fact because the repository is a concept which is central to the inner workings of Confix, as will be explained in Module Repository — Inter-Package Knowledge Transfer.)

2.7 Distributing the Package

So far we covered the life cycle of our test package from its raw source form to its installed incarnation. However this is not the only way a package can be treated. Rather, this is only the way the package's maintainer uses to write, build and test his package.

If you ever compiled an open source package, then you probably have taken the following steps.

1. Download the source distribution from the Internet, likely in compressed form, as a .tar.gz file.
2. Unpack it, thereby creating the source tree rooted by the package root.
3. Create a separate package build directory, chdir to that directory, and call configure, make, and make install.

So how do these steps correspond to the maintainer's steps which were described in this chapter? Actually, the package user performs a subset of the maintainer's work. Remember the explanation in Bootstrapping the Package, where we pointed out that the user is by no means required to have the Autotools available. Bootstrapping is the point after which the package user's work and the maintainer's work are the same. configure; make; make install.

What is still unclear is how the source distribution is made. Clearly this is not done by hand, as this job is tedious and error prone. Automake implements a set of makefile targets for this,

• dist creates a .tar.gz distribution.
• dist-bzip2 creates a .tar.bz2 distribution.
• dist-shar creates a shell archive distribution.
• dist-zip create a zip file (popular on Windows).
• dist-tarZ creates an old-style .tar.Z distribution.

Consult the Automake documentation for details about the various dist targets and how they work in detail. For example, to create a source distribution of our wx-utils package, we call

     $ pwd
     /home/jfasch/tmp/build/wx-utils
     $ make dist

or, through Confix,

     $ confix.py --make --targets=dist

This will create the distribution tar file in the package's build directory. The resulting distribution file's name has the form <packagename>-<packageversion>.tar.gz. Unless otherwise specified, <packagename> is the name of the package root directory, and <packageversion> is 0.0.0. For example, if we made a source distribution from our wx-utils package, the file name would be wx-utils-0.0.0.tar.gz. See The Toplevel Makefile.py Interface for ways to specify the package name and version.

2.8 Summary

In this chapter we have primarily learned about what you have to do when you are a package maintainer, and how Confix helps you do your job. We also covered shortly the role of the package user. We also touched a few concepts involved: dependency calculation and the module description files.

We presented lots of command lines as well as one helpful feature, the configuration file. Here's a recap of the most important commands we saw. (We assume a .confix file with contents about as in The Configuration File (Short Introduction).)

• Bootstrap. This is the process of instrumenting the package and generating all the files which are necessary for the end user.

            $ confix.py --bootstrap
       

  in the package root directory, or, as always,

            $ confix.py --packageroot=/home/jfasch/work/wx-utils \
                    --bootstrap
       

  in any directory.

• Configure. Preparing the package for the ultimate build. Performing checks and creating a Makefile in every directory of the package.

            $ confix.py --configure
       

• Make, Install. Compiling the package and installing it.

            $ confix.py --make
       

  or

            $ confix.py --make --targets=-k
       

  or

            $ confix.py --make --targets='-k install'
       

  or, without Confix, in the package build directory,

            $ make install
       

3 How Confix Works

This chapter gives an overview of the concepts involved. It describes each step Confix takes when it instruments a package. The description may look more technical than necessary to you. But, on the other hand, I am quite certain that it is easier for the user to understand the rest of the manual if the concepts are clear from the beginning. Confix offers an increasing range of possibilities where the user can hook in to influence the build process. To fully utilize this flexibility, it is crucial to understand how Confix works.

Don't be scared though, we only scratch the surface here. The details are buried in the appendixes which you can consult if there is need to. (If the appendixes do not contain enough material to satisfy you, then I haven't yet found time to do it right. Complain to me, or continue to ask questions, in either case the probability increases that I write it down.)

• Scanning --- Gathering the Pieces
• Resolving --- Determining the Build Order
• Checks --- The Autoconf Way of Portability
• Module Repository --- Inter-Package Knowledge Transfer

3.1 Scanning — Gathering the Pieces

Before Confix can instrument a package it must first determine the modules which have to be instrumented, and the properties of these modules.

We saw in the example in the preceding chapter that Confix traverses the package beginning at the package root. It examines every directory on the way, and if that directory contains a Makefile.py file, the directory is considered a module of the package and is remembered for later instrumentation. (See The Module Makefile.py Interface for a detailed explanation of what that file can contain.)

Once Makefile.py is found in a directory, Confix further examines this directory by interpreting its Makefile.py, and by examining the directory at the file system level (i.e., looking what files it contains). From this, the module's properties are determined. Properties include such things as

• Buildable objects. Confix uses these to generate build instructions for anything that can be built. For example, there is one buildable object for every C++ source file in a module; the same holds for other file types, such as C source files, C header files, and Lex and Yacc files. Apart from these buildable types that comprise single source files, there are more complicated composite buildable types — a library of compiled object files, for example, or an executable.

  Confix uses a great deal of heuristics to determine what has to be built, and how⁸. For example, if it finds out that a C or C++ source file contains the main() function, then it concludes that the package maintainer wants this file to become an executable, and links it with all other files in the module that do not contain the main() function. Contrariwise, if it finds that no source file of a module contains main(), then it decides to build a library instead. However, some of Confix's heuristics are a bit, well, simplistic, and you need to help it a bit. See What Will Be Built for a more detailed explanation of Confix's heuristics, and how to help out if something goes wrong.

• Provide and Require objects. These important concepts are central to computing the inter-module dependencies. A module A is said to depend on another module B if module B provides something that module A requires. For example, consider a file a.c in module A which #includes a file b.h which is located in module B. This is a situation where module A requires (through the #include <b.h> directive in its buildable file a.c) a file b.h which is provided by module B.

  
  

  

  In the remainder of this document, the words provide and require will often be used as a noun. This is to clarify that we mean the properties of a module which are the fundamentals of the resolving process (which will be explained in the next section).

  The fact that a buildable in module A contains #include <b.h> is said to be a require, whereas the fact that module B contains b.h is said to be a provide. The latter satisfies — resolves — the former, and thus module A is said to depend on module B.

• Checks. One of the most basic albeit powerful features of Autoconf is its ability to check the availability of certain features, and to provide that information (available yes/no/what/how) to the build process. Without Confix, these checks have to specified in the configure.in file by the maintainer. With Confix, checks are properties of modules and are mostly determined by the module's buildable objects. See Checks — The Autoconf Way of Portability further down in this chapter for more about checks, and Module Repository — Inter-Package Knowledge Transfer for how Confix carries checks across package boundaries.
• Module content. A module content is an object that describes what another module has to do if it uses this module. For example, the content of a module whose outcome is a C library describes to a using module what the name of the library is and what include path must be set to find the library's accompanying header files.

So much for the internal structures of Confix. Sure there is more to it, but this is beyond the scope of this manual. However, if you are interested: it is actually implemented this way, and you should easily be able to find the associated Python classes by using something like grep -i in the source code.

3.2 Resolving — Determining the Build Order

Resolving is the task of building a directed acyclic graph of modules, the dependency graph. In the preceding section we saw an example which detailed the concept of provide and require and what these have to do with module dependencies. According to this example, if the modules A and B are nodes in the dependency graph, then a graph edge leads from A to B if (and only if) A depends on B.

Confix builds this dependency graph by matching the requires of all modules against the provides of all modules. If a require of module A is matched (resolved) by a provide of module B, then we have an edge from A to B in the graph.

Note that the resolving process is not local to the package that is being instrumented. A module of the package may require something that is not provided locally, but rather by a module of another package. How can this dependency be resolved? Recall how we came across the module description files and the module repository multiple times in this manual (an indication that it must somehow be central to Confix). It's these repository files that carry the information what items an installed module provides and requires. When Confix builds the dependency graph, it not only determines the properties of the modules in this package, but also reads the module description repository⁹ to determine the properties of installed modules.

Now what has a dependency graph to do with building a package? The answer is build order. For example, consider our example wx-utils package from the preceding chapter. There is a module called bin which contains executables. On of these, built from the source file crypt.cc, is a simple program which accept two arguments from the commandline and passes these to the WX::Utils::Crypt::crypt() method, which is defined in the crypt module of the wx-utils package. Through this relationship, we say that module bin depends on module crypt.

Confix sees dependencies through the occurence of include directives in source files. bin/crypt.cc contains a line #include <WX/Utils/crypt.h> — in other words, module bin requires a file WX/Utils/indent.h. On the other hand, the module crypt contains a file crypt.h which is provided as WX/Utils/crypt.h¹⁰

Now consider what happens if module bin was built before module crypt. bin contains a C++ executable, with the main() routine in bin/crypt.cc. This executable contains a reference to a routine WX::Utils::Crypt::crypt() which is defined in crypt/crypt.cc. When it comes to linking the executable, the linker will try to satisfy that reference, which it cannot because module crypt has not yet been built.

Hence, module crypt must be built before bin. Confix ensures this by examining the dependency graph and calculating a correct build order from it¹¹. Dependency cycles (see Build Order — How Things Can Easily Go Wrong) are clearly an error; this error is detected by Confix, and reasonable messages are output so that the user will be able to break up the cycle.

3.3 Checks — The Autoconf Way of Portability

One approach to cross-platform portability is to wrap platform dependent code in #ifdefs, and to create an if-then-else ladder where you switch across architectures like this,

     #ifdef Solaris
     // Solaris code here
     #elif (defined Linux)
     // Linux code here
     #elif
     // blah
     #endif

Autoconf's (and, as Confix sits on top of it, Confix's) approach to portability is a bit different. It checks for features instead of for architectures, so that you do not anymore switch across architectures, but rather react on the presence or non-presence of features. Autoconf and Automake provide a bunch of so-called checks: M4 macros which expand to Bourne shell code. What they do is best explained with an example.

A C++ file (which is buildable object) certainly requires the availability of a C++ compiler. So, the buildable automatically contributes the appropriate Autoconf check to its enclosing module. Confix takes care that all checks gathered this way be written to the configure.in file in the package root, as if the maintainer had written them.

The effect of this check is seen when the user calls configure. The configure script executes code that tries out several things, such as looking if the CXX environment variable is set, checking if it points to a valid executable, checking if that executable can be used as a C++ compiler, what the flags and options are that have to be passed to that compiler to function properly, and so on. The effect of this particular check is that some well-defined sh variables are set for further use in configure, that code is generated to substitute specific patterns in the various Makefile.in's in the package tree, and so on. As for sh variables: C++ specific checks need them to be able to call the appopriate compiler to perform their work. For example, a check that determines if stringstream is available and works correctly (sigh!) needs to generate code that uses stringstream, and then call the C++ compiler to compile and eventually link the program.

This leads us to another feature of Confix¹² associated with Autoconf checks: it respects ordering of checks. For example, the check for the C++ compiler has to be performed before a check for a particular language feature, such as the stringstream class.

Checks are not only contributed implicitly. There are several ways that checks can be contributed explicitly by the maintainer. See Explicitly Adding Checks, The Toplevel Makefile.py Interface, The Module Makefile.py Interface, and The Buildable Interface for various ways to explicitly add checks¹³.

See also Module Repository — Inter-Package Knowledge Transfer how the build process of one module is automatically crafted to perform all checks of all installed modules it uses.

3.4 Module Repository — Inter-Package Knowledge Transfer

Suppose that you are in the situation where you have, say, ten installed packages which amount to twenty or so libraries. You are writing an executable which directly uses features from three of these libraries, and carefully craft your link line to contain these three libraries. Most likely this is not enough because one of these libraries is not self-contained — it depends on another four libraries. And so on.

Without Confix, you'd have to explicitly know about these dependencies, and hand-craft your link lines and include paths accordingly. Confix's process of computing a correct build order has already been outlined in Resolving — Determining the Build Order. There we pointed out that computing the build order involves topologically sorting the dependency graph. Now, that graph does not only contain modules which are local to the package, but also those modules which have been installed — whose module description files are available in the module repository. In other words, the graph presents a global view of module dependencies.

So, a topologically sorted list of the graph's nodes (the modules) gives enough information to compute the following important items.

• A correct build order. Only the modules of the current package must be taken, of course, as it doesn't make any sense to build an installed module.
• A correct include path.
• A correct link line.

This is only one example (although the most important) of how Confix coordinates packages and their modules. Autoconf checks are another example.

Say one module (a C or C++ library, as always) performs a check for, say, the availability of the JPEG library. That library is not maintained by Confix, but rather part of the Linux distribution (they don't manage their stuff with Confix yet). This check succeeds, and the module ends up referencing a function which is defined in the JPEG library. The check not only checked for the availability, but also set a sh variable that contains the name of the library. This variable normally has to be put on the link line of an executable so that the library is seen by the linker and the symbol can be resolved.

Now, sadly, the check is only performed locally, in the package that contains the module that directly uses the JPEG library. Any executable, of another package, that uses that module will get a linker error because it does not perform the same check (and thus does not add the JPEG library to the link line).

Here's where Confix and its module description repository helps again. Autoconf checks are properties of the module and thus show up in the module description file which is read by Confix. Whenever a module uses another module, Confix automatically arranges that it performs all the checks of that module.

4 The Coder's Role in the Build Process

One of the advantages of Confix is that you are not required to write any explicit build instructions to control the build process of a package and its modules. Confix tries to determine everything it needs to know from the files it has to write build instructions for. However, nothing is for free. Automatisms generally tend to cut down flexibility of the underlying mechanisms, and unfortunately Confix is not an exception to this. Confix imposes a few rules on the package maintainer, and for sure you cannot do everything you could do if you used the Autotools directly¹⁴. But read on to get a better judgement.

In this section we will explain basic properties of the build process, mainly by outlining how Confix works. As opposed to the previous section, How Confix Works (where we explained how Confix works), in this chapter we will put more focus on the package maintainer. We will try to make clear what the relevant package properties are, and how the maintainer can/should direct Confix to best determine these. Most of the package's build properties can be explicitly configured by the maintainer. However, this chapter deals with the properties and the involved automatisms Confix implements, and refers to Interfaces for Coder and Maintainer otherwise. The chapter is a bit lengthy as it goes to technical details whenever I feel that I need to justify why a particular restriction is the way it is. If you are curious, I suggest you read the chapter. If you are impatient, I suggest you only read the summary at the end of the chapter (Summary of Rules and Restrictions) and uncritically accept what is there.

To get an overview, here are some of the package properties we will be talking about.

• What files will be built. Just like the Autotools, Confix uses file extensions for its decision what is built and how.
• Installation locations. It makes sense to install header files (for example) into subdirectories of the public $(prefix)/include directory, primarily to prevent file name clashes when several packages are installed in the same location.
• Build order. We have touched this several times — the maintainer implicitly specifies the build order through provides and requires.
• What is public, what is private? For example, you may have header files which are only used during build, whereas others should be made publicly available by installing them.
• Checks. Confix provides various ways for the maintainer to add Autoconf checks to the build process.

Further, we will mostly be talking about C and C++ in this chapter because these are the only languages which are supported at the time of this writing¹⁵. The chapter will get deeper structure as other languages arrive.

• What Will Be Built
• Installation --- Going Public
• Build Order --- How Things Can Easily Go Wrong
• Explicitly Adding Checks
• Performing Automatic Tests
• Using Third Party Libraries
• Restrictions Imposed on the Package Structure
• Using GNU Libtool
• Summary of Rules and Restrictions

4.1 What Will Be Built, and How

Building C¹⁶ code is usually done in several steps. First, source files are compiled into object files. Then, object files are grouped together to form some kind of compound object. One example of a compound object (the simplest) is an archive, also known as static library. Another form of a compound object is a shared library, which is similar to a static library, but not quite¹⁷. The incarnation of a compound object that most computer users know is an executable.

The following section describes how Confix decides how source code is built, and how to group the compiled object files into compound objects.

• Building Source Files
• Library or Executables?
• Generated and Explicit Names

4.1.1 Building Source Files

From the file names in a directory, Confix determines what source files should be built. If a directory contains files which boil down to compiled object code, Confix assumes that either a library or one or more executables will be built. (See below for heuristics that help Confix decide which.) Confix uses much of the Autotools functionality to accomplish the task of building object code.

The Autotools recognize several filename extensions and generate make rules accordingly, and so does Confix. Currently, the file extensions recognized are

• .c: Plain C code
• .cc,.cpp: C++ code
• .l: lex, generating C code. Autoconf will determine the lex derivative, so you may end up compiling with flex under the hood, for example.
• .ll: lex, generating C++ code. Note that you may encounter portability problems if you are using this feature because not every lex version on every platform will be capable of generating C++ output. You will then have to force your users to install one lex version that is capable of that. The alternative is to accept that lex is somewhat, well, stupid, and assume that the user's lex version can do no more than K&R C, and name your lex files .l. Note also that there is no general advice as to which lex version or derivative your code is best compiled with. This topic is completely left to Autoconf which will determine the version — proper or improper.
• .y: yacc, generating C code. Again Automake determines the derivative and reacts properly.
• .yy: generating C++ code. Same problem as with lex.

By default, unless stated otherwise, Confix tries to build every file in a directory that it recognizes. Sometimes however you want to keep source files in a directory that you do not want to have built, probably because they do not compile yet. You can tell Confix to ignore these files; see the IGNORE_FILE() function in The Module Makefile.py Interface.

4.1.2 Library or Executables?

Before Confix can generate build instructions for compound objects, it has to decide which compound object(s) should be built. Recall section Scanning — Gathering the Pieces where we explained the concept of buildable objects for single files. Internally, every source file is wrapped into a (Python language) object which is responsible for generating build instructions for that file. Another responsibility of a single-file buildable object of C language type is to maintain a certain property that tells Confix whether the source file contains the main() function.

• Library. Confix puts all compiled files of the module together into a library if none of the source files has the main() funtion defined. By default, Confix generates a static library; see Using GNU Libtool for explanation of how to build shared libraries.
• Executable(s). If any of the files has main() defined, Confix divides the files of the module in two groups: those which have main() and those which don't. Every file that has main() becomes an executable, and it is linked together with those files that don't have main().

So how does Confix determine this “have the main() function” property? It scans every source file and employs a simple minded parser to search for patterns that look like as if they were the definition of main(). But this parser is not a real C/C++ parser, and it is likely to produce false results. Mostly it will see a main() definition where there is none, rather than the other way around. If this is the case, you'll probably see a linker error message that tries to tell you that linking failed because main is undefined. You then have two choices; either search in the offending source file for occurences of main and change them to primary or something; or you use one of the Confix interfaces to explicitly set the source file's 'MAIN' property to tell Confix that there's no main() in the file¹⁸. See the MAIN() and FILE_PROPERTIES() functions described in The Buildable Interface, or the FILE_PROPERTIES() function described in The Module Makefile.py Interface for ways to influence Confix in this respect.

• Generated and Explicit Names

4.1.3 Generated and Explicit Names

Confix tries to generate instructions to successfully build a package, ideally without any input from the package's maintainer. However, in doing so, Confix tries to not bring the package in conflict with other packages. It is likely that more than one package is installed in the same location — this is the common case (the default installation prefix is /usr/local). Now, for example, consider that two packages build executables, and that the names of two different executables are equal. These two executables would overwrite each other.

To prevent this situation, Confix does not simply take the name of the source file that has the main() function defined, as this would easily lead to the situation deescribed above. Rather, Confix “mangles” the name from the package name, the module name, and the source file name. While this is ok for test programs that are run automatically (see Performing Automatic Tests for more), you probably want to explicitly specify a name for these executables that you want to make visible to users. To do this, you set the 'EXENAME' source file property; see the EXENAME() and FILE_PROPERTIES() functions described in The Buildable Interface, or the FILE_PROPERTIES() function described in The Module Makefile.py Interface.

A similar conflict could happen with libraries, and Confix solves this problem the same way. Mangled library names are not normally a problem for the user because Confix recognizes module dependencies and generates link lines automatically. However, until Confix achieves world domination, it is likely that the libraries that are built by Confix are used by code that is not built by Confix¹⁹. In this case you have to give your library a publicly understandable name by using the MODULE_PROPERTIES() or LIBNAME() functions described in The Module Makefile.py Interface.

4.2 Installation — Going Public

One important part of build management is the installation location. The Autotools are pretty clear about what goes where; they state, for example, that header files go into $(prefix)/include, libraries go into $(prefix)/lib, executables go into $(prefix)/bin, and so on.

There are more aspects of installation, two of which we will cover in this section. First, you need a way to differentiate between files which will go public and files which won't. Second, you may want to put your header files into subdirectories of $(prefix)/include.

• What Goes Public
• Namespaces
• Installation --- A Short Example
• Restrictions

4.2.1 What Goes Public

When you are building a library, you will most likely have header files which you want to install along with that library. But, on the other hand, you will likely have header files whose purpose is an internal one, and which no-one needs anymore once the library is built. You do not want these to be installed.

To have Confix automatically differentiate between public and private header files, you simply prefix the private files with an underscore. Private files are then visible to modules within the same package, but they don't get installed and thus are not visible to modules from other packages.

4.2.2 Namespaces and #include

In C and C++, one often has the problem that certain symbols in one module conflict with symbols in modules of fellow developers. People end up adding prefixes to their symbol names to make them unique, which works but makes typing the names rather annoying. To help with this, C++ namespaces were invented. You put all the names which would otherwise have a common prefix into a namespace with that name instead. This way you only need to keep namespace names unique which is much easier.

A similar problem arises when all developers install their header files to a common location. People have to keep header file names unique, much like they have to keep symbol names unique in C.

While the former problem can be solved using namespaces, the latter problem lacks a standard solution. Confix tries to solve it by installing header files into subdirectories of that common location. These subdirectories are determined from namespaces that occur in the file. For example, if a file public.h contains the following,

     namespace MyOwnCode {
     namespace MyLibrary {
     
     // ...
     
     } // /namespace
     } // /namespace

then Confix guesses that the file should be installed as MyOwnCode/MyLibrary/public.h. Note the terminating comments which are vital to Confix as it is not a C++ parser. Other modules will have to include this file with #include <MyOwnCode/MyLibrary/public.h>. Note that files in the same directory still must include it with #include "public.h" because at this time the file is not yet installed.

4.2.3 Installation — A Short Example

Here's a little example that demonstrates the issues outlined above. The package contains a module my_library which builds a library. The library's interface is inside the C++ namespace MyLibrary which is itself inside a namespace MyOwnCode, and it is defined in a public header file public.h. During the build process of the module a second header file _private.h is used, but this header file is not part of the interface and thus must not be installed.

The source tree layout is as follows.

     <packageroot>
     `-- my_library
         |-- Makefile.py
         |-- _private.h
         |-- impl.cc
         `-- public.h

The content of public.h is (note the /namespace marker)

     #ifndef my_library_public_h
     #define my_library_public_h
     
     namespace MyOwnCode {
     namespace MyLibrary {
     
     // whatever interface definition
     
     } // /namespace
     } // /namespace
     
     #endif

After bootstrapping, configuring, and installing the package, the $(prefix) directory looks as follows,

     $(prefix)
     |-- include
     |   `-- MyOwnCode
     |       `-- MyLibrary
     |           `-- public.h
     |-- lib
     |   `-- lib<packageroot>_my_library.a
     `-- repo
         `-- <packageroot>_my_library.py

Note how the library interface public.h has been installed in a hierarchy MyOwnCode/MyLibrary under the include directory. The directory hierachy an exact mirror of the namespace hierarchy which is defined in the module. Foreign code (and other modules from the same package) that uses our library generally pulls in the definitions like this

     #include <MyOwnCode/MyLibrary/public.h>

4.2.4 Restrictions

One important restriction is imposed through this (and through the way Automake lets the maintainer hook into the build process): you can have only one namespace per directory.

Why is this? Say you have a second interface definition for the library, in a file public2.h in the same directory, which looks as follows.

     #ifndef my_library_public2_h
     #define my_library_public2_h
     
     #include "public.h"
     
     namespace MyOwnCode {
     namespace MyLibrarySecond {
     
     // second interface definition
     
     } // /namespace
     } // /namespace
     
     #endif

This file includes public.h. As that file is in the same directory as public2.h, public2.h includes public.h like #include "public.h". However, due to the different namespace hierarchies, the installed incarnation of the interface definitions is spread into two directories, like

     $(prefix)
     `-- include
         `-- MyOwnCode
             |-- MyLibrary
             |   `-- public.h
             `-- MyLibrarySecond
                 `-- public2.h

Now, consider user code that says,

     #include <MyOwnCode/MyLibrarySecond/public2.h>

This code will get a compilation error. The C preprocessor will complain about the #include "public.h" in public2.h which cannot be found. The include path generally only points in $(prefix)/include, and never in any subdirectories thereof. Hence, what works during compilation of the package, where public.h and public2.h are in the same directory (even with different namespaces), will cease to work for the installed case.

This restriction is not that stringent though. If for some reason you are forced to have different namespaces in files that are in the same directory, you can always disable the offending automatism. Edit the Makefile.py in that directory, and specify the installation location for all headerfiles in that directory. Say you want both public.h and public2.h installed into the directory MyOwnCode/MyLibrary, then you add the following to Makefile.py.

     INSTALLDIR_H('MyOwnCode/MyLibrary')

4.3 Build Order — How Things Can Easily Go Wrong

Confix uses requires and provides to match modules against each other, and to compute a correct build order. Now what is a correct build order? A formal definition could be, for example, “If module A depends on module B, then module B must not use any bit of module A”.

This may seem trivial, but if you think of it more deeply, you'd come to the question, “Why shouldn't module B #include header files from module A? They are present all the time, even if module A is not yet built.”. The latter is not quite true. Remember that we said above that even modules within the same package must include another module's include files as if they were foreign code. They have to apply the full namespace-to-include-directory mapping, simply because they must compile even after they have been installed (see the example above, where public.h was included from another public header file, public2.h). In the local case, however, this hierarchy is not present, and big tricks are applied by Confix to to make that constellation compile happily. These tricks are outside the scope of this manual²⁰, suffice it to say that the include files are not present all the time for all modules in a package. Rather, a module has to be built before its header files can be correctly included.

What follows is the restriction as stated above: If module A depends on module B, then module B must not use any bit of module A.

4.4 Explicitly Adding Checks (and Programming with Them)

With stock Autoconf, you normally list checks in configure.in. For example, if your code were using C++ string streams, you'd run into problems getting it to compile with both a pre-standard and a standard C++ compiler. To get around this, you'd have to use conditional code. The Autoconf way for this is to check for availability of one or the other, and to define the neessary preprocessor macro accordingly. In our case, the check is is AC_CXX_HAVE_SSTREAM (from the Autoconf macro archive which you get with Confix), and the macro it defines is HAVE_SSTREAM.

With stock Autoconf, you'd write the check in configure.in,

     ...
     AC_CXX_HAVE_SSTREAM
     ...

Then, in the code, you react on what the check determined for you. In particular, AC_CXX_HAVE_SSTREAM sets the HAVE_SSTREAM macro to 1 if the standard C++ header file <sstream> is available and defines std::stringstream, else the macro is left undefined. Your code looks as follows²¹,

     #ifdef HAVE_CONFIG_H
     # include <config.h>
     #endif
     
     #ifdef HAVE_SSTREAM
     # include <sstream>
     #else
     # include <strstream.h>
     #endif

One problem with this approach is that the check is placed in configure.in, whereas the results of the check (in this case, the HAVE_SSTREAM macro) are used in the various source files of the package. You do not automatically correlate between these locations. For example, if you change the source to not check HAVE_SSTREAM for some reason (e.g., you do not use stringstream anymore), you might want to consider removing the check from configure.in. However, you cannot be sure that no other source file in the package uses HAVE_SSTREAM, so you have to grep over the package source to be sure. Or the other way around: you remove your usage of HAVE_SSTREAM and don't care about the check. The result is a bloated configure.in file with lots of unnecessary checks, slowing down the build process.

The Confix approach, as always, is to determine from the source what checks have to be applied. However, it cannot, for example, guess from your usage of HAVE_SSTREAM that it must apply the check AC_CXX_HAVE_SSTREAM — this is, well, impossible. Confix simply cannot know that HAVE_SSTREAM is the result of the check AC_CXX_HAVE_SSTREAM²². The maintainer must direct Confix to apply the right check. This is ok because the maintainer usually knows about the effects of checks.

So, with Confix, you write code similar to above, but with one comment line more.

     #ifdef HAVE_CONFIG_H
     # include <config.h>
     #endif
     
     // CONFIX:CHECK(AC_CXX_HAVE_SSTREAM())
     #ifdef HAVE_SSTREAM
     # include <sstream>
     #else
     # include <strstream.h>
     #endif

Note the comment line: it starts with CONFIX:, followed by an instruction which adds the check AC_CXX_HAVE_SSTREAM to the containing package²³. Remember how we saw in Checks — The Autoconf Way of Portability that Confix takes care of the check's correct placement in configure.in, and of its uniqueness there.

So much for adding checks in the source code. Generally, these comment things are the interface to the buildable objects which resemble the respective source file. The part of the comment line which reads “CHECK(AC_CXX_HAVE_SSTREAM())” is actually nothing but Python code which calls a function CHECK() which is defined as part of that interface, and which expects a check object as a parameter. The “CONFIX:” prefix is C++ specific and marks occurences of such interface utilization. See The Buildable Interface for a detailed description of the interface and a list of what else you can write in source files.

4.5 Performing Automatic Tests

Automake provides support for automatically testing your code. All you have to do is to write test programs (of arbitrary complexity) which take no arguments, and which return a zero exit status on success and non-zero exit status on failure. Confix recognizes test programs by their filename prefix: you prefix the files containing the main() routine with _check.

Once you have bootstrapped, configured and built your package, you run the test suite by simply saying,

     $ pwd
     /home/jfasch/tmp/build/wx-utils
     $ make check

or, as we all know,

     $ confix.py --packageroot=/home/jfasch/work/wx-utils \
                 --make --targets=check

When the tests are finished, a statistic is given about the number of tests passed and tests failed.

The wx-utils package contains a growing number of self-tests, they are part of the check module²⁴.

4.6 Using Third Party Libraries

So far we have only talked about instrumenting source code packages with Confix. We have heard about the module description repository and its role in building relationships between the packages that are maintained with Confix. This is fine if all of the software in a project is built using Confix. While this would be nice, it is not quite realistic. For example, you would not easily convince the maintainers of the GNU Readline library to convert their build to Confix²⁵.

We heard that the module description repository contains information about every module that was built (and installed) using Confix. This information consists of things like

• Provides and requires: these are the basic building blocks of module dependencies. We have learned about them already.
• Checks: a collection of Autoconf checks that were performed when the respective module was built. As we heard in Module Repository — Inter-Package Knowledge Transfer, any module that uses another module should perform the checks of that module, and this is what the module description repository ensures.
• Instructions on how the module will participate in the build process of a module that uses it. For example, if the module is a C/C++ library, then it will have to appear on the link line of the using module at a certain position. Confix does this by calculating the dependencies and sorting the modules accordingly, thereby getting a correct link line.

Note that the module description does not contain build instructions itself — these are only of interest while building the modules of a package. In the end, we do not need anything more than the three items listed above to hook a third party library into one package's dependency graph and build process.

For Confix, an external module behaves exactly like a “native” module that has been built by Confix — it's just that a bit of handwork is necessary in that you'll have to write a few lines in the module's Makefile.py. As opposed to a native module, you have to at least manually provide the module somehow. Else the module will not get a chance to get into the dependency graph, and thus it will not contribute to the link line or the include path of other modules. Remember, a module provides itself through provide objects which you manually add to the module. The remainder of this section consists of examples, accompanied with explanations of what's going on.

• A Simple External Module
• A More Complicated External Module

4.6.1 A Simple External Module

With stock Autoconf, you'll link in the Unix crypt library like so,

     AC_CHECK_LIB(crypt,crypt)

The M4 macro AC_CHECK_LIB will check if passing -lcrypt on a program's link line will make the crypt() function available, and add -lcrypt to the link line of the programs that are built in the respective package. Furthermore, it will define the C preprocessor macro HAVE_LIBCRYPT if crypt() is available.

Without Confix it is the package maintainer's responsibility to place AC_CHECK_LIB in configure.in. The Confix way is to provide a “library” of external modules that provide appropriate external module definitions. User code then refers to modules from that library by — either implicitly or explicitly — requiring them.

The wx-utils package that accompanies Confix contains an external module definition for the crypt library in the subdirectory ext-crypt. A native module, in the crypt subdirectory of that package contains a small C++ wrapper around the crypt() function. The advantage of this constellation is that the native module can serve as an automatic entry point into the external library. The native module provides a header file which is included (required) by other modules, with Confix taking care of the dependencies automatically. The native module, in the subdirectory crypt, manually requires the external module.

But first, let's see how the external module is defined in ext-crypt/Makefile.py.

     PROVIDE_SYMBOL(symbol='crypt')
     
     CONFIGURE_IN(
         lines=['AC_CHECK_LIB(crypt,crypt)'],
         order=AC_LIBRARIES)

The first line adds a provide object to the module. So far we have only seen header files that act as provide objects. However the crypt library has no header file of its own that declares the crypt() function. Rather, crypt() is declared in the standard C library header file unistd.h which declares zillions of other symbols as well, and which is likely to be required by code that does not want to make use of crypt(). So we have to use an artificial provide object to announce our ext-crypt module. The drawback to this is that we must also require it artificially, which is what we do in the native crypt module²⁶.

The call to CONFIGURE_IN() adds our AC_CHECK_LIB macro invocation to configure.in. See Interfaces for Coder and Maintainer for an explanation of the parameters.

Now how's that module being used? The native module, crypt, that we've been talking about, has to reference it somehow. Normally this is done implicitly by #includeing a header file. However, we have no header file to include, so we have to manually require the external module. This is easiest done in source file, crypt/crypt.cc, using the REQUIRE_SYMBOL() from The Buildable Interface, like so,

     // CONFIX:REQUIRE_SYMBOL('crypt')

That symbolic require object added to the crypt module will match the symbolic provide object we have used to announce the external ext-crypt module, and thus introduce an edge in the dependency graph. Every module that has a source file with the line #include <WX/Utils/crypt.h> (this is what the native crypt module provides) will link in the crypt module. That module has a (symbolic) dependency on ext-crypt, which leads to ext-crypt (the Unix crypt library) to be linked in as well.

But that's nothing special to external modules, that's how Confix works. What's new here is that we've manually added provide ad require objects, and that we have added Autoconf code.

In your C/C++ code you should react on the outcome of the check, of course. This is nothing special to Confix, it applies to all coders who manage their packages with Autoconf. In our case, the crypt() function may or may not be available. AC_CHECK_LIB defines the preprocessor macro HAVE_LIBCRYPT if it is available, else the macro is undefined. The code will look about like this (see crypt/crypt.cc for the complete code),

     #ifdef HAVE_CONFIG_H
     #  include <config.h>
     #endif
     
     std::string
     Crypt::crypt(
        const std::string& key,
        const std::string& salt)
     {
     #ifdef HAVE_LIBCRYPT
        return ::crypt(key.c_str(), salt.c_str());
     #else
        THROW_ERROR_MSG(Error, "crypt(3) not available (HAVE_LIBCRYPT not defined)");
     #endif
     }

4.6.2 A More Complicated External Module

Not all third party code is as easy to integrate as the crypt library. If you want to, for example, embed the Python interpreter, you'd have to set the include and library path to somewhere inside the Python installation directory. Furthermore, you'd have to find out which libraries you have to link in order to get the necessary symbols defined, and then add these libraries to your link lines, at the correct position.

The wx-utils package contains a module, ext-python, which has all necessary magic for you to use the Python interpreter library. There is also a simple test program available, check/_check_python.cc, which demonstrates — quite rudimentarily though — usage of the Python interpreter.

Before we dissect ext-python/Makefile.py, let's do a little background first. Confix's main task in generating build instructions is to get link lines and include paths right. For this, it builds a dependency graph of modules, and then topologically sorts the graph. The outcome is a list of modules from which Confix computes link lines and include paths.

To participate in the build of another module, a module must contribute certain things such as the base names of the libraries it offers, or the include path which has to be set. For example, the error module of the wx-utils package offers the library libwx-utils_error.a (or similar, the actual name depends on the operating system, and Libtool usage). If an executable depends on that module, then -lwx-utils_error — the library's base name, prepended with -l — will have to appear on the linker commandline that builds that executable, on the correct position.

Normally, when you build all of your source code with Confix, you don't have to care about all this. Anyway, this is what's going on