Bruce Eckel on Build Systems

In "Why We Use Ant", Bruce argues why we should continue use Ant, even though it has some significant flaws as a build system for large, complex projects. His arguments are:

1. It’s never as easy as it seems [writing a replacement for Ant].
2. It’s probably not the battle you should be fighting.
3. Lots of people have tried and failed.
4. Politics matter.

Number four is actually a very strong argument, especially in a group doing Java development.

Let me quote an interesting passage in Bruce’s article.

Another important proglem with Ant is that its XML-influenced declarative syntax is limited. A build system is really a little computer program. Normally, you look for conditions (a source file is out of date with its target is the primary one for make) and you execute commands, but sometimes you need to do things that are more sophisticated, conditional logic, that kind of thinkg. Both make and Ant fall down here because, while it’s possible to do these things it suddenly gets a lot more complicated because that’s not normally what they do.

Exactly! It was my need to dynamically generate some make targets that started my original make rant that sparked the rake project. make, by itself, just wasn’t up to the task

Bruce continues …

I suppost if I were give a budget and a bevy of programmers, I would ask them to take make, which is open-source, eliminate the silly tab-vs.-space thing, and meld it together with Python, also open-source. make would only retain the simplest of its functionalites (primariy the way you set up dependencies and execute simple commands), and as soon as you wanted to do something more complex you’d be able to drop seamlessly into Python syntax, using any Python libraries you want. I think that would probably be my dream build system.

Bruce, you really need to check out rake. Its exactly what you describe above (well, if you substitute Ruby for Python). And it didn’t take a bevy of programmers. The core functionality was an evening’s worth of work.

Dropping seamlessly into Ruby is trivial, because a Rakefile is a Ruby program, specified in standard (although idiomatic) Ruby syntax. Since we never leave Ruby, we have no parsers to deal with, leaving a very small code base for the core functionality. The core of Rake (dependency management, task execution and rule matching) is less than 200 lines of code. The minimal rake install (a single file) is around 500 lines of Ruby code and adds common file commands (system independent versions of cp, rm, mv, etc.) and a flexible FileList object that mimics Ant’s include and exclude semantics.

Martin Fowler on Build Systems

Martin Fowler echos many of Bruce’s comments and reiterates that a build system really needs a full blown programming language. Martin has been dabbling in Ruby, and it seems that he has discovered rake and has been using in it some small projects. Great!

A Build Language Needs to be a Programming Language

Why should your build system support a full blown programming language? Here is one small example taken from a real situation at work.

When we are ready to install a project at work, we create a staging directory containing the files to be installed into an environment (e.g. testing or production). As problems are discovered in testing, corrections are made in a development area and copied to the staging directory for reinstallation.

The following makefile copies 2 jar files from the developement area to the staging area. Yes, I know that a makefile is probably overkill for this problem, but in real life there are other tasks as well. For example, using make (or rake) allows me to ensure that a refresh step always occurs before an install step, but only if the refresh is needed.

So here is the makefile …

  # Makefile to copy jars

  DEV=/home/jim/development
  JARDIR=jars

  refresh: $(JARDIR)/a.jar $(JARDIR)/b.jar

  $(JARDIR)/a.jar: $(DEV)/AProject/lib/a.jar $(JARDIR)
          cp $@(DEV)/AProject/lib/a.jar $@

  $(JARDIR)/b.jar: $(DEV)/BProject/lib/b.jar $(JARDIR)
          cp $(DEV)/BProject/lib/b.jar $@

  $(JARDIR):
          mkdir -p $(JARDIR)

Notice that there are several duplications in the file. The copy command for the two jar files are remarkably similar, yet different enough that you can’t capture both cases with a generic make rule. Also, knowledge about a particular jar file is spread out in two places; once in the copy rule itself, and again in refresh task line.

First we will translate the above Makefile into an equivalent Rakefile. See the Rakefile documents for details on the syntax of a Rakefile.

  # Rakefile translated directly from the Makefile

  DEV = "/home/jim/development"
  JARDIR = "jars"

  task :refresh => [ "#{JARDIR}/a.jar", "#{JARDIR}/b.jar" ]

  file "#{JARDIR}/a.jar" => [ "#{DEV}/AProject/lib/a.jar", JARDIR ] do |task|
    cp "#{DEV}/AProject/lib/a.jar", task.name
  end

  file "#{JARDIR}/b.jar" => [ "#{DEV}/BProject/lib/b.jar", JARDIR ] do |task|
    cp "#{DEV}/BProject/lib/b.jar", task.name
  end

  directory JARDIR

All the duplication from the original makefile still exists in the rake version. Lets remove it by writing a function that will generate the dependency (and copy command) based on the jar file name and the project name.

  # Rakefile with (some) redundencies removed

  DEV = "/home/jim/development"
  JARDIR = "jars"

  task :refresh
  directory JARDIR

  def make_jar_copy_task(jarname, project)
    stagejar = "#{JARDIR}/#{jarname}.jar"
    devjar   = "#{DEV}/#{project}/lib/#{jarname}.jar"

    task :refresh => [ stagejar ]

    file stagejar => [ devjar, JARDIR ] do
      cp devjar, stagejar
    end
  end

  make_jar_copy_task("a", "AProject")
  make_jar_copy_task("b", "BProject")

Notice that the refresh task is now declared without any explicit dependencies. The individual jar file dependencies are added to the refresh task inside the make_jar_copy_task function.

If we had a lot of jar files, we could put them in an explicit list and replace the last two lines with something like this …

  jarfiles = [ ["a", "AProject"], ["b", "BProject"], ["c", "CProject"] ]
  jarfiles.each { |jar, proj|  make_jar_copy_task(jar, proj) }

Or perhaps we have to deduce the jar files from the current contents of the JARDIR directory. (Assume that projects is a lookup table mapping jarfiles to project names.)

  Dir["#{JARDIR}/*.jar"].each { |jar|
    jarname = File.basename(path).sub(/\.jar$/, "")
    make_jar_copy_task(jarname, projects[jarname])
  }

Or perhaps we need to pull our jar file names from a database …

  DBI.connect(DBI_STRING, USER, PASSWORD) do |db|
    sql = "SELECT jarfile, project FROM install_table WHERE install_date = ?",
    db.select_all(sql, INSTALL_DATE) do |row|
      make_jar_copy_task(row['jarfile'], row['project'])
    end
  end

Ok, that last example might have been a little over the top. The point is that builds can become arbitrarily complex, and anything less than a full programming language is just doesn’t cut it.