Mark Probert on the Ruby-Talk mailing lists asks: “I am not sure how to create a set of Rake rules to do the following. Can anyone prove assistance?”

I had planned for the next Rake tutorial to go into using prepackaged task libraries, but Mark’s question highlights an interesting problem (and resolution) with rules. I posted an answer to Mark on the list, but why waste a perfectly good explaination when I can recycle it here.

The Problem

Mark has two separate source directories (he calls them src_a and src_b, but I suspect they are more creatively named in real life. Both directories contain .c files. However, the kicker is that all the object files are to be placed in a single directory (named obj) no matter which source directory contained the original C code.

The rule we introduced in the last tutorial isn’t powerful enough to move the .o file into the obj directory. We need to tweek it just a bit.

The .o File Rule

Here is the rule in question.

rule '.o' => '.c' do |t|
  sh "cc -c -o #{t.name} #{t.source}" 
end

To recap, the rule specifies how to create a .o file from a similarly named .c file. But as noted above, the .c are in a different location. We can fix this by giving the rule a general purpose function that transforms the object file name into the correct source file name.

Finding the Source File.

But how do we find the source file? Assuming we have a constant named SRC that contains a list of all our source files, this simple find command will do the trick (assume objfile is the name of the object file):

SRC.find { |s| File.basename(s, '.c') == File.basename(objfile, '.o') }

Wrapping this in a method (named find_source) gives us a nice way to find the source file.

Tweeking the Rule

We can now write the rule like this…

rule '.o' => lambda { |objfile| find_source(objfile) } do |t|
  sh "cc -c -o #{t.name} #{t.source}" 
end

The Whole Rakefile

Just a couple notes about the Rakefile

1. Note that we invoke the OBJDIR task directly in the rule. Because it is a rule, there is no opportunity to list OBJDIR as an explicit dependency. By invoking it directly inside the rule, we will build that directory if it is needed (but only if it is needed).
2. If searching the SRC list has performance problems (because SRC is very long), then an alternative is to create a mapping of object names to source names at the top of the file. Then finding the source name is a simple hash lookup.

Rakefile

require 'rake/clean'

PROG = "foo" 
LIBNAME = PROG
LIBFILE = "lib#{LIBNAME}.a" 

SRC = FileList['**/*.c']

OBJDIR = 'obj'
OBJ = SRC.collect { |fn| File.join(OBJDIR, File.basename(fn).ext('o')) }

CLEAN.include(OBJ, OBJDIR, LIBFILE)
CLOBBER.include(PROG)

task :default => [:build, :run]

task :build => [PROG]

task :run => [PROG] do
  sh "./#{PROG}" 
end

file PROG => [LIBFILE] do
  sh "cc -o #{PROG} -L . -l#{LIBNAME}" 
end

file LIBFILE => OBJ do
  sh "ar cr #{LIBFILE} #{OBJ}" 
  sh "ranlib #{LIBFILE}" 
end

directory OBJDIR

rule '.o' => lambda{ |objfile| find_source(objfile) } do |t|
  Task[OBJDIR].invoke
  sh "cc -c -o #{t.name} #{t.source}" 
end

def find_source(objfile)
  base = File.basename(objfile, '.o')
  SRC.find { |s| File.basename(s, '.c') == base }
end

Alternatives

On possible alternative is to replace the rule with a loop that explicitly creates tasks to compile each .c file. It might look something like this:

SRC.each do |srcfile|
  objfile = File.join(OBJDIR, File.basename(srcfile).ext('o'))
  file objfile => [srcfile, OBJDIR] do
    sh "cc -c -o #{objfile} #{srcfile}" 
  end
end

What I like about this solution is the ability to put the OBJDIR dependency directly in the task definition.

• Tagsurf Feedback

Rake is a tool for controlling builds. In this part of the Rake tutorial, we see how to organize the Rake actions to apply to many similar tasks.

In the RakeTutorialIntroduction, we talked about the basics of specifying dependencies and associating actions to build the files. We ended up with a nice Rakefile that built our simple C program, but with some duplication in the build rules.

But First, Some Extra Rake Targets

But before we get into all that, lets add some convience targets to our Rakefile. First of all, it would be nice to have a default target that is invoked when we don’t give any explicit task names to rake. The default target looks like this:

   task :default => ["hello"]

Until now, the only kind of task we have seen in Rake are file tasks. File tasks are knowledgable about time stamps on files. A file task will not execute its action unless the file it represents doesn’t exist, or is older than any of its prerequisites.

A non-file task (or just plain “task”) does not represent the creation of a file. Since there is no timestamp for comparison, non-file tasks always execute their actions (if they have any). Since the default task does not represent a file named “default”, we use a regular non-file task for this purpose. Non-file tasks just use the task keyword (instead of the file keyword).

Here are a couple of other really useful tasks that I almost always include in a Rakefile.

Remove temporary files created during the build process.

Remove all files generated during the build process.

clean tidies up the directories and removes any files that generated as part of the build process, but are not the final goal of the build process. For example, the .o files used to link up the final executable hello program would fall in this category. After the executable program is built, the .o files are no longer needed and will be removed by saying “rake clean”.

clobber is like clean, but even more aggressive. “rake clobber” will remove all files that are not part of the original package. It should return a project to the “just checked out of CVS” state. So it removes the final executable program as well as the files removed by clean.

In fact, these tasks are so common, Rake comes with a predefined library that implements clean and clobber.

But every project is different, how do we specify which files are to be cleaned and clobbered on a per project basis?

The answer is File lists.

File Lists to the Rescue

A file list is simply a list of file names. Since a lot of what Rake does involves files and lists of those files, a file list has some special features to make manipulating file names rather easy.

Suppose you want a list of all the C files in your project. You could add this to your rake file:

  SRC = FileList['*.c']

This will collect all the files ending in ”.c” in the top level directory of your project. File lists understand glob patterns (i.e. things like "*.c") and will find all the matching files.

By the way, no matter where you invoke it, rake always executes in the directory where the Rakefile is found. This keeps your path names consistent without depending on the current directory the user interactive shell.

The clean and clobber tasks use file lists to manage the files to remove. So if we want to clean up all the .o files in a project we could try …

  CLEAN = FileList['*.o']

(CLEAN is the file list associated with the clean task. I’ll let you guess the name of the file list associated with clobber).

The Rakefile So Far …

With the addtion of a few extra tasks, our Rakefile now looks like this. Notice the require ‘rake/clean’ line used to enable the clean and clobber tasks.

  require 'rake/clean'

  CLEAN.include('*.o')
  CLOBBER.include('hello')

  task :default => ["hello"]

  file 'main.o' => ["main.c", "greet.h"] do
    sh "cc -c -o main.o main.c" 
  end

  file 'greet.o' => ['greet.c'] do
    sh "cc -c -o greet.o greet.c" 
  end

  file "hello" => ["main.o", "greet.o"] do
    sh "cc -o hello main.o greet.o" 
  end

Ok, now its time to address the redundant compile commands.

Dynamically Building Tasks

The command to compile the main.c and greet.c files is identical, except for the name of the files involved. The simpliest and most direct way to address the problem is to define the compile task in a loop. Perhaps something like this …

  SRC = FileList['*.c']
  SRC.each do |fn|
    obj = fn.sub(/\.[^.]*$/, '.o')
    file obj  do
      sh "cc -c -o #{obj} #{fn}" 
    end
  end

Just a couple things to note about the above code.

• The dependencies are not specified. This is a common where we specify the dependents at one place and the actions in another. Rake is smart enough to combine the dependencies with the actions.

• Although the task was named after the .o (which is, after all, what we want to generate), the file list is defined in terms of the .c files. Why?

The simple reason is that file lists search for file names that exist in the file system. We have no guarantee that the .o files even exist at this point (indeed, the will not after invoking the clean task). The .c are source and will always be there.

Rake Can Automatically Generate Tasks

Defining tasks in a loop is pretty cool, but is really not needed in a number of simple cases. Rake can automatically generate file based tasks according to some simple pattern matching rules.

For example, we can capture the above logic in a single rule … no need to find all the source files and iterate through them.

  rule '.o' => '.c' do |t|
    sh "cc -c -o #{t.name} #{t.source}" 
  end

The above rule says that if you want to generate a file ending in .o, then you if you have a file with the same base name, but ending in .c, then you can generate the .o from the .c.

t.name is the name of the task, and in file based tasks will be the name of the file we are trying to generate. t.source is the name of the source file, i.e. the one that matches the second have of the rule pattern. t.source is only valid in the body of a rule.

Rules are actually much more flexible than you are led to believe here. But that’s an advanced topic that we will save for another day.

Final Rakefile

Here is our final resule. Notice how we use the SRC and OBJ file lists to manage our lists of scource files and object files.

  require 'rake/clean'

  CLEAN.include('*.o')
  CLOBBER.include('hello')

  task :default => ["hello"]

  SRC = FileList['*.c']
  OBJ = SRC.ext('o')

  rule '.o' => '.c' do |t|
    sh "cc -c -o #{t.name} #{t.source}" 
  end

  file "hello" => OBJ do
    sh "cc -o hello #{OBJ}" 
  end

  # File dependencies go here ...
  file 'main.o' => ['main.c', 'greet.h']
  file 'greet.o' => ['greet.c']

Up Next

In our next tutorial, we will look at using Rake to handle some tasks other than compiling C code.

• Tagsurf Feedback

Received via EMail:

I have just started using the excellent Rake tool (thanks, Jim!) and I am at a bit of a loss on how to proceed. I am attempting to create unit test for some C++ code I am creating, […]

To adderess this lack, I’m going to post several Rake tutorial articles that will take you through some of the basics. Eventually, I’ll organize the articles into a document somewhere.

Here’s the first one!

The Problem

We will start with a very simple build problem, the type of problem that make (and now rake) were desiged to deal with.

Suppose I am a C programmer and I have a simple C program consisting of the following files.

main.c

  #include "greet.h"
  int main() {
      greet ("World");
      return 0;
  }

greet.h

  extern void greet(const char * who);

greet.c

  #include <stdio.h>
  void greet (const char * who) {
      printf ("Hello, %s\n", who);
  }

(Yes, it really is the old standard "Hello, World" program. I did say we were starting with the basics!)

To compile and run this collection of files, a simple shell script like the following is adequate.

build.sh

  cc -c -o main.o main.c
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o

For those not familiar with compiling C code, the cc command is the C compiler. It generates an output file (specified by the -o flag) from the source files listed on the command line.

Running it gives us the following results …

  $ build.sh
  $ ./hello
  Hello, World

Building C Programs

Compiling C programs is really a two step process. First you compile all the source code file into object files. Then you take all the object files and link them together to make the executable.

The following figure illustrates the progression from source files to object files to executable program.

Our program is so small that there is little benefit in doing more than the three line build script above. However, as projects grow, there are more and more source files and object files to manage. Recompiling everything for a simple one line change in a single source file gets old quickly. It is much more efficient to just recompile the few files that change and then relink.

But how do we know what to recompile? Keeping track of that would be quite error prone if we tried to do that by hand. Here is where Rake become useful.

File Dependencies

First, lets take a look at when files need to be recompiled. Consider the main.o. Obviously if the main.c file changes, then we need to rebuild main.o. But are the other files that can trigger a recompile of main.o?

Actually, yes. Looking at the source of main.c, we see that it includes the header file greet.h. That means any changes in greet.h could possibly effect the main.o file as well.

We say that main.o has a dependency on the files main.c and greet.h. We can capture this dependency in Rake with the following line:

  file "main.o" => ["main.c", "greet.h"]

The rake dependency declaration is just regular Ruby code. We take advantage of the fact that we can construct hash arguments on the fly, and that Ruby doesn’t require parenthesis around the method arguement to create a file task declaration that reads very naturally to the humans reading the rake file. But its still just Ruby code.

Likewise, we can declare the dependencies for creating the "greet.o" file as well.

  file "greet.o" => ["greet.c"]

greet.c does include stdio.h, but since that is a system header file and not subject to change (often), we can leave it out of the dependency list.

Finally we can declare the dependencies for the executable program hello. It just depends on the two object files.

  file "hello" => ["main.o", "greet.o"]

Notice that we only have to declare the direct dependencies of hello. Yes, hello depends on main.o which in turn depends on main.c. But the .c files are not directly used in building hello, so they can safely be omitted from the list.

Building the Files

We have carefully specified how the files are related. Now we need to say what Rake would have to do to build the files when needed.

This part is pretty simple. The three line build script that we started with contains all the commands needed to build the program. We just need to put those actions with the right set of dependencies. Use a Ruby do / end block to capture actions …

The result looks like this:

Rakefile

  file 'main.o' => ["main.c", "greet.h"] do
    sh "cc -c -o main.o main.c"
  end

  file 'greet.o' => ['greet.c'] do
    sh "cc -c -o greet.o greet.c"
  end

  file "hello" => ["main.o", "greet.o"] do
    sh "cc -o hello main.o greet.o"
  end

Trying it out

So, let’s see if it works!

  $ rake hello
  (in /home/jim/pgm/rake/intro)
  cc -c -o main.o main.c
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o

The command line rake hello instructs rake to look through its list of tasks and find one called "hello". It then checks hello’s dependencies and builds them if required. Finally, when everything is ready it builds hello by executing the C compiler command.

Rake dutifully reports what it is doing as it goes along. We can see that each compiler invocation is done in the correct order, building the main program at the end. So, does the program work? Let’s find out.

  $ ./hello
  Hello, World

Success!

But what happens when we change a file. Lets change the greet function in greet.c to print "Hi" instead of hello.

  $ xemacs greet.c
  $ rake hello
  (in /home/jim/pgm/rake/intro)
  cc -c -o greet.o greet.c
  cc -o hello main.o greet.o
  $
  $ ./hello
  Hi, World

Notice that it recompiles greet.c making a new greet.o. And then it needs to relink hello with the new greet.o. Then it is done. There is no need to recompile main.c since it never changed.

What do you think will happend if we run Rake again?

  $ rake hello
  (in /home/jim/pgm/rake/intro)
  $

That’s right … nothing. Everything is up to date with its dependencies, so there is no work for Rake to do.

Ok, sure. Rake is a bit of overkill for only two source files and a header. But imagine a large project with hundreds of files and dependencies. All of a sudden, a tool like Rake becomes very attractive.

Summary

What have we learned? Building a Rakefile involves identifying dependencies and the actions required to create the target files. Then declaring the dependencies and actions are as simple as writing them down in standard Ruby code. Rake then handles the details of building

What’s Up Next

We notice that even our small example has a bit of duplication in it. We have specify how to compile both C file separately, even though the only difference is the files that are used. The next installment will look at fixing that problem as well as introduce non-file based tasks, rules and file lists.

Until then … Code Red, Code Ruby.

• Tagsurf Feedback