(send comments to Johan Boulé <bohan.wonderbuild@retropaganda.info>) The benchmarks were generated with this script. The script produces reproductible seeded-random setups. All these tests were run on the same gen-bench output, but each copied to its own directory to ensure they don't interfere with each others.

The gen-bench script was run with the arguments "50 100 15 5".
This gen-bench invocation generates 10000 tiny interdependent C++ source and header files, to be built into 50 static libs.

All tools were configured to use the same compiler flags (-g -O0 -Wall -DBENCH -DBENCH_LIB=x -Idir), and to produce static libs only.
Configuration time was not counted for tools that have a separate configuration phase.

All tests were run at least seven times (first run not counted) and the average taken (sorry, didn't measure incertitude). Initial builds were not always run seven times, but the first run was never counted.
All tools were run with silent/quiet option and/or > /dev/null.
The command lines used to run the tests can be seen in this script. The toy makefiles generated were recursive, and C++ dependencies were tracked using the makedepend X utility. The autotools generated only one makefile with everything (no recursive make invocation).
C++ dependencies were tracked with gcc's generated deps as a side-effect of compilation (-MMD option).
Libtool was enabled, even for building static libs. Despite a lot of efforts, i wasn't able to find out why jam is missing some header depencencies. While it's possible this could be fixed with a trivial change, i must point out that the documentation is rather confusing. Several people have reported problems on jam's mailing list, but it didn't help me. I decided to keep jam's benchmark results, since it's one of the fastest build tool, and it's missing only a few dependencies (precisely, it's rebuilding all libs but lib17 itself), so real numbers should actually be a bit higher, but not much. Scons was tuned to use file timestamps comparisons rather than content hashing.
C++ dependencies were tracked with scons' own full-featured custom preprocessor.
Cache of implicit deps was enabled.
Automatic fetching of source files from SCMs was disabled.
Built target cache was disabled.
Copying of source files to the build dir was disabled. Waf was tuned to use file timestamps comparisons rather than content hashing (in Utils.py, def h_file(filename): return str(os.stat(filename).st_mtime) ).
C++ dependencies were tracked with either gcc's generated deps as a side-effect of compilation (-MMD option), or waf's own full-featured custom preprocessor.
The parallelism algorithm of the task scheduler was set to "normal" (supposedly faster than "max parallel").
The CMake part of the script was contributed by Richard Quirk.
CMake seems to be unable to track system-wide headers. FBuild was tuned to use only file timestamps comparisons, by disabling content hashing. (in fbuild.path.Path, import time; def digest(self): return time.time() ).
The fbuild script finds source files dynamically instead of relying on a hard-coded file list. This is far more maintainance-free, and the impact on speed is negligible. The Fabricate part of the script was contributed by Michael Haggerty. The Tup part of the script was contributed by Jed Brown.
The tup scripts find source files dynamically instead of relying on a hard-coded file list. This is far more maintainance-free, and the impact on speed is negligible.
Tup itself was built with gcc's -O2 optimisation. Wonderbuild used either gcc's generated deps as a side-effect of compilation (-MMD option), or a custom dumb include scanner, with no condition nor macro evaluation.
The wonderbuild script finds source files dynamically instead of relying on a hard-coded file list. This is far more maintainance-free, and the impact on speed is negligible.

If you have an insterest in developping build tools, you can try wonderbuild with this benchmark with:


					svn co https://psycle.svn.sourceforge.net/svnroot/psycle/branches/bohan/wonderbuild
	cd wonderbuild/benchmarks
	./gen-bench bench 50 100 15 5
	cd bench
	time ../../wonderbuild.py --silent --check-missing=no # initial full build
	time ../../wonderbuild.py --silent --check-missing=no # no-op rebuild
	touch lib_17/class_17.hpp && time ../../wonderbuild.py --silent --check-missing=no # small partial rebuild
	touch lib_17/*.hpp && time ../../wonderbuild.py --silent --check-missing=no # big partial rebuild

Alternatively, you may use the helper script, which does the same thing, with:


					svn co https://psycle.svn.sourceforge.net/svnroot/psycle/branches/bohan/wonderbuild
	cd wonderbuild/benchmarks
	./gen-bench.default
	./run-bench reset wonderbuild # initial full build
	./run-bench big wonderbuild # big partial rebuild
	./run-bench small wonderbuild # small partial rebuild
	./run-bench no-op wonderbuild # big partial rebuild

Using the latter helper script, you can easily compare wonderbuild with other tools, as was done in these benchmarks. For example:


					./run-bench small make # small partial rebuild with make
	./run-bench small autotools # small partial rebuild with autotools
	./run-bench small cmake # small partial rebuild with cmake
	./run-bench small waf # small partial rebuild with waf

All tests were run with 4 jobs on the same quad core system, ext2 filesystem on a 4-disk lvm/md-raid0 device, linux 2.6.24, gcc's C++ compiler.

Tup version was 0.3-107-g8d046aa
Scons version was 1.2.0
Waf version was somewhere after 1.5.7 (svn rev 6260)
FBuild version was 0.1
Fabricate version was subversion revision 48
Python version was 2.5.2, except for FBuild where it was version 3.1
Jam version was 2.5
Autotools versions were autoconf 2.61, automake 1.10.1, libtool 1.5.26
CMake version was 2.4.7
Make version was gnu make 3.81
Makedepend was version 1.0.1
Shell was dash (not bash) version 0.5.4
Compiler tool chain versions were gcc 4.2.4, binutils 2.18.0

Using batch compilation

gcc deps (same with 1 batch/lib or max <jobs> batches/lib), (dynamically looking for sources)

Not using batch compilation

without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) missed header deps! without scan of system-wide headers (like gcc -MMD) with or without monitor gcc deps, timestamp signatures waf cpp, timestamp signatures waf cpp, timestamp signatures, dynamically looking for sources (timestamp signatures only, content hashing disabled, dynamically looking for sources) without monitor with fast tuning options with default options (non recursive)

Not using batch compilation, with only one CPU

(bizarrely 172s the first run) (hash sum signatures) (only one CPU supported) After touch lib_17/*.hpp => rebuild of 585 object files from c++ and 33 static libs

Using batch compilation

gcc deps (same with 1 batch/lib or max <jobs> batches/lib), (dynamically looking for sources)

Not using batch compilation

missed header deps! without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) with or without monitor without scan of system-wide headers (like gcc -MMD) (same with gcc deps and waf cpp), timestamp signatures (same with gcc deps and waf cpp), timestamp signatures, dynamically looking for sources (timestamp signatures only, content hashing disabled, dynamically looking for sources) (non recursive) (with fast tuning options)

Not using batch compilation, with only one CPU

(hash sum signatures) (only one CPU supported) After touch lib_17/class_17.hpp => rebuild of 26 object files from c++ and 8 static libs

Using batch compilation

gcc deps (same with 1 batch/lib or max <jobs> batches/lib), (dynamically looking for sources)

Not using batch compilation

with monitor without monitor missed header deps! without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) without scan of system-wide headers (like gcc -MMD) (same with gcc deps and waf cpp), timestamp signatures (same with gcc deps and waf cpp), timestamp signatures, dynamically looking for sources (non recursive) (timestamp signatures only, content hashing disabled, dynamically looking for sources) (with fast tuning options)

Not using batch compilation, with only one CPU

(hash sum signatures) (only one CPU supported) with monitor without monitor gcc deps, (dynamically looking for sources) missed header deps! without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) without scan of system-wide headers (like gcc -MMD) (timestamp signatures) (non recursive) (timestamp signatures), dynamically looking for sources (timestamp signatures only, content hashing disabled, dynamically looking for sources) with fast tuning options with default options

Not using batch compilation, with only one CPU

(hash sum signatures) (only one CPU supported) All tests were run with 1 job on the same system, linux 2.6.24, gcc's C++ compiler.

Scons version was 1.2.0
Waf version was somewhere after 1.5.2 (svn rev 5345)
Python version was 2.4.4
Jam version was 2.5
Autotools versions were autoconf 2.61, automake 1.10, libtool 1.5.22
CMake version was 2.6.0
Make version was gnu make 3.81
Makedepend was version 1.0.0
Shell was bash version 3.1.17
Compiler tool chain versions were gcc 4.1.2, binutils 2.17

Using batch compilation

gcc deps

Not using batch compilation

(recursive) missed header deps! waf cpp, timestamp signatures without scan of system-wide headers (like gcc -MMD) (non recursive) After touch lib_17/*.hpp => rebuild of 585 object files from c++ and 33 static libs

Using batch compilation

gcc deps

Not using batch compilation

missed header deps! (recursive) waf cpp, timestamp signatures without scan of system-wide headers (like gcc -MMD) (non recursive) After touch lib_17/class_17.hpp => rebuild of 26 object files from c++ and 8 static libs

Using batch compilation

gcc deps

Not using batch compilation

missed header deps! (recursive) waf cpp, timestamp signatures without scan of system-wide headers (like gcc -MMD) (non recursive) gcc deps missed header deps! (recursive) waf cpp, timestamp signatures without scan of system-wide headers (like gcc -MMD) (non recursive) All tests were run with 1 job on the same system, ms-windows-xp inside virtualbox, gcc's C++ compiler.

Scons version was 1.2.0
Waf version was somewhere after 1.5.4 (svn rev 6104)
Python version was 2.6.1
CMake version was 2.6.4
Compiler tool chain versions were gcc 3.4.5, binutils 2.17.50

Using batch compilation

gcc deps

Not using batch compilation

waf cpp without scan of system-wide headers (like gcc -MMD) with fast tuning options After touch lib_17/*.hpp => rebuild of 585 object files from c++ and 33 static libs

Using batch compilation

gcc deps

Not using batch compilation

waf cpp with fast tuning options without scan of system-wide headers (like gcc -MMD) After touch lib_17/class_17.hpp => rebuild of 26 object files from c++ and 8 static libs

Using batch compilation

gcc deps

Not using batch compilation

waf cpp without scan of system-wide headers (like gcc -MMD) with fast tuning options gcc deps waf cpp without scan of system-wide headers (like gcc -MMD) with fast tuning options All tests were run with 1 job on the same system, ms-windows-xp inside virtualbox, microsoft's C++ compiler.

Waf version was 1.5.6
Python version was 2.6.1
CMake version was 2.6.4
Compiler was MS Visual C++ version 15 (Visual Studio 9.0 aka 2008)

Using batch compilation

(roughly) (roughly)

Not using batch compilation

After touch lib_17/*.hpp => rebuild of 585 object files from c++ and 33 static libs

Using batch compilation

(roughly) (roughly)

Not using batch compilation

After touch lib_17/class_17.hpp => rebuild of 26 object files from c++ and 8 static libs

Using batch compilation

(roughly) (roughly)

Not using batch compilation

(roughly) (roughly) All tests were run with 1 job on the same system, cygwin running on ms-windows-xp inside virtualbox, gcc's C++ compiler.

Waf version was somewhere after 1.5.4 (svn rev 6104)
Python version was 2.5.2
Shell was bash version 3.2.48
Compiler tool chain versions were gcc 4.3.2, binutils 2.18.50

Using batch compilation

gcc deps

Not using batch compilation

waf cpp, timestamp signatures After touch lib_17/*.hpp => rebuild of 585 object files from c++ and 33 static libs

Using batch compilation

gcc deps

Not using batch compilation

waf cpp, timestamp signatures After touch lib_17/class_17.hpp => rebuild of 26 object files from c++ and 8 static libs

Using batch compilation

gcc deps

Not using batch compilation

waf cpp, timestamp signatures gcc deps waf cpp, timestamp signatures All tests were run with 4 jobs on the same quad core system, ext2 filesystem on a 4-disk lvm/md-raid0 device, linux 2.6.24, gcc's C++ compiler.
The filesystem was unmounted and remounted just before running the build.

Scons version was 1.2.0
Waf version was somewhere after 1.5.9 (svn rev 6671)
Python version was 2.5.2
Jam version was 2.5
Autotools versions were autoconf 2.61, automake 1.10.1, libtool 1.5.26
CMake version was 2.4.7
Make version was gnu make 3.81
Makedepend was version 1.0.1
Shell was dash (not bash) version 0.5.4
Compiler tool chain versions were gcc 4.2.4, binutils 2.18.0

Using batch compilation

gcc deps (same with 1 batch/lib or max <jobs> batches/lib)

Not using batch compilation

without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) missed header deps! without scan of system-wide headers (like gcc -MMD) waf cpp, timestamp signatures gcc deps, timestamp signatures with default options with fast tuning options (non recursive) gcc deps (timestamp signatures) without scan of system-wide headers, (recursive) with scan of system-wide headers, (recursive) without scan of system-wide headers (like gcc -MMD) missed header deps! (non recursive) with fast tuning options with default options Here we compare two different invocations:
- the reference one called with the parameters 50 100 15 5 (50 libs, 10000 files, 104500 #includes)
- with another one called with the parameters 100 200 30 10 (100 libs, 40000 files, 818000 #includes).
The results are the time ratio between the latter and the former.

All tests were run with 4 jobs on the same quad core system, ext2 filesystem on a 4-disk lvm/md-raid0 device, linux 2.6.24, gcc's C++ compiler.

Tup version was 0.3-107-g8d046aa
Waf version was somewhere after 1.5.9 (svn rev 6671)
Fabricate version was subversion revision 48
Python version was 2.5.2
Jam version was 2.5
CMake version was 2.4.7
Make version was gnu make 3.81
Makedepend was version 1.0.1
Shell was dash (not bash) version 0.5.4
Compiler tool chain versions were gcc 4.2.4, binutils 2.18.0

( 131.77s / 30.8s) gcc deps (same with 1 batch/lib or max <jobs> batches/lib) ( 154.35s / 35.6s) missed header deps! ( 164.50s / 37.5s) without scan of system-wide headers (like gcc -MMD) ( 221.52s / 41.3s) without monitor ( 226.56s / 47.0s) gcc deps, timestamp signatures ( 323.08s / 47.3s) waf cpp, timestamp signatures (1209.33s / 188.3s) (bizarrely 932s the first run) (hash sum signatures) (only one CPU supported) After touch lib_17/*.hpp => rebuild of 3291 object files from c++ and 83 static libs
(The reference benchmark rebuilt 585 object files from c++ and 33 static libs)

( 25.59s / 4.40s) gcc deps ( 27.98s / 4.25s) missed header deps! ( 33.59s / 5.54s) without monitor ( 36.86s / 6.15s) without scan of system-wide headers (like gcc -MMD) ( 52.06s / 7.42s) gcc deps, timestamp signatures ( 59.35s / 7.42s) waf cpp, timestamp signatures (247.82s / 26.90s) (hash sum signatures) (only one CPU supported) After touch lib_17/class_17.hpp => rebuild of 37 object files from c++ and 13 static libs
(The reference benchmark rebuilt 26 object files from c++ and 8 static libs)

( 1.17s / 0.40s) without monitor ( 3.44s / 0.84s) gcc deps ( 4.38s / 0.81s) missed header deps! ( 7.84s / 1.70s) without scan of system-wide headers (like gcc -MMD) (10.55s / 2.43s) waf cpp, timestamp signatures (10.63s / 2.43s) gcc deps, timestamp signatures (47.12s / 6.01s) (hash sum signatures) (only one CPU supported) ( 0.31s / 0.08s) without monitor ( 2.96s / 0.63s) gcc deps ( 4.10s / 0.71s) missed header deps! ( 6.77s / 1.39s) without scan of system-wide headers (like gcc -MMD) ( 7.31s / 1.65s) waf cpp, timestamp signatures ( 7.35s / 1.65s) gcc deps, timestamp signatures (44.57s / 4.99s) (hash sum signatures) (only one CPU supported) (recursive) UNFINISHED: makedepend did not manage to compute the include dependencies (error: out of space: increase MAXFILES) Here we used to stress the dependencies between libraries.
The generate-src script was asked to generate N libraries, each library n depending on all libraries from 0 through n - 1.
This leads to a total of N * (N -1) / 2 dependencies.
Thus:
- with 200 libs => 19900 dependencies,
- with 100 libs => 4950 dependencies,
- with 50 libs => 1225 dependencies.

All tests were run with 4 jobs on the same quad core system, ext2 filesystem on a 4-disk lvm/md-raid0 device, linux 2.6.24, gcc's C++ compiler.

Waf version was somewhere after 1.5.9 (svn rev 6674 with issue #522's optim patch)
Python version was 2.5.2
Autotools versions were autoconf 2.61, automake 1.10.1, libtool 1.5.26
Make version was gnu make 3.81
Shell was dash (not bash) version 0.5.4
Compiler tool chain versions were gcc 4.2.4, binutils 2.18.0

With 50 libs => 1225 dependencies, using static libs

gcc deps waf cpp, timestamp signatures, manually-ordered deps (non recursive)

With 50 libs => 1225 dependencies, using shared libs

gcc deps waf cpp, timestamp signatures (non recursive)

With 100 libs => 4950 dependencies, using static libs

gcc deps waf cpp, timestamp signatures, manually-ordered deps (non recursive)

With 100 libs => 4950 dependencies, using shared libs

gcc deps waf cpp, timestamp signatures (non recursive)

With 200 libs => 19900 dependencies, using static libs

gcc deps waf cpp, timestamp signatures, manually-ordered deps (non recursive) UNFINISHED: did not manage to complete within reasonable time, due to libtool being extremely slow!

With 200 libs => 19900 dependencies, using shared libs

gcc deps waf cpp, timestamp signatures (non recursive) UNFINISHED: did not manage to complete within reasonable time, due to libtool being extremely slow!