Consultant Eric Minick says that CI should be focused on testing, not on builds.

He makes the following interesting points:

• automated tests verify successful integration, not the compilation/build process
• running a build multiple times with different test “suites” is complicated and error prone
• performing many different kinds of builds in an enterprise environment is wasteful and much of the testing is outside of source control

His recommendation is that a given software “build” (a set of packaged software bits that represent the output of the compile/generation process) should be passed through multiple testing stages over time. Personally, I visualize this like a car going through a car wash - we don’t build a new car for each brush and spray hose - we use the same car over time.

My take: I think, conceptually, that there’s a lot of merit here - because if you have an active project with a lot of commits, and you’re firing off all sorts of different builds with different test profiles, you are dealing with a constantly changing code-base as you run different types of tests. In other words - the code that you run the “database” integration tests on might be subtly different from the code you run the “web services” integration tests on.

And that’s a fair point - there are situations where you end up with a bug that “slides between” two types of builds. Eric’s solution certainly addresses that issue, in a reasonably elegant way.

However:
A) those issues don’t happen that often, in my experience (if they happen all the time to you, then definitely consider Eric’s idea)
B) Most sites I know of have at least one “Mother of all Test Suites” to cover and catch bugs that slip past the “lesser” test suites.
c) The amount of “waste” involved in fairly meaningless

I applaud UrbanCode’s innovative thinking. If this is something that speaks to you, by all means, give it a try. But don’t trash your existing build model just yet.