Production-ready software, on-demand

The general notion of the practice of Continuous Integration is that it’s performed during the “development cycle”. I think we need to expand this into areas typically thought of as “latter lifecycle” activities. In my experience, even the most Agile of projects aren’t considering all aspects of what it takes to make software “production ready”.

Over the past year, I’ve been advocating a much more intensive criteria for Continuous Integration with my peers and customers, which is more like “Continuous Production” meaning you can produce production-ready software on a daily or even a continuous basis. Effectively, we’re talking about extending the Continuous Integration model into all environments (Test, Stage and Production), not just the development environment. In fact, at Stelligent, our purpose is to get our customers to production-ready software on a daily basis.

A posting on Wikipedia describes Continuous Production as “a method used to manufacture, produce, or process any product without interruption. There is no discrete rate at which goods are produced, as opposed to a batch production process, or job production.” I’m suggesting we move toward a similar model for developing software.

A lot is bundled in this term, Continuous Production, though. In my experience, it means that you automate most everything it takes to create software so that it can be deployed into the user’s environment any time. It means you may automate some or all of the following:

1. Compilation of source code and tests - this is a given for CI.Tools: Ant, MsBuild, rake, NAnt or similar build tool.
2. Database Integration/Migration to target database server (e.g. Oracle) - Some shops are doing this with CI. Tools: Ant sql task or ORM-based Ant tasks, dbdeploy
3. Testing - execution of unit, component, system, functional/acceptance, security, load and performance tests - Some shops are doing this with CI. Tools: JUnit, DbUnit, Selenium, JUnitPerf, FindBugs, JMeter, NUnit, NDbUnit
4. Inspections - execution of static/dynamic analysis checkers: code coverage, code duplication checks, coding standards, cyclomatic complexity and dependency analysis - Some shops are doing this with CI. Tools: PMD, CheckStyle, FindBugs, JavaNCSS, JDepend, Cobertura, NCover, NProf, Source Monitor, NCover
5. Deployment - Deployment and repeatable configuration to servers (e.g. JBoss) - Some shops are doing this with CI. Tools: Application Server-specific Ant tasks (e.g. WebLogic)
6. Remote Deployments to target environments - For example, from a build machine to staging environment (for web containers and database servers). Tools: Ant, Java Secure Channel and SmartFrog, Capistrano
   1. You will need a way to rollback to a previous deployment (this includes database changes)
7. (Optionally) Creation of installation distribution - Few shops are doing this with CI. Tools: NSIS and antinstaller, InstallShield .
8. (Optionally) Generation of distribution media - Few shops are doing this with CI
9. Source labeling and (possibly) build labeling - Some shops are doing this with CI
10. Automated notification of build status - This is a given for CI via Email, X10 devices, etc.
11. Automating the promotion through QA, Staging and Production environments - Promote the binary and accompanying artifact through each target environment. Tools: Build Management servers such as AntHill Pro are useful for build promotion
12. Documentation Generating the readme docs, installation and user guides. Tools: JavaDoc, NDoc, Doxygen, GraphViz, Grand, etc.

Of course, the actual steps will vary depending the type of software application you are producing whether it be a rich client, web application, embedded or other…but you get the point. Continuous Production, to me, means you have the capability to create production-ready software whenever necessary .

What I am saying is that the creation of production-ready software should become a “nonevent “.

The benefits are numerous, but here are a few:

• Decisions can be made based on working software , not on a task item on a project management chart - no more “it’s 80% done”
• Eliminate the reliance on an individual with specialized knowledge to produce the software for production
• There’s “done”, there’s “done, done” (coded with acceptance tests). With Continuous Production, it’s “done, done, done” and ready for release to users (code, acceptance tests, operating in the customer’s environment)

Note: This does not mean that you will be taking up cycles producing production-ready software with every change to the version control repository. Test, Stage and Production builds are more likely to be performed on demand (by a person). However, the point is that your build environments are capable of building production-ready software on a continuous basis.

There are very, very few projects that are doing this right now. I’d have to guess it’s much less than 1%. It makes me think of the saying “it’s not the destination, it’s the journey”. If your team sets this as a goal, the closer you get to it, the less you’ll be spending time on environment-related, repetitive and error-prone issues and the more time you spend on new features and improvements to the software. Is it possible, right now, for every project to do this? Probably not. However, once you start asking questions like “What do I need to do to make this feature production ready?”, it changes your perspective on everything from coding to testing and deployment.

In reality, I think even the most automated development environments still need (or should) have a manual component when creating software that is production-ready. This may include usability testing or even a simple manual sanity check. But, I think the more we move toward this model, the less headaches we’ll have and the better our estimates.

In my book on Continuous Integration, I make a reference to something Tim O’Reilly mentioned regarding Flickr’ s ability to deliver updates to its web software as frequently as every 30 minutes. This is only possible if you’ve automated many of the deployment steps.

Let me know if you or someone you know is doing this or moving toward this type of model. I’m curious to hear your experiences with it.

We received over 80,000 unique hits to my blog post, “Fire your ‘best’ people, reward the ‘lazy’ ones”. Based on comments here and postings on other sites, it’s fair to say that many of you seem to agree that if you have too many people on a project that focus exclusively on troubleshooting, you’ve got a problem. I figured this post deserved further explanation…

I wrote this based on my experiences in working on dozens of projects as either a developer, tester, PM, architect or consultant. Some of the behavior that teams are rewarding will not lead to getting better software out the door faster. There’s no way you can prevent every problem from occurring on a project. However, when problems do occur, it’s important to ask two questions:

“Why did this happen?”

-and -

“What are we going to do to prevent it from happening again?”

That’s it. People need to troubleshoot issues, but they must also be asking these questions after fixing the problem. In the post, I pit the troubleshooter against the troublepreventer, but I go on to say that, without question, you need troubleshooters on your project. I am poking at the people that focus so much on troubleshooting that they forget to look to prevent the problem from occurring in the first place. The time you invest in fixing or preventing a problem should be based on risk. It’s probably not worth investing several days on preventing a problem from occurring if the likelihood of its occurrence is low and the impact of the realizing the risk is minimal. However, it may be worth your time if the likelihood is low/medium and the impact is high/severe (e.g. application will not work). Obviously, if it’s already happened, the likelihood is rather high, don’t you think?

Even today, there are many that firmly believe that if a team works to improve/introduce software quality, it will delay the schedule. The antithesis is a false economy . In my experience, effectively employing practices/techniques such as developer testing, creating a fully automated build, continuous integration (CI), and automated deployment speed up quality and delivery with teams. Of course, I used a trick (some may say ‘clever’) word in the previous sentence…”effectively”. For instance, if you try to do CI without a fully automated build, you will have problems. Or, try to write developer tests that rely on specific values in the database that aren’t aren’t updated/integrated into scripts, you’ll have problems. There’s no silver bullet here.

People will ask, so “If I do TDD, CI and automated builds, will this improve my speed and quality of delivery?” I must, in all good conscience, give you the consultant’s answer: “It depends!” If you’re incorrectly implementing these practices, it doesn’t matter what you call it, they’re not going to magically solve problems just because you gave it a name. Unfortunately, the practices themselves can be misappropriated as the culprit of the schedule delay or quality problems (e.g. I always get a chuckle out of the: “We stopped using CI because our builds kept failing”).

Fail fast is a good mantra to follow. Learn from your mistakes and improve. Reacting to the latest emergency is necessary and inevitable. However, teams should seek to always learn and improve to lessen these occurrences in the future. People may say “Well, that’s just obvious!” Then, I will leave you with one of my favorite blog posts from Kathy Sierra. There’s a difference between saying and doing .

Recently, Steve McConnell was interviewed by PM Boulevard about Agile development.

I appreciate his non-dogmatic, sensible approach to software development practices. Here’s an excerpt of McConnell’s take on Agile:

I’ve always been pro-whatever-practices-work-best. In many situations the practices that work best are the practices that today are associated with agile development. And in some circumstances, other older practices still work best.

So I’m not pro-agile or anti-agile. I’m not pro-CMM or anti-CMM, pro-BDUF or anti-BDUF, or pro-pair programming or anti-pair programming. For that matter I’m not even pro-waterfall or anti-waterfall.

Hear, hear! My post Am I Agile or Not? speaks to a similar philosophy about software development practices as well.

In “Am I Alone Here?”, Tim Goodwin has a nice blog commenting on his desire to run a single command build using only assets from the version control repository. I could not agree more. As Martin Fowler mentions in his Continuous Integration article “…anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.”

Tim points out some of the challenges he experienced while working in a Microsoft environment. It is imperative that vendors give us the capability to run everything from the command line with minimal effort. GUI tools are fine as long as you still have the capability to run the same from the command line or API. Wizards or other GUI tools provide a level of automation (the first time) but not when we need to repeat a series of activities. In particular, when we need to setup an automated Continuous Integration server.

I am fond of saying if you can’t run (a tool: server, program, etc.) from the command line, it doesn’t exist (as a tool).

P.S. When is Microsoft going to eliminate the Visual Studio dependency on MSTest? This means I must install an IDE on any machine I wish to build software when using MSTest…not an acceptable option to me.

Fire your “best” people…reward the “lazy” ones

In my experience, what most people consider to be their “best” people are often the root of most problems. It’s the difference between troubleshooters and troublepreventers.

Let me explain…

Bill is a troubleshooter (only). Many consider Bill to be the best employee in the company because he solves almost any problem that comes into the door. However, unbeknownst to many (especially management), Bill is the one causing most of the problems. Bill knows the purpose of every hard-coded lookup value and is the fastest to manually deploy the application to WebLogic using its web console. For instance, he knows that “14″ means a “transaction that is waiting for approval by second-line management”. However, instead of creating a human-readable constant, Bill hard codes the number “14″ in the source code. Therefore, Bill is the only one that can easily maintain these methods. In my personal experience, someone like Bill doesn’t do this for nefarious reasons, he’s just not thinking about how the next developer may need to maintain it. For every time a method is written, it’s read (and maintained) ten times (10:1). Therefore, people like Bill need to be writing source code for other humans, not the computer.

Let’s have a look at Peter…

Peter is a “lazy” developer (a troublepreventer). Because Peter is “lazy”, he hates doing the same thing twice, so he generalizes common functionality into a delegating class, creates interfaces for implementing classes to adhere to, or he writes build scripts that compile, integrate the database, runs tests and inspections, and deploys the software the same way every time. Peter finds it mind-numbing to repeat himself. Again, Peter is lazy so he doesn’t mind spending two hours automating something that normally takes 10 minutes to run manually. Why? Because, he very quickly reaps the benefits of this extra effort since every time the process is run he gets those 10 minutes back and even more importantly, he eliminates the possibility of human error (which, again, saves time and money).

The (”lazy”) troublepreventer thinks ahead. He extracts variable information into common properties files, seeks to reduce complex code, and automates repetitive, error-prone activities such as the build and deployment processes. He also ensures that others can very easily repeat what he has done. Anyone that has worked with me for even a day knows that I often sound like a broken record when I say “Is it in Subversion?” or “Have you updated the Wiki?” To me, if the knowledge is locked in your head, you are a less valuable, not more valuable, resource. Troublepreventers put their knowledge in the system, not just their heads, so that it runs the same way every time.

People like troubleshooters because they can solve a problem when a project is under pressure such as getting that emergency fix out the door immediately. Without question, you need troubleshooters on your project. However, many times the (exclusive) troubleshooters are the ones that cause the problem in the first place, be it a hard-coded value, duplication of code or a large complex method only they can understand.

Before you start thinking that I’m trying to gather together a group of slackers, I’m suggesting the complete opposite of this. I just want people to think about the total time involved, not just fixing the symptom. There are people that are both troublepreventers and troubleshooters. These are the people you want to keep and reward. However, on a given team, I’d opt for more troublepreventers than troubleshooters as they save everyone time, money and headaches.

This is one of the more annoying questions I get. Nothing against the people that ask this question. It’s a perfectly reasonable question for someone to ask. However, for me, the implication of this question can be off putting. It’s as if some people think there’s an Agile committee waiting to sign off on whether a project is “Agile” or not. Who knows, maybe there is at their company?? <tangent>Reminds me of the CMMI. Nothing against CMMI — that is, if you like paying lots of money for someone to tell you nothing about whether you can deliver functioning software.</tangent>

On with the story…

Recently, someone asked me: “If we’re not doing TDD, is our project ‘Agile’?” First, why is it relevant if someone else considers your project to be Agile or not? Second, and how I answered this particular “developer testing” question was that it depends upon whether you want to learn about a defect as soon as it’s introduced when combined with a Continuous Integration system. Or…whether you want to learn quickly that a new feature caused a previously implemented component to break. To me, that’s more relevant. Not whether you’re “Agile” or not - in this context. In my experience, this question is often not about whether it’s in the “box” called Agile (a fine question, by my standards), but, instead, whether someone else would “label” me/my project as Agile or not. To me, this is an important subtlety as it makes me think they’re looking at it like it’s a checkbox rather than a way to improve software development. So, I guess I’d feel better if they’d ask me “Paul, what is the value of the test-driven development practice in the context of the Agile methodology?” or “Name some problems when using TDD”. Now, we’re getting somewhere.

I appreciate, use and advocate many, but not all, of the “Agile” practices because they make sense to me and the teams I’ve worked with. Personally, I probably lean more toward the “collection of best practices” approach (used in context) than pure methodologist. Furthermore, I’ve adapted (agile, no?) these practices based on the team environment and skill sets. For example, I’ve started teams out with a daily build because I felt that Continuous Integration was going to be too much for this particular team — at first. After a few weeks, we moved to Continuous Integration with compilation and unit tests. Later we added inspections and deployment and so on.

Let me beat my point into the ground…For instance, I believe that running a scheduled build that occurs once an hour is not “Continuous Integration”. However, how much does it really matter if it’s called “Continuous Integration” (I consider this to be “Scheduled Integration”)? I mean, will the authors of Continuous Integration use you as an example in the 2nd edition of the book? Let’s, instead, discuss the pluses and minuses of this approach. A minus, to me, is that a build runs regardless of whether you committed a change to your version control repository (seems like waste to me). Another minus is that it could distract you from something you are working on if it fails. However, there are a lot of pluses to it. For instance, as long as you’re keeping up with it , you’re ensuring your software is building successfully on a periodic basis and you’re learning about this more often than alternatives (such as 1) never 2) weekly 3) daily, etc.)

Don’t miss my point, labels do matter. In fact, my company helps teams optimize Agile software production . I appreciate and believe in the value of wrapping a word/label around something (e.g. “Agile”, “SCRUM”, “Unctuous” and so on) Further, it can help you determine what a practice/methodology is and what it is not. But, utilize the practice because it makes sense not because some “thought leader” told you that you better do this or I won’t consider you “Agile” (Look into my eyes…now at the swinging watch…you muuuust listen to me on this one ). Also, I think some are hesitant to say “well, it depends” because they may be afraid that people won’t follow their advice. Oh, if “they” could only see it my way…

A client I worked with recently had a very astute characterization of Continuous Integration. I will paraphrase it as “CI is a reflection of reality”. He was referring to the immediate feedback you receive from a successful or failed integration build. Using a single source (your version control repository), you’re able to quickly determine if any of your recent changes “played well with others”. Not tomorrow, not two weeks from now, but immediately after a change is made. It reminds me of something Bill Cosby used to say in one of his stand-up routines when someone mentioned they liked using cocaine because it “…intensifies your personality”. Cosby’s response was “Yes, but what if you’re an a**hole?”. CI provides you with a mirror of your software under development. It’s not the fault of CI if it comes back and tells you your code is not working as intended. If fact, you should embrace it before it tells you it’s ugly.

Since I’ve heard many conflicting definitions on what a software “build” actually is/is not, I figured I’d come up with my own definition to muddy things even more! Someone on Wikipedia has described a software build as “the process of converting source code files into executable code or the result of doing so.” Without parsing words, I think that this is, currently, the generally accepted definition of a build. However, I’d like to expand this definition to include much more than the mere compilation of source code files into something that can be executed. Unfortunately (for many it seems), compilation is often the end of the development line as a developer moves onto his next task. Some people will also “unit test” their code, which means they test a small part of the application functionally (btw, this is NOT a unit test). I think a build is converting source code files into executable code and validating and verifying that it works as expected. What does this mean? To me, it means that a build should consist of compilation coupled with execution of automated developer tests (unit, component, system, and so on), execution of automated inspections (code coverage, cyclomatic complexity, duplication checks, and so on), and packaging/deployment (to a server or other environment). The “compilation” piece may include rebuilding the database and test data to provide a consistent environment from which to execute tests. You’ll often hear people refer to “build and test”. It seems to me this terminology is used because most people still think of a build as nothing more than the compilation (and linking, etc.) piece. I think “builds” should refer to much more than the compile and, in fact, the flimsy use of the word leads many to think they’re done with their development once they click the “Run Build” button in their IDE. What do you think?

The co-founder of CruiseControl, Paul Julius, shares an experience that demonstrates how practicing CI saved $12,535 for one defect on one project by discovering and fixing the defect shortly after it was introduced. Some people ask “what’s the return on investment of CI?”. Although, software quality is a great benefit of CI, this experience demonstrates the amount of money that can be saved by effectively using the practice.

I was honored that Paul wrote one of the forewords for the upcoming book on Continuous Integration due out next month. In the foreword, he shares an excellent perspective of his experiences and contribution to the CI practice.

I was at CITCON last weekend and someone asked me about managing the CI server configuration files (such as with CruiseControl). I told them that I like to manage my configuration files just like any other software project. The build and CI scripts are not “second-class citizens” when I’m developing software.

First, I create a project in my version control system to manage the files that manage my CI server. Then, I create a project within my CI server to poll for changes against the project in the version control repository. If any changes are found, it retrieves the files and overwrites any relevant files in the CI server directory location (e.g. /opt/cruisecontrol-bin-2.6.2/config.xml). In this case, you’re using the CI server to look for changes and then overwrite its own configuration files….it reminds me of the ‘Dancin’ with Myself’ song by Billy Idol (1982). The code listing below shows a CruiseControl project called cc-config that polls a Subversion repository which contains CruiseControl configuration files. If changes are detected it calls the delegating build, ${build.config.file}.

...
<project name="cc-config" buildafterfailed="true">
  <listeners>
    <currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
  </listeners>
  <modificationset quietperiod="${qp.interval}">
    <svn RepositoryLocation="${svn.project.cc-config}"/>
  </modificationset>
   <schedule interval="${interval}">
     <ant anthome="apache-ant-1.6.5" buildfile="${build.config.file}"/>
   </schedule>
   <log dir="logs/${project.name}" />
</project>

The next code example is the delegating build in Ant. This build cleans any local files and gets the new files from Subversion. Finally, it calls the cc-config project’s build.xml.

<project name="build-cc-config" default="build" basedir="projects/cc-config">
  <property file="/opt/cruisecontrol-bin-2.6.2/config.properties" />
  <target name="build" depends="clean">
    <taskdef resource="svntask.properties" />
    <svn>
      <checkout url="${svn.project.cc-config}" revision="HEAD" destPath="." />
    </svn>
    <ant antfile="build.xml" target="all" />
</target>
<target name="clean" description="Deletes all configuration files">
  <delete includeemptydirs="true" quiet="true">
    <fileset dir="." includes="**/*" />
  </delete>
</target>
</project>

The final example is the Ant script that does the actual work, the build.xml. It copies the files retrieved from Subversion and updates these files to the CruiseControl configuration directory, /opt/cruisecontrol/cruisecontrol-bin-2.6.2.

<?xml version="1.0" encoding="iso-8859-1"?>
  <project name="cc-build" default="all" basedir=".">
    <target name="all">
      <copy todir="/opt/cruisecontrol/cruisecontrol-bin-2.6.2">
        <fileset dir=".">
          <include name="**/*.xml"/>
	  <exclude name="build.xml"/>
        </fileset>
      </copy>
  </target>
</project>

There are a couple of things to keep in mind if you are using this technique. If you make any changes to configuration files (e.g. /opt/cruisecontrol/cruisecontrol-bin-2.6.2/build-project.xml, etc.), directly on the server, they’ll be overwritten by these changes. The other thing is that if you make changes to the config.xml, they won’t be effective until the next build because it uses the config.xml to find changes to “itself”.

Is anyone using a different technique to solve this issue? Are you managing your CI server changes in your version control system? Is this too extreme?