Production-ready software, on-demand

The general notion of the practice of Continuous Integration is that it’s performed during the “development cycle”. I think we need to expand this into areas typically thought of as “latter lifecycle” activities. In my experience, even the most Agile of projects aren’t considering all aspects of what it takes to make software “production ready”.

Over the past year, I’ve been advocating a much more intensive criteria for Continuous Integration with my peers and customers, which is more like “Continuous Production” meaning you can produce production-ready software on a daily or even a continuous basis. Effectively, we’re talking about extending the Continuous Integration model into all environments (Test, Stage and Production), not just the development environment. In fact, at Stelligent, our purpose is to get our customers to production-ready software on a daily basis.

A posting on Wikipedia describes Continuous Production as “a method used to manufacture, produce, or process any product without interruption. There is no discrete rate at which goods are produced, as opposed to a batch production process, or job production.” I’m suggesting we move toward a similar model for developing software.

A lot is bundled in this term, Continuous Production, though. In my experience, it means that you automate most everything it takes to create software so that it can be deployed into the user’s environment any time. It means you may automate some or all of the following:

1. Compilation of source code and tests - this is a given for CI.Tools: Ant, MsBuild, rake, NAnt or similar build tool.
2. Database Integration/Migration to target database server (e.g. Oracle) - Some shops are doing this with CI. Tools: Ant sql task or ORM-based Ant tasks, dbdeploy
3. Testing - execution of unit, component, system, functional/acceptance, security, load and performance tests - Some shops are doing this with CI. Tools: JUnit, DbUnit, Selenium, JUnitPerf, FindBugs, JMeter, NUnit, NDbUnit
4. Inspections - execution of static/dynamic analysis checkers: code coverage, code duplication checks, coding standards, cyclomatic complexity and dependency analysis - Some shops are doing this with CI. Tools: PMD, CheckStyle, FindBugs, JavaNCSS, JDepend, Cobertura, NCover, NProf, Source Monitor, NCover
5. Deployment - Deployment and repeatable configuration to servers (e.g. JBoss) - Some shops are doing this with CI. Tools: Application Server-specific Ant tasks (e.g. WebLogic)
6. Remote Deployments to target environments - For example, from a build machine to staging environment (for web containers and database servers). Tools: Ant, Java Secure Channel and SmartFrog, Capistrano
   1. You will need a way to rollback to a previous deployment (this includes database changes)
7. (Optionally) Creation of installation distribution - Few shops are doing this with CI. Tools: NSIS and antinstaller, InstallShield .
8. (Optionally) Generation of distribution media - Few shops are doing this with CI
9. Source labeling and (possibly) build labeling - Some shops are doing this with CI
10. Automated notification of build status - This is a given for CI via Email, X10 devices, etc.
11. Automating the promotion through QA, Staging and Production environments - Promote the binary and accompanying artifact through each target environment. Tools: Build Management servers such as AntHill Pro are useful for build promotion
12. Documentation Generating the readme docs, installation and user guides. Tools: JavaDoc, NDoc, Doxygen, GraphViz, Grand, etc.

Of course, the actual steps will vary depending the type of software application you are producing whether it be a rich client, web application, embedded or other…but you get the point. Continuous Production, to me, means you have the capability to create production-ready software whenever necessary .

What I am saying is that the creation of production-ready software should become a “nonevent “.

The benefits are numerous, but here are a few:

• Decisions can be made based on working software , not on a task item on a project management chart - no more “it’s 80% done”
• Eliminate the reliance on an individual with specialized knowledge to produce the software for production
• There’s “done”, there’s “done, done” (coded with acceptance tests). With Continuous Production, it’s “done, done, done” and ready for release to users (code, acceptance tests, operating in the customer’s environment)

Note: This does not mean that you will be taking up cycles producing production-ready software with every change to the version control repository. Test, Stage and Production builds are more likely to be performed on demand (by a person). However, the point is that your build environments are capable of building production-ready software on a continuous basis.

There are very, very few projects that are doing this right now. I’d have to guess it’s much less than 1%. It makes me think of the saying “it’s not the destination, it’s the journey”. If your team sets this as a goal, the closer you get to it, the less you’ll be spending time on environment-related, repetitive and error-prone issues and the more time you spend on new features and improvements to the software. Is it possible, right now, for every project to do this? Probably not. However, once you start asking questions like “What do I need to do to make this feature production ready?”, it changes your perspective on everything from coding to testing and deployment.

In reality, I think even the most automated development environments still need (or should) have a manual component when creating software that is production-ready. This may include usability testing or even a simple manual sanity check. But, I think the more we move toward this model, the less headaches we’ll have and the better our estimates.

In my book on Continuous Integration, I make a reference to something Tim O’Reilly mentioned regarding Flickr’ s ability to deliver updates to its web software as frequently as every 30 minutes. This is only possible if you’ve automated many of the deployment steps.

Let me know if you or someone you know is doing this or moving toward this type of model. I’m curious to hear your experiences with it.