The Integrate Button website (from Paul Duvall’s book on Continuous Integration) recently published a screencast on using the Hudson Continuous Integration server - along with Subversion, Ant, HSQL and other tools. Click the image below to get started.

The screencast demonstrates the following steps:

• Checkout source files from the Subversion repository - locally
• Run the automated build locally
• Commit files to Subversion
• Download, install and configure Hudson - from Hudson website or from IntegrateButton scripts
• Make a code change (with error) and checkin files
• Get notified of error, automatically, via Hudson
• Fix code errors, commit change and see results in Hudson

I like repeatable processes. To me, it’s a waste of time to do the same thing more than once. I’m also reminded of this when I start repeating answers to the same questions. When considering knowledge companies, virtually everything valuable to us are our people and information, not physical assets. As I said in “Fire your Best People…”, if information is locked inside someone’s head, they are a less valuable resource. It’s all about sharing: files, knowledge, pertinent information from emails and so on. If the asset only exists on one person’s machine, or worse: their head, and is relevant to others, your company has a problem.

When considering software, if all of the assets necessary to create working software haven’t been committed to the version-control repository, the software doesn’t exist. Sounds simple doesn’t it? Then, why do so many projects and companies fail to follow this simple rule? If I was cynical, I might think it’s because some think that by harboring this information, they can essentially hold the organization “hostage” - and increasing job security. In my experience, this is no longer the primary reason in the organizations I’ve seen (although there are always a few exceptions). I think the main problem is leadership. It takes discipline for each person in an organization to ask themselves after they’ve sent instructions on how to do something in an email…

Is this something I want to keep permanent for all current and future employees?

Other examples include:

• Realizing that a configuration file used to deploy the software should be checked into the version-control system so that the software can be built in any target environment, not just on their machine.
• The workaround for that strange bug in your Content Management System should be posted to the Wiki
• That pointing to a file in a version-control repository may be preferred over sending email attachments with every change .
• The email that documented the steps that Human Resources performs when a new employee starts with the company, should be published to the company’s Intranet .

Here are some possible tools where you might capture these assets:

• Version-control repository
• Corporate Intranet
• Hosted Wiki
• Electronic Corporate bulletin board
• Company blog

Leaders in knowledge organizations understand their most vital assets are their people and the information produced by these people. When people leave, you need to have a way to continue to leverage this knowledge. Leaders understand that this philosophy of sharing information must be institutionalized. Just remember…

If it’s not in the System, it doesn’t exist

It can be a significant burden to manually apply upgrades to an existing database. Moreover, it’s often an error-laden activity. IBM developerWorks’ recently published an article on this topic for Paul Duvall’s (Stelligent CTO) popular Automation for the people series called Hands-free database migration.

Databases are often out of sync with the applications they support, and getting the database and data into a known state is a significant challenge to manage. Learn how the open source LiquiBase database-migration tool can reduce the pain of managing the constant of change with databases and applications.

In the last two articles, “UML Diagrams within Javadocs” and “Visual Documentation of Ant Dependencies in 3 Simple Steps” we saw how easy and valuable it was to automate technical documentation. By using open source tools, we were easily able to provide good technical documentation within a few minutes, and at no cost at all. We were also able to keep this up-to date by adding additional tasks to our Ant build files, and run them from our CI Server(Hudson in our case) on commit and nightly builds, and also publish the results.

In this article, I will be showing you how to use yet another tool called Dxoygen for generating technical documentation based on your source code. We all have used Javadoc and have been using it for a long time, right? So, you may ask what’s the need to have another tool which produces the same HTML documentation? Doxygen has a slight edge over Javadoc and here are a few reasons why you should consider using the same:

1. With Javadoc you have to remember all the HTML tags, you need to embed within your code comments. However, with Doxygen code comments are much more concise and polished, without the need for any HTML.

2. Doxygen can also generate a variety of diagrams, we will take a look at some of them later.

3. Doxygen also provides a structured view on the source code. As I mentioned in 2 above in the form of various diagrams, cross-referenced and syntax highlighted code.

4. You get all the above benefits even if the code does not have any comments at all.

5. Last but not the least, Doxygen is a documentation system not for just Java but also for various other languages like C++, C, Java, Objective-C, Python, IDL (Corba and Microsoft flavors), Fortran, VHDL, PHP, C#.

So, without wasting further time, lets see what we need to get started with Doxygen.

Step 1. Download, Install Doxygen

Download the binary distribution for Doxygen for the operating system you are using. I downloaded the binary distribution for Mac OS X called Doxygen-1.5.6.dmg. Installation is very simple, just drag the doxygen icon from this folder to the Applications folder, or wherever you want to keep it; as shown below. I dropped it within my Applications folder. Just be sure to remember where you dragged it. To uninstall, just delete the file. It is completely self-contained.

Step 2: Configure Doxygen.
To generate documentation using Doxygen, you will need a configuration file called the Doxyfile. You can generate this file in two ways; either by using the Doxygen wizard or by using the command line option. Lets see how to use both these options to generate the configuration file:

a. Command line.
Open a command window and type the following as shown below:

You should be able to locate the configuration file created within your default user folder. The file looks like this:

# Doxyfile 1.5.6

# This file describes the settings to be used by the documentation system
# doxygen (www.doxygen.org) for a project
#
# All text after a hash (#) is considered a comment and will be ignored
# The format is:
# TAG = value [value, …]
# For lists items can also be appended using:
# TAG += value [value, …]
# Values that contain spaces should be placed between quotes (” “)

#—————————————————————————
# Project related configuration options
#—————————————————————————

# This tag specifies the encoding used for all characters in the config file
# that follow. The default is UTF-8 which is also the encoding used for all
# text before the first occurrence of this tag. Doxygen uses libiconv (or the
# iconv built into libc) for the transcoding. See
# http://www.gnu.org/software/libiconv for the list of possible encodings.

DOXYFILE_ENCODING = UTF-8

# The PROJECT_NAME tag is a single word (or a sequence of words surrounded
# by quotes) that should identify the project.

PROJECT_NAME =

# The PROJECT_NUMBER tag can be used to enter a project or revision number.
# This could be handy for archiving the generated documentation or
# if some version control system is used.

PROJECT_NUMBER =

b. Wizard Option.
Launch the Doxygen application, and you should be able to create the configuration file using the wizard approach as shown below.

The user interface is quite intuitive so I am going to skip explaining this in detail.
The wizard approach was the one I used to get the initial settings for the configuration file, which you can always modify later.

A few options in my Doxygen configuration file are as follows:

# Doxyfile 1.5.6

#—————————————————————————
# Project related configuration options
#—————————————————————————
DOXYFILE_ENCODING = UTF-8
PROJECT_NAME = PetStore
PROJECT_NUMBER = 1.0
#—————————————————————————
# Build related configuration options
#—————————————————————————
EXTRACT_ALL = NO
EXTRACT_PRIVATE = NO
EXTRACT_STATIC = NO
EXTRACT_LOCAL_CLASSES = YES
#—————————————————————————
# configuration options related to the HTML output
#—————————————————————————
GENERATE_HTML = YES
HTML_OUTPUT = html
HTML_FILE_EXTENSION = .html
GENERATE_TREEVIEW = YES

Step 3. Doxygen and Ant.

In order to use Doxygen, we need an Ant task. There is already an Ant task written for Doxygen which you can download from here.
As always, since using Mac, when I downloaded the binaries and tried to use them, I got the ever famous error message :

java.lang.UnsupportedClassVersionError: Bad version number in .class file

So, had to do download the source, compile, and jar it up. Copy this library to your projects folder. So, lets start making changes to our build file.

3.a: Lets define the Doxygen task:

<taskdef name="doxygen" classname="org.doxygen.tools.DoxygenTask" classpath="lib/ant_doxygen.jar" />

3.b: To generate HTML documentation:

<target name="generate-doxygen-docs">
<taskdef name="doxygen" classname="org.doxygen.tools.DoxygenTask"
classpath="lib/ant_doxygen.jar" />
<doxygen configFilename="reports/Doxyfile">
<property name="INPUT" value="${srcdir}" />
<property name="RECURSIVE" value="yes" />
</doxygen>
</target>

3.c: Lets combine them in a target and run the same:

[doxygen] Exec: /Applications/Doxygen.app reports/Doxyfile

BUILD FAILED
/CodeMetricsProject/build.xml:91: Doxygen not found on the PATH.

Total time: 7 seconds

3.d: So, to launch Doxygen not in the path, we make changes to the doxygen task as shown below:

<doxygen doxygenPath="/Applications/Doxygen.app/Contents/Resources/doxygen" configFilename="reports/Doxyfile">

Lets run the target again and see if it fixed things. Yes indeed.

generate-doxygen-docs:
[doxygen] Exec: /Applications/Doxygen.app/Contents/Resources/doxygen reports/Doxyfile

BUILD SUCCESSFUL
Total time: 8 seconds

4. Integrate with Hudson.

4.a Hudson Job and Ant Target.
Once you have an Ant target working, calling this from your CI server is trivial. Within Hudson, select your Job, click on configure and add this new target to be called when running the build.

4.b Publish the Reports.

4. c: Sample Reports.

Force a build, and take a detailed look at the reports generated by Doxygen as shown below:

I have given you a brief overview of Doxygen in this article, how to configure the same, and use it effectively to generate technical documentation on a continuous basis; either on commit builds or nightly builds. The Doxygen web site has lots of information on how to use it with other programming languages and also has tutorials in languages other than English as well.

As always, if you are having trouble getting Doxygen to work, leave a comment or check out the Doxygen web site.

One of the primary goals of every developer should be to prevent or drastically limit the number of bugs or defects from being introduced in their source code. It is also our responsibility to write good, extensible, testable, and maintainable code. This however seems like a herculean task to many.

John Smart, has very well said in his latest and greatest "Java Power Tools" book:

When used well, software metrics can provide valuable feedback on the quality of the code being written, and allow the development team to learn and progress. To be successful, software metrics should be considered a team sport.

So, lets automate the task of finding defects early by using open source tools which can look at your source code and in most cases find, if not all, at least some potential defects. There are numerous open source tools like:

1. CheckStyle - is a development tool to help programmers write Java code that adheres to a coding standard.

2. JDepend - traverses Java class file directories and generates design quality metrics for each Java package.

3. JavaNCSS - cyclomatic complexity tool.

4. Simian - identifies duplication in Java. Not open source.

5. Emma - Code coverage tool.

There are several other open source tools as well, like PMD, Cobertura, FindBugs and many more. All these tools are highly configurable as well. I mentioned the above five for a reason. I will get to it soon.

Imagine how much work it is to setup, configure, and run these metric tools as part of your continuous integration. It really is a huge task doing it for each and every project you have.

However, there is just one tool which you need to download: Panopticode. Panopticode includes all the above mentioned tools preconfigured. If you have read Neal Ford’s "The Productive Programmer", chapter 7: Static Analysis" page 113, the author talks about "Generating Metrics with Panopticode". That’s how I got to know about this amazing tool. There is no detailed instructions in the book on how to get this up and running, the web site does have.

So, I decided to give it a try and here are the steps I followed.

Rather than creating a new project, I re-used the PetStore project which I have been using for a long time for all the articles I write.

Follow these steps to get Panopticode up and running:

1. Download Panopticode from here.

2. Unzip and copy to your project folder.

The folder structure will look like this:

3. Make changes to the build.xml file.

In the Panopticode web site, there are 4 steps listed which you can follow. However, rather than copying the entire build file, I modified my existing build file, and ran the new target metrics within the Hudson(Continuous Integration Server) Job .

Add the following lines from the panopticode/build-example.xml to your build file.

3.a

<import file="panopticode/panopticode-imports.xml"></import>

3.b

<target name="metrics" depends="clean">
		<panopticode projectdir="${basedir}" projectname="${ant.project.name}"
                           projectversion="${version}" srcdir="${srcdir}"></panopticode>
	                           </target>
             

In the above target, specify the path to your source directory.

3.c Modify your target which compiles and runs your unit tests to include the following lines:

<panopticode-junit unitTestClasspathId="unittest.path"
	                           outputDir="target">
	            <batchtest todir="target/rawmetrics/xml/junit">
	                <fileset dir="target/classes" includes="**/*Test.class" />
	            </batchtest>
	       

Make changes to the unitTestClasspathId to include the libraries required to run your tests. In my case, I included the JEE libraries required for my EJB application.

4. Run the metrics Ant target.

Run the metrics target either from the command line. You should see an output like this:

[junit] Testcase: validNumberOfLifecycleAnnotations took 0.008 sec
[report] processing input files …
[report] 2 file(s) read and merged in 4 ms
[report] writing [xml] report to [../CodeMetricsProject/target/rawmetrics/xml/emma.xml] …
[report] writing [html] report to [../CodeMetricsProject/target/rawmetrics/html/emma/emma-coverage.html] …
[emmaPanopticode] Loading panopticode structure: panopticode.xml
[emmaPanopticode] Loading supplement: org.panopticode.supplement.emma.EmmaSupplement
[emmaPanopticode] Writing back to panopticode structure: panopticode.xml
[mkdir] Created dir: ../CodeMetricsProject/target/reports/svg
[echo] Building Static Reports
[echo] Building Interactive Reports
BUILD SUCCESSFUL
Total time: 8 seconds

5. Force a Build.
Now that we have all the xml files generated for the metrics tool, we need to make sure run the above target metrics from the Hudson Job, and force a build. You should be able to see the following xml as well as HTML files generated:

6. Configure the XML files.
Finally, configure all these xml files within the Post-Build actions of the Hudson Job, and you should be able to force a build and see the reports in your dashboard. See screen shots below for various metrics:

a. Hudson Dashboard:

b. Emma Coverage Trend:

c. Check Style

d. JavaNCSS

7. Troubleshooting.
A few problems I found getting this up and running:

If you see the build failed with the following exception, copy the JDepend library to your ANT_HOME/lib folder.

compile:
    [mkdir] Created dir: /Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/target/classes
    [javac] Compiling 22 source files to /Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/target/classes
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
     [echo] Panopticode - Using JDepend for OO metrics
     [echo] If you get an error stating 'Could not create task or type of type: jdepend' then you need to copy the jdepend-2.9.1.jar file to your ANT_HOME/lib directory.

BUILD FAILED
/Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/build.xml:54: The following error occurred while executing this line:
/Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/panopticode/panopticode-imports.xml:125: The following error occurred while executing this line:
/Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/panopticode/supplements/jdepend/jdepend-imports.xml:38: Problem: failed to create task or type jdepend
Cause: the class org.apache.tools.ant.taskdefs.optional.jdepend.JDependTask was not found.
        This looks like one of Ant's optional components.
Action: Check that the appropriate optional JAR exists in
        -/usr/share/ant/lib
        -/Users/meerasubbarao/.ant/lib
        -a directory added on the command line with the -lib argument

Do not panic, this is a common problem.
The commonest cause is a missing JAR.

This is not a bug; it is a configuration problem

Total time: 5 seconds
Publishing Javadoc
Recording Emma reports CodeMetricsProject/reports/emma.xml
finished: FAILURE

In the Panopticode-imports.xml file, you see the following comments:

    <!--
        Panopticode supports multiple competing code coverage tools.  You can switch which tool you use at will and your
        build files will not change.  You must choose exactly one though.  If you do not want to collect coverage data
        then choose the 'nocoverage' import.  You choose your coverage tool by using exactly one of the following import
        statements:

            <import file="plugins/emma/emma-imports.xml" />
            <import file="plugins/cobertura/cobertura-imports.xml" />
            <import file="plugins/nocoverage-imports.xml" />

        The default is Emma.  Try them all to see which one you prefer.
    -->
	<import file="supplements/emma/emma-imports.xml" />

However, if you try to comment the import for emma and use cobertura import definition, you get an error as shown below:

Buildfile: ../CodeMetricsProject/build.xml

BUILD FAILED
../CodeMetricsProject/build.xml:3: The following error occurred while executing this line:
../CodeMetricsProject/panopticode/panopticode-imports.xml:42: Cannot find plugins/cobertura/cobertura-imports.xml imported from ../CodeMetricsProject/panopticode/panopticode-imports.xml

Total time: 223 milliseconds

Now, if I go back and change it to the directory where I found the cobertura-imports.xml file (supplements), I get an exception like this:

[cobertura-report] Report time: 263ms
     [move] Moving 1 file to /Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/target/rawmetrics/xml
   [delete] Deleting directory /Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/target/rawmetrics/xml/cobertura
   [delete] Deleting: /Users/meerasubbarao/Development/ci-jobs/jobs/CodeMetricsProject/workspace/CodeMetricsProject/cobertura.ser

BUILD FAILED
../CodeMetricsProject/build.xml:54: The following error occurred while executing this line:
../CodeMetricsProject/panopticode/panopticode-imports.xml:145: emmaFile emma.xml does not exist

Total time: 7 seconds

Next, I changed the report file to cobertura.xml, and the build complained about:

[cobertura-report] Report time: 264ms
     [move] Moving 1 file to ../CodeMetricsProject/target/rawmetrics/xml
   [delete] Deleting directory ../CodeMetricsProject/target/rawmetrics/xml/cobertura
   [delete] Deleting: ../CodeMetricsProject/cobertura.ser
[emmaPanopticode] Loading panopticode structure: panopticode.xml
[emmaPanopticode] Loading supplement: org.panopticode.supplement.emma.EmmaSupplement

BUILD FAILED
../CodeMetricsProject/panopticode/panopticode-imports.xml:145: null

Total time: 8 seconds

Finally, I downloaded the source code to see the classname for this taskdef, couldn’t find one and just gave up at this point.
You should be able to generate coverage report for Emma with no problems at all.

As you can see from this article, it is just a few minutes to setup and start using Panopticode. As I mentioned earlier also, there are many open source tools available, that can improve the quality of your code and also keep track of the same if they are automated and are part of your continuous integration. I am sure once you have these open source tools integrated, you will keep wondering how you developed code without them.

I have been on several teams where we studiously designed UML diagrams at the beginning of the project. As the project progressed, and deadlines approached, the UML diagrams were left somewhere behind, not to be updated in months. When a new developer joined the team, we showcased the old UML diagrams, and kept telling ” Oh, we never had time to update them, please see the source code to get an idea. And, don’t hesitate to ask if you have any doubt’s”. I am sure, you all have gone through the same scenario.
However, we don’t have to keep making up stories anymore, since this article shows how easy and simple it is to include UML diagrams within your Javadoc and also keep them updated with every change in the source code repository. We can do these in less than a few minutes, and in a few simple steps.

Getting started with UmlGraph takes five steps:

1. Download the source code for UMlGraph.
2. Download and install Graphviz.
3. Make changes to your Ant build file.
4. Run the Ant target.
5. Add this target to your CI job.

Step 1: Download the source code for UMLGraph here. Unzip the contents. To compile the Java doclet from the source code run ant on the build.xml file. Copy the UmlGraph.jar file to your projects library. If there is a version mismatch between the different versions of JDK you are using you get an exception like this:

java.lang.UnsupportedClassVersionError: Bad version number in .class file

Make sure you recompile the UMLGraph source code, and copy the library to your project.

Step 2 : Download and install Graphviz from here. The dot file needs to be post-processed with Graphviz to produce the actual UML diagram. Running the UmlGraph doclet will generate a Graphviz diagram specification that can be automatically processed to create png drawings. You can also generate other formats using Graphviz as well. If Graphviz isn’t installed you will get an exception as shown below:

BUILD FAILED
/Users/meerasubbarao/Development/webservices-samples/build.xml:107:
Execute failed: java.io.IOException: dot: not found
Total time: 269 milliseconds

Step 3. Changes to your build.xml file.
Assuming you already have a working project, with Ant build file. Add the following target to your build.xml file as shown below:

	<target name="javadocs" depends="build" description="generates javadoc and also UML Diagram">
		<mkdir dir="${reports.dir}/javadoc"/>
	        <javadoc sourcepath="${src.dir}" packagenames="com.stelligent.*" destdir="${reports.dir}/javadoc"
	        	classpathref="java.classpath" private="true">
	        	   <doclet name="org.umlgraph.doclet.UmlGraphDoc"
	        		  path="lib/UMLGraph.jar">
	        	        <param name="-attributes" />
	        	        <param name="-operations" />
	        	        <param name="-qualify" />
	        	        <param name="-types" />
	        	        <param name="-visibility" />
	        	    </doclet>
	        	  </javadoc>
		  <apply executable="dot" dest="${reports.dir}" parallel="false">
		    <arg value="-Tpng"/>
		    <arg value="-o"/>
		     <targetfile/>
		     <srcfile/>
		     <fileset dir="${reports.dir}" includes="*.dot"/>
		     <mapper type="glob" from="*.dot" to="*.png"/>
		  </apply>
	</target>

A number of options control the operation of UMLGraph class diagram generator. These can be specified as parameters within your build file as shown above.

Details about a few options are:

-output
Specify the output file (default graph.dot).
-d
Specify the output directory (defaults to the current directory).
-qualify
Produce fully-qualified class names.
-horizontal
Layout the graph in the horizontal direction.
-attributes
Show class attributes (Java fields)
-operations
Show class operations (Java methods)
-constructors
Show a class's constructors
-visibility
Adorn class elements according to their visibility (private, public, protected, package)
-types
Add type information to attributes and operations
-enumerations
Show enumarations as separate stereotyped primitive types.
-enumconstants
When showing enumerations, also show the values they can take.
-all
Same as -attributes -operations -visibility -types -enumerations -enumconstants

Step 4. Run the ant target:
Open a command window and run the ant target: ant javadocs and you should see output as such in your console window:

meera-subbaraos-macbook-9:webservices-samples meerasubbarao$ ant javadocs
Buildfile: build.xml

init:

cleanGenerated:

build:
    [javac] Compiling 22 source files to /Users/meerasubbarao/Development/ci-jobs/jobs/PetStore_Nightly/workspace/webservices-samples/classes
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

javadocs:
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package com.stelligent.biz.ws...
  [javadoc] Loading source files for package com.stelligent.ent.jpa...
  [javadoc] Constructing Javadoc information...
  [javadoc] UmlGraphDoc version 5.0, running the standard doclet
  [javadoc] Standard Doclet version 1.5.0_13
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...
  [javadoc] Generating /Users/meerasubbarao/Development/ci-jobs/jobs/PetStore_Nightly/workspace/webservices-samples/reports/javadoc/stylesheet.css...
  [javadoc] UmlGraphDoc version 5.0, altering javadocs
  [javadoc] Building Package view for package com.stelligent.biz.ws
  [javadoc] Building Package view for package com.stelligent.ent.jpa
  [javadoc] Building Context view for class com.stelligent.biz.ws.SupplierManagerBean
  [javadoc] Building Context view for class com.stelligent.biz.ws.SupplierManager
  [javadoc] Building Context view for class com.stelligent.biz.ws.SignonManagerBean
  [javadoc] Building Context view for class com.stelligent.biz.ws.SignonManager
.....

BUILD SUCCESSFUL
Total time: 8 seconds
meera-subbaraos-macbook-9:webservices-samples meerasubbarao$ 

The javadoc generated is pretty neat with UML diagrams on the top:

Step 5: Add this target to your CI Job.
If you already have a CI server like Hudson up and running, which runs commit builds and nightly builds, adding this new target is a one step process. In my case, I already have a nightly job running as shown below. I have added this ant target to my default target as shown below:

<target name="all" depends="cleanAndDeployForCoverage, javadocs" />

Next, force a build on the Hudson job, publish the javadocs, and you can see the results on the hudson dashboard.

The Javadoc displayed from within the Hudson dashboard:

Now that we have UML diagram integrated within our build file and also our CI job, we can ensure that our code base and the UML diagrams are always in sync. We saw how to include these ant targets in our commit builds or nightly builds of our CI jobs, and also published these artifacts as part of our post build process.

Resources:

• Ant
• UMLGraph
• Graphviz
• Hudson

ThoughtWorks, announced the general release of Cruise, a continuous integration and release management system. ThoughtWorks, are also creators of the open source CI Server CruiseControl. Cruise introduces the powerful concept of pipelines, which makes it simple to manage and test changes in an application from check-in through deployment.

ThoughtWorks is offering a free 30-day trial of Cruise (on Windows, Mac OS X and Linux), following which it will remain free to use for teams requiring two or fewer software agents. In addition, Cruise will be available at reduced or no cost for most open source projects, academic institutions and non-profit organizations.

For more details visit:

1. Cruise

An automated build process with Ant is one of the most crucial things required in any CI process. Ant is the build tool of choice for many Enterprise Java projects. This is an XML file, usually called build.xml, which describes a project’s dependencies. At the beginning of any project, this build file will be somewhere around 70-80 lines, which would include targets for compiling source and tests, running these tests, creating the libraries, and so on… But, as the project grows, this file grows as well and becomes as huge as several thousand lines; which is complicated and hard to maintain.

At this point, you might decide to refactor the build file, but before you being to do so, it would really help if you had a bigger picture of this several thousand lines of XML file, right? As the proverb goes “A picture is worth a thousand words,” I will show you how this one picture we are just going to create shows several lines of XML files.

1. Download the libraries.
I have tested diagrams from both Grand as well Vizant. I didn’t see any significant difference in either one of them. So, based on your preference and the license, download the libraries from here:

• Grand
• Vizant
• Graphviz The dot file needs to be post-processed with Graphviz to produce the actual graph.

2. Create an Ant Target.
I have two listings show below here; one for Grand and one for Vizant. If you have used custom ant tasks, the listing shown below is self explanatory. Before using either Grand or Vizant, define a task. Next, give the name of the build file, its location, and the output. To view the graph you need to transform the dot file into something else using the dot command. In our case, we are generating a PDF file, but you can generate any other format as well; like JPG, PNG and so on.

Listing 1 for Grand:

<target name="grandAntDiagram">
    <typedef resource="net/ggtools/grand/antlib.xml" classpath="${basedir}/lib/grand-1.8.jar" />
        <grand output="build.dot" buildfile="${basedir}/build.xml" />
		<exec executable="dot">
		  <arg line="-Tpdf -Gsize=11.69,8.27 -Grotate=90 -o build.pdf ${basedir}/build.dot" />
		</exec>
</target>

Listing 2 for Vizant:

<taskdef name="vizant" classname="net.sourceforge.vizant.Vizant" classpath="${basedir}/lib//vizant-0.1.2.jar" />
<target name="vizantAntDiagram">
		<vizant antfile="${basedir}/build.xml" outfile="vizant-build.dot">
			<attrstmt type="graph">
				<attr name="ranksep" value="1.2" />
				<attr name="nodesep" value="0.5" />
			</attrstmt>
		</vizant>
		<exec executable="dot">
			<arg line="-Tjpg vizant-build.dot -o build.jpg" />
		</exec>
</target>

3. Run the Target.
You can run the targets from the command lie, from the IDE of your choice or from your CI server by just calling the above targets.

mobile-166-217-041-119:webservices-samples meerasubbarao$ ls
build.dot		dist			run.bat
build.pdf		instrumented		src
build.properties	lib			test
build.xml		passfile		vizant-build.dot
classes			qalab.xml
cobertura.ser		reports
mobile-166-217-041-119:webservices-samples meerasubbarao$ ant grandAntDiagram
Buildfile: build.xml

grandAntDiagram:
    [grand] Loading project /Users/meerasubbarao/Development/webservices-samples/build.xml
    [grand] Setting up filter chain
    [grand] Writing output to /Users/meerasubbarao/Development/webservices-samples/build.dot
    [grand] Outputing to /Users/meerasubbarao/Development/webservices-samples/build.dot

BUILD SUCCESSFUL
Total time: 0 seconds
mobile-166-217-041-119:webservices-samples meerasubbarao$

Here are some sample diagrams for the build files I had.

1. A simple one, this was the build file I used for my article I wrote for Javalobby.

2. And, this one which was generated by an IDE for another article I was working on. Does your build file look like this? If it does, than it is time to refactor the build files heavily.

P.S: The listings shown above in itself are sufficient for you to get the diagrams working in your build files, the one thing you will need to change is the location of the libraries. Also, don’t forget to download and install Graphviz, if not you will get an exception like this:

BUILD FAILED
/Users/meerasubbarao/Development/webservices-samples/build.xml:107:
      Execute failed: java.io.IOException: dot: not found
Total time: 269 milliseconds

Consultant Eric Minick says that CI should be focused on testing, not on builds.

He makes the following interesting points:

• automated tests verify successful integration, not the compilation/build process
• running a build multiple times with different test “suites” is complicated and error prone
• performing many different kinds of builds in an enterprise environment is wasteful and much of the testing is outside of source control

His recommendation is that a given software “build” (a set of packaged software bits that represent the output of the compile/generation process) should be passed through multiple testing stages over time. Personally, I visualize this like a car going through a car wash - we don’t build a new car for each brush and spray hose - we use the same car over time.

My take: I think, conceptually, that there’s a lot of merit here - because if you have an active project with a lot of commits, and you’re firing off all sorts of different builds with different test profiles, you are dealing with a constantly changing code-base as you run different types of tests. In other words - the code that you run the “database” integration tests on might be subtly different from the code you run the “web services” integration tests on.

And that’s a fair point - there are situations where you end up with a bug that “slides between” two types of builds. Eric’s solution certainly addresses that issue, in a reasonably elegant way.

However:
A) those issues don’t happen that often, in my experience (if they happen all the time to you, then definitely consider Eric’s idea)
B) Most sites I know of have at least one “Mother of all Test Suites” to cover and catch bugs that slip past the “lesser” test suites.
c) The amount of “waste” involved in fairly meaningless

I applaud UrbanCode’s innovative thinking. If this is something that speaks to you, by all means, give it a try. But don’t trash your existing build model just yet.

Hudson is an open source CI server that is by far the easiest one to configure. Second to ease of use is Hudson’s impressive plug-in framework, which makes it easy to add features. For instance, Hudson has a plug-in for tracking FindBugs issues, PMD issues, and CheckStyle issues over time as well as code coverage. It also trends test results from JUnit, as well as build results and corresponding execution times. In spite of all these cool features, we had to find ways to get around some common issues we faced at work using Hudson.

This article describes a few real-life tips and tricks that we have found at work and will assist in configuring Hudson to work most effectively in your environment as well:

1. Changing Hudson home directory
2. OutOfMemoryError
3. Securing Hudson
4. Hudson Views
5. Hudson and Groovy
6. Spaces in directory names
7. Browser to use

Changing Hudson home directory:
Hudson by default uses the $USER/.hudson directory as its home directory. If you want to change the home directory, set the HUDSON_HOME environment variable to some other location before startup. This is where Hudson maintains its configuration files and manages its jobs. If you are using a batch script to run Hudson, you can set the HUDSON_HOME there as well:

set HUDSON_HOME=c:\hudson\ci_jobs

OutOfMemoryError:
If you are seeing this error, set the maximum heap size as such:
java -Xmx512m -jar hudson.war If you are seeing the same error in spite of increasing the memory, report the issue to Hudson mailing list and take a look at the following link:
http://hudson.gotdns.com/wiki/display/HUDSON/I’m+getting+OutOfMemoryError

Securing Hudson:
Hudson does not perform any security check in its default mode. This means any person accessing the website can configure Hudson and jobs, and perform builds, and download all the files from within the workspace. Hudson can be configured to authenticate users and enforce access control. In the “Matrix-based security”, you can basically configure which users have what permissions in a big table. From the Hudson dashboard, click on Manage Hudson link on the left hand side which will take you to a page as shown below:

It is also a good idea to secure the Workspace as shown above. If not, once you have a successful build, anyone can download the source code from the Hudson Dashboard as such:

Hudson Views:
Rather than having a long list of all the Jobs, organize your jobs into different views; like Commit Builds, Nightly Builds and so on.

Hudson and Groovy:

The Groovy plug-in which can be found here is very useful. In our case, we are using it for many purposes. To name a few:
1. To unlock all the files which are locked by the SCM. Download a free utility called unlocker.exe and run the groovy script to unlock files as such:

new File("directory where files are getting locked").eachFileRecurse{file ->
def command = """"C:\\Program Files\\Unlocker\\Unlocker.exe" ${file} -s"""
def proc = command.execute()
proc.waitFor()
}

2. And also to change the file permissions on directories.

def ant = new AntBuilder()
ant.chmod(dir:"directory which needs the permission to be changed", perm:"ugo+rw", includes:"**/*.*")
println "changed permissions on dirs"

Troubleshoot Hudson:

If you want to troubleshoot Hudson, you can run arbitrary Groovy scripts. In order to do so go to the Hudson Dashboard and click on the Manage Hudson which brings up the screen as shown below:

Click on the Script Console, which will bring you another page where in you can run any Groovy script to troubleshoot Hudson.

Hudson Environment Variables:
You can easily incorporate the environment variables which are easily available within your build scripts as such:

<target name="retrieve-env-variables">
<property environment="env" />
<echo message="${env.BUILD_TAG}"/>
<echo message="${env.BUILD_NUMBER}"/>
<echo message="${env.BUILD_ID}"/>
</target>

Spaces in directory names:
If you use the default setting for Hudson-Home; it is in the “Documents and Setting” folder. This causes Ant build files to fail. Make sure you set your Hudson-Home to a directory without spaces.
Also, while creating a new Job in Hudson, it has been seen that Job names which have spaces cause trouble while running the Ant build scripts.

Use anything but IE:
Last but not the least, if you are using Internet Explorer as the default browser, switch to some other browser. I spent a day trying to fix the matrix based security and also trying to rearrange the Build Steps, and enter a Groovy Command. All worked seamlessly when I switched to Firefox.

Links to additional tips and tricks already posted here:

1. Retrieving error level
2. Versioning Hudson job cofigurations
3. Hudson’s so Groovy
4. Matrix based security and IE