This was discovered with Clover 2.2.1

We have a build.xml file with the following pseudo-flow:

• clover-setup
• // this causes all future compilations to use the clover compiler to instrument the files.
• compile
• javac _all_source_
• test_A
• javac _unit_test_source_ // yes, this is a subset of all source
• junit _unit_tests_
• 
  
• test_B
• javac _unit_test_source_ // yes, this is a subset of all source, and redundant with test_A
• junit _integration_tests_
• clover-report
• NO DATA!
  

By inserting < clover-log / > at the end of test_A and test_B, I figured out what was going on.

1. after test_A, we have a nice coverage level
2. but at the end of test_B, we have been reset back to 0

My hypothesis is that there is a bug in Clover that causes it to:

a) wipe out all of the coverage data thus far when new code is compiled.

b) prevent new code from updating the coverage database.

And sure enough, by removing the javac compilation step from test_B, we have an accurate combined coverage report.

Moral of the story:

1. Do all of your compiling before you run any of your automated tests
2. < clover-log / > is your friend.

Typical code coverage tools give you line and path coverage information (which is certainly helpful depending on how you examine the data); however, what most tools don’t covey is directness. That is, what was the distance from the test to a particular path?– the thinking being that the closer the test, the more reliable the quality of that test. If method doOrder is 5 paths of indirection away from a test, then one could infer that this particular method may require better testing.

Project PEA is a code coverage tool that’ll monitor your JUnit tests in an attempt to gather call stacks so as to give you an idea of indirection– with PEA you can ascertain how well a particular test actually verifies say, that doOrder method.

For more information, check out PEA or view a powerpoint presentation that Matt Harrah put together explaining it.

JavaWorld has an interesting forum discussion entitled “Heinz Kabutz on Bad Code, Unit Testing, and More” in which the following thoughts are posed as a series of questions:

Kabutz also gives Java developers a hard time for not unit testing. He argues that it’s a major problem and that few Java developers do it. I’m curious: Has anyone gotten into big trouble for not unit testing? Is it a major source of bugs in your experience?

The discussion then centers around the merits of unit testing and code coverage (much like the excellent discussions from Stelligent’s TDD Horror stories evening!)– for instance, one reader states:

Striving for 100% unit test coverage is nice for libraries, but overkill for the avarage bussiness application that is written by most development teams.

What are your thoughts? Is TDD worth the effort or not?

So you want to make MSTest work on a .Net 2.0 projects and have CruiseControl.NET monitor everything for you. Well maybe “want” is a strong word. Maybe someone (your employer) told you to do so. It is possible, actually not even that hard. So here’s what I did.

My tools of choice here are MSBuild and MSTest. I agree Microsoft should distribute MSTest separately from the Visual Studios Team Edition (VSTS) for Testers. However Microsoft does not, so for now, the path of least resistance is to give Bill his pound of flesh and install the IDE on the build server. I actually installed the Visual Studios Team Suite on the build server that way I had all my favorite tools handy. The build server didn’t even whine.

I started off with an MSBuild file Build.xml that:

1. Compiles two solution files
2. Cleans out the TestResults directory
3. Runs the tests
4. Converts the code coverage to a XML file
5. Deploys the out put to a MSI file

The CCNet.config file:

1. Pulls source from SVN
2. Labels source
3. Runs a MSBuild task using the Build.xml from above
4. Merges the XML output from the MSBuild task
5. Emails the results to the appropriate people

The Dashboard displays:

1. The MSTest results
2. The Code Coverage results

Lets look at the build.xml first.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets ="Deploy" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" >
	<ItemGroup>
    <Project Include="$(MSBuildProjectDirectory)\csla20cs\cslacs.sln" />
    <Project Include="$(MSBuildProjectDirectory)\SourceMonitorSummary.sln" />
  </ItemGroup>
	<Target Name="BuildAll">
		<MSBuild Projects="@(Project)" Targets="Build"/>
	</Target>
  <ItemGroup>
    <TestFiles Include="$(MSBuildProjectDirectory)\**\*.*"/>
  </ItemGroup>
  <Target Name ='DeletetestResults' DependsOnTargets ='BuildAll'>
    <Delete Files="@TestFiles"  />
    <RemoveDir Directories="$(MSBuildProjectDirectory)\TestResults"  />
  </Target>
  <Target Name="RunTests" DependsOnTargets ="DeletetestResults">
    <RemoveDir Directories="$(MSBuildProjectDirectory)\TestResults" />
   <Exec Command='"C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\MSTest.exe" /testmetadata:SourceMonitorSummary1.vsmdi /testlist:"BuildTests" /runconfig:localtestrun.testrunconfig' />
	</Target>
  <Target Name ='TestCoverageToXML' DependsOnTargets ='RunTests'>
    <Exec Command='"C:\Program Files\MSTestCoverageToXML\bin\MSTestCoverageToXML.exe" $(MSBuildProjectDirectory)' WorkingDirectory ='$(MSBuildProjectDirectory)\TestResults'/>
  </Target>
  <Target Name ='Deploy' DependsOnTargets ='TestCoverageToXML'>
    <Exec Command='"$(MSBuildProjectDirectory)\CreateSetup.bat"'/>
  </Target>
</Project>

I’ll break it down step by step. First I build up an item group of solutions to compile. I have two solutions to deal with, SourceMonitorSummary.sln which depends on cslacs.sln. Next I execute a MSBuild task named “BuildAll” to compile the solutions listed in @(Project). That will compile cslacs.sln first and then SourceMonitorSummary.sln.

Next I delete all the files and directories in the TestResults folder. That guarantees that when the tests run I will have one and only one folder inside TestResults directory. This is to support the code coverage report. John Cunningham has a wonderful blog The Visual Studio Team System “off-road” code coverage experience on converting the code coverage data from a test run into an XML file. I have a simple version of John’s code running that navigates the TestResult directory tree to find and move the data.coverage file to the out directory and then creates a Coverage.xml file that is later merged into the CruiseControl.net build log for the project.

Now for the good part. I execute the Exec Command task named “RunTests” that runs MSTest.exe from the command line. The key here is to setup the test meta data ahead of time. You need a test list, in my case I created BuildTests. Use the “Test/Create New Test List…” menu options in VSTE for Testers. When MSTest runs it will create a test results file in the TestResults directory. The file has a .trx extension. It is an XML file that can be merged directly into the build log.

You can also enable code coverage found under “Test/Edit Test Run Configurations/Local Test Run”. Pick the artifacts that you want code coverage against.

After that I execute the MSTestCoverageToXML.exe which converts the code coverage data to an XML file. If you have not enable code coverage for the test configuartion skip this task.

Finally I run the CreateSetup.bat for deployment. I’m using a Microsoft WIX file that gathers the output from the compile and packages it into an MSI. We have a repeatable build that can be executed from a command line “MSBuild Build.xml”.

Lets look at the CCNet.config file now, and see what is needed there.

<cruisecontrol>
  <project name="SourceMonitorSummary">
     <workingDirectory>C:\CruiseControl\WorkingDirectory\dev\SourceMonitorSummary</workingDirectory>
     <artifactDirectory>C:\CruiseControl\ArtifactDirectory\devSourceMonitorSummary</artifactDirectory>
     <category>Category 1</category>
     <webURL>http://localhost/ccnet/Projects/dev/SourceMonitorSummary</webURL>
     <modificationDelaySeconds>2</modificationDelaySeconds>
     <triggers>
        <intervalTrigger name="continuous" seconds="60" buildCondition="IfModificationExists"/>
     </triggers>
     <state type="state" directory="C:\CruiseControlState" />
     <sourcecontrol type="svn">
        <trunkUrl>svn://localhost/svn/dev/SourceMonitorSummary</trunkUrl>
        <workingDirectory>C:\CruiseControl\WorkingDirectory\dev\SourceMonitorSummary</workingDirectory>
     </sourcecontrol>
     <labeller type="defaultlabeller">
        <prefix>SourceMonitorSummary</prefix>
        <incrementOnFailure>true</incrementOnFailure>
     </labeller>
     <tasks>
        <msbuild>
           <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727MSBuild.exe</executable>
           <workingDirectory>C:\CruiseControlWorkingDirectory\dev\SourceMonitorSummary</workingDirectory>
           <projectFile>Build.xml</projectFile>
           <buildArgs>/noconsolelogger /p:Configuration=Debug /v:diag</buildArgs>
           <logger>ThoughtWorks.CruiseControl.MsBuild.XmlLogger,C:\Program Files\CruiseControl.NET\webdashboard\bin\ThoughtWorks.CruiseControl.MsBuild.dll</logger>
        </msbuild>
     </tasks>
     <publishers>
        <merge>
           <files>
              <file>C:\CruiseControl\WorkingDirectory\dev\SourceMonitorSummary\TestResults\*.trx</file>
              <file>Coverage.xml</file>
           </files>
        </merge>
        <xmllogger logDir="C:\CruiseControl\buildlogs" />
       <email from="BuildMaster@MailServer.com" mailhost="MailServer" mailhostUsername="BuildMaster" mailhostPassword="pwd" includeDetails="TRUE">
       <users>
          <user name="Matt.Roswurm" group="Developers" address="Matt.Roswurm@MailServer.com"/>
          <user name="BuildMaster" group="Build" address="BuildMaster@MailServer.com"/>
          <user name="ProjectManager" group="Developers" address="ProjectManager@MailServer.com"/>
       </users>
       <groups>
          <group name="Developers" notification="failed"/>
          <group name="Build" notification="always"/>
       </groups>
       </email>
     </publishers>
   </project>
</cruisecontrol>

The interesting stuff happens in the MSBuild task. CruiseControl is executing exactly the same build that the developer executes locally. Note the custom logger in the <buildArgs>, this is required to convert the MSBuild output in to XML. Next CruiseControl will merge the output files that we created in the automated build into the build log, this will make the test results and code coverage output available to the build report. The *.trx file contains the test results and the Coverage.xml file holds the code coverage output.

Finally we have the dashboard.config changes that will display the MSTest output.

<?xml version="1.0" encoding="utf-8" ?>
<dashboard>
  <remoteServices>
    <servers>
      
      <server name="local" url="tcp://localhost:21234/CruiseManager.rem" allowForceBuild="true" allowStartStopBuild="true" />
    </servers>
  </remoteServices>
  <plugins>
    <farmPlugins>
      <farmReportFarmPlugin />
      <cctrayDownloadPlugin />
    </farmPlugins>
    <serverPlugins>
      <serverReportServerPlugin />
      <serverLogServerPlugin />
      <serverInformationServerPlugin />
    </serverPlugins>
    <projectPlugins>
      <projectReportProjectPlugin />
      <latestBuildReportProjectPlugin />
      <viewAllBuildsProjectPlugin />
      <projectStatisticsPlugin xslFileName="xslstatistics.xsl" />
      <serverLogProjectPlugin />
      <viewConfigurationProjectPlugin />
    </projectPlugins>
    <buildPlugins>
      <buildReportBuildPlugin>
        <xslFileNames>
          <xslFile>xslheader.xsl</xslFile>
          <xslFile>xslmodifications.xsl</xslFile>
          <xslFile>xslcompile.xsl</xslFile>
         
          <xslFile>xslMsTestSummary.xsl</xslFile>
          <xslFile>xslfxcop-summary.xsl</xslFile>
          <xslFile>xslNCoverSummary.xsl</xslFile>
          <xslFile>xslSimianSummary.xsl</xslFile>
          <xslFile>xslfitnesse.xsl</xslFile>
          <xslFile>xslMSTestCoverage.xsl</xslFile>
        </xslFileNames>
      </buildReportBuildPlugin>
      <buildLogBuildPlugin />
      <xslReportBuildPlugin description="NUnit Details" actionName="NUnitDetailsBuildReport" xslFileName="xsltests.xsl" />
      <xslReportBuildPlugin description="NUnit Timings" actionName="NUnitTimingsBuildReport" xslFileName="xsltiming.xsl" />
      <xslReportBuildPlugin description="NAnt Output" actionName="NAntOutputBuildReport" xslFileName="xslNAnt.xsl" />
      <xslReportBuildPlugin description="NAnt Timings" actionName="NAntTimingsBuildReport" xslFileName="xslNAntTiming.xsl" />
      <xslReportBuildPlugin description="FxCop Report" actionName="FxCopBuildReport" xslFileName="xslFxCopReport.xsl" />
      <xslReportBuildPlugin description="NCover Report" actionName="NCoverBuildReport" xslFileName="xslNCover.xsl" />
      <xslReportBuildPlugin description="Simian Report" actionName="SimianBuildReport" xslFileName="xslSimianReport.xsl"/>
      <xslReportBuildPlugin description="Fitnesse Report" actionName="FitnesseBuildReport" xslFileName="xslFitnesseReport.xsl"/>
      <xslReportBuildPlugin description="MSTest Report" actionName="MSTESTReport" xslFileName="xslMsTestSummary.xsl"/>
      <xslReportBuildPlugin description="MSTest Coverage Report" actionName="MSTestCoverageReport" xslFileName="xslMSTestCoverage.xsl"/>
    </buildPlugins>
  </plugins>
</dashboard>

There is another great blog entry covering MSTest and CruiseControl .NET at BM Bloggers. In that blog Richard talks about MSTest and displaying the test results in the build report. I have added his MSTestCoverage.xsl to the <xslFileNames> and commented out the unitttest.xsl as I’m not using NUnit. For code coverage report, John Cunningham to the rescue, Team System “off-road” code coverage analysis and reporting experience has a great description of how to format the code coverage data using XSLT transformation.

I don’t think that I have done anything really new here, I’ve just pulled together information from several sources that allow MSTest to co-exist with CruiseControl.net. I’d would love to here about other peoples experences with MSBuild/MSTest in their environments.

Paul Duvall and Levent Gurses have posted a vidcast (video podcast) discussing Levent’s presentation at the Better Software Conference next week in Vegas. Levent’s talk covers Eclipse and how a carefully selected set of plugins (JDepend4Eclipse, Checkstyle, Coverlipse, PMD, and Metrics) can help you create better quality code.

If you can’t check out his live presentation in Vegas, simply have a watch below and let us know what you think. Enjoy the show!

For those who like juggling multiple tasks, Eclipse provides a great opportunity. Take developer testing for example. JUnit comes standard out of the box and the agile developer can literally start practicing Test Driven Development (TDD) right off the bat. As individual developers increase the number and detail of unit tests the team may want to know how much of the code base is actually being tested. This is where test coverage comes into the picture.

The agile teams practicing Continuous Integration gain insight of their test coverage through tools such as CruiseControl and Cobertura. Since test coverage reports are generated as part of the CI cycle, the latter two provide an aggregate view of the test coverage of the entire code base. The reports, while open to the entire team, are mostly analyzed by the architects or project managers who would like to gain insight of the team’s coverage and trends.

Individual developers working with Eclipse can also gain the same insight into their test coverage without having to commit changes and wait for the CI server to complete the build. Coverlipse is an Eclipse plugin designed to measure the test coverage of Java code. It’s fully integrated into the Eclipse environment and allows the agile developer to see which parts of the code are actually tested. The ability to see test coverage immediately provides the shortest path to knowing what each developer’s individual test coverage is, even before a single line of code is committed to the repository.

Quick Tutorial

1. Download and start Eclipse 3.2
2. Navigate to Help | Software Updates | Find and Install
3. Select Search for new features to install
4. Add a new remote site Coverlipse and point it to http://coverlipse.sf.net/update/
5. Finish the wizard and restart Eclipse
6. Create a new Java project
7. Add the following Java class:

   public class NumberComparator {
     public NumberComparator() {
       super();
     }
     public boolean isGreater(int pFirst, int pSecond) {
       if (pFirst > pSecond) {
         return true;
       }
       return false;
     }
   }
   

8. Add the following JUnit test case:

   import junit.framework.TestCase;
   public class NumberComparatorTest extends TestCase {
     public void testIsGreater() {
       NumberComparator nComp = new NumberComparator();
       assertTrue(nComp.isGreater(5, 4));
     }
   }

9. In the Package Explorer, right-click on NumberComparatorTest and select Run As | JUnit w/ Coverlipse
10. Wait for the test to complete
11. Open the NumberComparator class. You should see the Coverlipse markers as in the following picture:
    
12. Notice the red mark on the left. This indicates that part of the method is not covered by the unit test.
13. Add a second assert to the JUnit test case as follows:

    assertFalse(nComp.isGreater(5, 6));

14. Rerun JUnit w/ Coverlipse
15. Notice there are no red marks now as the entire code is covered

Code coverage statistics are often heavily relied upon in an effort to gauge testing efforts and infer relative code quality. But, as I’ve written about before , code coverage reports can often fool you. As such, I highly recommend you use these reports not as a means for judging what’s tested but in judging what’s not tested.

It turns out that code conditionals are a bit tricky when it comes to measuring code coverage and in certain scenarios, tests can trigger high line and branch coverage rates but fail to uncover nefarious defects. In fact, I recently posted a code example that demonstrates short-circuiting code coverage via a simple OR condition .

In my posting’s code example, the second clause in an if conditional throws an exception; however, because one can easily write a test that only logically executes the first clause of the conditional, the second part of the conditional is skipped, yet a coverage tool may end up reporting 100% branch coverage as it did for me. The question then arises– how can one infer the quality of tests if code coverage reports are often liberal in their calculations?

One highly effective technique that yields much more effective testing (and therefore more precise coverage) is the notion of path testing. By measuring paths through a method (which, in essence, is the Cyclomatic complexity value of the method) one can quickly gauge how many tests one would need to achieve adequate coverage.

For example, running the Eclipse Metrics plug-in against this method yields a CC of 3. This means there are 3 logical paths through this method and accordingly, I’d need to write at least 3 tests to accurately test the branchIt method.

The branchIt method isn’t terribly complex, but look closely at its logic and see if you can spot the 3 paths.

public void branchIt(int value){
 if((value > 100) || (HiddenObject.doWork() == 0)){
  this.dontDoIt();
 }else{
  this.doIt();
 }
}  

It seems the 3 paths are divided as follows:

• The first conditional has two clauses, hence there’s 2 paths
• The later else conditional

The good news is that you’d find the defect rather quickly in this case– the mere act of trying to trigger the else clause will almost force the defect to show up, won’t it? What’s also interesting is that if you look at the Cobertura report for this code, you’ll notice that it correctly notes that the else conditional wasn’t covered in my first test. If I were trying to achieve 100% line coverage, I’d quickly uncover the defect lurking in the first clause by trying to force the else.

Paul Duvall published an article, Improving code with Eclipse plugins, the fifth installment of his “Automation for the people” series by IBM developerWorks, where he provides examples of installing, configuring, and using static analysis Eclipse plugins focused on helping developers in key areas such as:

• Coding standards
• Code duplication
• Code coverage
• Dependency analysis
• Complexity monitoring

These code analysis areas can be uncovered with five helpful plugins:

• CheckStyle: For coding standards
• PMD’s CPD: Enables discovering code duplication
• Coverlipse: Measures code coverage
• JDepend: Provides dependency analysis
• Eclipse Metrics plugin: Effectively spots complexity

Regardless if you chose one of the profiled tools or other Eclipse plugins, incorporating them into your work environment will contribute to real-time visual feedback on the state of the quality of your code so that you may prevent problems early in the development life cycle.

Despite our best intentions, many of us don’t always get to work on the latest and greatest language or technology. In some cases, we may need to make our way around a code base that has been written by another team. Given that the average software system lives for 11 years, it’s probably safe to say that there are, in fact, many such cases where we need to quickly assess a new code base to find its flaws (i.e. risks) and determine which portions of the code may be difficult to maintain and extend and which should be easier. If you’re looking at a code base that was developed by another team or even code you just wrote yesterday, once it’s written, it’s now code that you must maintain.

If you need to analyze a code base and determine which portions of the code are more risky than others, what would you do? For me, I use tools that help me quickly assess the code quality. For instance, I’d like to know something I call the “big five” measures:

• Code complexity
• Amount of duplication
• The resilience of the architecture
• Whether coding standards are being applied
• Amount of test code coverage

As I mentioned, I like to use tools to help me assess these measures. However, there are sometimes concerns when people consider using code metrics on their projects:

1) Don’t know which tools to use
2) Can be easily abused
3) Not sure what the data means

The table below links a measure to a tool (.NET and Java), the code smell associated with this measure, the refactoring for the smell and possible patterns that may be applied to refactor the code smell.

This is not a detailed list, but should give you some ideas. I plan to add to this list in the future. Do you have additional measures|tools|refactorings|patterns? Please share them with us.

Next time you need to target risky parts of your code base, use these tools to obtain metrics so that you can monitor and improve your code quality (through refactoring and patterns) throughout your development lifecycle.

Eric Sink recently wrote an article advocating the use of code coverage; where he has managed to achieve 100% coverage. I’m a huge believer that coverage data is quite valuable in the testing process, but is 100% such a good thing?

Earlier this year in his “In pursuit of code quality: Don’t be fooled by the coverage report” article, Andrew Glover noted that:

High coverage rates simply mean that a lot of code was exercised. High coverage rates do not imply that code was exercised well.

A common misconception of code coverage is that it is the final point of your quality development. Steve McConnell advises in his book, Code Complete, that “testing by itself does not improve software quality. Test results [and code coverage] are an indicator of quality, but in and of themselves, they don’t improve it. Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often.” Sound familiar?

Code coverage tools work as supplements, not substitues, for overall code quality. 100% code coverage certainly does not mean 100% tested. In the end, coverage measure is just an indicator of how various parts of your program were exercised - but gives you little confidence in your overall picture of quality.