Build Pipeline on Jenkins with incremental Maven builds and automatic versioning

Creating robust build pipelines on Continuous Integration servers such as Jenkins is still quite a challenge. Especially when it comes to versioning and performance (duration of the builds).

I’m going to show here how to use the nonsnapshot-maven-plugin together with Maven 3.2.x to build fully automated build pipelines with automatic version updates and incremental (fast) builds.

The pipeline will have the following features:

  • Update upstream dependencies versions (from a previous build step)
  • Update the version of all modules with code changes or updated (upstream) dependencies
  • Build (compile, test and deploy) of all modules with a new version
  • Commit changed POM files

Here a diagram how the steps of our pipeline shall work:
build_pipeline10

Example

Let’s say we have two projects depending on each other (Project 2 depends on Project 1, therefore the build pipeline consists of two consecutive steps):

build_pipeline5

The build of Project 1 starts and the nonsnapshot-maven-plugin detects code changes in Module 1. So, it updates the version and also the version of Module 2, because it has a dependency to Module 1. The two changed Modules will be compiled, tested and deployed.
At the end the build of Project 2 will be triggered, because it is a downstream project. The nonsnapshot-maven-plugin detects a newer version of Project 1 Module 2 in the Maven repository und updates the version of Module 2 accordingly. Maven will compile, test and deploy only this module. Like this:

build_pipeline6

Maven Configuration

For the incremental build you need at least Maven 3.2.1. Additionally you have configure the nonsnapshot-maven-plugin for each root (aggregator) project of your build pipeline. Below an example configuration for GIT, which considers all Maven dependencies with the groupId at.nonblocking and org.springframework as upstream dependencies (built by previous build steps):

<build>
    <plugins>
        <plugin>
            <groupId>at.nonblocking</groupId>
            <artifactId>nonsnapshot-maven-plugin</artifactId>
            <version>2.0.9</version>
            <configuration>
                <baseVersion>${myproject.base.version}</baseVersion>
                <scmType>GIT</scmType> <!-- GIT or SVN -->
                <!-- <scmUser></scmUser> -->
                <scmPassword>${git.passphrase}</scmPassword>
                <deferPomCommit>true</deferPomCommit>
                <generateIncrementalBuildScripts>true</generateIncrementalBuildScripts>        
                <!-- <generateChangedProjectsPropertyFile>true</generateChangedProjectsPropertyFile> -->
                <upstreamDependencies>
                    <upstreamDependency>at.nonblocking:*:LATEST</upstreamDependency>
                    <upstreamDependency>org.springframework:*:4.0</upstreamDependency>
                </upstreamDependencies>
            </configuration>
        </plugin>
    </plugins>
</build>

<pluginRepositories>
	<pluginRepository>
		<id>jcenter</id>
		<url>http://jcenter.bintray.com</url>
	</pluginRepository>
</pluginRepositories>

The baseVersion defines the version applied to changed modules. The option generateIncrementalBuildScripts enables the generation of a script for an incremental build.
The upstream dependency configuration org.springframework:*:4.0 means: Try to find newer versions of artifacts with the groupId org.springframework, but consider only 4.0.x versions.

On the command line execute a build step as follows:

cd <myproject-root>
mvn nonsnapshot:updateVersions -Dmyproject.base.version=1.2.3
./nonsnapshotBuildIncremental.sh clean install
mvn nonsnapshot:commitVersions

Note, you don’t have to touch any code to update the base version. Just change the property on the command line.

Jenkins Configuration

To create a build step on Jenkins, enable the option generateChangedProjectsPropertyFile in the nonsnapshot-maven-plugin configuration. It will create a property file with the changed modules, which can be passed to the Maven build for an incremental build. Furthermore you’ll need the EnvInject Jenkins plugin.

Configure the Jobs as follows:

build_pipeline1

(The Check-out strategy could also be Always check out a fresh copy)

build_pipeline2

build_pipeline3

build_pipeline4

build_pipeline5

Something similar should be possible on other CI servers.

Writing a Maven plugin with Scala

When you’re going to write Maven plugins with Scala you’ve to consider two things:

  1. XDoclet plugin descriptors won’t work with Scala, so you’ve to use Java 5 annotations (in org.apache.maven.plugin-tools:maven-plugin-annotations)
  2. You have to make sure that the scala:compile goal is executed prior to plugin:descriptor, otherwise the maven-plugin-plugin is not able to detect the annotations. So, either you change the default phase of plugin:descriptor (which is generate-resources) to compile (or later), or you compile the Scala files before generate-resources, e.g. in the process-sources phase (which is BTW the same you would do in a mixed Java/Scala environment – see here).

Here an example Mojo written in Scala:

import java.io.File
import org.apache.maven.plugin.AbstractMojo
import org.apache.maven.plugins.annotations.Mojo
import org.apache.maven.plugins.annotations.Parameter

@Mojo(name = "test")
class TestMojo extends AbstractMojo {

  @Parameter(required = true)
  var myParam1: String = _

  @Parameter(required = false)
  var myParam2: String = _
  
  @Parameter(defaultValue = "${project.basedir}")
  var baseDirectory: File = _

  def execute() = {
    getLog.debug("Running Mojo test");
    getLog.debug("Base dir: " + baseDirectory)
    
    //TODO
  }
}

And the corresponding pom.xml:

<dependencies>
	<dependency>
		<groupId>org.apache.maven</groupId>
		<artifactId>maven-plugin-api</artifactId>
		<version>3.0.4</version>
	</dependency>
	<dependency>
		<groupId>org.apache.maven.plugin-tools</groupId>
		<artifactId>maven-plugin-annotations</artifactId>
		<version>3.2</version>
		<scope>provided</scope>
	</dependency>
	<dependency>
		<groupId>org.scala-lang</groupId>
		<artifactId>scala-library</artifactId>
		<version>${scala.version}</version>
	</dependency>
</dependencies>

<build>
	<plugins>
		<plugin>
			<groupId>org.scala-tools</groupId>
			<artifactId>maven-scala-plugin</artifactId>
			<version>2.15.2</version>
			<executions>
				<execution>
					<id>scala-compile-first</id>
					<phase>process-sources</phase>
					<goals>
						<goal>compile</goal>
					</goals>
				</execution>
				<execution>
					<id>scala-test-compile</id>
					<phase>process-test-sources</phase>
					<goals>
						<goal>testCompile</goal>
					</goals>
				</execution>
			</executions>
		</plugin>

		<plugin>
			<artifactId>maven-plugin-plugin</artifactId>
			<version>3.2</version>
			<configuration>
				<goalPrefix>myPlugin</goalPrefix>
			</configuration>
		</plugin>
	
		
		<plugin>
			<groupId>org.codehaus.mojo</groupId>
			<artifactId>build-helper-maven-plugin</artifactId>
			<version>1.8</version>
			<executions>
				<execution>
					<id>add-source</id>
					<phase>generate-sources</phase>
					<goals>
						<goal>add-source</goal>
					</goals>
					<configuration>
						<sources>
							<source>src/main/scala</source>
						</sources>
					</configuration>
				</execution>

				<execution>
					<id>add-test-source</id>
					<phase>generate-test-sources</phase>
					<goals>
						<goal>add-test-source</goal>
					</goals>
					<configuration>
						<sources>
							<source>src/test/scala</source>
						</sources>
					</configuration>
				</execution>
			</executions>
		</plugin>
	</plugins>
</build>

Pimp your unit tests

A lot of people nowadays wonder if there might be a field of application for alternative programming languages in their Java applications. A good and relatively safe approach is to allow the usage of other languages for unit tests. In fact, when it comes to testing there is really no reason to bother with Java’s verbosity and strictness.

A good candidate in this field is Groovy, because it is very simple to use, both for Java developers and non programmers (e.g. the testers in your team). Also, the interoperability with Java is excellent and Groovy is easy to integrate into an existing Maven build process.

Just create a new folder src/test/groovy in your Maven project and add the following to pom.xml:

<dependencies>
  <dependency>
    <groupId>org.codehaus.groovy</groupId>
    <artifactId>groovy-all</artifactId>
    <version>2.1.1</version>
    <scope>test</scope>
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>build-helper-maven-plugin</artifactId>
      <version>1.7</version>
      <executions>
        <execution>
          <id>add-source</id>
          <phase>generate-test-sources</phase>
          <goals>
            <goal>add-test-source</goal>
          </goals>
          <configuration>
            <sources>
              <source>src/test/groovy</source>
            </sources>
          </configuration>
        </execution>
      </executions>
    </plugin>
    <plugin>
      <groupId>org.codehaus.gmaven</groupId>
      <artifactId>gmaven-plugin</artifactId>
      <version>1.5</version>
      <executions>
        <execution>
          <goals>
            <goal>testCompile</goal>
          </goals>
          <configuration>
            <sources>
              <fileset>
                <directory>${pom.basedir}/src/test/groovy</directory>
                <includes>
                  <include>**/*.groovy</include>
                </includes>
              </fileset>
            </sources>
          </configuration>
        </execution>
      </executions>
    </plugin>
</plugins>

If you’re using Eclipse you might also want to install the Eclipse Groovy IDE: http://groovy.codehaus.org/Eclipse+Plugin

And now you’re ready to start: Create a file src/test/groovy/Test.groovy:

import org.junit.Test
import static org.junit.Assert.*

class Test {
  @Test void firstTest() {
    def str = "test" + 1
    def bd = 22.3 * 3
    assertNotNull str
    assertEquals "test1", str
    assertTrue str == 'test1'
    assertEquals 66.9, bd
    assertTrue bd.class == BigDecimal
  }
}

And start the test case as usual in your favourite IDE or with the Maven test goal. And that’s basically it.

One of the interesting features of Groovy is, that you can omit semicolons and parentheses, which makes JUnit appear like a neat internal DSL. Other Groovy features handy for testing are:

  • The possibility to compare string with ==
  • Simplified handling of BigDecimal’s
  • Support for closures, which can be used to test multiple values in one step:
    [0, 3, 5, 7, 9, 10].each{ 
      assertNotNull classUnderTest.calculate(it)) 
    }
    
  • Simplified testing of collection content. For example this assertion checks if a person with name “Foo” exists in the collection:
    def persons = //Query for a list of persons
        
    assertTrue "Foo" in person*.name