Build Pipeline on Jenkins with incremental Maven builds and automatic versioning

Creating robust build pipelines on Continuous Integration servers such as Jenkins is still quite a challenge. Especially when it comes to versioning and performance (duration of the builds).

I’m going to show here how to use the nonsnapshot-maven-plugin together with Maven 3.2.x to build fully automated build pipelines with automatic version updates and incremental (fast) builds.

The pipeline will have the following features:

  • Update upstream dependencies versions (from a previous build step)
  • Update the version of all modules with code changes or updated (upstream) dependencies
  • Build (compile, test and deploy) of all modules with a new version
  • Commit changed POM files

Here a diagram how the steps of our pipeline shall work:
build_pipeline10

Example

Let’s say we have two projects depending on each other (Project 2 depends on Project 1, therefore the build pipeline consists of two consecutive steps):

build_pipeline5

The build of Project 1 starts and the nonsnapshot-maven-plugin detects code changes in Module 1. So, it updates the version and also the version of Module 2, because it has a dependency to Module 1. The two changed Modules will be compiled, tested and deployed.
At the end the build of Project 2 will be triggered, because it is a downstream project. The nonsnapshot-maven-plugin detects a newer version of Project 1 Module 2 in the Maven repository und updates the version of Module 2 accordingly. Maven will compile, test and deploy only this module. Like this:

build_pipeline6

Maven Configuration

For the incremental build you need at least Maven 3.2.1. Additionally you have configure the nonsnapshot-maven-plugin for each root (aggregator) project of your build pipeline. Below an example configuration for GIT, which considers all Maven dependencies with the groupId at.nonblocking and org.springframework as upstream dependencies (built by previous build steps):

<build>
    <plugins>
        <plugin>
            <groupId>at.nonblocking</groupId>
            <artifactId>nonsnapshot-maven-plugin</artifactId>
            <version>2.0.9</version>
            <configuration>
                <baseVersion>${myproject.base.version}</baseVersion>
                <scmType>GIT</scmType> <!-- GIT or SVN -->
                <!-- <scmUser></scmUser> -->
                <scmPassword>${git.passphrase}</scmPassword>
                <deferPomCommit>true</deferPomCommit>
                <generateIncrementalBuildScripts>true</generateIncrementalBuildScripts>        
                <!-- <generateChangedProjectsPropertyFile>true</generateChangedProjectsPropertyFile> -->
                <upstreamDependencies>
                    <upstreamDependency>at.nonblocking:*:LATEST</upstreamDependency>
                    <upstreamDependency>org.springframework:*:4.0</upstreamDependency>
                </upstreamDependencies>
            </configuration>
        </plugin>
    </plugins>
</build>

<pluginRepositories>
	<pluginRepository>
		<id>jcenter</id>
		<url>http://jcenter.bintray.com</url>
	</pluginRepository>
</pluginRepositories>

The baseVersion defines the version applied to changed modules. The option generateIncrementalBuildScripts enables the generation of a script for an incremental build.
The upstream dependency configuration org.springframework:*:4.0 means: Try to find newer versions of artifacts with the groupId org.springframework, but consider only 4.0.x versions.

On the command line execute a build step as follows:

cd <myproject-root>
mvn nonsnapshot:updateVersions -Dmyproject.base.version=1.2.3
./nonsnapshotBuildIncremental.sh clean install
mvn nonsnapshot:commitVersions

Note, you don’t have to touch any code to update the base version. Just change the property on the command line.

Jenkins Configuration

To create a build step on Jenkins, enable the option generateChangedProjectsPropertyFile in the nonsnapshot-maven-plugin configuration. It will create a property file with the changed modules, which can be passed to the Maven build for an incremental build. Furthermore you’ll need the EnvInject Jenkins plugin.

Configure the Jobs as follows:

build_pipeline1

(The Check-out strategy could also be Always check out a fresh copy)

build_pipeline2

build_pipeline3

build_pipeline4

build_pipeline5

Something similar should be possible on other CI servers.

Bringing the Java and Javascript world together

Many companies with Java as primary platform nowadays start to write pure HTML5/JavaScript front-ends, which is basically a good idea, since most Java Web Frameworks are quite painful.

The question is, how to integrate such an App into the existing Maven/Graddle build and how to deploy it on the application server. Since JavaScript has its own ecosystem of build and packaging systems, such as Node.js, Grunt or Bower.

One approach, I see a lot, is to integrate HTML5/JavaScript completely into the Java Project structure and let Maven trigger the JavaScript build. The idea is to just put all the JavaScript code and assets into a Maven WAR module and use the grunt-maven-plugin, or something similar, to launch the Grunt build.

Well, that works, but I’d advice against this approach, because:

  • It’s odd to couple two full featured build systems
  • Their build phases are not synchronized: The whole Grunt build, including the JavaScript tests, is executed in the Maven Test phase (default)
  • It slows things up to launch Maven and the Java VM to just start-up V8 and Grunt
  • The developers will start to use the Java Application Server instead of node.js for development, which is significantly slower and doesn’t allowed to run jshint and the JavaScript unit tests in the background
  • As your JavaScript applications grow you might want to re-use some Code and CSS in other Apps. That’s not possible with the integrated approach.
  • You might hire some JavaScript hackers which are not pretty familiar with the Java tooling landscape.

I present here an approach, which builds the Java and JavaScript stuff separately and merges it together at the end. The steps are:

  1. On the JavaScript side:
    1. Create separated HTML5/JavaScript modules with Grunt and Bower as build and packaging system
    2. Use a Grunt compress task to create Bower compatible ZIP file as the last build step
    3. Use a custom Grunt task to generate a Maven POM file, which just includes a build-helper:attach-artifact goal to turn the Bower ZIP file into a Maven artifact
    4. Add the project as dependency to another Bower module by just stating the full URL to the generated ZIP file. Which could of course be an URL of a Nexus or Artifactory repository.
  2. On the Build Server:
    1. On your build server run mvn install in the folder of the generated POM to install the Bower ZIP as Maven artifact in your local repository
    2. Install the artifact on Nexus or Artifactory
  3. On the Java side:
    1. Add the ZIP as dependency and unpack it into your WAR module in the package phase

Create the HTML5/JavaScript projects

To make things a bit more interesting we’re going to creating two Bower modules: A Theme, with just a single CSS file, and a AngularJS App.

The Theme project contains just a css/theme.css and a bower.json configuration like this:

{
  "name": "myTheme",
  "version": "1.0.0-SNAPSHOT",
  "main": "theme.css",
  "dependencies": {
    "bootstrap": "~3.1"
  }
}

In package.json we need to add grunt-contrib-compress as a development dependency:

{
  "dependencies": {
  },
  "devDependencies": {
    "grunt": "~0.4.1",
    "bower": "~1.3",
    "grunt-contrib-cssmin": "~0.7.0",
    "grunt-contrib-compress": "~0.7.0"
  },
  "scripts": {
    "postinstall": "bower install"
  },
  "engines": {
    "node": ">=0.8.0"
  }
}

In Gruntfile.js we have to add a compress task and a custom task to generate a pom.xml:

module.exports = function (grunt) {
  grunt.loadNpmTasks('grunt-contrib-cssmin');
  grunt.loadNpmTasks('grunt-contrib-compress');
  grunt.initConfig({   
   cssmin: {
    combine: {
    files: {
      'dist/theme.css': ['css/theme.css']
    }
    }
  },
  compress: {
    main: {
     options: {
       archive: 'deploy/myTheme.zip'
     },
      files: [
        { expand: true, cwd: 'dist/', src: ['**'], dest: '/'},
        { src: ['bower.json'], dest: '/'}
      ]}
    }
  });
  
  grunt.registerTask('mavenpom', function () {
   var bowerJson = grunt.file.readJSON('bower.json');
   grunt.log.writeln('Determined project version: ' + bowerJson.version);
 
   var pom = '<?xml version="1.0" encoding="UTF-8"?>\n' +
     '<project xmlns="http://maven.apache.org/POM/4.0.0"' +
     ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' +
     ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">\n' +
     '  <modelVersion>4.0.0</modelVersion>\n' +
     '  <groupId>at.nonblocking.bower</groupId>\n' +
     '  <artifactId>myTheme</artifactId>\n' +
     '  <packaging>pom</packaging>\n' +
     '  <version>' + bowerJson.version + '</version>\n' +
     '  <build>\n' +
     '    <plugins>\n' +
     '      <plugin>\n' +
     '        <groupId>org.codehaus.mojo</groupId>\n' +
     '        <artifactId>build-helper-maven-plugin</artifactId>\n' +
     '        <version>1.8</version>\n' +
     '        <executions>\n' +
     '          <execution>\n' +
     '            <id>attach-artifacts</id>\n' +
     '            <phase>package</phase>\n' +
     '            <goals>\n' +
     '              <goal>attach-artifact</goal>\n' +
     '            </goals>\n' +
     '            <configuration>\n' +
     '              <artifacts>\n' +
     '                <artifact>\n' +
     '                  <file>deploy/myTheme.zip</file>\n' +
     '                   <type>zip</type>\n' +
     '                </artifact>\n' +
     '              </artifacts>\n' +
     '            </configuration>\n' +
     '          </execution>\n' +
     '        </executions>\n' +
     '      </plugin>\n' +
     '    </plugins>\n' +
     '  </build>\n' +
     '</project> ';
 
   grunt.log.writeln('Writing deploy/pom.xml');
   grunt.file.write('deploy/pom.xml', pom);
  });
  
  grunt.registerTask('default', [
    'cssmin',
    'compress',
    'mavenpom'
  ]);
};

Please note, the POM artifactId and version is taken from bower.json!

If we now execute:

npm install
grunt
cd deploy
mvn install

a Bower module is generated and installed it as Maven artifact.

For the actual app we can use Yeoman to generate a AngularJS hello world example:

npm install -g yo
npm install -g generator-angular  
yo angular

And should get a structure like this:

angular_app_structure

Let’s say, the myTheme project is already on an artifactory repository, then you can add it as dependency like this:

{
  "name": "myApp",
  "version": "1.0.0-SNAPSHOT",
  "dependencies": {
    "angular": "1.2.16",
    "angular-resource": "1.2.16",
    "angular-cookies": "1.2.16",
    "angular-sanitize": "1.2.16",
    "angular-route": "1.2.16"
    "myTheme": "http://myserver/artifactory/release/at/nonblocking/bower/myTheme-1.0.0-SNAPSHOT/myTheme-1.0.0-SNAPSHOT.zip"
  }
}

Furthermore you have to add the same compress and mavenpom task to Gruntfile.js as for the myTheme module.

Create the Jenkins Jobs

On Jenkins we will need two jobs to build myTheme and myApp. Just create two free-style projects, configure the source code management section accordingly and add node.js to the PATH:

jenkins_nodejs

Then create two build tasks for each job:

  1. An Execute Shell build step to start the grunt default task
  2. And a Maven step to install the generated bower ZIP locally in your repository

jenkins_grunt

The last step on Jenkins is to publish the ZIP artifact in your remote repository:
jenkins_artifactory

Create the Java WAR module

The configuration of the WAR module is straight forward: Add the ZIP files as dependencies and use the dependency plugin to unzip it into the target/projectName folder:

<dependencies>
    <dependency>
        <groupId>at.nonblocking.bower</groupId>
        <artifactId>myTheme</artifactId>
        <version>1.0.0-SNAPSHOT</version>
        <type>zip</type>
    </dependency>
    <dependency>
        <groupId>at.nonblocking.bower</groupId>
        <artifactId>myApp</artifactId>
        <version>1.0.0-SNAPSHOT</version>
        <type>zip</type>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.2</version>
            <executions>
                <execution>
                    <id>unpack-bower-bundles</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>unpack-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeTypes>zip</includeTypes>
                        <outputDirectory>
                            ${project.build.directory}/${project.build.finalName}/
                        </outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

That’s it! Now your HTML5 front-end team can work completely independently from your Java back-end team. And both teams will use the existing Maven repository.