This guide walks you through using Gradle to build a simple Spring YARN project.

What you’ll build

You’ll create a simple app and then build it using Gradle.

In this guide we are not trying to create fully working YARN application, instead we focus on project and build model.

What you’ll need

  • About 15 minutes

  • A favorite text editor or IDE

  • JDK 6 or later

How to complete this guide

Like most Spring Getting Started guides, you can start from scratch and complete each step, or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.

To start from scratch, move on to Set up the project.

To skip the basics, do the following:

When you’re finished, you can check your results against the code in gs-gradle-yarn/complete.

If you’re not familiar with gradle or don’t have it installed, refer to Building Java Projects with Gradle.

Set up the project

First you set up a Java project for Gradle to build. To keep the focus on Gradle, make the project as simple as possible for now.

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure:

├── gs-gradle-yarn-appmaster
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── appmaster
├── gs-gradle-yarn-container
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── container
├── gs-gradle-yarn-client
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── client
└── gs-gradle-yarn-dist

for example, on *nix systems, with:

mkdir -p gs-gradle-yarn-appmaster/src/main/resources
mkdir -p gs-gradle-yarn-appmaster/src/main/java/hello/appmaster
mkdir -p gs-gradle-yarn-container/src/main/resources
mkdir -p gs-gradle-yarn-container/src/main/java/hello/container
mkdir -p gs-gradle-yarn-client/src/main/resources
mkdir -p gs-gradle-yarn-client/src/main/java/hello/client
mkdir -p gs-gradle-yarn-dist

First, let’s create a ContainerApplication class.

gs-gradle-yarn-container/src/main/java/hello/container/ContainerApplication.java

package hello.container;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class ContainerApplication {

	public static void main(String[] args) {
		SpringApplication.run(ContainerApplication.class, args);
	}

}

Next, we create an AppmasterApplication class.

gs-gradle-yarn-appmaster/src/main/java/hello/appmaster/AppmasterApplication.java

package hello.appmaster;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class AppmasterApplication {

	public static void main(String[] args) {
		SpringApplication.run(AppmasterApplication.class, args);
	}

}

The last Java class we need to create is a ClientApplication class.

gs-gradle-yarn-client/src/main/java/hello/client/ClientApplication.java

package hello.client;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class ClientApplication {

	public static void main(String[] args) {
		SpringApplication.run(ClientApplication.class, args);
	}

}

Now we need to create an application YAML configuration file for all sub-projects.

gs-gradle-yarn-container/src/main/resources/application.yml gs-gradle-yarn-appmaster/src/main/resources/application.yml gs-gradle-yarn-client/src/main/resources/application.yml

spring:
    yarn:
        appName: gs-yarn-gradle

Understanding Gradle Usage with Spring YARN

We need to create a build.gradle file where we define everything needed for the build.

We will be using the Spring Boot Gradle plugin so repository and dependency needs to be defined within a buildscript section.

buildscript {
    repositories {
        maven { url "http://repo.spring.io/libs-release" }
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:1.3.3.RELEASE")
    }
}

We apply gradle base plugin to all projects. Technically it’s only needed for a root project order to have i.e. clean task in it. But it doesn’t matter if we add it to all projects.

allprojects {
    apply plugin: 'base'
}

Below that section we add the following — project version as 0.1.0, apply plugins for java, eclipse, idea and add repositories to resolve all of the third party jar dependencies via Maven repositories. Next we add spring-boot plugin, spring-yarn-boot maven dependency to all sub-projects. Additionally we create a copyJars task to copy created jars from a sub-projects into a distribute project named gs-yarn-testing-dist. This will make things easier to test and run your application.

subprojects { subproject ->
    apply plugin: 'java'
    apply plugin: 'eclipse'
    apply plugin: 'idea'
    version =  '0.1.0'
    repositories {
        mavenCentral()
        maven { url "http://repo.spring.io/libs-release" }
    }
    dependencies {
        compile("org.springframework.data:spring-yarn-boot:2.4.0.RELEASE")
    }
    task copyJars(type: Copy) {
        from "$buildDir/libs"
        into "$rootDir/gs-gradle-yarn-dist/target/gs-gradle-yarn-dist/"
        include "**/*.jar"
    }
    assemble.doLast {copyJars.execute()}
}

We add configure section for each sub-project. These are empty for now but needed in further guides when more specific dependencies are added. The Gradle plugin for Spring Boot automatically creates a task to repackage a main jar file created from a project.

project('gs-gradle-yarn-client') {
    apply plugin: 'spring-boot'
}

project('gs-gradle-yarn-appmaster') {
    apply plugin: 'spring-boot'
}

project('gs-gradle-yarn-container') {
    apply plugin: 'spring-boot'
}

Next, related to what we previously did, in project gs-gradle-yarn-dist we add a compile dependency for all sub-projects. This is required later guides when we create tests to this project.

project('gs-gradle-yarn-dist') {
    dependencies {
        compile project(":gs-gradle-yarn-client")
        compile project(":gs-gradle-yarn-appmaster")
        compile project(":gs-gradle-yarn-container")
        testCompile("org.springframework.data:spring-yarn-boot-test:2.4.0.RELEASE")
        testCompile("org.hamcrest:hamcrest-core:1.2.1")
        testCompile("org.hamcrest:hamcrest-library:1.2.1")
    }
    test.dependsOn(':gs-gradle-yarn-client:assemble')
    test.dependsOn(':gs-gradle-yarn-appmaster:assemble')
    test.dependsOn(':gs-gradle-yarn-container:assemble')
    clean.doLast {ant.delete(dir: "target")}
    jar.enabled = false
}

It is worth to go through what we did in above section for gs-gradle-yarn-dist:

  • Eventually we want to use this project to collect other artifacts together and create some tests. That is why we added dependencies to other projects.

  • We used test.dependsOn for all other projects assembly task to make sure that Spring Boot’s gradle plugin does a repackage before any tests are run.

  • We ask clean task to wipe up our target directory. This sounds very maven centric but in our samples we try to support same project structure for both maven and gradle. Also due to limitations in Hadoop’s testing classes, some files are hard coded to be created under target directory.

  • We disabled jar task for this project to reduce noise. Disabling this really doesn’t matter but if there is only tests classes in this project this jar would be empty and thus unnecessary.

Last we just add a normal gradle wrapper task which is not needed for project itself but allows to use this build file without installing gradle binaries.

task wrapper(type: Wrapper) {
    gradleVersion = '1.11'
}

We need to create a settings.gradle file where we define a name of a root project and include sub-projects.

include 'gs-gradle-yarn-client'
include 'gs-gradle-yarn-appmaster'
include 'gs-gradle-yarn-container'
include 'gs-gradle-yarn-dist'

Declaring Hadoop dependencies

Spring for Apache Hadoop provides necessary transitive dependencies via the project’s Maven POM files. The default is using Apache Hadoop 2.6.x but there are additional versions available for other Hadoop distributions like Pivotal HD, Hortonworks Data Platform or Cloudera CDH. By specifying the desired version, your project will automatically resolve the correct Hadoop distribution dependencies.

Apache Hadoop 2.6.x
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE")
}
Pivotal HD 2.1
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE-phd21")
}
Hortonworks Data Platform 2.2
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE-hdp22")
}
Cloudera CDH 5.x
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE-cdh5")
}
Apache Hadoop 2.4.x
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE-hadoop24")
}
Apache Hadoop 2.5.x
dependencies {
    compile("org.springframework.data:spring-yarn-boot:2.1.0.RELEASE-hadoop25")
}

Build Application Packages

Run build.

gradle clean build

You should see three jar files created.

gs-gradle-yarn-dist/target/gs-gradle-yarn-dist/gs-gradle-yarn-client-0.1.0.jar
gs-gradle-yarn-dist/target/gs-gradle-yarn-dist/gs-gradle-yarn-container-0.1.0.jar
gs-gradle-yarn-dist/target/gs-gradle-yarn-dist/gs-gradle-yarn-appmaster-0.1.0.jar

You can run the project with java -jar gs-gradle-yarn-dist/target/dist/gs-gradle-yarn-client-0.1.0.jar but we haven’t added the code to actually submit the YARN application yet. So, all you will see is some logging mesages. You will build a complete application that you can submit to YARN in the other Getting Started Guides for Spring YARN.

Summary

Congratulations! You have now created a simple yet effective Gradle build file for building Spring YARN projects.

Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.

All guides are released with an ASLv2 license for the code, and an Attribution, NoDerivatives creative commons license for the writing.