This guide walks you through using Maven to build a simple Spring YARN project.

What you’ll build

You’ll create a simple app and then build it using Maven.

In this guide we are not trying to create fully working YARN application, instead we focus on project and build model.

What you’ll need

  • About 15 minutes

  • A favorite text editor or IDE

  • JDK 6 or later

How to complete this guide

Like most Spring Getting Started guides, you can start from scratch and complete each step, or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.

To start from scratch, move on to Set up the project.

To skip the basics, do the following:

When you’re finished, you can check your results against the code in gs-maven-yarn/complete.

If you’re not familiar with gradle, refer to Building Java Projects with Maven.

Set up the project

First you set up a Java project for Maven to build. To keep the focus on Maven, make the project as simple as possible for now.

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure:

├── gs-maven-yarn-appmaster
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── appmaster
├── gs-maven-yarn-container
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── container
├── gs-maven-yarn-client
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── client
└── gs-maven-yarn-dist

for example, on *nix systems, with:

mkdir -p gs-maven-yarn-appmaster/src/main/resources
mkdir -p gs-maven-yarn-appmaster/src/main/java/hello/appmaster
mkdir -p gs-maven-yarn-container/src/main/resources
mkdir -p gs-maven-yarn-container/src/main/java/hello/container
mkdir -p gs-maven-yarn-client/src/main/resources
mkdir -p gs-maven-yarn-client/src/main/java/hello/client
mkdir -p gs-maven-yarn-dist

Here you create a ContainerApplication class.

gs-maven-yarn-container/src/main/java/hello/container/ContainerApplication.java

package hello.container;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class ContainerApplication {

	public static void main(String[] args) {
		SpringApplication.run(ContainerApplication.class, args);
	}

}

Here you create an AppmasterApplication class.

gs-maven-yarn-appmaster/src/main/java/hello/appmaster/AppmasterApplication.java

package hello.appmaster;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class AppmasterApplication {

	public static void main(String[] args) {
		SpringApplication.run(AppmasterApplication.class, args);
	}

}

Here you create a ClientApplication class.

gs-maven-yarn-client/src/main/java/hello/client/ClientApplication.java

package hello.client;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

@EnableAutoConfiguration
public class ClientApplication {

	public static void main(String[] args) {
		SpringApplication.run(ClientApplication.class, args);
	}

}

Here you create a application yaml configuration file for all sub-projects.

gs-maven-yarn-container/src/main/resources/application.yml

spring:
    yarn:
        appName: gs-yarn-maven

gs-maven-yarn-appmaster/src/main/resources/application.yml

spring:
    yarn:
        appName: gs-yarn-maven

gs-maven-yarn-client/src/main/resources/application.yml

spring:
    yarn:
        appName: gs-yarn-maven

Understanding Maven Usage with Spring YARN

Here you create a main parent pom.xml file.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-maven-yarn</artifactId>
    <version>0.1.0</version>
    <packaging>pom</packaging>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.3.RELEASE</version>
    </parent>

    <modules>
        <module>gs-maven-yarn-container</module>
        <module>gs-maven-yarn-appmaster</module>
        <module>gs-maven-yarn-client</module>
        <module>gs-maven-yarn-dist</module>
    </modules>

    <dependencies>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-yarn-boot</artifactId>
            <version>2.4.0.RELEASE</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptors>
                        <descriptor>assembly.xml</descriptor>
                    </descriptors>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>spring-release</id>
            <url>http://repo.spring.io/libs-release</url>
            <snapshots><enabled>false</enabled></snapshots>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>spring-release</id>
            <url>http://repo.spring.io/libs-release</url>
            <snapshots><enabled>false</enabled></snapshots>
        </pluginRepository>
    </pluginRepositories>

</project>

We created a project structure placing appmaster, container and client in its own sub-projects. Additionally we added a fourth project named dist which only purpose is to use assembly plugin to collect all artifacts together when project is build.

Here you create an appmaster pom.xml file.

gs-maven-yarn-appmaster/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-maven-yarn-appmaster</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework</groupId>
        <artifactId>gs-maven-yarn</artifactId>
        <version>0.1.0</version>
    </parent>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Here you create a container pom.xml file.

gs-maven-yarn-container/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-maven-yarn-container</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework</groupId>
        <artifactId>gs-maven-yarn</artifactId>
        <version>0.1.0</version>
    </parent>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Here you create a client pom.xml file.

gs-maven-yarn-client/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-maven-yarn-client</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework</groupId>
        <artifactId>gs-maven-yarn</artifactId>
        <version>0.1.0</version>
    </parent>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Here you create a distribution pom.xml file.

gs-maven-yarn-dist/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-maven-yarn-dist</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework</groupId>
        <artifactId>gs-maven-yarn</artifactId>
        <version>0.1.0</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>gs-maven-yarn-client</artifactId>
            <version>0.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>gs-maven-yarn-appmaster</artifactId>
            <version>0.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>gs-maven-yarn-container</artifactId>
            <version>0.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-yarn-boot-test</artifactId>
            <version>2.4.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-core</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.hamcrest</groupId>
            <artifactId>hamcrest-library</artifactId>
            <version>1.2.1</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <id>distro-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <finalName>${project.name}</finalName>
                            <appendAssemblyId>false</appendAssemblyId>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

Here you create an assembly assembly.xml file.

gs-maven-yarn-dist/assembly.xml

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">

    <id>dist</id>
    <formats>
        <format>dir</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <moduleSets>
        <moduleSet>
            <useAllReactorProjects>true</useAllReactorProjects>
            <includes>
                <include>org.springframework:gs-maven-yarn-client</include>
                <include>org.springframework:gs-maven-yarn-appmaster</include>
                <include>org.springframework:gs-maven-yarn-container</include>
            </includes>
            <binaries>
                <unpack>false</unpack>
            </binaries>
        </moduleSet>
    </moduleSets>
</assembly>

Declaring Hadoop dependencies

Spring for Apache Hadoop provides necessary transitive dependencies via the project’s Maven POM files. The default is using Apache Hadoop 2.6.x but there are additional versions available for other Hadoop distributions like Pivotal HD, Hortonworks Data Platform or Cloudera CDH. By specifying the desired version, your project will automatically resolve the correct Hadoop distribution dependencies.

Apache Hadoop 2.6.x
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE</version>
    </dependency>
</dependencies>
Pivotal HD 2.1
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE-phd21</version>
    </dependency>
</dependencies>
Hortonworks Data Platform 2.2
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE-hdp22</version>
    </dependency>
</dependencies>
Cloudera CDH 5.x
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE-cdh5</version>
    </dependency>
</dependencies>
Apache Hadoop 2.4.x
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE-hadoop24</version>
    </dependency>
</dependencies>
Apache Hadoop 2.5.x
<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-yarn-boot</artifactId>
        <version>2.1.0.RELEASE-hadoop25</version>
    </dependency>
</dependencies>

Build Application Packages

Run build.

mvn clean package

You should see three jar files created.

gs-maven-yarn-dist/target/gs-maven-yarn-dist/gs-maven-yarn-client-0.1.0.jar
gs-maven-yarn-dist/target/gs-maven-yarn-dist/gs-maven-yarn-container-0.1.0.jar
gs-maven-yarn-dist/target/gs-maven-yarn-dist/gs-maven-yarn-appmaster-0.1.0.jar

You can run the project with java -jar gs-maven-yarn-dist/target/dist/gs-maven-yarn-client-0.1.0.jar but we haven’t added the code to actually submit the YARN application yet. So, all you will see is some logging mesages. You will build a complete application that you can submit to YARN in the other Getting Started Guides for Spring YARN.

Summary

Congratulations! You have now created a simple yet effective Maven build file for building Spring YARN projects.

Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.

All guides are released with an ASLv2 license for the code, and an Attribution, NoDerivatives creative commons license for the writing.