This guide walks you through testing a simple Spring YARN project.

What you’ll build

You’ll use an existing sample application and create a simple JUnit test to verify application is executed properly.

What you’ll need

Testing this sample application don’t need existing or running Hadoop instance.

How to complete this guide

Like most Spring Getting Started guides, you can start from scratch and complete each step, or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.

To start from scratch, move on to Set up the project.

To skip the basics, do the following:

When you’re finished, you can check your results against the code in gs-yarn-testing/complete.

If you’re not familiar with gradle, refer to Building Java Projects with Gradle.

Set up the project

Focus on this guide is to use JUnit to test your sample application. We have already prepared a project for gradle and application files based on sample YARN Basic Sample. What is left for you is to create a JUnit test class.

First you set up a basic build script. You can use any build system you like when building apps with Spring, but the code you need to work with Gradle and Maven is included here. If you’re not familiar with either, refer to Building Java Projects with Gradle or Building Java Projects with Maven.

We also have additional guides having specific instructions using build systems with Spring YARN. If you’re not familiar with either, refer to Building Spring YARN Projects with Gradle or Building Spring YARN Projects with Maven.

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure:

├── gs-yarn-testing-appmaster
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── appmaster
├── gs-yarn-testing-container
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── container
├── gs-yarn-testing-client
│   └── src
│       └── main
│           ├── resources
│           └── java
│               └── hello
│                   └── client
└── gs-yarn-testing-dist
    └── src
        └── test
            └── java
                └── hello

for example, with

mkdir -p gs-yarn-testing-appmaster/src/main/resources
mkdir -p gs-yarn-testing-appmaster/src/main/java/hello/appmaster
mkdir -p gs-yarn-testing-container/src/main/resources
mkdir -p gs-yarn-testing-container/src/main/java/hello/container
mkdir -p gs-yarn-testing-client/src/main/resources
mkdir -p gs-yarn-testing-client/src/main/java/hello/client
mkdir -p gs-yarn-testing-dist/src/test/java/hello

on *nix systems:

Create a Gradle build file

Create a JUnit Test Class

gs-yarn-testing-dist/src/test/java/hello/AppIT.java

package hello;

import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.notNullValue;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.greaterThan;
import hello.client.ClientApplication;

import java.io.File;
import java.util.List;

import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.junit.Test;
import org.springframework.core.io.Resource;
import org.springframework.yarn.boot.test.junit.AbstractBootYarnClusterTests;
import org.springframework.yarn.test.context.MiniYarnClusterTest;
import org.springframework.yarn.test.junit.ApplicationInfo;
import org.springframework.yarn.test.support.ContainerLogUtils;

@MiniYarnClusterTest
public class AppIT extends AbstractBootYarnClusterTests {

	@Test
	public void testApp() throws Exception {
		String[] args = new String[] {
			"--spring.yarn.client.files[0]=file:target/gs-yarn-testing-dist/gs-yarn-testing-container-0.1.0.jar",
			"--spring.yarn.client.files[1]=file:target/gs-yarn-testing-dist/gs-yarn-testing-appmaster-0.1.0.jar" };
		ApplicationInfo info = submitApplicationAndWait(ClientApplication.class, args);
		assertThat(info.getYarnApplicationState(), is(YarnApplicationState.FINISHED));

		List<Resource> resources = ContainerLogUtils.queryContainerLogs(
				getYarnCluster(), info.getApplicationId());
		assertThat(resources, notNullValue());
		assertThat(resources.size(), is(4));

		for (Resource res : resources) {
			File file = res.getFile();
			String content = ContainerLogUtils.getFileContent(file);
			if (file.getName().endsWith("stdout")) {
				assertThat(file.length(), greaterThan(0l));
				if (file.getName().equals("Container.stdout")) {
					assertThat(content, containsString("Hello from HelloPojo"));
				}
			} else if (file.getName().endsWith("stderr")) {
				assertThat("stderr with content: " + content, file.length(), is(0l));
			}
		}
	}

}

Let’s go through step by step what’s happening in this JUnit class. As already mentioned earlier we don’t need any existing or running Hadoop instances, instead testing framework from Spring YARN provides an easy way to fire up a mini cluster where your tests can be run in an isolated environment.

We named a class as AppIT which plays nice with both gradle and maven when tests are executed during the build. Tests will need a real artifacts to be build which means test can only run after jar files has been build and processed by a Spring Boot’s repackage plugin. In gradle build file we make sure artifacts are build before tests are run. In maven build file we use failsafe plugin which is meant for integration testing and tests handled by it are executed after all project artifacts has been build.

  • We used spring.yarn.client.files property to override settings defined in application.yml file because test is run from gs-yarn-testing-dist project.

  • @MiniYarnClusterTest is a composed annotation telling Spring to start a Hadoop’s mini cluster having components for HDFS and YARN. Hadoop’s configuration from this minicluster is automatically injected into your testing context.

  • AbstractBootYarnClusterTests is a class containing a lot of base functionality what you need in your tests.

Then it’s time to deploy the application into a running minicluster

  • submitApplicationAndWait() method simply runs your ClientApplication and expects it to do an application deployment. On default it will wait 60 seconds an application to finish and returns a current state.

  • We make sure that we have a correct application state

We use ContainerLogUtils to find our container logs files from a minicluster.

  • We assert count of a log files

  • We expect some specified content from log file

  • We expect stderr files to be empty

Run Test

If using gradle:

./gradlew clean build

If using maven:

mvn clean package

If test is executed succesfully you’d be able to see YARN container logs within the minicluster logging directory:

$ find gs-yarn-testing-dist/target/|grep std
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stderr
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stdout
target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000001/Appmaster.stderr
$ grep Hello gs-yarn-testing-dist/target/yarn--1502101888/yarn--1502101888-logDir-nm-0_0/application_1392395851515_0001/container_1392395851515_0001_01_000002/Container.stdout
[2014-02-14 16:37:54.278] boot - 18453  INFO [main] --- HelloPojo: Hello from HelloPojo
[2014-02-14 16:37:54.278] boot - 18453  INFO [main] --- HelloPojo: About to list from hdfs root content
[2014-02-14 16:37:55.157] boot - 18453  INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}
[2014-02-14 16:37:55.157] boot - 18453  INFO [main] --- HelloPojo: FileStatus{path=hdfs://localhost:33626/app; isDirectory=true; modification_time=1392395854968; access_time=0; owner=jvalkealahti; group=supergroup; permission=rwxr-xr-x; isSymlink=false}

Summary

Congratulations! You have now created a simple yet effective JUnit test for building Spring YARN projects.

Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.

All guides are released with an ASLv2 license for the code, and an Attribution, NoDerivatives creative commons license for the writing.