This guide walks you through the process of creating a basic batch-driven solution.
What you’ll build
You’ll build a service that imports data from a CSV spreadsheet, transforms it with custom code, and stores the final results in a database.
What you’ll need
-
About 15 minutes
-
A favorite text editor or IDE
-
JDK 1.8 or later
-
You can also import the code straight into your IDE:
How to complete this guide
Like most Spring Getting Started guides, you can start from scratch and complete each step, or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.
To start from scratch, move on to Build with Gradle.
To skip the basics, do the following:
-
Download and unzip the source repository for this guide, or clone it using Git:
git clone https://github.com/spring-guides/gs-batch-processing.git
-
cd into
gs-batch-processing/initial
-
Jump ahead to Create a business class.
When you’re finished, you can check your results against the code in gs-batch-processing/complete
.
Build with Gradle
Build with Gradle
First you set up a basic build script. You can use any build system you like when building apps with Spring, but the code you need to work with Gradle and Maven is included here. If you’re not familiar with either, refer to Building Java Projects with Gradle or Building Java Projects with Maven.
Create the directory structure
In a project directory of your choosing, create the following subdirectory structure; for example, with mkdir -p src/main/java/hello
on *nix systems:
└── src └── main └── java └── hello
Create a Gradle build file
Below is the initial Gradle build file.
build.gradle
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:2.0.5.RELEASE")
}
}
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
bootJar {
baseName = 'gs-batch-processing'
version = '0.1.0'
}
repositories {
mavenCentral()
}
sourceCompatibility = 1.8
targetCompatibility = 1.8
dependencies {
compile("org.springframework.boot:spring-boot-starter-batch")
compile("org.hsqldb:hsqldb")
testCompile("junit:junit")
}
The Spring Boot gradle plugin provides many convenient features:
-
It collects all the jars on the classpath and builds a single, runnable "über-jar", which makes it more convenient to execute and transport your service.
-
It searches for the
public static void main()
method to flag as a runnable class. -
It provides a built-in dependency resolver that sets the version number to match Spring Boot dependencies. You can override any version you wish, but it will default to Boot’s chosen set of versions.
Build with Maven
Build with Maven
First you set up a basic build script. You can use any build system you like when building apps with Spring, but the code you need to work with Maven is included here. If you’re not familiar with Maven, refer to Building Java Projects with Maven.
Create the directory structure
In a project directory of your choosing, create the following subdirectory structure; for example, with mkdir -p src/main/java/hello
on *nix systems:
└── src └── main └── java └── hello
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.springframework</groupId>
<artifactId>gs-batch-processing</artifactId>
<version>0.1.0</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
</parent>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>hsqldb</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
The Spring Boot Maven plugin provides many convenient features:
-
It collects all the jars on the classpath and builds a single, runnable "über-jar", which makes it more convenient to execute and transport your service.
-
It searches for the
public static void main()
method to flag as a runnable class. -
It provides a built-in dependency resolver that sets the version number to match Spring Boot dependencies. You can override any version you wish, but it will default to Boot’s chosen set of versions.
Build with your IDE
Build with your IDE
-
Read how to import this guide straight into Spring Tool Suite.
-
Read how to work with this guide in IntelliJ IDEA.
Business Data
Typically your customer or a business analyst supplies a spreadsheet. In this case, you make it up.
src/main/resources/sample-data.csv
Jill,Doe
Joe,Doe
Justin,Doe
Jane,Doe
John,Doe
This spreadsheet contains a first name and a last name on each row, separated by a comma. This is a fairly common pattern that Spring handles out-of-the-box, as you will see.
Next, you write a SQL script to create a table to store the data.
src/main/resources/schema-all.sql
DROP TABLE people IF EXISTS;
CREATE TABLE people (
person_id BIGINT IDENTITY NOT NULL PRIMARY KEY,
first_name VARCHAR(20),
last_name VARCHAR(20)
);
Spring Boot runs [email protected]@[email protected]@.sql automatically during startup. -all is the default for all platforms. |
Create a business class
Now that you see the format of data inputs and outputs, you write code to represent a row of data.
src/main/java/hello/Person.java
package hello;
public class Person {
private String lastName;
private String firstName;
public Person() {
}
public Person(String firstName, String lastName) {
this.firstName = firstName;
this.lastName = lastName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getFirstName() {
return firstName;
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
@Override
public String toString() {
return "firstName: " + firstName + ", lastName: " + lastName;
}
}
You can instantiate the Person
class either with first and last name through a constructor, or by setting the properties.
Create an intermediate processor
A common paradigm in batch processing is to ingest data, transform it, and then pipe it out somewhere else. Here you write a simple transformer that converts the names to uppercase.
src/main/java/hello/PersonItemProcessor.java
package hello;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.ItemProcessor;
public class PersonItemProcessor implements ItemProcessor<Person, Person> {
private static final Logger log = LoggerFactory.getLogger(PersonItemProcessor.class);
@Override
public Person process(final Person person) throws Exception {
final String firstName = person.getFirstName().toUpperCase();
final String lastName = person.getLastName().toUpperCase();
final Person transformedPerson = new Person(firstName, lastName);
log.info("Converting (" + person + ") into (" + transformedPerson + ")");
return transformedPerson;
}
}
PersonItemProcessor
implements Spring Batch’s ItemProcessor
interface. This makes it easy to wire the code into a batch job that you define further down in this guide. According to the interface, you receive an incoming Person
object, after which you transform it to an upper-cased Person
.
There is no requirement that the input and output types be the same. In fact, after one source of data is read, sometimes the application’s data flow needs a different data type. |
Put together a batch job
Now you put together the actual batch job. Spring Batch provides many utility classes that reduce the need to write custom code. Instead, you can focus on the business logic.
src/main/java/hello/BatchConfiguration.java
package hello;
import javax.sql.DataSource;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider;
import org.springframework.batch.item.database.JdbcBatchItemWriter;
import org.springframework.batch.item.database.builder.JdbcBatchItemWriterBuilder;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.jdbc.core.JdbcTemplate;
@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
@Autowired
public JobBuilderFactory jobBuilderFactory;
@Autowired
public StepBuilderFactory stepBuilderFactory;
// tag::readerwriterprocessor[]
@Bean
public FlatFileItemReader<Person> reader() {
return new FlatFileItemReaderBuilder<Person>()
.name("personItemReader")
.resource(new ClassPathResource("sample-data.csv"))
.delimited()
.names(new String[]{"firstName", "lastName"})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
setTargetType(Person.class);
}})
.build();
}
@Bean
public PersonItemProcessor processor() {
return new PersonItemProcessor();
}
@Bean
public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<Person>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)")
.dataSource(dataSource)
.build();
}
// end::readerwriterprocessor[]
// tag::jobstep[]
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
@Bean
public Step step1(JdbcBatchItemWriter<Person> writer) {
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer)
.build();
}
// end::jobstep[]
}
For starters, the @EnableBatchProcessing
annotation adds many critical beans that support jobs and saves you a lot of leg work. This example uses a memory-based database (provided by @EnableBatchProcessing
), meaning that when it’s done, the data is gone.
Break it down:
src/main/java/hello/BatchConfiguration.java
@Bean
public FlatFileItemReader<Person> reader() {
return new FlatFileItemReaderBuilder<Person>()
.name("personItemReader")
.resource(new ClassPathResource("sample-data.csv"))
.delimited()
.names(new String[]{"firstName", "lastName"})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
setTargetType(Person.class);
}})
.build();
}
@Bean
public PersonItemProcessor processor() {
return new PersonItemProcessor();
}
@Bean
public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<Person>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)")
.dataSource(dataSource)
.build();
}
. The first chunk of code defines the input, processor, and output. - reader()
creates an ItemReader
. It looks for a file called sample-data.csv
and parses each line item with enough information to turn it into a Person
. - processor()
creates an instance of our PersonItemProcessor
you defined earlier, meant to uppercase the data. - write(DataSource)
creates an ItemWriter
. This one is aimed at a JDBC destination and automatically gets a copy of the dataSource created by @EnableBatchProcessing
. It includes the SQL statement needed to insert a single Person
driven by Java bean properties.
The next chunk focuses on the actual job configuration.
src/main/java/hello/BatchConfiguration.java
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
@Bean
public Step step1(JdbcBatchItemWriter<Person> writer) {
return stepBuilderFactory.get("step1")
.<Person, Person> chunk(10)
.reader(reader())
.processor(processor())
.writer(writer)
.build();
}
. The first method defines the job and the second one defines a single step. Jobs are built from steps, where each step can involve a reader, a processor, and a writer.
In this job definition, you need an incrementer because jobs use a database to maintain execution state. You then list each step, of which this job has only one step. The job ends, and the Java API produces a perfectly configured job.
In the step definition, you define how much data to write at a time. In this case, it writes up to ten records at a time. Next, you configure the reader, processor, and writer using the injected bits from earlier.
chunk() is prefixed <Person,Person> because it’s a generic method. This represents the input and output types of each "chunk" of processing, and lines up with ItemReader<Person> and ItemWriter<Person> . |
src/main/java/hello/JobCompletionNotificationListener.java
package hello;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.listener.JobExecutionListenerSupport;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
@Component
public class JobCompletionNotificationListener extends JobExecutionListenerSupport {
private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);
private final JdbcTemplate jdbcTemplate;
@Autowired
public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
@Override
public void afterJob(JobExecution jobExecution) {
if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
log.info("!!! JOB FINISHED! Time to verify the results");
jdbcTemplate.query("SELECT first_name, last_name FROM people",
(rs, row) -> new Person(
rs.getString(1),
rs.getString(2))
).forEach(person -> log.info("Found <" + person + "> in the database."));
}
}
}
This code listens for when a job is BatchStatus.COMPLETED
, and then uses JdbcTemplate
to inspect the results.
Make the application executable
Although batch processing can be embedded in web apps and WAR files, the simpler approach demonstrated below creates a standalone application. You package everything in a single, executable JAR file, driven by a good old Java main()
method.
src/main/java/hello/Application.java
package hello;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class Application {
public static void main(String[] args) throws Exception {
SpringApplication.run(Application.class, args);
}
}
@SpringBootApplication
is a convenience annotation that adds all of the following:
-
@Configuration
tags the class as a source of bean definitions for the application context. -
@EnableAutoConfiguration
tells Spring Boot to start adding beans based on classpath settings, other beans, and various property settings. -
Normally you would add
@EnableWebMvc
for a Spring MVC app, but Spring Boot adds it automatically when it sees spring-webmvc on the classpath. This flags the application as a web application and activates key behaviors such as setting up aDispatcherServlet
. -
@ComponentScan
tells Spring to look for other components, configurations, and services in thehello
package, allowing it to find the controllers.
The main()
method uses Spring Boot’s SpringApplication.run()
method to launch an application. Did you notice that there wasn’t a single line of XML? No web.xml file either. This web application is 100% pure Java and you didn’t have to deal with configuring any plumbing or infrastructure.
For demonstration purposes, there is code to create a JdbcTemplate
, query the database, and print out the names of people the batch job inserts.
Build an executable JAR
You can run the application from the command line with Gradle or Maven. Or you can build a single executable JAR file that contains all the necessary dependencies, classes, and resources, and run that. This makes it easy to ship, version, and deploy the service as an application throughout the development lifecycle, across different environments, and so forth.
If you are using Gradle, you can run the application using ./gradlew bootRun
. Or you can build the JAR file using ./gradlew build
. Then you can run the JAR file:
If you are using Maven, you can run the application using ./mvnw spring-boot:run
. Or you can build the JAR file with ./mvnw clean package
. Then you can run the JAR file:
The procedure above will create a runnable JAR. You can also opt to build a classic WAR file instead. |
The job prints out a line for each person that gets transformed. After the job runs, you can also see the output from querying the database.
Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE) Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE) Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE) Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE) Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE) Found <firstName: JILL, lastName: DOE> in the database. Found <firstName: JOE, lastName: DOE> in the database. Found <firstName: JUSTIN, lastName: DOE> in the database. Found <firstName: JANE, lastName: DOE> in the database. Found <firstName: JOHN, lastName: DOE> in the database.
Summary
Congratulations! You built a batch job that ingested data from a spreadsheet, processed it, and wrote it to a database.
See also
The following guides may also be helpful:
Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.
All guides are released with an ASLv2 license for the code, and an Attribution, NoDerivatives creative commons license for the writing. |