Get ahead
VMware offers training and certification to turbo-charge your progress.
Learn moreOn behalf of the team and all contributors, I am pleased to announce that Spring Batch 6.1.0-RC1 and 6.0.6 are available now from Maven Central!
These releases bring a number of new features, enhancements and bug fixes.
For the complete list of changes, please check the release notes: 6.1.0-RC1 and 6.0.6.
This blog post walks you through the major changes in 6.1.0-RC1:
@JobParameter annotation for injecting job parametersPageableItemReaderJsonLineMapper agnostic to the JSON libraryJsonLineAggregatorRemoteChunkingManagerBuilderDefaultChunkProcessorJdbcClientSpring dependencies are upgraded to the following versions:
@JobParameter annotation for injecting job parametersInjecting a job parameter into a job or step scoped bean used to require a SpEL expression, which is only evaluated at runtime and is easy to get wrong. The new @JobParameter annotation is a shorter alternative that defaults the name of the job parameter to the name of the annotated field or parameter:
@Bean
@StepScope
public FlatFileItemReader<Foo> fooReader(@JobParameter String inputFile) {
return new FlatFileItemReaderBuilder<Foo>()
.resource(new FileSystemResource(inputFile))
// other properties omitted
.build();
}
This replaces @Value("#{jobParameters['inputFile']}"). The name can also be specified explicitly, for example @JobParameter("input.file.name"). Like the expression it replaces, it is meant for beans defined in the job or step scope. When the name is omitted on a method or constructor parameter, the code must be compiled with the -parameters flag, which is the default with the Spring Boot Maven and Gradle plugins. See issue #5563 for more details.
PageableItemReaderPageableItemReader is a type-safe alternative to RepositoryItemReader. Instead of invoking a repository method reflectively by name, it takes a function from a Spring Data Pageable to a Slice, which is checked at compile time:
PageableItemReader<Customer> reader = new PageableItemReaderBuilder<Customer>()
.query(customerRepository::findAll)
.sort(Sort.by(Sort.Direction.ASC, "id"))
.pageSize(100)
.name("customerReader")
.build();
The sort can also be expressed with Spring Data's type-safe property references, for example Sort.by(Sort.Direction.ASC, Customer::getId), which eliminates string-based property names.
JsonLineMapper agnostic to the JSON libraryJsonLineMapper could only map each line of a JSON Lines file to a Map<String, Object>, and only with Jackson. It is now generic and delegates the conversion of a line to a new JsonObjectUnmarshaller, which makes it independent of the JSON library and able to map lines to any type, such as a domain class or a record. Implementations based on Jackson and Gson are provided:
record Trade(String isin, int quantity, double price, String customer) {
}
FlatFileItemReader<Trade> reader = new FlatFileItemReaderBuilder<Trade>()
.name("tradeReader")
.resource(resource)
.lineMapper(new JsonLineMapper<>(new JacksonJsonObjectUnmarshaller<>(Trade.class)))
// or .lineMapper(new JsonLineMapper<>(new GsonJsonObjectUnmarshaller<>(Trade.class)))
.build();
The no-argument constructor and the constructor that takes a JsonMapper are deprecated in favor of JsonLineMapper(JsonObjectUnmarshaller), and are scheduled for removal in version 7.0. This is a source and binary breaking change for code that uses the raw JsonLineMapper type and calls mapLine directly on it. Code that only passes the mapper to a reader needs no changes beyond a recompile. See the reference documentation for more details.
JsonLineAggregatorJsonLineAggregator is the writer-side counterpart of JsonLineMapper: a LineAggregator that makes it possible to write items to a JSON Lines file with a FlatFileItemWriter. Like the mapper, it is agnostic to the JSON library, as it delegates the conversion of an item to a JsonObjectMarshaller, so either the Jackson or the Gson implementation can be used:
FlatFileItemWriter<Trade> writer = new FlatFileItemWriterBuilder<Trade>()
.name("tradeWriter")
.resource(new FileSystemResource("trades.jsonl"))
.lineAggregator(new JsonLineAggregator<>(new JacksonJsonObjectMarshaller<>()))
.build();
RemoteChunkingManagerBuilderRemoteChunkingManagerBuilder is the counterpart of RemoteChunkingWorkerBuilder for configuring the manager step of a remote chunking setup. It only holds the settings that are specific to remote chunking (channels, messaging template, maximum number of wait timeouts and throttle limit), and chunk(int) returns a standard ChunkOrientedStepBuilder for everything else. The resulting step is a ChunkOrientedStep, and the builder does not expose any deprecated API:
@Bean
public Step managerStep(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
return new RemoteChunkingManagerBuilder("managerStep", jobRepository)
.outputChannel(requests())
.inputChannel(replies())
.chunk(100)
.reader(itemReader())
.transactionManager(transactionManager)
.build();
}
RemoteChunkingManagerStepBuilder and RemoteChunkingManagerStepBuilderFactory are deprecated in favor of this builder, and are scheduled for removal in version 7.0. Note that the transaction manager defaults to a resourceless one, whereas the factory used the transaction manager of the application context, so it should be set explicitly if the item reader of the manager step needs a transactional resource.
DefaultChunkProcessorSimpleChunkProcessor has been deprecated since 6.0 as part of the legacy chunk-oriented processing engine, but it was still the way to process chunks outside of a step, for example in a remote chunking worker or with a ChunkTaskExecutorItemWriter. DefaultChunkProcessor is its replacement for those use cases. It is a minimal ChunkProcessor that applies an optional ItemProcessor to each item of a chunk and writes the result with an ItemWriter, for example new DefaultChunkProcessor<>(itemProcessor, itemWriter). Unlike SimpleChunkProcessor, it does not support listeners and does not record metrics. RemoteChunkingWorkerBuilder now uses it internally.
JdbcClientThe JDBC implementations of the job repository DAOs (JdbcJobInstanceDao, JdbcJobExecutionDao, JdbcStepExecutionDao and JdbcExecutionContextDao) now use Spring Framework's JdbcClient instead of JdbcOperations to access the batch metadata tables. This gives them named parameters and a single fluent API for queries, updates and batch updates. The client is a required collaborator, so it is now passed to the constructor:
JdbcJobInstanceDao jobInstanceDao = new JdbcJobInstanceDao(JdbcClient.create(dataSource));
Applications that configure the job repository through @EnableBatchProcessing, @EnableJdbcJobRepository, JdbcDefaultBatchConfiguration, JdbcJobRepositoryFactoryBean or JobExplorerFactoryBean are not affected, as these build the DAOs themselves. This is however a source and binary breaking change for code that instantiates these DAOs directly, since their no-argument constructors have been removed. Accordingly, AbstractJdbcBatchMetadataDao#setJdbcTemplate(JdbcOperations) and #getJdbcTemplate() are deprecated in favor of the new constructor and of #getJdbcClient(). See issue #4804 for more details.
This release makes several parts of the job repository and of partitioned steps faster and lighter on memory. Here are the biggest gains.
Restarting a job used to read the full history of its job instance, loading every past execution with all its steps. So the more often a job was restarted, the more SQL statements each restart needed: on 6.0.4, the 12th restart of the same failed job issued 130 statements, 8 more for each past execution. Now only the last execution is checked, so the cost of a restart no longer depends on the history. This is also available in 6.0.6.
Finding the executions of a job instance is faster too. JdbcJobExecutionDao#findJobExecutions used to run 3 extra queries for each execution (1 + 3N queries for N executions). It now needs 2 queries, whatever N is. For a job instance with 10 executions, that is 2 queries instead of 31, which is about 15 times fewer. getLastJobExecution goes from 4 queries to 2, which is half as many.
MessageChannelPartitionHandler regularly asks the job repository if its workers are done. Each time, it used to load the step execution of every worker, and each of them came with the whole job execution and all its step executions. With many partitions, this adds up quickly: in one reported case, a step with 837 partitions kept 837 × 837 = 700,569 StepExecution objects in memory, and the node polling the workers used 15 GB of memory, 10 times more than another node doing the same amount of work (1.5 GB).
The handler now only counts the running workers, with the new JobRepository#countRunningStepExecutions(Collection<Long>), and loads the job execution once, when all workers are done. In the same case, that is 837 objects instead of 700,569, which is 837 times fewer, and it removes the extra memory of the polling node: about 10 times less memory. The new methods are default methods, so existing JobRepository implementations still work, and the handler falls back to the previous behavior if counting is not supported.
A few more queries were removed from the JDBC DAOs. Each one saves one query every time it applies:
JOB_EXECUTION table.SELECT COUNT(*) first to check that the row exists, since the update itself reports a missing row.JdbcJobExecutionDao#synchronizeStatus(StepExecution) saves one query.The last two are also available in 6.0.6.
I would like to thank all contributors who had a role in this release! As we get closer to the 6.1 GA release, we look forward to your feedback on Github Issues, Github Discussions and X.