On behalf of the Spring Integration and Spring Cloud Stream teams, I’m pleased to announce that the
spring-kafka (Spring for Apache Kafka) Release Candidate for version
1.0 is now available.
org.springframework.kafka:spring-kafka-test:1.0.0.RC1 are available in the Milestone repository.
First of all many thanks to everyone involved, to active community members, who provided feature requests and contributions.
Special thanks to Martin Dam, who spent a lot of time helping us with the
pause/resume algorithm to handle slow listeners.
There have been a number of reworks and housekeeping since the Second Milestone including:
ConsumerRebalanceListenercan be injected into the
KafkaProducercan be customized with
ProducerFactory, respectively; this is an alternative to configuring using the properties.
JsonDeserializers are provided, based on the Jackson library;
FilteringMessageListenerAdapterare provided to allow skipping records or dealing with duplicate deliveries;
ListenerContainerIdleEventis emitted, after a configurable time, when the
MessageListenerContainerhas moved to an idle state;
TopicPartitionInitialOffsetis provided to allow configuring the
MessageListenerContainerto be assigned to specific partitions in the topic(s) and, optionally, seek to the desired offset when starting;
@TopicPartitionattribute is enhanced with the
@PartitionOffsetproperty for initial offset configuration;
Improvements for consumer rebalance events.
The main feature that has been implemented for this Release Candidate, is pausing/resuming the
KafkaConsumer when the target listener is slow to process records.
If we don’t poll
KafkaConsumer within the
session.timeout.ms, Kafka presumes our consumer is defective and initiates a partition rebalance process, when using group management for partition assignment.
To stay alive, we have to poll consumer.
Consider the case where Kafka returned a large number of records on a poll and it takes longer than the session timeout to process them.
For this purpose, the
MessageListenerContainer can be configured with
queueDepth - and the
KafkaConsumer will be paused (polling continues but no records will be received) until the listener completes its work and the consumer is resumed.
One more time thanks to Martin Dam for his help with this feature and his testing.
See also the Reference Manual for more information.
Don’t miss the Release Candidate for Spring Integration Kafka 2.0 which is reworked to be based on this Spring for Apache Kafka foundation.
We expect to release the GA in about a week, so please give the candidate a try and report any issues as soon as possible.