A streaming data pipeline orchestrated as a series of microservice applications has always been the core value of Spring Cloud Data Flow’s design. In Data Flow 1.3 we have provided the ability to update sources, processors, and sinks independently without having to undeploy and redeploy the entire stream.
The stream update and rollback functionality is implemented by delegating the deployment process to a new Spring Cloud project called Skipper. Skipper is a lightweight Spring Boot application, purpose-built to fill this feature gap in Data Flow. Skipper defines a package format, much like
brew and can also deploy/undeploy applications to multiple cloud platforms: Local, Cloud Foundry, and Kubernetes. It uses the same Spring Cloud Deployer libraries that have been part of Data Flow since the beginning. Recent presentations at SpringOne 2017 introduces Skipper and the integration with Data Flow in more depth.
When deploying a Stream, Data Flow creates Skipper package describing the Stream and the applications that are part of the Stream definition. Skipper then deploys the applications to the desired platform. When requesting a stream update, only the application or applications that need to be changed are automatically redeployed. A simple strategy managed by a Spring Statemachine instance performs the update or rollback steps.
Data Flow includes new stream commands to make upgrade and rollback operations.
dataflow:>app register --name transform --type processor --uri maven://com.eg:transformer:0.0.1
dataflow:>stream create mystream --definition "jdbc | transform | mongodb"
dataflow:>app register --name transform --type processor --uri maven://com.eg:transformer:0.0.2
dataflow:>stream update mystream --properties “version.transform=0.0.2”
dataflow:>stream rollback mystream
In this series of commands, the stream is deployed using version 0.0.1 of the transformer. The
mongodb source and sink are already registered. The stream is then updated to use version 0.0.2 of the transformer. Only the
transform application is updated, with version 0.0.2 being deployed and version 0.0.1 being undeployed. The
mongodb applications are left as-is. The rollback command does the opposite, bringing the stream back to the state with version 0.0.1 of the transformer.
Note: To use Data Flow and Skipper, Data Flow’s feature toggle for Skipper must be enabled in both the Data Flow Server and shell.