Revamped metrics and monitoring of deployed applications on Kubernetes
Updated analytics using micrometer
Database migration support
Data Flow 2.0 introduces a new architecture for collecting and displaying application metrics for Streams. The Data Flow Metrics Collector introduced in Data Flow 1.4 has been removed.
This new architecture is based on using the Micrometer library in the deployed applications to send metrics to popular monitoring systems and then visualizing the metrics with Grafana. Each application in the Einstein release train of the application starters contains Micrometer libraries for Prometheus, InfluxDB and DataDog.
In this release we provide instructions on how to get started using Prometheus as the monitoring system when running Data Flow on Kubernetes. Two Grafana dashboards are also provided, one that shows a more application centric view and other that show a more stream based view. Here is a screenshot of the Grafana dashboard based on Prometheus showing the message rates for the applications in a stream.
Selectors for applications and streams allows you to navigate down to specific areas of interest.
In Data Flow 2.0, the use of Redis counters has been replaced with counters provided by the Micrometer library. The Einstein release train of application starters now provide a Counter Processor and a Counter Sink. The twitter analytics sample application has been updated to use the new counter implementation and a Grafana Dashboard is provided to show the equivalent view of what the Data Flow Analytics UI had previously shown.
The server uses OAuth 2.0 authentication to secure the REST endpoints. Documentation and sample code is now available that shows how to configure the
DataFlowTemplate for use with OAuth2.0.
Dependent libraries that had recent CVEs reported were updated.
Flyway migration code has been improved and tested across all supported databases using a suite of Docker based integration tests.
We postponed the feature of storing the detailed history of Task deployments for 2.1. Various documentation issues and additional testing is planned leading up to the 2.0 GA release. General themes for 2.1 will be a focus on documentation, getting started guides, videos and overall user experience. A new site for Data Flow is planned as well.
Also note that the 1.x line of Spring Cloud Data Flow will cease maintenance twelve months from the 2.0 GA announcement date. The 2.0 GA is tentatively planned for end of February 2019, so in February 2020, the 1.x line would reach end-of-general-support status.