Spring Team
Mark Pollack

Mark Pollack

Spring Cloud Data Flow lead

New York, NY

Mark Pollack is a software engineer with Pivotal and is the lead of the Spring Cloud Data Flow project. He has been a contributor to many Spring projects dating back to the Spring Framework in 2003 as well as founding the Spring.NET and Spring Data projects.
Blog Posts by Mark Pollack

Spring Cloud Data Flow 1.7 M1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7 M1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights

  • Improved UI

  • Stream Application DSL

  • Audit trail

  • Concurrent Task Launch Limiting

  • Stream and Task validation

  • Force upgrade for Streams

Improved UI

The UI has a completely new look. The navigation has moved from tabs to a left side navigation system. This gives increased screen real estate for creating streams with the Flo designer and even more screen real estate can be obtained by minimizing the left side navigation. There is a quick search feature that searches across all the different Data Flow categories. Additional colors and overall theme changes have been added to make the UI look more lively. Deeper in the core, the route management has been improved and we have increased our end to end testing coverage using BrowserStack/SauceLabs.

Stream Create
Read more...

Spring Cloud Data Flow 1.6 GA Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6.0. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Feature highlights for 1.6 GA

  • Task Scheduling on PCF

  • Dashboard improvments

  • Kubernetes support enhancements

  • App hosting tool

  • Composed Task Runner security

  • DSL and deployment property parsing refinements

  • Batch Database Schema and Optimization

Task Scheduling on PCF

We are happy to introduce the native integration of PCF Scheduler in the SCDF for Cloud Foundry implementation!

A typical workflow for batch data processing involves scheduling batch applications. For example, the scheduler system accepts a cron expression and launches the application whenever the expression matches the current time.

Data Flow provides the ability to schedule and unschedule a task definition. The schedule is based on a cron expression. Building upon the PCF Java Client the team has created a portable scheduler interface in the Spring Cloud Scheduler SPI project (Service Provider Interface) and an implementation for PCF, Spring Cloud Scheduler for Cloud Foundry. The Dashboard provides access to schedule and unschedule a task as shown in the screenshot below.

Create Schedule
List and Delete Schedules
Read more...

Spring Cloud Data Flow 1.6 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the release highlights:

PCF Scheduler

The Pivotal Cloud Foundry implementation of Scheduler improved on a few fronts to enhance the developer experience. Validation of the cron-expression and proactive measures to prevent the scheduler service from creating incorrect schedules is now part of this release.

Dashboard

The stream deployment history is available for review from the Dashboard. It is convenient to review the context-specific history of a stream from a central location; especially, when the CI/CD systems continually deploy new version application artifacts that belong to the stream.

Read more...

Spring Cloud Data Flow 1.6 M2 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6 M2. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights

  • Task Scheduling on PCF

  • Angluar 6 update

  • App Hosting Tool

Task Scheduling on PCF

We are happy to introduce the native integration of PCF Scheduler in the SCDF for Cloud Foundry implementation!

A typical workflow for batch data processing involves scheduling batch applications. For example, the scheduler system accepts a cron expression and launches the application whenever the expression matches the current time.

Data Flow provides the ability to schedule and unschedule a task definition. The schedule is based on a cron expression. Building upon the PCF Java Client the team has created a portable scheduler interface in the Spring Cloud Scheduler SPI project (Service Provider Interface) and an implementation for PCF, Spring Cloud Scheduler for Cloud Foundry. The Dashboard provides access to schedule and unschedule a task as shown in the screenshot below.

Create Schedule
List and Delete Schedules
Read more...

Spring Cloud Data Flow 1.6 M1 and 1.5.2 released

The Spring Cloud Data Flow team is pleased to announce the 1.6 M1 release and 1.5.2 release.

For 1.6 M1, follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

For 1.5.2, follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Areas of improvement for 1.6 M1:

  • DSL and deployment property parsing

  • Task Execution status

  • Composed Task Runner security

  • Dashboard

  • Kubernetes deployments

DSL and deployment property parsing

Launching Tasks with custom arguments is a great approach to influence the Task application with differing behaviors at runtime. Imagine influencing the batch-job (running as a Task) that accepts timezone as an argument to perform timezone specific data processing. In this release, we have adapted the parsing logic to include key-value pairs as values. Thanks to the community for reporting, giving us feedback, and sharing of their use-cases.

While reviewing the parsing rules for in-line vs. property files based properties for stream and task definitions, the community has found a difference in behavior, and that we have documented it for general guidance.

Read more...

Spring Cloud Data Flow 1.5.1 Released

The Spring Cloud Data Flow team is pleased to announce the 1.5.1 GA release. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

This is a bug fix release. The server improves the handling of special characters in stream definitions and passing of comma delimited strings in the Task launch argument list. It should be used with Skipper 1.0.5.RELEASE. The UI has been improved to support stream update functionality.

As always, we welcome feedback and contributions, so please reach out to us on Stackoverflow or GitHub or via Gitter.

Read more...

Spring Cloud Skipper 1.0.5 released

On behalf of the team, I am pleased to announce the release of Spring Cloud Skipper 1.0.5 GA

Skipper is a lightweight tool that allows you to discover Spring Boot applications and manage their lifecycle on multiple Cloud Platforms. You can use Skipper standalone or integrate it with Continuous Integration pipelines to help implement the practice of Continuous Deployment.

The getting started section in the reference guide is the best place to start kicking the tires.

This is primarily a bug fix release. Significant changes since the 1.0 GA release are:

Read more...

Spring Cloud Data Flow 1.5.0 Released

The Spring Cloud Data Flow team is pleased to announce the 1.5.0 GA release. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights:

  • UI Improvements

  • Spring Boot, Spring Cloud Stream 2.0, and Spring Cloud Task 2.0 Support

  • Updated Application Starters

  • Metrics Improvements

  • Nested splits for Composed Tasks

  • Kubernetes Improvements

  • Updated File Ingest sample

UI Improvements

We have continued to improve the UI/UX of the Dashboard. We hope that you will immediately notice an overall lighter weight design. The Tasks tab has been rewritten to match the UX styling of the other tabs. A new paginator component has been added to all the list pages. Switching from a list of 20, 30, 50, or 100 items per page is possible. This further simplifies the bulk operation workflows.

The updated Stream Builder tab makes is easy to deploy Stream Definitions and update deployed streams. You can edit application and deployment properties as well as change the version of individual applications in the stream and re-deploy. Data Flow’s integration with Skipper handles the upgrade process, allowing for easy rollback in case the upgrade doesn’t go as planned. The Stream Builder tab also includes many optimizations, including better form validation and eager error reporting. Try it out!

Stream Builder Tab

There has also been a significant amount of refactoring to optimize the code base and prepare for future extensions and feature additions. End-to-end testing with Selenium and SauceLabs has also been added.

Read more...

Spring Cloud Data Flow 1.5 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.5.0 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights:

General Improvements

  • Switch to Hikari connection pool and restructure code to use fewer connections.

  • Several bug fixes in underling deployer libraries.

Dashboard

  • Editing a created/deployed stream is now possible from the Stream Builder. The application and deployment properties can be edited and re-deployed. The App version can be switched, too.

  • A new paginator component is added to all the list page. Switching from a list of 20, 30, 50, or 100 items per page is possible. This further simplifies the bulk operation workflows.

  • Introduction of end-to-end testing via Selenium and SauceLabs.

Read more...

Spring Cloud Data Flow 1.5 M1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.5.0 M1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights:

  • UI Improvements

  • Spring Boot & Spring Cloud Stream 2.0 Support

  • Nested splits for Composed Tasks

  • Metrics Collector 2.0 M1

  • Stream Application Starters Darwin M1 release train

  • Support for deploying to multiple Kubernetes clusters

UI Improvements

We have continued to improve the UI/UX of the Dashboard. You will immediately notice an overall lighter weight design. The Tasks tab has been rewritten to match the UX styling of the other tabs. The stream-builder view includes many optimizations ranging from better form validation and eager error reporting. Try it out!

There has also been a significant amount of refactoring to optimize the codebase and prepare for future extensions and feature additions.

Read more...