Spring Team
Mark Pollack

Mark Pollack

Spring Cloud Data Flow lead

New York, NY

Mark Pollack is a software engineer with Pivotal and is the lead of the Spring Cloud Data Flow project. He has been a contributor to many Spring projects dating back to the Spring Framework in 2003 as well as founding the Spring.NET and Spring Data projects.
Blog Posts by Mark Pollack

Project Update: Spring Cloud Data Flow for Apache Mesos and Apache YARN

Dear Spring community,

The Spring Cloud Data Flow team have been happy stewards of the Spring Cloud Deployer and Spring Cloud Data Flow implementations of Apache Mesos and Apache YARN.

We now feel that TrustedChoice.com will be a better home for Apache Mesos implementation of Spring Cloud Deployer and Spring Cloud Data Flow, and we are donating the projects to them to carry it forward.

The development will now be managed directly by the team (Adam J. Weigold, Phil Egelston, Justin Mathieu, and Cole Anderson) at TrustedChoice.com, as the Spring Cloud Data Flow team will no longer maintain it.

Read more...

Spring Cloud Data Flow 1.7 GA Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7.0. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes. Look for an updated Cloud Foundry Tile for SCDF release in the coming weeks.

Here are the highlights

  • Improved UI

  • Stream Application DSL

  • Audit trail

  • Concurrent Task Launch Limiting

  • Stream and Task validation

  • Force upgrade for Streams

  • Task Scheduling on Kubernetes

Improved UI

The UI has a completely new look. The navigation has moved from tabs to a left side navigation system. This gives increased screen real estate for creating streams with the Flo designer and even more screen real estate can be obtained by minimizing the left side navigation. There is a quick search feature that searches across all the different Data Flow categories. Additional colors and overall theme changes have been added to make the UI look more lively. Deeper in the core, the route management has been improved and we have increased our end to end testing coverage using BrowserStack/SauceLabs. The property whitelisting functionality has been refined to not display all application properties by default if the whitelist is empty. Check out the video for a UI walkthough.

Read more...

Spring Cloud Data Flow 1.7 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

The RC1 release builds on the core features introduced in 1.7 M1 with a few refinements.

Here are the highlights

Read more...

Spring Cloud Data Flow 1.7 M1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7 M1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights

  • Improved UI

  • Stream Application DSL

  • Audit trail

  • Concurrent Task Launch Limiting

  • Stream and Task validation

  • Force upgrade for Streams

Improved UI

The UI has a completely new look. The navigation has moved from tabs to a left side navigation system. This gives increased screen real estate for creating streams with the Flo designer and even more screen real estate can be obtained by minimizing the left side navigation. There is a quick search feature that searches across all the different Data Flow categories. Additional colors and overall theme changes have been added to make the UI look more lively. Deeper in the core, the route management has been improved and we have increased our end to end testing coverage using BrowserStack/SauceLabs.

Stream Create
Read more...

Spring Cloud Data Flow 1.6 GA Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6.0. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Feature highlights for 1.6 GA

  • Task Scheduling on PCF

  • Dashboard improvments

  • Kubernetes support enhancements

  • App hosting tool

  • Composed Task Runner security

  • DSL and deployment property parsing refinements

  • Batch Database Schema and Optimization

Task Scheduling on PCF

We are happy to introduce the native integration of PCF Scheduler in the SCDF for Cloud Foundry implementation!

A typical workflow for batch data processing involves scheduling batch applications. For example, the scheduler system accepts a cron expression and launches the application whenever the expression matches the current time.

Data Flow provides the ability to schedule and unschedule a task definition. The schedule is based on a cron expression. Building upon the PCF Java Client the team has created a portable scheduler interface in the Spring Cloud Scheduler SPI project (Service Provider Interface) and an implementation for PCF, Spring Cloud Scheduler for Cloud Foundry. The Dashboard provides access to schedule and unschedule a task as shown in the screenshot below.

Create Schedule
List and Delete Schedules
Read more...

Spring Cloud Data Flow 1.6 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the release highlights:

PCF Scheduler

The Pivotal Cloud Foundry implementation of Scheduler improved on a few fronts to enhance the developer experience. Validation of the cron-expression and proactive measures to prevent the scheduler service from creating incorrect schedules is now part of this release.

Dashboard

The stream deployment history is available for review from the Dashboard. It is convenient to review the context-specific history of a stream from a central location; especially, when the CI/CD systems continually deploy new version application artifacts that belong to the stream.

Read more...

Spring Cloud Data Flow 1.6 M2 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6 M2. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights

  • Task Scheduling on PCF

  • Angluar 6 update

  • App Hosting Tool

Task Scheduling on PCF

We are happy to introduce the native integration of PCF Scheduler in the SCDF for Cloud Foundry implementation!

A typical workflow for batch data processing involves scheduling batch applications. For example, the scheduler system accepts a cron expression and launches the application whenever the expression matches the current time.

Data Flow provides the ability to schedule and unschedule a task definition. The schedule is based on a cron expression. Building upon the PCF Java Client the team has created a portable scheduler interface in the Spring Cloud Scheduler SPI project (Service Provider Interface) and an implementation for PCF, Spring Cloud Scheduler for Cloud Foundry. The Dashboard provides access to schedule and unschedule a task as shown in the screenshot below.

Create Schedule
List and Delete Schedules
Read more...

Spring Cloud Data Flow 1.6 M1 and 1.5.2 released

The Spring Cloud Data Flow team is pleased to announce the 1.6 M1 release and 1.5.2 release.

For 1.6 M1, follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

For 1.5.2, follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Areas of improvement for 1.6 M1:

  • DSL and deployment property parsing

  • Task Execution status

  • Composed Task Runner security

  • Dashboard

  • Kubernetes deployments

DSL and deployment property parsing

Launching Tasks with custom arguments is a great approach to influence the Task application with differing behaviors at runtime. Imagine influencing the batch-job (running as a Task) that accepts timezone as an argument to perform timezone specific data processing. In this release, we have adapted the parsing logic to include key-value pairs as values. Thanks to the community for reporting, giving us feedback, and sharing of their use-cases.

While reviewing the parsing rules for in-line vs. property files based properties for stream and task definitions, the community has found a difference in behavior, and that we have documented it for general guidance.

Read more...

Spring Cloud Data Flow 1.5.1 Released

The Spring Cloud Data Flow team is pleased to announce the 1.5.1 GA release. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

This is a bug fix release. The server improves the handling of special characters in stream definitions and passing of comma delimited strings in the Task launch argument list. It should be used with Skipper 1.0.5.RELEASE. The UI has been improved to support stream update functionality.

As always, we welcome feedback and contributions, so please reach out to us on Stackoverflow or GitHub or via Gitter.

Read more...

Spring Cloud Skipper 1.0.5 released

On behalf of the team, I am pleased to announce the release of Spring Cloud Skipper 1.0.5 GA

Skipper is a lightweight tool that allows you to discover Spring Boot applications and manage their lifecycle on multiple Cloud Platforms. You can use Skipper standalone or integrate it with Continuous Integration pipelines to help implement the practice of Continuous Deployment.

The getting started section in the reference guide is the best place to start kicking the tires.

This is primarily a bug fix release. Significant changes since the 1.0 GA release are:

Read more...