Spring Team
Mark Pollack

Mark Pollack

Spring Cloud Data Flow lead

New York, NY

Mark Pollack is a software engineer with Pivotal and is the lead of the Spring Cloud Data Flow project. He has been a contributor to many Spring projects dating back to the Spring Framework in 2003 as well as founding the Spring.NET and Spring Data projects.
Blog Posts by Mark Pollack

Introducing java-cfenv: A new library for accessing Cloud Foundry Services

Introduction

The Spring Cloud Connectors library has been with us since the launch event of Cloud Foundry itself back in 2011. One of the main goals of the connector library and Cloud Foundry’s Java buildpack was to “reduce the initial investment when you want to get started with Cloud Foundry”. The connector library creates the Spring bean definitions required to connect to backing services, like databases, using information contained in the VCAP_SERVICES environment variable. The buildpack then replaces these bean definitions you had in your application with those created by the connector library through a feature called ‘auto-reconfiguration’. You may have seen it mentioned in the logs when you pushed an app to Cloud Foundry…

Read more...

Spring Cloud Data Flow 1.7.4 and Skipper 1.1.4 Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7.4 of Data Flow for Local and Cloud Foundry. The Kubernetes release is at version 1.7.3.

Follow the Getting Started guides for running on Local, Cloud Foundry, and Kubernetes.

Various bugs since the 1.7 GA release have been fixed. You can see the issues for 1.7.1, 1.7.2, 1.7.3, and 1.7.4 for more details.

We have also released 1.1.4 release of Spring Cloud Skipper. The getting started section in the reference guide is the best place to start if you want to use Skipper separately from Data Flow.

Read more...

Spring Cloud Data Flow and Skipper 2.0 M2 Released

The Spring Cloud Data Flow team is pleased to announce the release of 2.0 M2 of Data Flow. Follow the Getting Started guides for running on Local, Cloud Foundry, and Kubernetes.

Hand in hand is the 2.0 M2 release of Spring Cloud Skipper. The getting started section in the reference guide is the best place to start if you want to use Skipper separately from Data Flow.

Here are the highlights for Data Flow

  • Launch tasks against multiple platforms

  • Revamped metrics and monitoring of deployed applications

  • Additional security roles

  • Dashboard improvements

  • Database migration support

Read more...

Spring Cloud Data Flow and Skipper 2.0 M1 Released

The Spring Cloud Data Flow team is pleased to announce the release of 2.0 M1 Follow the Getting Started guides for running on Local, Cloud Foundry, and Kubernetes.

Hand in hand is the 2.0 M1 release of Spring Cloud Skipper which has been updated to depend upon Spring Boot 2.1. The getting started section in the reference guide is the best place to start kicking the tires.

Here are the highlights for Data Flow

  • Stream deployment always delegates to Skipper

  • Single server that runs on all supported platforms

  • Update to Boot 2.1

  • Standardize on OAuth2 and OpenID Connect for Security

  • Update internals to use JPA

  • UI improvements

  • Task/Job Execution and Performance improvements

Read more...

Project Update: Spring Cloud Data Flow for Apache Mesos and Apache YARN

Dear Spring community,

The Spring Cloud Data Flow team have been happy stewards of the Spring Cloud Deployer and Spring Cloud Data Flow implementations of Apache Mesos and Apache YARN.

We now feel that TrustedChoice.com will be a better home for Apache Mesos implementation of Spring Cloud Deployer and Spring Cloud Data Flow, and we are donating the projects to them to carry it forward.

The development will now be managed directly by the team (Adam J. Weigold, Phil Egelston, Justin Mathieu, and Cole Anderson) at TrustedChoice.com, as the Spring Cloud Data Flow team will no longer maintain it.

Read more...

Spring Cloud Data Flow 1.7 GA Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7.0. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes. Look for an updated Cloud Foundry Tile for SCDF release in the coming weeks.

Here are the highlights

  • Improved UI

  • Stream Application DSL

  • Audit trail

  • Concurrent Task Launch Limiting

  • Stream and Task validation

  • Force upgrade for Streams

  • Task Scheduling on Kubernetes

Improved UI

The UI has a completely new look. The navigation has moved from tabs to a left side navigation system. This gives increased screen real estate for creating streams with the Flo designer and even more screen real estate can be obtained by minimizing the left side navigation. There is a quick search feature that searches across all the different Data Flow categories. Additional colors and overall theme changes have been added to make the UI look more lively. Deeper in the core, the route management has been improved and we have increased our end to end testing coverage using BrowserStack/SauceLabs. The property whitelisting functionality has been refined to not display all application properties by default if the whitelist is empty. Check out the video for a UI walkthough.

Read more...

Spring Cloud Data Flow 1.7 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

The RC1 release builds on the core features introduced in 1.7 M1 with a few refinements.

Here are the highlights

Read more...

Spring Cloud Data Flow 1.7 M1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.7 M1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the highlights

  • Improved UI

  • Stream Application DSL

  • Audit trail

  • Concurrent Task Launch Limiting

  • Stream and Task validation

  • Force upgrade for Streams

Improved UI

The UI has a completely new look. The navigation has moved from tabs to a left side navigation system. This gives increased screen real estate for creating streams with the Flo designer and even more screen real estate can be obtained by minimizing the left side navigation. There is a quick search feature that searches across all the different Data Flow categories. Additional colors and overall theme changes have been added to make the UI look more lively. Deeper in the core, the route management has been improved and we have increased our end to end testing coverage using BrowserStack/SauceLabs.

Stream Create
Read more...

Spring Cloud Data Flow 1.6 GA Released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6.0. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Feature highlights for 1.6 GA

  • Task Scheduling on PCF

  • Dashboard improvments

  • Kubernetes support enhancements

  • App hosting tool

  • Composed Task Runner security

  • DSL and deployment property parsing refinements

  • Batch Database Schema and Optimization

Task Scheduling on PCF

We are happy to introduce the native integration of PCF Scheduler in the SCDF for Cloud Foundry implementation!

A typical workflow for batch data processing involves scheduling batch applications. For example, the scheduler system accepts a cron expression and launches the application whenever the expression matches the current time.

Data Flow provides the ability to schedule and unschedule a task definition. The schedule is based on a cron expression. Building upon the PCF Java Client the team has created a portable scheduler interface in the Spring Cloud Scheduler SPI project (Service Provider Interface) and an implementation for PCF, Spring Cloud Scheduler for Cloud Foundry. The Dashboard provides access to schedule and unschedule a task as shown in the screenshot below.

Create Schedule
List and Delete Schedules
Read more...

Spring Cloud Data Flow 1.6 RC1 released

The Spring Cloud Data Flow team is pleased to announce the release of 1.6 RC1. Follow the Getting Started guides for Local Server, Cloud Foundry, and Kubernetes.

Here are the release highlights:

PCF Scheduler

The Pivotal Cloud Foundry implementation of Scheduler improved on a few fronts to enhance the developer experience. Validation of the cron-expression and proactive measures to prevent the scheduler service from creating incorrect schedules is now part of this release.

Dashboard

The stream deployment history is available for review from the Dashboard. It is convenient to review the context-specific history of a stream from a central location; especially, when the CI/CD systems continually deploy new version application artifacts that belong to the stream.

Read more...