We are pleased to announce the first release milestone of Spring Shell 2.x!

Two years in the making, Spring Shell 2 is a complete rewrite of Spring Shell, leveraging newer components (such as JLine 3) and applying better modularization. Spring Shell 2 is also built with Spring Boot in mind, making use of auto configuration and other boot features.

The internal architecture now uses a pluggable model to discover what methods to turn into commands, how to parse user input into argument values and how to handle return values. This is very similar to the approach taken by Spring MVC for example and allows extensions of the framework in ways that were not previously possible. Users of Spring Shell would typically not care though, only dealing with the new "standard" command API…

On behalf of the Data Flow team, I am pleased to introduce you to the first of a hopefully long series of posts highlighting features of Data Flow and related projects.

We’ll start gently with a short video discussing the Data Flow shell and some of its features (some of them coming directly from the Spring Shell project) by yours truly.

Topics include:

• TAB completion

• Keyboard shortcuts

• Quotes handling in Spring Shell

• Quotes handling in Data Flow DSL

Links

The reference documentation section mentioned in the video can be found here, while reference on the general Data Flow DSL is here…

The 1.1.1.RELEASE maintenance release of Spring Cloud Data Flow for Cloud Foundry is now available and includes a variety of stability and performance improvements. Following are the highlights:

• Stability Improvements The instability problems when using SCDF’s 1.1.0.RELEASE on PCF manifested to the users in the following forms.

  • stream list operation would hang and eventually time out with “IllegalStateException: 503 Bad Gateway” or “IllegalStateException: Timeout on block read” exceptions. This is addressed in the cf-java-client library.

  • With SCDF server sitting cold/idle for a period of time and with OAUTH token refresh routine happening in the background, any new operation to the server would never complete. The OAUTH refresh token routine had a bug in successfully negotiating for new tokens. This is addressed in the cf-java-client library.

  …

We are pleased to announce the general availability of Spring Cloud Data Flow for Cloud Foundry version 1.1.0.RELEASE.

In addition to all the bells and whistles of Spring Cloud Data Flow 1.1.2 that this release builds upon, the Cloud Foundry specifics include:

• Memory Optimizations: After much troubleshooting, we are pleased to deliver a memory efficient SCDF server in this release. With the changes now, regardless of how many stream/task deployments and concurrency, we have observed the memory usage of the server stay within the 200MB range as opposed to 1G or over to an eventual crash in previous 1.0.0.RELEASE. If you experienced spurious crashes previously, we highly recommend upgrading to 1.1.0.RELEASE right away.

• Maven Cache and Disk Optimizations: We have improved the way Maven caching is applied and used. There is a new LRU (Least Recently Used) design on top of the maven cache to allow us to be more proactive about the cache size as opposed to letting it grow indefinitely, which if not governed, leads to out of disk space errors for the application container where the server is running.

  …

Spring Cloud Data Flow for Cloud Foundry gets two new releases today, detailed below:

1.0.1.GA is a maintenance release that

• Builds upon 2.0.1.RELEASE of Cloud Foundry’s Java Client and 3.0.2.RELEASE of Reactor

• Builds upon 1.0.1.RELEASE of Spring Cloud Data Flow core

• Gets general clarifications and improvements in the reference documentation

• Fixes a bug pertaining to multiple ORG + Space combination causing task launch failures

• Fixes a bug where environment variables were not propagated correctly when launching a task

1.1.0.M1 is the first milestone in the new line of development…

We are pleased to announce the general availability of Spring Cloud Data Flow for Cloud Foundry version 1.0.0.RELEASE.

Spring Cloud Data Flow for Cloud Foundry provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices on Cloud Foundry runtime.

As the successor to Spring XD, this project benefits from a much more decoupled architecture, leveraging the Spring Cloud Deployer for Cloud Foundry library, which also goes GA today. More details about Spring Cloud Data Flow’s architecture and its ecosystem can be found in this blog.

• Stream and Batch/Task Processing are the primary functionalities in Spring Cloud Data Flow and they map to Cloud Foundry Diego’s LRPs and Tasks1 respectively.

• Includes developer toolkits to build streaming and batch/task pipelines using the DSL, Shell, REST-APIs, Dashboard, Flo Designer, or any combination of those.

• Facilitates test-driven-development at individual data pipeline components along with test fixtures to develop and test "data-centric" apps in isolation.

• Leverages Cloud Foundry’s runtime capabilities such as security, metrics, operational monitoring, scaling, and reliable execution of streaming and batch/task pipelines.

  …

We are pleased to announce the 1.0.0.RC1 release of Spring Cloud Data Flow for Cloud Foundry.

As we near completion towards a GA release in the upcoming days, this milestone brings the following improvements:

• Builds upon the 1.0.0.RC1 release of Spring Cloud Deployer Cloud Foundry, which itself builds upon Project Reactor 3.0.0.RELEASE and of CF-Java-Client 2.0.0.RELEASE

• Adds the ability to orchestrate short-lived tasks including Spring Batch Jobs in Cloud Foundry, which can be enabled as an experimental feature toggle

• Adds support for command line arguments as a separate set of properties to be passed to a Task when it’s launched

• Adds support to separate stream and task specific service bindings. This allows pinning stream and tasks specific services to stream and task applications respectively

  …

We are pleased to announce the release of Spring Cloud Data Flow’s Cloud Foundry 1.0.0.M4.

In this milestone release, we have few improvements to the APIs and the general stability of the overall design constructs.

• Builds upon 1.0.0.RELEASE of Spring Cloud Data Flow Core (shell, UI, REST-APIs, etc.)

• Streamlines application vs. deployment property semantics

• Adds a migration guide and other documentation fragments to the reference documentation

• Adds a project site

Journey Ahead

• We are looking forward to Reactor and Cloud Foundry Java Client’s RC and GA releases and by this month, we shall release 1.0.0.RELEASE of Spring Cloud Data Flow for Cloud Foundry.

• We have exciting work-in-progress to orchestrate short-lived microservices through TaskLauncher constructs in the Spring Cloud Deployer for Cloud Foundry project. We are hopeful to deliver the BETA release of this functionality alongside the upcoming Pivotal Cloud Foundry’s 1.7.x release.

…

I am pleased to announce the 1.0.0.M3 release of Spring Cloud Data Flow for Cloud Foundry, a team effort that encompasses many new features under the hood:

This third milestone builds upon the recent RC1 release of Spring Cloud Data Flow and also adds Cloud Foundry specific refinements. Some highlights include:

• Alignment with Cloud Foundry’s global environment properties such as ORG, API, and URL as opposed to previous naming variants

• Adds security integration to authenticate against OAUTH backend server that’s compatible with Cloud Foundry

• Publishes security configurations for easier overrides when connecting to OAUTH backend server that’s compatible with Cloud Foundry

• Preliminary iterations on performance testing on Cloud Foundry with and without taps’s in real-time streaming pipeline

• Builds upon feature-toggle support from Spring Cloud Data Flow’s core project to conditionally refine what features get included in each release

  …

On behalf of the Spring Cloud Data Flow team, I am pleased to announce the 1.0.0.M2 release of Spring Cloud Data Flow for Cloud Foundry.

Spring Cloud Data Flow for Cloud Foundry allows one to use all the goodness of Spring Cloud Data Flow (like the Shell, UI and Flo) while targeting Cloud Foundry as a backend. Stream components are deployed as individual apps in Cloud Foundry, leveraging the power of the platform to handle scaling and health monitoring.

This second milestone

• builds upon the 1.0.0.M1 release of Cloud Foundry implementation of spring-cloud-deployer
• builds upon the 1.0.0.M3 release of Spring Cloud Data Flow
• builds upon the 2.0.0.M4 release of the Cloud Foundry java client, leveraging its reactive programming style
…