Spring Team
Eric Bottard

Eric Bottard

Spring Cloud Stream, Spring Cloud Dataflow & Spring XD

Paris, France

Blog Posts by Eric Bottard

Spring Cloud Data Flow for Cloud Foundry 1.1.1 Maintenance release available

The 1.1.1.RELEASE maintenance release of Spring Cloud Data Flow for Cloud Foundry is now available and includes a variety of stability and performance improvements. Following are the highlights:

  • Stability Improvements The instability problems when using SCDF’s 1.1.0.RELEASE on PCF manifested to the users in the following forms.

    • stream list operation would hang and eventually time out with “IllegalStateException: 503 Bad Gateway” or “IllegalStateException: Timeout on block read” exceptions. This is addressed in the cf-java-client library.

    • With SCDF server sitting cold/idle for a period of time and with OAUTH token refresh routine happening in the background, any new operation to the server would never complete. The OAUTH refresh token routine had a bug in successfully negotiating for new tokens. This is addressed in the cf-java-client library.

  • Error Handling Improvements The improved error handling and logging support include the operations and its call chain details. This is especially useful when performing 100s of stream/task operations in SCDF, and if anything fails in the process, it will be easy to correlate and determine the root-cause more efficiently. There has been error handling improvements in SCDF’s cf-deployer, cf-java-client, and reactor-core projects.

  • Network Call Optimizations While invoking stream list from the shell or navigating to the equivalent stream list page in the Dashboard, we were performing a network call for each application associated with the stream(s). This is now refactored and replaced to efficiently query for all the application statuses in a single network call if the backing runtime supports it (and the Cloud Foundry deployer certainly does). This change significantly improves the user-experience of the stream listing operation.

  • Stress/Load Testing In the process of addressing above mentioned instability patterns, we have built a comprehensive integration and acceptance test harness. With the load-testing routines simulating 100s of concurrent operations on the server including “stream creates”, “stream deploys”, “stream destroys”, and “stream lists”, we were able to monitor the overall health of the SCDF server and as well correlate to confirm the successful execution of operations in entirety.

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.1 GA released

We are pleased to announce the general availability of Spring Cloud Data Flow for Cloud Foundry version 1.1.0.RELEASE.

In addition to all the bells and whistles of Spring Cloud Data Flow 1.1.2 that this release builds upon, the Cloud Foundry specifics include:

  • Memory Optimizations: After much troubleshooting, we are pleased to deliver a memory efficient SCDF server in this release. With the changes now, regardless of how many stream/task deployments and concurrency, we have observed the memory usage of the server stay within the 200MB range as opposed to 1G or over to an eventual crash in previous 1.0.0.RELEASE. If you experienced spurious crashes previously, we highly recommend upgrading to 1.1.0.RELEASE right away.

  • Maven Cache and Disk Optimizations: We have improved the way Maven caching is applied and used. There is a new LRU (Least Recently Used) design on top of the maven cache to allow us to be more proactive about the cache size as opposed to letting it grow indefinitely, which if not governed, leads to out of disk space errors for the application container where the server is running.

  • Tasks Support: In the latest PCF 1.9 release, the Task component graduated from being experimental to production-ready feature capability. Spring Cloud Data Flow’s Task support for Cloud Foundry closely followed this development and with this release, we have removed the “experimental” clause from our side, too. Spring Cloud Data Flow’s Task support is production-ready when used on PCF 1.9. One can still try this functionality on older PCF releases, PCF 1.8 for example, but the feature capabilities neither guaranteed nor supported on older PCF releases since is it classified as ”experimental”.

    The Task database schema changed slightly with this release. For folks upgrading from 1.0.0.RELEASE to 1.1.0.RELEASE, please review the migration steps included in the reference guide.

  • Improved Error Handling: Building on the latest version of the Spring Cloud Deployer for Cloud Foundry project, interaction with the Cloud Controller REST API has been made more resilient, being more pro-active in dealing with the occasional server-side error.

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.1M1 and 1.0.1 released

Spring Cloud Data Flow for Cloud Foundry gets two new releases today, detailed below:

1.0.1.GA is a maintenance release that

  • Builds upon 2.0.1.RELEASE of Cloud Foundry’s Java Client and 3.0.2.RELEASE of Reactor

  • Builds upon 1.0.1.RELEASE of Spring Cloud Data Flow core

  • Gets general clarifications and improvements in the reference documentation

  • Fixes a bug pertaining to multiple ORG + Space combination causing task launch failures

  • Fixes a bug where environment variables were not propagated correctly when launching a task

Read more...

Spring Cloud Data Flow for Cloud Foundry goes 1.0 GA

We are pleased to announce the general availability of Spring Cloud Data Flow for Cloud Foundry version 1.0.0.RELEASE.

Spring Cloud Data Flow for Cloud Foundry provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices on Cloud Foundry runtime.

As the successor to Spring XD, this project benefits from a much more decoupled architecture, leveraging the Spring Cloud Deployer for Cloud Foundry library, which also goes GA today. More details about Spring Cloud Data Flow’s architecture and its ecosystem can be found in this blog.

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.0.0.RC1 now available

We are pleased to announce the 1.0.0.RC1 release of Spring Cloud Data Flow for Cloud Foundry.

As we near completion towards a GA release in the upcoming days, this milestone brings the following improvements:

  • Builds upon the 1.0.0.RC1 release of Spring Cloud Deployer Cloud Foundry, which itself builds upon Project Reactor 3.0.0.RELEASE and of CF-Java-Client 2.0.0.RELEASE

  • Adds the ability to orchestrate short-lived tasks including Spring Batch Jobs in Cloud Foundry, which can be enabled as an experimental feature toggle

  • Adds support for command line arguments as a separate set of properties to be passed to a Task when it’s launched

  • Adds support to separate stream and task specific service bindings. This allows pinning stream and tasks specific services to stream and task applications respectively

  • Adds the ability to override the buildpack used on a per-app deployment basis

  • Significant improvements to the Spring Cloud Deployer Cloud Foundry TCK test coverage allowed refinements in app-state calculation

  • Adds documentation about rolling upgrades and how to integrate with Spring Cloud Config Server "service" in Pivotal Cloud Foundry

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.0.0.M4 now available

We are pleased to announce the release of Spring Cloud Data Flow’s Cloud Foundry 1.0.0.M4.

In this milestone release, we have few improvements to the APIs and the general stability of the overall design constructs.

Journey Ahead

  • We are looking forward to Reactor and Cloud Foundry Java Client’s RC and GA releases and by this month, we shall release 1.0.0.RELEASE of Spring Cloud Data Flow for Cloud Foundry.

  • We have exciting work-in-progress to orchestrate short-lived microservices through TaskLauncher constructs in the Spring Cloud Deployer for Cloud Foundry project. We are hopeful to deliver the BETA release of this functionality alongside the upcoming Pivotal Cloud Foundry’s 1.7.x release.

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.0.0.M3 released

I am pleased to announce the 1.0.0.M3 release of Spring Cloud Data Flow for Cloud Foundry, a team effort that encompasses many new features under the hood:

This third milestone builds upon the recent RC1 release of Spring Cloud Data Flow and also adds Cloud Foundry specific refinements. Some highlights include:

  • Alignment with Cloud Foundry’s global environment properties such as ORG, API, and URL as opposed to previous naming variants

  • Adds security integration to authenticate against OAUTH backend server that’s compatible with Cloud Foundry

  • Publishes security configurations for easier overrides when connecting to OAUTH backend server that’s compatible with Cloud Foundry

  • Preliminary iterations on performance testing on Cloud Foundry with and without taps’s in real-time streaming pipeline

  • Builds upon feature-toggle support from Spring Cloud Data Flow’s core project to conditionally refine what features get included in each release

Read more...

Spring Cloud Data Flow for Cloud Foundry 1.0.0.M2 released

On behalf of the Spring Cloud Data Flow team, I am pleased to announce the 1.0.0.M2 release of Spring Cloud Data Flow for Cloud Foundry.

Spring Cloud Data Flow for Cloud Foundry allows one to use all the goodness of Spring Cloud Data Flow (like the Shell, UI and Flo) while targeting Cloud Foundry as a backend. Stream components are deployed as individual apps in Cloud Foundry, leveraging the power of the platform to handle scaling and health monitoring.

This second milestone
- builds upon the 1.0.0.M1 release of Cloud Foundry implementation of spring-cloud-deployer
- builds upon the 1.0.0.M3 release of Spring Cloud Data Flow
- builds upon the 2.0.0.M4 release of the Cloud Foundry java client, leveraging its reactive programming style

Read more...