On behalf of the team, I am pleased to announce the release of the first milestone of Spring Cloud Data Flow for Kubernetes 1.1 along with a 1.0.1 maintenance release.

Spring Cloud Data Flow for Kubernetes provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices on Kubernetes.

Note: A great way to start using this new release(s) is to follow the release matrix on the project page, which includes the download coordinates and the links to the reference guide.

The most significant change for both of these releases is the way we handle task launching. We used to orchestrate them as a Job with associated Pods, but that proved problematic in terms of failed jobs getting restarted even with a RestartPolicy specified as "Never". To resolve this, we now launch the tasks using bare pods…

On behalf of the team, I am excited to announce the General Availability of Spring Cloud Data Flow for Apache Mesos 1.0.

Spring Cloud Data Flow for Apache Mesos provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices on Apache Mesos. We launch stream apps using Application Groups on Marathon and tasks as Chronos one-time jobs. The release includes a template JSON script for deploying the Spring Cloud Data Flow server on Marathon. We also include sample scripts for deploying Redis, MySQL and Rabbit MQ on Marathon to be used for testing a…

On behalf of the team, I am excited to announce the release of the first milestone of Spring Cloud Data Flow 1.1 along with a 1.0.1 maintenance release for the 1.0 version.

Note: A great way to start using this new release(s) is to follow the release matrix on the project page, which includes the download coordinates and the links to the reference guide.

1.1 M1 release

Over the last few weeks, we have added new features and improvements to the overall orchestration of data microservices. The following new features were included in the 1.1.0.M1 release:

Security

• Adds LDAP backend-authentication
• Adds Basic backend-authentication
• Adds File backend-authentication
• Improvements to OAUTH backed-authentication
…

We are pleased to announce the 1.0.0.RC2 release candidate of Spring Cloud Data Flow for Mesos, a team effort that encompasses many new features under the hood.

This release candidate builds upon the recent 1.0 GA release of Spring Cloud Data Flow. Some highlights include:

• We now run the Spring Cloud Data Flow Server as a Docker image on Marathon, a container orchestration platform for Mesos.
• This release adds features to support stream partitioning and scaling

• Currently partitioning and scaling of sinks and processors are handled by using multiple application deployments, one for each app instance, identified by an index appended to the name.
• Scaling of sources is handled by using additional application instances.

* Streams are now deployed using Marathon [Application Groups](https://mesosphere.github.io/marathon/docs/application-groups.html) so it is easier to identify the different apps making up a stream. * We have added support for launching tasks using Chronos, a fault tolerant job scheduler for Mesos.

As part of this effort we have developed a simple Java client for interacting with the Chronos API. This Java client is included in the latest 1.0.2.RELEASE version of the Spring Cloud Deployer for Mesos project…

We are pleased to announce the release of Spring Cloud Data Flow's Kubernetes 1.0.0.RELEASE.

Spring Cloud Data Flow for Kubernetes provides support for orchestrating long-running (streaming) and short-lived (task/batch) data microservices on Kubernetes.

This project was originally conceptualized by the community and we are thankful to Florian Rosenberg for his early contributions that eventually made it into the official Spring Cloud Deployer for Kubernetes project. Building upon this theme, we recently bumped into Donovan Muller's blog, where he walks through his experience extending the…

We are pleased to announce the Spring for Apache Hadoop 2.4.0 GA release.

In addition to version upgrades for Hadoop distros Spring for Apache Hadoop 2.4 adds the following improvements:

• Add build support for HDP 2.4 [SHDP-557]
• Create utils sub-project [SHDP-556]
• Fix issue where NetworkUtils may find wrong default address [SHDP-546]
• Missing trailing slash on spring.yarn.applicationDir path fails to localize [SHDP-562]

See the release changelog for details.

We continue to provide version specific artifacts with their respective transitive dependencies in the Spring IO milestone repository:

• 2.4.0.RELEASE (default - Apache Hadoop stable 2.7.1)
• 2.4.0.RELEASE-phd30 (Pivotal HD 3.0)
• 2.4.0.RELEASE-cdh5 (Cloudera CDH 5.7)
• 2.4.0.RELEASE-hdp23 (Hortonworks HDP 2.3)
• 2.4.0.RELEASE-hdp24 (Hortonworks HDP 2.4)
…

I am pleased to announce the 1.0.0.RC1 release candidate of Spring Cloud Data Flow for Kubernetes, a team effort that encompasses many new features under the hood.

This release candidate builds upon the recent 1.0.0.RC1 release of Spring Cloud Data Flow and also adds Cloud Foundry specific refinements. Some highlights include:

• Implements the TaskLauncher SPI to support launching of short-lived applications a.k.a tasks. They will be created as Kubernetes jobs and each task launch will create a new pod as part of the Kubernetes job
• Adds support to configure command line args as a separate set of properties to be passed to an application when it's deployed. This is an important requirement for running Spring Batch jobs orchestrated as tasks using Spring Cloud Task programming model
• Adds support for RDBMS “datasource” to supplement persistent repository requirements
• Adds /info…

I am pleased to announce the 1.0.0.RC1 release candidate of Spring Cloud Data Flow for Mesos, a team effort that encompasses many new features under the hood.

This release candidate builds upon the recent 1.0.0.RC1 release of Spring Cloud Data Flow. Some highlights include:

• Improves orchestration mechanics by inheriting latest improvements to spring-cloud-deployer contracts and the supported functionalities.
• Builds upon feature-toggle support from Spring Cloud Data Flow’s core project to conditionally refine what features get included in each release

To get started using Spring Cloud Data Flow for Mesos follow these steps outlined…

On behalf of the team, I am pleased to announce 1.0.0.M2 releases of Spring Cloud Data Flow for Apache Mesos and Spring Cloud Data Flow for Kubernetes.

Spring Cloud Data Flow for Apache Mesos allows one to use all the goodness of Spring Cloud Data Flow (like the Shell, UI and Flo) while targeting Apache Mesos as a backend. Stream components are deployed as individual apps using Marathon, leveraging the power of the platform to handle scaling and health monitoring.

This second milestone

• Builds upon 1.0.0.M1 release of the Spring Cloud Deployer Mesos/Marathon implementation
• Builds upon 1.0.0.M3 release of Spring Cloud Data Flow
• Replaces spring-cloud-marathon-connector with environment variables for service connection parameters. We will improve the service discovery in future release and tie in to Mesos/Marathon’s native service discovery mechanism for runtime credentials
• Adds support to resolve, register, and run OOTB and custom apps as docker images
…

We are pleased to announce the Spring for Apache Hadoop 2.3.0 GA release.

Spring for Apache Hadoop 2.3 adds the following new features and improvements:

• Added support for Hive 1.x and updated HiveTemplate to work with HiveServer2
• New batch tasklet for Apache Spark, see docs for a usage example
• Added FlushTimeoutTrigger to StoreObjectSupport to allow to flush to disk during writes
• Replaced internal state machine implementation with new "spring-statemachine" project
• Added jobHistoryAddress to SpringHadoopProperties for Boot configuration
• Update build to use Spring Framework 4.2.4, Spring Batch 3.0.6.RELEASE, Spring Boot 1.3.1.RELEASE and Spring Integration 4.2.4.RELEASE.
…