close

Case Study: HTTP Request Function and Processor

Introduction

We began this series by introducing the new stream applications based on Java functions, and function composition. The previous entry presented a tutorial for building a simple stream application and running it in Spring Cloud Data Flow. Today we explore the HTTP Request Function and present examples of how to use it.

In case you missed it, the prior posts in this series are:

The HTTP Request Function

This is an updated implementation of the legacy HTTP Client Processor Stream App Starter, based on the reactive Spring WebClient. The function is an all purpose web client that submits HTTP requests to a URL and returns the response. Designed primarily for streaming applications, it is able to extract the URL, HTTP method, request body, desired response type, and contents, using configured SpEL expressions evaluated against each incoming Message. Also, to support efficient stream processing, the function uses reactive streams. Its signature is:

Function<Flux<Message<?>>, Flux<?>>

That is, it accepts a Flux (stream) of Messages and returns a Flux of any type.

Configuration Properties

The HttpRequestFunction is configured through the following configuration properties:

http.request.body-expression
A SpEL expression to derive the request body from the incoming message. (Expression, default: <none>)

http.request.expected-response-type
The type used to interpret the response. (Class<?>, default: String)

http.request.headers-expression
A SpEL expression used to derive the http headers map to use. (Expression, default: <none>)

http.request.http-method-expression
A SpEL expression to derive the request method from the incoming message. (Expression, default: GET)

http.request.maximum-buffer-size
Maximum buffer size in bytes allocated for input stream buffers. Defaults to 256k. Increase, as necessary, for posting or getting large binary content. (Integer, default: 256 * 1024)

http.request.reply-expression
A SpEL expression used to compute the final result, applied against the whole http {@link org.springframework.http.ResponseEntity}. (Expression, default: ResponseEntity::getBody)

http.request.timeout
Request timeout in milliseconds. (Long, default: 30000)

http.request.url-expression
A SpEL expression against the incoming message to determine the URL to use. (Expression, default: <none>)

The SpEL expressions are applied to the incoming Message. So fields like body and headers[name] can be used to evaluate message contents. I say "can be…" because sometimes it is more desirable to use static values. In this case, literal values must be enclosed in single quotes, for example:

http.request.url-expression='https://start.spring.io'
http.request.http-method-expression='POST'

Example 1: Using the HTTP Request Function in a Standalone Application

Let’s look at an example of how to use this function in a simple Spring Boot web application. In this example, we will use it in an app that retrieves an image from a URL and renders a thumbnail of the image. The complete code for this example is here.

We will build the application using Spring Boot and Spring Web Flux, along with our function to retrieve the image, and some code to generate a thumbnail.

The relevant dependencies are:

  • org.springframework.cloud.fn:http-request-function - The HTTP request function transitively includes spring-boot-starter-webflux

  • io.spring.example:image-thumbnail-processor - A simple Java function, included in this example, that creates thumbnails. We won’t get into the details here, just note that it is a separate component which we will reuse in a later example.

We first need to set some configuration properties for our function:

http.request.url-expression=payload
http.request.expected-response-type=byte[]
http.request.maximum-buffer-size=2097152

Thus, the message payload contains the target URL, the image(response body) will be returned as a byte array. And since these images might be fairly large, we will increase the size of the buffer holding the response body to 2GB (2 * 1024 * 1024).

Here is the code:

@SpringBootApplication
@Controller
@Import(HttpRequestFunctionConfiguration.class)
public class ThumbnailStandaloneApplication {
  private static Logger logger = LoggerFactory.getLogger(ThumbnailStandaloneApplication.class);

  public static void main(String[] args) {
    SpringApplication.run(ThumbnailStandaloneApplication.class, args);
  }

  private ThumbnailProcessor thumbnailProcessor = new ThumbnailProcessor();

  @Autowired
  private HttpRequestFunction httpRequestFunction;

  @Bean
  RouterFunction<?> routes() {
    return RouterFunctions.route()
        .GET("/thumbnail", this::createThumbnail)
        .build();
  }

  private Mono<ServerResponse> createThumbnail(ServerRequest serverRequest) {
    String url = serverRequest.queryParam("url").orElseThrow(
                           () -> new RuntimeException("URL required"));

    return Mono.from(httpRequestFunction.apply(Flux.just(new GenericMessage<>(url)))
        .flatMap(image -> {
          Map<String, Object> model = new HashMap<>();
          byte[] thumbnail = thumbnailProcessor.apply((byte[]) image);
          logger.info("creating thumbnail for {}", url);
          model.put("url", url);
          model.put("thumb", new String(Base64.getEncoder().encode(thumbnail)));
          Mono<ServerResponse> serverResponse = ServerResponse.ok()
              .render("thumbnail", model);
          return serverResponse;
        }));
  }

We apply the HttpRequestFunction to retrieve the image. Then we apply the thumbnailProcessor to the returned byte array and encode it to base 64 so we can render it on the page.

standalone

Example 2: Using the HTTP Request Processor in a streaming application

Now that we know how our function works, let’s put together a streaming application, using Spring Cloud Stream, to do something similar. In this case, we will use the pre-packaged HTTP Request Processor and File Source stream applications. This processor wraps the HTTP request function in a Spring Cloud Stream processor application that simply invokes the function, binding the input and output to a message broker destination (a Kafka topic, or a Rabbit MQ exchange, for example). Our application, expressed in stream definition DSL, looks like:

file-source | http-request-processor | image-thumbnail-sink

where the | represents I/O using a message broker.

Here, we are using a user-developed sink that uses the file-consumer function to write each thumbnail to a file. The sink uses Spring Cloud Function’s declarative composition to compose the thumbnail-processor, from the previous example, with a header enricher, and finally the standard fileConsumer. So our composed function is defined by:

spring.cloud.function.definition=thumbnailProcessor|filenameEnricher|fileConsumer

Our composite function definition is conceptually and syntactically similar to the above stream definition. But in this case the | represents in-process communication.

We will explore the ins and outs of the File Source in a future post. For now, we will use it to poll a source directory and produce messages whenever a new file is added to the directory. In this case, we want to process a text file with an image URL per line. We will configure the source to produce a message per line, containing the URL in the payload. We already know what the HTTP request processor does. The sink generates a thumbnail and writes it to a file.

The fully configured stream definition is:

file-source --file.consumer.mode=lines --file.consumer.mode=lines --file.supplier.directory=<source-directory> | http-request-processor --http.request.url-expression=payload --http.request.expected-response-type=byte[] --http.request.maximum-buffer-size=2097152| image-thumbnail-sink --file.consumer.directory=<target-directory>

If we run this and drop a text file into the source directory, we will see the thumbnails written to the target directory:

thumbnail files

If you want to run this on your local machine, complete instructions are here.

Summary

We just did a deep dive on the HTTP Request Function, demonstrating how to use it in a standalone web application and in a streaming pipeline to process images. We also used function composition, composing user-written and out of the box functions, to great effect.

Stay Tuned…​

In the coming weeks we will present many more case studies for Spring Cloud Stream and Spring Cloud Data Flow, each will highlight different stream applications and capabilities.

Read more

Creating Efficient Docker Images with Spring Boot 2.3

This is an update to the original blog post about creating docker images with Spring Boot 2.3.
There were a few things related to image creation that changed between the first milestone of Spring Boot 2.3 and the GA release.

The two new features introduced in Spring Boot 2.3 to help improve image creation techniques were: layered jars and buildpack support.

Layered Jars

The layered jar feature evolved quite a bit as we started adding support for custom layers. While the need to express layers that the jar should be split into for image creation was evident, changing the format of the jar to do so no longer seemed necessary.
Spring Boot 2.3 includes support for layering a jar using a layers.idx file. The layers index file provides a list of layers and the parts of the jar that should be contained within them. Layers are written in the order that they should be added to the Docker/OCI image.

Read more

A Bootiful Podcast: RSocket everywhere and Maciej Walkowiak on Spring Cloud AWS

Hi, Spring fans! In this installment Josh Long (@starbuxman) has RSocket on the brain with his latest talk, RSocket Revolution, the new RSocket chapter in his book Reactive Spring, and the work he’s doing on Spring Retrosocket, a declarative Feign-like RSocket client. Then, Josh talks to Spring ecosystem luminary Maciej Walkowiak (@maciejwalkowiak) about what he’s doing at the helm of Spring Cloud AWS.

Read more

Config file processing in Spring Boot 2.4

Spring Boot 2.4.0.M2 has just been released, and it brings with it some interesting changes to the way that application.properties and application.yml files are loaded.

If your applications use the fairly typical setup of only using a single application.properties or application.yml file, then you’ll probably not notice any difference. If, however, you have applications that use more complex setups (such as profile specific properties), you might want to read on to learn about what we’ve changed and why.

Read more

Spring Boot 2.4.0-M2 is now available

On behalf of the team and everyone that contributed, I am pleased to announce that the second milestone of Spring Boot 2.4 has been released and is available from our milestone repository. This release closes just over 250 issues and pull requests.

If you’re trying this release with an existing Spring Boot 2.3 project, please note the slight change in the format of the version. 2.4.0-M2 is using our new versioning scheme.

Highlights of this second milestone include:

Read more

Spring Security 5.4.0-RC1 Released

On behalf of the community, I’m pleased to announce the release of Spring Security 5.4.0-RC1! You can find the complete details in the release notes and the highlights below:

OAuth 2.0

gh-8903 - Allow for custom ClientRegistration.clientAuthenticationMethod
gh-6489 - Simplify retrieving Introspection-specific attributes

Web

gh-8804 - Remove need for WebSecurityConfigurerAdapter
gh-8599 - Reactive SwitchUserWebFilter for user impersonation
gh-8854 - Add AuthenticationConverterServerWebExchangeMatcher
Read more

Spring Boot 2.3.3 available now

On behalf of the team and everyone who has contributed, I’m happy to announce that Spring Boot 2.3.3 has been released and is now available from repo.spring.io and Maven Central.

This release includes 67 bug fixes, enhancements, documentation improvements, and dependency upgrades. Thanks to all those who have contributed with issue reports and pull requests.

How can you help?

If you’re interested in helping out, check out the “ideal for contribution” tag in the issue repository. If you have general questions, please ask on stackoverflow.com using the spring-boot tag or chat with the community on Gitter.

Read more

Spring Batch 4.3.0-M2 is out!

On behalf of the Spring Batch team, I am pleased to announce that Spring Batch 4.3.0-M2 is now available from our milestone repository.

What’s new?

This release comes with a number of new features and enhancements that you can find in the release notes, but here are the major highlights:

1. Kafka support enhancements

By default, the KafkaItemReader reads a topic from the beginning. When we initially introduced this reader in v4.2, it was not possible to configure it to read records from a custom offset. In this release, we added this feature. As a result of this addition, it is now possible to configure the reader to start from the offset stored in Kafka.

Read more

Premiering: The RSocket Revolution

Hi, Spring fans! I have been developing a new talk and I finally got a chance to do a pre-recorded version of it for a keynote for a conference (in China) recently, and want to share the content with you all, so I’m premiering it here, too. This talk, The RSocket Revolution, is about 38 minutes and looks at:

  • some of the new features in Spring Boot 2.3x including GraalVM-capable native images
  • the basics of RSocket
  • fundamentals with RSocket @Controllers in Spring Framework
  • Toshiaki Maki’s fantastic rsc RSocket client
  • Declarative, Feign-like clients with Spring Retrosocket
  • The RSocketRequester to support making requests of RSocket services on the JVM
  • Spring Integration’s reactive and RSocket support
  • Bidirectional communication - where both client and service separately initiate the communication - using RSocket
  • Securing RSocket endpoints with Spring Security and RSocket metadata
Read more