This guide walks you through the process of using Spring Integration to create a simple application that retrieves data from Twitter, manipulates the data, and then writes it to a file.

What you’ll build

You’ll create a flow using Spring Integration.

What you’ll need

How to complete this guide

Like most Spring Getting Started guides, you can start from scratch and complete each step, or you can bypass basic setup steps that are already familiar to you. Either way, you end up with working code.

To start from scratch, move on to Build with Gradle.

To skip the basics, do the following:

When you’re finished, you can check your results against the code in gs-integration/complete.

Build with Gradle

Build with Gradle

First you set up a basic build script. You can use any build system you like when building apps with Spring, but the code you need to work with Gradle and Maven is included here. If you’re not familiar with either, refer to Building Java Projects with Gradle or Building Java Projects with Maven.

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure; for example, with mkdir -p src/main/java/hello on *nix systems:

└── src
    └── main
        └── java
            └── hello

Create a Gradle build file

build.gradle

buildscript {
    repositories {
        maven { url "http://repo.spring.io/libs-release" }
        mavenLocal()
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:1.1.6.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'spring-boot'

jar {
    baseName = 'gs-integration'
    version =  '0.1.0'
}

repositories {
    mavenLocal()
    mavenCentral()
    maven { url "http://repo.spring.io/libs-release" }
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-integration")
    compile("org.springframework.integration:spring-integration-twitter:4.0.3.RELEASE")
    testCompile("junit:junit")
}

task wrapper(type: Wrapper) {
    gradleVersion = '1.11'
}

The Spring Boot gradle plugin provides many convenient features:

  • It collects all the jars on the classpath and builds a single, runnable "über-jar", which makes it more convenient to execute and transport your service.

  • It searches for the public static void main() method to flag as a runnable class.

  • It provides a built-in dependency resolver that sets the version number to match Spring Boot dependencies. You can override any version you wish, but it will default to Boot’s chosen set of versions.

Build with Maven

Build with Maven

First you set up a basic build script. You can use any build system you like when building apps with Spring, but the code you need to work with Maven is included here. If you’re not familiar with Maven, refer to Building Java Projects with Maven.

Create the directory structure

In a project directory of your choosing, create the following subdirectory structure; for example, with mkdir -p src/main/java/hello on *nix systems:

└── src
    └── main
        └── java
            └── hello

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.springframework</groupId>
    <artifactId>gs-integration</artifactId>
    <version>0.1.0</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.6.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-integration</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.integration</groupId>
            <artifactId>spring-integration-twitter</artifactId>
            <version>4.0.3.RELEASE</version>
        </dependency>
    </dependencies>

    <properties>
        <start-class>hello.Application</start-class>
    </properties>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>http://repo.spring.io/libs-release</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <url>http://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>

</project>
You might notice a specific version of maven-compiler-plugin. This is NOT recommended in general. Instead, it’s meant to solve an issue with our CI system that defaulted to a very old (pre-Java5) version of this plugin.

The Spring Boot Maven plugin provides many convenient features:

  • It collects all the jars on the classpath and builds a single, runnable "über-jar", which makes it more convenient to execute and transport your service.

  • It searches for the public static void main() method to flag as a runnable class.

  • It provides a built-in dependency resolver that sets the version number to match Spring Boot dependencies. You can override any version you wish, but it will default to Boot’s chosen set of versions.

Build with Spring Tool Suite

Build with Spring Tool Suite

If you have Spring Tool Suite, then you can simply import this guide directly.

Define an integration plan

For this guide’s sample application, you will define a Spring Integration plan that reads tweets from Twitter, transforms them into an easily readable String, and appends that String to the end of a file.

To define an integration plan, you simply create a Spring XML configuration with a handful of elements from Spring Integration’s XML namespaces. Specifically, for the desired integration plan, you work with elements from these Spring Integration namespaces: core, twitter, and file.

The following XML configuration file defines the integration plan:

src/main/resources/hello/integration.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:int="http://www.springframework.org/schema/integration"
    xmlns:twitter="http://www.springframework.org/schema/integration/twitter"
    xmlns:file="http://www.springframework.org/schema/integration/file"
    xsi:schemaLocation="http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
        http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file-2.2.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/integration/twitter http://www.springframework.org/schema/integration/twitter/spring-integration-twitter-2.2.xsd">

    <twitter:search-inbound-channel-adapter id="tweets"
            query="#HelloWorld"
            twitter-template="twitterTemplate">
        <int:poller fixed-rate="5000"/>
    </twitter:search-inbound-channel-adapter>

    <int:transformer
            input-channel="tweets"
            expression="payload.fromUser + '  :  ' + payload.text + @newline"
            output-channel="files"/>

    <file:outbound-channel-adapter id="files"
            mode="APPEND"
            charset="UTF-8"
            directory="/tmp/si"
            filename-generator-expression="'HelloWorld'"/>

</beans>

As you can see, three integration elements are in play here:

  • <twitter:search-inbound-channel-adapter>. An inbound adapter that searches Twitter for tweets with "#HelloWorld" in the text. It is injected with a TwitterTemplate from Spring Social to perform the actual search. As configured here, it polls every 5 seconds. Any matching tweets are placed into a channel named "tweets" (corresponding with the adapter’s ID).

  • <int:transformer>. Transformed tweets in the "tweets" channel, extracting the tweet’s author (payload.fromUser) and text (payload.text) and concatenating them into a readable String. The String is then written through the output channel named "files".

  • <file:outbound-channel-adapter>. An outbound adapter that writes content from its channel (here named "files") to a file. Specifically, as configured here, it will append anything in the "files" channel to a file at /tmp/si/HelloWorld.

This simple flow is illustrated like this:

A flow plan that reads tweets from Twitter

The integration plan references two beans that aren’t defined in integration.xml: the "twitterTemplate" bean that is injected into the search inbound adapter and the "newline" bean referenced in the transformer. Those beans will be declared separately in JavaConfig as part of the main class of the application.

Make the application executable

Although it is common to configure a Spring Integration plan within a larger application, perhaps even a web application, there’s no reason that it can’t be defined in a simpler standalone application. That’s what you do next, creating a main class that kicks off the integration plan and also declares a handful of beans to support the integration plan. You also build the application into a standalone executable JAR file.

src/main/java/hello/Application.java

package hello;

import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.ImportResource;
import org.springframework.core.env.Environment;
import org.springframework.social.oauth2.OAuth2Operations;
import org.springframework.social.oauth2.OAuth2Template;
import org.springframework.social.twitter.api.Twitter;
import org.springframework.social.twitter.api.impl.TwitterTemplate;

@Configuration
@ImportResource("/hello/integration.xml")
public class Application {

    public static void main(String[] args) {
        new AnnotationConfigApplicationContext(Application.class);
    }

    @Bean
    public String newline() {
        return System.getProperty("line.separator");
    }

    @Bean
    public Twitter twitterTemplate(OAuth2Operations oauth2) {
        return new TwitterTemplate(oauth2.authenticateClient().getAccessToken());
    }

    @Bean
    public OAuth2Operations oauth2Template(Environment env) {
        return new OAuth2Template(env.getProperty("clientId"), env.getProperty("clientSecret"), "", "https://api.twitter.com/oauth2/token");
    }

}

As you can see, this class provides a main() method that loads the Spring application context. It’s also annotated as a @Configuration class, indicating that it will contain bean definitions.

Specifically, three beans are created in this class:

  • The newline() method creates a simple String bean containing the underlying system’s newline character(s). This is used in the integration plan to place a newline at the end of the transformed tweet String.

  • The twitterTemplate() method defines a TwitterTemplate bean that is injected into the <twitter:search-inbound-channel-adapter>.

  • The oauth2Template() method defines a Spring Social OAuth2Template bean used to obtain a client access token when creating the TwitterTemplate bean.

The oauth2Template() method references the Environment to get "clientId" and "clientSecret" properties. Those properties are ultimately client credentials you are given when you register your application with Twitter. Fetching them from the Environment means you don’t have to hardcode them in this configuration class. You’ll need them when you Run the application, though.

Finally, notice that Application is configured with @ImportResource to import the integration plan defined in /hello/integration.xml.

Build an executable JAR

You can build a single executable JAR file that contains all the necessary dependencies, classes, and resources. This makes it easy to ship, version, and deploy the service as an application throughout the development lifecycle, across different environments, and so forth.

./gradlew build

Then you can run the JAR file:

java -jar build/libs/gs-integration-0.1.0.jar

If you are using Maven, you can run the application using mvn spring-boot:run. Or you can build the JAR file with mvn clean package and run the JAR by typing:

java -jar target/gs-integration-0.1.0.jar
The procedure above will create a runnable JAR. You can also opt to build a classic WAR file instead.

Run the application

Now you can run the application from the jar:

java -DclientId={YOUR CLIENT ID} -DclientSecret={YOUR CLIENT SECRET} -jar build/libs/{project_id}-0.1.0.jar

... app starts up ...

Make sure you specify your application’s client ID and secret in place of the placeholders shown here.

Once the application starts up, it connects to Twitter and starts fetching tweets that match the search criteria of "#HelloWorld". The application processes those tweets through the integration plan you defined, ultimately appending the tweet’s author and text to a file at /tmp/si/HelloWorld.

After the application has been running for awhile, you should be able to view the file at /tmp/si/HelloWorld to see the data from a handful of tweets. On a UNIX-based operating system, you can also choose to tail the file to see the results as they are written:

tail -f /tmp/si/HelloWorld

You should see something like this (the actual tweets may differ):

BrittLieTjauw  :  Now that I'm all caught up on the bachelorette I can leave my room #helloworld
mishra_ravish  :  Finally, integrated #eclim. #Android #HelloWorld
NordstrmPetite  :  Pink and fluffy #chihuahua #hahalol #boo #helloworld http://t.co/lelHhFN3gq
GRhoderick  :  Ok Saint Louis, show me what you got. #HelloWorld

Summary

Congratulations! You have developed a simple application that uses Spring Integration to fetch tweets from Twitter, process them, and write them to a file.