What you’ll build

You’ll set up Spring XD (eXtreme Data), create a stream to monitor a live twitter feed, and pipe it into a file.

What you’ll need

  • About 15 minutes

  • JDK 1.7 or later

How to complete this guide

If you are on a Mac there are some simple instructions to Install Homebrew.

If you don’t have a Mac or you aren’t interested in installing Homebrew, then you can skip that and jump right to Installing Spring XD.

Install Homebrew

There are several package managers available for Mac OS X, but one of the most popular is Homebrew. And we have strong support for Homebrew! If you haven’t already, use these steps to set it up.

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"

This installation script will provide extra information.

Homebrew has a base set of "formulas" to install many tools. It also supports 3rd party sets called "taps". Pivotal has a tap that provides several tools including Spring XD, which you’ll see in the next section.

Installing Spring XD

If you are using a Mac with Homebrew, the process is pretty simple.

brew tap pivotal/tap && brew install springxd

That’s it! It might take a little bit of time to install.

If you aren’t using Homebrew or you’re on a different platform:

  1. Download Spring XD 1.2.0.RELEASE.

  2. Move it to your preferred folder and unzip it. For convenience, we will refer to this location as $XD_INSTALL_DIR

Running Spring XD

To start Spring XD:

$XD_INSTALL_DIR/xd/bin/xd-singlenode
singlenode is Spring XD’s simplest mode and more suited for demonstration purposes. See https://github.com/spring-projects/spring-xd/wiki for more options and details.

You should see something like this:

 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
1.2.0.RELEASE                    eXtreme Data

Started : SingleNodeApplication
Documentation: https://github.com/spring-projects/spring-xd/wiki

In another terminal, start Spring XD’s management shell:

$XD_INSTALL_DIR/shell/bin/xd-shell

This will produce a prompt:

 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
eXtreme Data
1.2.0.RELEASE | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>

Inside Spring XD’s shell create a twitter stream:

xd:> stream create --name twittersearchjava --definition "twittersearch --consumerKey=afes2uqo6JAuFljdJFhqA --consumerSecret=0top8crpmd1MXGEbbgzAwVJSAODMcbeAbhwHXLnsg --query='java' | file" --deploy

Here you are creating a stream which consists of a source and a sink. In addition to creating a stream definition, You are also deploying the stream using the --deploy option.

  • The stream is named twittersearchjava.

  • The source is twittersearch, using credentials required by the Twitter API, querying on the token java.

  • The results are piped into the file sink, which defaults to /tmp/xd/output/[streamName].out

In another terminal:

$ cd /tmp/xd/output
$ tail -f twittersearchjava.out

Spring XD is capturing live data from twitter about java and writing it to twittersearchjava.out. You should see the tweets rendered as JSON. While Spring XD runs and the stream is deployed, the file sink will continue to grow as it accumulates more data. You can stop the stream by undeploying it from the Spring XD shell:

xd:>stream undeploy twittersearchjava

Summary

Congratulations! You’ve just installed Spring XD and created a stream to capture a live twitter feed and pipe it into a file.

Want to write a new guide or contribute to an existing one? Check out our contribution guidelines.