What you’ll build
You’ll set up Spring XD (eXtreme Data), create a stream to monitor a live twitter feed, and pipe it into a file.
What you’ll need
About 15 minutes
JDK 1.7 or later
How to complete this guide
There are several package managers available for Mac OS X, but one of the most popular is Homebrew. And we have strong support for Homebrew! If you haven’t already, use these steps to set it up.
ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"
This installation script will provide extra information.
Homebrew has a base set of "formulas" to install many tools. It also supports 3rd party sets called "taps". Pivotal has a tap that provides several tools including Spring XD, which you’ll see in the next section.
Installing Spring XD
If you are using a Mac with Homebrew, the process is pretty simple.
brew tap pivotal/tap && brew install springxd
That’s it! It might take a little bit of time to install.
If you aren’t using Homebrew or you’re on a different platform:
Download Spring XD 1.2.0.RELEASE.
Move it to your preferred folder and unzip it. For convenience, we will refer to this location as
Running Spring XD
To start Spring XD:
|singlenode is Spring XD’s simplest mode and more suited for demonstration purposes. See https://github.com/spring-projects/spring-xd/wiki for more options and details.|
You should see something like this:
_____ __ _______ / ___| (-) \ \ / / _ \ \ `--. _ __ _ __ _ _ __ __ _ \ V /| | | | `--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | | /\__/ / |_) | | | | | | | (_| | / / \ \ |/ / \____/| .__/|_| |_|_| |_|\__, | \/ \/___/ | | __/ | |_| |___/ 1.2.0.RELEASE eXtreme Data Started : SingleNodeApplication Documentation: https://github.com/spring-projects/spring-xd/wiki
In another terminal, start Spring XD’s management shell:
This will produce a prompt:
_____ __ _______ / ___| (-) \ \ / / _ \ \ `--. _ __ _ __ _ _ __ __ _ \ V /| | | | `--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | | /\__/ / |_) | | | | | | | (_| | / / \ \ |/ / \____/| .__/|_| |_|_| |_|\__, | \/ \/___/ | | __/ | |_| |___/ eXtreme Data 1.2.0.RELEASE | Admin Server Target: http://localhost:9393 Welcome to the Spring XD shell. For assistance hit TAB or type "help". xd:>
Inside Spring XD’s shell create a twitter stream:
xd:> stream create --name twittersearchjava --definition "twittersearch --consumerKey=afes2uqo6JAuFljdJFhqA --consumerSecret=0top8crpmd1MXGEbbgzAwVJSAODMcbeAbhwHXLnsg --query='java' | file" --deploy
Here you are creating a stream which consists of a source and a sink. In addition to creating a stream definition, You are also deploying the stream using the
The stream is named twittersearchjava.
The source is twittersearch, using credentials required by the Twitter API, querying on the token java.
The results are piped into the file sink, which defaults to /tmp/xd/output/[streamName].out
In another terminal:
$ cd /tmp/xd/output $ tail -f twittersearchjava.out
Spring XD is capturing live data from twitter about java and writing it to
twittersearchjava.out. You should see the tweets rendered as JSON. While Spring XD runs and the stream is deployed, the file sink will continue to grow as it accumulates more data. You can stop the stream by undeploying it from the Spring XD shell:
xd:>stream undeploy twittersearchjava
Congratulations! You’ve just installed Spring XD and created a stream to capture a live twitter feed and pipe it into a file.