The Streaming APIs

Updated on Mon, 2012-09-24 13:47

Overview

The set of streaming APIs offered by Twitter give developers low latency access to Twitter's global stream of Tweet data. A proper implementation of a streaming client will be pushed messages indicating Tweets and other events have occurred, without any of the overhead associated with polling a REST endpoint.

Twitter offers several streaming endpoints, each customized to certain use cases.

Public streams Streams of the public data flowing through Twitter. Suitable for following specific users or topics, and data mining.
User streams Single-user streams, containing roughly all of the data corresponding with a single user's view of Twitter.
Site streams The multi-user version of user streams. Site streams are intended for servers which must connect to Twitter on behalf of many users.

Differences between Streaming and REST

Connecting to the streaming API requires keeping a persistent HTTP connection open. In many cases this involves thinking about your application differently than if you were interacting with the REST API. For an example, consider a web application which accepts user requests, makes one or more requests to Twitter's API, then formats and prints the result to the user, as a response to the user's initial request:

Diagram of a HTTP server issuing Twitter API requests in response to user requests

An app which connects to the Streaming APIs will not be able to establish a connection in response to a user request, as shown in the above example. Instead, the code for maintaining the Streaming connection is typically run in a process separate from the process which handles HTTP requests:

Diagram of two server processes, where one process receives streamed Tweets, while the other handles HTTP requests

The streaming process gets the input Tweets and performs any parsing, filtering, and/or aggregation needed before storing the result to a data store. The HTTP handling process queries the data store for results in response to user requests. While this model is more complex than the first example, the benefits from having a realtime stream of Tweet data make the integration worthwhile for many types of apps.