Search Tweets

Integrating premium search

Introduction

Welcome to Twitter's premium Search Tweets API. This API has two sets of endpoints, 30-day and Full-archive, that can provide either the most recent month of Tweets or Tweets from as early as 2006. The purpose of this guide is to provide an orientation for developers integrating with this new premium search API. Your API integration path will depend on your experience with other Twitter search APIs. If you are completely new to Twitter search APIs, then reviewing this guide is a great first step to get started with the Search Tweets API. 

If you are migrating from either the standard or enterprise search APIs, both have fundamental differences and similarities to the premium Search Tweets API. Authenticating requests with the Search Tweets API is similar to the standard search API, while fundamental API details, such as query syntax, request parameters, pagination, availability of counts endpoint, and response payloads are the same as the enterprise APIs. See below for more details.

Key documentation for any integration effort includes the premium search overview and the premium search API reference. 

The premium search overview provides an introduction to the following fundamental topics:

• New Twitter developer's portal where you create the development environments necessary to start working with the API.
• There are two types of development environments: Sandbox and Premium.
• Key API characteristics that are important to consider as you start your integration.

The following material will provide a review of request parameters, examples of API JSON responses, and example requests using code and common HTTP tools.

Migrating from standard search

If you are transitioning from the standard search API (referred to as the Twitter Search API, and part of the Twitter ‘public’ platform), migration to the Search Tweets API is a multi-step process. Differences in the APIs include request parameters, many response JSON details, availability of a counts endpoint, pagination details, available query operators and available requests per month window (vs. 15 minute window).  As their names imply, the Search Tweets 30-day and Full-archive endpoints enable access to significantly more historical data than the 7-day standard search.

Although the returned Tweet JSON payloads are similar in structure, there are fundamental differences. Both response payloads consist of an array of Tweet objects and an object providing metadata about the request made. With the Search Tweets API, there is a ‘results’  array (instead of ‘statuses’), and a ‘requestParameters’ hash (instead of ‘seach_metadata’). There are also differences in the Tweet attributes returned. For example, the Search Tweets API payloads provide reply_count, quote_count, and matching_rules attributes. See HERE for an introduction to the Tweet JSON returned by the Search Tweets API. If you want to jump straight into Tweet JSON details, see HERE.

There are also some notable similarities between the standard and Search Tweets APIs. Fundamental HTTP ‘plumbing’ details are the same, including OAuth authentication. If your code is already using application-only OAuth for authentication, then you can quickly move on to the fun stuff. The querying languages are similar enough that being familiar with standard query operators provides for quick learning of the new syntax.

Code and data parsers written for the standard search API will likely provide a foundation for Search Tweets API integration.

Note: There is currently not a usage API to track premium usage, do not use the rate_limit_status API to try to track premium usage.

Migrating from enterprise search

If you have experience with Twitter’s enterprise search APIs, you’ll find many similarities. Details such as request parameters, pagination, and the fundamentals of building queries are the same. Key differences include a different method of authentication, a different list of available query operators, and search queries can have a maximum of 256 characters. Also, if you are ingesting Tweets in the Activity Stream JSON format, you will need to adjust your parser to handle Twitter’s enriched native format.

Next, we will dig into the details of integrating with the Search Tweets API. We’ll start off by discussing authentication, then cover request parameters, response payloads, and provide some example requests.

Note: There is currently not a usage API to track premium usage, do not use the enterprise Usage API to try to track premium usage.

Integrating with the Search Tweets API

From a developer’s viewpoint, this new search API is similar in design and behavior to the enterprise search APIs, of which there are also two endpoints: 30-Day and Full-Archive. Many details of those APIs are identical in the Search Tweets API, including request parameters, query syntax, availability of a Tweet counts endpoint, pagination details, and response payloads. Like the enterprise search APIs, Search Tweets API supports POST requests (and are generally recommended over GET requests).

One key difference between those enterprise search APIs and this premium API is the method of authentication. The enterprise APIs curre∂jntly use Basic authentication, while the Search Tweets API uses two flavors of OAuth. So, if you have code written for an enterprise API, a first step is updating the authentication details by teaching it OAuth.

We’ll split the following integration information into these sections:

• Request parameters

• Handling responses

• Example requests

So, we’ll start off this section with a discussion of authentication, the first step for any Twitter API request. Then we’ll dig into details about making search requests and show some examples.

Request parameters

The premium Search Tweets API shares the same API request parameters as the enterprise search APIs.

Data requests

• query - Required. Filter/rule syntax used to match Tweets. The query can be up to 1,024 characters long (256 with Sandbox tier).

• fromDate, toDate - Optional. Used to specify a time range (UTC. The format is YYYYMMDDhhmm. If not supplied, the time period defaults to the past 30 days (true for both endpoints).

• maxResults - Optional. Specifies how many Tweets (10-100) to return per response. The default is 100.

• tag - Optional. Tags can be used to segregate rules and their matching data into different logical groups. If a rule tag is provided, the rule tag is included in a matching_rules JSON attribute.

• next - This parameter is used to get the next "page" of results as described HERE. The value used with the request parameter is pulled directly from the API response, and should not be modified.

  For more data request details and examples, see HERE.
  

Counts requests (Premium tier only)

• query - Required. Filter/rule syntax used to match Tweets. The query can be up to 1,024 characters long (256 with Sandbox tier).

• fromDate, toDate - Optional. Used to specify a time range (UTC). The format is YYYYMMDDhhmm. If not supplied, the time period defaults to the past 30 days (true for both endpoints).

• bucket - Optional. The unit of time for which count data will be provided. By default, hourly counts will be provided. Options: "day", "hour", "minute"

• tag - Optional. Tags can be used to segregate rules and their matching data into different logical groups. If a rule tag is provided, the rule tag is included in a matching_rules JSON attribute.

• next - This parameter is used to get the next "page" of results as described HERE. The value used with the request parameter is pulled directly from the API response, and should not be modified.

For more counts request details and examples, see HERE.

Handling responses

Data responses

Data requests to the Search Tweets API return matching Tweets in a “results” top-level JSON array of Tweet objects. When making data requests it is likely that there is more data than can be returned in a single response. When that is the case the response will include a ‘next’ token, provided as a root-level JSON attribute. Whenever a ‘next’ token is provided, there is additional data to retrieve so additional API requests are necessary.

There is also a “requestParameters” key that echoes the request made. Note that the if you pass in a rule tag, that is contained in the Tweets’ ‘matching_rules’ metadata.

Here is the general structure of Tweet JSON payloads returned by Tweet Search:

{
  "results": [
       {Tweet},
       {Tweet},
       {Tweet}
  ],
  "next": "TOKEN",
  "requestParameters": {
      "maxResults": 10,
      "fromDate": "201709020000",
      "toDate": "201710021814"
  }
}

For more Tweet data response details and examples, see these enterprise search API response JSON examples. The structure shown above is the same in both APIs.  

Count responses

Count requests to the Search Tweets API return the amount of matching Tweets in a “results” top-level JSON array. Array items contain the date/time stamp marking the beginning of the time ‘bucket’ along with the number of matching Tweets. The time-series interval is specified with the ‘bucket’ request parameter, which can be set to ‘minute’, ‘hour’, or ‘day’ (default).

In some cases, the Tweet counts may require more than one response. When that is the case the response will include a ‘next’ token, provided as a root-level JSON attribute. Whenever a ‘next’ token is provided, there is additional counts to retrieve so additional API requests are necessary.

There is also a “requestParameters” key that echoes the request made. Note that the if you pass in a rule tag, that is contained in the Tweets’ ‘matching_rules’ metadata.

Here is the general structure of Tweet count JSON payloads returned by Tweet Search:

For more count response details and examples, see these enterprise API response JSON examples. The structure shown above is the same in both APIs. 

For descriptions of potential error responses, see this list of error responses, with error codes and messages.

Example requests

In this section, we’ll present some example API requests. We’ll use curl to demonstrate both GET and POST requests, and also show some sample code to help get started. If you update the curl examples below with your authentication Bearer token, you will be able to quickly start exercising the API.

All Search Tweets API requests request a query, and for the following examples the following rule will be submitted:

(snow OR sleet OR hail OR (freezing rain)) has:images

The examples below reference the 30-day endpoint. If you will be using the Full-archive endpoint, you would replace the '30day' URL token with 'fullarchive'.

curl examples

With GET requests, all request parameters are URL encoded and appended to the API endpoint. 

Here is an example setting the query parameter to the rule above and maxResults set to 10:

With POST requests, all the request parameters are encoded in JSON and provided as a request data body. The two request parameters would be encoded as: 

With curl, we specify the POST method with the ‘-X’ parameter (curl’s default method is GET), and use the -d parameter to pass in the JSON data body:

In general, POST requests with JSON-encoded request parameters are recommended since URL encoding can get messy, and JSON is generally more readable in code and logs (as well as documentation).

In this example, a search date range is specified. Here is the JSON data body:

Example curl command (with HTTP POST):

httpie example

Example httpie command (with HTTP GET):

Code examples

Below are code snippets that illustrate the steps for making API requests in Python and Ruby. In these examples Bearer tokens are passed in as request headers. Note that you should ensure you use the correct environment name, and also that you specify a valid date range for your search (for 30-day search, the range must be within the past 30 days for the query to work).

Request rate limits

Search Tweets API has the following technical and access rate limits:

Technical Rate limits:

Requests per second: 10

Requests per minute:

• Sandbox tier: 30
  
• Premium tier: 60

Access Rate Limits:

Requests per month:

• Sandbox tier: 250
• Premium tier: 500-10,000

Note: Do not use the rate_limit_status to track premium API usage.

Next Steps

• Check out these example search API clients: Python, Ruby.
• Learn more about Search Tweets operators.   
• Learn more about building Search queries. See our tutorial on translating plain language to PowerTrack rules. The Search Tweets API’s querying language is based on Twitter’s enterprise syntax (PowerTrack).
• Review our OAuth documentation.
• Learn more about Tweet JSON.