COVID-19 stream

Interested in exploring Labs?
The endpoints we release in Labs are previews of tools that may be released more broadly in the future, but will likely undergo changes before then. We encourage you to take that into consideration as you explore. Before getting started, please read more about Twitter Developer Labs.

Overview

We want developers and researchers to be able to study the conversation about COVID-19 in real-time. To do so, we’re releasing a streaming endpoint into Twitter Developer Labs to do just that.

The COVID-19 streaming endpoint will return Tweets based on our internal COVID-19 Tweet annotation and a set of defined parameters we believe deliver a comprehensive view of the conversation around this topic.

This endpoint is a real-time stream of public Tweets we believe deliver a comprehensive view of the conversation about COVID-19. Compared to other Twitter streaming endpoints, it does not have data volume or throughput limitations, and it offers full-fidelity.

Compared to our other streaming endpoints, the COVID-19 stream features the following:

  • Is configured with a set of filters we believe deliver a comprehensive view of the conversation about this topic
  • Delivers the full conversation about this topic, rather than just a sample
  • Does not have throughput limitations (as opposed to 50 Tweets per second of the filtered stream endpoint)
  • Does not have a daily or monthly quota of Tweets you can receive – it delivers complete results
  • Delivers results in the legacy enriched native Tweet format – useful to ingest Tweets in a data pipeline already compatible with Historical PowerTrack or Sample or Decahose streams.

This streaming endpoint is a GET connection authenticated using a Bearer Token. Once a connection is established, Tweets are delivered in the legacy enriched native JSON response format through a persistent HTTP streaming connection. 

Support for partitioning

Due to the high volume of data that it provides, the COVID-19 stream endpoint utilized 4 partitions to split the overall volume of data over multiple connections. This is achieved using the required partition parameter. This means that you will need to connect to all 4 partitions, each containing about 25% of the total volume, to consume the full stream.

Access requirements

The volume of data this stream delivers is significant, such that prior experience working with Twitter data and having the necessary infrastructure in place to process, store, and analyze the many millions of Tweets being generated every day is critical. We have introduced a dedicated application to ensure you have access to the capabilities required to effectively use the endpoint in service of the public good. Prerequisites to that application include:

  • You have an approved Twitter developer account
  • You have demonstrable experience working with Twitter data in JSON format
  • You have a clear plan and ability to support the ingestion, processing, storage, and analysis of streaming data at a scale of tens of millions of Tweets per day
  • Your project is adherent with our developer terms and restricted use cases
  • Your use case is non-commercial (for example, you do not plan to monetize the use of this data)
  • Your use case supports the public good

When the application is approved, you will need to activate the “COVID-19 stream” preview in the Labs dashboard.

 

Additional resources