The Twitter REST API utilizes a technique called ‘cursoring’ to paginate large result sets. Cursoring separates results into pages (the size of which are defined by the count request parameter) and provides a means to move backwards and forwards through these pages.
To retrieve cursored results, you initially pass a
cursor with a value of -1 to the endpoint. By default, an API endpoint that supports cursoring will assume -1 was passed as
cursor if you do not provide one. The response to a cursored request will contain
next_cursor_str. As is the case with most ID values in Twitter’s APIs, the
next_cursor is the cursor that you should send to the endpoint to receive the next batch of responses, and the
previous_cursor is the cursor that you should send to receive the previous batch. You will know that you have requested the last available page of results when the API responds with a next_cursor = 0.
The pseudocode to iterate over all responses from a cursored endpoint is:
cursor = -1
api_path = "https://api.twitter.com/1.1/endpoint.json?screen_name=targetUser"
url_with_cursor = api_path + "&cursor=" + cursor
response_dictionary = perform_http_get_request_for_url( url_with_cursor )
cursor = response_dictionary[ 'next_cursor' ]
while ( cursor != 0 )
Let’s see a real-life example of cursors in action. Consider the common scenario where we want to obtain the list of IDs who follow a user who has a large amount of followers. Instead of returning all IDs in one response set, the endpoint will instead return pages of results.Request
We now have a means to move forwards through our dataset, via the
next_cursor. To get the next batch of at least 5,000 results, perform the same request, but set the
cursor to the value of the
Notice that we now have a
next_cursor as well as a
previous_cursor. This means that we can now move back and forth through our results. Let's keep advancing through the data with the next cursor.
next_cursor is now 0, which indicates that there are no more remaing pages. We’ve now completed iterating over the accounts’ followers.