401 on rate limit, and can't get X-RateLimit-Class api-identified

tweepsmap
@tweepsmap TweepsMap

Hi guys,
We have an app that has been working for a long time, all of the sudden we started getting errors about a week ago.
The errors appear when there is a rate limit, the error returned now is:
401 Basic authentication is not supported
instead of rate limit error. We can deal with that as a special case I guess but it appear that the API is not returning the correct error result for rate limiting.

We also noticed that using oAuth, users/lookup (of the REST API 1.0) the rate limit is 150 instead of 350, X-RateLimit-Class is "api" and there is no X-Warning. The same oAuth call works perfectly when using a different REST API methods ex: users/show.

Any idea what happened in the last week or so? And why is api-identified no longer working for users/lookup?

Thanks

[Edited: the error is 401, not 403]

1 year 19 weeks ago

Replies

episod
@episod Taylor Singletary

The X-Warning thing has started to be phased out as our infrastructure changes.

Recently we've begun getting more strict with both HTTP and OAuth across the board. If you're seeing that your rate limit is 150, then that means that your OAuth is likely being rejected. Verify that your credentials are correct, that you're following the HTTP 1.1 and OAuth spec nearly to the letter (encoding all reserved characters, using all the appropriate HTTP headers like Host, Content-Type, and Content-Length as appropriate).

1 year 19 weeks ago
tweepsmap
@tweepsmap TweepsMap

Thanks Taylor for the reply.

I am pretty sure that the credentials are correct. Shouldn't oAuth be rejected for all end points if it was invalid, not just users/lookup?

Also, I am finding that the 150 limit is per OAuth token, so we are having multiple users exhausting their 150 limit at the same time, not 150 limit per server, which is strange.

1 year 19 weeks ago
episod
@episod Taylor Singletary

That is indeed strange.

users/lookup is a little special in that it uses frequently uses commas, which are a character that often is handled incorrectly in HTTP & OAuth implementations.

Can you share a request-and-response cycle with the exact URL you're executing, the HTTP headers you send, and the response you get back? For bonus points, if you know how to access it, could you also send a signature base string?

1 year 19 weeks ago
tweepsmap
@tweepsmap TweepsMap

Hi @episod,
Here is a curl dump, also signature base string is below it
curl --get "https://api.twitter.com/1/users/lookup.json" --data "user_id=108710952" --header "Authorization: OAuth oauth_consumer_key=\"zZdI96q9UbBOCQOpZbGlg\", oauth_nonce=\"3adc99975e3a083f2bd4465a413d37ed\", oauth_signature=\"9SyirYJb0dz8vi44d9xtL9Xxx9Q%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1354686775\", oauth_token=\"355123823-5M6bOOl4O6ORHSIz3DJtuyPtEf2Jq6UaxifPpi0f\", oauth_version=\"1.0\"" --verbose -k
* About to connect() to api.twitter.com port 443 (#0)
* Trying 199.16.156.104...
* connected
* Connected to api.twitter.com (199.16.156.104) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
* start date: 2012-05-02 00:00:00 GMT
* expire date: 2013-05-03 23:59:59 GMT
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA- G2
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET /1/users/lookup.json?user_id=108710952 HTTP/1.1
User-Agent: curl/7.28.1
Host: api.twitter.com
Accept: /
Authorization: OAuth oauth_consumer_key="zZdI96q9UbBOCQOpZbGlg", oauth_nonce="3adc99975e3a083f2bd4465a413d37ed", oauth_signature="9SyirYJb0dz8vi44d9xtL9Xxx9Q%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354686775", oauth_token="355123823-5M6bOOl4O6ORHSIz3DJtuyPtEf2Jq6UaxifPpi0f", oauth_version="1.0"

< HTTP/1.1 200 OK
< Date: Wed, 05 Dec 2012 05:54:17 GMT
< Status: 200 OK
< X-Runtime: 0.03396
< X-Transaction-Mask: a6183ffa5f8ca943ff1b53b5644ef114828f11c4
< X-MID: 74cb3b54f6c64e76bc9a61951d95d6533688cae1
< Expires: Tue, 31 Mar 1981 05:00:00 GMT
< X-Frame-Options: SAMEORIGIN
< Content-Length: 2120
< X-RateLimit-Class: api
< X-Access-Level: read-write
< X-Transaction: fb961d6bd69ce1f2
< Content-Type: application/json; charset=utf-8
< ETag: "559848478709074aa6c81518424a2e11"
< X-RateLimit-Remaining: 147
< Last-Modified: Wed, 05 Dec 2012 05:54:17 GMT
< Pragma: no-cache
< X-RateLimit-Limit: 150
< X-RateLimit-Reset: 1354690347
< Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
< Set-Cookie: k=10.40.21.112.1354686857349326; path=/; expires=Wed, 12-Dec-12 05:54:17 GMT; domain=.twitter.com
< Set-Cookie: guest_id=v1%3A135468685735476622; domain=.twitter.com; path=/; expires=Fri, 05-Dec-2014 17:54:17 GMT
< Set-Cookie: dnt=; domain=.twitter.com; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
< Set-Cookie: lang=en; path=/
< Set-Cookie: lang=en; path=/
< Set-Cookie: lang=en; path=/
< Set-Cookie: twid=u%3D355123823%7CCTmjWnSh5JiL2%2Bpyt8EI8M3Csgc%3D; domain=.twitter.com; path=/; secure
< Set-Cookie: twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCJYAoWk7ASIKZmxhc2hJQzonQWN0aW9uQ29u%250AdHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7ADoHaWQiJWE0%250ANDg3NTFhOWM2MGU5ZmFiNDNhNGFiMjlhMjdkMDIw--1e27630ba6daf7e1b2cfb9df42887186c94e8a57; domain=.twitter.com; path=/; HttpOnly
< Vary: Accept-Encoding
< Server: tfe
<
[{"id":108710952,"profile_sidebar_border_color":"99CC33","profile_image_url":"ht
tp:\/\/a0.twimg.com\/profile_images\/723426874\/newGreen2_normal.png","screen_na
me":"RentCompass","created_at":"Tue Jan 26 20:40:00 +0000 2010","following":true
,"default_profile":false,"profile_background_tile":false,"id_str":"108710952","p
rofile_sidebar_fill_color":"FE9800","utc_offset":-18000,"url":"http:\/\/www.Rent
Compass.com","name":"RentCompass","listed_count":20,"protected":false,"notificat
ions":false,"profile_background_color":"000000","contributors_enabled":false,"ti
me_zone":"Eastern Time (US & Canada)","profile_image_url_https":"https:\/\/si0.t
wimg.com\/profile_images\/723426874\/newGreen2_normal.png","profile_background_i
mage_url":"http:\/\/a0.twimg.com\/profile_background_images\/108879549\/twitter-
background.png","geo_enabled":false,"friends_count":1339,"location":"Canada","fo
llow_request_sent":false,"followers_count":1014,"statuses_count":527,"profile_li
nk_color":"0084B4","is_translator":false,"default_profile_image":false,"lang":"e
n","favourites_count":0,"profile_use_background_image":true,"profile_background
image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/108879549\
/twitter-background.png","profile_text_color":"333333","status":{"in_reply_to_us
er_id_str":null,"favorited":false,"possibly_sensitive":false,"contributors":null
,"id_str":"259072858901323776","coordinates":null,"geo":null,"created_at":"Thu O
ct 18 23:25:58 +0000 2012","retweet_count":0,"retweeted":false,"truncated":false
,"text":"Eugene E. Jones, Jr. (Gene Jones) \u00b7 Toronto Community new CEO spea
king \nGTAA town hall dinner. http:\/\/t.co\/TA8mzVdE","source":"\u003Ca href=\"
http:\/\/tapbots.com\/tweetbot\" rel=\"nofollow\"\u003ETweetbot for iOS\u003C\/a
\u003E","place":null,"in_reply_to_screen_name":null,"in_reply_to_user_id":null,"
in_reply_to_status_id":null,"id":259072858901323776,"in_reply_to_status_id_str":
null},"verified":false,"description":"The first Canadian rental listing service
on the iPhone, iPad & Android. Find houses and apartments for rent across Canada
. List your rental property for free."}]* Connection #0 to host api.twitter.com
left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

GET&https%3A%2F%2Fapi.twitter.com%2F1%2Fusers%2Flookup.json&oauth_consumer_key%3DZrcKya6D7FvhigbmXdD1w%26oauth_nonce%3Dc7933b9b37b699fd76324a38731047af%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1354684481%26oauth_token%3D368711188-vgrKvZ83dLPuBFPx7rNkfKlvN3X8G9zV9qLFI0Sy%26oauth_version%3D1.0%26user_id%3D242047518

Note that there is no X-Warning, but the limit is 150. When I put an invalid token it gives a 401

1 year 19 weeks ago
semifor
@semifor Marc Mims

Taylor, if it's an OAuth request problem, then Twitter itself has the same problem.

Using the OAuth tool at https://dev.twitterc.com/apps, the Twitter generated curl request results in exactly the same problem.

The original poster was getting 403 responses when the rate limit is reached. I'm getting 401 responses with error text "Basic authentication not supported".

  1.  curl -i --get 'https://api.twitter.com/1/users/lookup.json' --data 'screen_name=twitterapi%2Ctwitter' --header 'Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="9581ac08146df54401d80240275a609c", oauth_signature="2%2F7%2FAw4xOia75od1aZrsUfAdVQE%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354654202", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"' --verbose
  2. * About to connect() to api.twitter.com port 443 (#0)
  3. *   Trying 199.59.150.41...
  4. * connected
  5. * Connected to api.twitter.com (199.59.150.41) port 443 (#0)
  6. * SSLv3, TLS handshake, Client hello (1):
  7. * SSLv3, TLS handshake, Server hello (2):
  8. * SSLv3, TLS handshake, CERT (11):
  9. * SSLv3, TLS handshake, Server finished (14):
  10. * SSLv3, TLS handshake, Client key exchange (16):
  11. * SSLv3, TLS change cipher, Client hello (1):
  12. * SSLv3, TLS handshake, Finished (20):
  13. * SSLv3, TLS change cipher, Client hello (1):
  14. * SSLv3, TLS handshake, Finished (20):
  15. * SSL connection using RC4-SHA
  16. * Server certificate:
  17. *     subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
  18. *   start date: 2012-05-02 00:00:00 GMT
  19. *   expire date: 2013-05-03 23:59:59 GMT
  20. *      subjectAltName: api.twitter.com matched
  21. *   issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
  22. *   SSL certificate verify ok.
  23. > GET /1/users/lookup.json?screen_name=twitterapi%2Ctwitter HTTP/1.1
  24. > User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
  25. > Host: api.twitter.com
  26. > Accept: /
  27. > Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="9581ac08146df54401d80240275a609c", oauth_signature="2%2F7%2FAw4xOia75od1aZrsUfAdVQE%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354654202", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"
  28. > 
  29. < HTTP/1.1 200 OK
  30. HTTP/1.1 200 OK
  31. < Date: Tue, 04 Dec 2012 20:50:36 GMT
  32. Date: Tue, 04 Dec 2012 20:50:36 GMT
  33. < Status: 200 OK
  34. Status: 200 OK
  35. < X-RateLimit-Class: api
  36. X-RateLimit-Class: api
  37. < Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
  38. Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
  39. < X-RateLimit-Reset: 1354657207
  40. X-RateLimit-Reset: 1354657207
  41. < X-Access-Level: read-write-directmessages
  42. X-Access-Level: read-write-directmessages
  43. < Pragma: no-cache
  44. Pragma: no-cache
  45. < X-MID: 47ca871da6c7dee8fb495d2949a2903b6b25b1d5
  46. X-MID: 47ca871da6c7dee8fb495d2949a2903b6b25b1d5
  47. < X-RateLimit-Remaining: 148
  48. X-RateLimit-Remaining: 148
  49. < X-Transaction: 658c2438e694b392
  50. X-Transaction: 658c2438e694b392
  51. < Content-Length: 4787
  52. Content-Length: 4787
  53. < X-Transaction-Mask: a6183ffa5f8ca943ff1b53b5644ef1149d8456a6
  54. X-Transaction-Mask: a6183ffa5f8ca943ff1b53b5644ef1149d8456a6
  55. < Expires: Tue, 31 Mar 1981 05:00:00 GMT
  56. Expires: Tue, 31 Mar 1981 05:00:00 GMT
  57. < Content-Type: application/json; charset=utf-8
  58. Content-Type: application/json; charset=utf-8
  59. < Last-Modified: Tue, 04 Dec 2012 20:50:36 GMT
  60. Last-Modified: Tue, 04 Dec 2012 20:50:36 GMT
  61. < X-RateLimit-Limit: 150
  62. X-RateLimit-Limit: 150
  63. < ETag: "8e298a4c3c69427d2fc1ce2d7b5f8419"
  64. ETag: "8e298a4c3c69427d2fc1ce2d7b5f8419"
  65. < X-Runtime: 0.05091
  66. X-Runtime: 0.05091
  67. < X-Frame-Options: SAMEORIGIN
  68. X-Frame-Options: SAMEORIGIN
  69. < Set-Cookie: k=10.36.21.130.1354654236179201; path=/; expires=Tue, 11-Dec-12 20:50:36 GMT; domain=.twitter.com
  70. Set-Cookie: k=10.36.21.130.1354654236179201; path=/; expires=Tue, 11-Dec-12 20:50:36 GMT; domain=.twitter.com
  71. < Set-Cookie: guest_id=v1%3A135465423618446820; domain=.twitter.com; path=/; expires=Fri, 05-Dec-2014 08:50:36 GMT
  72. Set-Cookie: guest_id=v1%3A135465423618446820; domain=.twitter.com; path=/; expires=Fri, 05-Dec-2014 08:50:36 GMT
  73. < Set-Cookie: dnt=; domain=.twitter.com; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  74. Set-Cookie: dnt=; domain=.twitter.com; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  75. < Set-Cookie: lang=en; path=/
  76. Set-Cookie: lang=en; path=/
  77. < Set-Cookie: lang=en; path=/
  78. Set-Cookie: lang=en; path=/
  79. < Set-Cookie: lang=en; path=/
  80. Set-Cookie: lang=en; path=/
  81. < Set-Cookie: twid=u%3D14251368%7CI%2B9MSVqtt6RLI%2BrLyAPMMZ4KLQo%3D; domain=.twitter.com; path=/; secure
  82. Set-Cookie: twid=u%3D14251368%7CI%2B9MSVqtt6RLI%2BrLyAPMMZ4KLQo%3D; domain=.twitter.com; path=/; secure
  83. < Set-Cookie: _twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCCU%252Br2c7AToHaWQiJTg1NmY4Mjg3ODU1Nzk2%250AZGZlOTAwMjNhNTQ4ZWUzNjgwIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--39b9eb6570d6396e67dda1bd7f2e0fed79f0492f; domain=.twitter.com; path=/; HttpOnly
  84. Set-Cookie: _twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCCU%252Br2c7AToHaWQiJTg1NmY4Mjg3ODU1Nzk2%250AZGZlOTAwMjNhNTQ4ZWUzNjgwIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--39b9eb6570d6396e67dda1bd7f2e0fed79f0492f; domain=.twitter.com; path=/; HttpOnly
  85. < Vary: Accept-Encoding
  86. Vary: Accept-Encoding
  87. < Server: tfe
  88. Server: tfe
  89.  
  90. < 
  91. [{"id":783214,"listed_count":74223,"profile_background_color":"ACDED6","contributors_enabled":true,"time_zone":"Pacific Time (US & Canada)","verified":true,"following":true,"profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/657090062\/l1uqey5sy82r9ijhke1i.png","utc_offset":-28800,"geo_enabled":true,"friends_count":1142,"follow_request_sent":false,"followers_count":14938125,"statuses_count":1485,"profile_link_color":"038543","name":"Twitter","screen_name":"twitter","created_at":"Tue Feb 20 14:35:54 +0000 2007","default_profile_image":false,"protected":false,"favourites_count":20,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/657090062\/l1uqey5sy82r9ijhke1i.png","url":"http:\/\/blog.twitter.com\/","profile_text_color":"333333","is_translator":false,"notifications":false,"profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/783214\/1347405327","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2284174758\/v65oai7fxn47qv9nectx_normal.png","profile_sidebar_border_color":"EEEEEE","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2284174758\/v65oai7fxn47qv9nectx_normal.png","lang":"en","default_profile":false,"profile_background_tile":true,"location":"San Francisco, CA","id_str":"783214","profile_sidebar_fill_color":"F6F6F6","status":{"retweet_count":132,"possibly_sensitive":false,"truncated":false,"retweeted_status":{"retweet_count":132,"possibly_sensitive":false,"truncated":false,"coordinates":null,"geo":null,"created_at":"Mon Dec 03 18:08:07 +0000 2012","in_reply_to_screen_name":null,"retweeted":false,"in_reply_to_status_id_str":null,"contributors":[17033908],"in_reply_to_user_id":null,"in_reply_to_status_id":null,"text":"Today at 2pm ET President Obama will be tweeting from @whitehouse to answer #my2k questions about tax cuts. https:\/\/t.co\/A1FN2h1j","in_reply_to_user_id_str":null,"id_str":"275662716327309313","source":"web","place":null,"id":275662716327309313,"favorited":false},"coordinates":null,"geo":null,"created_at":"Mon Dec 03 18:19:11 +0000 2012","in_reply_to_screen_name":null,"retweeted":false,"in_reply_to_status_id_str":null,"contributors":null,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"text":"RT @gov: Today at 2pm ET President Obama will be tweeting from @whitehouse to answer #my2k questions about tax cuts. https:\/\/t.co\/A1FN2h1j","in_reply_to_user_id_str":null,"id_str":"275665498044243968","source":"web","place":null,"id":275665498044243968,"favorited":false},"description":"Your official source for news, updates and tips from Twitter, Inc."},{"id":6253282,"listed_count":11090,"is_translator":false,"profile_background_color":"C0DEED","contributors_enabled":true,"time_zone":"Pacific Time (US & Canada)","verified":true,"following":true,"profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/656927849\/miyt9dpjz77sc0w3d4vj.png","utc_offset":-28800,"geo_enabled":true,"friends_count":31,"follow_request_sent":false,"followers_count":1358101,"statuses_count":3360,"profile_link_color":"0084B4","name":"Twitter API","screen_name":"twitterapi","created_at":"Wed May 23 06:01:13 +0000 2007","default_profile_image":false,"protected":false,"favourites_count":25,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/656927849\/miyt9dpjz77sc0w3d4vj.png","url":"http:\/\/dev.twitter.com","profile_text_color":"333333","notifications":false,"profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/6253282\/1347394302","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2284174872\/7df3h38zabcvjylnyfe3_normal.png","profile_sidebar_border_color":"C0DEED","lang":"en","default_profile":false,"profile_background_tile":true,"location":"San Francisco, CA","id_str":"6253282","profile_sidebar_fill_color":"DDEEF6","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2284174872\/7df3h38zabcvjylnyfe3_normal.png","status":{"in_reply_to_user_id_str":null,"retwee* Connection #0 to host api.twitter.com left intact
  92. t_count":39,"truncated":false,"coordinates":null,"geo":null,"created_at":"Mon Dec 03 21:53:08 +0000 2012","in_reply_to_screen_name":null,"possibly_sensitive":false,"retweeted":false,"contributors":[7588892],"in_reply_to_user_id":null,"in_reply_to_status_id":null,"text":"We now have a page which tracks upcoming and recent changes to the platform: https:\/\/t.co\/3gMjdnBp ^ARK","id_str":"275719344317685760","source":"web","in_reply_to_status_id_str":null,"place":null,"id":275719344317685760,"favorited":false},"description":"The Real Twitter API. I tweet about API changes, service issues and happily answer questions about Twitter and our API. Don't get an answer? It's on my website."}]* Closing connection #0
  93. * SSLv3, TLS alert, Client hello (1):
1 year 19 weeks ago
episod
@episod Taylor Singletary

Are you able to make calls to account/verify_credentials with the same access token? v1's users/lookup doesn't require auth and will try to satisfy your request if you're using an invalid access token. The OAuth tool on this site doesn't care (and doesn't check) whether your token is valid or not before it generates a request, it just follows a recipe. If you've changed the access token representing your own relationship with your application at some point, it's easy to have invalidated the "cached" token that might be used in that function.

1 year 19 weeks ago
semifor
@semifor Marc Mims

Yes, verify_credentials succeeds.

This problem first appeared in our application logs an 2012/11/29 09:41:51. The prior call to /users/lookup at 09:06:02 had a rate limit of 350.

I haven't tested all endpoints, but I know /users/lookup and /statuses/retweets have this problem. And I know other endpoints, like /users/show do not.

1 year 19 weeks ago
tweepsmap
@tweepsmap TweepsMap

Thanks Marc for sharing request/response. I haven't had a chance yet to grab a trace myself.
And yes, the problem started happening on the 29th, not sure the exact time, we only have few rate limited calls per day.

@episode, the tokens we are using are not cached in any way, they are generated using the full OAuth cycle request/access tokens.
Also, we are getting 401, not 403... that was a typo (now corrected)

1 year 19 weeks ago
semifor
@semifor Marc Mims

@episod are you able to confirm this, yet? It's still a problem on our end.

Should be easy to confirm with the OAuth tool at https://dev.twitter.com/apps.

1 year 18 weeks ago
episod
@episod Taylor Singletary

I'm wholly unable to reproduce as of yet. I have some questions out to internal teams about what could potentially cause this besides an auth failure. Your request and response cycle look like what I'd expect from an OAuth request to this particular endpoint with an expired access token. If you're sure you're able to use account/verify_credentials with this exact same access token, then there's some other quirk going on here. Have you tried the v1.1 version of this method?

1 year 18 weeks ago
semifor
@semifor Marc Mims

Yes, I can call verify_cedentials. Did so with the OAuth tool and curl:

  1. curl --get 'https://api.twitter.com/1/account/verify_credentials.json' --header 'Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="8596853ff9d010e4d3fc31108b8e8eaf", oauth_signature="ABAW3nwmB%2FGjG9B0uU20tVbCFfc%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354745111", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"' --verbose

Output:

  1. * About to connect() to api.twitter.com port 443 (#0)
  2. *   Trying 199.59.150.9...
  3.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  4.                                  Dload  Upload   Total   Spent    Left  Speed
  5.  
  6.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* connected
  7. * Connected to api.twitter.com (199.59.150.9) port 443 (#0)
  8. * SSLv3, TLS handshake, Client hello (1):
  9. } [data not shown]
  10. * SSLv3, TLS handshake, Server hello (2):
  11. { [data not shown]
  12. * SSLv3, TLS handshake, CERT (11):
  13. { [data not shown]
  14. * SSLv3, TLS handshake, Server finished (14):
  15. { [data not shown]
  16. * SSLv3, TLS handshake, Client key exchange (16):
  17. } [data not shown]
  18. * SSLv3, TLS change cipher, Client hello (1):
  19. } [data not shown]
  20. * SSLv3, TLS handshake, Finished (20):
  21. } [data not shown]
  22. * SSLv3, TLS change cipher, Client hello (1):
  23. { [data not shown]
  24. * SSLv3, TLS handshake, Finished (20):
  25. { [data not shown]
  26. * SSL connection using RC4-SHA
  27. * Server certificate:
  28. *      subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
  29. *   start date: 2012-05-02 00:00:00 GMT
  30. *   expire date: 2013-05-03 23:59:59 GMT
  31. *      subjectAltName: api.twitter.com matched
  32. *   issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
  33. *   SSL certificate verify ok.
  34. > GET /1/account/verify_credentials.json HTTP/1.1
  35. > User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
  36. > Host: api.twitter.com
  37. > Accept: /
  38. > Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="8596853ff9d010e4d3fc31108b8e8eaf", oauth_signature="ABAW3nwmB%2FGjG9B0uU20tVbCFfc%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354745111", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"
  39. > 
  40.  
  41.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
  42. < Date: Wed, 05 Dec 2012 22:05:51 GMT
  43. < Status: 200 OK
  44. < X-RateLimit-Class: api
  45. < X-Access-Level: read-write-directmessages
  46. < Content-Length: 2120
  47. < Pragma: no-cache
  48. < X-RateLimit-Remaining: 148
  49. < Expires: Tue, 31 Mar 1981 05:00:00 GMT
  50. < Content-Type: application/json; charset=utf-8
  51. < ETag: "756c6dd9e23b4f95bb534fee1a441043"
  52. < X-Runtime: 0.04028
  53. < X-RateLimit-Limit: 150
  54. < X-Frame-Options: SAMEORIGIN
  55. < Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
  56. < X-MID: 4baa5616c3c6310e5a5b6183c3e0998c34983ea1
  57. < X-Transaction-Mask: a6183ffa5f8ca943ff1b53b5644ef114828f11c4
  58. < Last-Modified: Wed, 05 Dec 2012 22:05:51 GMT
  59. < X-RateLimit-Reset: 1354745607
  60. < X-Transaction: dc4b25ba2b023559
  61. < Set-Cookie: k=10.36.59.101.1354745151169069; path=/; expires=Wed, 12-Dec-12 22:05:51 GMT; domain=.twitter.com
  62. < Set-Cookie: guest_id=v1%3A135474515117373498; domain=.twitter.com; path=/; expires=Sat, 06-Dec-2014 10:05:51 GMT
  63. < Set-Cookie: dnt=; domain=.twitter.com; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  64. < Set-Cookie: lang=en; path=/
  65. < Set-Cookie: lang=en; path=/
  66. < Set-Cookie: lang=en; path=/
  67. < Set-Cookie: twid=u%3D14251368%7CI%2B9MSVqtt6RLI%2BrLyAPMMZ4KLQo%3D; domain=.twitter.com; path=/; secure
  68. < Set-Cookie: _twitter_sess=BAh7CSIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCNV%252BGm07AToHaWQiJWFh%250AM2UwOTFmZTE5YWY4MzdjMTYzZThiNWI3NDRlNTUxOgxjc3JmX2lkIiVkYmQ5%250AZGYxYTQzMTdhZjhkNjQ3MTk1NDc0ZDhkYzdmYw%253D%253D--49ff094b27c99deb66e8c2c5cd480c988cd5a746; domain=.twitter.com; path=/; HttpOnly
  69. < Vary: Accept-Encoding
  70. < Server: tfe
  71. < 
  72. { [data not shown]
  73.  
  74. 100  2120  100  2120    0     0   2071      0  0:00:01  0:00:01 --:--:--  7386
  75. * Connection #0 to host api.twitter.com left intact
  76. {"id":14251368,"profile_sidebar_border_color":"C0DEED","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/57590298\/profile_normal.jpg","screen_name":"semifor","created_at":"Sat Mar 29 14:55:25 +0000 2008","following":false,"listed_count":49,"profile_background_tile":false,"id_str":"14251368","profile_sidebar_fill_color":"DDEEF6","utc_offset":-28800,"url":"http:\/\/blog.questright.com\/","name":"Marc Mims","friends_count":342,"protected":false,"notifications":false,"statuses_count":2193,"profile_background_color":"C0DEED","contributors_enabled":false,"time_zone":"Pacific Time (US & Canada)","default_profile_image":false,"is_translator":false,"profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","geo_enabled":true,"location":"Spokane Valley, WA, USA","favourites_count":555,"follow_request_sent":false,"followers_count":647,"profile_link_color":"0084B4","lang":"en","profile_use_background_image":true,"profile_text_color":"333333","status":{"possibly_sensitive_editable":true,"favorited":false,"contributors":null,"id_str":"276433024105734145","in_reply_to_status_id_str":null,"coordinates":null,"geo":null,"created_at":"Wed Dec 05 21:09:03 +0000 2012","in_reply_to_user_id_str":null,"retweet_count":0,"retweeted":false,"truncated":false,"text":"Booked my first ever #perl conference http:\/\/t.co\/jSQ51fJ7 where I plan to heckle @perigrin in the #Moose class.","source":"\u003Ca href=\"http:\/\/itunes.apple.com\/us\/app\/twitter\/id409789998?mt=12\" rel=\"nofollow\"\u003ETwitter for Mac\u003C\/a\u003E","possibly_sensitive":false,"place":null,"in_reply_to_screen_name":null,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"id":276433024105734145},"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/57590298\/profile_normal.jpg","verified":false,"default_profile":true,"description":"Perl hacker, Software Engineer at SEOmoz, author of Net::Twitter - Perl's Twitter API lib, Linux enthusiast, avid cyclist. Director, Board of Directors @BikeWA"}* Closing connection #0
  77. * SSLv3, TLS alert, Client hello (1):
  78. } [data not shown]

Followed by a call to /users/lookup:

  1. curl --get 'https://api.twitter.com/1/users/lookup.json' --data 'screen_name=twitterapi%2Cepisod' --header 'Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="e6f3dcb2e4e0e01efbff5d72d8de4296", oauth_signature="CwcitDCqbuh9C6oSeUcvN%2Fd5VX4%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354745424", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"' --verbose
  1. * About to connect() to api.twitter.com port 443 (#0)
  2. *   Trying 199.59.148.20...
  3.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  4.                                  Dload  Upload   Total   Spent    Left  Speed
  5.  
  6.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* connected
  7. * Connected to api.twitter.com (199.59.148.20) port 443 (#0)
  8. * SSLv3, TLS handshake, Client hello (1):
  9. } [data not shown]
  10. * SSLv3, TLS handshake, Server hello (2):
  11. { [data not shown]
  12. * SSLv3, TLS handshake, CERT (11):
  13. { [data not shown]
  14. * SSLv3, TLS handshake, Server finished (14):
  15. { [data not shown]
  16. * SSLv3, TLS handshake, Client key exchange (16):
  17. } [data not shown]
  18. * SSLv3, TLS change cipher, Client hello (1):
  19. } [data not shown]
  20. * SSLv3, TLS handshake, Finished (20):
  21. } [data not shown]
  22. * SSLv3, TLS change cipher, Client hello (1):
  23. { [data not shown]
  24. * SSLv3, TLS handshake, Finished (20):
  25. { [data not shown]
  26. * SSL connection using RC4-SHA
  27. * Server certificate:
  28. *    subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=Twitter Security; CN=api.twitter.com
  29. *   start date: 2012-05-02 00:00:00 GMT
  30. *   expire date: 2013-05-03 23:59:59 GMT
  31. *      subjectAltName: api.twitter.com matched
  32. *   issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
  33. *   SSL certificate verify ok.
  34. > GET /1/users/lookup.json?screen_name=twitterapi%2Cepisod HTTP/1.1
  35. > User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
  36. > Host: api.twitter.com
  37. > Accept: /
  38. > Authorization: OAuth oauth_consumer_key="agdvsZFSuZP0AqFJzOJtgA", oauth_nonce="e6f3dcb2e4e0e01efbff5d72d8de4296", oauth_signature="CwcitDCqbuh9C6oSeUcvN%2Fd5VX4%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1354745424", oauth_token="14251368-vZJDCvYgS91pMGFLv2djgzNOUiISyGthxUnj2o9k", oauth_version="1.0"
  39. > 
  40. < HTTP/1.1 200 OK
  41. < Date: Wed, 05 Dec 2012 22:10:58 GMT
  42. < Status: 200 OK
  43. < X-RateLimit-Limit: 150
  44. < Pragma: no-cache
  45. < ETag: "a83e4b97ee94df78673cb58534969cd0"
  46. < X-MID: 99ec4c04917ecaa76a816f333e6abd3ffe644722
  47. < Content-Type: application/json; charset=utf-8
  48. < X-Transaction: 8901047c535526a6
  49. < X-Transaction-Mask: a6183ffa5f8ca943ff1b53b5644ef114828f11c4
  50. < Expires: Tue, 31 Mar 1981 05:00:00 GMT
  51. < X-RateLimit-Remaining: 147
  52. < X-Frame-Options: SAMEORIGIN
  53. < Content-Length: 4455
  54. < X-Runtime: 0.06314
  55. < Last-Modified: Wed, 05 Dec 2012 22:10:58 GMT
  56. < X-RateLimit-Class: api
  57. < Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
  58. < X-Access-Level: read-write-directmessages
  59. < X-RateLimit-Reset: 1354745607
  60. < Set-Cookie: k=10.36.54.126.1354745458814755; path=/; expires=Wed, 12-Dec-12 22:10:58 GMT; domain=.twitter.com
  61. < Set-Cookie: guest_id=v1%3A135474545882096368; domain=.twitter.com; path=/; expires=Sat, 06-Dec-2014 10:10:58 GMT
  62. < Set-Cookie: dnt=; domain=.twitter.com; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  63. < Set-Cookie: lang=en; path=/
  64. < Set-Cookie: lang=en; path=/
  65. < Set-Cookie: lang=en; path=/
  66. < Set-Cookie: twid=u%3D14251368%7CI%2B9MSVqtt6RLI%2BrLyAPMMZ4KLQo%3D; domain=.twitter.com; path=/; secure
  67. < Set-Cookie: _twitter_sess=BAh7CCIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNo%250ASGFzaHsABjoKQHVzZWR7ADoPY3JlYXRlZF9hdGwrCJIwH207AToHaWQiJTFj%250AYmQyNWM0YWE4MzdmNTc1MjI3ODc0ZDYwZTVkM2M0--3155c347a2ac9090284a4a89896c91b741f9a46d; domain=.twitter.com; path=/; HttpOnly
  68. < Vary: Accept-Encoding
  69. < Server: tfe
  70. < 
  71. { [data not shown]
  72.  
  73. 100  4455  100  4455    0     0  12198      0 --:--:-- --:--:-- --:--:-- 15967
  74. * Connection #0 to host api.twitter.com left intact
  75. [{"profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/6253282\/1347394302","id":6253282,"profile_sidebar_border_color":"C0DEED","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2284174872\/7df3h38zabcvjylnyfe3_normal.png","screen_name":"twitterapi","created_at":"Wed May 23 06:01:13 +0000 2007","following":true,"default_profile":false,"profile_background_tile":true,"id_str":"6253282","profile_sidebar_fill_color":"DDEEF6","utc_offset":-28800,"url":"http:\/\/dev.twitter.com","name":"Twitter API","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2284174872\/7df3h38zabcvjylnyfe3_normal.png","listed_count":11093,"protected":false,"notifications":false,"profile_background_color":"C0DEED","contributors_enabled":true,"is_translator":false,"time_zone":"Pacific Time (US & Canada)","profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/656927849\/miyt9dpjz77sc0w3d4vj.png","geo_enabled":true,"friends_count":31,"location":"San Francisco, CA","follow_request_sent":false,"followers_count":1360045,"statuses_count":3361,"profile_link_color":"0084B4","default_profile_image":false,"lang":"en","favourites_count":25,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/656927849\/miyt9dpjz77sc0w3d4vj.png","profile_text_color":"333333","status":{"possibly_sensitive":false,"favorited":false,"contributors":[14927800],"in_reply_to_status_id_str":null,"id_str":"276413601651245056","coordinates":null,"geo":null,"created_at":"Wed Dec 05 19:51:52 +0000 2012","in_reply_to_user_id_str":null,"retweet_count":46,"retweeted":false,"truncated":false,"text":"\"@urbandictionary grows daily followers 6.5x by implementing the Follow Button\" - https:\/\/t.co\/oisy5Rlu   ^JC","source":"web","place":null,"in_reply_to_screen_name":null,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"id":276413601651245056},"verified":true,"description":"The Real Twitter API. I tweet about API changes, service issues and happily answer questions about Twitter and our API. Don't get an answer? It's on my website."},{"profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/819797\/1354385404","id":819797,"profile_sidebar_border_color":"000000","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2919709289\/4700bbd4e1659797a812ed114b134284_normal.png","screen_name":"episod","created_at":"Wed Mar 07 22:23:19 +0000 2007","following":false,"default_profile":false,"profile_background_tile":false,"id_str":"819797","profile_sidebar_fill_color":"FBFBFB","utc_offset":-28800,"url":"http:\/\/soundcloud.com\/reality-technician","name":"Taylor Singletary","listed_count":356,"protected":false,"notifications":false,"profile_background_color":"000000","contributors_enabled":true,"time_zone":"Pacific Time (US & Canada)","profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/686878932\/6447abb9f83c76fb4fbd68e626c6c8c1.png","geo_enabled":true,"friends_count":5606,"location":"San Francisco, CA","is_translator":false,"follow_request_sent":false,"followers_count":8050,"statuses_count":18923,"profile_link_color":"941594","default_profile_image":false,"lang":"en","favourites_count":17710,"profile_use_background_image":false,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/686878932\/6447abb9f83c76fb4fbd68e626c6c8c1.png","profile_text_color":"D20909","status":{"favorited":false,"contributors":null,"id_str":"276430883160985600","coordinates":null,"geo":null,"created_at":"Wed Dec 05 21:00:33 +0000 2012","in_reply_to_status_id_str":"276428345959387136","retweet_count":0,"in_reply_to_user_id_str":"564331210","retweeted":false,"truncated":false,"text":"@ttessirak it's true, they are. but if you go far enough in any direction, you'll eventually find a cow. maybe a wife. a love story.","source":"\u003Ca href=\"http:\/\/sites.google.com\/site\/yorufukurou\/\" rel=\"nofollow\"\u003EYoruFukurou\u003C\/a\u003E","place":null,"in_reply_to_screen_name":"ttessirak","in_reply_to_user_id":564331210,"in_reply_to_status_id":276428345959387136,"id":276430883160985600},"verified":false,"description":"Reality Technician, Twitter API team, synth enthusiast. An almost excellent adventure in timelines. Narrator in question. I undo.","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2919709289\/4700bbd4e1659797a812ed114b134284_normal.png"}]* Closing connection #0
  76. * SSLv3, TLS alert, Client hello (1):
  77. } [data not shown]

I get the some result with every app/user I try and with every Twitter API library I try.

v1.1 works as expected. But we really need v1 to function as documented while we make the switch.

1 year 18 weeks ago
tweepsmap
@tweepsmap TweepsMap

Actually when you query with an expired token you also get:
< X-Warning: Invalid OAuth credentials detected

1 year 18 weeks ago
tweepsmap
@tweepsmap TweepsMap

BTW, account/verify_credentials works fine for me.
v1.1 of the API also works fine, I get 180 which is correct. But here is the weird part once I use my token for v1.1 I start to get X-Warning for that token on v1, when I create a new token for the X-Warning to go away, but still I can't get X-RateLimit-Class api-identified

Also, if I use a new token on v1.0 I can't use it on v1.1 anymore I get error "Could not authenticate you"

The problem is very easy to reproduce, no coding required:
1. Create A new test app
2. Create access token from the app page
3. Generate OAuth signature from OAuth tool. Use https://api.twitter.com/1/users/lookup.json and user_id=108710952 for url/data
4. run the cURL command from the OAuth tool

1 year 18 weeks ago
tweepsmap
@tweepsmap TweepsMap

Hi @episod,
Any updates from the dev team? Where you able to reproduce with the steps provided above either by me, or @semifor ?

Thanks

1 year 18 weeks ago
episod
@episod Taylor Singletary

Hey everyone,

Still unable to reproduce but I'm continuing to have engineers look into this.

1 year 18 weeks ago
semifor
@semifor Marc Mims

@episod can you post your OAuth tool curl results? It's surprising and frustrating that you're unable to reproduce it on your end. It's reproducible on our end with every application an user we've tried.

Happy to offer any information or assistance I can to help.

1 year 18 weeks ago
tweepsmap
@tweepsmap TweepsMap

Another dev complaining about the same thing:
dev.twitter.com/discussions/13241

1 year 17 weeks ago
creatorul
@creatorul Daniel

Same problem here, getting 150 on authenticated calls on v 1. Probably twitter is trying to push everybody to v1.1 with these weird limitations

1 year 14 weeks ago
creatorul
@creatorul Daniel

any solutions to this ?

1 year 14 weeks ago
tweepsmap
@tweepsmap TweepsMap

Given that they haven't fixed it 6 weeks after it broke, I don't think we have many options:
1. Migrate to 1.1
2. Ignore the rate limit reported by the rate limit API and just handle the errors once the API rate limit gets exceeded.

1 year 14 weeks ago
semifor
@semifor Marc Mims

I'm really disappointed Twitter hasn't even acknowledged the problem, let alone fixed it. :-/

1 year 14 weeks ago
JustinSPowers
@JustinSPowers Justin Powers

I am seeing this issue as well. Making calls to /1/account/verify_credentials.json work as expected, as do requests to endpoints that require authentication. Calls to methods that require authentication return headers that indicate the proper rate limiting. However, calls to /1/friendships/show.json are being rate limited by IP, even when authenticated.

1 year 11 weeks ago