When trying to POST to update_with_media a status 500 is returned

danpalmer
@danpalmer Dan Palmer

Update: I have found what the issue is. The update_with_media API does not conform to the RFC specification for multipart/form-data correctly.

The headers should finish with a blank line, and the first boundary of the form-data includes another CLRF as a part of it. This means 2 blank lines (as shown in my sample request below). It is this extra CLRF, that is required by the specification, that causes the error 500 response from the API.

The specification also says that any data between the headers and multipart/form-data boundary should be ignored, but including data here will cause the same issues with the API.

This is therefore not a client issue and I believe it should be resolved with the service instead. The simple change of allowing the extra (required) CLRF as a part of the first boundary could possibly fix many clients that using libraries for the construction of their requests.

RFC specification for multipart/form-data: http://tools.ietf.org/html/rfc2046#section-5.1.1

  1. POST /1/statuses/update_with_media.xml HTTP/1.1
  2. Host: upload.twitter.com
  3. User-Agent: Analog/25 CFNetwork/520.0.13 Darwin/11.1.0 (x86_64) (MacBookPro8%2C2)
  4. Content-Length: 596419
  5. Accept: /
  6. Content-Type: multipart/form-data; boundary=D71D0913C8234CF4B262FB83B681ADC0473670002680C2317EDFB
  7. Authorization: OAuth oauth_signature="ZDvmnGXq7ntJf%2F5yWaG2yWNSYr8%3D",oauth_nonce="14AC0E17B1FA4376A4347A6C8E33C799473670002680C232393FA",oauth_timestamp="1315498721",oauth_consumer_key="La8awocq7tSBJgYtYiaDw",realm="",oauth_token="369435172-rf4VCgPYnYo5OVr8gj71Hq3CDOQBLAFqePhkPaQB",oauth_version="1.0",oauth_signature_method="HMAC-SHA1"
  8. Accept-Language: en-us
  9. Accept-Encoding: gzip, deflate
  10. Cookie: __utma=4XXXXXXX8.15XXXXXXX.XXXXXX46.1XXXXXXXX1.1XXXXXXX.58; __utmb=43838368.1.10.1315497261; __utmv=43838368.lang%3A%20en; __utmz=43838368.1315497261.58.58.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=%22missing%20or%20invalid%20url%20parameter%22; twll=l%3D1315469481; k=80.4.205.162.1315224193186587; guest_id=v1%3AXXXXXXX14671096
  11. Connection: keep-alive
  12.  
  13.  
  14. --D71D0913C8234CF4B262FB83B681ADC0473670002680C2317EDFB
  15. Content-Disposition: form-data; name="status"
  16. Content-Transfer-Encoding: binary
  17. Content-Type: text/plain; charset=utf-8
  18.  
  19. This is my tweet text
  20. --D71D0913C8234CF4B262FB83B681ADC0473670002680C2317EDFB
  21. Content-Disposition: form-data; name="media[]"; filename="DSC03644.jpg"
  22. Content-Transfer-Encoding: binary
  23. Content-Type: image/jpeg
  24.  
  25. Image Data
  26. --D71D0913C8234CF4B262FB83B681ADC0473670002680C2317EDFB--
2 years 32 weeks ago

Replies

kurrik
@kurrik Arne Roomann-Kurrik

It looks like there was an elevated number of 500s coming from media uploads over the weekend, which should be resolved now. Are you able to submit photos at this time?

2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

I started trying on Thursday and have been trying since. Haven't yet submitted a single image successfully with my code. Obviously something is wrong with it, but the 500 still indicates a problem on the Twitter end too.

2 years 32 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

There are certain cases where the API will 500 for a malformed POST body, so that's the first place I would double check (particularly around too few/many \r\n lines). Generally mime multipart messages can be pretty tricky to get absolutely correct if you're writing an implementation by hand (if you're using a library, please let me know). I'll take a deeper look at your request though.

2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

Using a private library for constructing requests. The library was written to conform exactly to the RFC spec. From what I can tell, the POST body it is generating is correct according to the specification.

Are there any things which the API doesn't like which we could try removing? I could make the POST exactly match what (for example) Tweetbot (only client I know of that uses this API) does. However it would require modifying the library quite significantly or 'special casing' Twitter in some way which we really don't want to do.

2 years 32 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

I've been able to successfully post a media upload using the library referenced in the announcement blogpost: https://dev.twitter.com/discussions/1059 which uses curl's multipart implementation. Unfortunately, this means that I can't do much to try and recreate your request with this approach (curl handles setting all of the multipart boundaries and such). As a next step, I'll try implementing the multipart request building by hand in order to try recreating as much of your request as I can.

It did occur to me that you may be signing your status parameter as part of the OAuth signature, which you shouldn't do. That shouldn't return a 500 error, but it's worth double checking.

Until I can repro, here's a dump of my successful request, which may be of help. One notable difference is that there is only one \r\n between the last header and the first part of the message.

  1. POST /1/statuses/update_with_media.json HTTP/1.1
  2. User-Agent: themattharris' HTTP Client
  3. Host: upload.twitter.com
  4. Accept: /
  5. Authorization: OAuth oauth_consumer_key="xvz1evFS4wEEPTGEFPHBog", oauth_nonce="7f9f12fe2ebed42396f8b723a66ae69a", oauth_signature="8xdNLIzkDhKJDd4QPQzwgl%2B9FWI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1315870735", oauth_token="370773112-s8hjUxMHerwdUjFKNFlYZaUZboIaVXPI4OvdItqH", oauth_version="1.0"
  6. Content-Length: 221274
  7. Content-Type: multipart/form-data; boundary=----------------------------d67b3e07c45e
  8.  
  9. ------------------------------d67b3e07c45e
  10. Content-Disposition: form-data; name="media[]"; filename="./image.jpg"
  11. Content-Type: application/octet-stream
  12.  
  13. â ...bunch of image data...
  14. ê!∂∑fiZ¶≤=xÇú∏°€dEhÏ*f®ò^q•Àƒ#ø,yƒE-««1ú+#A‚—Ü8"¿÷6iµˇqzûºÄ5IENDÆBÇ
  15. ------------------------------d67b3e07c45e
  16. Content-Disposition: form-data; name="status"
  17.  
  18. This is a status update
  19. ------------------------------d67b3e07c45e--
2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

Thanks, I will try removing the 'extra' \r\n from the body of my requests, although that is valid under the specification.

I am not signing the status, I know that. I have checked and the way the library I am using works it cannot do that, partly because the OAuth spec says that multipart/form-data should be ignored for signing.

2 years 32 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

Would it be possible to host a raw dump of a failing request somewhere?

2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

I sent an email to @jkalucki at twitter-failing-photos at twitter.com. It details the fix. It is a server side issue.

2 years 31 weeks ago
coanor
@coanor coanor

While the status is in Unicode, for example, the Chinese "通", its UTF-8 code is "%E9%80%9A", and the POST body looks like that:

------------------------------d67b3e07c45e
Content-Disposition: form-data; name="status"

%E9%80%9A
------------------------------d67b3e07c45e--

But the status still display as "%E9%80%9A" on Twitter, how to make it display as Chinese "通"?

2 years 6 weeks ago
cmpinfy
@cmpinfy CM Sample

I am trying using Rest Client in Mozilla.

https://upload.twitter.com/1/statuses/update_with_media.json?status=test&&media[]="C:/Pictures/images.jpg;type=image/jpg;filename=flicker images"

When posting the above request I am getting Could not authenticate with OAuth error :(

Although i set OAuth details properly.
https://api.twitter.com/1/statuses/update.json?status=Test_Status --> This is executing fine.

Can you tell what will be the problem in above?

2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

Your problem does not appear to be relevant to this discussion, it might be more appropriate to create your own discussion for it, but here are a few things to try:

  1. You must provide the OAuth details in EVERY request to an authenticated API such as this.
  2. If you are going to POST the data in the URL encoded format you MUST use all of the fields to generate the OAuth signature, everything must be signed. If you used multipart/form-data as I mentioned in a comment to your other post and which is require by the API then you would not need to do this.
  3. Posting the filename in the media[] field will not work, you should post the actual file.
2 years 32 weeks ago
jkalucki
@jkalucki John Kalucki

Can you send the time of one of these failures, in UTC, and the account name that caused the failure? Either reply here or mail to: twitter-failing-photos at twitter dot com.

2 years 32 weeks ago
cpatil
@cpatil Chetan Patil

The 500 error is likely raised while parsing the post body in the Rails core.
Please confirm the content_length header takes the correct multibyte string length into account in the post. We have seen such issues before.
If you provide the request time / account info, I can look through the logs.

2 years 32 weeks ago
danpalmer
@danpalmer Dan Palmer

I have sent an email to @jkalucki so I guess he is on the case. The details of the solution are in the email, but as our request was, to our knowledge, compliant with the specification, we do not want to change the code on our end and instead see it as a server-side issue.

Edit: Just for anyone coming here who wants to know the result of the discussions. As of when I last tried (quite a while ago now) the reason for my 500 errors was that the Twitter API doesn't support the RFC multipart/form-data specification fully. The spec says that there should be 2 blank lines (3x CLRF) between the end of the headers and the beginning of the first boundary, but Twitter would see the 3rd as an early message termination and therefore generate an error. By creating a special case in our framework to circumvent Twitter's problem and only produce 2 CLRFs, we solved the problem.

2 years 17 weeks ago
pradeep_dey
@pradeep_dey Pradeep Dey

Dan Palmer & Arne Roomann-Kurrik

Can you guys please help me out, I am still getting response 500 despite trying options out!!

using java, below are the headers and body contents i am posting, if you can throw some light on this, it would be great...
Req Headers: {Authorization=OAuth oauth_signature="jzin%2BhgnpnsdfghPTgWG2dmJFpA%3D", oauth_version="1.0", oauth_nonce="1524235035", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="MLqudfgJtshcKozhEoLFuQ", oauth_token="443552025-nyJZBnl45aUPORS0mhOO5fmjED9Ot2BsQPOAuLw", oauth_timestamp="1324978284", Content-Length=13, Content-Type=multipart/form-data; boundary=------------PRABOUNDARY2011}
Body Params: {}
Verb: POST
Body Contents:

Content-Disposition:form-data;name="media[]"; filename="abc.png"
Content-Type:application/octet-stream

PNG data in bytes
--------------PRABOUNDARY2011
Content-Disposition:form-data; name="status"

test321
--------------PRABOUNDARY2011--

2 years 17 weeks ago
danpalmer
@danpalmer Dan Palmer

This has quite a few issues with it. Your headers should be clear key/value pairs with no lines separating them like this:

POST /1/statuses/update_with_media.json HTTP/1.1
Host: upload.twitter.com
Accept: /
Content-type: multipart/form-data; boundary= gc0p4Jq0M2Yt08jU534c0p

You will also need the OAuth headers. After the headers, you should have a blank line and then the first boundary, multipart section headers, blank line, data, boundary, and so on like that until you include the closing boundary. There are some good examples from the RFC on page 8: http://www.ietf.org/rfc/rfc1867.txt.

2 years 17 weeks ago
MediatekInc
@MediatekInc Mediatek

By using update_with_media, I got a problem that is the image need to be encrypted by SSL? My question is post here:
http://stackoverflow.com/questions/9613797/twitter-api-status-update-with-media-return-404-not-found

2 years 6 weeks ago
mdsadiqm
@mdsadiqm sadiq

I am getting error :- {"request":"\/1\/statuses\/update_with_media.json","error":"Could not authenticate you."};

Below is my code let me know where i am wrong.

  1. $timestamp = time ( );
  2.        $nonce = md5 ( uniqid ( rand( ), true ) );
  3.      $twitter_url = 'https://upload.twitter.com/1/statuses/update_with_media.'.$format;
  4.        $url = 'https://api.twitter.com/1/account/verify_credentials.'.$format;
  5.       $sbs ="POST&https%3A%2F%2Fupload.twitter.com%2F1%2Fstatuses%2Fupdate_with_media.".$format."&oauth_consumer_key%3D".$this->consumer_key."%26oauth_nonce%3D".$nonce."%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D".$timestamp."%26oauth_token%3D".$this->access_token."%26oauth_version%3D1.0";
  6.       $key = $this->consumer_secret. '&'. $this->access_secret;
  7.       $signature = urlencode ( base64_encode ( hash_hmac ( 'sha1', $sbs, $key, true ) ) );
  8. $headers =
  9.             <<<EOF
  10. OAuth oauth_consumer_key=$this->consumer_key, oauth_signature_method="HMAC-SHA1", oauth_token=$this->access_token, oauth_timestamp=$timestamp, oauth_nonce=$nonce, oauth_version="1.0", oauth_signature=$signature
  11. EOF;
  12.  
  13.        //Everything must be on a single line...
  14.        $curl = curl_init ( );
  15.         curl_setopt ( $curl, CURLOPT_URL, $twitter_url );
  16.        curl_setopt ( $curl, CURLOPT_RETURNTRANSFER, true );
  17.         curl_setopt ( $curl, CURLOPT_FAILONERROR, false );
  18.         // Set our OAuth Echo headers
  19.         curl_setopt ( $curl, CURLOPT_HTTPHEADER, array (
  20.             'X-Verify-Credentials-Authorization: ' . $headers,
  21.             'X-Auth-Service-Provider: ' . $url,
  22.           "Content-type: multipart/form-data;",
  23.             "Expect:" 
  24.         ) );
  25.         //post your data $post_data array with you api key, message, and media...
  26.         curl_setopt ( $curl, CURLOPT_POST, 1 );
  27.         curl_setopt ( $curl, CURLOPT_POSTFIELDS, $post_data );
  28.      curl_setopt ( $curl, CURLOPT_SSL_VERIFYPEER, true ); 
  29.       curl_setopt ( $curl, CURLOPT_SSL_VERIFYHOST, 2 ); 
  30.      curl_setopt ( $curl, CURLOPT_CONNECTTIMEOUT, 30 );
  31.      curl_setopt ( $curl, CURLOPT_TIMEOUT, 10 );
  32.         $response = curl_exec ( $curl );
  33.        if ( ! $response ) {
  34.             $response = curl_error ( $curl );
  35.         }
  36.         curl_close ( $curl );
  37.         return ( $response );

Thanks

1 year 47 weeks ago
episod
@episod Taylor Singletary

I'm not sure exactly what your problem is (there are a lot of resources on this site for you to research issues like this yourself and debug incrementally) but this code contains a number of things that aren't at all pertinent to what you're trying to accomplish -- like setting OAuth Echo headers (X-Verify-Credentials-Authorization, etc.) when trying to talk to upload.twitter.com. You're also inconsistently wrapping your parameters in quotes in your OAuth Authorization. header.

Have you considered leveraging an OAuth library to accomplish this to better increase your chance of success?

1 year 47 weeks ago
StarlineVenture
@StarlineVenture Starline Ventures

I have been problems posting photos to twitter. I download the photo from a url to my server, then I post it to twitter but I keep getting a status error. Here is the discussion https://dev.twitter.com/discussions/9424

1 year 39 weeks ago
VellumGames
@VellumGames Vellum Interactive

Has this issue been resolved? I am getting a 500 error when posted to update_with_media.json, and it is driving me bonkers.

Here are the things I have doubled checked:
1. I am NOT signing the status or post body.
2. I have all the correct Authorization header
3. I doubled checked my /r/n, and it seems to match the working copy posted.

1 year 34 weeks ago
MikhaWrdha
@MikhaWrdha RendraPratama

Buatlah daftar semua hal yang Anda bisa lakukan untuk menghasilkan uang. Dan kemudian mencoret dua pertiga dari daftar itu. Meninggalkan hanya apa yang Anda sangat bersemangat yang ada di daftar itu...

1 year 3 weeks ago
nyxtom
@nyxtom Thomas Holloway

For the life of me I can't get my posts to work either:

  1. POST /1/statuses/update_with_media.json HTTP/1.1
  2. Host: api.twitter.com
  3. Accept: /
  4. Authorization: oauth stuff here…………….
  5. User-Agent: C# Twitter Agent
  6. Content-Type: multipart/form-data;boundary=8d11111c598fc30
  7. Content-Length: 215669
  8. Connection: keep-alive
  9.  
  10.  
  11. --8d11115c0e0e9ec
  12. Content-Disposition: form-data; name="media[]"; filename="fhyS4yoDLQPU31VqYxOLdw.png"
  13. Content-Type: application/octet-stream
  14.  
  15. rfjasdklfjdsjfdsjfl raw bytes here and such...............
  16. --8d11115c0e0e9ec
  17. Content-Disposition: form-data; name="status"
  18.  
  19. Late night hackathon on some publishing features with media support
  20. --8d11115c0e0e9ec--
5 weeks 1 day ago
jaakkosf
@jaakkosf jacob petrie

@nyxtom, the v1 endpoint is deprecated. You'll want to use the v1.1 endpoint. Take a look at the endpoint doc and our uploading media guide.

https://dev.twitter.com/docs/api/1.1/post/statuses/update_with_media

https://dev.twitter.com/docs/uploading-media

2 weeks 5 days ago