Access-Control-Allow-Origin

fabianonline
@fabianonline Fabian Schlenz

Hi there.

Will there ever by an Access-Control-Allow-Origin header in the responses from the Twitter / Streaming API? That would enable us developers to create Twitter clients in pure JavaScript... Currently the only way to get around the Same-Origin-Policy is by providing a proxy...

Fabian Schlenz

2 years 33 weeks ago

Replies

Nico1485
@Nico1485 Nicolás Muñoz

Yes, please, I request the same header. It's VERY important.

2 years 26 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

I don't think this is as useful as you may want.

The streaming API will not work with a standard XHR call, since there's no specified way to flush the request buffer. I believe browsers just hold the response data in memory until the request closes, which is not supposed to happen in a streaming request. I also suspect that such a client will cause a lot of connection churn due to page refreshes, something we want clients to limit. In general, a HTML page is not a good candidate for a streaming API client.

For the REST API, we offer JSONP for public read operations. But for write operations, OAuth 1.0a is not suitable for HTML clients, since there is no way to keep a consumer secret hidden, meaning anyone could spoof your application. This is also a practice we want to limit.

In general, it sounds like it would be an interesting experiment, but the reality is that such a client would not be useful as a production application.

2 years 26 weeks ago
fabianonline
@fabianonline Fabian Schlenz

Well... I'm using this construction as my main Twitter client for months now.

You can use the Streaming API surprisingly easy - all browsers trigger an readyStateChanged event (number 3) when receiving new data. Only exception is Opera, but there you can just use a timer to periodically check the buffer for new data.
Once the client is up and running, there is just one open connection to the streaming API with no more "connection churn".
And on "keeping consumer secrets hidden": It has been argued many, many times that keeping those keys secret just isn't possible. Since the app has to use this keys, they have to be stored somewhere. A JS client just makes it a bit easier to find this keys, but it's not as if it would be impossible to get this keys from other apps.

On the other hand, adding this Header would for example enable people to easily show their latest tweets on their homepage. The widgets are a nice thing - but they just aren't customizable enough...

2 years 20 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

Can you post the code you're using to check the XHR buffer? I still think this is probably a huge waste of RAM, as the request will never complete and the page will not flush the response buffer.

2 years 20 weeks ago
dtinth
@dtinth Thai Pangsakulyanont

I once done that on my web-based Twitter client. I just make a normal XMLHttpRequest and and then periodically check the length of responseText. Process new data when they become longer.

That was a huge waste of RAM. But you can disconnect and reconnect the stream like every 5 minutes, or when what's in the buffer is reasonably large.

Also for "keeping consumer secrets hidden", when the users of my web-based app wants to retrieve data from the API, it sends the request to my server, then the server signs it, but lets the client request the resource from Twitter API itself, to reduce load on the server. That way the consumer secrets are still hidden. CORS would be useful in this case, so my app can tell the browser to contact the Twitter API directly with needing a browser extension.

2 years 20 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

Disconnecting and reconnecting the stream that often is connection churn and behavior we don't want clients to practice.

How does the server prevent a malicious app from obtaining freely signed requests?

2 years 20 weeks ago
dtinth
@dtinth Thai Pangsakulyanont

My bad for the 5 minutes example, it's too short. As @fabianonline said, it could be like hours, or depending on the size of the buffer.

About how the server prevents a malicious app from obtaining freely signed requests, the API on the server is meant to be used only by the client, so CORS isn't needed there, but the Twitter API is meant to be used by developers. Additionally I also prevent CSRF on the server side code to prevent XSS or freely signed requests.

2 years 19 weeks ago
fabianonline
@fabianonline Fabian Schlenz

The Client is open source and can be found at http://github.com/fabianonline/geotweeter (and yes, I know that the code is horrible - it just grew in a sort of organic way ;-) ).

The code for checking the buffer is in function parseResponse() (curently line 529 of geotweeter.js). It is set as onReadyStateChange handler of the request (startRequest(), line 485). All browsers (except for opera) call this handler with a readyState of 3 every time new data is received via the stream. In Opera I just call this function every 5 seconds via a timer.

Regarding RAM usage: Yeah, the buffer will grow and grow, but in my case it doesn't get that large that quickly - I'd say about 500KB/hour, tops. If I remember it, I'll run a Heap Snapshot the next time the client has been running for several hours.

2 years 20 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

Well thanks for providing the code. This is still a practice we don't really want to encourage (particularly for pages which may be refreshed or open/closed often) but it's interesting to know that such a thing exists :)

2 years 20 weeks ago
fabianonline
@fabianonline Fabian Schlenz

Well... I don't see much of a difference between my JS base client and, for example, Echofon, which I can as easily start and quit periodically which will also lead to lots of opened and closed streaming connections...

Anyway, going back to topic, is there any chance of Twitter providing Access-Control-Allow-Origin headers? Just adding these will make accessing the API a lot easier. You theoretically could even add a field to applications' settings to enable / disable this header.

2 years 19 weeks ago
Louis_Remi
@Louis_Remi Louis-Rémi Babé

I think I haven't seen any problem listed in this page that can't be solved.
I'm amongst the growing number of developers who think that (mobile) web apps represent the future of cross-platform app development. Twitter has to find solutions for developers willing to create clients using web technologies.

The first problem I had to solve was how to sign requests while keeping my secret key... secret. I opted for the same mechanism @dtinth is using: as server-side script signs request that are then handled on the client-side. Communication with the signing script is done through SSL, and every request has to include oauth tokens. This way no malicious app can obtain freely signed requests.

The second problem I have to deal with is the inability to use the streaming APIs. CORS would be very helpful here. As it as been said, a web app is no different than a native app: it is absolutely possible to build a "single-page" web app that keeps streaming connections open as long as the app is in use. Just like it is possible to create a bad native app that closes and open streaming connections more often than necessary.
Regarding memory consumption, if this is really a problem, this could be solved by providing a "server-sent events" version of the streaming API: prepend "data:" at the beginning of each JSON line and add an empty line at the end of the message

  1. data: {"JSON": "a new status"}
  2.  
  3. data: {"JSON": "another status"}
  4.  
  5. ...

See the event-source specification: http://www.w3.org/TR/eventsource/
and the demo I made while working at Mozilla: http://hacks.mozilla.org/2011/06/a-wall-powered-by-eventsource-and-server-sent-events/

The last problem I have to deal with is the inability to store profile pictures locally. If those images were served with CORS headers too, it could be possible to serialize them using canvasContext.toDataUrl() and store them as a string in localStorage or indexedDB. Currently, I have to use a lib that generates a dynamic appcache: mediaCache ( http://github.com/louisremi/mediaCache ).

2 years 19 weeks ago
Louis_Remi
@Louis_Remi Louis-Rémi Babé

We might be focusing too much on possible complications with the Streaming API here.
The first thing to do would be to enable CORS on the REST API.

I'm sure there is a way to make that happen.

Lr

2 years 18 weeks ago
LeaVerou
@LeaVerou Lea Verou

I'd guess that the REST API, excluding OAuth, could support CORS with no security issues. Github does exactly that: Supports CORS in everything except OAuth. Everything else, including writing (as long as you have either the oauth access token or username & password) is supported just fine.
They don't allow it for every domain, just for the ones registered with Github as OAuth applications, and of course rate limiting applies too.
I've used the Github API through CORS extensively in dabblet.com and it was a lifesaver.
Maybe Twitter could do something similar? I know many front-end developers that would celebrate. :)

2 years 14 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

I've made the request internally, although I think there would be more work to get to the point where where registered domains would be possible. It would make sense to support CORS anywhere we support JSONP though.

I'll follow up here if there's any change.

2 years 13 weeks ago
fabianonline
@fabianonline Fabian Schlenz

Whoa, that sounds great... But as I can see from the documentation, only the endpoints at geo/* and search currently can be used with JSON-P...?
Please use the scheme @leaverou posted: Anything but oauth stuff. Of course including the streaming API (for which JSONP would be somewhat useless... ;-) ).

2 years 13 weeks ago
kurrik
@kurrik Arne Roomann-Kurrik

Any public read-only endpoint should support the "callback" parameter - it's probably a failing of the documentation that this isn't clearer. Something else for me to work on :)

2 years 13 weeks ago
LeaVerou
@LeaVerou Lea Verou

Thank you so much Arne! I'm looking forward to any steps towards this direction. I hope Twitter keeps in mind that anything is better than nothing. Even CORS access restricted in a number of different ways, is way better than none.

2 years 13 weeks ago
Louis_Remi
@Louis_Remi Louis-Rémi Babé

I'm glad to hear that @kurrik, thanks a lot.

2 years 13 weeks ago
randallb
@randallb Randall Bennett

+1 on the "anywhere JSONP exists CORS should too". FB's API is much simpler in this regard.

2 years 2 weeks ago
hetblauweknopje

Any news on the CORS headers for public read-only endpoints?

1 year 34 weeks ago
tktredstar
@tktredstar Tom Tavernier

Also waiting for this to complete, I want to load in twitter via a webworker!!

1 year 30 weeks ago
jniktweets
@jniktweets John Nikolakis

So it seems there is absolutely no way to get data from the v1.1 API without a back-end at the moment. Any news on if / when that is going to change?

1 year 4 weeks ago