140 Character Scripts!

A long time pioneer in the OSS world, Nat Friedman, recently wrote out a call for people to Take the Tweetable Script Challenge.

That is, to post interesting scripts in 140 characters or less.

He documented some of the results here: Ten Tweetable Scripts.

.

Don't Cross the Domains!

You may have caught this on the Twitter Development Talk Google Group, but last week we made a change to our crossdomain.xml file.

Previously, we allowed Flash applications hosted anywhere on the web to make requests to the Twitter API. Unfortunately, a proof-of-concept security flaw was demonstrated that would allow malicious Flash applications to make requests on behalf of users logged-in to Twitter. So, for the time being, our crossdomain.xml only allows applications hosted at Twitter domains or other domains approved by our team to talk to our API.

If your Flash app needs to talk to the Twitter API, the primary way around this is to host a proxy to the Twitter API on your domain. Beyond that, we're open to suggestions, and we've been collecting them on the aforementioned Google Group. We don't want to make Flash developers second-class citizens, but we also need to act with our users' security in mind.

Track your [stolen] iPhone with Twitter.

Ahh. The iPhone craze is in the air. Although Apple is coming out with an official SDK for writing native apps this month, a huge community has grown around developing indie apps for the JailBroken iPhone. Some truly amazing pieces of software are cropping up, as almost all aspects of the SDK have already been reverse engineered.

Earlier today I worked a bit with Erica Sadun of TUAW on an app of her devise that runs in the background and automatically sends the triangulated geo-coordinates of your iPhone to a private Twitter account of your choice.

Now when someone slyly walks off in possession of a recently-made-theirs phone, you can literally 'follow' them and ... well, politely ask for it back?

http://www.tuaw.com/2008/02/21/tuaw-responds-iphone-lojack/

How to Build a Twitter Agent

Dominiek ter Heide had a dream. A dream of a "Twitter Service that will allow us to track the Blogosphere." So he built it, and blogged about the process! Dive in to his post to learn about building Twitter bots that talk our service via Jabber/XMPP.

If Ruby's your thing, you'll also learn about some great tools for XMPP, managing background tasks, and more in Dominiek's post. If it's not, you can take much of Dominiek's approach and apply it in your favorite language with XMMP libraries like python-xmpp for Python, XMPP.NET for C#, Gloox for C++, Smack for Java, xmpp4moz on the Mozilla platform, or one of many other libraries out there.

Jabber is great for way more than just chatting with your friends, and until now it's been up to talented developers like Dominiek to muddle their way through building Twitter bots and applications with the protocol. I'd like to put together an "official" guide to developing Jabber/XMPP applications for Twitter in the near future. Until then, soak up the knowledge that our great developer community is sharing!

Solving "The Case of the Missing Updates"

The Twitter engineering and operations team had a long week. Just several days after moving our cluster of server to a new host, we started to get reports of users missing updates. This, of course, is exactly the sort of thing we've been working to avoid by increasing our capacity and operating in a predictable, thoroughly-tested, reliable environment. So we got on the case like the Scooby gang.

Tracing

While the Twitter code base has thorough test coverage for the majority of the application, many of our daemons—small programs that work on tasks retrieved from Starling queues— lack transparency into their inner workings. Tracing a message as it passes through the Twitter stack, traversing a variety of machines and services, is no mean feat. But trace we did, and we now have more "observability" built into Twitter's internals than ever before. (Plus, watching a message wind its way through our servers isn't just useful, it's also kinda neat.)

Mediating Between Starling and memcache-client

The insight gained from the tracing work helped us narrow down the potential culprits to Starling and the memcache-client library. Our newest hire, Robey, discovered that the event-driven version of Starling disconnects idle clients after 60 seconds of inactivity. Makes sense, right? Don't want to keep a zillion old connections around when you're a frequently-contacted network service...

Problem is, memcache-client doesn't check for dropped connections when doing a SET operation. We do a lot of those. Not only do we use memcache for typical cache duties, but we employ the memcache-client library as the base of our own StarlingClient library. Diagnosis: we'd inherited a couple of nasty bugs that were wrecking communication between client and server ends of the queuing system that we rely on to keep everything going. Yikes!

The solution ended up being a patch to the memcache-client library that raises a MemCacheError if an operation is unable to be completed. This allows code that wraps memcache-client to rescue and retry. Additionally, Starling has been improved with more logging and the ability to set the client timeout duration. A new public release of Starling that incorporates these fixes and improvements is coming soon.

Wrapping Up

We also fixed a handful of other minor issues with our system that should generally improve the stability of our database and memcache connections, and identified areas in which we were relying on replicated database that contained stale or inaccurate data. All this should contribute to more solid and dependable service from Twitter. With these bugs behind us, we're hoping that users and third-party developers can really see our new cluster shine!

If You Build It, They Will Riff

The best part about open source development is seeing people riff on your code and ideas. Since opening up Starling, we've already seen a couple great contributions.

First off, righteous Ruby hackers Chris Wanstrath (of Err The Blog fame) and Kevin Clark (of Powerset) has mashed up Starling with the EventMachine library. Rather than mucking about with threads and locks, EventMachine allows Starling to operate with tidy, speedy asynchronous ease. We're already using the "evented" version of Starling in production here at Twitter, and we watched our CPU usage graphs drop as soon as we rolled it out. Thanks, Chris!

Next, the hackers at German startup play/type have created Workling, which offers several ways to ease the use of Starling with Rails. Nice work, guys!

Announcing Starling

In various presentations throughout 2007, the Twitter team has made reference to a pure Ruby message queue server called Starling, written by our own Blaine Cook.

Starling is at the core of what we do at Twitter; it moves small messages around to daemons that work on jobs like processing updates, delivering messages, archiving user accounts, and so forth. An asynchronous messaging solution is becoming a necessity for big web applications, and Starling fits the particular needs we have at Twitter. It's fast, it's stable, it speaks the memcache protocol so it doesn't need a new client library, and it's disk-backed for persistence. When other parts of the Twitter site go down, Starling stays up. It's a champ, and we love it.

Until now, Starling has lived a sheltered life in the Twitter code base. We're happy to announce that Starling is now open source and freely available for anyone to use, modify, and improve. We're eager to see patches and to start a proper open source community around Starling.

To give Starling a try today, just sudo gem install starling on your favorite Ruby development box. Let's see some serious queues!