Updated Topify Gmail Filters
Well, Twitter got prettier follow and direct message notifications today. Bully for them. Now I have to publish this update.
Update (05/07, 22:45): My update was broken, so the update had to be updated. The filter should now catch DMs, too. Believe it or not, I was wrong that Twitter changed the address that direct message notifications come from; it stayed the same. So that part of the filter didn't need to change. All's well that ends well, right?
I'll leave this post up just for posterity, but please don't try to use these new filters and then complain that they don't work. 
To make things easier for myself, I'll assume that everyone's seen the old filter setup I published at the end of last month.
The old method was quite convenient for those of us with multiple Twitter accounts, because the email addresses in the From headers changed depending on the address associated with each account (after October 30, 2008 and before this afternoon). Now they all come from noreply@twitter.com (as they used to last year), with the account-specific email addresses tucked away in the reply-to headers (which I can't filter on in Gmail, so that sucks).
Not only did the addresses again become uniform, but that was basically the only easy way to tell the difference between my personal account (which has Topify set up) and the others I run (which don't). Now I have to go through several hoops, and the filter string is longer.
Anyway, here's the updated updated filter string; put this all in the "Has the words" field in Gmail's filter settings:
(to:(you@yourdomain.tld) from:(noreply@twitter.com) subject:"is now following you on Twitter") OR from:twitter-dm-you=yourdomain.tld
As before, you@yourdomain.tld is the email address set in your Twitter account settings, the address to which all your notifications are sent.
I won't bother making an XML file for the new filters, because it's only one field. I'll probably leave the old one for posterity — at least until my Google Page Creator account is completely borked by the transition to Google Sites — because it's easier than deleting it and then updating my old post to reflect that.
Just for the record, Twitter, I'm not happy that I'm having to retool my filters this soon. If you want to make me happy again, put back your email headers the way they were last week. kthx
How To: Safely Use Twitter Notification Enhancement Services
As you all have probably heard, Twitter is gaining popularity in leaps and bounds. All the new users mean more follower notifications arriving in my inbox, and Twitter's default messages aren't very useful. (The direct message notifications are pretty bare-bones, too, but I don't get many of those so it wasn't a priority.)
Update (05/06): Twitter prettified their emails, but I still think Topify's are better. Unfortunately, Twitter also went back to using the same address (noreply@twitter.com) for all users' notifications, putting the email-address – specific addresses in the reply-to header. So the filter setup in this post doesn't work any more. I had to come up with a new, more complicated filter... Stupid Twitter...
The Beginning
The first enhanced-notification service I discovered was Twimailer, created by a British developer named Jon Wheatley (and apparently later sold—shortly after I signed up — to a Romanian named Toni).
Despite the admonitions in the article above (on ReadWriteWeb) about changing passwords and all kinds of security precautions, I'm not worried about my own account. There's one simple reason for that: I never actually switched my email address in Twitter's settings. Instead, I created a Gmail filter to auto-archive follow notifications from Twitter and forward them to Twimailer. That way, I:
- had all my follow notifications even if the service went down (it did for several days) or glitched (sometimes I get messages with no information)
- only forwarded the messages Twimailer needed to be useful, rather than everything
- made sure to keep password resets (which I haven't used for my main account in the last few months anyway) completely out of Twimailer's hands
I was very comfortable with this system. I can only guess that Jon's original intent was to simplify the setup process. After all, most people don't bother with email filters, and wouldn't necessarily know how to set one up. Changing settings on Twitter's website is a lot easier.
A New Age
Thanks to a TechCrunch post about fighting Twitter spam I read tonight, I discovered Topify, an invitation-only (for now) service based in Israel that offers all of what Twimailer did — and more.
I found an invite on the Topify blog (sorry, no link; you gotta dig through their site too so it's fair for everyone) and quickly signed up. The Twitter password field distressed me a little, but it's obviously necessary for all the extra features (like follow-back, reply to direct message, and block), all of which can be done via email with Topify. (In the future, I hope Topify will implement support for Twitter's OAuth authentication and delete users' passwords from their system. Consider this a request, Arik. 
Anyway, switching was pretty painless. All I had to do was change the address to which my Gmail filter forwards and add my direct-message notification From address to the filter. I'm currently waiting for something to happen on my Twitter account so I can try out the new service. (I considered running Twimailer and Topify side-by-side for a bit, but decided against it; redundant emails would increase my processing time, the opposite of the intended effect.)
Filter Details
For those who want to copy my setup (I'm telling you, it's a lot more resilient than the default instructions from either service), here are the filter settings to enter. Update (05/06): These settings are left here for posterity; they won't work anymore thanks to Twitter's changes from today. See the first update near the top for more detail. Update (05/07): These should now work again, since Twitter appears to have gone back to the old email headers.
In the filter's From box, enter:
- Topify: twitter-follow-you=yourdomain.tld OR twitter-dm-you=yourdomain.tld
- Twimailer: twitter-follow-you=yourdomain.tld
Replace you=yourdomain.tld with your email address, using = in place of @.
That's all you need to do for filter criteria. (If you have only one Twitter account coming into your inbox, it's even easier; you can omit the -you=yourdomain.tld part(s) of the filter criteria. It doesn't hurt to include them, though.)
For actions, I selected "Skip Inbox" and "Mark as read", and told Gmail to forward these messages to my secret Twimailer/Topify address.
Click the Create filter button, scroll down your filter list, and you should see something like the following (image is linked to full-size version):
(There's also an XML file available to import, for those with the Filter Import/Export feature enabled in Gmail Labs, but creating the filter from scratch is pretty easy. The file link might go dead in a month or two when my Google Page Creator site is moved to Google Sites, but I'll know because things like the site logo will stop working. If that happens, I'll definitely fix it.)
Note: As I was writing this, I discovered Chris Messina's post about this, published almost two months ago. My little hack is nothing new, I guess; but I'll publish anyway because his instructions are focused on Twimailer and Twimailer only.
Wrap
Let me know if you find this little hack useful. I haven't time to make a bunch of pretty screenshots (unlike Chris , so if you have questions, post in the comments.
Incidentally, this is my 500th blog post. If that means anything.
Comments On Google Changes
Image via WikipediaOn Wednesday, Google announced changes to or shutdowns of several services. Google also shut down Lively, the 3D chatroom that I always looked upon as a rather silly Second Life knock-off, beginning January 1 — just a few short weeks ago. ("Reasons Google should kill Lively" was actually a topic in my to-blog list for several months, but it looks like I didn't have to blog about it for Google to see that it wasn't a good fit with their other projects. Nice work saving effort there, self. Anyway...)
I'm rather unconcerned with the fate of Google Catalog Search, which was (I believe) really just a good way to work on the OCR technology Google now uses in Book Search. (Catalog Search's former homepage at catalogs.google.com now redirects to Google's main site.) I never used it; catalogs are pretty useless these days anyway what with online shopping and Froogle (now known by the much-less-punny appellation of Google Product Search and accessed by a link in the Google header called "Shopping", though I wish they'd bring back the old name).
Image via CrunchBaseAlso of little real consequence to me, personally, is the development stoppage on Google Notebook. I don't use it much, and my service will be unaffected anyway. Well, relatively unaffected, at any rate. When I do use Notebook, it's usually in conjunction with the "Clip" function of the accompanying Firefox extension — which will no longer work. But the service will continue for now as long as one already has an account.
I'm actually somewhat glad to hear of Dodgeball.com closing and Jaiku being open-sourced. Dodgeball was a premature service somewhat like Brightkite — which I occasionally use — that stagnated almost immediately after being acquired by Google in 2005. Its interface has always been phone-only (Brightkite allows use via text message, Web interface, or iPhone/iPod-Touch – optimized site). In my opinion, Google would do well to encourage Dodgeball's users to move to Brightkite. An agreement with Brightkite to ease the transition for users willing to make the switch would likely make Dodgeball's death as swift and painless as possible.
Image via CrunchBaseJaiku has also stagnated, the most notable annoyance being that it has been invite-only since its acquisition in 2007. With the transition to open-source (the service will continue to be run by a team of volunteer Googlers), Laconica1 might get some new features, competition, or perhaps both. Twitter may also be encouraged to develop long-overdue features like OAuth support, since Jaiku is slated to support OAuth right out of the gate when it is released to the open-source community. (Securely logging into Twitter from third-party applications and websites has long been a point of contention in the community, because the only option continues to be giving every app your username and password. Not a very secure solution, especially because there isn't even the layer of security provided by API keys such as used by FriendFeed and, yes, Jaiku. Google itself has supported OAuth authentication for its own services since as long as I can remember.)
Overall I think Jaiku's fate is the best and most (potentially) beneficial of all those announced this week. The potential for competition and improvement in the entire microblogging and status-update ecosystem is wonderful. However, potential users of the newly – open-sourced Jaiku Engine (hopefully that's the address where it will live — it 404s at the moment, but I'll watch and update if necessary) will still be dependent upon other Google amenities, namely Google App Engine. Jaiku was ported to App Engine last year, and the code base is now dependent upon being run in that environment. (This could be a ploy to get money, since App Engine charges — or will charge; I'm not sure of the time frame of the fee structure—nominal fees for applications that move beyond moderate-scale deployment — not a bad business move, if I do say so myself.)
Image via WikipediaThe one impending closure that I am really and truly saddened by is the shutdown of uploads to Google Video, which is now (and will shortly be, but for already-uploaded content) solely a meta – video-search site. I've always preferred Google Video to YouTube for a variety of reasons. Some of them are:
- cleaner interface
- more professional player appearance (nice for embedding on sites like swrobotics.com)
- especially the lack of related videos and pop-down search bar
- less cluttered site
- fewer extraneous features
- more focus on videos, less on social networking
(As an aside, there is also some uncertainty how the closure will affect users of Blogger who use the platform's video upload feature, which uses Google Video for hosting. I have never uploaded a video for a post through Blogger, so I'm somewhat detached from this particular concern — but I thought it was relevant nonetheless.)
I have many more reasons and thoughts on this comparison that I can't easily articulate, but I will be very sad to lose the ability to add new content to Google Video. Failing the motivation to deal with YouTube (which has limits on content length and filesize that may or may not be lifted in the aftermath of Video's shutdown), I suppose I might have to start uploading somewhere else entirely, like Vimeo (whose player I like quite a bit).
YouTube has always kind of irked me as a place to upload content. It's a great place to go to watch videos, almost always, but for hosting videos intended for display on another site... Despite the number of sites that do so, its player has always seemed out of place on the sites I'm involved with. The in-built social network (which includes profile pages, a messaging feature, "friendship", and so forth) has always seemed like an unnecessary layer to me.
I suppose I should have seen the writing on the wall when Google's own blogs, which used Google Video uploads themselves for a while even after the YouTube acquisition, all switched over to YouTube videos and left Google Video in the dust. Maybe I'm a video Scrooge, or maybe I'm just being resistant to change (who isn't?). Whatever the reason, I — honest and truly — will miss being able to add my videos to such a simplistically elegant site as Google Video has been.
Update (19:46): Some good news (depending on how you look at it) came this week, too: FeedBurner now shows a link to migrate your account to Google on the My Feeds page. That wasn't there a couple days ago...
Notes:
Twitter’s Problems: Getting Worse
I once told myself I'd never write about Twitter's downtime, because everyone in the tech blogosphere writes about it. Today I'm going against that. The problems have simply become ridiculous.
Back when I first started using Twitter, it was very responsive. (I tried to find my first tweet as an example, but there's currently a paging limit of 10 which blocks everything but the 200 most recent tweets.) Posting was nearly instantaneous, the API allowed 70 requests every hour, one could get replies to one's own tweets using a convenient tab on the site or simple call to the API, you could add keywords you wanted to track and have matching tweets sent to you (via IM or SMS), and you could even use Twitter via IM alone!
Utopia Shattered
Now all that has changed. With the thousands -- nay, millions -- of new users that have joined over the last several months, plus the increases in highly tweetlific (tweeting prolifically) power users, Twitter has had lots of downtime. There's been more downtime than I could possible list here; even linking to reports of that downtime is something I'll leave up to the reader (search for "twitter is down" in Google and see how many results you get).
Not only has the service gone down a lot, but many features have been crippled or disabled. The API is limited to 20 req/h out of the original 70 (and has been for weeks); keyword tracking has been shut off for even longer; the IM bot has been offline for so long I can't even remember what using it was like; pagination is limited to the latest 10 pages (of 20 tweets) for each section; the Replies tab has been disabled, requiring the use of search services like Summize to gather responses to one's own messages; the list goes on and on.
Perhaps the most annoying part of all this is the fact that, no matter what the Twitter developers do, the site still goes down. Twitter was founded by the engineers who wrote Blogger, for crying out loud; it should be able to handle a little scaling. But maybe the fact that Blogger engineers wrote the service is the very problem.
Code-Level Problems
The fundamental issue is this: Twitter was written to be a content management system (CMS), not a messaging service. Blogger is a CMS, too. The capability to handle hundreds or thousands of inputs every second is not part of the normal CMS design pattern. Twitter needs to be fundamentally rewritten.
I know I'm not saying anything new; in fact, some of my ideas were inspired or taken directly from other blogs (too many to even begin to remember, unfortunately). It's just that, with today's outage, I realized just how true all the criticisms are. Twitter may disable feature after feature in an effort to reduce the load on the servers that make the site run, but the underlying architecture is a huge (er, very narrow) bottleneck. Technically speaking, you cannot make a messaging system out of a CMS; it's just not possible, code-wise. The two system types have vastly different ways of handling things.
Secondary Effects
Things might not be so bad, though, if being written as the wrong kind of service was Twitter's only problem. The excessive load caused by the incorrect architecture has caused other technical failures in the system, including the loss of an entire database about a month ago and, just this morning, an overloaded load balancer (how's that for irony?). But at least they launched that nice Twitter Status Blog so they can tell us that they're down after we've already known for half an hour.
Conclusion
If I didn't love Twitter to death I'd probably have given up on it by now. Lots of people already have. The latest darling in the social media space is Plurk, which I personally can't stand (the UI is ugly). I like Jaiku better, but it's been invite-only ever since being acquired by Google, which means it's hard to get an account. Pownce is just too weird for me. All I can do for now is deal with the ridiculous bugs, outages, glitches, and all the other crap we Twitter users have to deal with. Then, during the downtimes, I can hope that when I get back in August, Twitter will be back to normal.
If only I had any confidence that it'll happen that quickly.
Update (08/17): Well, actually, Twitter got a whole ton better over the last six weeks or so. After continuing to use it for almost two weeks (back in normal, twhirl-using mode), I'm finding the upped API limit (100 req/h) to be absolutely great, and the site is much faster than it was in June. Looks like a lot of my complaints from this post are no longer relevant. Yay! I just wish that some people (*cough* possible248 *cough*) hadn't moved to other sites in the interim...
Users to Twitter: Get Stable
I myself am included in those users. While I probably won't be participating* in the event, I thought I'd still call attention to it, within my small sphere of influence, in case some of you want to take part.
Anyway, this Twit-Out, as it's being called, was started by a couple people on FriendFeed: Shey Smith and Bwana McCall. Shey came up with an initial what-if question that sparked Bwana's idea to actually do something. Then Andrew Dobrow came along and made the logo you see at the beginning of this post.
So this Wednesday, May 21, is the first Twit-Out event. If you want to join in and try to get Twitter to realize that reliability is important (I'm sure they do already, to play the devil's advocate -- at least to some extent), just don't tweet on May 21. It's pretty simple. If you have accounts at Pownce and Jaiku, just use them. Heck, use Twhirl to post to FriendFeed instead of Twitter.
I have cross-posting set up within Twhirl so my stuff shows up in three places, but I'm not sure if I could kill posting to Twitter (with an incorrect password or somesuch) while retaining the other two services. Since I don't plan to officially participate*, I'll let you come up with your own solution. Unfortunately I have other things that need doing tonight.
While I'm not sure it'll accomplish anything, it's certainly a worthy cause. We've already seen Twitter become a very useful communication tool, and some people and even businesses rely on it daily. If it keeps going down, that's a lot of potential lost profits and time for users (however legally exempt Twitter is from actually caring about that). Though people not using Twitter is kind of the opposite thing one would expect to get them to beef up their systems...
* - May 21 will be the first performance of a show I'm in, so I anticipate being busy and away from Twitter for much of the day anyway. My tweeting will likely be pretty low all next week, actually.
[Image credit: Jersey, Suburbia by Andrew Dobrow; used with permission. Notes: I converted the image to PNG format; that is the only modification. Originally Andrew's blog had the ability to link to specific comments, but in the last few days the theme has been changed and that feature removed.]
Social Me
Badges
Subscription Options
Latest Comments
- Voyagerfan5761 on “No Evil”: My NET10 Wireless Experiences
- dogbonz on “No Evil”: My NET10 Wireless Experiences
- Voyagerfan5761 on “No Evil”: My NET10 Wireless Experiences
Latest Posts
Archives
Categories
Creations
Southwest High Robotics Team My first non-personal design work
Promotions, Causes, Etc.
- Yes, it's a shameless plug for money. A strange thing coming from me, but it supports one of the Web's greatest sites.
- I'm sure you saw this one coming. You know what I want.
- Set yourself up with OpenDNS, the fastest DNS service around! You can also set up shortcuts for frequently visited sites and filter content.
- Download the best multi-protocol IM client I've ever seen! So it crashes once in a while, but it's in active development, so they're definitely working on it. (And really, what program doesn't crash once in a while?)
- Don't let Microsoft push through a redundant, broken standard for office files! Chaos and confusion could ensue!
- This is just in case spam-bots visit this site. Click it to find out more (it only does bad things to email harvesters, don't worry).
Blogged.com says I have to leave its badge here to claim ownership of this site. I can't take it off, apparently. Whatever. What would be nice is linking to my own site's page rather than generic "technology news" listings...
blogarama and Blog Flux have similar, but implied, issues. And why does everyone link me to a generic listing page?!