I've been thinking on and off about propagating blog update notifications, in an attempt to get some solution that sucks far less than the current "poll the list of things you're interested in" scheme. Granted, shouting across string-connected tin cans would be better than that, but I was shooting for something a touch more sophisticated than that. I think I've found one. And, even better, there's a pretty good template to build the system design around.
Usenet News.
More specifically, Usenet News served by INN and other stream-capable servers.
Now, I'm not proposing that blogs get served over NNTP, though that's actually pretty darned close to what's needed, but think about it. What we really need is twofold. First, there needs to be a way to get pinged when things change. And second there needs to be a way for clients to get notification of all the blogs of interest that have changed without having to go and ping each blog in turn. NNTP handles both scenarios just fine.
Most people haven't run news servers, or have at best run leaf servers that don't have any special connection to their upstream servers, but I have (albeit a small one) There are a number of nice features in the system that could easily be pulled into the core feature set, things that are both simple and reasonably scalable, which'd allow for a distributed system of a size sufficient to manage what we're seeing now. Things like known subscribers with cached content change notifications, so a reconnect gets a blitz of queued change notices (which in a blog scheme doesn't even require separate spool directories if there's a server subscription list)
Plus, of course, usenet's show us all sorts of ways to not do things, which is often as important as showing what to do.
I think it's doable. Heck, I think an updated NNTP (BTP, anyone? :) scheme could work for distribution of blog content, as well as change notification, but that strikes me as a Phase 2 sort of thing.
Maybe I'll bodge something together at some point. Shouldn't actually be all that tough, other than the shortness of bandwidth I have here at home if it actually works and starts taking off...
Posted by Dan at March 24, 2003 01:48 PM | TrackBack (1)Totally doable.
Think about this:
1. create a group blog.feeds.rss. Set espire to a low n of days. probably a hiearchy by subject/topic...
2. everytime you change the your blog, post it to that newsgroup. Options:
subject should be your ID, the URL of the feed;
Optionally PGP sign the content, to avoid fake rss posts, no need to use your pgp key, create a new one for that. The client's can retrive your public_key from the site direcly once (the first time it's needed) with a .pgp appended to the original URL
3. pool your local newsgroup for those URL's you are interested on.
Totally feasible today... Next step's:
1. create the group;
2. convince at least radio and moveable type to support NNTP posting automagically;
3. convince a coupple of rss agregators (Netnewswire and ampthedesk, probably?) to support it;
4. sit back, relay and see where this is going :)
Want help? :)
Posted by: Pedro Melo at March 24, 2003 03:51 PMTotally doable.
Think about this:
1. create a group blog.feeds.rss. Set espire to a low n of days. probably a hiearchy by subject/topic...
2. everytime you change the your blog, post it to that newsgroup. Options:
subject should be your ID, the URL of the feed;
Optionally PGP sign the content, to avoid fake rss posts, no need to use your pgp key, create a new one for that. The client's can retrive your public_key from the site direcly once (the first time it's needed) with a .pgp appended to the original URL
3. pool your local newsgroup for those URL's you are interested on.
Totally feasible today... Next step's:
1. create the group;
2. convince at least radio and moveable type to support NNTP posting automagically;
3. convince a coupple of rss agregators (Netnewswire and ampthedesk, probably?) to support it;
4. sit back, relay and see where this is going :)
Want help? :)
Posted by: Pedro Melo at March 24, 2003 03:51 PMWhile I think NNTP and usenet in general is a good starting point, both as a template and object lesson, I think that a separate protocol is the way to go here.
We don't, for example, need to transfer information, merely metainformation. (Though transferring actual blog entries would be useful as an extension) We do have to deal with a potentially untrustworthy set of data feeds, and a very strong separation between data sources, data transporters, and data consumers.
There's also the issues of potential validation or verification of data sources, and a single source for each data stream, rather than news' multiple sources.
NNTP's worth looking at, but I don't think it's the right answer. Just closer to the right answer than what we have now. I think, though, that I will put something together and see what I can do about a lower-impact system than we have now.
Posted by: Dan at March 24, 2003 05:40 PMThis was also on my mind the other day. My solution involved subscribing to a feed that sends you an e-mail when something changes.
E-mail, in this case, could be something like SMS or a Jabber message, or whatever you prefer. It's just an asynchronous message pushed from a server.
Posted by: chromatic at March 24, 2003 08:16 PMWhile an async ping notification using current schemes (email, SMS, Jabber, IM, IRC, whatever) would work pretty well as a plug-in on the end of the distribution net, what I'm currently interested in is the injection of notifications into the network and the means of verification, validation, and transport of the notifications across the network. Once that's set, the end notification, whether it's a daemon that fetches the new entries or a program that pings someone's pager, is relatively trivial.
Posted by: Dan at March 25, 2003 11:58 AMI saw that in the apache logs. It does look interesting, though as I said I don't think NNTP is ultimately the right way to go for this. NNTP was designed for a gentler, less obnoxious time, when abuses of the system could be dealt with by calling up the offender's system knowing full well that something appropriately BOFHish would happen. These days, well...
There's just too much trust built into NNTP. It's a shame that it can't be any more, but there you go.
Posted by: Dan at March 26, 2003 04:39 PM