I can't believe I missed Yahoo's Pipeline beta. Chris from Touchstone actually dropped me a email and asked if I've seen it. But all I get now is…
Our Pipes are clogged! We've called the plumbers!
Well in the meantime a lot of people are talking about it (Techmeme). Tim O'Reilly has a long piece about it on his Radar blog. He starts with,
Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as “an interactive feed aggregator and manipulator” that allows you to “create feeds that are more powerful, useful and relevant.” While it's still a bit rough around the edges, it has enormous promise in
turning the web into a programmable environment for everyone.
In agreement, but I'm worried Yahoo might be focusing too much on aggregation that general purpose pipelining of any data source online. Tim then talks about why he's excited and points at some of my also favorite posts in this area. Jon Udell's keynote at the 8th Python conference and the JavaOne keynote which really gelled with my thoughts about
Pipelines at the time. This is also another reason why I got fed up of the Gillmor Gang without Jon Udell. Anyway back to Tim's post, here's a couple of other things I found interesting.
But perhaps more significantly, to develop a mashup, you already needed to be a programmer. Yahoo! Pipes is a first step towards changing all that, creating a programmable web for everyone.
This is certainly very true, coming from a design background I just couldn't understand why pipelines were not used more in application development. I actually thought the move towards objects in programming would be the start of this, but I guess not.
Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique (with the “ue” this time:-), count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops.
RSS and XML are easy targets for a beta service. But whats really needed is more input adapters. Microformats, FOAF, S5, WebAPIs, XMPP, etc. The transformers are predictable bar the user-defined filters (which I would assume would be XSL?). There's other services like RSS Mix and Feed Rinse which do the same thing. Chris is right filters are old hat.
Talking of Chris, in his post he seems quite down on his own pipeline: Touchstone. Personally I think their further down the line because the interesting part of the pipeline is being able to mix local and remote content not just remote. Also the widget style UI is very powerful. You could use Yahoo Pipes and I guess Yahoo Widget Engine to create something like Touchstone but your missing the Relevancy engine (APML) which did a great job of finding me screenshots of Windows
Mobile 6.
I'm a little worried about the focus on the GUI used for Yahoo Pipes. It sounds good but there needs to be thoughts about interopability. I don't want to create a great Pipeline and then be locked into Yahoo Pipes forever more.
Anyway, I can't talk much more about it till I get a chance to play with it first hand. Good work Yahoo.
