Thoughts and ideas of a dyslexic designer/developer
What was waiting for me in my inbox today…
To: Ian Forrester
We are pleased to accept the following proposal for XTech 2007.
- Pipelines: Plumbing for the next web
It has been scheduled for 16:45 on 16 May 2007.
Please confirm that you have received this acceptance and can deliver the presentation.
Thank you,
Edd Dumbill
So my presentation at BarCampLondon2 will be a very early draft for whats to come in May.
I can't believe I missed Yahoo's Pipeline beta. Chris from Touchstone actually dropped me a email and asked if I've seen it. But all I get now is…
Our Pipes are clogged! We've called the plumbers!
Well in the meantime a lot of people are talking about it (Techmeme). Tim O'Reilly has a long piece about it on his Radar blog. He starts with,
Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as “an interactive feed aggregator and manipulator” that allows you to “create feeds that are more powerful, useful and relevant.” While it's still a bit rough around the edges, it has enormous promise in
turning the web into a programmable environment for everyone.
In agreement, but I'm worried Yahoo might be focusing too much on aggregation that general purpose pipelining of any data source online. Tim then talks about why he's excited and points at some of my also favorite posts in this area. Jon Udell's keynote at the 8th Python conference and the JavaOne keynote which really gelled with my thoughts about
Pipelines at the time. This is also another reason why I got fed up of the Gillmor Gang without Jon Udell. Anyway back to Tim's post, here's a couple of other things I found interesting.
But perhaps more significantly, to develop a mashup, you already needed to be a programmer. Yahoo! Pipes is a first step towards changing all that, creating a programmable web for everyone.
This is certainly very true, coming from a design background I just couldn't understand why pipelines were not used more in application development. I actually thought the move towards objects in programming would be the start of this, but I guess not.
Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique (with the “ue” this time:-), count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops.
RSS and XML are easy targets for a beta service. But whats really needed is more input adapters. Microformats, FOAF, S5, WebAPIs, XMPP, etc. The transformers are predictable bar the user-defined filters (which I would assume would be XSL?). There's other services like RSS Mix and Feed Rinse which do the same thing. Chris is right filters are old hat.
Talking of Chris, in his post he seems quite down on his own pipeline: Touchstone. Personally I think their further down the line because the interesting part of the pipeline is being able to mix local and remote content not just remote. Also the widget style UI is very powerful. You could use Yahoo Pipes and I guess Yahoo Widget Engine to create something like Touchstone but your missing the Relevancy engine (APML) which did a great job of finding me screenshots of Windows
Mobile 6.
I'm a little worried about the focus on the GUI used for Yahoo Pipes. It sounds good but there needs to be thoughts about interopability. I don't want to create a great Pipeline and then be locked into Yahoo Pipes forever more.
Anyway, I can't talk much more about it till I get a chance to play with it first hand. Good work Yahoo.
If you want prove design can make a huge difference, check out trustedplaces.com's new web 2.0 design. Its something very beautiful and ever so sweet to use. Also very clever putting the survey in after you login. My drink selection says it all. Good work guys, now I feel so much happier logging in and using the site.
I think it was about a year ago when Yahoo tried to sweet talk everyone who used Flickr into upgrading to a Yahoo ID. Old Skool flickr members revolted and staged a large protest. Well its happening again, I wonder if there will be a large protest again? I just received this email from Yahoo.
Dear Old Skool Account-Holding Flickr Member,
On March 15th we'll be discontinuing the old email-based Flickr sign in system. From that point on, everyone will have to use a Yahoo! ID to sign in to Flickr.
We're making this change now to simplify the sign in process in advance of several large projects launching this year, but some Flickr features and tools already require Yahoo! IDs for sign in — like the mobile site at m.flickr.com or the new Yahoo! Go program for mobiles, available at: http://go.yahoo.com.95% of your fellow Flickrites already use this system and their experience is just the same as yours is now, except they sign in on a different page. It's easy to switch: it takes about a minute if you already have a Yahoo! ID and about five minutes if you don't.
You can make the switch at any time in the next few months, from today till the 15th. (After that day, you'll be required to merge before you continue using your account.) To switch, start at this page:
http://flickr.com/account/associate/
Nothing else on your account or experience of Flickr changes: you can continue to have your FlickrMail and notifications sent to any email address at any domain and your screenname will remain the same.
Complete details and answers to most common questions are available here: http://flickr.com/help/signin/
Thanks for your patience and understanding – and even bigger thanks for your continued support of Flickr: if you're reading this, you've been around for a while and that means a lot to us!
Warmest regards,
– The Flickreenos
So as Neil and others have pointed out, the Yahoo/Flickr protest is back and this time Yahoo don't seem to be rolling over. So whats my beef with Yahoo? Well let me tell you in a couple of points.
I just posted up a review of Touchstone's Alpha on my pipeline blog. While I can't put my finger on exactly what it is, it's self described as a alerts/updates and attention management platform. What ever that is – it is, it's certainly a move forward beyond the standard RSS reader or online aggregator. I just can't wait for it to use less resources and
add more adapters.
Technorati Tags: alpha, atom, attention, banner, desktop, pipeline, platform, rss, flow, touchstone
When you see it like this it makes a lot of sense. But I've had to set 3 messages almost exactly the same 3 times over. I should be able to set it once and that be the end of it – surely?
You can read more about this on my Flow * blog
meta-send-pingbacks=true meta-auto-trackback=true
Technorati Tags: api, im, interop, jabber, message, notification, presence, skype, twitter
So I've been pretty quiet about 24ways to impress your friends this year. The reason why is because I've been writing a tip titled Making XML beautiful again to go into the 24ways collection for 2006. The tip centres around a client side XSL transformation on a ATOM feed. I thought this would be simplier that RSS because I would have to create templates to support RSS 0.91, 0.92, 1.0 and 2.0. ATOM 1.0 feeds are also very much a like so a safe ground to start on (although a lot of ATOM 0.3 feeds use a different date element).
The actual XSL took me all of about 5mins to write but the explaining took a good few weeks. I have spell checked it, grammer checked and run it past the eyes of Sheila (my XSL friend). Sheila helped a lot on making it sound less like me talking and more like me writing, but there is still bits of my twisted humour in there. I also wanted to explain the difference between client-side and server-side transformations but decided it was out of scope. As was spending 3 paragraphs on what XSL is, which I finally cut down to 1. There is also something else which has been bugging me while writing the tip. Firefox 2.0, I can not work out if its actually broken when it comes to client side transformations or not. Some people seem to think so, but I'm getting just odd results like the output escaping not working. I've tried to install Firefox 1.5 in a Virtual Machine but I can't get it online (for many reasons). So I'm currently loading up a old Ubuntu Boot CD on a spare machine. I'm sure the comments will come flooding in soon.
Anyway big thanks to Drew McLellan for thinking about me when relaunching the 24ways project this year. Maybe Drew took it to heart when (in a good way of course) I asked why he didn't use XSL for parsing Microformats at BarCampLondon. I was still amazed he used a non rules based language to parse Microformats. It shows talent. Thanks again Drew, I just hope it comes across as well, as the other excellent authors on 24ways..
After my huge mind dump for Pipelines, I've now decided to start a new blog called Flow *. It will be about everything to do with pipelines. I'll sometimes cross post some gems here but generally the flow blog will form part of my research and development into pipelines.
I've been sitting on this blog post for bloody ages. But Tom's post has tipped me over.
Want to see something cool that's coming soon? Take a gander at XProc – the XML Pipeline Language. It's a way of defining a series of processes that operate on an XML file – for instance, running it through XInclude, schema validation, XSLT and making choices etc. It is great in as much as it's abstracting yet another layer out of the processing systems (SAX, DOM etc.) and their implementations (Java, PHP etc.) – obviously there are problems with that. Norman Walsh says that it's quite likely to be finished early next year. Kurt Cagle of XML.com thinks this is a good thing, and should fit in to the XML+REST ecosystem nicely.
So I've been thinking about some presentations and talks I'm planning on giving next year. I can't quite put my finger on the exact term but I know through blogging it and being very open about my thoughts I might reach a set of conclusions or at least points worthy of talking about with others..In my usual style a lot of the stuff is scattered around all over the place, so I'm going to try and use a wiki or something else to tie things togther.
My abstract for Etech 2007, which didn't get accepted.
API's are a great way of developers being able to access data and content from one provider. But with the trend of the mash-up has come the ability to join two or more providers together to the benefit of the user. This level of interoperability means people can start offering automation and new business opportunities by chaining services together. As many of us look towards the social benefits of a somewhat centralised Web 2.0, I can see how our single provider habits will be broken by the user generated pipelines.
Like Unix Pipelines, a user generated pipeline can be used to send content through a series of pipes. But unlike UNIX pipelines these pipes can be a series of remote or local webservices, services, applications, transformers, etc. A simple example could be, uploading a photo from your mobile phone to Flickr, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. Thats the magic of pipelines.
This is not a new concept but how we manage this has existed in the domain of Apple-scripters, Perl and Python hackers. Automator by Apple is an example of this, but fails due to its proprietary nature.
I'm proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. A whole eco-system will grow out of this decentralised user driven behaviour, which I call Web 2.5.
The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request.
This is all possible now with simple AppleScript or some other scripting language like Perl but requires a intimate knowledge of the scripting language. A user generated pipeline would be the higher level language to describe the Flickr example
Addy Santo of Santomania once wrote this quite fantastic application called Blogwave which he has not been updated for at least 3 years now. Its a multi purpose .net application which can consume RSS feeds (generator), transform them with some parameters like sort. It would then send them somewhere else, for example FTP, Email, SMB, etc in a RSS or Text form. What I found interesting about it was actually, it would create timed batch tasks in the standard Windows scheduler (something not many people use on there desktop). So in actual fact, it was a GUI for the command line in Windows. The application was a head of its time and unfortually not open source, so its kind of died but can still be used if you find the right link. But the concept is key, a GUI creates scripts or manages the complex pipeline process. The different pipes are already defined so you don't need some low level code to manage it. It seems Touchstone will take over from where Blogwave went, but I'm not on the alpha programme so I can't actually play with it.
I have tons of other examples but I'm now saving them up for the wiki and for my talk at Xtech 2007 which I'm currently rewriting my failed Etech proposal for right now.
After hearing Ryan Freitas talk about Twitter among other things, I've been trying out it out. If you don't know what Twitter is, check out Derek Powazek's description.
Twitter lets me SMS to a group all at once and creates a handy 'what I'm up to right now' insert for my site. A kind of in-situ, realtime, status message blogging. Fun!
So yeah its a good way to send short timely messages to a pool/group of friends. Those messages can be text messages (sms) or even im using jabber. It sounds really good but it has problems. Tantek posted up shots of ways to improve the pretty poor signup process. But I've been having problems registering my work mobile phone which is on O2 with Twitter. What also bugs me is the fact you can only register one mobile number. Yes I know its rare people have more that one mobile, but those who would use Twitter are much more likely to have 2 phones or 2 lines. My other issue is around adding your friends. Please please, allow me to add either my flickr network, upcoming network (why flickr and upcoming don't interop, is still beyond me), Plaxo contacts or even a Foaf file. I just can't be bothered to setup another network of friends in Twitter.
I do see the use of Twitter, specially for setting up adhoc geekdinners and getting people organised. But I would prefer to see Twitter used as an output point for messages. So I could send messages to it from almost any application or service and it would amplfy it to a certain group. Maybe it can do this already, it has an API so its certainly possible.
My username on Twitter is cubicgarden under the name of ian forrester, as you'd expect.
David Czarnecki has wrote a Twitter Plugin for Blojsom 3.0. Its currently in CVS but it will update twitter when there is an update or new entry to a blojsom blog. Damm I need upgrade to blojsom 3.0.
I wish I had a XML desktop. Why? Well I really want to do some of things which can be easily done online, with my desktop machine too. Kind of application development using web technologies. Yes it would be slower that writing in C++ or something like that, but it would mean more people could write stuff for there machines. So whats prompted this observation? Well I'm already thinking about this for my Xtech Proposal (which I should be working on instead of blogging) plus I saw the OSX atmosphere concept on electro plankton just recently.
So what I'm thinking is this is kind of possible using a weather feed or api and Flickr pictures. Obviously I've not seen the application running but the general idea of what the wather is like outside on your desktop is very achiveable. RSS Screensavers currently are pretty lame, for example my current favorate displays the headlines and a random picture from the local machine. But it will also display markup as actual non escaped markup, yeah sucks when you get a feed with pictures. You would have thought it would be clever enough to display that picture or something. I mean imagine subscribing to a feed like engadget or gizmodo which are heavy with high quality images. The experience would be a lot different.
But back to the XML desktop idea, yes Vista with XAML looks/sounds like what I'm after but knowing Microsoft its not going to live up to the promises. Geez this is certainly geting close to my proposal but wouldn't it be great to have read only (at the moment) XML feeds for commonly used APIs on your own machine? Its kind of like a widget engine and how they make common desktop api's available for use. Well extend that out so you don't have to build just widgets. I know for a fact this has security and privicy implications but say we could find a way through those very serious concerns? Wouldn't that be fantastic?
Update: It looks Adobe'e Apollo could be the solution to this? Thanks Gareth for the heads up on this. I started think about Apollo a lot more while reading the PDF and thought of the parallels between it, Xulrunner and a widget engine. And came up with this matrix.
Jeremy Keith writes about how everything he uses outputs RSS of some kind which can easily be mashed up. Yes this is pretty straight forward and I hope commonly known now but what prompted me to blog was this bit.
At the recent Take Back The Web event here in Brighton, Rob Purdie talked about RSS being the vaseline that’s greasing the wheels of Web 2.0. He makes a good point.
Over the course of any particular day, I could be updating five or six RSS feeds, depending on how much I’m blogging, how many links I’m posting, or how much music I’m listening to. I’d like to take those individual feeds and mush ‘em all up together.
I think were finally at the stage where its accepted that RSS and ATOM can be like RESTful API's. I remember having a email exchange with Jeff Barr about this and he disagreed. Well I'm sorry but it looks I was right.
What Jeremy also talks about is why I love XSL so much. As long as its valid XML and web accessable I can do something with it. I've been asked to be involved in a special project for Christmas to do with XSL, so look out for that soon.
Quick heads up on where to get really iindepth information on the O'reilly Web 2.0 Summit which just finished recently. Richard MacManus from the Read/Write Web has tons of posts which seem to cover pretty much everything. His wrap up post links to everything he's written in notes.
I also have to point to Ben Metcalfe's post about his ignoyance about the people attending the web 2.0 summit. Richard also picks up on this.
Some folks, like Ben Metcalfe, think the conference has lost its edge. Maybe it has, but the Web 2.0 Summit is a different beast now than it was last year – and that's a sign of the times.
What is google up to? That seems to be the question on a lot of peoples lips. A while ago they did a deal with Sony, then bought YouTube with stock (this is the intimate details link). Not long ago Jotspot was bought by Google and now Google have launched there mobile gmail service. Its not that amazing, Plaxo has offered the same kind of service for quite some time but alas only for there paying users. However its going to be secure and free for anyone who has a Gmail account. Its a bit of a bum deal if you live outside north america and don't have a all you can reasonably eat data plan. But as the taoofmac describes, this could be the email for the masses. Once they start pushing calendar, contacts, etc through that same gateway, it will be even more useful. I hate to say it but this is what Plaxo do right now.
Generally Google is doing the loosey joined pieces pretty well. You can certainly see how everything will fit together in the future but right now, there very keen to just develop the pieces. There pieces are not always cutting edge and best of breed but there getting traction and gaining users.
Cheers Miles for a lot of the links.