Pipelines and the flow of automation

Water Pipes

I've been sitting on this blog post for bloody ages. But Tom's post has tipped me over.

Want to see something cool that's coming soon? Take a gander at XProc – the XML Pipeline Language. It's a way of defining a series of processes that operate on an XML file – for instance, running it through XInclude, schema validation, XSLT and making choices etc. It is great in as much as it's abstracting yet another layer out of the processing systems (SAX, DOM etc.) and their implementations (Java, PHP etc.) – obviously there are problems with that. Norman Walsh says that it's quite likely to be finished early next year. Kurt Cagle of XML.com thinks this is a good thing, and should fit in to the XML+REST ecosystem nicely.

So I've been thinking about some presentations and talks I'm planning on giving next year. I can't quite put my finger on the exact term but I know through blogging it and being very open about my thoughts I might reach a set of conclusions or at least points worthy of talking about with others..In my usual style a lot of the stuff is scattered around all over the place, so I'm going to try and use a wiki or something else to tie things togther.

My abstract for Etech 2007, which didn't get accepted.

API's are a great way of developers being able to access data and content from one provider. But with the trend of the mash-up has come the ability to join two or more providers together to the benefit of the user. This level of interoperability means people can start offering automation and new business opportunities by chaining services together. As many of us look towards the social benefits of a somewhat centralised Web 2.0, I can see how our single provider habits will be broken by the user generated pipelines.

Like Unix Pipelines, a user generated pipeline can be used to send content through a series of pipes. But unlike UNIX pipelines these pipes can be a series of remote or local webservices, services, applications, transformers, etc. A simple example could be, uploading a photo from your mobile phone to Flickr, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. Thats the magic of pipelines.

This is not a new concept but how we manage this has existed in the domain of Apple-scripters, Perl and Python hackers. Automator by Apple is an example of this, but fails due to its proprietary nature.

I'm proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. A whole eco-system will grow out of this decentralised user driven behaviour, which I call Web 2.5.

flickr authentication list

The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request.

This is all possible now with simple AppleScript or some other scripting language like Perl but requires a intimate knowledge of the scripting language. A user generated pipeline would be the higher level language to describe the Flickr example

blogwave sources

Addy Santo of Santomania once wrote this quite fantastic application called Blogwave which he has not been updated for at least 3 years now. Its a multi purpose .net application which can consume RSS feeds (generator), transform them with some parameters like sort. It would then send them somewhere else, for example FTP, Email, SMB, etc in a RSS or Text form. What I found interesting about it was actually, it would create timed batch tasks in the standard Windows scheduler (something not many people use on there desktop). So in actual fact, it was a GUI for the command line in Windows. The application was a head of its time and unfortually not open source, so its kind of died but can still be used if you find the right link. But the concept is key, a GUI creates scripts or manages the complex pipeline process. The different pipes are already defined so you don't need some low level code to manage it. It seems Touchstone will take over from where Blogwave went, but I'm not on the alpha programme so I can't actually play with it.


I have tons of other examples but I'm now saving them up for the wiki and for my talk at Xtech 2007 which I'm currently rewriting my failed Etech proposal for right now.

Comments [Comments]
Trackbacks [0]

Like watching a baby playing with a loaded gun…

Baby face closeup

…Is what Miles said about me setting up my own virtual private server on the weekend. Yep I finally took bull by the horns and slapped down my credit card and decided to go with Hub.org for Cubicgarden.com's new resting place. To be fair I didn't really know what I was getting myself into. See I kind of thought Tomcat and Apache would be installed and ready to go. But nope I finally logged into my FreeBSD box and quickly found out that it was a barebones box and I would need to do the configuration of applications, permissions and users. Well trust me, this is no easy thing. I mean there something very different about running unix on the desktop and running it as a server. In a server environment permissions and applications running all need to be kept under tight wraps. I would agree this should be the case for a Desktop environment too, but you can be a little more flexable with the configuration of a desktop machine. Put it this way, being a admin with root access to your own server is certainly compareable to building your first F1 car then racing it along the streets of Monte Carlo. Or as Miles puts it a baby with a loaded gun.

Either way, with thanks to Miles and tons of resources online like this one, I'm almost up, running and hopefully pretty secure. Rather than the usual Apache 2.x and Tomcat 5.5.x type configuration with mod_jk, I've gone for Tomcat 5.5.x with Pen in front for a reverse proxy and load balancer. Miles suggested Pound and Balance but I couldn't get Pound to compile without seriously messing with OpenSSL and Balance didn't seem to forward HTTP traffic without stripping away the header information. Pen is just like Pound it would seem, but also runs on Windows which is good to remember for other projects I may have in mind with my old Windows 2000 box. So yeah its a pretty sweet setup so far and means I loose the overhead of running Apache when all I really want is Tomcat. By the way, I was very close to installing Resin 3.x but decided against it for now.

Compiling Cocoon only took 2 mins once I finally untared and gunzipped the source. Can I just say how much of a nightmare Compression is? This guide was very useful for not only uncompressing files (tar -xvvzf cocoon2.1.8.tar.gz) but also compressing them. It took me a while to work out the correct parameters to compress a directory of files and its contains but keep the permissions and modified dates (which is extreamly useful for moving blojsom blog entries) . tar -cRvzf archive.tar foldertocompress/. Anyway, Cocoon is running happily in Tomcat now and Blojsom is also running fine with everything this blog has up till Feb 26th. So I'll have to do another update just before I swap over to the new server. This will also mean there will be a period of maybe 2 days when the blog and RSS feeds may time out or seem out of date. Don't worry I'll warn you in advance of the exact day.

So what next on the horizon? Well I need to do some more securing and enable Log4J on Tomcat and Cocoon. I've also still got to sort out basic Unix type things. For example while I was setting up the server, the only text editor I had was vi and the only shells were tsch and some other weird ones. Yep thats right no Nano or Bash. I don't know how I managed, but trust me I'll be avoiding vi when ever possible. I've already chpass all the users and made Bash the default shell. Beyond this, I'm considering Hamachi for Linux which would mean I could securely login to Blojsom, Tomcat and anything else from anywhere without setting up that crazy port forwarding in Putty. This sounds over kill but I'm tempted to at least run Hamachi on my Smoothwall Firewall server at home.

In regards to Cocoon, well my next step which I had planned to do if I was not writing this long blog post, would be to install Saxon 8.7 (good to see a .net version btw) in Cocoon using this guide (I know it works, i already installed Saxon 8.4 on the development machine at work). While with Blojsom, I will start trimming down some of the outstanding issues I had.

Oh before I finish, did I say how great Wget and Sudo are? Loaded gun indeed.

Comments [Comments]
Trackbacks [0]