Tag: pipelines

Hooray, audacity files are XML

Plumbing for the next web, by ian forrester

I’ve been looking for a way to create SMIL files with an editor for a while. Main reason being to speed up the creation of creating podcasts for the Perceptive Podcast client and make it easier for those who don’t understand markup/code.

One of the techniques we deployed during the Visual Perceptive Media project was to export final cut xml out of final cut/premiere pro then transform the lot with XSL/Python/etc to something else more usable. Its something I’ve had in mind for a long time, as you can see with this paper/presentation I wrote 12 years ago.

There was a point when Wmas, could create an editor for our director/writer (Julius) or allow him to use tools he was familiar with (non-linear editor like Finalcut/Premiere). Of course we choose the latter and converted the final cut xml (which isn’t really an official spec) into json using python. We were able to use markers and zones to great effect, indicating the interactive intentions of the director in a non-linear editor. This meant the intentions can exist and run completely through to the very end, rather than tacking it on at the end.

So with all that in mind, I started thinking if I could turn Audacity into a editor in a similar way? Is there a final cut xml format for audio? Thats when I came across this article which made perfect sense – Audacity files are just XML documents, sooo…

Structure of a empty project

<?xml version=”1.0″ standalone=”no” ?>
<!DOCTYPE project PUBLIC “-//audacityproject-1.3.0//DTD//EN” “http://audacity.sourceforge.net/xml/audacityproject-1.3.0.dtd” >
<project xmlns=”http://audacity.sourceforge.net/xml/” projname=”blank-audacity_data” version=”1.3.0″ audacityversion=”2.2.1″ sel0=”0.0000000000″ sel1=”0.0000000000″ vpos=”0″ h=”0.0000000000″ zoom=”86.1328125000″ rate=”44100.0″ snapto=”off” selectionformat=”hh:mm:ss + milliseconds” frequencyformat=”Hz” bandwidthformat=”octaves”>
<tags/>
</project>

Just the title ignited my mind, the actual content of the blog is less interesting but I realised I may have a free & open-source editor which runs on every platform and with a bit of XSL magic could be the start of the editor I was looking for? The idea of it being a pipe, which leads on to more is something which fits in the bigger pipeline chain

I also found a GIT project to Parse audio track times from an audacity .aup projects. Its uses XSL to do the processing, so I may spend a bit of time playing with it to make something useful.

Just need to dust off my old XSL development skills… Which reminds me what happened to XPROC (XML pipeline language)?

If this happens do this to my laptop…

I have already talked about IFTTT to death and how I liken it to some ideas and work I had around pipelining.

All the new movement in this area has been in the online space but I found this little app for Linux which operates in a similar fashion to the very old conduit (Conduit) but its focused around system events rather than webservices.

Cuttlefish is a tool which can execute various actions when specific events are triggered. For example, you can change the proxy mode depending on the currently connected wireless network, unlock your computer when a specific Bluetooth or USB device is connected or disconnected and so on.

I can easily see how webservices can be written into the application, although there is no roadmap yet.

Pipeling our reality

Hadn’t really thought hard about how the internet of things and pipe-lining work together. But listening to John Kestner on ITConversations recently, I’ve been re-thinking about some of thoughts around pipe-lining.

Those way back might remember I have a massive thing about pipeling born from my love of using Apache Cocoon and XML. Theres actually a very cool XML syntax called Xproc which I’ve yet to really check out in any detail.

I thought about ways to extend Yahoo! Pipes and declared my love for if this then that. But reaching out into the physical world is something else all together…

“Twine is a wireless module tightly integrated with a cloud-based service. The module has WiFi, on-board temperature and vibration sensors, and an expansion connector for other sensors.” Developer John Kestner describes the device and its development. He reviews how it works, what decisions were made in its design, and how It allows you to connect things to the Internet. He also discusses the community that is working to make the product better.

I saw pachube.com a while ago but never really blogged it or thought hard about it. But you can certainly think about how ifttt and pachube could work together really nicely.

The ability to automate and manipulate the physical world is something very special. I’d certainly like to investigate a little deeper in the near future.

If this then…. web pipelines are back again?

ifthisthenthat workflow setup for #workbetter tweets

I first read about if this then… on lifehacker today while riding into work on the tram.

If This Then That is a simple to use tool to keep your social networks, to-do lists, RSS feeds, imaging sites, and all your other webapps on the same page.

The core of If This Then That works off of different channels you can use to create your own recipes for sharing by using the basic instruction, “if this, then that.” For instance, you can set it so that if you text message something with a particular hashtag to a specified number, then it will automatically create a note in Evernote. You can also set it to automatically pull your photos off Instagram and save them to Dropbox, send you a text message when you have a Google Calender event, or even send you an email if it’s supposed to rain tomorrow. Provided you’re using one of the 35 different services and webapps currently supported, you can cross-link and integrate them in almost anyway you see fit.

It strikes me as a good stab at what I wrote about in 2006. Pipelines for the web…

That was quite some time ago and I’ve been hoping Conduit would get worked on but things have gone very quiet on that front. Then came along, Ping.FM which was also interesting for slightly different reasons before they got aquired. So I was surprised when I learned about IFTT, so I gave it a try.

So my first pipe or task, useful so I can save links from anywhere just by sending a correctly formatted instant messenger to the right bot.

my first task

I made it into a recipe, then decided to checked out the pre-existing recipes other people have made.

Here’s some of the ones I’m finding interesting…

So right now I’m pretty impressed with iftt but I’m hoping they add even more services soon. Don’t get me wrong the list is pretty huge right now, but I’d like to have some things like NewstoBook, DeskSMS, Top10, Google plus, etc…

Plumbing for the next web at Xtech 2007

I have uploaded my presentation, pipelines: plumbing for the next web fresh from the first day of Xtech 2007 today to Slideshare.

The general view is that the presentation went down well and made sense. However I think people really wanted to see something which worked instead of slideware.

Comments [Comments]
Trackbacks [0]

Titbits from the last few weeks

So there is a few things which I've twittered but not blogged recently. Rather that use up tons of entries with one liners (tom), I prefer to roll them into one large entry.

Linked-in

I caved in the other day when Plaxo added support for Linked-in. I had been thinking about it for years but choose not to be involved because I simply didn't see the point. The one which almost tipped me over the top was Linked-in support in Particls (touchstone). I wanted to see what extra information it might add to the APML file but didn't have an account to try it with. But now I do. I would just add, Chris and the team you should add linked-in support as a input adapter not bolt it on to the Feed adapter. This code might be useful too, because I and others might be able to hack together other input adapters for Flickr, Delicious, etc. I still want to play a lot more with my FOAF profile because I think thats so much more powerful. So anyway, you can link to me using my bbc email address or search for my name.

Particls

Recently something has gone wrong with Particls (touchstone) it happened when it tried to swap my account from Touchstone to Particls directory. I need to sort it out as my APML file is too neutral now and I'm not getting those good sharp posts coming through like before.  The RSS screensaver (sent to lifehacker btw) is also doing lots of repeating, so I suspect the Pebble output isn't work correctly. I got a feeling I might end up doing a complete post with screenshots because its changed so much since I last looked at it.

Trusted Places and Dopplr

I've been using reusing Trustedplaces recently when I go away to a different cities. Now I have trusted places in New York, San Francisco and of course London. I have also started using Dopplr which tracks your trips and shows your friends trips. Its really cool but I wish it would keep your old trips so you can compare in the future and maybe hang things off of it like flickr pictures, trusted places reviews, etc. I look forward to when Trustedplaces and Dopplr get APIs or tons of feeds.

Dopplr my map

Xtech: Pipelines

I'm wrote the paper for Xtech recently and I'm in the process of writing the presentation. I'm still playing with my flow * setup but the presentation is the most important thing. I'm hoping to get the Eurostar to Paris but it looks like I'll be getting a plane because the BBC don't have a eurostar account. I'm in Paris for a week but the last part will be with friends from Minneapolis.

Comments [Comments]
Trackbacks [0]

Proposal accepted for XTech 2007 – The Ubiquitous Web

What was waiting for me in my inbox today…

To: Ian Forrester

We are pleased to accept the following proposal for XTech 2007.

  • Pipelines: Plumbing for the next web

It has been scheduled for 16:45 on 16 May 2007.

Please confirm that you have received this acceptance and can deliver the presentation.

Thank you,
Edd Dumbill

So my presentation at BarCampLondon2 will be a very early draft for whats to come in May.

Comments [Comments]
Trackbacks [0]

Yahoo catches on to the idea of internet pipelines

Yahoo Pipes

I can't believe I missed Yahoo's Pipeline beta. Chris from Touchstone actually dropped me a email and asked if I've seen it. But all I get now is…

Our Pipes are clogged! We've called the plumbers!

Well in the meantime a lot of people are talking about it (Techmeme). Tim O'Reilly has a long piece about it on his Radar blog. He starts with,

Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output. Yahoo! describes it as “an interactive feed aggregator and manipulator” that allows you to “create feeds that are more powerful, useful and relevant.” While it's still a bit rough around the edges, it has enormous promise in
turning the web into a programmable environment for everyone.

In agreement, but I'm worried Yahoo might be focusing too much on aggregation that general purpose pipelining of any data source online. Tim then talks about why he's excited and points at some of my also favorite posts in this area. Jon Udell's keynote at the 8th Python conference and the JavaOne keynote which really gelled with my thoughts about
Pipelines at the time. This is also another reason why I got fed up of the Gillmor Gang without Jon Udell. Anyway back to Tim's post, here's a couple of other things I found interesting.

But perhaps more significantly, to develop a mashup, you already needed to be a programmer. Yahoo! Pipes is a first step towards changing all that, creating a programmable web for everyone.

This is certainly very true, coming from a design background I just couldn't understand why pipelines were not used more in application development. I actually thought the move towards objects in programming would be the start of this, but I guess not.

Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique (with the “ue” this time:-), count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops.

RSS and XML are easy targets for a beta service. But whats really needed is more input adapters. Microformats, FOAF, S5, WebAPIs, XMPP, etc. The transformers are predictable bar the user-defined filters (which I would assume would be XSL?). There's other services like RSS Mix and Feed Rinse which do the same thing. Chris is right filters are old hat.

Talking of Chris, in his post he seems quite down on his own pipeline: Touchstone. Personally I think their further down the line because the interesting part of the pipeline is being able to mix local and remote content not just remote. Also the widget style UI is very powerful. You could use Yahoo Pipes and I guess Yahoo Widget Engine to create something like Touchstone but your missing the Relevancy engine (APML) which did a great job of finding me screenshots of Windows
Mobile 6.

I'm a little worried about the focus on the GUI used for Yahoo Pipes. It sounds good but there needs to be thoughts about interopability. I don't want to create a great Pipeline and then be locked into Yahoo Pipes forever more.

Anyway, I can't talk much more about it till I get a chance to play with it first hand. Good work Yahoo.

Comments [Comments]
Trackbacks [0]

Pipelines and the flow of automation

Water Pipes

I've been sitting on this blog post for bloody ages. But Tom's post has tipped me over.

Want to see something cool that's coming soon? Take a gander at XProc – the XML Pipeline Language. It's a way of defining a series of processes that operate on an XML file – for instance, running it through XInclude, schema validation, XSLT and making choices etc. It is great in as much as it's abstracting yet another layer out of the processing systems (SAX, DOM etc.) and their implementations (Java, PHP etc.) – obviously there are problems with that. Norman Walsh says that it's quite likely to be finished early next year. Kurt Cagle of XML.com thinks this is a good thing, and should fit in to the XML+REST ecosystem nicely.

So I've been thinking about some presentations and talks I'm planning on giving next year. I can't quite put my finger on the exact term but I know through blogging it and being very open about my thoughts I might reach a set of conclusions or at least points worthy of talking about with others..In my usual style a lot of the stuff is scattered around all over the place, so I'm going to try and use a wiki or something else to tie things togther.

My abstract for Etech 2007, which didn't get accepted.

API's are a great way of developers being able to access data and content from one provider. But with the trend of the mash-up has come the ability to join two or more providers together to the benefit of the user. This level of interoperability means people can start offering automation and new business opportunities by chaining services together. As many of us look towards the social benefits of a somewhat centralised Web 2.0, I can see how our single provider habits will be broken by the user generated pipelines.

Like Unix Pipelines, a user generated pipeline can be used to send content through a series of pipes. But unlike UNIX pipelines these pipes can be a series of remote or local webservices, services, applications, transformers, etc. A simple example could be, uploading a photo from your mobile phone to Flickr, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. Thats the magic of pipelines.

This is not a new concept but how we manage this has existed in the domain of Apple-scripters, Perl and Python hackers. Automator by Apple is an example of this, but fails due to its proprietary nature.

I'm proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. A whole eco-system will grow out of this decentralised user driven behaviour, which I call Web 2.5.

flickr authentication list

The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request.

This is all possible now with simple AppleScript or some other scripting language like Perl but requires a intimate knowledge of the scripting language. A user generated pipeline would be the higher level language to describe the Flickr example

blogwave sources

Addy Santo of Santomania once wrote this quite fantastic application called Blogwave which he has not been updated for at least 3 years now. Its a multi purpose .net application which can consume RSS feeds (generator), transform them with some parameters like sort. It would then send them somewhere else, for example FTP, Email, SMB, etc in a RSS or Text form. What I found interesting about it was actually, it would create timed batch tasks in the standard Windows scheduler (something not many people use on there desktop). So in actual fact, it was a GUI for the command line in Windows. The application was a head of its time and unfortually not open source, so its kind of died but can still be used if you find the right link. But the concept is key, a GUI creates scripts or manages the complex pipeline process. The different pipes are already defined so you don't need some low level code to manage it. It seems Touchstone will take over from where Blogwave went, but I'm not on the alpha programme so I can't actually play with it.

Touchstone

I have tons of other examples but I'm now saving them up for the wiki and for my talk at Xtech 2007 which I'm currently rewriting my failed Etech proposal for right now.

Comments [Comments]
Trackbacks [0]