Easy! mashups! with! Pipes! (and PEE-POO-BUM)

echo hello | IT_Blogwatch: in which we look into Yahoo! Pipes. Not to mention the world's most unfortunately coded airports...

Juan Carlos Perez reports:

Yahoo Inc. has launched a service called Pipes designed to let regular users mix different RSS and Atom feeds and create data "mashups," a process that so far has required programming knowledge. Pipes features a drag-and-drop interface that the company hopes non-technical users will find simple and intuitive as they manipulate content syndication feeds to combine data in new and useful ways.

An example of a Pipe is this one, which meshes listings from Craigslist with data from Yahoo's local search engine to display apartments for rent near any business ... Another one collates news about topics chosen by the user from a variety of sources. While Pipes today lets users mix data from RSS and Atom feeds, Yahoo hopes to extend the service to support other data formats, Web services, processing modules and output renderings

Jeremy Zawodny scribbled this random note:

For far too long now RSS has been used in ways that don't really tap its true potential. Being able to syndicate my favorite headlines or blog posts is great. In fact, it helped to kick off a revolution in personal on-line publishing that is still growing and evolving. But I want so much more ... It's not for lack of data either. You can get RSS output from lots of non-news and non-blog stuff. Everything from classifieds on eBay and craigslist to Bugzilla, Wikis, and so on.

The problem has been a lack of good tools for pulling it all together. In the Unix world, we often connect sources of data to filters and utilities using pipes. A pipe is a way of constructing ad-hoc workflows composed of any number of inputs, filters, and manipulation tools. And the beauty of the whole system is that they all use a very simple input and output method, so there's a nearly infinite set of ways you can combine and recombine them.

On the web, however, it's harder. There are data sources and feeds, but until now we've had no pipes! Pulling together and integrating data sources using JavaScript isn't on the client for the faint of heart. The browser isn't the same as a Unix command-line, so building mashups has been more frustrating and time consuming that it needs to be--especially for Unix people like me. Yahoo! Pipes is one of a very small set of completely amazing on-line data manipulation and data mashup environments that can really change the way we work with on-line data sources. (The others are DabbleDB and Dapper.)

Tim O'Reilly:

Yahoo!'s new Pipes service is a milestone in the history of the internet. It's a service that generalizes the idea of the mashup, providing a drag and drop editor that allows you to connect internet data sources, process them, and redirect the output ... While it's still a bit rough around the edges, it has enormous promise in turning the web into a programmable environment for everyone ... I'm so excited. This is something I've been waiting nearly ten years for.
...
Mashups have generally been limited in their scope, pairwise combinations with their output typically being simply another web site. That is, the pipes and filter mechanism had not been generalized. But perhaps more significantly, to develop a mashup, you already needed to be a programmer ... [but] using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique, ... count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops. In short, it's a good start on the Unix shell for mashups.

Edward Ho will work for Yahoo's doughnuts:

I’m pleased to say that we’ve got an early beta of Pipes available today. This is a tool for developers to remix data on the web and we are hoping that you find it useful. Please bear in mind that the initial version is here for feedback, so don’t hesitate to use the links on the site. I’m really looking forward to where this service will lead.
...
As for me, my favorite pipe is European Performance Car News. When Pasha Sadri first told me about his idea, I immediately wanted to have this pipe. It pulls automotive news from my favorite sites (Autoblog, Fourtitude, and Jalopnik) and filters them for the makes that I’m interested in. It’s a simple pipe, but it’s an example of how you can use pipes to quickly scratch an itch you’ve been having.

Chad Dickerson has more background:

I work in the row of cubes right beside the team and it’s been inspiring to watch Pipes go from idea to reality ... [here's] a mashup that I built. The most exciting thing is that I didn’t really have to know how any of the APIs worked that I used — they are rolled into the product and all I have to do is feed data into the mix from my choice of RSS feeds, set up the pipeline with various parameters, and the data I want transforms and flows out of the other end like magic ... I took the Upcoming.org RSS feeds for the Fillmore and the Warfield in San Francisco (two music venues) and joined them into one feed, then piped the unified feed through the Content Analysis Term Extraction API to pull out the keywords in the RSS feeds. Then I looped through the Flickr API to run queries for photos on those keywords. My logic wasn’t perfect, but all I know is that I saw a photo of George Clinton and sure enough, he’s playing at the Fillmore on March 9. It’s 1:20 am as I write this, and although I have been meaning to go to bed for about three hours now, I’ve just been having too much fun reconfiguring these pipes. Yes, fun.

Matt Cutts' got an itch to scratch:

What’s an example use case? Well, I’ve got one right here in my back pocket. Suppose you had multiple RSS calendar feeds, and wanted to combine those multiple feeds into one feed. Yahoo Pipes has a “union” operator, so I’m assuming you could pipe in (say) four feeds and get back one feed that was a superset of all the items that you fed in. A couple months ago, I went searching for a package that did set operations on RSS feeds (specifically looking to combine multiple different calendar feeds), and didn’t really see anything great at that point. This tool would solve that problem.

It's Richard MacManus's pleasure:

The real power is in the filtering options. Opening up the editing environment reveals a lot of choices to manipulate the individual feeds ... The idea is to then subscribe to the remix feed in your favorite Feed Reader. This is a neat service from Yahoo and I'll be playing around with it more tonight! The UI seems a little geeky and kind of reminds me of Ning (not sure if that's a compliment or not, as Ning never took off). But I've long thought that RSS remix feeds are the future of RSS - and certainly one way to try and filter information overload. So this is a great move by Yahoo to release an RSS remix service to the early adopter crowd.

Chris Mohney brings us back to Earth:

Looks nifty, anyway, though a trifle nerdy. Have someone from IT look into it.

Buffer overflow:

Around the Net

• Kelly Conley: The Unexpected – A Spin-Off of the "Tricky New Spam Tactic"
• Michael Santo: Ban Proposed on iPod Listening While Crossing Streets
• Jean Hollis Weber: Plugins for converting between OpenDocument and Microsoft Office file formats
• Linux-Watch: Linux hackers tackle WiFi hassles
• Peter Lattman: Revolving Door: A New GC at HP & a Former MA AG Joins GT
• Aaron Adams: Month of Apple Bugs Overview
• Paul McNamara: Did Gates fib about H1-B business?
• Mike,Techdirt: Amazon Patents Showing You Your Own Books
• Ryan Paul: Rumors of Novell Linux ban greatly exaggerated

Around Computerworld

• Robert L. Mitchell: HP to IBM: Take that!
• Computerworld TechCast: Flash Animation
• Robert L. Mitchell: Printer sharing glitch stumps the chump
• Martin McKeay: Administrator at RSA
• David Haskin: Microsoft: Make WinMo 6 more usable
• Shark Tank: Power play
• Douglas Schweitzer: The insider threat "should" be easy to spot!
• Shark Bait: I doesn't work!
• Shark Bait: Internet On The Air (OH YEA!!)
• Shark Bait: I told You So!
• Shark Bait: Right-click or left-click?
• Shark Bait: Transport Claim

Previously in IT Blogwatch

• We have fun at RSA conference in SF (and Dr. Python)
• Microsoft's anti-virus failure (and V-day)
• Apple <3 Apple (and Super Ads)
• Novell SuSE Linux threat from FSF ban (and code != comments)
• Dell's eponym-CEO returns (and DOS sub shock)
• Older posts

And finally... Destinations of Beavis & Butt-head Airlines

Richi Jennings is an independent technology and marketing consultant, specializing in email, blogging, Linux, and computer security. A 20 year, cross-functional IT veteran, he is also an analyst at Ferris Research. Contact Richi at blogwatch@richi.co.uk.