A Blog About Making Culture
Yahoo Pipes
February 8, 2007
Background: Yahoo's launched an interesting and innovative new service, Pipes, which lets users with a relatively low degree of technological expertise combine structured sources of web data such as feeds. In this way, it's possible for non-experts to create new web services for their own use or for public consumption. Pipes combines a remarkably sophisticated development environment with some core social features such as the ability to clone or share the web services you produce. The service is fairly approachable, but somewhat complex once you get just under the surface, and should be moderately successful while radically raising the bar for other tools in its category.
Some quick links if you're interested in this topic:
- Plagger: an open-source, installable feed routing system created by Tatsuhiko Miyagawa which performs much of the core functionality of Pipes and is customizable, but lacks the user interface and integrated development environment (IDE) which distinguish Pipes. (Disclaimer, for what it's worth: I work with Tatsuhiko. But Plagger's free and not a Six Apart product, so... shrug.)
- Ning: Perhaps the archetypal social application platform for the web. Headed by Gina Bianchini, Ning has thus far defined the feature set for end-user creation of web applications, though the focus has not been on creating web services.
- An introduction to Unix pipes. Not merely the inspiration for the name of pipes, the powerful idea of routing data through a series of loosely-connected applications is one of the core concepts that powers most lightweight automation and data processing by non-programmers.
- The Mario-inspired image you see here is available on a Threadless T-shirt. My wife has one and it is very cute.
So, what is Pipes?
Okay, with all the introduction out of the way -- what the heck is Pipes? Yahoo's overview page offers the following explanation:
Pipes is a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.
In practice, this means you can use Pipes to pick a few feeds or APIs to retrieve data from, set up rules for processing the data which is provided by those feeds, prompt for user input in your processing, and then output the processed results as another feed or object format for use in your own applications. In the simplest case, you can apply some straightforward rules to a feed and then subscribe to the end result.
Pipes also has a full web-based IDE. Even more amazingly, the IDE is incredibly rich and powerful, with the usual complement of editing tools you'd expect from a visual editor in a desktop application. I'd first seen a browser-based IDE for editing LiveJournal's S2 style templates, but even that impressive effort pales next to the Pipes IDE. I'll never learn all of the Pipes IDE's features, but I will always admire its gee-whizzery.
Passing the Pipe
Most importantly, and perhaps most key to the success or failure of Pipes, are the social functions that underpin the application. With Pipes, it's easy to make your own web services public, to clone web services that others have made, or to offer your own services for others to clone. That element of social sharing of code, first pioneered by platforms like Ning, makes the open source ethos much simpler to participate in. Instead of setting up complex version control systems and submitting patches to a central repository, application cloning works on a principal of infinite forking, taking the idea of embracing failure and building it into the platform. Code 'em all, and let blogs sort 'em out.
There's also another key accomodation of social functionality: Pipes is pretty. As I mentioned in the introduction, much of this type of functionality is technologically possible with tools like Plagger. But, much as I love Plagger, I just don't have the patience to install half of CPAN to get it running just so I can hand-code an application on top of it. And that's even though I think it's a cool idea -- imagine if I weren't already familiar with the concept of routing feeds around.
Pipes is attractive without being overly pretty; There are the requisite nods to Web 2.0 design (it's blue!), but overall the site is refreshingly straightforward. The IDE is, frankly, a little cluttered unless you're running on a gigantic monitor, but that's been true of IDEs since Visual Studio was still Visual C++.
The Bottom Line
Is Pipes going to be a success? In many ways it already is. It lets Yahoo unequivocally be first at something, and if you count the broader market of web-based application development tools, it lets Yahoo be best at something, too. It's innovative, exciting, and well-done. There are still rough edges and inexplicable nods to the Big Purple Monster. (Whose idea was it to have the Yahoo Messenger anime avatars next to developer names on the site?) But the web needs a way to rip, mix and burn feeds, and Yahoo has stepped up to provide an essential platform in a way that seems open and approachable.
So, take a look at the docs, browse some pipes, and let me know if any of you can get it hooked up to the firehose that is the Six Apart Update Stream (that's an endless Atom feed of blog posts, flowing into your pipes at 30 posts a second). Because routing all these streams to the right place is exactly what pipes are for.
Previously: Represent Your Set
Next: Clay Feats
Jeremy Zawodny's blog
Yahoo! Pipes: Unlocking the Data Web: For far too long now RSS has been used in ways that don't really tap its true potential. Being able to syndicate my favorite headlines or blog posts is great. In fact, it helped to kick off a revolution in personal on-line publishing that is still gro... read more »Robin Good's Latest News
Beyond NewsMastering: Yahoo! Pipes Is The Internet RSS Remixer - Overview And Reports: Yahoo! Pipes, is essentially a very powerful RSS feed remixer, which goes well and beyond the original newsmastering concept I described a few years ago. Potentially, Yahoo! Pipes is a highly disruptive visual programming environment that puts in the h... read more »Peter Miller
Isn’t Dapper in this space too?
Ivan Pope
Anil, what about Dappit and their Dapplications? http://dappit.com
Byrne Reese
Are you baiting me or something?
Is Pipes going to be a success? In many ways it already is. It lets Yahoo unequivocally be first at something, and if you count the broader market of web-based application development tools, it lets Yahoo be best at something, too.
Simply put, this is not something Yahoo is first at. Period. You can spin it anyway you like, but it simply is not true. They might be best at it, but not first. Sorry.
Harish TM
I really think the best way to use this new product is to have an entire community of users interacting with each other and creating content.
I have tried to kick start something of the sort. Guess we will just have to wait and see where it goes.
And I personally don’t think the market cares about who is first. Its all about who succeeds…
Matt
feedGod has had a simple version of “Pipes” available for some time. Not as many options as Yahoo’s but it’s a lot easier to use and includes thousands of built in feeds.
UsefulVideo
Indeed, the main thing that Yahoo Pipes has done is that it’s made a difficult programming chore more accessible to mainstream users. Just as so many programming languages make assembly language more accessible. However, there is still quite a learning curve for the average end-user. I have created some video tutorials to help lessen this curve. With your permission, here is the link: http://usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html
Brad Gessler
I’ve been playing with Pipes now quite a bit and it opens up some interesting possibilities. I came up with an interesting approach to using pipes to create a “Life Feed”; a feed that combines all of my blog posts, flickr photos, and diggs so that my friends and family can follow my online life through one feed. You can read the tutorial on my site at http://www.bradgessler.com/2007/03/07/my-life-feed-the-killer-yahoo-pipes-app/
Enjoy!
Sherif
Thanks for the great article.
I watched a great Google Tech Talk video on Yahoo Pipes - it was brilliant. Thought this would help: http://blog.sherifmansour.com/?p=85
Bo Aphran
Pipes is even more powerful now when has the reg exp operator. I’m reading this tutorial ( http://theytookmystapler.blogspot.com/2007/09/yahoo-pipes-case-study-of-using-regex.html ) on it, I will use Pipes with the reg exp operator to change the titles of my Flickr pictures.
Dan N. Moldovan
Good article! In my opinion Yahoo Pipes make the first step to the next web 3.0.
But if you want to see some Yahoo Pipes implementations in a site take a look on my MacrosReader
Post a comment