PubSub: Prospective search for the knowledge economy

I'm sure that if you're like me, you probably feel overwhelmed at times with too much information. I do my best not to act as if I'm plugged in during all of my waking hours. But I can't resist the urge to frequently check my BlackBerry (although only 20 seconds have passed) or to read certain websites or blogs to keep current. Unfortunately, in this day and age, it isn't possible to stay up-to-date on everything. There are just too many data sources and too many ways to access information.

It's good news, then, that there's research and development underway to find ways to help filter the noise and convert what's remaining into useful knowledge. I'm glad that Computerworld chose to profile PubSub as a leading example of a technology that saves time by finding what we need and nothing more. PubSub is solving what I consider to be a hard problem: matching user generated interests against what the company characterizes as a fire hose of streaming, real-time information. I wish I could figure out how to apply this to my job, where anecdotally, a venture capitalist may hear of perhaps 500 companies before finding one that matches his or her interests and investment style.

The art of the solution requires tuning for high relevancy in the face of volatile, and perhaps interpretative data. For volatility, I'm referring to whether the data changes regularly (such as a sports score), and for interpretative, I'm referring to whether there's universality in agreement of the interpretation of the data, or whether it's opinionated (Was that a romantic comedy or a comic drama?). The computational challenges are hard at both extremes. At one end, you have the problem of indexing billions of somewhat static web pages. At the other, you have the challenge of sorting through millions of real time feeds. My solution for sorting through less volatile, singular information, is to use Google or sources like Wikipedia. For all the other stuff, and perhaps the stuff that gives me more of an edge in this shifting knowledge environment, I resort to trolling around and wondering what I'm missing.

Best of luck to PubSub, and also to similar services that combine matching and searching, such as Feedster, another service I use a lot.