Digging out the golden nuggets

How do you find valuable and relevant information hidden in the millions of blogs today? How did you find this blog? There are several services that attempt to sift through all this and rank the best content. Digg, Reddit, Techmemeorandum, Slashdot, Findory, and others use collaborative filtering, user voting, and various algorithms to find and rank popular content by category.

The Wall Street Journal has a story today "Digging Out The News" "In December, Digg drew 698,000 unique visitors, according to Nielsen/NetRatings. By comparison, tech news blog Slashdot had 1.3 million visitors that month and sites like CNET Networks Inc.'s News.com and Lycos Inc.'s Wired News had close to two million unique visitors each. Anyone can view Digg.com, but only the sites 140,000 registered users can submit stories and vote.

The site gets about 1,400 story submissions each day, said Chief Executive Jay Adelson, and about 50 get enough votes to show up on the home page. Each item has a "submitted by" line that includes the online alias of the contributor."

Digg allows registered users to submit stories that they think are noteworthy. Other users cast votes called "diggs" if they agree the story is good. The highest rated stories rise to the top of the home page.

Richard McManus compares Digg to Reddit in his story on ZDnet. Reddit works on a peer ranking system called Karma. "When a particular item is promoted or demoted, the user who posted it is either rewarded or punished — a system of editorial karma. In the same way that popular submissions are voted to the top, the individuals who post them get increases in karma. Every redditor affects one another's karma equally, regardless of his/her karma. Although democracy isn't perfect, this experiment should supply the public with the information they demand while also rewarding those who provide it."

Tech-meme-orandum takes a completely different approach. One that still baffles me, but it is highly effective. It appears that Techmemeorandum starts with published stories in the traditional media, then looks for blogs from prominent bloggers who link to the story. If a story gets enough important thought leader bloggers linking to it, then the story and the associated blogs get published to the home page. It is very rare that a blog story leads without a major published story. Techmemeorandum covers Tech news and Politics. They appear to have a list of a hundred or so top bloggers in each category and rank stories based on how many of those top bloggers link to it. All bloggers are not created equal. If Robert Scoble links to a story it goes right to the top of the list...even if he is the only link. Robert has earned his reputation as a thought leader and Techmemeorandum rightfully gives him more weight in their ranking algorithm.

Pure collaborative filtering works in a slightly different and more personal way. You create a personal profile where your "ranking votes" are stored. You rank lots of stories, or music, or books, etc. The system learns what you like as you continue to rank content. In addition the system compares the profile of what you like to the profiles of all the users. It finds users who have very similar preferences and suggests content from these users that you have not yet seen. Netflix uses a crude implementation of collaborative filtering to suggest new movies you might enjoy.

I use all of the services to scan popular content. They all do a reasonably good job of exposing "popular" content. There is no clear winner in my mind, although one is bound to emerge over the next year or so. All of them are private and independent at this point. I expect that will change over the next year as well. The big players are likely to look for acquisitions in this space.