Digg, Dug, Buried: How Linux news disappears

Like it or lump it, the major reason that determines whether any given online story will get read or not is how much play it gets on news link sharing sites and social networks like Digg, reddit, and StumbleUpon. Unlike earlier news sharing sites like Slashdot, these sites have no central editorial control. Instead, the stories that get prominent play on these sites is determined entirely by readers. That sounds like democracy in its most basic form, but in practice what it really means that stories can be buried from sight by abusive users with an ax to grind.

I became aware of this because in the last few weeks I've had several stories that were pro-Linux and anti-Microsoft-Linux, it doesn't get any faster and Macs, Windows 7, and Linux--first became popular on Digg, and, an hour later they were buried.

On Digg, what this means is that, unless you already know the Digg link, or specifically search for a tale with the buried story option on, the story link disappears. In short, no matter how many people thought it was a good story, other people will no longer be able to see it.

Digg staffers explained that this happens because "The Digg.com community has collectively voted to bury the story to which you refer. Though the story has been buried, it is still available on Digg.com. If you or a friend Dugg, submitted or commented on the story, there will be a link on your respective profiles. If you did not have the chance to Digg the story, you can search for it at http://digg.com/search - just be sure to select 'Include Buried Stories' when you search."

When asked further about how this works, a Digg staffer replied, "Stories get buried from the Popular page all the time. We don't have a definitive percentage. It makes sense that stories will be buried from the homepage from time to time since that's where most people hang out on Digg; more people "vetting" (Digging and burying) content on the homepage than anywhere else on Digg, so to speak."

Fair enough, but he continued, "There are users who abusively bury content in coordination with other users for a variety of reasons." You think!?

In checking with other technology writers, editors, and publishers, I discovered that there was a pattern to what stories get buried. Pro-Linux and anti-Microsoft stories tend to be buried. It's not just Digg though. I, and others, have found similar patterns with other story popularity contest sites.

Digg says that "like our promotional algorithm, our bury algorithm requires diverse sets of users and burying behavior in order for a story to be buried. In other words, we have logic coded into our also that works to determine whether stories should or shouldn't be buried based on their diverse or non-diverse actions."

But, that said, Digg admits that group of users-say Microsoft employees, partners, and supporters-can "abusively bury content." I'd add, not just 'can, but do.'

With Digg, "once a story reaches the threshold to bury a story, yes, that story will be buried whether it's on the homepage or in the sea of submissions in our Upcoming section. And once stories are buried, they do not get reinstated. This is all done by the Digg community, as Digg has no editors. The only thing we provide is an algorithm that looks at user behavior to help prevent coordinated patterns of abuse and fraud."

That sounds good, but since these sites use broken CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) security for user accounts, it's trivial for any organized group to make sure the news that they want kept out of the public's eye is buried.

This is not new. You can find similar stories about Digg's "Bury Brigades" from 2007. In 2008, Digg addressed this issue, among others, by banning the use of scripts.

Unfortunately, the abuse continues. And, what's far more unfortunate is that it matters so much.

You see a plurality of readers of this blog, and indeed almost all online news and opinion sites, now come to them from sites like Digg. To be exact, 28.1% of traffic to my stories is driven by such Websites. Those numbers are about the same for all online news sites.

In other words, when Microsoft supporters bury stories, they're making sure tens to hundreds of thousands of readers never see them.

This is far more though than just a 'inside baseball' technology news story though. I'm told by friends in other branches of journalism that they see the same kind of thing, except in their cases, its things like stories about political unrest in Pakistan. You see while Iran's unrest is getting coverage in the online news popularity contest, Pakistan's troubles aren't, so they're invisible. Both are important.

So, while personally I certainly have a problem with stories being buried, I have a far bigger beef with people believing in the validity of 'popularity' news sites as determining what is, or isn't, important.

I also wish that newspapers weren't dying. How annoying to actually agree with Steve Ballmer for once! There's nothing I can do about either trend though.

What I can do though is to encourage all the news-link sites to reconsider how they're handling both how stories are voted up and how they're buried. All these sites are far too important to how people get their news for their owners to allow them to be so easily gamed by companies, groups or politicians.

For starters, I'd suggest that they start using open algorithms and revealing who votes stories down as well up. It works for open source; I don't see why it can't work for news sites as well.