Blocked by Digg

Remember Digg mobile? From all the digg-mobile online efforts, this was the most successful of the bunch. Apparently is now blocking us completely! They check for the user agent if it’s “php” and they check our HTTP_REFERER if it’s, and if yes, they block our access to their RSS feed.

I think what they do is pretty sneaky, because all we do is using their free RSS feed. We don’t grab bits and pieces of their HTML code (like some others do), but we used the legitimacy of the RSS feed (so all clicks end up on their site and not at the sites their news items are linking to). We also had a disclaimer to say that our site was not affiliated with And yet, they blocked us. As far as I know, we have not received a complain email by them about anything. If we had, we would have complied with requested changes.

My assumption is that they want to promote their own mobile site now,, so they block community effort to provide mobile access to Digg. Problem with diggriver is though that it’s not as well tested with many mobile browsers and so it bombs out on many phones.

Does anyone know how we can fake the http_referer variable our server sends to them using PHP? That would be the only way to go around their blockage. My google search didn’t reveal any way to do that although I think it’s possible because some firewalls strip such HTTP headers.

UPDATE: Thanks to “Grendelboogie” we are back in business. The mobile digg page is back up with some caching (thanks Adam), although without proper paging. Some phones will bomb out as we now serve 20 items per page instead of 10 (some motorola phones will run out of memory).


Alex Forster wrote on March 8th, 2007 at 8:37 AM PST:

Absolutely. The HTTP 1.1 rfc only requires the following to be sent with a request:

GET /rss/index.xml HTTP/1.1

How exactly are you making this request? I assume you’re not using PHP’s sockets?

This is the admin speaking...
Eugenia wrote on March 8th, 2007 at 8:49 AM PST:

ini_set(’allow_url_fopen’, ‘1′);
$data = @file_get_contents($url);

Alex Forster wrote on March 8th, 2007 at 9:26 AM PST:

Meep. Well, okay, you have several avenues to take…

1- $data = exec(’wget …’);

I can help you with either of these if necessary. I have a /lot/ of experience with PHP’s sockets (including easy to use, well written wrapper functions). AIM: i3x171um, ICQ: 231766197

Adam wrote on March 9th, 2007 at 1:08 AM PST:

Do you think it might be because with every load, you request their RSS? I added caching to mobile quo some time ago because many sites ban any host that requests too many times.

I think if you cache, they may allow you back.

Slashdot’s etiquette is 1 request per 30 minutes.

This is the admin speaking...
Eugenia wrote on March 9th, 2007 at 1:09 AM PST:

Yeah, haven’t updated that code at all since that time. It could be that.

grendelboogie wrote on March 9th, 2007 at 4:15 AM PST:

Hey, I’m the Director of Ops at I don’t think we intentionally block your efforts. However, we like to see User-Agent on requests. If you send me the IP address you are coming from, I can investigate.

Comments are closed as this blog post is now archived.

Lines, paragraphs break automatically. HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

The URI to TrackBack this blog entry is this. And here is the RSS 2.0 for comments on this post.