403 Access Denied errors - why so many?

[ldesign]

I am seeing a lot more 403 - Access Denied errors since the move to the new servers. My guess is that these are from server wide blocking of requests based on IP addresses or other criteria. If so, I'd like to know more about them so I can get a better handle on them.

All of the http requests in the 403's I'm getting are for valid web page URLs. They have no extra GET parameters (URL variables) tacked on. In other words, they look like simple, legitimate http requests for my site pages. They don't come flooding in. They come in at a rate that appears to be an actual person trying to view my site.

My concern is that Page-Zone could get over zealous in banning suspicious visitors to the servers, therefore deflecting honest visitors to my site.

I'd really like more access to information about these 403's

- What triggered them?
I'd like to know when they are triggered by system wide controls so I can understand how often that happens, ... and ideally, I'd like to be able to handle them with a custom error page.

- If system wide, what criteria has caused the request to be denied?

- I really want the ability to look up the IP addresses to see if they are banned by Page-Zone - and find out why they are banned.

I know we all have to be diligent in this complex, dynamic network, but my concern is that in an effort to shut out the "bad guys" we could end up paying too big a price in accessibility to "good guys."

Why do I get so many Access denied errors from legitimate page requests (not probes for cgi scripts or formmail, but simple request for my home page)?

Chuck

[ldesign]

Okay, here's one example. Over the last two days, 403's have been trickling in (I email myself) like this:

From: cycletourist.com
Server Addr = 8.9.24.34
Date and time = Thu, 18 Jan 2007 23:54:38 -0500 • EST
Request URI = /robots.txt
referer =
User Agent = Mozilla/5.0 (compatible; BecomeBot/3.0; +Become: Site Owners)
Remote Addr = 64.124.85.80
Remote Host = 64.124.85.80.become.com

become.com tries to read my robots.txt file (which is there), but gets a 403 error instead. Then it requests my home page, like this:

From: cycletourist.com
Server Addr = 8.9.24.34
Date and time = Thu, 18 Jan 2007 22:08:18 -0500 • EST
Request URI = /
referer =
User Agent = Mozilla/5.0 (compatible; BecomeBot/3.0; +Become: Site Owners)
Remote Addr = 64.124.85.80
Remote Host = 64.124.85.80.become.com

... but gets a 403 error instead (with related 403's for my favicon and style sheet file).

Why does 64.124.85.80 get redirected to my 403 error page whenever it tries to access my site?

Chuck

[ldesign]

Here's another example:

From: cycletourist.com
Server Addr = 8.9.24.34
Date and time = Fri, 19 Jan 2007 20:43:21 -0500 • EST
Request URI = /
referer =
User Agent = Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
Remote Addr = 83.160.254.102
Remote Host = goudeket.demon.nl

Every time 83.160.254.102 tries to access my site it is redirected to 403 error. This IP address made 6 valid page requests to my site yesterday over the course of one hour and all were redirected to 403 .... Why?

Chuck

[edwurster]

Originally Posted by ldesign

Here's another example:

From: cycletourist.com
Server Addr = 8.9.24.34
Date and time = Fri, 19 Jan 2007 20:43:21 -0500 • EST
Request URI = /
referer =
User Agent = Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)
Remote Addr = 83.160.254.102
Remote Host = goudeket.demon.nl

Every time 83.160.254.102 tries to access my site it is redirected to 403 error. This IP address made 6 valid page requests to my site yesterday over the course of one hour and all were redirected to 403 .... Why?

Normally, the robot reads the text file, and hopefully follows what you've put in there.

What happens if the robot is misconfigured? Maybe it finds robots.txt, misinterprets, something here, and then a 403?

You should put in an empty file favicon.ico. That eliminates a lot of 403's. And of course you need a 403 file, or requests for it show as an error too.

What happens if the robots file is bad? Could we have a situation where most robots handle the error, but this robot does not.

[ldesign]

Originally Posted by edwurster

Normally, the robot reads the text file, and hopefully follows what you've put in there.

What happens if the robot is misconfigured? Maybe it finds robots.txt, misinterprets, something here, and then a 403?

You should put in an empty file favicon.ico. That eliminates a lot of 403's. And of course you need a 403 file, or requests for it show as an error too.

What happens if the robots file is bad? Could we have a situation where most robots handle the error, but this robot does not.

Hi Ed,

Thanks for joining in the discussion.

First of all, not all of these trails of 403's start with reading robots.txt. Also, ... my robots.txt file is in the correct format .... .... and the 403's I get are requests for pages that are not disallowed in robots.txt. The 403's I get are for requests as simple as a request for the home page.

The lack of a favicon.ico file would cause many 404 errors (file not found), but not 403 errors. Besides I do have a favicon file in place. That's why a 403 error when requesting /favicon.ico makes absolutely no sense (as say a sub-request for the home page). And I do have a custom 403 error page, that's how I get an email every time someone gets there.

The robots.txt file is not really a factor in this. It is merely one of the files that sometimes results in a 403 for some requesters. Since it exists, there is really no reason at all to server an error .....

.... Unless they are being blocked by system-wide controls.

Chuck