[tlug] Monkey vs Apache!!! Fight!

[Sach Jobb <sach@example.com>]

Hello TLUG,

>From around 18:00 on friday, our main front end webserver, which runs
apache started running into problems. Specifically it suddenly gets
extremely busy, processing requests until it hits the limit, the new
clients then get a connection error message, and then if no staff
intervene (by restarting the service in time), apache just dies
entirely (no core dump either!).

I have a snap from munin of what it looks like here (# of processes):
http://imgserv.net/f5aeed39e034b252.png
(the skyscrapers are when the problem happens).

And here is a relevant snip from the apache access log:
(I changed the request uri to "/someurl/" and changed the client ip to
114.x.y.z.)

114.x.y.z - - [02/Apr/2011:17:49:03 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"
114.x.y.z - - [02/Apr/2011:17:49:05 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"
114.x.y.z - - [02/Apr/2011:17:49:02 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"1
114.x.y.z - - [02/Apr/2011:17:49:11 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"
114.x.y.z - - [02/Apr/2011:17:49:14 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"
114.x.y.z - - [02/Apr/2011:17:49:15 +0900] "GET /someurl/ HTTP/1.1"
200 25728 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6;
en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4
Safari/533.20.27"

If you grep and sort the log for IP and URI you'll see around 4,5
connections to the same URI from the same IP per second, with up to
3000 in total depending on when it was restarted (or when it died).

Upon better investigation of the logs, I figured out that it was
actually all coming from the same IP address (well, three of the times
correlate to the problem) and managed to temporarily fix the problem
by simply blocking that IP in the apache config. Tracking the IP back
to logins, I figured out that it was a real user, not a bot, and
managed to get in contact with him.

Speaking with him on the phone, I learned that he has a regular OCN
connection, somewhere in Tokyo, and some 10 client machines or so that
share through a cheap firewall. Most of them are macs. As far as I can
tell he wasn't doing any kind of behavior that would possibly take
apache, or any other service down. He's just a normal user. So, that
just left me with more questions... is there some sort of virus on one
of his machines? Is the cheap fw getting confused and rapid fire
sending the same request uri over and over? Is apache just buggered?

So, my questions are:
1) Has anyone else experienced this sort of behavior before, and
2) What to you do to protect apache against somebody that just
suddenly goes nuts with the connections, intentionally or not?

I should mention that this server is just a direction connection.
There is no reverse proxy, no load-balancer, no firewall or anything
else mucking around with IP in any way. I also double-checked with the
ISP at the colo just to make sure they didn't have any changes.

Comments appreciated.

BTW, I am the monkey, if you haven't figured that one out yet....
(pointless reference: http://www.youtube.com/watch?v=I_QsCXm1vrk)

Cheers,
Sach