Re: [tlug] DDB/CJKV-E Web Host under DDOS attack

[Jim Breen <jimbreen@example.com>]

On 8 March 2016 at 19:33, Curt Sampson <cjs@example.com> wrote:
> On 2016-01-07 07:41 +1100 (Thu), Jim Breen wrote:
>
>> On looking at it more closely I realise that wwwjdic is probably
>> not a good CDN candidate after all. It doesn't have static content;
>> everything is coming out of dictionary files with a humungous number
>> of potential query terms. I don't think there really are identifiable
>> common queries, or at least not enough to pre-compose.
>
> I thought you were right, for a bit, but then I had an idea. The
> individual dictionary entries are all fairly constant, most of them
> rarely changing, right? One could set up a query system that returns
> only the set of entry IDs of interest (as some small JSON object, no
> doubt) and then have the page itself (i.e., JavaScript running in the
> browser) pull these entries from the CDN servers (that would store them
> as static files) and arrange them nicely on the page.
>
> This would minimize data served by the servers that actually do
> searches. How useful this would be, I'm still not sure.

I'm not sure either. The server that fields the lookup requests would
get much the same incoming traffic; just the outgoing would be cut.
Whether this improves its position in a DDOS is questionable.

> Certainly it makes DOS attacks more difficult. You could effectively
> limit the bandwidth of every IP of a potential attacker to something
> negligible, meaning that he's going to have to have a lot of IP
> addresses available to run up your CPU and/or IO load to a problematic
> level. (You can safely limit an individual IP to something like 10
> requests/minute, so a single query server should be able to handle
> O(100,000) clients without difficulty.)

Hmmm. I can't see that change has any impact on the application level
traffic coming *from* requesting IP addresses.  At the IP level it would,
but trying to limit that at the IP address level would mean very messy
stuff right down where web applications don't or can't go.

We've been having some problems in recent months which have looked
rather like DOS attacks, but in fact have just been cases of innocent
events leading to a lot of traffic which has damaged the rather
lightly-provisioned
server. They've mainly been to do with the database maintenance system,
which unlike wwwjdic(*) use python and postgresql. A rush of requests has
lead to a heap of processes (httpd, etc.) being spawned, a sudden runout
of RAM and swap, and the system eventually thrashes itself to death.
We've stopped the problem by limiting the number of httpd processes
allowed and by having the python routines block requests if the available
swap is below a threshold. Fingers crossed. Seems to have worked.

JIm

(*) wwwjdic is in C and has all its data in read-only static files which are
usually cached, so it can breeze through a lot of requests with very
little load on the server.

-- 
Jim Breen
Adjunct Snr Research Fellow, Japanese Studies Centre, Monash University