Re: [tlug] Server shows strange load and memory spikes

["Jean-Christian Imbeault" <jean_christian@example.com>]

>From: B0Ti <9915104t@example.com>
>
>Check out if/what cron was running in /var/log/cron or similar depending
>on your distro. Logrotate comes to mind.
>If you can't find out, then just extend that script to capture the process
>list every second or so.

Yesterday I stopped anacron, since it runs logrotate, hoping that would 
solve the problem of my load average going at weird times. The load on my 
machine did go down a little but I thinks that only becuase yesterday was a 
Monday and not as many people were accessing the web server.

I did extend my script to capture which jobs were running when the load 
average was high but I can't see any "unusual" jobs running. Here is a 
sample output (minus the http procs):


10:37:31 CPU load: 1.46 #http procs: 218

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  1416   76 ?        S    May31   0:06 init [3]
root         2  0.0  0.0     0    0 ?        SW   May31   0:00 [keventd]
root         3  0.0  0.0     0    0 ?        SWN  May31   0:22 
[ksoftirqd_CPU0]
root         4  0.0  0.0     0    0 ?        SW   May31  11:50 [kswapd]
root         5  0.0  0.0     0    0 ?        SW   May31   0:00 [bdflush]
root         6  0.0  0.0     0    0 ?        SW   May31   0:02 [kupdated]
root       128  0.0  0.0     0    0 ?        SW   May31   0:00 [kjournald]
root       129  0.0  0.0     0    0 ?        SW   May31   0:06 [kjournald]
root       130  0.0  0.0     0    0 ?        SW   May31   0:00 [kjournald]
root       131  0.0  0.0     0    0 ?        SW   May31   0:00 [kjournald]
root       132  0.0  0.0     0    0 ?        SW   May31   0:09 [kjournald]
root       133  0.0  0.0     0    0 ?        SW   May31   0:05 [kjournald]
root       134  0.0  0.0     0    0 ?        SW   May31   8:14 [kjournald]
root       467  0.0  0.0  1476  320 ?        S    May31   0:01 syslogd -m 0
root       472  0.0  0.0  1892  108 ?        S    May31   0:00 klogd -2
root       631  0.0  0.0  5196  412 ?        S    May31   0:00 sendmail: 
accepting connections
root       650  0.0  0.0  1604  444 ?        S    May31   0:00 crond
daemon     686  0.0  0.0  1448   52 ?        S    May31   0:00 /usr/sbin/atd
root       714  0.0  0.0  1388   16 tty3     S    May31   0:00 
/sbin/mingetty tty3
root       715  0.0  0.0  1388   16 tty4     S    May31   0:00 
/sbin/mingetty tty4
root       716  0.0  0.0  1388   16 tty5     S    May31   0:00 
/sbin/mingetty tty5
root       717  0.0  0.0  1388   16 tty6     S    May31   0:00 
/sbin/mingetty tty6
root       877  0.0  0.0  1388   16 tty2     S    May31   0:00 
/sbin/mingetty tty2
root     29555  0.0  0.0  2968  416 ?        S    Jun14   0:03 
/usr/local/sbin/sshd
root     16604  0.0  0.0  1388   32 tty1     S    Jul08   0:00 
/sbin/mingetty tty1
root     30597  0.0  0.1  3100  920 ?        S    Aug16   0:04 snmpd -c 
/usr/src/net-snmp-5.0.3/snmpd.conf
root      5062  0.0  0.2  2856 1320 ?        SN   Aug26   0:00 /usr/bin/perl 
-w ./t2.pl
root     24107  0.0  0.0  1512  352 ?        S    Aug26   0:01 
/usr/local/sbin/cronolog /www/logs/%Y/%m/%d/error.log
root     24108  0.1  0.0  1516  364 ?        S    Aug26   0:57 
/usr/local/sbin/cronolog /www/logs/%Y/%m/%d/access.log
root     13755  0.0  0.2  2844 1368 ?        SN   09:56   0:00 /usr/bin/perl 
-w ./mem.pl
root     14708  0.0  0.2  2860 1380 ?        SN   10:19   0:00 /usr/bin/perl 
-w ./load.pl
root     14834  0.0  0.2  2856 1380 ?        SN   10:20   0:00 /usr/bin/perl 
-w ./load60.pl
root     16543  0.0  0.2  3028 1052 ?        RN   10:37   0:00 ps -auwx


Thre three perl scripts are what I use to monitor theload and memory usage 
at 1, 30, and 60 second intervals (the basically loop forever and sleep 1,30 
or 60 seconds). The load was high at strange time even before I ran these 
scripts.

Can anyone see anything strange in the ps output? The output is pretty much 
always the same as this is a barebones apache server.

There are always about the same 30 (+/- 2) non-http processes running on the 
machine. Yet the load somthing spikes as high as 3.0 at strange time like 
10:30am and 15:30.

I really don't know what to make of it. Any advice on how I can pinpoint 
what is making the load shoot so high?

Thanks,

Jc

PS I still get that strange ~40Mb memory spike at 4am too ...

_________________________________________________________________
MSN Photos is the easiest way to share and print your photos: 
http://photos.msn.com/support/worldwide.aspx