News:

Wondering if this will always be free?  See why free is better.

Main Menu

httpd --> very high CPU usage

Started by Mike Bobbitt, November 01, 2006, 07:18:56 PM

Previous topic - Next topic

Mike Bobbitt

SMF Version: SMF 1.1 RC3
Hi all,

I'm back again with another CPU problem. Sometime today, apache took off and started eating up all my CPU.    When I restart apache, it goes away... for a short period. But then it ramps right back up again. Server reboot has not helped. The system load goes above 100 if I let apache keep running.

I make any config changes, but now my httpd processes are running absolutely wild on me and I can't keep my server online for more than 10-20 minutes before it becomes totally unusable.

strace shows a lot of "-1 EAGAIN (Resource temporarily unavailable)" messages. Not having run strace on apache when it was working fine, I'm not sure if that's indicative of a problem, or just apache waiting for more data.

Any suggestions would be appreciated.


  • A link to your forum.
http://forums.army.ca

  • A link to your status.php
http://army.ca/deleteme/status.php

  • a link to your phpinfo.php
http://army.ca/deleteme/status.php?php

  • Is this a dedicated server or shared host. If it is a dedicated server, what spec is the server and what else runs on the server? If it is a shared host post a link to the hosts site the package you are on
Dedicated: Fedora Core 5, Pentium 4 CPU 2.80GHz, 2 Gb RAM. Also runs a mail server and several smaller (very low traffic) sites.

  • The Number of posts your forum has.
457,309 Posts in 28,365 Topics by 10,556 Members

  • The Average number of users you have online during peak time and over what time period that is over, e.g 400 users over 15 mins.
200-300 over 30 minutes

  • A list of mods which you have installed
Oldiesmann's SMF+G2 mod (http://smf.oldiesmann.us/galleryproject/index.php)
Custom theme changes applied via mod

  • Do you still have performance issues if you use the default theme?
Yes

I realize this is not likely SMF related, but I'm hoping some of the apache experts here can give me some guidance on troubleshooting.

Christian A. Herrnboeck

Hello,

First off, login to SSH, and post the output of ps aux and top while apache is running, and load is high. (Use code tags please ;) ).  Also, have you tried compiling Apache with PHP SuExec?

-Christian


Farmers:Producing food for the world!

Mike Bobbitt

Hi Christian,

Here's a sample top screen from just after httpd was restarted:

top - 22:49:16 up  4:47,  2 users,  load average: 7.48, 5.66, 13.72
Tasks: 182 total,   9 running, 173 sleeping,   0 stopped,   0 zombie
Cpu(s): 98.0% us,  1.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.2% si,
Mem:   2073760k total,  1166068k used,   907692k free,    30744k buffers
Swap:  2031608k total,   390216k used,  1641392k free,   380048k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
4379 apache    19   0 34268  13m 4884 R   71  0.7   0:21.60 httpd
4544 apache    16   0 37232  16m 4984 R   33  0.8   0:01.29 httpd
4390 apache    21   0 34520  13m 4960 R   19  0.7   0:03.91 httpd
4381 apache    25   0 35204  14m 5008 R   15  0.7   0:05.46 httpd
4385 apache    25   0 37344  16m 5160 R   13  0.8   0:06.88 httpd
4533 apache    25   0 35140  14m 4868 R   10  0.7   0:12.80 httpd
4393 apache    25   0 34520  13m 4860 R    9  0.7   0:14.19 httpd
4377 apache    18   0 37244  16m 5116 S    8  0.8   0:00.94 httpd
4460 apache    16   0 37332  16m 5000 S    8  0.8   0:01.18 httpd
4447 apache    15   0 34452  13m 4952 S    4  0.7   0:00.82 httpd
4378 apache    15   0 38936  18m 5208 S    3  0.9   0:01.47 httpd
3783 mysql     15   0  457m  62m 5480 S    2  3.1   0:01.88 mysqld
4181 root      15   0  2260 1052  784 R    1  0.1   0:00.87 top
1760 root      15   0  1656  552  504 S    0  0.0   0:00.78 syslogd
2325 cyrus     15   0  5240 1052  948 S    0  0.1   0:00.43 cyrus-master
4373 root      16   0  8708 2984 1936 S    0  0.1   0:00.10 sendmail
4659 root      15   0  8688 2904 1876 S    0  0.1   0:00.01 sendmail


Here is a process list:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2012   548 ?        Ss   18:01   0:01 init [3]
root         2  0.0  0.0      0     0 ?        S    18:01   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   18:01   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    18:01   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S    18:01   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        SN   18:01   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    18:01   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<   18:01   0:00 [events/0]
root         9  0.0  0.0      0     0 ?        S<   18:01   0:00 [events/1]
root        10  0.0  0.0      0     0 ?        S<   18:01   0:00 [khelper]
root        11  0.0  0.0      0     0 ?        S<   18:01   0:00 [kthread]
root        15  0.0  0.0      0     0 ?        S<   18:01   0:00 [kblockd/0]
root        16  0.0  0.0      0     0 ?        S<   18:01   0:00 [kblockd/1]
root        17  0.0  0.0      0     0 ?        S<   18:01   0:00 [kacpid]
root        80  0.0  0.0      0     0 ?        S<   18:01   0:00 [cqueue/0]
root        81  0.0  0.0      0     0 ?        S<   18:01   0:00 [cqueue/1]
root        84  0.0  0.0      0     0 ?        S<   18:01   0:00 [khubd]
root        86  0.0  0.0      0     0 ?        S<   18:01   0:00 [kseriod]
root       146  0.9  0.0      0     0 ?        S<   18:01   2:44 [kswapd0]
root       147  0.0  0.0      0     0 ?        S<   18:01   0:00 [aio/0]
root       148  0.0  0.0      0     0 ?        S<   18:01   0:00 [aio/1]
root       300  0.0  0.0      0     0 ?        S<   18:01   0:00 [kpsmoused]
root       322  0.0  0.0      0     0 ?        S<   18:01   0:00 [ata/0]
root       323  0.0  0.0      0     0 ?        S<   18:01   0:00 [ata/1]
root       324  0.0  0.0      0     0 ?        S<   18:01   0:00 [ata_aux]
root       330  0.0  0.0      0     0 ?        S<   18:01   0:00 [scsi_eh_0]
root       331  0.0  0.0      0     0 ?        S<   18:01   0:00 [scsi_eh_1]
root       357  0.0  0.0      0     0 ?        S<   18:02   0:00 [kmirrord]
root       370  0.0  0.0      0     0 ?        S<   18:02   0:08 [kjournald]
root       409  0.0  0.0      0     0 ?        S<   18:02   0:00 [kauditd]
root       434  0.0  0.0   2356   360 ?        S<s  18:02   0:00 /sbin/udevd -d
root      1365  0.0  0.0      0     0 ?        S<   18:19   0:00 [kjournald]
named     1719  0.1  0.2  50644  4916 ?        Ssl  18:19   0:19 /usr/sbin/named -u named -t /var/named/chroot
root      1760  0.0  0.0   1656   552 ?        Ss   18:19   0:00 syslogd -m 0
root      1763  0.0  0.0   1600   388 ?        Ss   18:19   0:00 klogd -x
root      1773  0.0  0.0   1600   284 ?        Ss   18:19   0:00 irqbalance
rpc       1788  0.0  0.0   1732   400 ?        Ss   18:19   0:00 portmap
root      1799  0.0  0.0  12012   560 ?        S<sl 18:19   0:00 auditd
root      1839  0.1  1.0 752476 22088 ?        Sl   18:19   0:24 /opt/wildfire/jre/bin/java -server -Dinstall4j.jvmDir=/opt/wildfire/jre -Dinstall4j.appDir=/opt/wildfire -D
dbus      1847  0.0  0.0   3092   312 ?        Ss   18:19   0:00 dbus-daemon --system
root      1906  0.0  0.0   1820   388 ?        Ss   18:19   0:00 /usr/bin/hidd --server
root      1996  0.0  0.0   1876   612 ?        Ss   18:19   0:00 /usr/sbin/automount --timeout=60 /net program /etc/auto.net
root      2009  0.0  0.0   1596   332 ?        Ss   18:19   0:00 /usr/sbin/acpid
root      2018  0.0  0.0   5012   616 ?        Ss   18:19   0:00 ./hpiod
root      2025  0.0  0.0  12504  1068 ?        S    18:19   0:00 python ./hpssd.py
root      2037  0.0  0.0   9336  1460 ?        Ss   18:19   0:00 cupsd
root      2047  0.0  0.0   4992   816 ?        Ss   18:19   0:00 /usr/sbin/sshd
root      2057  0.0  0.0   2228   532 ?        Ss   18:19   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       2091  0.0  0.2   4244  4244 ?        SLs  18:20   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid
root      2102  0.0  0.0   4388   240 ?        Ss   18:20   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
cyrus     2325  0.0  0.0   5240  1052 ?        Ss   18:20   0:00 /usr/lib/cyrus-imapd/cyrus-master -d
root      2337  0.0  0.0   2800   332 ?        Ss   18:20   0:00 /usr/sbin/dhcpd
cyrus     2343  0.0  0.0  29192   160 ?        S    18:20   0:00 idled
cyrus     2345  0.0  0.0  30712  1232 ?        S    18:20   0:00 imapd
cyrus     2346  0.0  0.0  30716  1232 ?        S    18:20   0:00 imapd -s
cyrus     2348  0.0  0.0  29324   880 ?        S    18:20   0:00 notifyd
cyrus     2350  0.0  0.0  30712  1228 ?        S    18:20   0:00 imapd
defang    2381  0.0  0.0   3620   492 ?        S    18:20   0:00 /usr/bin/mimedefang-multiplexor -p /var/spool/MIMEDefang/mimedefang-multiplexor.pid -m 2 -x 10 -U defang -b
defang    2396  0.0  0.0  50080  1652 ?        Sl   18:20   0:02 /usr/bin/mimedefang -P /var/spool/MIMEDefang/mimedefang.pid -m /var/spool/MIMEDefang/mimedefang-multiplexor
cyrus     2399  0.0  0.0  30712  1232 ?        S    18:20   0:00 imapd
cyrus     2400  0.0  0.0  30716  1232 ?        S    18:20   0:00 imapd
cyrus     2414  0.0  0.0  30712  1232 ?        S    18:20   0:00 imapd
cyrus     2415  0.0  0.0  30712  1232 ?        S    18:20   0:00 imapd
cyrus     2416  0.0  0.0  30712  1232 ?        S    18:20   0:00 imapd
cyrus     2418  0.0  0.0  30712  1280 ?        S    18:20   0:00 imapd
cyrus     2419  0.0  0.0  30716  1232 ?        S    18:20   0:00 imapd
cyrus     2421  0.0  0.0  30716  1224 ?        S    18:20   0:00 imapd
root      2430  0.0  0.0   8492  1688 ?        Ss   18:20   0:00 sendmail: accepting connections
root      2431  0.0  0.1   8872  2144 ?        S    18:20   0:00 sendmail: ./kA1AVFX2032172 cluster2a.us.messagelabs.com.: client EHLO
smmsp     2438  0.0  0.0   7336   800 ?        Ss   18:20   0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
defang    2443  0.6  4.7 149504 98488 ?        S    18:20   1:52 /usr/bin/perl /usr/bin/mimedefang.pl -server
root      2454  0.0  0.1 133752  2904 ?        Ss   18:20   0:09 /usr/bin/spamd -d -c -m5 -H -r /var/run/spamd.pid
root      2463  0.0  0.0   1820   372 ?        Ss   18:20   0:00 gpm -m /dev/input/mice -t imps2
root      2483  0.0  0.0   5196   580 ?        Ss   18:20   0:00 crond
xfs       2521  0.0  0.0   3588   580 ?        Ss   18:20   0:00 xfs -droppriv -daemon
root      2550  0.0  0.0  13220  1400 ?        Ss   18:20   0:00 smbd -D
root      2561  0.0  0.0  13220   196 ?        S    18:20   0:00 smbd -D
root      2568  0.0  0.0   8548   972 ?        Ss   18:20   0:00 nmbd -D
root      2585  0.0  0.0   2176   372 ?        Ss   18:20   0:00 /usr/sbin/atd
root      2594  0.0  0.0   5208  1172 ?        Ss   18:20   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
root      2597  0.0  0.0   5208  1156 ?        S    18:20   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
root      2598  0.0  0.0   5208  1156 ?        S    18:20   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
root      2600  0.0  0.0   5208  1156 ?        S    18:20   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
root      2601  0.0  0.0   5208  1156 ?        S    18:20   0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
avahi     2612  0.0  0.0   2800   688 ?        Ss   18:20   0:00 avahi-daemon: running [www.local]
avahi     2613  0.0  0.0   2800   140 ?        Ss   18:20   0:00 avahi-daemon: chroot helper process
root      2646  0.0  0.0   3136   424 ?        Ss   18:20   0:00 cups-config-daemon
68        2661  0.0  0.0   5052  1096 ?        Ss   18:20   0:01 hald
root      2662  0.0  0.0   3140   496 ?        S    18:20   0:00 hald-runner
68        2668  0.0  0.0   2236   552 ?        S    18:20   0:00 /usr/libexec/hald-addon-acpi
root      2671  0.0  0.0 133752  1364 ?        S    18:20   0:00 spamd child
root      2672  0.0  0.0 133752  1304 ?        S    18:20   0:00 spamd child
68        2676  0.0  0.0   2232   548 ?        S    18:20   0:00 /usr/libexec/hald-addon-keyboard
root      2692  0.0  0.0   2192   412 ?        D    18:20   0:00 /usr/libexec/hald-addon-storage
root      2735  0.0  0.5  12236 12232 ?        SLsl 18:21   0:00 mdmpd
root      2741  0.0  0.0   1584   352 tty1     Ss+  18:21   0:00 /sbin/mingetty tty1
root      2742  0.0  0.0   1588   352 tty2     Ss+  18:21   0:00 /sbin/mingetty tty2
root      2743  0.0  0.0   1584   352 tty3     Ss+  18:21   0:00 /sbin/mingetty tty3
root      2744  0.0  0.0   1588   352 tty4     Ss+  18:21   0:00 /sbin/mingetty tty4
root      2745  0.0  0.0   1588   352 tty5     Ss+  18:21   0:00 /sbin/mingetty tty5
root      2747  0.0  0.0   1584   352 tty6     Ss+  18:21   0:00 /sbin/mingetty tty6
root      2982  0.0  0.0   4104   580 ?        Ss   18:21   0:00 /usr/bin/ssh-agent
root      8799  0.0  0.1   8872  2236 ?        S    19:20   0:00 sendmail: ./kA16MqTn001732 cluster1.us.messagelabs.com.: client EHLO
cyrus    17769  0.0  0.1  31544  2620 ?        S    20:19   0:00 imapd -s
root     26394  0.0  0.1   8864  2300 ?        S    21:20   0:00 sendmail: ./kA1NKI6P002431 cluster2a.us.messagelabs.com.: client EHLO
root      1400  0.0  0.0   5712   528 ?        S    22:15   0:00 crond
root      1401  0.0  0.0  18028  1924 ?        Ss   22:15   0:00 /usr/bin/php /home/bobbitt/bin/dreamhost-backup-database.php
root      1469  0.0  0.0   2324   460 ?        S    22:15   0:00 sh -c /usr/bin/mysqldump -CceKq --single-transaction -h db.bobbitt.ca -u bobbitt --password=pw4dream bobbit
root      1470  0.0  0.0   8104  1384 ?        S    22:15   0:00 /usr/bin/mysqldump -CceKq --single-transaction -h db.bobbitt.ca -u bobbitt --password=x xxxxxx bobbitt_gall
root      1471  0.0  0.0   8964   280 ?        S    22:15   0:00 bzip2 -c -9
root      1564  0.0  0.0   8688  1600 ?        S    22:16   0:00 sendmail: kA23GOQU001564 [218.56.58.108]: DATA
defang    1636  1.8  3.9 131624 82208 ?        S    22:16   0:37 /usr/bin/perl /usr/bin/mimedefang.pl -server
root      2512  0.0  0.0      0     0 ?        S    22:24   0:00 [pdflush]
root      2566  0.0  0.1   7840  2388 ?        Ss   22:27   0:00 sshd: bobbitt [priv]
bobbitt   2580  0.0  0.0   7980  1716 ?        S    22:27   0:00 sshd: bobbitt@pts/0
bobbitt   2581  0.0  0.0   5428  1908 pts/0    Ss   22:27   0:00 -tcsh
root      2633  0.0  0.1   5796  2248 pts/0    S    22:28   0:00 -sh
root      3103  0.0  0.0      0     0 ?        S    22:33   0:00 [pdflush]
defang    3276  1.7  6.0 127784 125648 ?       S    22:35   0:15 /usr/bin/perl /usr/bin/mimedefang.pl -server
cyrus     3294  0.0  0.1  31476  2488 ?        S    22:35   0:00 lmtpd
root      3514  0.0  0.1   7840  2388 ?        Ss   22:37   0:00 sshd: bobbitt [priv]
bobbitt   3549  0.0  0.0   7980  1704 ?        S    22:37   0:00 sshd: bobbitt@pts/1
bobbitt   3550  0.0  0.0   5460  1960 pts/1    Ss   22:37   0:00 -tcsh
root      3615  0.0  0.0   5416  1860 pts/1    S    22:38   0:00 -sh
root      3750  0.0  0.0   4404  1120 pts/1    S    22:38   0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --log-error=
mysql     3783  0.4  3.1 470260 65128 pts/1    Sl   22:38   0:02 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-
cyrus     3861  0.0  0.1  31496  2468 ?        S    22:39   0:00 lmtpd
cyrus     4062  0.0  0.1  31488  2512 ?        S    22:42   0:00 lmtpd
cyrus     4065  0.0  0.1  31484  2524 ?        S    22:42   0:00 lmtpd
root      4181  0.3  0.0   2260  1052 pts/1    S+   22:44   0:01 top
root      4232  0.0  0.1   8708  2408 ?        Ss   22:45   0:00 sendmail: ./kA23jLoL004229 localhost: client DATA 354
cyrus     4277  0.0  0.1  31488  2488 ?        S    22:46   0:00 lmtpd
root      4336  0.0  0.1   8688  2940 ?        S    22:47   0:00 sendmail: server raq6.hiwaay.net [216.180.102.66] cmd read
root      4353  0.0  0.1   8688  2940 ?        S    22:47   0:00 sendmail: server mailbarlovento.fishnavy.inf.cu [200.55.133.85] cmd read
root      4374  0.2  0.5  28400 11716 ?        Ss   22:47   0:00 /usr/sbin/httpd
apache    4377  0.9  0.8  37244 16780 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4378  1.1  0.8  38936 18572 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4379 18.7  0.6  34056 13516 ?        S    22:47   0:25 /usr/sbin/httpd
apache    4380  2.6  0.8  37372 16888 ?        S    22:47   0:03 /usr/sbin/httpd
apache    4381  7.1  0.6  34284 13776 ?        S    22:47   0:09 /usr/sbin/httpd
apache    4382  0.7  0.8  37344 16900 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4384  0.2  0.5  31216 11844 ?        S    22:47   0:00 /usr/sbin/httpd
apache    4385  7.5  0.8  37344 16956 ?        S    22:47   0:10 /usr/sbin/httpd
apache    4388  0.4  0.8  37304 16808 ?        S    22:47   0:00 /usr/sbin/httpd
apache    4389  1.0  0.8  37568 17108 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4390  8.6  0.7  35052 14540 ?        S    22:47   0:11 /usr/sbin/httpd
apache    4391 11.7  0.6  34056 13524 ?        S    22:47   0:15 /usr/sbin/httpd
apache    4392  1.3  0.7  35296 14792 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4393 17.2  0.6  34960 14448 ?        S    22:47   0:23 /usr/sbin/httpd
cyrus     4395  0.0  0.1  30712  2112 ?        S    22:47   0:00 imapd -s
apache    4426  0.5  0.8  38948 18140 ?        S    22:47   0:00 /usr/sbin/httpd
apache    4433  0.8  0.8  37228 16704 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4438  0.8  0.8  38924 18424 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4446  0.2  0.7  37224 16396 ?        S    22:47   0:00 /usr/sbin/httpd
apache    4447  1.0  0.8  37336 16860 ?        S    22:47   0:01 /usr/sbin/httpd
apache    4457  0.9  0.8  37344 16880 ?        S    22:48   0:01 /usr/sbin/httpd
apache    4459  6.1  0.8  37320 16684 ?        S    22:48   0:07 /usr/sbin/httpd
apache    4460  1.0  0.8  37332 16708 ?        S    22:48   0:01 /usr/sbin/httpd
apache    4461  0.8  0.7  35660 15020 ?        R    22:48   0:01 /usr/sbin/httpd
apache    4462  0.3  0.6  34604 14044 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4528  0.2  0.6  34764 14140 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4532 10.1  0.6  34728 13940 ?        S    22:48   0:11 /usr/sbin/httpd
apache    4533 14.9  0.8  37348 16796 ?        S    22:48   0:16 /usr/sbin/httpd
apache    4534  1.2  0.8  37252 16820 ?        S    22:48   0:01 /usr/sbin/httpd
apache    4536 11.3  0.8  37244 16840 ?        S    22:48   0:12 /usr/sbin/httpd
apache    4537  0.4  0.6  34560 14024 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4542  0.4  0.7  37208 16272 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4543  1.3  0.8  37692 17144 ?        S    22:48   0:01 /usr/sbin/httpd
apache    4544 11.4  0.8  37232 16696 ?        S    22:48   0:11 /usr/sbin/httpd
apache    4545 11.0  0.6  34308 13680 ?        S    22:48   0:11 /usr/sbin/httpd
apache    4546  0.4  0.6  34328 13704 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4548  0.6  0.8  37368 16976 ?        S    22:48   0:00 /usr/sbin/httpd
apache    4554  0.3  0.7  37264 16492 ?        S    22:48   0:00 /usr/sbin/httpd
root      4581  0.0  0.1   8840  2672 ?        Ss   22:48   0:00 sendmail: ./kA23mYU1004581 mail.greatlooking.com.: user open
cyrus     4584  0.0  0.1  31496  2400 ?        S    22:48   0:00 lmtpd
apache    4723  1.0  0.6  34288 13580 ?        S    22:49   0:00 /usr/sbin/httpd
root      4770  0.0  0.0   8516  1436 ?        S    22:50   0:00 sendmail: startup with 221.1.206.4
apache    4783  0.0  0.0  35552  1696 ?        Rl   22:50   0:00 host -W 1 206.75.114.145
root      4787  0.0  0.0   4180   932 pts/0    R+   22:50   0:00 ps aux


I'm using RPMs for php and apache:

php-5.1.6-1.1
httpd-2.2.2-1.2

I think what I really need is to find out why httpd has suddenly started sucking up so much CPU. Are there any troubleshooting tools that might let me peer into the inner workings of httpd? Any insight would be appreciated.


Thanks
Mike

Ben_S

If you are using eAccelerator or APC, try disabling them temporarrily and see if that helps, sometimes they have a tendency to screw up and do weird things, especially eAccelerator.

That said, you load currently looks fine according to status.php
Liverpool FC Forum with 14 million+ posts.

Mike Bobbitt

Hi Ben,

Yep, sometime through the night the issue seemed to go away on it's own. Mysterious and a bit disturbing.

No accelerators are enabled at the moment, I suspected a faulty apache module so I tried to disable everything I could.

Christian A. Herrnboeck

That looks almost like someone is trying to do a DOS attack on your box. Apache is starting up way to many threads, when you just turn it on. The best way to pinpoint where it's comming from would be to install phpSuExec (there should be a package called php-phpsuexec in your repo).

Regards,
Christian


Farmers:Producing food for the world!

Mike Bobbitt

Hi Christian,

mod_suexec is already loaded as an apache module. I'm starting to agree re: the DOS, as it comes and goes regardless of what I'm doing on the server. To me, that indicates something external.

Having said that, the logs don't indicate an upsurge in traffic or an odd patter from any particular IP(s).

Christian A. Herrnboeck

Hi Mike,


With forums, you really don't need too much of an upswing. For instance, requesting a topic 2,600 times in one hour is enough to slow down a server. (Believe me, I've had to deal with that more than once already :D)

Can you post the output of:

/etc/init.d/httpd_OR_apache_OR_apache2 fullstatus

Make sure to replace httpd_OR_apache_OR_apache2 with your actual apache executable. On most Debian systems, this is apache for the 1.x series, and apache2 for the 2.x series. On RPM/Redhat/CentOS/Fedora systems, it'll usually be httpd

Regards,
Christian A. Herrnboeck


Farmers:Producing food for the world!

Mike Bobbitt

#8
Thanks Christian, I didn't know about that, but it looks extremely useful! Here's the output:

Apache Server Status for army.ca

Server Version: Apache
Server Built: Jul 26 2006 11:12:08

Current Time: Thursday, 02-Nov-2006 17:29:58 EST
Restart Time: Thursday, 02-Nov-2006 17:07:03 EST
Parent Server Generation: 6
Server uptime: 22 minutes 55 seconds
Total accesses: 15971 - Total Traffic: 69.2 MB
CPU Usage: u657.86 s14.76 cu9.11 cs0 - 49.6% CPU load
11.6 requests/sec - 51.5 kB/second - 4542 B/request
55 requests currently being processed, 20 idle workers

W_..K_WWWW._KCWKWKKWWWW___WW.WWWW.KWKC_WW..CWCWW_..WK.WKCKW_WWW_
_W__WK__W_WW__KW_.WW_W..........................................
................................................................
................................................................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-6 22291 0/200/269 W 23.34 0 0 0.0 0.74 1.20 216.58.41.139 army.ca GET /server-status HTTP/1.1
1-6 22292 0/170/268 _ 14.41 4 4993 0.0 0.59 0.96 68.147.40.10 forums.army.ca GET /forums/index.php/topic,52093.15.html HTTP/1.1
2-6 - 0/0/390 . 18.87 35 0 0.0 0.00 1.44 ::1 army.ca GET / HTTP/1.0
3-6 - 0/0/236 . 11.94 27 0 0.0 0.00 1.13 ::1 army.ca GET / HTTP/1.0
4-6 22440 1/254/353 K 6.70 0 871 1.2 0.64 0.95 24.69.176.244 army.ca GET /forums/index.php?sa=news;type=rss;action=.xml HTTP/1.1
5-6 23401 0/117/339 _ 2.36 3 5742 0.0 0.52 1.82 70.68.70.197 forums.army.ca GET /forums/index.php/topic,52696.0/topicseen.html HTTP/1.1
6-6 23416 0/42/257 W 10.09 98 0 0.0 0.08 0.88 24.150.135.88 army.ca GET // HTTP/1.1
7-6 23686 0/32/274 W 1.95 53 0 0.0 0.13 1.37 142.22.186.13 army.ca GET / HTTP/1.0
8-6 23832 0/14/232 W 1.59 98 0 0.0 0.12 1.35 69.197.17.50 army.ca GET / HTTP/1.1
9-6 23936 0/25/257 W 1.59 15 0 0.0 0.21 1.07 64.228.0.33 army.ca GET / HTTP/1.1
10-6 - 0/0/270 . 10.47 35 66 0.0 0.00 1.32 194.72.70.106 forums.army.ca GET /forums/index.php?PHPSESSID=nhlihec2vj9nohniiv8lbl9v43&amp;
11-6 23340 0/46/208 _ 17.45 1 0 0.0 0.25 0.97 70.71.56.184 army.ca GET /forums/Themes/default/images/valid-css.gif HTTP/1.0
12-6 23417 1/89/309 K 3.21 0 477 9.1 0.45 1.45 83.70.248.248 forums.army.ca GET /gallery2/main.php?g2_itemId=17191 HTTP/1.1
13-6 23696 6/24/329 C 9.82 0 0 0.0 0.14 1.35 74.104.104.208 army.ca GET /forums/Themes/default/xml_board.js HTTP/1.1
14-6 23341 0/53/213 W 10.14 82 0 0.0 0.17 0.84 131.137.245.200 army.ca GET / HTTP/1.0
15-6 22065 1/255/296 K 25.49 0 12 0.0 0.82 1.09 207.232.97.25 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=21&clientid=14&zoneid=7&s
16-6 24142 0/0/217 W 28.37 39 0 0.0 0.00 0.95 64.5.252.186 army.ca GET / HTTP/1.0
17-6 23833 5/66/287 K 1.35 0 0 0.0 0.29 0.94 207.232.97.25 army.ca GET /ads/images/armyposter.jpg HTTP/1.1
18-6 23001 3/63/226 K 27.13 0 0 0.0 0.26 0.72 207.232.97.25 forums.army.ca GET /forums/Themes/default/images/titlebg.jpg HTTP/1.1
19-6 23698 1/25/255 W 10.01 1 0 0.8 0.07 0.93 24.69.176.244 army.ca GET /forums/index.php?sa=recent;type=rss;action=.xml HTTP/1.1
20-6 23087 0/41/210 W 2.89 268 0 0.0 0.18 0.66 74.119.103.12 army.ca GET / HTTP/1.1
21-6 23343 0/9/177 W 0.90 326 0 0.0 0.03 0.98 69.196.146.128 army.ca GET // HTTP/1.1
22-6 23699 0/6/175 W 0.82 215 0 0.0 0.02 0.69 74.119.103.12 army.ca GET / HTTP/1.1
23-6 23720 0/39/342 _ 10.54 3 0 0.0 0.09 1.09 70.68.70.197 army.ca GET /forums/Themes/default/spellcheck.js HTTP/1.1
24-6 23423 0/80/301 _ 2.84 0 18 0.0 0.30 1.09 142.162.197.127 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=58&clientid=2&zoneid=7&so
25-6 22449 0/250/379 _ 10.37 3 0 0.0 0.88 1.58 70.68.70.197 army.ca GET /forums/Themes/default/xml_topic.js HTTP/1.1
26-6 23005 0/55/210 W 14.71 108 0 0.0 0.22 1.27 24.89.237.153 army.ca GET // HTTP/1.1
27-6 24042 0/22/214 W 0.24 35 0 0.0 0.11 0.79 64.12.116.195 army.ca GET / HTTP/1.1
28-6 - 0/0/273 . 13.53 61 0 0.0 0.00 0.83 ::1 army.ca GET / HTTP/1.0
29-6 23006 0/37/249 W 11.24 341 0 0.0 0.16 0.83 64.39.167.156 army.ca GET // HTTP/1.1
30-6 23092 0/38/180 W 1.97 305 0 0.0 0.18 0.65 69.196.146.128 army.ca GET // HTTP/1.1
31-6 23093 0/51/277 W 1.08 331 0 0.0 0.25 1.16 64.12.116.195 army.ca GET / HTTP/1.1
32-6 23425 0/49/203 W 2.93 4 0 0.0 0.51 1.12 68.147.40.10 forums.army.ca GET /forums/index.php/topic,52093.15.html HTTP/1.1
33-6 - 0/0/185 . 25.67 32 0 0.0 0.00 1.23 ::1 army.ca GET / HTTP/1.0
34-6 23721 22/46/236 K 9.34 3 0 0.0 0.07 1.26 216.168.120.63 army.ca GET /forums/Smileys/Armyca/sm_bullet.gif HTTP/1.1
35-6 23448 0/22/265 W 0.34 253 0 0.0 0.15 1.38 70.48.217.191 army.ca GET / HTTP/1.1
36-6 22455 1/174/246 K 21.91 1 111 0.0 0.52 1.02 207.47.230.189 forums.army.ca GET /forums/index.php?action=keepalive;1162506594651 HTTP/1.1
37-6 22456 4/193/247 C 20.10 0 0 0.0 0.73 0.98 74.104.104.208 army.ca GET /forums/Themes/default/images/topic/my_normal_post.gif HTTP
38-6 23449 0/61/179 _ 10.27 2 561 0.0 0.24 0.64 24.70.95.203 forums.army.ca GET /forums/index.php/topic,51833.30/topicseen.html HTTP/1.1
39-6 23450 0/45/204 W 10.68 65 0 0.0 0.22 0.76 24.70.95.204 army.ca GET / HTTP/1.1
40-6 23008 2/94/186 W 21.26 62 0 47.7 0.56 1.40 24.70.95.204 army.ca GET / HTTP/1.1
41-6 - 0/0/225 . 18.28 11 0 0.0 0.00 0.75 199.213.122.87 forums.army.ca GET /forums/Themes/default/images/maintab_active_first.gif HTTP
42-6 - 0/0/152 . 3.42 65 0 0.0 0.00 0.49 ::1 army.ca GET / HTTP/1.0
43-6 23453 1/74/234 C 11.61 1 0 8.0 0.20 1.27 142.162.197.127 army.ca GET /forums/avatars/User-Uploaded/avatar_5333.png HTTP/1.1
44-6 23010 0/75/230 W 4.29 244 0 0.0 0.28 0.89 142.22.186.13 army.ca GET / HTTP/1.0
45-6 23011 2/86/281 C 19.64 1 107 10.6 0.38 1.54 142.162.197.127 forums.army.ca GET /forums/index.php/topic,49515.0/prev_next,next.html HTTP/1.
46-6 23722 0/37/108 W 2.18 18 0 0.0 0.32 0.65 70.48.231.244 army.ca GET // HTTP/1.1
47-6 22743 0/133/197 W 7.06 15 0 0.0 0.60 0.83 216.186.72.88 army.ca GET /inf/torscot.php HTTP/1.1
48-6 23454 0/51/229 _ 17.14 1 259 0.0 0.26 0.86 24.70.95.203 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=27&clientid=2&zoneid=7&so
49-6 - 0/0/311 . 11.86 33 0 0.0 0.00 1.51 ::1 army.ca GET / HTTP/1.0
50-6 - 0/0/245 . 4.02 64 0 0.0 0.00 1.52 ::1 army.ca GET / HTTP/1.0
51-6 23099 0/53/187 W 1.99 275 0 0.0 0.22 0.81 64.39.167.156 army.ca GET // HTTP/1.1
52-6 23100 35/166/229 K 10.92 3 0 0.0 0.28 0.48 216.168.120.63 army.ca GET /forums/Themes/default/images/bbc/bbc_bg.gif HTTP/1.1
53-6 - 0/0/255 . 1.60 52 14 0.0 0.00 1.19 216.208.67.182 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=72&clientid=35&zoneid=7&s
54-6 23101 0/122/125 W 9.54 12 0 0.0 0.38 0.38 131.137.245.199 army.ca GET / HTTP/1.0
55-6 23459 1/75/259 K 11.78 1 0 0.0 0.23 0.95 205.206.175.230 army.ca GET /album/Humour/thmb_Drill_Yells.png HTTP/1.1
56-6 23723 3/46/141 C 2.30 1 0 35.2 0.12 0.39 142.162.197.127 army.ca GET /forums/avatars/User-Uploaded/avatar_17855.png HTTP/1.1
57-6 23460 1/51/64 K 10.69 0 0 0.0 0.24 0.30 207.232.97.25 army.ca GET /forums/Themes/default/images/upshrink.gif HTTP/1.1
58-6 23835 0/16/35 W 9.88 4 0 0.0 0.07 0.12 64.12.116.195 army.ca GET / HTTP/1.1
59-6 24043 0/13/65 _ 0.46 6 0 0.0 0.02 0.25 142.163.162.54 army.ca GET /forums/Themes/default/images/icons/login_sm.gif HTTP/1.1
60-6 23836 0/52/188 W 0.66 58 0 0.0 0.15 0.81 64.5.246.234 army.ca GET / HTTP/1.0
61-6 23468 0/57/89 W 2.76 77 0 0.0 0.37 0.76 24.70.95.204 army.ca GET / HTTP/1.1
62-6 23837 0/35/183 W 1.38 34 0 0.0 0.09 0.67 131.137.245.197 army.ca GET / HTTP/1.0
63-6 23469 0/86/90 _ 4.15 4 295 0.0 0.41 0.41 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
64-6 24044 0/36/130 _ 0.58 1 12 0.0 0.08 0.69 207.112.49.18 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=87&clientid=2&zoneid=7&so
65-6 23839 0/14/25 W 0.74 90 0 0.0 0.06 0.08 72.58.29.205 army.ca GET / HTTP/1.1
66-6 23840 0/23/30 _ 0.99 5 14 0.0 0.06 0.06 70.68.70.197 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=23&clientid=2&zoneid=7&so
67-6 23121 0/60/162 _ 13.12 5 5 0.0 0.37 0.66 64.26.171.13 army.ca GET /server-status HTTP/1.1
68-6 23845 0/21/156 W 0.90 59 0 0.0 0.06 0.66 206.132.51.144 army.ca GET / HTTP/1.1
69-6 24055 2/9/42 K 1.11 2 12 5.2 0.02 0.22 142.163.162.54 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=76&clientid=42&zoneid=7&s
70-6 21862 0/361/361 _ 36.56 5 630 0.0 1.44 1.44 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
71-6 23124 0/42/45 _ 9.69 0 951 0.0 0.19 0.20 66.249.65.109 army.ca GET /forums/index.php/topic,25548.0.html HTTP/1.1
72-6 23847 0/51/216 W 1.96 18 0 0.0 0.11 1.48 24.150.75.192 army.ca GET / HTTP/1.1
73-6 24056 0/9/76 _ 2.44 6 13 0.0 0.26 0.50 142.163.162.54 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=74&clientid=38&zoneid=7&s
74-6 24057 0/6/19 W 0.00 61 0 0.0 0.02 0.03 64.39.167.156 army.ca GET // HTTP/1.1
75-6 23127 0/31/31 W 9.22 300 0 0.0 0.28 0.28 64.12.116.195 army.ca GET / HTTP/1.1
76-6 23860 0/48/48 _ 1.15 1 0 0.0 0.24 0.24 70.71.56.184 army.ca GET /forums/Themes/default/images/valid-xhtml10.gif HTTP/1.0
77-6 24058 0/17/17 _ 0.70 1 12 0.0 0.21 0.21 216.168.120.63 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=76&clientid=42&zoneid=7&s
78-6 24064 1/7/7 K 1.57 0 924 14.2 0.06 0.06 66.249.65.109 forums.army.ca GET /forums/index.php?topic=41980.msg374678 HTTP/1.1
79-6 24065 0/24/24 W 2.17 9 0 0.0 0.18 0.18 209.226.192.120 army.ca GET / HTTP/1.1
80-6 24066 0/10/10 _ 1.59 0 29 0.0 0.07 0.07 74.104.104.208 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=21&clientid=14&zoneid=7&s
81-6 - 0/0/1 . 0.00 63 1 0.0 0.00 0.00 64.26.171.13 army.ca GET /forums/avatars/User-Uploaded/avatar_1.png HTTP/1.1
82-6 24083 0/4/4 W 0.09 38 0 0.0 0.02 0.02 64.5.246.234 army.ca GET / HTTP/1.0
83-6 24084 0/7/7 W 0.51 6 0 0.0 0.00 0.00 24.70.95.205 army.ca GET / HTTP/1.1
84-6 24085 0/8/8 _ 1.26 1 274 0.0 0.09 0.09 207.232.97.25 forums.army.ca GET /forums/index.php/topic,52703.new.html HTTP/1.1
85-6 24088 0/7/7 W 0.31 20 0 0.0 0.02 0.02 216.186.72.88 army.ca GET /inf/torscot.php HTTP/1.1
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
CPU CPU usage, number of seconds
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 1
sub-caches: 32, indexes per sub-cache: 133
time left on oldest entries' SSL sessions: avg: 232 seconds, (range: 232...232)
index usage: 0%, cache usage: 0%
total sessions stored since starting: 3
total sessions expired since starting: 2
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 9 hit, 2 miss
total removes since starting: 0 hit, 0 miss
Apache Server at army.ca Port 443


Load average is a mere 4 right now, so I may not be cathing the problem in the act, but I'll keep looking.

Edit: Updated it with the output during a higher load average (around 25).

Christian A. Herrnboeck



40-6 23008 2/94/186 W 21.26 62 0 47.7 0.56 1.40 24.70.95.204 army.ca GET / HTTP/1.1


This line intrigues me. It seems that it's transferring an unusually high amount of data, especially for the GET requests it is processing.

Other than that, it looks OK... It may be easier to find the problem when load gets closer to 50 -  75...


Regards,
Christian



Farmers:Producing food for the world!

Mike Bobbitt

Here's a version that shows the problem more clearly, I believe:

Apache Server Status for army.ca

Server Version: Apache
Server Built: Jul 26 2006 11:12:08

Current Time: Friday, 03-Nov-2006 10:54:28 EST
Restart Time: Thursday, 02-Nov-2006 19:59:15 EST
Parent Server Generation: 7
Server uptime: 14 hours 55 minutes 12 seconds
Total accesses: 342208 - Total Traffic: 1.6 GB
CPU Usage: u13421.5 s460 cu553.63 cs0 - 26.9% CPU load
6.37 requests/sec - 31.6 kB/second - 5082 B/request
117 requests currently being processed, 139 idle workers

__WWWW__W_W__WWW__W__C_W_WWWW_W_CWCW___WWW_W_WWWWW_____LWLWWW_WW
L__WRW__WWW__W___WW_CWWWWR___WWW______W____W__R___WWWW__CW_W_WW_
__LWLWWW_W___W______W_W__WWR__W_W_WWWW_______RC_W____W____C____W
_C___CC_WCR_W___W_C_C_C_________W___C___W____W___W_WR__C_RW_W_W_

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-7 6093 0/1513/7750 _ 139.37 12 374 0.0 8.13 35.46 131.137.245.200 forums.army.ca GET /images/blank404.jpg HTTP/1.0
1-7 22599 0/7487/10526 _ 666.47 16 419670 0.0 36.53 52.10 216.209.114.174 army.ca GET / HTTP/1.1
2-7 17450 0/3568/7892 W 316.59 797 0 0.0 18.92 38.71 64.230.35.66 army.ca GET /forums/ HTTP/1.1
3-7 14496 0/2/6170 W 0.05 767 0 0.0 0.06 27.71 137.94.192.214 forums.army.ca GET /forums/index.php?action=unread HTTP/1.1
4-7 19197 0/7941/10673 W 793.18 743 0 0.0 39.09 51.94 69.198.194.208 army.ca GET / HTTP/1.1
5-7 31057 0/6149/9406 W 607.54 740 0 0.0 31.70 47.76 63.243.150.26 army.ca GET // HTTP/1.1
6-7 14507 0/5/5965 _ 0.35 8 153 0.0 0.00 27.64 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_first.gif HTTP/1.1
7-7 17453 0/3584/6894 _ 326.94 12 321 0.0 18.03 31.95 131.137.245.197 forums.army.ca GET /images/bar404.gif HTTP/1.0
8-7 14508 0/1/2825 W 0.37 482 0 0.0 0.01 12.35 216.209.114.174 forums.army.ca GET /forums/index.php/board,8.80.html HTTP/1.1
9-7 14509 0/4/5773 _ 0.99 8 157 0.0 0.03 27.25 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_active_last.gif HTTP/
10-7 17456 0/3525/7725 W 335.31 780 0 0.0 18.35 36.55 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51554.msg474116;topicseen HTTP/1.1
11-7 17457 0/3479/6839 _ 354.70 16 528928 0.0 17.98 33.57 66.160.186.32 forums.army.ca GET /forums/index.php?topic=51805.0;all HTTP/1.0
12-7 14524 0/2/5649 _ 0.98 14 1600 0.0 0.01 27.86 69.158.65.169 forums.army.ca GET /forums/index.php?action=unread HTTP/1.1
13-7 18092 0/8543/9531 W 771.53 772 0 0.0 43.71 48.59 131.137.245.199 forums.army.ca GET /forums/index.php/topic,51916.15.html HTTP/1.0
14-7 17459 0/3546/7502 W 367.91 59 0 0.0 18.93 36.41 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
15-7 14536 0/2/5062 W 0.35 17 0 0.0 0.01 23.10 216.209.114.174 army.ca GET / HTTP/1.1
16-7 14537 0/3/5574 _ 0.77 13 563 0.0 0.02 25.32 142.166.239.108 army.ca GET /forums/Themes/default/images/smflogo.gif HTTP/1.1
17-7 14541 0/4/2039 _ 0.59 0 0 0.0 0.01 9.39 ::1 army.ca GET / HTTP/1.0
18-7 17470 0/3568/6828 W 305.56 767 0 0.0 19.13 34.55 205.250.206.150 forums.army.ca GET /forums/index.php?action=profile;u=2443;sa=showPosts;start=
19-7 14546 0/3/5530 _ 1.09 8 1431 0.0 0.02 25.89 196.207.32.38 forums.army.ca GET /forums/index.php?topic=36923.0;wap2 HTTP/1.1
20-7 14547 0/9/5399 _ 0.98 16 448 0.0 0.02 24.64 216.209.114.174 army.ca GET / HTTP/1.1
21-7 14548 1/3/4060 C 0.79 2 27173 0.0 0.01 20.09 207.179.148.147 army.ca GET / HTTP/1.1
22-7 14551 0/4/6772 _ 0.57 16 13267 0.0 0.02 30.96 70.26.147.197 army.ca GET /forums/ HTTP/1.1
23-7 31141 0/6105/9275 W 545.09 776 0 0.0 31.59 45.87 131.137.245.197 army.ca GET /forums/index.php HTTP/1.0
24-7 14552 0/4/3364 _ 0.54 16 1643 0.0 0.02 15.15 69.159.152.61 forums.army.ca GET /forums/index.php?topic=31342.msg263672 HTTP/1.1
25-7 17491 0/3597/6103 W 333.59 761 0 0.0 18.86 30.32 131.137.245.197 forums.army.ca GET /forums/index.php/topic,51916.0.html HTTP/1.0
26-7 17496 0/3599/7574 W 307.89 796 0 0.0 18.80 37.17 209.146.168.34 army.ca GET / HTTP/1.1
27-7 31143 0/6164/9747 W 513.94 799 0 0.0 31.87 48.77 24.76.123.82 forums.army.ca GET /forums/index.php/topic,51916.0.html HTTP/1.1
28-7 14559 0/1/2143 W 0.58 719 0 0.0 0.01 10.78 63.243.150.26 army.ca GET // HTTP/1.1
29-7 14563 0/3/5960 _ 0.55 16 14153 0.0 0.01 29.44 74.119.48.151 army.ca GET / HTTP/1.1
30-7 14564 0/2/5290 W 0.09 736 0 0.0 0.06 25.61 81.230.102.80 forums.army.ca GET /forums/index.php/topic,50320.0.html HTTP/1.1
31-7 17498 0/3578/4727 _ 339.87 8 334 0.0 19.21 24.92 142.166.239.108 army.ca GET /forums/Themes/default/images/powered-mysql.gif HTTP/1.1
32-7 20684 1/7682/7864 C 770.41 0 160 0.0 39.37 40.28 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_active_first.gif HTTP
33-7 17499 0/3554/4458 W 273.39 786 0 0.0 19.64 23.78 69.198.194.208 army.ca GET / HTTP/1.1
34-7 14565 1/4/985 C 0.45 13 0 0.0 0.01 4.85 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/titlebg.jpg HTTP/1.1
35-7 14568 0/0/929 W 0.34 750 0 0.0 0.00 4.99 142.167.0.184 army.ca GET / HTTP/1.1
36-7 14569 0/5/3415 _ 0.68 0 123 0.0 0.01 16.49 ::1 army.ca GET / HTTP/1.0
37-7 24659 0/7181/8234 _ 638.47 16 491248 0.0 33.84 38.65 66.160.186.32 forums.army.ca GET /forums/index.php?topic=30267.60 HTTP/1.0
38-7 14594 0/2/1017 _ 0.52 16 495836 0.0 0.01 4.95 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
39-7 31159 0/6101/9601 W 529.30 65 0 0.0 32.57 48.08 131.137.245.197 forums.army.ca POST /gallery2/main.php HTTP/1.0
40-7 31160 0/6021/6886 W 581.20 176 0 0.0 32.39 37.13 147.240.16.51 army.ca GET / HTTP/1.1
41-7 17531 0/3588/6575 W 311.41 774 0 0.0 18.19 32.42 198.103.104.12 army.ca GET / HTTP/1.1
42-7 14595 0/4/4326 _ 1.06 16 13184 0.0 0.02 19.62 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
43-7 20810 0/7945/7992 W 638.16 780 0 0.0 39.20 39.56 24.141.117.163 forums.army.ca GET /forums/index.php?topic=51916.new;topicseen HTTP/1.1
44-7 14596 0/1/3556 _ 0.42 0 740134 0.0 0.00 17.02 131.137.245.197 forums.army.ca GET /forums/index.php/topic,40792.165.html HTTP/1.0
45-7 14613 0/0/5857 W 421.50 736 0 0.0 0.00 27.49 142.167.0.184 army.ca GET / HTTP/1.1
46-7 14618 0/1/1118 W 0.01 732 0 0.0 0.00 4.80 192.197.178.2 forums.army.ca GET /forums/index.php?board=1.0 HTTP/1.1
47-7 14619 0/2/227 W 0.36 226 0 0.0 0.01 0.95 63.243.150.22 forums.army.ca GET /forums/index.php?action=unread HTTP/1.0
48-7 14620 0/0/1166 W 0.16 732 0 0.0 0.00 5.53 131.137.245.199 forums.army.ca POST /forums/index.php?action=pm;sa=send2 HTTP/1.0
49-7 31169 0/6169/6428 W 438.91 736 0 0.0 31.94 33.24 209.115.236.89 army.ca GET / HTTP/1.1
50-7 14621 0/4/3761 _ 0.35 0 1363 0.0 0.00 16.38 ::1 army.ca GET / HTTP/1.0
51-7 14622 0/2/1775 _ 0.50 12 707857 0.0 0.00 8.14 216.161.236.187 forums.army.ca GET /forums/index.php?topic=41980.0;all HTTP/1.1
52-7 14623 0/6/136 _ 1.06 16 0 0.0 0.02 0.76 67.149.76.130 forums.army.ca GET /favicon.ico HTTP/1.1
53-7 14625 0/1/1168 _ 0.58 9 714008 0.0 0.00 4.95 140.184.72.180 forums.army.ca GET /forums/index.php/topic,45008.90.html HTTP/1.1
54-7 17547 0/3622/3756 _ 313.59 16 13256 0.0 17.73 18.29 131.137.245.197 forums.army.ca GET /forums/images/bar404.gif HTTP/1.0
55-7 14626 0/0/1671 L 0.16 1 0 0.0 0.00 7.26 24.72.1.32 army.ca GET / HTTP/1.1
56-7 17558 0/3547/4756 W 410.02 68 0 0.0 19.85 24.88 69.180.189.66 forums.army.ca GET /forums/index.php/topic,24003.0.html HTTP/1.1
57-7 14627 0/1/1816 L 0.07 2 0 0.0 0.00 7.62 131.137.245.199 forums.army.ca GET /forums/index.php/topic,40792.165.html HTTP/1.0
58-7 14628 0/4/138 W 0.11 707 0 0.0 0.07 0.67 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51554.msg474116;topicseen HTTP/1.1
59-7 14629 0/0/802 W 0.40 715 0 0.0 0.00 4.16 131.137.245.198 forums.army.ca GET /forums/index.php/topic,40792.165.html HTTP/1.0
60-7 24844 0/7077/7232 W 628.13 12 0 0.0 35.48 36.25 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_active_first.gif HTTP
61-7 14639 0/4/279 _ 0.30 0 7846 0.0 0.01 1.34 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_last.gif HTTP/1.1
62-7 14640 0/0/1252 W 0.36 715 0 0.0 0.00 5.82 199.212.250.95 army.ca GET /inf/rcr.php HTTP/1.1
63-7 14641 0/2/71 W 0.36 29 0 0.0 0.01 0.56 74.119.48.151 army.ca GET / HTTP/1.1
64-7 14642 0/0/2708 L 0.36 50 0 0.0 0.00 11.66 63.243.150.26 forums.army.ca GET /forums/index.php/board,6 HTTP/1.1
65-7 14643 0/11/942 _ 0.66 0 124 0.0 0.01 3.91 ::1 army.ca GET / HTTP/1.0
66-7 14644 0/3/56 _ 0.35 13 0 0.0 0.00 0.38 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/catbg.jpg HTTP/1.1
67-7 14645 0/0/44 W 0.00 705 0 0.0 0.00 0.19 24.76.123.82 forums.army.ca GET /forums/index.php HTTP/1.1
68-7 14646 0/2/745 R 0.30 16 251214 0.0 0.01 3.59 ? ? ..reading..
69-7 24940 0/7013/8721 W 675.97 803 0 0.0 35.21 42.19 70.49.39.249 army.ca GET / HTTP/1.1
70-7 17607 0/3595/3721 _ 335.61 16 42339 0.0 18.43 18.92 192.168.1.7 army.ca GET /quotes/index.php?ssi=1 HTTP/1.0
71-7 17609 0/3596/5215 _ 347.45 15 479359 0.0 18.43 25.35 66.160.186.32 army.ca GET /forums/index.php/topic,16339.15.html HTTP/1.0
72-7 14647 0/1/82 W 0.04 700 0 0.0 0.00 0.55 131.137.245.197 army.ca GET / HTTP/1.0
73-7 14648 0/2/1042 W 0.28 57 0 0.0 0.02 4.62 24.150.89.123 forums.army.ca GET /forums/index.php?action=keepalive;1162568691531 HTTP/1.1
74-7 14649 0/0/2278 W 102.99 701 0 0.0 0.00 9.32 131.137.245.200 army.ca GET / HTTP/1.0
75-7 14650 0/4/90 _ 0.91 8 976 0.0 0.06 0.56 67.149.76.130 forums.army.ca GET /forums/index.php?topic=36209.msg292812 HTTP/1.1
76-7 14652 0/4/65 _ 0.46 16 14225 0.0 0.01 0.30 64.231.231.248 army.ca GET / HTTP/1.1
77-7 14653 0/3/78 W 0.40 29 0 0.0 0.01 0.43 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
78-7 14654 0/3/75 _ 0.36 12 936 0.0 0.02 0.42 142.166.239.108 army.ca GET /forums/Themes/default/images/upshrink.gif HTTP/1.1
79-7 14664 0/6/19 _ 0.69 0 4365 0.0 0.00 0.08 ::1 army.ca GET / HTTP/1.0
80-7 14665 0/10/37 _ 1.74 16 43753 0.0 0.01 0.12 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
81-7 14668 0/0/0 W 0.00 690 1367559027 0.0 0.00 0.00 146.214.103.8 forums.army.ca GET /forums/index.php/topic,52255.0/all.html HTTP/1.0
82-7 14669 0/2/2 W 0.78 30 0 0.0 0.02 0.02 66.249.65.109 forums.army.ca GET /gallery2/main.php?g2_itemId=24387&g2_highlightId=24391 HTT
83-7 14670 0/6/6 _ 0.79 8 669 0.0 0.01 0.01 131.137.245.197 army.ca GET / HTTP/1.0
84-7 14671 0/3/3 C 1.42 1 11837 0.0 0.01 0.01 140.184.72.180 forums.army.ca GET /forums/index.php/board,10.20.html HTTP/1.1
85-7 14672 0/0/0 W 0.00 691 1367559770 0.0 0.00 0.00 209.82.105.122 army.ca GET / HTTP/1.1
86-7 14673 0/1/1 W 0.03 685 0 0.0 0.00 0.00 24.69.165.55 forums.army.ca GET /forums/index.php/topic,51554.15.html HTTP/1.1
87-7 14674 0/8/8 W 1.06 17 0 0.0 0.01 0.01 70.104.234.162 forums.army.ca GET /gallery2/main.php?g2_itemId=17191 HTTP/1.1
88-7 14676 0/0/0 W 0.00 684 1367552812 0.0 0.00 0.00 216.209.114.174 forums.army.ca GET /forums/index.php/topic,48452.0.html HTTP/1.1
89-7 14677 0/3/3 R 0.38 16 1802 0.0 0.01 0.01 ? ? ..reading..
90-7 14684 0/3/3 _ 0.88 16 42778 0.0 0.01 0.01 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
91-7 14685 0/9/9 _ 0.31 10 197 0.0 0.02 0.02 ::1 army.ca GET / HTTP/1.0
92-7 14686 0/7/7 _ 0.42 16 546832 0.0 0.02 0.02 156.34.176.184 forums.army.ca GET /forums/index.php?PHPSESSID=s25b5jh3pteo1a1cg0nmd2d5u1&boar
93-7 14687 0/0/0 W 0.00 682 1367551019 0.0 0.00 0.00 140.184.72.180 forums.army.ca GET /forums/index.php/topic,45008.90.html HTTP/1.1
94-7 14688 0/0/0 W 0.00 681 1367550425 0.0 0.00 0.00 74.6.69.13 forums.army.ca GET /forums/index.php?topic=40708.0 HTTP/1.0
95-7 14689 0/0/0 W 0.00 680 1367548659 0.0 0.00 0.00 24.226.86.249 forums.army.ca GET /forums/index.php/topic,51554.15.html HTTP/1.1
96-7 14690 0/1/1 _ 0.28 16 666600 0.0 0.00 0.00 192.197.178.2 forums.army.ca GET /forums/index.php?board=1.0 HTTP/1.1
97-7 14691 0/7/7 _ 0.44 15 548174 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=13652.0;all HTTP/1.0
98-7 14692 0/2/2 _ 0.18 0 123 0.0 0.00 0.00 142.166.239.108 army.ca GET /forums/Themes/default/images/valid-xhtml10.gif HTTP/1.1
99-7 14693 0/4/4 _ 0.89 15 302 0.0 0.03 0.03 142.166.239.108 forums.army.ca GET /forums/index.php/board,49.0.html HTTP/1.1
100-7 14694 0/1/1 _ 0.19 16 665644 0.0 0.00 0.00 131.137.245.199 forums.army.ca GET /forums/index.php?action=unreadreplies HTTP/1.0
101-7 14695 0/8/8 _ 0.65 0 3583 0.0 0.03 0.03 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_last.gif HTTP/1.1
102-7 14696 0/5/5 W 0.21 567 0 0.0 0.00 0.00 131.137.245.198 army.ca GET / HTTP/1.0
103-7 14697 0/5/5 _ 0.68 0 17958 0.0 0.01 0.01 131.137.245.199 army.ca GET /inf/rdusag.php HTTP/1.0
104-7 14698 0/3/3 _ 0.36 15 572505 0.0 0.00 0.00 131.137.245.200 army.ca GET /forums/ HTTP/1.0
105-7 14699 0/3/3 _ 0.30 15 572349 0.0 0.00 0.00 216.209.114.174 forums.army.ca GET /forums/index.php/board,8.80.html HTTP/1.1
106-7 14700 0/3/3 _ 0.74 16 551076 0.0 0.00 0.00 65.92.153.28 forums.army.ca GET /forums/index.php?topic=36209.msg292812 HTTP/1.1
107-7 14701 0/4/4 W 0.11 574 0 0.0 0.00 0.00 68.148.193.152 army.ca GET / HTTP/1.1
108-7 14702 0/1/1 _ 0.59 10 636225 0.0 0.00 0.00 142.163.74.95 forums.army.ca POST /forums/index.php?action=search2 HTTP/1.1
109-7 14703 0/6/6 _ 0.45 16 7927 0.0 0.03 0.03 74.119.228.95 forums.army.ca GET /favicon.ico HTTP/1.1
110-7 14704 0/5/5 R 0.52 29 843 0.0 0.01 0.01 ? ? ..reading..
111-7 14705 0/22/22 _ 2.61 16 12936 0.0 0.02 0.02 131.137.245.200 forums.army.ca GET /forums/images/blank404.jpg HTTP/1.0
112-7 14706 0/2/2 _ 0.18 14 1 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /Army.css?rc3 HTTP/1.1
113-7 14707 0/4/4 _ 0.25 16 563108 0.0 0.01 0.01 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.90.html HTTP/1.0
114-7 14710 0/1/1 W 0.31 15 0 0.0 0.01 0.01 74.114.193.236 forums.army.ca GET /forums/index.php?action=keepalive;1162569217015 HTTP/1.1
115-7 14711 0/4/4 W 0.13 572 0 0.0 0.00 0.00 209.115.236.89 army.ca GET / HTTP/1.1
116-7 14712 0/4/4 W 1.33 55 0 0.0 0.01 0.01 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
117-7 14713 0/3/3 W 0.21 66 0 0.0 0.01 0.01 68.148.193.152 army.ca GET / HTTP/1.1
118-7 14714 0/3/3 _ 0.52 8 49940 0.0 0.01 0.01 131.137.245.198 forums.army.ca GET /forums/index.php?action=keepalive;1162568931980 HTTP/1.0
119-7 14715 0/1/1 _ 0.27 16 641007 0.0 0.00 0.00 206.75.50.143 forums.army.ca GET /forums/index.php/topic,51554.msg474111/topicseen.html HTTP
120-7 14716 1/2/2 C 0.43 10 535051 0.2 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=23622.msg286115 HTTP/1.0
121-7 14717 0/3/3 W 0.10 575 0 0.0 0.00 0.00 142.167.0.184 army.ca GET / HTTP/1.1
122-7 14718 0/1/1 _ 0.24 15 637566 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51554.msg474116;topicseen HTTP/1.1
123-7 14733 0/0/0 W 0.00 658 1367527003 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51554.msg474116;topicseen HTTP/1.1
124-7 14734 0/1/1 _ 0.51 16 630162 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51916.msg474112;topicseen HTTP/1.1
125-7 14735 0/3/3 W 0.45 28 0 0.0 0.01 0.01 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.90.html HTTP/1.0
126-7 14736 0/0/0 W 0.00 646 1367515078 0.0 0.00 0.00 205.250.206.150 forums.army.ca GET /forums/index.php?topic=51898.30 HTTP/1.1
127-7 14767 0/1/1 _ 0.33 8 637356 0.0 0.08 0.08 131.137.245.200 forums.army.ca GET /forums/index.php/topic,25680.15.html HTTP/1.0
128-7 14768 0/5/5 _ 0.52 8 0 0.0 0.01 0.01 ::1 army.ca GET / HTTP/1.0
129-7 14769 0/2/2 _ 0.21 0 8232 0.0 0.00 0.00 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_back.gif HTTP/1.1
130-7 14770 0/0/0 L 0.00 8 1367518576 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51916.msg474112;topicseen HTTP/1.1
131-7 14771 0/0/0 W 0.00 649 1367518531 0.0 0.00 0.00 131.137.245.197 forums.army.ca GET /forums/index.php?PHPSESSID=0v116hsfj6390fde1c67aqvri1&topi
132-7 14772 0/0/0 L 0.00 6 1367518459 0.0 0.00 0.00 131.137.245.200 army.ca GET / HTTP/1.0
133-7 14773 0/0/0 W 0.00 646 1367515099 0.0 0.00 0.00 198.103.104.12 army.ca GET / HTTP/1.1
134-7 14774 0/1/1 W 0.14 3 0 0.0 0.00 0.00 216.58.41.139 army.ca GET /server-status HTTP/1.1
135-7 14775 0/0/0 W 0.00 646 1367515098 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51916.msg474112;topicseen HTTP/1.1
136-7 14776 0/5/5 _ 0.66 16 13469 0.0 0.00 0.00 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=ra
137-7 14777 0/0/0 W 0.00 646 1367515078 0.0 0.00 0.00 206.75.50.143 forums.army.ca GET /forums/index.php/board,37.0.html HTTP/1.1
138-7 14778 0/1/1 _ 0.34 16 630020 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51916.msg474112;topicseen HTTP/1.1
139-7 14783 0/1/1 _ 0.50 14 626242 0.0 0.00 0.00 68.148.193.152 forums.army.ca GET /forums/index.php/topic,52597.0/all.html HTTP/1.1
140-7 14784 0/1/1 _ 0.30 0 622771 0.0 0.00 0.00 142.163.74.95 forums.army.ca POST /forums/index.php?action=search2 HTTP/1.1
141-7 14785 0/0/0 W 0.00 637 1367506033 0.0 0.00 0.00 131.137.245.198 forums.army.ca POST /forums/index.php?action=pm;sa=send2 HTTP/1.0
142-7 14786 0/1/1 _ 0.18 8 620904 0.0 0.00 0.00 198.161.51.186 forums.army.ca GET /forums/index.php?topic=51554.msg474116;topicseen HTTP/1.1
143-7 14787 0/1/1 _ 0.29 15 620695 0.0 0.00 0.00 206.75.50.143 forums.army.ca GET /forums/index.php/topic,52711.msg474105/topicseen.html HTTP
144-7 14788 0/3/3 _ 0.24 9 10 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /ads/phpAdsNew/adlog.php?bannerid=27&clientid=2&zoneid=7&so
145-7 14791 0/1/1 _ 0.34 15 610203 0.0 0.00 0.00 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.90.html HTTP/1.0
146-7 14792 0/1/1 _ 0.16 15 615093 0.0 0.01 0.01 74.12.162.25 army.ca GET / HTTP/1.1
147-7 14795 0/3/3 _ 0.89 16 12350 0.0 0.01 0.01 205.211.50.10 army.ca GET / HTTP/1.1
148-7 14796 0/4/4 W 0.36 66 0 0.0 0.00 0.00 204.40.1.129 forums.army.ca POST /forums/index.php?action=post2;board=22;preview;xml HTTP/1
149-7 14797 0/4/4 _ 0.69 16 542190 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=47156.msg410063 HTTP/1.0
150-7 14798 0/4/4 W 0.22 28 0 0.0 0.00 0.00 131.137.245.197 forums.army.ca GET /forums/index.php?topic=52757.msg474091;topicseen HTTP/1.0
151-7 14799 0/4/4 _ 0.29 15 566903 0.0 0.00 0.00 68.149.166.204 forums.army.ca GET /forums/index.php/topic,51652.0/all.html HTTP/1.1
152-7 14800 0/5/5 _ 0.35 0 15245 0.0 0.01 0.01 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_active_back.gif HTTP/
153-7 14801 0/8/8 W 0.37 56 0 0.0 0.09 0.09 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
154-7 14802 0/3/3 W 0.13 577 0 0.0 0.00 0.00 131.137.245.199 forums.army.ca GET /forums/index.php/topic,52006.45.html HTTP/1.0
155-7 14804 0/9/9 R 0.94 29 29369 0.0 0.01 0.01 ? ? ..reading..
156-7 14805 0/4/4 _ 0.57 12 100 0.0 0.02 0.02 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
157-7 14806 0/5/5 _ 0.86 8 1548 0.0 0.01 0.01 131.137.245.199 army.ca GET //images/bar404.gif HTTP/1.0
158-7 14808 0/0/0 W 0.00 596 1367465205 0.0 0.00 0.00 204.81.7.13 army.ca GET / HTTP/1.1
159-7 14809 0/10/10 _ 1.52 16 12456 0.0 0.02 0.02 74.119.168.246 army.ca GET / HTTP/1.1
160-7 14810 0/1/1 W 0.15 587 0 0.0 0.00 0.00 131.137.245.199 army.ca GET / HTTP/1.0
161-7 14811 0/3/3 _ 0.47 15 1968 0.0 0.00 0.00 131.137.245.200 army.ca GET / HTTP/1.0
162-7 14924 0/0/0 W 0.00 550 1367419563 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /forums/index.php?action=pm;sa=send HTTP/1.1
163-7 14926 0/0/0 W 0.00 567 1367436213 0.0 0.00 0.00 204.244.139.120 army.ca GET / HTTP/1.1
164-7 14930 0/0/0 W 0.00 566 1367435464 0.0 0.00 0.00 147.240.236.9 army.ca GET / HTTP/1.1
165-7 14931 0/1/1 W 0.14 57 0 0.0 0.00 0.00 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
166-7 14934 0/5/5 _ 0.59 16 455 0.0 0.03 0.03 24.76.123.82 army.ca GET /Army.css?rc3 HTTP/1.1
167-7 14936 0/2/2 _ 0.36 10 0 0.0 0.01 0.01 142.166.239.108 army.ca GET /forums/Themes/default/sha1.js HTTP/1.1
168-7 14937 0/2/2 _ 0.46 16 13442 0.0 0.00 0.00 131.137.245.197 army.ca GET / HTTP/1.0
169-7 14943 0/2/2 _ 0.75 14 52911 0.0 0.00 0.00 64.230.24.162 ruxted.ca GET / HTTP/1.1
170-7 14944 0/2/2 _ 0.33 0 1831 0.0 0.00 0.00 131.137.245.199 army.ca GET /inf/rdusag.php HTTP/1.0
171-7 14947 0/2/2 _ 0.27 0 8077 0.0 0.00 0.00 142.166.239.108 army.ca GET /forums/Themes/default/images/powered-php.gif HTTP/1.1
172-7 14948 0/3/3 _ 0.45 13 772 0.0 0.01 0.01 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_back.gif HTTP/1.1
173-7 14951 0/2/2 R 0.71 30 161258 0.0 0.00 0.00 ? ? ..reading..
174-7 14952 1/2/2 C 0.26 13 46 0.4 0.00 0.00 196.201.66.199 forums.army.ca GET /forums/index.php?topic=28632.5;wap2 HTTP/1.1
175-7 14953 0/2/2 _ 0.34 12 303 0.0 0.00 0.00 24.76.123.82 army.ca GET /ads/phpAdsNew/adlog.php?bannerid=27&clientid=2&zoneid=7&so
176-7 14954 0/0/0 W 0.00 551 1367420087 0.0 0.00 0.00 156.34.176.184 army.ca GET / HTTP/1.1
177-7 14955 0/1/1 _ 0.24 8 528641 0.0 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=44741.15 HTTP/1.0
178-7 14956 0/1/1 _ 0.60 0 545264 0.0 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=48873.0;all HTTP/1.0
179-7 14957 0/1/1 _ 0.48 16 529049 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=18519.msg116103 HTTP/1.0
180-7 14958 0/3/3 _ 0.38 13 781 0.0 0.01 0.01 24.76.123.82 army.ca GET /forums/Themes/default/images/titlebg.jpg HTTP/1.1
181-7 14969 0/0/0 W 0.00 541 1367410266 0.0 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=28895.15 HTTP/1.0
182-7 14979 0/6/6 _ 1.05 16 13478 0.0 0.02 0.02 216.209.114.174 army.ca GET / HTTP/1.1
183-7 14980 0/1/1 _ 0.20 16 510100 0.0 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=41498.msg467839 HTTP/1.0
184-7 14991 0/1/1 _ 0.20 16 508965 0.0 0.00 0.00 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
185-7 14992 0/4/4 _ 0.43 0 3553 0.0 0.01 0.01 142.166.239.108 army.ca GET /forums/Themes/default/images/valid-css.gif HTTP/1.1
186-7 14993 1/1/1 C 0.33 11 506099 0.2 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php/topic,43773.msg0.html HTTP/1.0
187-7 14994 0/1/1 _ 0.50 16 509075 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=41498.105 HTTP/1.0
188-7 14995 0/4/4 _ 0.32 16 7934 0.0 0.01 0.01 70.31.163.146 army.ca GET / HTTP/1.1
189-7 14996 0/2/2 _ 6.23 15 409103 0.0 0.01 0.01 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.0/prev_next,next.html HTTP/1.
190-7 14997 0/1/1 _ 0.35 16 507723 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=23622.msg307611 HTTP/1.0
191-7 14998 0/1/1 W 0.25 6 0 0.0 0.00 0.00 ::1 army.ca GET / HTTP/1.0
192-7 15000 0/3/3 _ 0.31 12 2513 0.0 0.05 0.05 24.76.123.82 army.ca GET /img/Army.ca-Header-16.jpg HTTP/1.1
193-7 15001 1/2/2 C 0.22 9 43 0.0 0.00 0.00 142.166.239.108 army.ca GET /ads/images/ad-army.ca-t-shirts-120x90.png HTTP/1.1
194-7 15008 0/1/1 _ 0.70 16 495483 0.0 0.01 0.01 66.160.186.32 forums.army.ca GET /forums/index.php?topic=18425.msg94456 HTTP/1.0
195-7 15009 0/2/2 _ 0.21 0 7217 0.0 0.00 0.00 ::1 army.ca GET / HTTP/1.0
196-7 15013 0/2/2 _ 0.25 0 5034 0.0 0.00 0.00 142.166.239.108 army.ca GET /forums/Themes/default/images/icons/login_sm.gif HTTP/1.1
197-7 15014 0/3/3 C 0.54 4 1147 0.0 0.02 0.02 196.207.40.213 forums.army.ca GET /forums/index.php?topic=51143.0;wap2 HTTP/1.1
198-7 15015 1/2/2 C 0.20 0 8189 0.3 0.00 0.00 ::1 army.ca GET / HTTP/1.0
199-7 15016 0/2/2 _ 0.27 12 339 0.0 0.00 0.00 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_first.gif HTTP/1.1
200-7 15017 0/3/3 W 0.83 29 0 0.0 0.02 0.02 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=re
201-7 15020 0/6/6 C 1.28 1 964 0.0 0.05 0.05 69.11.124.168 forums.army.ca GET / HTTP/1.1
202-7 15026 0/1/1 R 0.14 13 477913 0.0 0.00 0.00 ? ? ..reading..
203-7 15027 0/3/3 _ 0.82 9 197 0.0 0.02 0.02 24.76.123.82 army.ca GET /forums/Themes/default/images/maintab_active_back.gif HTTP/
204-7 15028 0/2/2 W 0.29 29 0 0.0 0.00 0.00 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=ra
205-7 15035 0/2/2 _ 0.26 0 7911 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_active_last.gif HTTP/
206-7 15036 0/4/4 _ 0.51 16 12505 0.0 0.01 0.01 74.119.228.95 forums.army.ca GET /favicon.ico HTTP/1.1
207-7 15043 0/3/3 _ 0.85 16 13741 0.0 0.01 0.01 192.168.1.7 forums.army.ca GET /gallery2/main.php?g2_view=imageblock.External&g2_blocks=ra
208-7 15044 0/0/0 W 0.00 482 1367351364 0.0 0.00 0.00 66.160.186.32 forums.army.ca GET /forums/index.php?topic=43095.0;all HTTP/1.0
209-7 15045 0/3/3 _ 0.76 10 1584 0.0 0.02 0.02 ::1 army.ca GET / HTTP/1.0
210-7 15046 0/3/3 C 0.44 0 66149 0.0 0.01 0.01 65.93.106.147 forums.army.ca GET /forums/index.php?action=pm HTTP/1.1
211-7 15047 0/4/4 _ 0.46 16 202 0.0 0.00 0.00 216.36.138.189 forums.army.ca GET /forums/index.php?topic=19718.15 HTTP/1.1
212-7 15048 1/4/4 C 0.72 12 4001 5.6 0.02 0.02 63.243.150.22 army.ca GET /album/Avatars/thmb_leafcan.png HTTP/1.0
213-7 15049 0/3/3 _ 0.68 15 1887 0.0 0.01 0.01 69.158.65.169 army.ca GET /forums/Themes/default/images/powered-mysql.gif HTTP/1.1
214-7 15050 0/2/2 C 0.22 15 464976 0.0 0.01 0.01 142.46.57.109 forums.army.ca POST /forums/index.php?action=post2;start=0;board=37 HTTP/1.1
215-7 15051 0/2/2 _ 0.25 0 9508 0.0 0.00 0.00 131.137.245.199 army.ca GET / HTTP/1.0
216-7 15052 0/3/3 _ 0.80 12 600 0.0 0.02 0.02 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_last.gif HTTP/1.1
217-7 15059 0/1/1 _ 0.53 16 463806 0.0 0.02 0.02 199.212.250.95 army.ca GET /inf/rcr.php HTTP/1.1
218-7 15060 0/2/2 _ 0.62 8 12 0.0 0.01 0.01 131.137.245.197 army.ca GET //images/blank404.jpg HTTP/1.0
219-7 15061 0/3/3 _ 0.67 15 1866 0.0 0.01 0.01 67.69.18.10 army.ca GET //inf/rwinr.gif HTTP/1.1
220-7 15062 0/2/2 _ 0.25 9 0 0.0 0.00 0.00 142.166.239.108 army.ca GET /forums/Themes/default/images/filter.gif HTTP/1.1
221-7 15064 0/8/8 _ 0.92 15 13195 0.0 0.03 0.03 204.40.1.129 army.ca GET / HTTP/1.1
222-7 15065 0/8/8 _ 1.13 7 1611 0.0 0.01 0.01 202.8.240.182 forums.army.ca GET /forums/index.php?action=keepalive;1162569185796 HTTP/1.1
223-7 15066 0/2/2 _ 0.21 0 7449 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_back.gif HTTP/1.1
224-7 15069 0/1/1 W 0.54 0 0 0.0 0.02 0.02 67.149.76.130 forums.army.ca GET /forums/ HTTP/1.1
225-7 15071 0/4/4 _ 0.46 16 14305 0.0 0.01 0.01 147.240.16.51 army.ca GET / HTTP/1.1
226-7 15073 0/2/2 _ 0.52 10 50 0.0 0.01 0.01 74.6.71.172 forums.army.ca GET /forums/index.php?board=20 HTTP/1.0
227-7 15074 0/2/2 _ 0.17 12 0 0.0 0.00 0.00 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_active_last.gif HTTP/
228-7 15075 0/3/3 C 0.52 4 1963 0.0 0.00 0.00 84.176.49.135 forums.army.ca GET /forums/index.php?topic=41652.msg459093 HTTP/1.1
229-7 15076 0/3/3 _ 0.29 10 688 0.0 0.00 0.00 ::1 army.ca GET / HTTP/1.0
230-7 15081 0/3/3 _ 0.65 16 41042 0.0 0.01 0.01 204.40.1.129 army.ca GET / HTTP/1.1
231-7 15082 0/2/2 _ 0.31 12 107 0.0 0.00 0.00 131.137.245.199 army.ca GET /inf/rdusag.php HTTP/1.0
232-7 15083 0/0/0 W 0.00 470 1367339129 0.0 0.00 0.00 216.209.114.174 forums.army.ca GET /forums/index.php/board,8.60.html HTTP/1.1
233-7 15084 0/2/2 _ 0.28 8 1252 0.0 0.00 0.00 131.137.245.199 forums.army.ca GET /forums/index.php?action=unread HTTP/1.0
234-7 15085 0/2/2 _ 0.45 16 184 0.0 0.01 0.01 66.160.186.36 forums.army.ca GET /forums/index.php?topic=51927.0;all HTTP/1.0
235-7 15086 0/4/4 _ 0.81 16 312 0.0 0.00 0.00 74.6.73.6 army.ca GET /ads/phpAdsNew/adimage.php?filename=ad-army.ca-chat-120x90.
236-7 15087 0/2/2 _ 0.21 10 49 0.0 0.01 0.01 131.137.245.198 forums.army.ca GET /forums/index.php?topic=25680.15 HTTP/1.0
237-7 15088 0/1/1 W 0.01 467 0 0.0 0.01 0.01 216.209.114.174 forums.army.ca GET /forums/index.php/board,8.80.html HTTP/1.1
238-7 15089 0/6/6 _ 0.78 8 800 0.0 0.01 0.01 74.99.189.226 forums.army.ca GET /forums/index.php?action=keepalive;1162569106510 HTTP/1.1
239-7 15090 0/3/3 _ 0.33 16 860 0.0 0.01 0.01 63.243.150.22 forums.army.ca GET /forums/index.php/topic,51916.90.html HTTP/1.0
240-7 15091 0/2/2 _ 0.23 12 5076 0.0 0.02 0.02 63.243.150.22 army.ca GET /album/Avatars/thmb_leafcan.png HTTP/1.0
241-7 15092 0/0/0 W 0.00 469 1367338303 0.0 0.00 0.00 131.137.245.197 army.ca GET / HTTP/1.0
242-7 15093 0/2/2 _ 0.18 14 3035 0.0 0.00 0.00 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_active_first.gif HTTP
243-7 15094 0/2/2 W 0.19 335 0 0.0 0.00 0.00 131.137.245.199 forums.army.ca POST /forums/index.php?action=gallery HTTP/1.0
244-7 15096 0/2/2 R 0.23 16 262027 0.0 0.00 0.00 ? ? ..reading..
245-7 15100 0/2/2 _ 0.30 0 7784 0.0 0.01 0.01 209.115.236.89 army.ca GET /favicon.ico HTTP/1.1
246-7 15101 0/2/2 _ 0.20 8 0 0.0 0.00 0.00 142.166.239.108 forums.army.ca GET /forums/Themes/default/images/maintab_active_back.gif HTTP/
247-7 15102 1/2/2 C 0.71 9 0 0.3 0.01 0.01 ::1 army.ca GET / HTTP/1.0
248-7 15103 0/2/2 W 0.23 0 0 0.0 0.02 0.02 24.64.223.204 forums.army.ca GET /forums/index.php?action=keepalive;1162569206254 HTTP/1.1
249-7 15104 0/1/1 R 0.38 12 449654 0.0 0.00 0.00 ? ? ..reading..
250-7 15105 0/15/15 W 1.43 65 0 0.0 0.00 0.00 64.26.171.13 army.ca GET /deleteme/status.php HTTP/1.1
251-7 15106 0/10/10 _ 0.90 14 1956 0.0 0.01 0.01 142.166.239.108 army.ca GET /forums/Themes/default/print.css?rc3 HTTP/1.1
252-7 15107 0/0/0 W 0.00 462 1367330960 0.0 0.00 0.00 156.34.176.184 army.ca GET / HTTP/1.1
253-7 15114 0/3/3 _ 0.41 8 1783 0.0 0.02 0.02 216.209.114.174 army.ca GET / HTTP/1.1
254-7 15115 0/0/0 W 0.00 462 1367330895 0.0 0.00 0.00 131.137.245.199 army.ca GET / HTTP/1.0
255-7 15116 0/2/2 _ 0.23 8 1149 0.0 0.00 0.00 64.230.35.66 forums.army.ca GET /forums/Themes/default/images/maintab_first.gif HTTP/1.1
Srv Child Server number - generation
PID OS process ID
Acc Number of accesses this connection / this child / this slot
M Mode of operation
CPU CPU usage, number of seconds
SS Seconds since beginning of most recent request
Req Milliseconds required to process most recent request
Conn Kilobytes transferred this connection
Child Megabytes transferred this child
Slot Total megabytes transferred this slot
SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 1
sub-caches: 32, indexes per sub-cache: 133
time left on oldest entries' SSL sessions: avg: 290 seconds, (range: 290...290)
index usage: 0%, cache usage: 0%
total sessions stored since starting: 9
total sessions expired since starting: 8
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 80 hit, 24 miss
total removes since starting: 0 hit, 0 miss
Apache Server at army.ca Port 443

Christian A. Herrnboeck

How many members did you have on when you took that?


Regards,
Christian


Farmers:Producing food for the world!

Mike Bobbitt

I don't know for sure... the server was pretty hosed... but my daily max for that day was 386, and that's not a whole lot.

Christian A. Herrnboeck

What exactly does mimedefang.pl do? Perl scripts are notorious for running up CPU load...  Other than that, I'd pretty much need access to the box to see what it's doing, real-time.

Sometimes, what happens is that Apache isn't built right, and causes intermittent high loads, but it's pretty hard to diagnose that off-the-bat.

-Christian


Farmers:Producing food for the world!

Mike Bobbitt

mimedefang is a mail filter. It ties spamassassin and a bunch of other tools together, and it definitely does consume high CPU/memory at times.

I think I'm going to shoot for an upgrade to FC6 to see if that changes things at all. I hate to do something like that "just in case" it fixes a problem but I was planning to do the upgrade anyway.

Christian A. Herrnboeck

Before you do that, try recompiling Apache from source. Packages are usually not good to use in server environments anyway...

-Christian


Farmers:Producing food for the world!

Mike Bobbitt

Hi all,

This problem has been back with a vengeance, and my site has been effectively unavailable for the last couple of weeks. Where the problem used to subside, now it never does.

It's definitely httpd taking up CPU and definitely "load based". When I kill the outside connection, the server load returns to normal. In fact, when I completely take my SMF forums offline, the load goes down too. (But if I just put the forums in maintenance mode, it doesn't help.)

Is anyone running a mid-large sized board with RPMs? I really don't want to get into compiling if there is some other way. I'm open to ideas, but I'm off to download the httpd source as we speak (and install compilers, etc... ugh.)


Thanks
Mike

Mike Bobbitt

As a side note... if I could track down a specific feature/script that was the "biggest contributor" I'd be happy to tweak that, at least temporarily. But all I see in top is "httpd" threads as the highest contributor... is there a way to get more detail? Any sort of PHP performance monitor would be ideal.

Christian A. Herrnboeck

Hi Mike,


Are you running a control panel of sorts? Which versions of Apache/PHP are you using right now?

Regards,
Christian A. Herrnboeck


Farmers:Producing food for the world!

Mike Bobbitt

Thanks, I found the immediate problem... the versions of my PHP and APC accelerators didn't match up so APC was failing to initialize and I didn't know it.

Advertisement: