Log of server problem

[jesse]

At about 8:57 PM CST the POA website quit responding. I started pinging the server and received consistent responses throughout the event.

Pinging pilotsofamerica.com [216.69.169.202] with 32 bytes of data:

Reply from 216.69.169.202: bytes=32 time=98ms TTL=40
Reply from 216.69.169.202: bytes=32 time=100ms TTL=40
Reply from 216.69.169.202: bytes=32 time=97ms TTL=40
Reply from 216.69.169.202: bytes=32 time=99ms TTL=40

That is about an average response time from my computer to the POA server. Based on this, I don't think this is a network issue.

I poked around a little more and at about 9:00 PM CST I had a decent test going. I was trying to download the main page from the POA server with wget. After 10 seconds it would timeout and try again. As you can tell the web server did not respond until 9:01:34. As soon as I was able to download a page the problem disappeared. Here is the test:

[root@server1 jangell]# date
Thu Oct  4 21:00:20 CDT 2007
[root@server1 jangell]# wget pilotsofamerica.com --timeout=10 --tries=0
--21:00:22--  http://pilotsofamerica.com/
Resolving pilotsofamerica.com... 216.69.169.202
Connecting to pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.pilotsofamerica.com/ [following]
--21:00:22--  http://www.pilotsofamerica.com/
Resolving www.pilotsofamerica.com... 216.69.169.202
Reusing existing connection to pilotsofamerica.com:80.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--21:00:33--  (try: 2)  http://www.pilotsofamerica.com/
Connecting to www.pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--21:00:45--  (try: 3)  http://www.pilotsofamerica.com/
Connecting to www.pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--21:00:58--  (try: 4)  http://www.pilotsofamerica.com/
Connecting to www.pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--21:01:12--  (try: 5)  http://www.pilotsofamerica.com/
Connecting to www.pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--21:01:27--  (try: 6)  http://www.pilotsofamerica.com/
Connecting to www.pilotsofamerica.com|216.69.169.202|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

    [  <=>                                                                                                                ] 70,792       258K/s   in 0.3s

21:01:34 (258 KB/s) - `index.html' saved [70792]

[root@server1 jangell]#

Based on this it's pretty obvious that whatever the problem is it is completely maxing out the resources of the server. The instant that the resource max out stops--is the instant POA responds. There isn't much in between--it pretty much either works or doesn't.

I really think it's a waste of time to sit here trying to optimize the database constantly. This would make sense if the problem wasn't at the exact same time every day. This isn't an issue of someone on POA generating too much load. This is an issue of another user on the physical server generating too much load.

I'm available to assist with resolving the issue and tools could be put in place that would allow us to graph when these problems are occurring in relation to other server resource specifications. But his is kind of a waste of effort. POA needs a new home.

 

[TangoWhiskey]

Does your root account allow you to run a process list (ps) to see what's running at that time? I'm surprised the hosting provider's support channel hasn't been able to give any indication of what's going on (have we asked them to investigate?).

Thanks for the work you all are doing!!

 

[jesse]

Does your root account allow you to run a process list (ps) to see what's running at that time? I'm surprised the hosting provider's support channel hasn't been able to give any indication of what's going on (have we asked them to investigate?).

Thanks for the work you all are doing!!

The above--is a server of mine. Not the POA server.

Looking at the current running processes is not going to do any good as you are only going to see the processes in your virtual machine. Let me explain how POA is setup:

There is a physical server with a software package called [SIZE=-1]Virtuozzo which is put out by a company named SWsoft. It allows you to take one physical server and resell multiple "virtual servers" on it. These virtual servers appear in every way to be their own physical server. If you rent one of these virtual servers you have full root access to the VIRTUAL server. You cannot go outside of your virtual server and quite frankly you can't even tell the difference.

In theory--things are in place to limit one virtual server from eating up all the physical server's resources and bringing the other virtual servers to their knees. But these things aren't perfect. So most likely what is happening is another virtual server on this physical server is eating up a crap ton of resources for a short period of time.

Could you sit there and argue with the host back and forth? Probably. But you get what you pay for--and I think POA pays about $30 per month. We are getting $30 per month of value.

There comes a point where you need to step back and decide what you want out of a server and then accept what you will pay to get that. This is why I maintain my own dedicated server for my own projects. Do I pay more then $30 per month? Yes. Do I have these problems? No.

Clearly the users of this site would like this problem to be fixed. But this is Chuck's site and it will be handled on his timeframe which I feel is fair.

There are several of us in the POA community that have the knowledge and the resources to fix this problem if given the opportunity.
[/SIZE]