What's up? (or down)?

J

JOhnH

Touchdown! Greaser!
Joined
May 20, 2009
Messages
14,217
Location
Florida
Display Name
Display name:
Right Seater
Last night POA appeared down or hung to the point my browser kept timing out. I figured someone would reboot it soon and all would be better tomorrow.

This morning it was still molasses. I thought perhaps I had been banned and nobody told me about it.

It took me several minutes just to bring up this feedback forum to report it, but suddenly, all seemed well. For a few minutes. Now it is hung again. I am typing this on a word processor and will try to post on POA if I ever get my prompt back.
 
It's still wonky. At times it works ok for a few minutes,
RyanB said:
@flyingron

I noticed it late last night. Seems back to normal now. Is it still running slow
Click to expand...
It's still wonky. Sometimes it works for a few minutes, then something happens, like this:
An unexpected database error occurred. Please try again later.
Click to expand...
Everything I typed in before disappeared then I got the above message.

It seems to be ok at this minute, but . . .

My guess is either a corrupted database, or possibly a DoS attack.
 
RyanB said:
@flyingron

I noticed it late last night. Seems back to normal now. Is it still running slow on your end?
Click to expand...
Groundpounder said:
Funny, two different threads about this yet nary a peep from management
Click to expand...
It seems to be working fine now on my end. That said, we are working to determine the source, not much else to report right now, but we’re aware of the troubles.
 
as of 1:44 EST, still wonky.
 
Ugh. It's like trying to download a picture over dialup in the 90's, but without the payoff :biggrin:
 
Still slow, sometimes get a message saying “database error.” Probably caused by the FAA......
 
Yeah. 4pm central and it works ok now, but didn’t about 5 min ago.

Edit: Never mind, there’s still something goofy going on.
 
7:02AM and so far, so good.
I wonder if somebody fixed something or did the problem just go away.
(or did I just jinx the site?)
 
 
The problem here was like an onion. It was smelly and made me cry a lot. Actually, I mean I kept peeling away layers.

The first obvious issue was the root file system got filled mostly due to logfiles betting really big. Nothing really bad in the log files, but we do log every page downloaded and we were getting lots of load. Ok, clean that up.

Second was that our database server was running out of the maximum number of simultaneous connections. Changed that configuration.

Then things started to pick up and the disk that the images/attachments was staged on suddenly rocketed to fullness (this has been an intermittant problem as it contains cached scaled things that do go up and down periodically). Anyhow, increased the size of that drive substantially.

Then the system was just way heavily loaded. I just think the site is getting more popular. I wasn't seeing any obnoxious access (bots or some sort of attack), just lots of people using the site. I increased the system size (we use Amazon Web Services, so we can just ask for more resources).

Of course, some of these changes required me to reboot the machine a few times, so if you were trying to use it between 10PM and Midnight last night, you might have observed times when it was just not up at all.

Anyhow, as of this morning, it seems better. We'll see when evening comes tonight (when we have our peak load) if we hold up.
 
flyingron said:
The problem here was like an onion. It was smelly and made me cry a lot. Actually, I mean I kept peeling away layers.

The first obvious issue was the root file system got filled mostly due to logfiles betting really big. Nothing really bad in the log files, but we do log every page downloaded and we were getting lots of load. Ok, clean that up.

Second was that our database server was running out of the maximum number of simultaneous connections. Changed that configuration.

Then things started to pick up and the disk that the images/attachments was staged on suddenly rocketed to fullness (this has been an intermittant problem as it contains cached scaled things that do go up and down periodically). Anyhow, increased the size of that drive substantially.

Then the system was just way heavily loaded. I just think the site is getting more popular. I wasn't seeing any obnoxious access (bots or some sort of attack), just lots of people using the site. I increased the system size (we use Amazon Web Services, so we can just ask for more resources).

Of course, some of these changes required me to reboot the machine a few times, so if you were trying to use it between 10PM and Midnight last night, you might have observed times when it was just not up at all.

Anyhow, as of this morning, it seems better. We'll see when evening comes tonight (when we have our peak load) if we hold up.
Click to expand...
Thanks Ron,
Hopefully you will be smiling instead of crying tonight.
 
Uh Oh.
Methinks there may be more layers to the onion to unpeel. Things are getting wonky again. Not as bad as yesterday, but obviously, something is wrong.
 
You must log in or register to reply here.
Back
Top