PoA Reliability

ArrowFlyer86

ArrowFlyer86

En-Route
PoA Supporter
Joined
Jul 17, 2019
Messages
3,285
Location
Around
Display Name
Flight
I know we have a thread on slowness but this is different. These days I'm finding PoA servers are generally unreliable. Sometimes my connection will time out or be unable to load, and unfortunately that's become pretty normal.

Idk what's happening but the impact on usability is extreme. I'd say a solid 25% of times I try to visit the site it just doesn't load, so I have to just try again later.

I've experienced this across all my devices and on and off WiFi, so I don't think it's idiosyncratic. I'm wondering if other people are having reliability problems? Not just an issue where it takes a second or two longer to load.
 
ArrowFlyer86 said:
I know we have a thread on slowness but this is different. These days I'm finding PoA servers are generally unreliable. Sometimes my connection will time out or be unable to load, and unfortunately that's become pretty normal.

Idk what's happening but the impact on usability is extreme. I'd say a solid 25% of times I try to visit the site it just doesn't load, so I have to just try again later.

I've experienced this across all my devices and on and off WiFi, so I don't think it's idiosyncratic. I'm wondering if other people are having reliability problems? Not just an issue where it takes a second or two longer to load.
Click to expand...
It's the same problem. Lately the bots have been attacking so hard that the connection just times out.
 
Is there such a thing as antibots?
 
172andyou said:
Presumably PoA management just has a hard time balancing the needs of their families and providing free labor to this site.
Click to expand...

Families? Families??!! You mean we allow the MC to reproduce?! Gad. I thought that was against the ROC. Next we'll let them moonlight outside POA, maybe even get real jobs, forcing us to hire bots to administer the forum.
 
Half Fast said:
Families? Families??!! You mean we allow the MC to reproduce?! Gad. I thought that was against the ROC. Next we'll let them moonlight outside POA, maybe even get real jobs, forcing us to hire bots to administer the forum.
Click to expand...

AI can do it, can’t they? After all, AI can do anything

edit to clarify: I'm being sarcastic
 
Last edited:
I've been active on one forum or another since the very early 2000s and these things have always been part of the experience. Sometimes the fix is a software upgrade, sometimes it's a server capacity thing, sometimes the database has just gone wonky and needs tweaking.

I did have some problems this morning, it wasn't working at all. Now it seems ok... most of the time it's working ok an usually when it's not it's just a little slow or comes back up within a few hours. Annoying when you really want to check something/post something/etc I know but it's working well more often than not, come back later in the day or the next day and it's usually good.
 
Not sure about 25% of the time. but it does happen more often than it used to.
This is unfortunately a whack a mole kind of thing.

Tim
 
Random thought - can the forum software be configured to add a small delay (a second? just guessing) to all requests from a guest user? Wouldn't really impact a human user, but would definitely slow down the bots.
 
AV8R_87 said:
Random thought - can the forum software be configured to add a small delay (a second? just guessing) to all requests from a guest user? Wouldn't really impact a human user, but would definitely slow down the bots.
Click to expand...
That would likely make it worse; the connection is still there, occupying a slot (that cannot be used by a real user) for even longer. We're not CPU- or IO- or network- or db-bound; it's literally a DDoS by virtue of 10s-100s of clients per CIDR, multiplied across multiple actors and multiple CIDRs per actor.

Something like anubis might help.

previously:
www.pilotsofamerica.com/community/threads/poa-problems.154332
 
POA seems to be inflicted much more than other web forums I frequent. Why the difference?
 
GeorgeC said:
That would likely make it worse; the connection is still there, occupying a slot (that cannot be used by a real user) for even longer. We're not CPU- or IO- or network- or db-bound; it's literally a DDoS by virtue of 10s-100s of clients per CIDR, multiplied across multiple actors and multiple CIDRs per actor.

Something like anubis might help.

previously:
www.pilotsofamerica.com/community/threads/poa-problems.154332
Click to expand...

Well I tried to reply, but I got a timeout :D
I hope you implement Anubis or something else soon.

Tim
 
Using Apache is your first mistake. Resource hog.
 
Half Fast said:
POA seems to be inflicted much more than other web forums I frequent. Why the difference?
Click to expand...
Perhaps it's because POA is a bigger target.

But every aviation web site I regularly visit (4 of them), all but one have gone through worse and have seemingly gotten it under control. At least for a little while. One of them (the smallest one) completely replaced their server software.

The problems on POA are not as bad as some others have had it Among others, I'm thinking BeechTalk, which has been completely shut down the most of a few days before getting it "mostly" under control But as web siet defenses improve, so do the attacker's methods.
 
As I said before, this is a game of whack a mole. As long as it is volunteers who have a other things in life, expect it to be reactive not proactive.

If you want proactive, the best option is companies (I am not aware of any non-profit organization) that have a broad customer base allowing them to play whack a mole across a much larger footprint. They therefore tend to be much more effective from the individual customer viewpoint; and much faster at responding. No idea on current pricing, but a few years ago it was not cheap.

Tim
 
JOhnH said:
Perhaps it's because POA is a bigger target.
Click to expand...

well, a different forum-that-shall-not-be-named was experiencing about the same number of guests as POA is, regularly seeing 4000+ "guests"
 
It all boils down to you get what you pay for. How much did you pay?
 
GeorgeC said:
Are you volunteering to port to nginx or whatever quantum serverless lambda ai blockchain is fashionable these days? :D
Click to expand...
Given you're talking about "slots", I assume this is still running using the "forking" model of Apache. Nginx's event-driven asynchronous model was released in 2004 and was very main-stream by 2009.....17 years ago! Hell, Apache introduced a threading model in 2012.

Of course, I could have moved the damn thing away from the forked model, given I was running it in those years. But it wasn't a priority as the site wasn't unusable during large portions of the day.
 
GeorgeC said:
Are you volunteering to port to nginx or whatever quantum serverless lambda ai blockchain is fashionable these days? :D
Click to expand...
I offered to help a couple years ago and was told my help wasn't needed...
 
" Total: 10,685 (members: 22, guests: 10,663) "

over 10,000 guests?

wow
 
I cranked up the adjustment on our antibot for now. We may have to go from doing this solely within XF to just shutting down connections from blacklisted sites.
 
Ironically, I read this whole thread with no issues, then the site hung when I left it.
 
Yeah, this sucks wind out of the sails.
 
One (of many) fixes would be to switch to a different (better) provider with a big pipe that is good at filters out a lot of the attacks before they even hit the PoA servers.

Not a total fix, but a start.
 
M20K said:
One (of many) fixes would be to switch to a different (better) provider with a big pipe that is good at filters out a lot of the attacks before they even hit the PoA servers.

Not a total fix, but a start.
Click to expand...
Bigger than AWS?
 
schmookeeg said:
Bigger than AWS?
Click to expand...
I didn't look, so didn't realize they were an AWS client. But the issue is still how much bandwidth PoA has (unlikely they have the full AWS pipe at their disposal :D) and who is monitoring and providing the security?

Are they doing it themselves or did they sign up for AWS Shield? If they are doing it themselves, then being hosted at AWS doesn't mean anything.
 
I've hosted a vehicle specific site online since 2006 and my solution to bots was to enable anti-spam features at login. Knowledge based captcha's and moderating the first 5 posts for content. These were added as the bots were found, so existing accounts were 'grandfathered' in.

There are many not-labor intensive things admins can do to alleviate the workload, one is pruning members with 0 posts. Most of those accounts are bots anyway. You sign up to post, if not, buh bye. PoA dosn't require a login to view attachments so an account with no activity is a bot in waiting.

Next is server side. My guess is that AWS sells capacity, not efficiency. Building a site from bare metal and making all the tweaks can find horsepower that you never knew you had. I bet this site could move to a modest virtual machine and out of AWS, save a ton of money and increase capacity... but that takes some work.. and that might not be something the owners want to do. All those web interfaces take CPU and disk/RAM. They make things easy, but not the most efficient.

Hosting a free site is both really interesting and really tiring. I understand the struggle but the powers that be need to make choices. If PoA falls, there will be another. The circle of life and whatnot

Doing a QUICK and dirty search:

Summary Cost Range​

  • Low-End Estimate (~$250 - $300/mo): Achieved if the site is highly optimized, utilizes aggressive caching so the database rarely sweats, and takes advantage of 1-year or 3-year AWS Reserved Instance commitments.
  • High-End Estimate (~$450 - $520/mo): Occurs if they are paying standard On-Demand prices, run multi-Availability Zone backups for maximum uptime, or use heavy internal search operations via an attached Elasticsearch/OpenSearch instance.

DigitalOcean: You could deploy a 4 vCPU / 16GB RAM "Droplet" for the XenForo web application ($96/mo) and pair it with a Managed MySQL Database ($60/mo). With 7 TB of combined bandwidth included for free, your total cost would be roughly $160/month.

Hetzner: If the site doesn't strictly require US-west/east low latency (though they have data centers in Virginia and Oregon now), Hetzner offers unmatched value. A dedicated AMD EPYC server with 4 vCPUs and 16GB RAM costs around $30–$40/mo. Even with managed databases or a self-hosted DB setup, the total monthly bill would be under $80/month.

If cost is the issue (aka they don't/can't migrate) then there are way cheaper options that AWS.
 
Last edited:
The site is hosted as an EC instance. I doubt increasing the pipe is going to help.
The bots are NOT logged in. This site is wide open for browsing by those without logging in. I suppose we could nail it down (some of the other sites have done this), but that would decrease a lot of the functionality.
We have been running a plugin that detects rapid fire searching or our files and blacklists the perpetrators, but trying to adjust the parameters so it doesn't stomp on regular users, can be problematic. We did have one user blacklisted already when I was turning things up.
 
What is the "site functionality" benefit of having 10,000 bots scrape PoA content exactly? This seems one captcha away from lasting improvement..
 
schmookeeg said:
What is the "site functionality" benefit of having 10,000 bots scrape PoA content exactly? This seems one captcha away from lasting improvement..
Click to expand...
poa has a ton of valuable content that is linked to from various search engines. many of the current day members found PoA, because, of its content being public. So, one doesn't want to hold back the major search engines.

That said...much of this bot activity these days is driven by AI training. No harm in blocking that, because its never going to bring members. It'll just steal them.
 
You must log in or register to reply here.
Back
Top