Forum Outages
Wuerfel_21
Posts: 5,687
The forums seem to be experiencing some major downtime lately!
Anyone have an idea what's going on?
Wuerfel_21
Posts: 5,687
The forums seem to be experiencing some major downtime lately!
Anyone have an idea what's going on?
Comments
I fear some people might give up on Parallax if this carries on.
Seems like some security settings in AWS that didn't agree with themselves after upgrading themselves.
Jeff is working on it with some AWS engineers. I'd guess it's related to some legacy settings that no one knows about and no one seems to be able to find, but they no doubt will soon
Could also be partially sketchy traffic related; For example, we had >18000 hits on the OBEX server last Thursday afternoon which wasn't ideal and I had to add some "stuff" to deal with that and keep it responsive. I suspect the learn and forum sites are seeing something similar. The traffic I can see on OBEX has been mental the last week, - someone or something (AI?) is downloading lots of stuff over and over. Though they are throttled to a calm Sunday afternoon pace now
I don't know if this is related, but it was just reposted by a former Parallax employee who is now working in a critical role at one of our country's security organizations.
Thought they might be upgrading the forum software... Guess not. Suppose that's good news.
Was a good test to see if I'm addicted to P2 forum or not...
Btw: was wondering about emergency alternatives …
Not completely sure how Discord works, but is there a P2 discord group?
Seem to recall something about that…
Yes, it's here: https://discord.com/invite/9TkW8jD
No one knows how Discord works, they keep changing it. Now they're trying to sell you ＴＨＥ ＯＲＢ or whatever. But alas, they killed off casual IRC, so it's what'll need to do for now.
Ok, this is getting ridiculous now...
I agree
When I was in CA Jeff told me that Parallax servers are constantly barraged with foreign DOS attacks.
And now AI crawlers as well, I bet. Those things grab everything they can get.
A lot of sites are using this software now to block scraping bots: https://anubis.techaro.lol
Downside is that javascript is required. It's apparently very difficult to properly block those guys otherwise, allegedly some of them proxy through some kind of botnet, so the requests come from random people's home IPs. And of course none of them read robots.txt or anything like that.
on the bright side.... maybe the AI will get better at Spin code syntax
Plural! There'll be hundreds if not thousands of them. And each one is employing tactics of deception, exploits, and DDOS to break into website databases. The sort of things "hackers" are sent to jail for doing even once.
LOL, oh man, I totally forgot I posted here ... So the latest round of forum outages is probably fallout from AWS falling over the other day then?
EDIT: Just did some searches and found that started early last Monday (GMT-5), more than a week ago. That's a lot of fallout!
@evanh and all - thank you for your continued support!
We too initially thought the major AWS issue had affected the forums since it happened at the same time, but that was reportedly only for an AWS region we don't use. What has affected the forums was transient and mysterious, all year, and was progressively getting worse. There was a legacy mis-configuration that we found and fixed back in June I think, and things were better until recently. (Got something connected to the Internet? It WILL BE attacked- sometimes relentlessly)
I now found that the incoming traffic followed an unnatural pattern - bursts of activity at regular intervals that would occasionally change in period; looks a bit like a square wave / pulse train when viewed at the right zoom level. Check out the most recent traffic request pattern leading up the vertical line at about 9:30 pm last night.
This is somewhat typical of search engine bots, so I looked past it at first, but then noticed I could only get the forum running successfully again if I just happened to restart it during the low points (and it'd run for hours sometimes, and more recently only minutes), and eventually never again at all (most of yesterday).
Cool pattern, huh? The gap after that pattern, and before 9:30-ish is where I'd completely shut off all traffic while I addressed the offenders and then opened it back up.
The conclusion here (I hope) is that I found there was an abusive system, or systems, making ridiculous requests of our forum server that eventually caused it to spin its wheels. I found the abuse and put a stop to it.
Now hopefully we can all get our forum "fix" again.
Just one offender? An LLM scraping bot maybe? They're known to be written to ignore etiquette then get tangled in links.
Well, regardless of the cause, I am glad I can connect now. I was starting to have withdrawals. LOL
All I can say about the outages is WTF. I noticed the parallax site never went down. In a trickle down way the forum generates revenue by keeping people interested in the products
Yeah... "You never miss it till it's gone away"
A bit offline-time can be a good thing is I don't waste so much time browsing the forum. But sometimes you're really lost if left alone. I have local copies of all the important documents but there's so much knowledge about the P2 that is only available in this forum. And asking some real person is often much more effective than searching code examples to find out how some special feature works.
I'm glad its back! As a noob there's so many internet searches about the P2 that end up being links to this forum!
Thanks for the effort to bring it back!
Though for a second about making my own forum. Then, started to realize that is a lot of thankless work!
Indeed !
Well, THANK YOU!
I almost thought of emailing you Rayman! I was thinking of seeing if there was a way to download an image of the forum from the Wayback Machine.