TheDude

Great question and I’m happy to provide some clarity.

When the server comes under significant load from this type of coordinated scraping activity, it becomes extremely difficult to investigate the issue while it is happening. Even with out-of-band access, the server can become so overloaded that it is nearly unresponsive.

To relieve some of that pressure, I temporarily enable Cloudflare’s Under Attack Mode. This helps reduce the amount of traffic reaching the server, but it also interfere with certain third-party applications, such as Voyager and others.

I enable Under Attack Mode to reduce the load and collect access logs showing what the filtered traffic looks like. I then disable it temporarily to capture additional traffic, including requests that were previously being blocked. This is why you may intermittently regain access during these incidents.

By comparing these two datasets, I can more quickly distinguish legitimate traffic from malicious or unwanted traffic patterns. From there, I can adjust the security rules to make it more difficult and costly for that bad traffic to reach the server in the first place.

Sorry if that got a little technical.

We are experiencing some kind of bot attack at the moment. Voyager and other mobile apps are unavailable while we attempt mitigate this. Only the web-ui is available. We’ll gave more to share soon. Thanks for your patience

Hey all,

We identified bot abuse on this endpoint and temporarily took it down until I can implement some bot restriction on it. I’ll bring it back online shortly.

Hey @westingham

Thanks for starting a post about this.

Our traffic today jumped 4x our previous daily high. I had to enable some extra protection to give the server some breathing room to recover. One of the side effects is the mobile apps are temporarily unavailable until I get everything tweaked. It will be back in a few days. The good news is that I already got my hands on a newer more powerful server that will be going into production in the next few weeks.

For the next few days if you want to scroll the site you will need to use your mobile browser. I understand its totally not the same experience as voyager but I promise you i’ll get everything back up as it was as soon as I can.

Thanks for your trust and patience!

Hey! these screenshots are taken from the cloudflare panel

The Tesseract frontend has been pulled.

For those interested you can see the announcement here: sh.itjust.works/post/64232628

alright, turns out it had nothing to do with cloudflare and more related to my nginx settings. I made some changes and it appears to have fixed the issue. Let me know if this has fixed it on your side as well.

Thank you

I believe this might be related to the fact that we are behind cloudflare. I notice other instances that are behind cloudflare appear to have the same issues while those that are not behind cloudflare appear to be ok. Lemmy.world appears to strip the embedded video function completely. See lemmy.world/post/49302693 as an example

I also noticed someone did open a github issue on the lemmy-ui repo github.com/LemmyNet/lemmy-ui/issues/4001

I’ll take a look and see if there’s something in the nginx settings that could fix this

Hey all, most of today’s outage was caused by me. I implemented Anubis in attempt to differ some bot/scrapping traffic. We are presently serving over 350 million requests per month and its starting to show in the stability in the instance. I will be switching to the horizontal scaling approach in hopes to stabilize things more.

I’ll also be providing an long overdue update once I get everything back under control.

Sorry for the stability issues as of late, you all deserve better.

@dbtng@eviltoast.org I made you mod on the community. Like others have mentioned its recommended that you have an account on the instance of the community but not a requirement.

Thank you for volunteering your time to the fediverse

Hey, we did an upgrade earlier today. All tests were passed but its possible something was missed.

Could you tell us which app you are using?