So nice to know that small websites are being taken down by AI crawlers burning the equivalent of an entire state worth of energy in pursuit of...what, exactly? More shitty useless LLMs? Jesus fuck, just give it up already. Your AI BS is nothing but waste.
this post was submitted on 21 Mar 2025
10 points (100.0% liked)
Videos
15412 readers
237 users here now
For sharing interesting videos from around the Web!
Rules
- Videos only
- Follow the global Mastodon.World rules and the Lemmy.World TOS while posting and commenting.
- Don't be a jerk
- No advertising
- No political videos, post those to [email protected] instead.
- Avoid clickbait titles. (Tip: Use dearrow)
- Link directly to the video source and not for example an embedded video in an article or tracked sharing link.
- Duplicate posts may be removed
Note: bans may apply to both [email protected] and [email protected]
founded 2 years ago
MODERATORS
This has happened to me personally. I have a very small git server sitting on a very small instance. I noticed my traffic was spiking for the past few days and thought nothing of it.
After I looked at the logs, it was 6 ip addresses. 1 Amazon LLM search and open AI with the other 5. I added them t the fail2ban and thought nothing of it. About 10 min later, openAI had a multitude of new IPs hitting the server again, specifically the git repos. I again looked up to confirm that, yes it totally was OpenAI.
I had to create an AI blackhole with python and create new rules just to stop the LLM madness. It worked but im tempted to put a very small capta like give me 2+2 on a form in order to see my code.
Worst part, I had a robot.txt that totally blocked the indexing of my sites. But they just didnt care.
This is so fricking scummy. Hope that legislature somehow get a hold on them and this strategy and they have to pay for the damage.
Its making the open web hard to keep up for small entities thats for sure.
all according to plan.