1079
AI companies are violating a basic social contract of the web and and ignoring robots.txt
(www.theverge.com)
this post was submitted on 14 Feb 2024
1079 points (100.0% liked)
Technology
68864 readers
4637 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
We need laws mandating respect of
robots.txt. This is what happens when you don’t codify stuffAI companies will probably get a free pass to ignore robots.txt even if it were enforced by law. That's what they're trying to do with copyright and it looks likely that they'll get away with it.
Turning that into a law is ridiculous - you really can’t consider that more than advisory unless you enforce it with technical means. For example, maybe put it behind a login or captcha if you want only humans to see it
Are you aware of what "unlisted" means?
Yes, and there’s also no law against calling an unlisted phone number
Also we already had this battle with robots.txt. In the beginning, search engines wouldn’t honor it either because they wanted the competitive advantage of more info, and websites trusted it too much and tried to wall off too much info that way.
There were complaints, bad pr, lawsuits, call for a law
It’s no longer the Wild West:
There's also no law against visiting an unlisted webpage? What?
you can't really make laws in the united states it's too hard
The battle cry of conservatives everywhere: It's too hard!
Except if it involves oppressing minorities and women. Then it's a moral imperative worth all the time and money you can shovel at it regardless of whether the desired outcome is realistic or not.
Seriously, could the party of "small government" get out of my business, please?
Sure as long as the party of law and order respects law. And order.
I just wish the push and pull of politics didn't have to be played as a zero sum game. I wish someone could take the initiative and just...
I think both parties in America sing pretty loud about "law and order." I haven't heard that cry particularly loudly from either side over the other. I don't think I've heard anyone who claims to be a Democrat saying the end goal is "small government" but I have heard it from Republican voices.
Honestly, I would really prefer if we were in a system that enabled more parties, so we didn't have "parties" that did such contradictory things as the current ones...
The GOP has historically been the party of law and order. Hence why they implied that blue lives matter more than black lives.
thatsthejoke.png
Just like how one party impeached a president of the other for obstruction and abuse of power, and the other impeached a president for checks notes lying about a blowjob.
Sounds like the type of thing that would either be unenforceable or profitable to violate compared to the fines.
Why? What would you like to achieve and how would that help?
I hope not, laws tend to get outdated real fast. Who knows robots.txt might not even be used in the future and it just there adding space because of law reasons.
robots.txt is a 30 year old standard. If we can write common sense laws around things like email and VoIP, we can do it for web standards too.
robots.txt has been an unofficial standard for 30 years and its augmented with sitemap.xml to help index uncrawlable pages, and Schema.org to expose contents for Semantic Web. I'm not stating it shouldn't not be a law, but to suggest changing norms as a reason is a pretty weak counterargument, man.
All my scrapping scripts go to shit...please no, I need automation to live...