this post was submitted on 24 Jul 2024
190 points (100.0% liked)
Technology
71355 readers
3827 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
iirc, isn't robots.txt more of a gentlemen's agreement? I vaguely recall bots being able to crawl a site regardless, it's just that most devs respect robots.txt and don't. Could be wrong though, happy to be corrected.
That's a good point, it's probably way less load and overhead if Reddit and Google just sent info back and forth instead of scraping. Good way for Google to keep their spot as the favoured search engine and beat the competition too, since everything that comes up these days are articles full of SEO nonsense at best, then AI generated nonsense at worst. If nobody else can read the actual human responses, Google has a huge leg up. Also interesting to see that Google's honouring the txt file even when nobody's holding them to it.
I had no idea Twitter's search updated their index immediately after a comment is posted though. That's a lot of updates considering the amount of posts they get daily.