this post was submitted on 01 Mar 2025
33 points (100.0% liked)
Privacy
1855 readers
244 users here now
Welcome! This is a community for all those who are interested in protecting their privacy.
Rules
PS: Don't be a smartass and try to game the system, we'll know if you're breaking the rules when we see it!
- Be civil and no prejudice
- Don't promote big-tech software
- No reposting of news that was already posted
- No crypto, blockchain, NFTs
- No Xitter links (if absolutely necessary, use xcancel)
Related communities:
Some of these are only vaguely related, but great communities.
founded 5 months ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I've tried scraping arXiv before and they blocked my access to the website stating suspicious activity and that I should contact arXiv owners if I want to scrape it, despite the reason for arXiv existing being for Open Access to scientific articles.
Scraping for scientific articles is still on my plans, but not arXiv anymore.
The gap between the common user and the big technology players is more than just a gap of knowledge, these agreements to keep everything on the hands of big companies is problematic, especially when dealing with important philosophical concepts that should guide some websites, like Open Science.
But it is what it is, no use sending an email to arXiv telling they're wrong and they should change their minds. I'll have to look for options.
arXiv has bulk access methods -- you shouldn't need to scrape their website to get the data: https://info.arxiv.org/help/bulk_data.html
If you really want everything (5TB+), that's available from their S3 bucket if you're willing to cover the transfer costs: https://info.arxiv.org/help/bulk_data_s3.html