520
this post was submitted on 27 Oct 2023
520 points (95.0% liked)
Technology
69600 readers
3337 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
by run his own models he means locally running a text generation ai on his computer, because sending all that data to openai is a privacy nightmare, especially if you use it for sensitive stuff
But that's still confusing because we already can. Yeah you might need a little bit more of hardware but... not that crazy. Plus some simpler models can be run with more normal hardware.
Might not be easy to setup that is true.
For large context models the hardware is prohibitively expensive.
I can run 4bit quantised llama 70B on a pair of 3090s. Or rent gpu server time. It's expensive but not prohibitive.
I’m trying to get to the point where I can locally run a (slow) LLM that I’ve fed my huge ebook collection too and can ask where to find info on $subject, getting title/page info back. The pdfs that are searchable aren’t too bad but finding a way to ocr the older TIFF scan pdfs and getting it to “see” graphs/images are areas I’m stuck on.
How many tokens can you run it for?
3k?Can't recall exactly, and I'm getting hardwarestability issues.