Technology
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
Oh yay another model I can't run on my computer :'(
i thought this was about the MUD server software, and got excited. alas.
That's nice and all, but what are some FOSS models I can run on GPU with only 4GB?
I've tried Deepseek Coder, and it's pretty nice for what I use it for. Then there's TinyLlama, which... well it's fast, but I need to be veeeery exact in how I prompt it.
Unfortunately LLMs need a lot of VRAM. You could try using koboldcpp, it runs on the CPU but let's you offload layers onto the GPU. That way you might be able to stay withing those 4gb even with larger models.
Edit: I forgot to mention there's a fork of koboldcpp with rocm for AMD cards, which is about twice as fast if I remember correctly. Only relevant if you have an AMD card tho.
Edit 2: This is the model I use btw
I'm currently playing around with the Jan client, which uses the nitro engine. I think I need to read up on it more, because when I set the ngl value to 15 in order to offload 50% to GPU like the Jan guide says, nothing happens. Though that could be an issue specific to Jan.
Maybe 50% GPU is already using too much VRAM and it crashes. You could try to set it to 0% GPU and see if that works.
4GB is practically nothing in this space. Ideally you want at least 10GB of dedicated vram if you can't get even more. Keep in mind you're also probably trying to share that vram with your operating system. So it's more like ~3GB before you even started.
Kolboldcpp is capable of using both your GPU and CPU together, you might wanna consider that. (Using a feature called layers) There's a trade-off that occurs between the memory available and the quality of its output and the speed of the calculation.
The model mentioned in this post can be run on the CPU with enough system ram or swap.
If you wanna keep it all on the GPU check out 4bit models. Also there's been a lot of work into trying to do this with the raspberry Pi. I suspect that their work could help you out here as well.
Depends on your needs. Best look around in [email protected] or similar. (I don't wanna say reddit but r/localLlama is much larger.)
If you're more into creative writing, maybe look for places that discuss SillyTavern (r/SillyTavernAI is an option). It's software for role-play chats, which may not be what you want. But the community is (relatively) large and likely to have good tips for non-coding/less technical applications.
If only I had the $ to get a rig that could run this locally
OOTL: What is a LLM and what does it do?
Large Language Model AI. Like ChatGPT.
And at 72 billion parameters it's something you can run on a beefy but not special-purpose graphics card.
Based on the other comments, it seems like this needs 4x as much ram than any consumer card has
It hasn't been quantized, then. I've run 70B models on my consumer graphics card at a reasonably good tokens-per-second rate.