DavidGarcia

latex exists to make your text look more professional, not to make you more productive, duh.

pig lipstick

ah I see, that makes sense somewhere probably

relationship with parents improves since I am finally passionate about something and applying myself

yeah right, lies, fabrications

your parents are supposed to castigate you for wasting your time with doing something you like doing

that's what real and good parents do

ok rude. you're creepy, miss author of this article

okay so the opposing sides of a cube die always add up to 7 right? Does that mean the other side of that 9 has a -2?

or is it not a cube and actually has >= 9 faces?

anyone who doesn't understand lemmy after using reddit must be a real pea brain

that one makes me so uncomfortable, similar to Made in Abyss, worst horror show ever

didn't the show say elves only live a few thousand years? I was under the impression she's like 1000-2000 years old and will die in another 1000-2000 or something

NiceNice

Investors poured completely insane amounts of money into thd endless money pit that is ClosedAI. then they realize betting everything on one horse was really stupid, since they have zero competitve advantage.

now they try to get as much loot off the sinking ship as possible lmao

they'll probably exit scam soon

Q4 will give you like 98% of quality vs Q8 and like twice the speed + much longer context lengths.

If you don't need the full context length, you can try loading the model at shorter context length, meaning you can load more layers on the GPU, meaning it will be faster.

And you can usually configure your inference engine to keep the model loaded at all times, so you're not loosing so much time when you first start the model up.

Ollama attempts to dynamically load the right context lenght for your request, but in my experience that just results in really inconsistent and long time to first token.

The nice thing about vLLM is that your model is always loaded, so you don't have to worry about that. But then again, it needs much more VRAM.

In my experience anything similar to qwen-2.5:32B comes closest to gpt-4o. I think it should run on your setup. the 14b model is alright too, but definitely inferior. Mistral Small 3 also seems really good. anything smaller is usually really dumb and I doubt it would work for you.

You could probably run some larger 70b models at a snails pace too.

Try the Deepseek R1 - qwen 32b distill, something like deepseek-r1:32b-qwen-distill-q4_K_M (name on ollama) or some finefune of it. It'll be by far the smartest model you can run.

There are various fine tunes that remove some of the censorship (ablated/abliterated) or are optimized for RP, which might do better for your use case. But personally haven't used them so I can't promise anything.