for coding tasks you need web search and RAG. It's not the size of the model that matters, since even the largest models find solutions online.
this post was submitted on 16 Jun 2025
121 points (100.0% liked)
Selfhosted
46677 readers
551 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
Any suggestions for solutions?
Not on top of my head, but there must be something. llama.cpp and vllm have basically solved the inference problem for LLMs. What you need is a RAG solution on top that also combines it with web search.
I have it roleplay scenarios with me and sometimes I verbally abuse it for fun.
As cool and neato as I find AI to be, I haven't really found a good use case for it in the selfhosting/homelabbing arena. Most of my equipment is ancient and lacking the GPU necessary to drive that bus.
Learning/practice, and any use that feeds in sensitive data you want to keep on-prem.
Unless you’re set to retire within the next 5 years, the best reason is to keep your resume up to date with some hands-on experience. With the way they’re trying to shove AI into every possible application, there will be few (if any) industries untouched. If you don’t start now, you’re going to be playing catch up in a few years.
Sorry, I am just gonne dump you some links from my bookmarks that were related and interesting to read, cause I am traveling and have to get up in a minute, but I've been interested in this topic for a while. All of the links discuss at least some usecases. For some reason microsoft is really into tiny models and made big breakthroughs there.
https://reddit.com/r/LocalLLaMA/comments/1cdrw7p/what_are_the_potential_uses_of_small_less_than_3b/
https://github.com/microsoft/BitNet
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
https://news.microsoft.com/source/features/ai/the-phi-3-small-language-models-with-big-potential/
I installed Llama. I've not found any use for it. I mean, I've asked it for a recipe because recipe websites suck, but that's about it.
you can do a lot with it.
I heated my office with it this past winter.
It'll work for quick bash scripts and one-off things like that. But there's not usually enough context window unless you're using a 24G GPU or such.
Yeah shell scripts are one of those things that you never remember how to do something and have to always look it up!
load more comments
(3 replies)
Have you tried RAG? I believe that they are actually pretty good for searching and compiling content from RAG.
So in theory you could have it connect to all of you local documents and use it for quick questions. Or maybe connected to your signal/whatsapp/sms chat history to ask questions about past conversations
load more comments
(4 replies)
7b is the smallest I've found useful. I'd try a smaller quant before going lower, if I had super small vram.
I've integrated mine into Home Assistant, which makes it easier to use their voice commands.
I haven't done a ton with it yet besides set it up, though, since I'm still getting proxmox configured on my gaming rig.
load more comments
(6 replies)
I've used smollm2:135m for projects in DBeaver building larger queries. The box it runs on is Intel HD 530 graphics with an old i5-6500T processor. Doesn't seem to really stress the CPU.
UPDATE: I apologize to the downvoter for not masochistically wanting to build a 1000 line bulk insert statement by hand.
How, exactly, do you have Intel HD graphics, found on Intel APUs, on a Ryzen AMD system?
load more comments
(1 replies)
Currently I've been using a local AI (a couple different kinds) to first - take the audio from a Twitch stream; so that I have context about the conversation, convert it to text, and then use a second AI; an LLM fed the first AIs translation + twitch chat and store 'facts' about specific users so that they can be referenced quickly for a streamer who has ADHD in order to be more personable.
That way, the guy can ask User X how their mothers surgery went. Or he can remember that User K has a birthday coming up. Or remember that User G's son just got a PS5 for Christmas, and wants a specific game.
It allows him to be more personable because he has issues remembering details about his users. It's still kind of a big alpha test at the moment, because we don't know the best way to display the 'data', but it functions as an aid.
Hey, you're treating that data with the respect it demands, right? And you definitely collected consent from those chat participants before you Hoover'd up their [re-reads example] extremely Personal Identification Information AND Personal Health Information, right? Because if you didn't, you're in violation of a bunch of laws and the Twitch TOS.
If I say my name is Doo doo head, in a public park, and someone happens to overhear it - they can do with that information whatever they want. Same thing. If you wanna spew your personal life on Twitch, there are bots that listen to all of the channels everywhere on twitch. They aren't violating any laws, or Twitch TOS. So, *buzzer* WRONG.
Right now, the same thing is being done to you on Lemmy. And Reddit. And Facebook. And everywhere else.
Look at a bot called "FrostyTools" for Twitch. Reads Twitch chat, Uses an AI to provide summaries of chat every 30 minutes or so. If that's not violating TOS, then neither am I. And thousands upon thousands of people use FrostyTools.
I have the consent of the streamer, I have the consent of Twitch (through their developer API), and upon using Twitch, you give the right to them to collect, distribute, and use that data at their whim.
Let's take a look at the Developer Agreement that you cited:
You must only retain chat logs as long as necessary for the operation of Your Services or to improve Your Services; do not do so for the purpose of creating public databases or websites, or, in general, to collect information about Twitch’s end users. You must enable, and process, all requests by end users to block, discontinue, delete, or otherwise opt-out of any retention of chat logs for Your Services.
This very clearly states that you are disallowed from retaining chat logs for the general purpose of collecting information about Twitch's end users.
You said that you, "store ‘facts’ about specific users so that they can be referenced quickly," but then later in a different thread state, "I’m not storing their data. I’m feeding it to an LLM which infers things and storing that data." You're retrieving information about specific users at a later time. You've built a database of structureless PII from chat logs. You've chosen to store the data as inferences, which makes it a bad database, but still a database.
I have questions:
When your streamer mentions something deeply personal, like, "how their mothers surgery went," that your tool helped them remember, do they disclose that your tool was involved in that transaction? When the viewer gets weirded out and asks your streamer to not mention that again, or forget it entirely, do you have a way to remove that information from your database and a way to prove it's been deleted? When other people in chat think it's gross, and ask to opt-out, can you even do it?
Regarding FrostyTools: I don't think it's storing the chat logs for a later time. They don't have a data retention section in their TOS or Privacy Policy that isn't related to the streamer. (As in, they hold on to the streamer's Twitch account and some other information for billing, authentication, etc.) I think it's taking the chat logs only for as long as it needs to output a response and then deleting it. Also, this excerpt from the FrostyTools TOS made me chuckle:
This means that you, and not FrostyTools, are entirely responsible for all Content that you upload, post, email, transmit, stream, or otherwise make available via the Service. FrostyTools does not control the Content posted via the Service and, as such, does not guarantee the accuracy, integrity or quality of such Content. You understand that by using the Service, you may be exposed to Content that is offensive, indecent or objectionable. Under no circumstances will FrostyTools be liable in any way for any Content, including, but not limited to, any errors or omissions in any Content, or any loss or damage of any kind incurred as a result of the use of any Content posted, emailed, transmitted, streamed, or otherwise made available via the Service.
You agree that you must evaluate, and bear all risks associated with, the use of any Content, including any reliance on the accuracy, completeness, or usefulness of such Content. In this regard, you acknowledge that you may not rely on any Content created by the Service or submitted to the Service.
This leads me to believe that you can violate the Twitch TOS quoted above using FrostyTools. It is apparent that FrostyTools has positioned itself as an application that creates User Generated Content (like Photoshop or Word).
You must only retain chat logs as long as necessary for the operation of Your Services or to improve Your Services
I'm not storing chat logs.
do not do so for the purpose of creating public databases or websites, or, in general, to collect information about Twitch’s end users.
Not creating any kind of public database either. It's a private tool. Its purpose isn't to massively-collect data about all of twitch either - it's to provide reminders for social situations. If anything, it's an accessibility tool for the disabled.
You must enable, and process, all requests by end users to block, discontinue, delete, or otherwise opt-out of any retention of chat logs for Your Services.
Again - Not storing chat logs. They are processed for information and that information inferred. I am storing reminders for the twitch streamer to talk about a certain subject at a certain time. If I put a reminder in my phone to remember to tell you happy birthday because I saw it on twitch; am I "creating a database of user information"? No. I'm creating a reminder for myself to remember to say happy birthday.
Having a computer help me remember those things isn't a violation. Hell, even something like Microsoft's new AI in windows does the same thing - are THEY violating twitch TOS when you have a browser window open? The answer is no.
When your streamer mentions something deeply personal, like, "how their mothers surgery went," that your tool helped them remember, do they disclose that your tool was involved in that transaction?
No, nor should they be required to.
When the viewer gets weirded out and asks your streamer to not mention that again, or forget it entirely, do you have a way to remove that information from your database and a way to prove it's been deleted? When other people in chat think it's gross, and ask to opt-out, can you even do it?
When they mention not wanting to talk about something, that's listed as something they don't like to talk about, so in a way, yes.
Additionally, I instruct the 'agent' to disregard anything political or religious. - Though so far it's not very good at distinguishing those things. Additionally it's easy to feed it false information though it usually fixes it over time.
So, buzzer WRONG.
Quite arrogant after you just constructed a faulty comparison.
If I say my name is Doo doo head, in a public park, and someone happens to overhear it - they can do with that information whatever they want. Same thing.
That's absolutely not the same thing. Overhearing something that is in the background is fundamentally different from actively recording everything going on in a public space. You film yourself or some performance in a park and someone happens to be in the background? No problem. You build a system to identify everyone in the park and collect recordings of their conversations? Absolutely a problem, depending on the jurisdiction. The intent of the recording(s) and the reasonable expectations of the people recorded are factored in in many jurisdictions, and being in public doesn't automatically entail consent to being recorded.
See for example https://www.freedomforum.org/recording-in-public/
(And just to clarify: I am not arguing against your explanation of Twitch's TOS, only against the bad comparison you brought.)
You're both getting side-tracked by this discussion of recording. The recording is likely legal in most places.
It's the processing of that unstructured data to extract and store personal information that is problematic. At that point you go from simply recording a conversation of which you are a part, to processing and storing people's personal data without their knowledge, consent, or expectation.
This was my main thrust.
load more comments
(4 replies)
You build a system to identify everyone in the park and collect recordings of their conversations? Absolutely a problem, depending on the jurisdiction.
Literally not. The police use this right now to record your location and time seen using license plates all over the nation - with private corporations providing the service.
and being in public doesn't automatically entail consent to being recorded.
And yes, it's called 'expectation to the right of privacy'. Public venues are not 'private' locations, and thus do not need consent. You can, quite literally, record anyone in public.
Even the link you provided agrees.
In the US maybe but not in Germany, Austria and probably most countries in Europe.
load more comments
(6 replies)
load more comments
(5 replies)
load more comments
(8 replies)
Nothing.
view more: next ›