this post was submitted on 20 Dec 2024
13 points (100.0% liked)
LocalLLaMA
2795 readers
7 users here now
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I do not know what SFT means. So I can't comment on that, I'm afraid.
Models only predict the distribution of the next token. So "partial response" vs "full response" is a consequence repeated inference untill the stop token is reached. It's mostly unrelated to the model parameters.
For training, it makes no difference.
The article you linked to uses SFT (supervised fine tuning, a specific training technique) as its alignment strategy. There are other ways to fine-tune a model.
I guess I'm wondering if you can train on these partial responses without needing the full rest of the output, without the stop token, or if you need full examples as the article hints to.
I was unaware of that acronym, thank you. It does make me wonder: is there unsupervised training of LLMs?
Yes, you can train without the stop token. The stop token is just that: another token. I do not expect the model to unlearn the usage of the stop token from training on a few 100 new examples.
Unsupervised training happens during the pre-training phase when you dump all kinds of quality documents and it learns the relationship between tokens
Could you perhaps share a reference for this? I'm eager to learn as I don't quite understand.
I've always trained LLM supervised: predict token N+1 based on tokens 1 to N.
This pre-training was done by Meta. It's what Llama-3.1-405B is (in contrast to Llama-3.1-405B-Instruct). https://huggingface.co/meta-llama/Llama-3.1-405B
Oh I see the origin of my confusion. The terminology "supervised learning" got repurposed.
It's all supervised learning if the model is learning the relationship between input and expected output (using supervised learning as described in (1)). The methodology of "pre-training" is the same as that of "supervised fine tuning".
There's no unsupervised learning happening, as described in (2)
No, it's unsupervised. In pre-training, the text data isn't structured at all. It's books, documents, online sources, all put together.
Supervised learning uses data with "ground truth" labels.
Have you worked with (variational) auto-encoders? I think they're a great example of what I would call unsupervised learning.
What are "ground truth" labels?
Ground truth labels are just prescriptive labels that we recognize as being true. The main thing that distinguishes unsupervised from supervised is that in unsupervised learning, what is "good" is learned from the unstructured data itself. In supervised learning, what is "good" is learned from some external input, like "good" human-provided examples.
Would you call token (N+1), given tokens (1 to N) as a ground truth?
No, in that case there's no labelling required. That would be unsupervised learning.
https://en.wikipedia.org/wiki/Unsupervised_learning
So supervised vs unsupervised, according to you, is a property of the dataset?
Sorry, I really don't care to continue talking about the difference between supervised and unsupervised learning. It's a pattern used to describe how you are doing ML. It's not a property of a dataset (you wouldn't call Dataset A "unsupervised"). Read the Wikipedia articles for more details.
It's alright :)