LocalLLaMA

2800 readers

20 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

founded 2 years ago

MODERATORS

[email protected]

Can you fine-tune on localized steering of an LLM? (lemmy.dbzer0.com)

submitted 3 months ago* (last edited 3 months ago) by [email protected] to c/[email protected]

19 comments fedilink hide all child comments

I want to fine tune an LLM to "steer" it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 "steers" on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these "steer" operations has value and could be used for training.

So maybe I'm looking for some kind of localized DPO. Does anyone know if something like this exists?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago) (1 children)

Oh I see the origin of my confusion. The terminology "supervised learning" got repurposed.

It's all supervised learning if the model is learning the relationship between input and expected output (using supervised learning as described in (1)). The methodology of "pre-training" is the same as that of "supervised fine tuning".

There's no unsupervised learning happening, as described in (2)

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago) (1 children)

No, it's unsupervised. In pre-training, the text data isn't structured at all. It's books, documents, online sources, all put together.

Supervised learning uses data with "ground truth" labels.

[–] [email protected] 1 points 3 months ago (1 children)

Have you worked with (variational) auto-encoders? I think they're a great example of what I would call unsupervised learning.

Supervised learning uses data with "ground truth" labels.

What are "ground truth" labels?

[–] [email protected] 1 points 3 months ago (1 children)

Ground truth labels are just prescriptive labels that we recognize as being true. The main thing that distinguishes unsupervised from supervised is that in unsupervised learning, what is "good" is learned from the unstructured data itself. In supervised learning, what is "good" is learned from some external input, like "good" human-provided examples.

[–] [email protected] 1 points 3 months ago (1 children)

Would you call token (N+1), given tokens (1 to N) as a ground truth?

[–] [email protected] 1 points 3 months ago (1 children)

No, in that case there's no labelling required. That would be unsupervised learning.

https://en.wikipedia.org/wiki/Unsupervised_learning

Conceptually, unsupervised learning divides into the aspects of data, training, algorithm, and downstream applications. Typically, the dataset is harvested cheaply "in the wild", such as massive text corpus obtained by web crawling, with only minor filtering (such as Common Crawl). This compares favorably to supervised learning, where the dataset (such as the ImageNet1000) is typically constructed manually, which is much more expensive.

[–] [email protected] 2 points 3 months ago (1 children)

So supervised vs unsupervised, according to you, is a property of the dataset?

[–] [email protected] 2 points 3 months ago (1 children)

Sorry, I really don't care to continue talking about the difference between supervised and unsupervised learning. It's a pattern used to describe how you are doing ML. It's not a property of a dataset (you wouldn't call Dataset A "unsupervised"). Read the Wikipedia articles for more details.

[–] [email protected] 1 points 3 months ago* (last edited 3 months ago)

It's alright :)