shamelessly stolen from nixCraft on mastodon
my meme now
Sure! You’ll probably want to look at train-text-from-scratch in the llama.cpp project, it runs on pure CPU. The (admittedly little docs) should help, otherwise ChatGPT is a good help if you show it the code. NanoGPT is fine too.
For dataset, maybe you could train on French Wikipedia, or scrape from a French story site or fan fiction or whatever. Wikipedia is probably easiest, since they provide downloadable offline versions that are only a couple gigs.
Thank you so much! Have a good break!
kid called EU anticompetitive laws:
The technology of compression a diffusion model would have to achieve to realistically (not too lossily) store “the training data” would be more valuable than the entirety of the machine learning field right now.
They do not “compress” images.
Are you using SDXL? If you are, you need to set the resolution to 1024x1024
I dunno. Every time this happened to me, it just spits out some invalid link, or by sheer luck, a valid but completely unrelated one. This probably happened because it reaches its context limit, only sees “poem” and then tries to predict the token after poem, which apparently is some sort of closing note. What I’m trying to argue is that this is just sheer chance, I mean you can only have so many altercations of text.
the hard-to-kill reptile
UN report says world is racing to well past warming limit as carbon emissions rise instead of plunge
If we just give up, then there is a 0% chance. If we try, then the chance of succession isn’t zero. We have to try to be optimistic. Yes, the world is fucked, but hey, giving up is just accepting that and allowing it.
UN report says world is racing to well past warming limit as carbon emissions rise instead of plunge
Yes, we did invent it. However, that was done by a small group of people that have been in power for generations, and kept it difficult to change to a better system.
What I’m trying to say is that I think most people probably don’t find it very fair that someone like Bezos can just be so ridiculously rich.
Maybe we can change this.
UN report says world is racing to well past warming limit as carbon emissions rise instead of plunge
I don’t really think communism in the extreme version is currently a solution, but there is a simpler solution for now for the ultra-rich if you tax them for a large amount of money proportional to the income let’s say 100% after 10 million per year you quickly fix (I guess bandaid-patch) a big problem with capitalism.
I (was :( ) wearing a cute dress
a person holding a cat with the caption: It’s dangerous to go alone, take this
The models after pruning can be used as is. Other methods require computationally expensive retraining or a weight update process.
Paper: https://arxiv.org/abs/2306.11695
Code: https://github.com/locuslab/wanda
Excerpts: The argument concerning the need for retraining and weight update does not fully capture the challenges of pruning LLMs. In this work, we address this challenge by introducing a straightforward and effective approach, termed Wanda (Pruning by Weights and activations). This technique successfully prunes LLMs to high degrees of sparsity without any need for modifying the remaining weights. Given a pretrained LLM, we compute our pruning metric from the initial to the final layers of the network. After pruning a preceding layer, the subsequent layer receives updated input activations, based on which its pruning metric will be computed. The sparse LLM after pruning is ready to use without further training or weight adjustment. We evaluate Wanda on the LLaMA model family, a series of Transformer language models at various parameter levels, often referred to as LLaMA-7B/13B/30B/65B. Without any weight update, Wanda outperforms the established pruning approach of magnitude pruning by a large margin. Our method also performs on par with or in most cases better than the prior reconstruction-based method SparseGPT. Note that as the model gets larger in size, the accuracy drop compared to the original dense model keeps getting smaller. For task-wise performance, we observe that there are certain tasks where our approach Wanda gives consistently better results across all LLaMA models, i.e. HellaSwag, ARC-c and OpenbookQA. We explore using parameter efficient fine-tuning (PEFT) techniques to recover performance of pruned LLM models. We use a popular PEFT method LoRA, which has been widely adopted for task specific fine-tuning of LLMs. However, here we are interested in recovering the performance loss of LLMs during pruning, thus we perform a more general “fine-tuning” where the pruned networks are trained with an autoregressive objective on C4 dataset. We enforce a limited computational budget (1 GPU and 5 hours). We find that we are able to restore performance of pruned LLaMA-7B (unstructured 50% sparsity) with a non-trivial amount, reducing zero-shot WikiText perplexity from 7.26 to 6.87. The additional parameters introduced by LoRA is only 0.06%, leaving the total sparsity level still at around 50% level.
NOTE: This text was largely copied from u/llamaShill
He's 15 years old now, and his ears really bother him, but he still brutally murders birds in our garden.
the fur on the sofa is from the other cats lol
view more: next ›
but it yummy
also expensive as fuck wtf the last time I had it it was like 12 bucks never again