This is an update to a previous post found at https://lemmy.world/post/27013201

Ollama uses the AMD ROCm library which works well with many AMD GPUs not listed as compatible by forcing an LLVM target.

The original Ollama documentation is wrong as the following can not be set for individual GPUs, only all or none, as shown at github.com/ollama/ollama/issues/8473

AMD GPU issue fix

1. Check your GPU is not already listed as compatibility at github.com/ollama/ollama/blob/main/docs/gpu.md#linux-support
2. Edit the Ollama service file. This uses the text editor set in the $SYSTEMD_EDITOR environment variable.

sudo systemctl edit ollama.service

3. Add the following, save and exit. You can try different versions as shown at https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides-on-linux

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

4. Restart the Ollama service.

sudo systemctl restart ollama

Ollama lets you download and run large language models (LLMs) on your device.

Install Ollama on Arch Linux (Windows guide coming soon)

1. Check whether your device has an AMD GPU, NVIDIA GPU, or no GPU. A GPU is recommended but not required.
2. Open Console, type only one of the following commands and press return. This may ask for your password but not show you typing it.

sudo pacman -S ollama-rocm    # for AMD GPU
sudo pacman -S ollama-cuda    # for NVIDIA GPU
sudo pacman -S ollama         # for no GPU (for CPU)

3. Enable the Ollama service [on-device and runs in the background] to start with your device and start it now.

sudo systemctl enable --now ollama

Test Ollama alone (Open WebUI guide coming soon)

1. Open localhost:11434 in a web browser and you should see Ollama is running. This shows Ollama is installed and its service is running.
2. Run ollama run deepseek-r1 in a console and ollama ps in another, to download and run the DeepSeek R1 model while seeing whether Ollama is using your slow CPU or fast GPU.

AMD GPU issue fix

https://lemmy.world/post/27088416

y2u.be/aVvkUuskmLY

Llama 3.1 (405b) seems 👍. It and Claude 3.5 sonnet are my go-to large language models. I use chat.lmsys.org. Openai may be scrambling now to release Chatgpt 5?

I'm an avid Marques fan, but for me, he didn't have to make that vid. It was just a set of comparisons. No new info. No interesting discussion. Instead he should've just shared that Wired podcast episode on his X.

I wonder if Apple is making their own large language model (llm) and it'll be released this year or next year. Or are they still musing re the cost-benefit analysis? If they think that an Apple llm won't earn that much profit, they may not make 1.

Click Here to be Taken to the Megathread!

from [email protected]

Vicuna v1.5 Has Been Released!

Shoutout to [email protected] for catching this in an earlier post.

Given Vicuna was a widely appreciated member of the original Llama series, it'll be exciting to see this model evolve and adapt with fresh datasets and new training and fine-tuning approaches.

Feel free using this megathread to chat about Vicuna and any of your experiences with Vicuna v1.5!

Starting off with Vicuna v1.5

TheBloke is already sharing models!

Vicuna v1.5 GPTQ

7B

• Vicuna-7B-v1.5-GPTQ
• Vicuna-7B-v1.5-16K-GPTQ

13B

• Vicuna-13B-v1.5-GPTQ

Vicuna Model Card

Model Details

Vicuna is a chat assistant fine-tuned from Llama 2 on user-shared conversations collected from ShareGPT.

Developed by: LMSYS

• Model type: An auto-regressive language model based on the transformer architecture
• License: Llama 2 Community License Agreement
• Finetuned from model: Llama 2

Model Sources

• Repository: https://github.com/lm-sys/FastChat
• Blog: https://lmsys.org/blog/2023-03-30-vicuna/
• Paper: https://arxiv.org/abs/2306.05685
• Demo: https://chat.lmsys.org/

Uses

The primary use of Vicuna is for research on large language models and chatbots. The target userbase includes researchers and hobbyists interested in natural language processing, machine learning, and artificial intelligence.

How to Get Started with the Model

• Command line interface: https://github.com/lm-sys/FastChat#vicuna-weights
• APIs (OpenAI API, Huggingface API): https://github.com/lm-sys/FastChat/tree/main#api

Training Details

Vicuna v1.5 is fine-tuned from Llama 2 using supervised instruction. The model was trained on approximately 125K conversations from ShareGPT.com.

For additional details, please refer to the "Training Details of Vicuna Models" section in the appendix of the linked paper.

Evaluation Results

Vicuna is evaluated using standard benchmarks, human preferences, and LLM-as-a-judge. For more detailed results, please refer to the paper and leaderboard.

I noticed there didn't seem to be a community about large language models, akin to r/localllama. So maybe this will be it.

For the uninitiated, you can easily try a bleeding edge LLM in your browser here.

If you loved that, some places to get started with local installs and execution are here-

https://github.com/ggerganov/llama.cpp

https://github.com/oobabooga/text-generation-webui

https://github.com/LostRuins/koboldcpp

https://github.com/turboderp/exllama

and for models in general, the renowned TheBloke provides the best and fastest releases-

https://huggingface.co/TheBloke