Hi All, I just got a P100, and although it appears to work, i cant quite get gemma4 to run on it properly.
My setup is:
- Ubuntu 22.04 VM
- PCI passthrough the GPU (Connected via thunderbolt) (I know this is unideal, and I might try cut this out later)
- Nvidia 580 driver installed.
- llama.cpp built from source, with cuda enabled.
I got some mostly normal output from qwen, but whenever I try to load gemma4 I get a lot of <unused49><unused49><unused49><unused49>. I did have it briefly working, no idea what changed, but reboot and its back to unused spam.
Anyone hit this before? Ideas on what I should try change?
You must log in or register to comment.
What version of llama.cpp are you using (exactly), what command are you using to run the model (exactly), and which version of Gemma are you trying to run (exactly)?
There are open issues on llama.cpp where people have gotten similar
<unused49>spam under various circumstances, but the cause is not clear – e.g. https://github.com/ggml-org/llama.cpp/issues/26088 – if this can be consistently reproduced, you may want to let them.Similarly, people mentioned it on some of unsloth’s quants a few months back: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2
As far as general workaround advice: try Vulkan instead of CUDA (or vice versa), try a different version of llama.cpp (especially if you’re running either a bleeding edge version or a very old version), and try a different version of the model.
llama.CPP version:
Lastest git, as of 24hrs ago
I did find that limiting context to 32k fixes it, 64k or larger doesnt work. But I’m also sure that it did work at 64k before it broke.
Current suspicion is that because I’m both passing through and using thunderbolt, the GPU isnt getting properly reset at reboot, and is getting to a broken state.
I’m gonna try removing the VM portion of the chain when I get a spare PC. And maybe if that doesnt work I’ll try drop the thunderbolt part as well, but that is a little harder to deal with.
Some GPU gore in the image:
Gave my VM a bit more ram, and right now, its currently working at 255k context. Who knows if it will keep working, but right now I am happy. I am planning on getting an old NUC with plenty of ram, so maybe this will be sorted with that upgrade.
Idk if this is a thing but there is no motherboard to control voltages and you may fry your GFx card that way
Nah, the PSU controls the voltages. All the motherboard does is turn on the PSU by shorting the green PS-ON wire in the mobo cable to ground. The thunderbolt card actually came with a switch to do that, normally I use a paperclip :D.
http://bitsavers.informatik.uni-stuttgart.de/pdf/intel/ATX/ATX_Specification_2.01_199702.PDF
PS-ON is an active low signal that turns on all of the main power rails including 3.3V, 5V, -5V, 12V, and -12V power rails. When this signal is held high by the PC board or left open circuited, outputs of the power rails should not deliver current and should be held at a zero potential with respect to ground. Power should be delivered to the rails only if the PS-ON signal is held at ground potential. This signal should be held at +5VDC by a pull-up resistor internal to the power supply.
Here is an example of a switch: https://a.aliexpress.com/_msxxnjv Mine doesn’t have the third wire, I think its just for powering the switch LED though.
Um. Try using lemonade instead. It’s a ton easier and you can use the api
Is it gui only? I dont have a gui on this VM, and not much space for it either. I’ll give it a go when i get a dedicated box. Does seem to be a wrapper around llama.cpp, vLLM and others though.
Definitely does not require only GUI.
I’ll have another go, but the install instructions didn’t work well on 2204.
The apt method doesnt work, they don’t seem to have a 2204 repo, and the snap method started to download gnome…
Building it from source was pretty fruitless, the cmake scripts give a version error despite having an “accepted” cmake version. Upgrading got me further, but still failed to build.
Increasing the system memory works, so I’m happy for now.
