Dalai Alpaca install failing to run, invalid model file 'models/7B/ggml-model-q4_0.bin'

Question

I started out trying to get Dalai Alpaca to work, as seen here, and installed it with Docker Compose by following the commands in the readme:

docker compose build
docker compose run dalai npx dalai alpaca install 7B
docker compose up -d

And it managed to download it just fine, and the website shows up. But when I try to prompt it, I get the error llama_model_load: invalid model file 'models/7B/ggml-model-q4_0.bin' (bad magic). Is there any way to fix this?

Full error in debug mode:

/root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello!" is "" exit root@06ca5ae31973:~/dalai/alpaca# /root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello!" is /root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello""" is "" main: seed = 1684196106 llama_model_load: loading model from 'models/7B/ggml-model-q4_0.bin' - please wait ... llama_model_load: invalid model file 'models/7B/ggml-model-q4_0.bin' (bad magic) main: failed to load model from 'models/7B/ggml-model-q4_0.bin' root@06ca5ae31973:~/dalai/alpaca# exit exit

I have already tried reinstalling it. The file appears to be in the right place (dalai\models\alpaca\models), so I am unsure as to what is going wrong.

Chromaticness

Answer 1

I was having the same problem - I didn't exactly solve it but worked around it by using the instructions from one of the README.md files that was installed when I installed the nodejs/python based solution.

git clone https://github.com/antimatter15/alpaca.cpp

cd alpaca.cpp

make chat

The model file can be fetched with curl - three locations were given but this one was the fastest:

curl -o ggml-alpaca-7b-q4.bin -C - https://gateway.estuary.tech/gw/ipfs/QmQ1bf2BTnYxq73MFJWu1B7bQ2UD6qG7D7YDCxhTndVkPC

Then just run

./chat

It works for me but is quite slow and takes a lot of CPU!

Rafaelof

Works for me too (on i7-10510U running Pop!_OS 22.04). Many thanks ! – Consubstantiation

The download link didn't work for me – Tights

Answer 2

I got it to work by downloading the model from Hugging Face: https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/tree/main

I then copied it to ~/dalai/alpaca/models/7B and renamed the file to ggml-model-q4_0.bin.

I was then able to run dalai, or run a CLI test like this one:

~/dalai/alpaca/main --seed -1 --threads 4 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "What color is the sky?"

Enjoy!

Keane

This did the trick for me. For anyone wondering: Dalai seems to work with different formatted models. This model was converted to the old GGML model format, which is why it works (see: huggingface.co/Sosaka/Alpaca-native-4bit-ggml). – Guyenne

Answer 3

I was not able to solve this problem too. I believe the cause is that the .bin model fails the magic verification which is checking the format of the expected model. I tried to change the model's first 4 bits to what it expects in magic verification error statement i.e "ggml" in ASCII. But that did not solve the problem. This indicates that the format of .bin file is wrong and probably dalai package failed at some step.

Angloamerican

Any solution? I downloaed two versions of the file based off some github discussions on llama.cpp repository and none of the models work. They fail to load. It's almost pointless :/ – Rennie