I only have 16GB of RAM in my laptop, but I would love to run a powerful model like that even at 0.5 tokens/s. For the questions I am asking, such a speed is more than enough. If anyone has any suggestions, please let me know! I am not an expert on LLMs, just a humble mathematician.
Someone posted this fork, which should work in just 10 GB RAM, at least for another big model. I thought of your request and came back to spread the news :)
It might be impossible to run such a large model on the laptop, if I were you I'd first start experimenting with running smaller models, using llama.cpp or lmstudio, just to start understanding how it works and learn about some of the concepts involved - quantization and all!
For example Qwen3.5-4B or Qwen3.5-9B could probably be run cpu-only on some quantizations at more reasonable speeds.
If you want a larger one, you could probably run Qwen3.6-27B or 35B-A3B decently slowly.
Now to be humble - maybe it's impossible to run colibri (this project in the OP) on your computer. Note the performance stories - 0.07 token/s on a 24 GB laptop with 24 CPU threads!
