mistral.rs

Fast, flexible LLM inference

curl -fsSL https://mistralrs.dev/install.sh | sh

Follow the installer, then open a new terminal.

Once installed, start a chat.

mistralrs serve -m Qwen/Qwen3-4B --quant 4

When ready, open localhost:1234/ui. Quickstart