Models and formats in Inferno

Inferno runs models that live on your own disk. The best-supported path is GGUF.

GGUF (recommended)

GGUF is the format used by the llama.cpp ecosystem, and it’s Inferno’s production backend (built in via --features gguf). Most quantized community models - the Q4, Q5, Q8 variants you’ll see on Hugging Face - are GGUF, which is what makes them small enough to run comfortably on a laptop.

Finding and installing models

Inferno can search and download from Hugging Face directly:

inferno models search mistral
inferno models install TheBloke/Mistral-7B-Instruct-v0.2-GGUF

You can also install from a direct URL, or point Inferno at a folder of models you already have with the INFERNO_MODELS_DIR environment variable. Check a file before loading it with inferno validate <file>.

Other formats

Inferno also has an ONNX backend (an optional --features onnx build). Beyond that, model-format conversion between GGUF, ONNX, and others is on the roadmap but not yet reliable, and non-GGUF/ONNX runtimes are still in progress - so if your goal is “run a model now,” reach for a GGUF model. We’ll expand this article as more paths become production-ready.

Choosing a size

• Smaller quantized models (2-7B, Q4) run fastest and use the least memory.
• If a model won’t load or you run out of memory, try a more aggressively quantized version or a smaller model, and reduce the context size.

Not sure which model fits your machine? Contact support and we’ll suggest a starting point.