Getting started with Inferno

Inferno is a local AI inference server written in Rust. It runs language models entirely on your own hardware - no cloud, no telemetry - and exposes them over a familiar HTTP API.

Installing

The most dependable way to run Inferno today is to build it from source. Prebuilt binaries, a Docker image, and a Homebrew formula are published on the project’s GitHub releases page as they become available; if you want the latest working build, source is the safe path:

git clone https://github.com/ringo380/inferno
cd inferno
cargo build --release --features gguf

The --features gguf flag matters: Inferno’s model-execution backends are optional build features, and a default build ships without one. If you build without a backend, models will appear to load but won’t actually run - so include gguf (the llama.cpp / GGUF path, which is the production one) when you build. The binary lands at target/release/inferno.

Run your first model

Find and download a model straight from Hugging Face, then run a prompt:

inferno models search llama
inferno models install TheBloke/Llama-2-7B-GGUF
inferno run --model MODEL_NAME --prompt "Hello!"

List what you have locally with inferno models list.

Where to go next

• Models and formats in Inferno covers what you can run.
• Serving Inferno and troubleshooting covers the API and common issues.

Inferno is open source under the MIT license. Hit a wall during setup? Send us a message and a human will reply by email.