Serving Inferno and troubleshooting

Once a model is in place, Inferno runs as a server you can point tools at.

Start the server

inferno serve

By default it listens on http://localhost:8080. Check it’s alive with curl http://localhost:8080/health.

An OpenAI-compatible API

Inferno exposes an OpenAI-style API, so a lot of existing tooling works by just changing the base URL. The chat endpoint is:

POST http://localhost:8080/v1/chat/completions

There’s also a native REST surface (/inference, streaming via /inference/stream, /embeddings) and a WebSocket endpoint for real-time streaming. If you enable API keys, send them as Authorization: Bearer <key>.

GPU acceleration

On Apple Silicon, Inferno uses Metal automatically through the GGUF backend, and the speedup over CPU is large - this is the tested, supported GPU path today. Other GPU backends (NVIDIA CUDA, AMD ROCm) are planned but not yet built, so on other platforms expect CPU execution for now.

Common issues

• A model loads but nothing generates: your build likely has no inference backend - rebuild with --features gguf.
• Out of memory: use a more quantized model, or lower the context and batch sizes.
• GPU looks disabled on a Mac: confirm you’re on Apple Silicon and that GPU offload is enabled in your config.
• Debugging config: settings resolve from CLI flags, then INFERNO_* env vars, then .inferno.toml. Run with INFERNO_LOG_LEVEL=debug for detail.

Still not serving? Send us a message with your platform and build command and a human will reply.