> ## Documentation Index > Fetch the complete documentation index at: https://docs.drkm.dev/llms.txt > Use this file to discover all available pages before exploring further. # Models and inference > Connect a provider, serve models on your own hardware, and let chat, agents and jobs pick from one catalog. ## How an outpost gets models Two sources: a hosted provider reached over HTTPS with an API key, and models the outpost serves itself on its CPU, its GPU or a rented GPU pool. Both appear in one catalog of model names that chat, the agent and jobs share. **Auto**, the default, routes each call to a model here or on a verified peer in the mesh: a responsive Grok route first, then each outpost’s preferred model, then every other route. Within each tier, the route expected to finish the call soonest goes first, and a route that failed recently goes last. A call fails over when its route breaks. An Auto turn with no reachable model waits for one, for up to an hour, rather than failing; a turn that names a model nothing holds is refused. ## Connect a provider The first `drkm start`, or `drkm models setup` at any time, offers three choices: connect an inference provider with an API key, run a local model sized for this device’s GPU, or set up later. The provider path offers OpenRouter, or any other OpenAI-compatible endpoint by its HTTPS base URL. It then takes the key, typed hidden, and a model ID, and sends a short test completion before saving the key encrypted. Administrators can also do this in the console under **Settings → Assistant**: **Default model** picks the model for new chats, **Context windows** shows each model’s window and where the figure came from, and **Your own API key** saves the provider connection with an optional context window and private CA. To connect more than one, **Settings → Models → Providers → Add provider** takes OpenRouter, OpenAI, Anthropic, Mistral, Google Gemini, xAI, Groq, Together AI or DeepSeek with just a key, or **Custom** for any OpenAI-compatible server. It then lists that provider’s own models with their context windows, and a switch puts each one in chat. ```bash Terminal theme={null} drkm start drkm models setup drkm console models # the chat catalog: models, served, holders, default ``` > [!NOTE] > > drkm models setup needs a running local outpost and an interactive terminal. The key is stored encrypted and never shown again. ## Run a model on your hardware The wizard’s local choice supports Linux NVIDIA GPUs: it sizes a pinned open model to the GPU it finds and launches the packaged serving runtime, which needs NVIDIA drivers, Docker and the NVIDIA Container Toolkit, 80% of one GPU free and tens of gigabytes of disk for the first build. Outside the wizard, a Mac serves a language model through its native engine with `drkm engine install`, and a CPU-only Linux host serves a quantized GGUF model with `drkm serve --engine llama-cpp` and an explicit engine profile. The model library names models as Ollama does. `drkm pull qwen3:8b` downloads that size, or `drkm pull qwen3` the family’s default, in a file sized to this outpost, and starts it. `drkm run` starts a model the outpost holds, or pulls one it lacks, then opens a `>>>` prompt; that prompt needs a Mac’s native engine, and elsewhere you chat with the model in the console. `drkm ls`, `ps`, `show`, `stop` and `rm` list, inspect, unload and remove models. One local language model runs at a time, so a pull replaces the one running. The lower-level host commands stage any model into the outpost’s registry and serve it: `drkm pull` also stages a registry snapshot by name or imports one from a hub with `hf://`, `drkm serve` starts an engine on it, and `drkm models list` shows what the registry holds. ```bash Terminal theme={null} drkm pull qwen3:8b # or qwen3 for its default size drkm run qwen3:8b # a >>> prompt; /bye leaves drkm ls # models this outpost holds drkm ps # the model it runs drkm stop qwen3:8b # drkm stop alone still stops the outpost drkm rm qwen3:8b drkm models list drkm pull hf:/// --name my-model --revision drkm serve my-model --engine vllm # or llama-cpp, mlx, detector # Mac: serve a language model through the native engine, persistently drkm engine install my-model drkm engine remove ``` ## The Darkmatter catalog In **Settings → Models**, **Set up a model** picks a language model that fits this outpost when none runs. The **Library** lists language model families and their sizes, as Ollama’s library lists them, each size marked for whether it fits this outpost. Clicking a size that fits pulls it and runs it. **Pull a model** searches Hugging Face, or takes a pasted `owner/model`: any public repository pulls without an account, and a gated or private one needs a connected Hugging Face account. **Downloaded models** lists what is stored or running here: the vision detectors, **Overwatch** (planes, ships, vehicles and infrastructure in satellite imagery) and **Silhouette** (people in drone footage), and language models. Each row shows where it runs and its state, and offers **Deploy** or **Load**, **Stop** and **Remove**. Everyone can read the page; administrators act. **Deploy** offers *This node · GPU*, *This node · CPU* or, under Advanced, a *Modal pool* of rented GPUs with a type, a count and how many to keep warm; the first pooled model adopts the pool and removing the last releases it. On an outpost with no serving controller, such as a cloud one, Deploy shows the Modal pool and selects it. An installed outpost downloads catalog models with its update grant; one that is neither signed in nor connected to Hugging Face shows a lock, because it cannot download models it lacks. ```bash Terminal theme={null} drkm start --detector # serve the Overwatch detector here drkm start --no-detector # stop serving it # Rent a GPU pool for the detectors from the host drkm pool adopt modal --model overwatch-obb-r3-int8 --gpus 4 --warm 0 drkm pool release ``` ## Choose a model in chat The composer’s **Model** menu lists *Auto* first, then every catalog name; a model that peers also hold shows *here +N* or the holder’s callsign. **Find model...** filters the list, and for an administrator a **Download** section offers library sizes that fit this outpost: picking one pulls it, shows its progress and selects it once it runs. Below it, **Thinking** runs from **Off** through **Light** and **Balanced** to **Deep**, or **Auto**, which keeps the model’s default. A new chat starts on the outpost’s default; reopening a conversation keeps its model. Each reply says which model answered. ```bash Terminal theme={null} drkm console chat --model auto --effort medium -p "Summarize today's jobs" ``` ## Models for jobs A worker declares the models it needs by alias in `package()`: a catalog name pins that model, a requirement such as `{"kind": "agent", "intelligence": "medium"}` states a floor that placement resolves, and `{"kind": "completion", "model": "vision"}` selects a configured completion lane. Inside the attempt, `context.models["alias"]` is a `drkm.Model` with `predict()`, `predict_batch()` and `analyze_image()`. A `predict` job needs no worker at all: it runs a served detector over each record’s one image, and the platform keeps every GPU the outpost has busy, pooled ones included. ```python Detections without a worker theme={null} import drkm drkm.connect() records = [{"id": "harbor-01", "files": ["tiles/harbor-01.png"]}] job = (drkm.parallelize(records) .predict("overwatch-obb-r3-int8", conf=0.5) .submit(name="Vessels in harbor imagery")) job.wait() print(job.rows()) ``` > [!NOTE] > > The model name must be a detector an eligible outpost serves or can load: overwatch-obb-r3-int8 or one of the Silhouette builds. Settings → Models shows what this outpost holds, and drkm console peer-models \ shows what a member serves now; drkm console models lists the chat catalog only. A detector loaded for a job is unloaded after 15 idle minutes. ## Serve across the mesh An administrator can ask another outpost to serve a catalog model by name, one of the vision detectors or an entry of that member’s pinned language list, never an arbitrary download. The member’s own rules decide: a detector must fit its free GPU memory, a language model starts only on an idle engine, nothing replaces what already serves, and another outpost stops only what it started. ```bash Terminal theme={null} drkm console peer-models drkm console request-model silhouette-a2-v1-fp16 drkm console stop-model silhouette-a2-v1-fp16 ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.