Architecture

Guild runs a mixture-of-experts model on one normal PC: one GPU (NVIDIA or AMD) plus system RAM, on Windows or Linux. The engine is C++ (CUDA or HIP kernels); a Python layer in serve/ provides the web app and the Anthropic-shaped API in front of the engine’s OpenAI-compatible server.

This file is the map. It says what the parts are, how a token flows through them, and which document has the detail - it repeats none of them.

The three layers, and the one rule

The rule that keeps them decoupled: resolution happens once, off the hot path. Core asks the KernelRegistry for one implementation per operation while the model loads, keeps the resulting function pointers and package ids, and then calls them directly. Inside a token or layer loop there is no virtual dispatch and no registry lookup - see MODULAR_RUNTIME.md for why that is a hard rule and not a preference.

The layers meet at two contracts:

• the C ABI in include/guild/abi/ (kernel_abi.h, cpu_expert_abi.h, runner and lifecycle headers) - plain values, explicit error info, no C++ types across the boundary;
• the .gkmetadata / .gkernel / .gkmodule package format, which is how a provider that was not compiled into this binary gets loaded, verified and leased (KERNEL_ABI.md, GKERNEL_FORMAT.md, GKMODULE.md).

A consequence worth knowing: the same Core builds a CPU-only engine, a CUDA engine and (experimental) a SYCL engine, and which kernels exist is a build-time and load-time question, not a set of #ifdefs in the loops.

Component map

flowchart TB
    subgraph clients["Clients"]
        oai["HTTP clients<br/>OpenAI / Anthropic wire formats"]
        web["serve/<br/>web app, MCP tools, Jinja chat templates"]
        cli["guild CLI"]
    end

    subgraph core["Core"]
        srv["src/server<br/>HTTP listener, SSE streaming<br/>src/runtime<br/>GuildModel / GuildSession"]
        plan["src/memory<br/>memory planner"]
        loops["src/core, src/prefill, src/spec<br/>layer loop, batched prefill, verifier, drafters"]
        reg["src/kernels/registry.cpp<br/>operation registry + leases"]
        pkg["src/abi<br/>package manager, trust, install"]
    end

    subgraph providers["Providers"]
        cpu["CPU expert pool<br/>+ native CPU arithmetic"]
        cuda["CUDA dense / attention / MoE"]
        hip["HIP backend"]
        ext["external .gkernel / .gkmodule"]
    end

    oai --> srv
    web -->|subprocess, line protocol| srv
    cli --> srv
    cli --> loops
    srv --> loops
    plan --> loops
    loops --> reg
    pkg --> reg
    reg --> cpu
    reg --> cuda
    reg --> hip
    reg --> ext

Artifacts: what a model is on disk

flowchart LR
    gguf["GGUF shards"] -->|guild convert| gpack["GPACK .gpack<br/>portable: tensors, metadata,<br/>tokenizer, expert blob"]
    gpack -->|guild optimize| gexec["GEXEC .gexec<br/>tuned for one host"]
    gpack --> run["guild run / bench / serve"]
    gexec --> run

• GGUF is the import format, and remains a valid one: the runtime can load shards directly.
• GPACK (GPACK_FORMAT.md) is the portable container: one memory-mapped file with the tensor index, dense weights, optional embedded expert blob, tokenizer assets and auxiliary models. The mapping is the model - weights are views, not copies.
• GEXEC (GEXEC_CONTRACT.md) is not a model: it is what this machine measured about running that model (kernel selection, expert layout, placement plan, workspace sizes), pinned to a hardware and ABI fingerprint. An artifact that does not match is ignored with a reason, never used approximately.

Memory hierarchy

The planner (src/memory/planner.cpp) places everything, and every tier below exists because VRAM is the scarce resource; CPU_GPU_WORKLOAD_ANALYSIS.md has the measurements that drove the split.

Where to read next