CodeC

codec-sglang (Docker)

The turnkey path, a pre-built SGLang server with the Codec patches applied and a control plane bolted on, in one GPU container. OpenAI-compatible.

codec-sglang is the easy way to stand up a Codec-speaking inference server. It’s a pre-built Docker image bundling:

  • SGLang with the Codec patches already applied, token-native binary transport on /v1/completions, a server-side ToolWatcher that emits tool-call boundaries on the ID stream, and the full compression stack (gzip + brotli + dict-zstd) negotiated via Accept-Encoding per the spec preference order zstd > br > gzip > identity.
  • codec-supervisor, a FastAPI admin sidecar that handles model uploads, Hugging Face pulls, hot-swaps, and reverse-proxies the inference backend.
  • All upstream sglang kernels (flash-attn, sgl_kernel, triton) intact, the patches are applied as an editable overlay.

If you’d rather build sglang yourself from upstream and apply the Codec patches by hand, see sglang vanilla setup.

Quick start

docker run -d --gpus all \
  -p 8080:8080 \
  -v codec-models:/models \
  -v codec-hf-cache:/root/.cache/huggingface \
  -e CODEC_INITIAL_MODEL=Qwen/Qwen2.5-0.5B-Instruct \
  --shm-size 8g \
  wdunn001/codec-sglang:latest

The container boots the supervisor on :8080. The supervisor then launches the sglang backend with Qwen/Qwen2.5-0.5B-Instruct (or whatever you set in CODEC_INITIAL_MODEL). First boot pulls weights from Hugging Face into the persistent cache volume.

GPU prereq: NVIDIA Container Toolkit installed, and --gpus all (or a specific device list). See the NVIDIA install docs. The image targets CUDA 12.

First request

The same endpoint speaks both formats. The client picks per-request:

# OpenAI-compatible (JSON-SSE if stream, JSON otherwise)
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"x","prompt":"Hello","max_tokens":20}'

# Codec wire format, msgpack frames of token IDs (with dict-zstd if a dict is loaded)
curl http://localhost:8080/v1/completions \
  -H "Accept-Encoding: zstd, br, gzip" \
  -d '{"model":"x","prompt":"Hello","max_tokens":20,"stream":true,"stream_format":"msgpack"}'

The full Codec stack on Qwen2.5-0.5B-Instruct at 2 K tokens lands at 291 bytes, 1,707× smaller than the 485 KB JSON-SSE baseline, at the same ~45 ms TTFB. The wire reduction is essentially free in latency.

The model field is ignored when only one model is loaded at a time. The supervisor proxies whichever model is currently loaded. Use /admin/load to swap.

Running your own models

Three ways to point the image at any model you like, progressively more “self-serve”:

1. Override CODEC_INITIAL_MODEL at docker run time

Any Hugging Face repo id works here, or any path inside the container. The supervisor downloads + boots on first start and caches in the volume.

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=meta-llama/Llama-3.1-8B-Instruct \
  -e HF_TOKEN=hf_xxxxx \
  -v hf-cache:/root/.cache/huggingface \
  wdunn001/codec-sglang:latest

HF_TOKEN is only required for gated models.

2. Mount a local model directory

For checkpoints or fine-tunes you don’t want to upload to HF, bind-mount the directory and point CODEC_INITIAL_MODEL at the in-container path:

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=/models/my-finetune \
  -v /path/to/my-finetune:/models/my-finetune:ro \
  wdunn001/codec-sglang:latest

The mount is read-only inside the container. The supervisor cannot mutate your weights.

3. Hot-swap via the admin API after boot

This is what the supervisor adds on top of stock sglang. The container stays up, only the model process swaps. See the Admin endpoints section below for the full surface; the short version:

# Pull from HF into the persistent registry
curl -X POST http://localhost:8080/admin/models/pull \
  -H "Content-Type: application/json" \
  -d '{"repo_id": "Qwen/Qwen2.5-7B-Instruct"}'

# Or upload a tarball of a local fine-tune (multipart, name as query param)
tar -cf my-finetune.tar -C ./checkpoints/my-finetune .
curl -X POST "http://localhost:8080/admin/models/upload?name=my-finetune" \
  -F "file=@my-finetune.tar"

# Hot-swap to it
curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name": "my-finetune"}'

# Or load an HF id directly without staging it first
curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name": "Qwen/Qwen2.5-7B-Instruct", "allow_remote": true}'

Pass CODEC_BACKEND_ARGS to tune sglang per-model (--tp 2 --quantization fp8 --mem-fraction-static 0.9, etc.); for per-load tuning, add extra_args to the /admin/load body to override the supervisor’s default for that single load.

Hot-swap caveat: there’s a few-second gap during the swap (terminate child → fork new → poll /health). For zero-downtime multi-model serving, run multiple containers behind a router. The supervisor is single-model-per-container by design.

Volumes and persistence

MountPurpose
/modelsLocal model store. Models pulled or uploaded here survive container restarts.
/root/.cache/huggingfaceHF download cache. Subsequent pulls of the same revision skip the network.

In the quick-start above, both are named Docker volumes. For production, mount a host directory or a shared filesystem so the cache survives image upgrades.

Environment

VariableDefaultEffect
CODEC_PORT8080Port the supervisor listens on (host-side via -p).
CODEC_INITIAL_MODEL(unset)Model to load on first boot. HF repo id, local name in /models, or absolute path. If unset, the supervisor starts but no backend is running until you call /admin/load.
CODEC_BACKEND_ARGS--mem-fraction-static 0.85 --attention-backend tritonVerbatim arguments appended to python3 -m sglang.launch_server. Use this for --tp, --quantization, mem fractions, etc.
CODEC_BACKEND_PORT30000Internal-only sglang port; rarely needs changing.
CODEC_MODELS_DIR/modelsWhere uploaded / HF-pulled models land. Volume-mounted in the quick-start.
CODEC_STARTUP_TIMEOUT_S1800How long to wait for sglang’s /health after launch (large models take a while).
HF_TOKEN(unset)Required only for gated HF models.

Admin endpoints

The supervisor on :8080 mixes admin routes with a transparent proxy:

MethodPathWhat
GET/healthsupervisor liveness
GET/admin/statuscurrent model + uptime
GET/admin/modelslist models in /models
POST/admin/models/pullsnapshot_download from Hugging Face
POST/admin/models/uploadmultipart tarball upload
DELETE/admin/models/{name}remove from /models
POST/admin/loadhot-swap to a different model
POST/admin/stopstop backend (supervisor stays up)
*/v1/*proxied to the sglang backend

Pull a model from Hugging Face

curl -X POST http://localhost:8080/admin/models/pull \
  -H "Content-Type: application/json" \
  -d '{"repo_id":"Qwen/Qwen2.5-7B-Instruct"}'

Optional fields: revision (branch / tag / commit) and token (per-request HF token; otherwise falls back to the HF_TOKEN env var).

Hot-swap to a different model

curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"qwen2.5-7b"}'

name resolves first against the local /models registry. To load a fresh HF id without staging it first, set allow_remote: true:

curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"Qwen/Qwen2.5-7B-Instruct","allow_remote":true}'

The supervisor stops the running backend, releases the GPU, and starts a new one with the requested model. Inflight requests against /v1/* see a brief unavailable window during the swap.

To override sglang flags for just this load (--tp, --quantization, etc.), pass extra_args:

curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"Qwen/Qwen2.5-7B-Instruct","allow_remote":true,
       "extra_args":["--tp","2","--quantization","fp8"]}'

Upload a model tarball

For air-gapped setups or pre-baked models. name is a query parameter; the tarball is the multipart file field:

tar -cf my-fine-tune.tar -C ./checkpoints/my-fine-tune .
curl -X POST "http://localhost:8080/admin/models/upload?name=my-fine-tune" \
  -F "file=@./my-fine-tune.tar"

The tarball is extracted into /models/<name>/ (path-traversal-safe; symlinks are rejected). Afterwards it’s loadable via /admin/load.

Pointing a Codec client at it

Once running, point any of the language bindings at http://your-host:8080/v1/completions. That’s the same endpoint pattern the language walkthroughs already document. The Codec client decides per-request whether to ask for stream_format: "msgpack" (binary frames) or omit the field (JSON-SSE).

// TypeScript example, same as the sglang walkthrough, just a different host
import { loadMap, Detokenizer, decodeStream } from "@codecai/web";

const map = await loadMap({
  url:  "https://cdn.jsdelivr.net/gh/wdunn001/codec-maps/maps/qwen/qwen2.json",
  hash: "sha256:62c2f94fcbdb9b49d51632314e64aa65894496bc39751cb90866049657a262ad",
});

// stream_format lives in the BODY (piggybacks on OpenAI's request schema);
// Accept-Encoding + Codec-Client-Version are the v0.4.1 Codec REQUEST HEADERS.
// See /docs/protocol/#request-vs-response-where-each-codec-knob-lives.
const resp = await fetch("http://localhost:8080/v1/completions", {
  method: "POST",
  headers: {
    "Content-Type":         "application/json",
    "Accept-Encoding":      "zstd, br, gzip, identity", // full v0.4.1 stack
    "Codec-Client-Version": "0.4",                       // v0.4 normative
  },
  body: JSON.stringify({
    model: "Qwen/Qwen2.5-0.5B-Instruct",
    prompt: "Explain entropy in one paragraph.",
    stream: true,
    stream_format: "msgpack",  // Codec opt-in (body, not header)
    max_tokens: 256,
  }),
});

const detok = new Detokenizer(map);
for await (const frame of decodeStream(resp.body!, "msgpack")) {
  process.stdout.write(detok.render(frame.ids, { partial: !frame.done }));
}

The same code works against vanilla sglang too. codec-sglang’s wire format is bit-identical to upstream-sglang-with-the-PRs.

When to use this vs vanilla sglang

  • Use codec-sglang when you want the protocol working in 30 seconds with no toolchain to babysit. The image bakes the right CUDA, the right sglang build, and the supervisor.
  • Use vanilla sglang when you have a bespoke build (custom kernels, weird CUDA version, internal mirror) or a deploy story that already pulls upstream sglang directly. Apply the Codec patches and you’re equivalent.

License

The codec-supervisor wrapper is published under Business Source License 1.1 by Quasarke LLC. Free for non-production use and for production use under US $5M annual gross revenue (combined with affiliates). Each release auto-converts to Apache-2.0 four years after publication.

The bundled SGLang itself is unchanged from upstream and remains under Apache-2.0. Commercial licensing for the wrapper above the threshold: licensing@quasarke.com.

Source & links

See also