frit
overview
benchmarks
architecture
github
☀
☾
architecture
stack
token path
observability
gpu fleet
reliability
git
gitops/ , the only
source of truth
polls
flux
1 infra
2 controllers
3 apps
dependsOn order
SINGLE K3S NODE , ONE GPU
1 . infra
gpu-operator , driver + dcgm
vault + external-secrets
cloudnative-pg , kuberay
ready before anything below
2 . controllers
kube-prometheus-stack , and the servicemonitor crd apps register to
3 . apps
ray serve + vllm , one model active
litellm , openai-compatible gateway
open webui
langfuse , per-request traces
guidellm cronjobs , benchmark sweeps
model-warm , pre-pull weights off the serve path
model choice = one line in kustomization.yaml
client
open webui
litellm
gateway
ray serve
proxy + replica
VLLM ENGINE
queue
grows under load
= queue wait
prefill
whole prompt at once
compute-bound
decode
one token per step
bandwidth-bound
kv cache , grows per token per request
TIME TO FIRST TOKEN
gateway hops + queue wait + prefill
INTER-TOKEN LATENCY
one decode step, repeated
a guard model inverts this , it prefills a prompt and emits almost nothing,
so a config tuned for chat is the wrong config for a classifier
EMITS
COLLECTS
ANSWERS
dcgm exporter
gpu util, memory, power,
temperature, sm occupancy
engine metrics
ttft, tpot, queue depth,
kv cache, running vs waiting
node + kube-state
cpu, memory, pod restarts
servicemonitor
prometheus
scrape + store
in-cluster only,
dies with the vm
dcgm dashboard , is the gpu actually busy
ray serve , replicas, routing, errors
serve-llm , engine internals per model
ttft / slo , hand-built, the one that matters
OFF THE METRICS PATH
langfuse
one slow request, not aggregates
guidellm
synthetic sweeps , breaking point, on demand
results pvc
survives teardown
t4 , turing sm_75
asu-sandbox , 16gb
works , small models, head_dim 64
blocks , every 7b tried
no attention backend does
head_dim 128 prefill on sm_75
l4 , ada sm_89
asu-l4 , 24gb , active
unlocks , flashattention, fp8
the 7b kv-pressure story
one card, so scaling is about
batching not sharding
h100 x2 , hopper
asu-h100 , 80gb each
unlocks , nvlink, tensor parallel
the only box where multi-gpu
collectives are real
~$7/hr , minutes not afternoons
COST GUARDRAILS , ALL THREE MACHINES
auto-stop
gcp policy, nightly
make guard
asserts the policy exists
make bill-check
fails if anything runs
make stop-all
loud on failure
SURVIVES A TEARDOWN
boot disk , cached model weights
results pvc , benchmark artifacts
git , every manifest and config
LOST, REBUILT FROM GIT
prometheus history
cluster state , vault needs unseal
make up
flux reconciles the
whole cluster back
CONTROL POINTS
availability
replicas, pdb, health probes,
fail-open vs fail-closed
scaling
queue depth, kv pressure,
batch size, replicas vs nodes
recovery
rebuild from git, warm cache,
vault unseal, spot preemption
the binding measurement
a stated ttft slo, so degraded
is a number not an opinion