Kubernetes
deploy/helm/loom runs Loom as N identical replicas on Kubernetes: one
Deployment of web pods over one ClickHouse service, each pod a full replica. It works on
any conformant cluster; values-eks.yaml sets it up on EKS, values-kind.yaml runs it on
a laptop, and values-clickhouse-cloud.yaml, over the EKS one, adds ClickHouse Managed
Postgres as the memory authority (ClickHouse Cloud and Managed
Postgres).
| Object | What it does |
|---|---|
Deployment <release> |
The web pods. Rolls one pod at a time and takes none down before its replacement is ready. |
| Service, optional Ingress | Routes to ready pods only. The readiness probe is /v1/health/ready. |
| Secret | The keys the Loom pods share (the extractor’s read only its token). Bring your own with secrets.existingSecret. The rest of the env is in the pod template. |
ConfigMap <release>-settings |
The settings file (settingsFile), mounted at /app/config/loom.yaml on every pod; each hook Job carries its own copy. |
| PodDisruptionBudget | One pod at a time on node drains and upgrades. |
| Optional HPA | CPU-based. Rate-limit shares resize as pods come and go. |
Optional <release>-workers |
Background workers in their own pods (workers.enabled). Off by default: one web pod holds the lease and runs them. |
Optional <release>-publish |
A pre-upgrade Job for a change that cannot roll (below). |
Optional <release>-analytics-export |
A daily CronJob and its own ServiceAccount (analyticsExport.enabled, below). |
Optional <release>-extractor |
The document extractor’s Deployment, Service and NetworkPolicy (extractor.enabled, below). |
NetworkPolicy <release> |
Full policy with networkPolicy.enabled; else all egress but egressExcept, unless it is [] or guardMetadata: false. |
Optional <release>-code-tiers |
A PrometheusRule on the failing code-index tiers (alerts.prometheusRule.enabled, below). |
| Optional dev ClickHouse | devClickhouse.enabled: one ClickHouse pod with embedded Keeper. For test clusters only. |
Before you install
- ClickHouse. One service for every pod: ClickHouse Cloud, or a self-managed server
with the
deploy/clickhouse-config/files mounted and Keeper (requirements). Create the database:CREATE DATABASE loom. On a self-managed replicated cluster, point every pod at one replica. - A root secret.
LOOM_JWT_SECRET, at least 32 random bytes (openssl rand -hex 32). Every pod shares it, and it seals the signing keys and stored credentials in ClickHouse, so keep a copy outside the cluster. - A public URL.
publicBaseUrl, what clients reach Loom on. Pods refuse to start without it.
Install
helm install loom deploy/helm/loom -n loom --create-namespace \
--set publicBaseUrl=https://loom.example.com \
--set clickhouse.host=abc123.us-east-1.aws.clickhouse.cloud \
--set secrets.jwtSecret=$(openssl rand -hex 32) \
--set secrets.clickhousePassword=... \
--set secrets.extra.OPENAI_API_KEY=...
kubectl -n loom rollout status deploy/loom
The first pod runs the migrations under the migrate lease and the others wait for it,
so the first install takes a few minutes. The startup probe allows 15 minutes
(probes.startup), which also covers cloning the git mirrors.
The embedding settings go through embedding.*, and the root secret and its previous
value through secrets.jwtSecret and secrets.jwtSecretPrevious; the chart guards these
(below). The root secret can instead live in the secrets.existingSecret Secret, whose
contents the chart cannot check. The chart refuses all of them in env, extraEnv,
settingsFile and secrets.extra, and an envFrom source must not carry them either. The
deployment probe port goes through cellProbe.port below. Every other
setting goes in env (plain values) or secrets.extra (keys), named as in
the configuration reference. settingsFile takes the same
settings as the YAML settings file holds them, lowercase and typed:
settingsFile:
memory_search_default_mode: rrf
memory_ask_recency_weight: 0.3
The chart always renders the settings file (an empty settings map when settingsFile
is empty), mounts it at /app/config/loom.yaml on every pod and on the publish, reseal
and rollback-check Jobs (each from its own copy, rendered with the Job), and rolls the
pods when it changes; env wins over it. A pod of the previous ReplicaSet that starts
during a rollout still finds the file, and reads the new revision’s copy: a pod of the
previous build logs a setting it does not have and ignores it, so a new setting can go
in the same upgrade, but it refuses to start on a value only the new build accepts (a
new choice of an existing setting, a widened bound). Set such a value in the upgrade
after the one that changes the image. A live setting can also change without a rollout
(runtime settings). values.yaml documents the chart’s own values.
Preparing a cell
For cell preparation, enable the private deployment-identity listener in the release’s values:
cellProbe:
port: 8777
Upgrade the release and wait for the web and, when enabled, workers Deployments to finish rolling before running prepare. Both pod types run the listener; a separate workers child does not. The listener requires Postgres and authentication; this value does not configure those prerequisites.
The chart creates a ClusterIP Service named <fullname>-cell-probe, shortening the
fullname to keep the suffix within 63 characters. It publishes unready pod addresses
and selects only those two workload types. Its configured origin is
http://<probe-service>.<namespace>.svc:<port>. A later cell passes that origin as
--probe-url; the router appends /v1/health/deployment. For the first cell, put the
origin in the router’s LOOM_ROUTER_UPSTREAM_PROBE_URL instead and omit --probe-url
from prepare. The full direct diagnostic endpoint is
http://<probe-service>.<namespace>.svc:<port>/v1/health/deployment.
The default port is zero, which disables the listener and Service. Enabled ports
must be whole numbers from 1 to 65535 except 7777. Set the port only through
cellProbe.port: the chart refuses it in env, extraEnv, secrets.extra and
settingsFile, and renders an explicit value, including zero, over envFrom.
The endpoint membership label loom.clickhouse.com/cell-probe is chart-owned when
enabled. Application traffic and health probes continue to use the HTTP port.
ClusterIP supplies routing, not authentication. The listener is unauthenticated and returns nonsecret deployment identity. With the chart’s ingress NetworkPolicy on, add a separate ingress policy allowing the probe port from router pods and the CLI’s location; without ingress isolation, cluster peers can reach it. The chart does not add that rule. A load-balanced response does not prove every process was sampled; the cell preparation checks remain required.
On a laptop
kind create cluster
helm install loom deploy/helm/loom -f deploy/helm/loom/values-kind.yaml \
--set secrets.jwtSecret=$(openssl rand -hex 32)
kubectl port-forward svc/loom 7777:7777
This runs ClickHouse inside the release, the fake embedder and no repository-job
sandbox. Loom’s pods meet the restricted Pod Security Standard; the dev ClickHouse
does not. Mint a token with
kubectl exec deploy/loom -- loom mint-token --subject me --org dev --namespaces '*' --roles admin.
On EKS
values-eks.yaml assumes the AWS Load Balancer Controller, the EBS CSI driver with a
gp3 StorageClass and ClickHouse Cloud. Replace its placeholders (hostname,
certificate, account, VPC range) and install with it:
helm install loom deploy/helm/loom -n loom -f deploy/helm/loom/values-eks.yaml
-
ALB to pod IPs.
target-type: ipsends to pods directly. Label the namespace before the install, so each new pod carries the controller’s readiness gate and a rollout removes no old pod before the ALB sends to its replacement:kubectl label namespace loom elbv2.k8s.aws/pod-readiness-gate-inject=enabledThe target group checks
/v1/health/readyevery 5 s. A stopping pod keeps serving for 20 s (shutdown.readyDelaySeconds) while the controller deregisters it, and the deregistration delay (50 s,readyDelaySeconds+gracefulSeconds) lets it finish what it holds. The 300 s idle timeout covers long calls, and Loom keeps idle connections open longer (keepAliveSeconds, 310 s), so the ALB never reuses one Loom has just closed. The image reads it asLOOM_KEEP_ALIVE_S. -
trustedProxies. Set it to the ALB’s subnets, or every client counts as the ALB for the per-address sign-in limits. Not the whole VPC: any pod there could then setX-Forwarded-For. -
Secrets. The overlay names a Secret
loom-secrets. With the External Secrets Operator and AWS Secrets Manager:apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: { name: loom-secrets, namespace: loom } spec: secretStoreRef: { name: aws-secrets-manager, kind: ClusterSecretStore } target: { name: loom-secrets } refreshInterval: "0" # sync on demand only (see below) dataFrom: - extract: { key: loom/production } # LOOM_JWT_SECRET, LOOM_CH_PASSWORD, OPENAI_API_KEY -
Identity. The service account carries an IRSA role annotation, for Bedrock, S3 backups or the document bucket. With EKS Pod Identity, bind the role to the service account and drop it.
-
Zones. Pods spread over zones, then nodes, and still schedule when a zone is short. The spread is kept at scheduling only: after a drain two pods can share a zone until the next rollout or the descheduler moves one.
-
Cache on gp3. Each pod gets its own ephemeral volume for
/app/var(git mirrors, the Go module cache), created with the pod and deleted with it. The StorageClass needsvolumeBindingMode: WaitForFirstConsumer, so each volume is made in its pod’s zone. -
NetworkPolicy. On, with ingress from the ALB’s subnets only. Pods reach DNS, ClickHouse, the EKS Pod Identity agent and port 443 anywhere but the instance metadata service; the cluster needs a policy engine (the VPC CNI’s network policy agent, Calico or Cilium). Also set the node groups’ IMDS hop limit to 1. A ClickHouse inside the cluster needs its own rule, as engines differ on matching pod IPs by range:
networkPolicy.extraEgress: [{to: [{namespaceSelector: …, podSelector: …}], ports: [{port: 8123}]}]. -
ClickHouse Cloud’s IP access list must allow the cluster’s NAT gateway addresses.
-
Image. Pin
image.digest, andextractor.image.digestwithextractor.enabled; each release’s run summary prints both, after each image passed a CVE scan. The assistant runs in the Loom web image.
Upgrades and scaling
helm upgrade with a new image.tag rolls the pods one at a time. A new pod starts
serving only after its startup probe passes, and the old one is removed after that. The
mixed-version window is safe: builds N-1 and N run side by side on N’s schema
(rolling upgrades). Each pod template carries its
env, so a pod of the previous ReplicaSet that restarts mid-rollout keeps its own. The
Secret is one for every pod, so a change to it reaches each pod as it next starts; a
chart-managed Secret change rolls the pods through its checksum, and an existingSecret
change needs kubectl rollout restart. Keep a reloader off the Secret that holds
LOOM_JWT_SECRET: a root change has to go through a publish (below). For an upgrade whose
release notes ask for every pod at once (state that moves between builds), add
--set strategy.rollingUpdate.maxSurge=100%, and set it back on the next upgrade:
--reuse-values keeps it.
A stopping pod answers not-ready and keeps serving for shutdown.readyDelaySeconds
(15 s) while the Service and the balancer drop it. uvicorn then waits up to
shutdown.gracefulSeconds (30 s, the longest task.pull wait) for requests in flight,
and the lifespan drains and retires the pod. terminationGracePeriodSeconds is the sum
of the three shutdown values, 95 s by default. Behind a balancer with a slower check,
raise readyDelaySeconds. A pod that is killed (OOM, node loss) fails only the requests
it held; a client’s retry lands on another pod.
Scaling is replicaCount, kubectl scale or the HPA. It changes no pod config, so it
rolls nothing. With autoscaling.enabled each revision records the Deployment’s live
replica count (at least autoscaling.minReplicas), so turning the HPA on does not first
scale it down. A helm rollback writes the count its revision recorded, and the HPA
then corrects it: size minReplicas for the load a rollback must carry. The workers lease moves when its holder stops: within seconds on a
graceful stop, and within the 30 s lease TTL when a pod or node dies. Every pod is
labelled deploy:k8s:<pod>, so a new leader runs the workers whose last pass is
overdue rather than waiting out their interval again.
A pod that dies without stopping leaves its provider participant row. A missing
or replaced pod object, a Failed/Succeeded API phase and a completed rollout do
not prove that an old process on a partitioned node stopped. Such rows remain
blockers until guarded shutdown or verified exact physical death with settled
queries. Loom requests no pod-read permission or service-account token for
participant retirement. The explicit serviceAccount.automountToken value stays
available for workloads that need a token for another integration.
Use loom participant retire --participant <id> on the trusted host where its
exact PID/namespace/start identity is verifiable. If that evidence is unavailable,
use the quiescent lifecycle recovery
after positively terminating or fencing every relevant runtime. API object
cleanup alone cannot authorize retirement.
There is no migration Job. Every pod migrates at boot under a ClickHouse lease, so a Job would only repeat that step, and a pre-upgrade hook would run the new schema while every old pod still serves, outside the window the carry barrier protects.
Changing what cannot roll
The embedding settings (embedding.*) and LOOM_JWT_SECRET must be the same on every
pod. A pod whose values differ from the deployment’s published ones is not ready, so
these do not roll. The chart refuses an upgrade that changes them: it keeps a hash of
the embedding settings on the Deployment and compares the root secret with its Secret.
Error: UPGRADE FAILED: ... this upgrade changes the embedding settings or LOOM_JWT_SECRET ...
To make the change on purpose, upgrade with --set mustMatchChange.publish=<token>, any
value the pods don’t carry yet, such as $(date +%s): the upgrade that brings a new token
publishes, and later ones that keep it (--reuse-values) don’t. A pre-upgrade Job runs
loom signing-key record-root and loom deployment publish --yes with the new values.
From that moment the old pods are not ready and answer calls on the connections they
still hold with 503 CONFIG_DRIFT, and the rollout replaces every pod at once
(maxSurge: 100%). The outage lasts until the first new pod is ready and registered with
the balancer: about 10 s with the image on the node, plus the image pull and the ALB’s
registration otherwise. Change nothing else in that upgrade (no new image, no pending
migration), and pull the image onto the nodes first. The Deployments keep that surge
strategy until the next upgrade: follow the publish with a helm upgrade --reuse-values
(the same token, so no publish), or a later kubectl rollout restart surges every pod.
A helm rollback to the publish revision brings that strategy back too: follow it the
same way.
Then:
- New root secret: tokens the old root signed (kid
default) stop verifying at once; mint new ones. Tokens signed with a keyring key (loom signing-key rotate) keep verifying. Setsecrets.jwtSecretPreviousto the old root in the same upgrade, and what it sealed keeps opening: after the rollout apost-upgradeJob runsloom secrets reseal, which seals the stored credentials, signing keys and rebuild recipes again under the new root (the publish Job checks first that it can, so a failing one publishes nothing). The previous root only opens sealed values; it never verifies a token. Remove it in a later upgrade oncekubectl exec deploy/loom -- loom secrets reseal --checkexits 0. Until then it also counts the SSO sign-ins in flight (a code, a device grant, an approval or link awaiting its click) whose identity-provider token only the old root opens: they are not rewritten, and expire within minutes; the check names when the last one does. A value it lists as opening with neither root was lost before this rotation: enter it again. Without it, the stored credentials stop opening: enter them again. If the old root leaked, also rotate the signing keys and retire the old kids (tokens and grants). - New embedding model: run
loom reembedin a pod (deployment).
Run the upgrade with --wait and a --timeout above the startup budget (15 minutes), and
never with --atomic: a rollback after the Job has published leaves every pod drifted. If
the upgrade fails after the Job ran (the Job half-done, or the apply refused), every pod
is drifted: run the same upgrade again with the same token, which publishes again. If
the upgrade timed out after the rollout started, the pods already carry the token and the
retry does not publish; while secrets.jwtSecretPrevious is set, every upgrade’s
post-upgrade Job reseals what is left, and loom secrets reseal does it by hand.
Every helm rollback first runs loom deployment publish --check with the target
revision’s env, and refuses a rollback whose root secret or embedding settings are not the
published ones: its pods would never be ready, and the Secret it re-applies would drift
every pod that restarts. Helm then leaves the release pending-rollback; a
helm rollback loom <current revision> passes the check and clears it. To go back past a
publish, upgrade with the old values and a new token instead.
Installing onto a ClickHouse whose deployment already published another root secret or embedding space (a move from another install, a restore) brings up only drifted pods, as hooks do not publish on install: follow the install with an upgrade that sets a token.
With secrets.existingSecret the chart cannot see the Secret’s contents, so it does not
guard a new root secret there. A pod that starts after the Secret changed and before the
publish is drifted and never ready, so change it only together with a publish: with the
External Secrets Operator, update Secrets Manager (the new LOOM_JWT_SECRET, and the old
one as LOOM_JWT_SECRET_PREVIOUS; secrets.jwtSecretPrevious applies to the chart’s own
Secret only), force a sync
(kubectl annotate externalsecret loom-secrets force-sync=$(date +%s) --overwrite), wait
for the ExternalSecret to be Ready, then upgrade with a new token, which rolls the pods
so they read it. The guard and the live replica count read the release with lookup,
which helm template (and so Argo CD) does not run: there the guard is off, and every
render with a token set would publish again, so never set mustMatchChange.publish there.
A publish is then loom secrets reseal --dry-run && loom signing-key record-root && loom deployment publish --yes in a pod with the new env (the dry run checks the previous root
while the old one is still the recorded root), then loom secrets reseal once the new pods
serve. A loom login code shown while the root changes stops matching (its key derives
from the root): run loom login again. So does an SSO sign-in started on a pod with the
other root: start it again.
Cache and git mirrors
Each pod clones its own mirrors at startup and keeps them fetched. A pod is ready before
its clones finish: until a mirror is there, history reads (blame, commit checks, PR
history) answer no_git_mirror and the rest of the code plane serves from ClickHouse.
A clone takes as long as the repository: a bare clone of a large monorepo (14 GB) is
about 12 minutes, two at a time per pod. To hold traffic until the mirrors are cloned,
set env.LOOM_READY_REQUIRES_MIRRORS: "true" and raise probes.startup.
The mirrors live in the pod’s cache at /app/var, with the Go module cache, and are
lost with the pod. The default emptyDir (cache.sizeLimit, 20 GiB) counts against the
node’s storage, and the kubelet evicts a pod that outgrows it. cache.volume: ephemeral
gives each pod its own volume from a StorageClass instead (cache.ephemeral). Size it
for full clones of every indexed repository plus the module cache. workers.cache is
merged over cache for the workers pods, which also keep a checkout per compiler tier:
about 30 GB beside the mirror for a ClickHouse-sized C++ entry with its build tree.
cache.ephemeral.dataSourceRef starts each new pod’s volume from a snapshot instead of
empty, e.g. a VolumeSnapshot of a warm workers volume (on EKS this needs the CSI
snapshot controller and a VolumeSnapshotClass for the EBS driver). The mirrors and
checkouts then fetch only what is new, and the C++ tier skips its CMake configure.
Refresh the snapshot now and then: an old one still works, it just leaves more to fetch.
Compiler tiers
Every compiler tier runs in the workers process: the per-merge deltas, the weekly C++
full re-index (code_scip_full, which also seeds a new entry), the SQL ↔ C++ crossref
and the checkout repacks. The image carries every indexer; scip-clang is in the amd64
image only, because upstream builds no arm64 Linux binary, so the chart schedules on
kubernetes.io/arch: amd64 nodes by default. On arm64 set nodeSelector to
{kubernetes.io/arch: null} and mount your own scip-clang
(env.LOOM_CODE_SCIP_CLANG_PATH), or the C++ tier skips with scip-clang missing.
A C++ full index is the workers pod’s peak. scip-clang runs workers.scipClangJobs (4)
translation units at once at a few GB each, and the default workers.resources (32 GiB
limit) is sized for that. Without the workers Deployment the web pod that holds the
lease runs them with one job inside its own limits. The index and its delta are written
under /app/var/scip-work on the cache volume (LOOM_CODE_SCIP_WORK_DIR), not /tmp:
several GB for a full run of a large tree.
Repository-job sandbox
Repository jobs run repository code (repository-job sandbox).
Loom runs them under bubblewrap, which needs unprivileged user namespaces. The default
pod security does not allow them, so by default (sandbox.required: false) the compiler
tiers of the repositories the operator lists in LOOM_CODE_REPOS_JSON run as plain
processes in the pod, and a repository added from the dashboard is refused. List only
repositories you trust. LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY and
LOOM_CODE_SELF_SERVICE_GIT_UNSANDBOXED in env let platform admins add repositories
with code.repo.add in such pods: their clone and fetches run as plain processes too,
and they get no compiler tier
(without the sandbox). With
sandbox.required: true a pod is not ready until the sandbox works, so set it only once
the nodes are prepared:
-
Run Loom on nodes whose OS allows unprivileged user namespaces.
-
Install
dev/seccomp/loom-repo-sandbox.jsonunder the kubelet’s seccomp directory (/var/lib/kubelet/seccomp/) and loaddeploy/apparmor/loom-repo-sandboxon those nodes, for example with a DaemonSet or the node image. -
Point the container at them:
sandbox: required: true hostUsers: false securityContext: seccompProfile: { type: Localhost, localhostProfile: loom-repo-sandbox.json } appArmorProfile: { type: Localhost, localhostProfile: loom-repo-sandbox } procMount: Unmasked
procMount: Unmasked needs a user namespace (sandbox.hostUsers: false, Kubernetes 1.33
or later, where ProcMountType and UserNamespacesSupport are on), and the namespace’s
Pod Security level must allow it (baseline and restricted do not). Run
dev/sandbox_probe.sh in a pod to check the result before setting sandbox.required: true.
Instance metadata service
By default, and with networkPolicy.enabled, a NetworkPolicy (enforced only where the
cluster runs a policy engine) keeps the Loom pods off networkPolicy.egressExcept (the
instance metadata service), so repository code a pod runs cannot take the node’s cloud
role. With the policy off, networkPolicy.guardMetadata renders an egress-only
NetworkPolicy whose one rule allows every pod and every address but those (none with
egressExcept: []). The policy’s effects:
- A pod has no cloud credentials from the metadata service: not the node’s instance
role, and not GKE Workload Identity on Dataplane V2. IRSA, and EKS Pod Identity on an
engine that matches node addresses by range, work. With the policy off, set
guardMetadata: falseto keep metadata access; withnetworkPolicy.enabled, open it throughextraEgress. egressExceptapplies with the policy off too: a range listed there (a VPC range, say) cuts the pods off every address in it but other pods, and an entry that is not a CIDR fails the upgrade.- A metadata proxy that redirects 169.254.169.254 to another address is reached at
that address, which the engine may check instead: add the address of one that passes
node credentials through to
egressExcept(outside the cluster’s pods, which stay reachable). - A service in the cluster that fetches URLs for its callers (a forward proxy, a
ClickHouse whose
url()reaches any host, such as the test-only dev ClickHouse) reachesegressExceptfor them: deny those addresses in its own policy. - NetworkPolicies add up: where your own policies limit the pods’ egress, this one
lets them reach everything else. Set
guardMetadata: falseand deny the metadata addresses in your own policy. Set it too when your own policy is named after the release: Helm refuses to take over an object it does not own, so the upgrade fails. - An engine that keeps address ranges off node IPs (Cilium’s default) blocks host-network
endpoints and the nodes’ own addresses: NodeLocal DNSCache (so DNS), the EKS Pod
Identity agent, where the engine
gives it an identity of its own. Turn on the engine’s node matching, or set
guardMetadata: falseand allow them in your own policy.
A helm upgrade --reuse-values from a chart with a shorter egressExcept keeps that
list: set networkPolicy.egressExcept again, or upgrade with --reset-then-reuse-values
(Helm 3.14 or later).
TLS on the Loom port
tls.secretName names a kubernetes.io/tls Secret in the release’s namespace. Every web
and workers pod mounts its tls.crt and tls.key (mode 0440, group
podSecurityContext.fsGroup), serves them on port 7777 with HTTPS only and probes over
HTTPS; the Service marks the port appProtocol: https. tls.minVersion sets
LOOM_TLS_MIN_VERSION. The chart refuses the LOOM_TLS_* settings anywhere else: the
mount and the HTTPS probes come with tls.secretName.
A controller that renews the Secret in place needs no restart. The kubelet updates the mount within its sync period, and each web process serves the new pair from its next handshake (TLS on the Loom port). The Secret must exist before the pods start.
The Ingress or proxy in front then connects over HTTPS. On EKS, add
alb.ingress.kubernetes.io/backend-protocol: HTTPS to ingress.annotations: the chart
refuses an alb Ingress without it, or with another backend-protocol or
healthcheck-protocol in ingress.annotations or service.annotations (the controller
reads both, the Service’s first). With ingress-nginx, set
nginx.ingress.kubernetes.io/backend-protocol: HTTPS; other controllers have their own
setting, which the chart does not check. Setting or
clearing tls.secretName on a running release breaks requests until the rollout ends:
the balancer and in-cluster clients switch scheme for every pod at once, while the pods
restart one at a time. Plan a window, or install a second release with TLS and move the
clients to it. A client in the cluster that calls the Service by URL, such as
deploy/okf-sync/cronjob.yaml (PRIVATE_URL), switches to https:// and trusts the CA
that issued the certificate. For the loom CLI, SSL_CERT_FILE names that CA and
replaces the system roots, so set it on the command that calls the Service only, or point
it at a bundle of the system roots and the CA. The assistant calls the port
over plain HTTP, so the chart refuses assistant.enabled with tls.secretName. The hop
from Loom to the release’s extractor (extractor.enabled) stays plain HTTP. Keep
tls.minVersion at "1.2" unless every client of the port, the balancer included,
speaks TLS 1.3. helm test calls the Service over HTTPS without verifying the
certificate.
Separate workers
workers.enabled: true moves the background workers to their own Deployment
(workers.replicaCount, workers.resources). Its pods take no traffic and elect one
leader among themselves. The web pods run no workers. Use it when the workers need
more memory than a web pod, or to scale them apart from it. With replicaCount: 2
the second pod is a standby with warm mirrors, and a PodDisruptionBudget keeps one of
the two through a drain.
Document extractor
extractor.enabled: true runs the document extractor service (document
extraction) as a Deployment and a Service of
its own, extractor.replicaCount pods (two by default), each running one extraction at
a time. The Loom pods get LOOM_DOCUMENT_EXTRACTOR_URL pointing at it, unless env or
extraEnv sets one, and both sides read the bearer token from the release Secret’s
LOOM_DOCUMENT_EXTRACTOR_TOKEN key: secrets.extractorToken (16 characters or more;
openssl rand -hex 24), or that key in secrets.existingSecret. The extractor pods
read that one key and nothing else of the Secret, mount no service-account token, run as
uid 10001 on a read-only root filesystem with every capability dropped, and carry labels
of their own, so no policy or Service that selects the Loom pods selects them. Their
NetworkPolicy, rendered whatever networkPolicy.enabled says, admits the release’s Loom
pods on 8790 and allows no egress; either form of the Loom pods’ policy allows egress to
them. extractor.env takes the service’s LOOM_EXTRACTOR_* limits
(extractor), and extractor.resources bounds each pod.
Each side holds one token, so changing secrets.extractorToken restarts the Loom
Deployments (web, and workers when enabled) and the extractor’s, and while they roll, a
Loom pod and an extractor pod may hold different tokens: extraction answers unavailable
until every roll finishes.
Assistant
assistant.enabled: true runs the native assistant in each Loom web process. Configure
its provider and model through env or settingsFile, and provider keys through the
release Secret. It needs no separate Deployment, database, Service, or ingress peer.
The existing model egress policy applies. See assistant setup for
verification and the retained-volume cutover procedure. TLS on Loom’s port is supported:
internal tools run through the authenticated in-process MCP transport.
Analytics export
analyticsExport.enabled: true adds a daily CronJob that writes the previous UTC day’s
usage and a namespace snapshot to an S3 bucket as Parquet, under a ServiceAccount of its
own (exporting usage to a data warehouse).
Alert rules
alerts.prometheusRule.enabled: true adds a PrometheusRule (the Prometheus Operator’s
CRD, which the chart does not install) for a deployment that exports Loom’s metrics over
OTLP into Prometheus: LoomCodeTierFailing for a code-index tier the code_tier_failing
health problem reports, and LoomCodeTierFailingFarBehind for a tier the problem
grades error. Their thresholds are the LOOM_CODE_TIER_FAILING_* settings
(when a tier fails).
alerts.prometheusRule.labels adds the labels your Prometheus selects rules by.
Checks
helm test loom -n loom # live, ready and /v1/health through the Service
kubectl -n loom exec deploy/loom -- loom doctor
kubectl -n loom logs -l app.kubernetes.io/instance=loom --tail=-1 | grep -E "WorkerSupervisor: starting|supervisor lease: acquired"
Sign-in
SSO is configured as on any host (single sign-on): the issuer and client id
in env, the client secret in the Secret.
env:
LOOM_OAUTH_LOGIN_PROVIDER: oidc
LOOM_OAUTH_ISSUER: https://loom.example.com # the ingress host; the callback is built from it
LOOM_OIDC_ISSUER: https://your-org.okta.com/oauth2/default
LOOM_OIDC_CLIENT_ID: your-client-id
secrets:
extra:
LOOM_OIDC_CLIENT_SECRET: your-client-secret # or a key of secrets.existingSecret
Register <LOOM_OAUTH_ISSUER>/oauth/callback with the identity provider. Every
replica must run the same provider: set it in one place, and roll it out like any
other change; a sign-in whose steps land on two replicas that disagree is started
again. Run loom sso adopt (see the SSO page) from one pod, after the rollout:
kubectl -n loom exec deploy/loom -- loom sso adopt --domain example.com
Model calls through an inference gateway
To send Loom’s own model calls through a gateway that takes a workload token instead of a provider key (models and keys), mount a projected service-account token and point Loom at it:
env:
LOOM_OPENAI_BASE_URL: https://gateway.example/v1
LOOM_ANTHROPIC_BASE_URL: https://gateway.example
LOOM_INFERENCE_TOKEN_URL: https://gateway.example/oauth/token
LOOM_INFERENCE_WORKLOAD_TOKEN_FILE: /var/run/secrets/inference/token
extraVolumes:
- name: inference-token
projected:
sources:
- serviceAccountToken:
audience: gateway.example # what the gateway expects
expirationSeconds: 3600
path: token
extraVolumeMounts:
- name: inference-token
mountPath: /var/run/secrets/inference
readOnly: true
The kubelet renews the file before it expires and Loom rereads it on each exchange,
so nothing restarts. The gateway trusts the cluster’s OIDC issuer for this service
account (serviceAccount.name); that trust is set up on the gateway’s side. The
NetworkPolicy’s egress to port 443 already covers the gateway and the identity
provider.
A gateway that verifies the projected token itself takes it as the bearer: drop the token URL and set the bearer mode. Loom rereads the file at most once a minute, and at once after a 401. To name the agent each call is for, add identity headers (models and keys):
env:
LOOM_INFERENCE_WORKLOAD_TOKEN_MODE: bearer
LOOM_INFERENCE_WORKLOAD_TOKEN_FILE: /var/run/secrets/inference/token
LOOM_INFERENCE_GATEWAY_HOSTS: gateway.example
LOOM_INFERENCE_IDENTITY_HEADERS_JSON: '{"X-Agent": "{agent}"}'
Git through a forward proxy
To send the code index’s network git through a forward proxy (git through a forward proxy), give every Loom pod the proxy’s address, a token file for its password and, for a proxy that terminates TLS under its own authority, the CA bundle:
env:
LOOM_CODE_GIT_PROXY_URL: http://git-proxy.example:3128
LOOM_CODE_GIT_PROXY_TOKEN_FILE: /var/run/secrets/git-proxy/token
LOOM_CODE_GIT_PROXY_CA_FILE: /etc/git-proxy/ca.crt
LOOM_CODE_GIT_PROXY_IDENTITY: loom
extraVolumes:
- name: git-proxy-token
projected:
sources:
- serviceAccountToken:
audience: git-proxy.example # what the proxy expects
expirationSeconds: 21600 # well above the longest git job
path: token
- name: git-proxy-ca
configMap:
name: git-proxy-ca
extraVolumeMounts:
- name: git-proxy-token
mountPath: /var/run/secrets/git-proxy
readOnly: true
- name: git-proxy-ca
mountPath: /etc/git-proxy
readOnly: true
env and the two mounts reach the web pods and the workers alike: both fetch mirrors.
The kubelet renews the token file and Loom reads it again for every git job, so nothing
restarts. A job presents the token it read at its start for as long as it runs (a fetch
up to LOOM_CODE_GIT_FETCH_TIMEOUT_S, an hour by default), so the proxy must accept a
token for that long after Loom read it. With networkPolicy.enabled, a proxy on a port
other than 443 needs a networkPolicy.extraEgress rule.
An http:// proxy URL, as above, sends the identity and the token to the proxy in clear
on every job: keep that path inside a network you trust, or give the proxy an https://
URL. Git then verifies the proxy’s certificate before it sends them, against its default
roots. For a certificate under an authority of your own, mount that authority’s bundle
and name it:
env:
LOOM_CODE_GIT_PROXY_URL: https://git-proxy.example:8443
LOOM_CODE_GIT_PROXY_TLS_CA_FILE: /etc/git-proxy/proxy-ca.crt
proxy-ca.crt is one more key of the git-proxy-ca config map here. It is not
LOOM_CODE_GIT_PROXY_CA_FILE, which verifies the remotes behind a proxy that terminates
their TLS: a proxy that does both under one authority gets the same file in both
settings. The proxy’s certificate must name git-proxy.example, the host the URL
spells.
The proxy and the repository-job sandbox are not used
together. sandbox.required: false alone does not keep the sandbox off: a pod whose
profiles let bubblewrap run uses it whatever that value says, and then every network git
job is refused. Where the proxy is set, leave sandbox.securityContext and
sandbox.hostUsers at their defaults as well. Repository jobs are then plain processes:
a compiler tier’s build runs as Loom’s user and can read the token file, so run compiler
tiers there only for repositories you trust. To add repositories with code.repo.add in
such pods, set LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY and
LOOM_CODE_SELF_SERVICE_GIT_UNSANDBOXED as well
(repository-job sandbox): their git goes through the proxy
under the identity the add names, else the default one.