> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Operating Streaming Hub
> Deploy Streaming Hub with Helm in BYOC, run out-of-band migrations, wire liveness and readiness probes, set the essential configuration, and observe the hub through OTLP.
This page is for operators running Streaming Hub in their own infrastructure (BYOC). It covers deployment, the configuration that matters, the health and shutdown contract, and how to observe the service.
## Deploying with Helm
***
Streaming Hub ships as a dedicated Helm chart, `streaming-hub-helm`, separate from any other Lerian product chart. The chart runs the hub in one of two shapes:
* **`all`** — a single deployment that runs every background worker. This is the default and the simplest to operate.
* **`split`** — separate **ingest** and **delivery** deployments that scale independently: ingest replicas share one Kafka consumer group, while delivery replicas work delivery jobs from Postgres.
The split shape is driven per process by `STREAMING_HUB_ROLE` (`all` | `ingest` | `delivery`). The role gates **which background workers run and which Kafka clients dial** — it does **not** gate which HTTP routes mount. Every role serves the full control-plane API and, crucially, the `/readyz` endpoint your orchestrator and metrics scrape depend on. There is one image and one binary; the role is a deployment input, not a build.
## Running database migrations
***
Streaming Hub is backed by a single hub-owned PostgreSQL database, and it **never migrates itself**. Schema migrations run **out of band** — as a separate migration step (for example, an ArgoCD PreSync hook) that applies the versioned migrations before the hub starts. At boot the hub only *verifies* that the schema version it expects is present; it never runs a migration as a side effect of starting.
The hub does provision its own weekly table partitions ahead of time as a routine background task — that is internal housekeeping, not a schema migration, and needs no operator action beyond leaving the partition cron running.
## Health and graceful shutdown
***
Streaming Hub exposes two distinct probe endpoints. Wire each to the matching Kubernetes probe:
| Endpoint | Probe | Behavior |
| ---------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/healthz` | Liveness | Returns `200` unconditionally once the process is serving, independent of Postgres, Kafka, or roster health. A failing liveness restarts the pod, so it must not depend on dependencies. |
| `/readyz` | Readiness | Folds the full probe set for the role. `Healthy` or `Degraded` → `200` (kept in rotation); `Down` → `503` (pulled from rotation). A `Degraded` replica keeps serving. |
`/readyz` distinguishes two failure classes. A **runtime-probe** failure — Postgres unreachable, consumer dead — takes the replica **Down** and out of rotation. A **degrader** — elevated latency, consumer lag, a thin partition buffer — clamps the replica to **Degraded** but keeps it serving, because an impaired replica should not refuse traffic. The probe set is role-aware: a delivery-role pod is not marked unready for having no ingest consumer.
On `SIGTERM` the hub drains gracefully. It flips `/readyz` to `NotReady` **first** — before it stops serving — and waits a bounded pre-stop window so the orchestrator can pull the pod from the service before connections are cut. `/healthz` stays `200` throughout, so the pod is not killed mid-drain. It then tears down in dependency-safe order (HTTP, then consumer, then dispatcher, then the background apps, then Kafka clients, then the pool, then telemetry).
Set the deployment's `terminationGracePeriodSeconds` at or above the hub's derived drain ceiling for your `STREAMING_HUB_SHUTDOWN_TIMEOUT`, not a fixed magic number. At the default 30-second shutdown timeout the ceiling is about **80 seconds**: the 5-second pre-stop window, plus the shutdown timeout itself, plus a worst-case dispatcher drain leg of `min(timeout, 55s)`, plus a fixed teardown margin for the remaining components. A grace period below the ceiling risks a `SIGKILL` of a still-draining replica — safe for correctness (in-flight jobs are reclaimed and redelivered, deduplicated at the consumer), but it forfeits the clean drain.
`/version` (build identity) and `/runtime` (a cheap Go-runtime snapshot) round out the unauthenticated operational surface for incident triage.
## Essential configuration
***
Streaming Hub reads its configuration from `STREAMING_HUB_*` environment variables (plus a few shared `PLUGIN_AUTH_*` and `OTEL_*` variables). The full inventory, with every default, lives in the service's environment reference. The variables you set most often:
| Variable | Default | Purpose |
| -------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `STREAMING_HUB_ENV` | `local` | Deployment environment (`local` \| `staging` \| `production`). Drives the production security gate that rejects the dev bypass flags. |
| `STREAMING_HUB_ROLE` | `all` | Deployable slice: `all` \| `ingest` \| `delivery`. Gates workers and Kafka clients, never routes. |
| `STREAMING_HUB_HTTP_LISTEN_ADDR` | `:8080` | The single control-plane bind address. |
| `STREAMING_HUB_POSTGRES_DSN` | *(required)* | The hub-owned PostgreSQL DSN. Boot fails if empty. |
| `STREAMING_HUB_KAFKA_BROKERS` | *(empty)* | Bootstrap broker list for the internal stream. |
| `STREAMING_HUB_TENANT_ID` | `default` | The BYOC tenant id. See the warning below before changing it. |
| `STREAMING_HUB_KEK_SOURCE` | `env` | Key-encryption-key provider (`env` \| `secretsmanager`). |
| `STREAMING_HUB_KEK_REF` | *(empty)* | The **name** of the env var holding the KEK material — never the material itself. |
| `STREAMING_HUB_MANIFEST_SOURCES` | *(empty)* | Producer manifest base URLs the event catalog is built from. |
| `PLUGIN_AUTH_ADDRESS` | *(plugin-auth default)* | The plugin-auth decision-point base URL for control-plane authorization. |
| `PLUGIN_AUTH_ENABLED` | `true` | Master auth switch. `false` is a local bypass, **rejected in production**. |
| `STREAMING_HUB_AUTODISABLE_ENABLED` | `true` | Kill switch for auto-disabling broken destinations. |
| `STREAMING_HUB_SHUTDOWN_TIMEOUT` | `30s` | Graceful-drain window. The pre-stop window plus `min(value, 55s)` must stay strictly below the 60-second dispatcher lease, or boot fails closed. |
| `STREAMING_HUB_MULTI_TENANT_ENABLED` | `false` | `false` is BYOC single-tenant; `true` wires the SaaS multi-tenant roster. |
| `STREAMING_HUB_AWS_HUB_PRINCIPAL_ARN` | *(empty)* | The hub's public IAM principal embedded in AWS setup artifacts. Required for AWS sinks. |
| `STREAMING_HUB_AWS_SETUP_TEMPLATE_URL` | *(empty)* | The public CloudFormation template URL for the AWS quick-create link. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | *(empty)* | The OTLP collector endpoint telemetry exports to (no `STREAMING_HUB_` prefix). |
Secret **values** never belong in these variables in production. The KEK is referenced by the *name* of the env var that the deploy layer injects it into (`STREAMING_HUB_KEK_REF`); the hub reads the material from that named variable and never logs it. SASL, TLS CA, and tenant-manager credentials follow the same rule — the variable holds the value at runtime, but the value comes from your secret store, not a committed config file.
**`STREAMING_HUB_TENANT_ID` is a zero-delivery trap in BYOC.** The hub only accepts events whose `ce-tenantid` matches this value exactly; every other event is silently dropped (`unknown_or_inactive_tenant`), advancing the offset with no poison row. If you set it to anything other than `default`, you **must** confirm the producer emits that same `ce-tenantid` — otherwise the hub drops 100% of the stream and delivers nothing, with no error. A non-default value emits one startup warning; heed it.
## DLQ forensics
***
`GET /admin/dlq` is the operator forensics surface for dead-letter observations. It is **cross-tenant by design**: it is gated by the lib-auth admin scope, carries **no** tenant shim, and returns records across all tenants, so it is not part of the customer-facing `/v1` API. The dead-letter observations it reads are **observability only** — they are captured from upstream producers' dead-letter topics and are never re-delivered by the hub. Use it to investigate why records failed upstream; it does not replay them.
## Topic reconciler
***
The topic reconciler is a **read-only** drift detector, enabled by default (`STREAMING_HUB_RECONCILER_ENABLED`). On each pass it compares the live broker topics, the event catalog, and the distinct subscription targets, and flags three kinds of drift: **ghost topics** (a followed topic with no catalog entry), **dead subscriptions** (a subscribed event type and major with no live catalog entry), and **lag-versus-retention** breaches.
It **detects, never corrects** — it emits count-only gauges and structured logs, and writes no broker or database state. When disabled, it spawns no goroutine and dials no admin client, so the disabled path costs nothing; disabling loses an operational alarm but never affects delivery.
## Observability
***
Streaming Hub exports its metrics, traces, and logs through **OTLP** (lib-observability), pointed at the collector in `OTEL_EXPORTER_OTLP_ENDPOINT`. That is where the real `streaming_hub_*` metrics live.
The `/metrics` endpoint is **almost empty by design**: it serves only the static `streaming_hub_build_info` gauge. Scrape the hub's real metrics from your OTLP collector, not from `/metrics`.
Tenant identity is never a metric label — it lives on span attributes and log fields — so the metric cardinality stays bounded no matter how many tenants a deployment serves.
## Next steps
***
The delivery internals behind the operational surfaces above.
The control-plane operations your tenants use.