Operating Streaming Hub

This page is for operators running Streaming Hub in their own infrastructure (BYOC). It covers deployment, the configuration that matters, the health and shutdown contract, and how to observe the service.

​

Deploying with Helm

---

Streaming Hub ships as a dedicated Helm chart, streaming-hub-helm, separate from any other Lerian product chart. The chart runs the hub in one of two shapes:

• all — a single deployment that runs every background worker. This is the default and the simplest to operate.
• split — separate ingest and delivery deployments that scale independently: ingest replicas share one Kafka consumer group, while delivery replicas work delivery jobs from Postgres.

The split shape is driven per process by STREAMING_HUB_ROLE (all | ingest | delivery). The role gates which background workers run and which Kafka clients dial — it does not gate which HTTP routes mount. Every role serves the full control-plane API and, crucially, the /readyz endpoint your orchestrator and metrics scrape depend on. There is one image and one binary; the role is a deployment input, not a build.

​

Running database migrations

---

Streaming Hub is backed by a single hub-owned PostgreSQL database, and it never migrates itself. Schema migrations run out of band — as a separate migration step (for example, an ArgoCD PreSync hook) that applies the versioned migrations before the hub starts. At boot the hub only verifies that the schema version it expects is present; it never runs a migration as a side effect of starting. The hub does provision its own weekly table partitions ahead of time as a routine background task — that is internal housekeeping, not a schema migration, and needs no operator action beyond leaving the partition cron running.

​

Health and graceful shutdown

---

Streaming Hub exposes two distinct probe endpoints. Wire each to the matching Kubernetes probe: /readyz distinguishes two failure classes. A runtime-probe failure — Postgres unreachable, consumer dead — takes the replica Down and out of rotation. A degrader — elevated latency, consumer lag, a thin partition buffer — clamps the replica to Degraded but keeps it serving, because an impaired replica should not refuse traffic. The probe set is role-aware: a delivery-role pod is not marked unready for having no ingest consumer. On SIGTERM the hub drains gracefully. It flips /readyz to NotReady first — before it stops serving — and waits a bounded pre-stop window so the orchestrator can pull the pod from the service before connections are cut. /healthz stays 200 throughout, so the pod is not killed mid-drain. It then tears down in dependency-safe order (HTTP, then consumer, then dispatcher, then the background apps, then Kafka clients, then the pool, then telemetry). /version (build identity) and /runtime (a cheap Go-runtime snapshot) round out the unauthenticated operational surface for incident triage.

​

Essential configuration

---

Streaming Hub reads its configuration from STREAMING_HUB_* environment variables (plus a few shared PLUGIN_AUTH_* and OTEL_* variables). The full inventory, with every default, lives in the service’s environment reference. The variables you set most often: Secret values never belong in these variables in production. The KEK is referenced by the name of the env var that the deploy layer injects it into (STREAMING_HUB_KEK_REF); the hub reads the material from that named variable and never logs it. SASL, TLS CA, and tenant-manager credentials follow the same rule — the variable holds the value at runtime, but the value comes from your secret store, not a committed config file.

​

DLQ forensics

---

GET /admin/dlq is the operator forensics surface for dead-letter observations. It is cross-tenant by design: it is gated by the lib-auth admin scope, carries no tenant shim, and returns records across all tenants, so it is not part of the customer-facing /v1 API. The dead-letter observations it reads are observability only — they are captured from upstream producers’ dead-letter topics and are never re-delivered by the hub. Use it to investigate why records failed upstream; it does not replay them.

​

Topic reconciler

---

The topic reconciler is a read-only drift detector, enabled by default (STREAMING_HUB_RECONCILER_ENABLED). On each pass it compares the live broker topics, the event catalog, and the distinct subscription targets, and flags three kinds of drift: ghost topics (a followed topic with no catalog entry), dead subscriptions (a subscribed event type and major with no live catalog entry), and lag-versus-retention breaches. It detects, never corrects — it emits count-only gauges and structured logs, and writes no broker or database state. When disabled, it spawns no goroutine and dials no admin client, so the disabled path costs nothing; disabling loses an operational alarm but never affects delivery.

​

Observability

---

Streaming Hub exports its metrics, traces, and logs through OTLP (lib-observability), pointed at the collector in OTEL_EXPORTER_OTLP_ENDPOINT. That is where the real streaming_hub_* metrics live. Tenant identity is never a metric label — it lives on span attributes and log fields — so the metric cardinality stays bounded no matter how many tenants a deployment serves.

​

Next steps

---