Operating Streaming Hub

This page is for operators who run Streaming Hub in their own infrastructure (BYOC). It covers deployment, the configuration that matters, the health and shutdown contract, and how to observe the service.

Deploying with Helm

Streaming Hub ships as a dedicated Helm chart, streaming-hub-helm, separate from any other Lerian product chart. The chart runs the hub in one of two shapes:

all: a single deployment that runs every background worker. This is the default and the simplest to operate.
split: separate ingest and delivery deployments that scale independently: ingest replicas share one Kafka consumer group, while delivery replicas work delivery jobs from Postgres.

The split shape is driven per process by STREAMING_HUB_ROLE (all | ingest | delivery). The role gates which background workers run and which Kafka clients dial. It does not gate which HTTP routes mount. Every role serves the full control-plane API and the /readyz endpoint your orchestrator depends on. There is one image and one binary.

Running database migrations

Streaming Hub uses a single hub-owned PostgreSQL database, version 15 or later, and it never migrates itself. Schema migrations run out of band: the chart runs them in a Helm pre-install and pre-upgrade Job before the hub starts. The hub only checks that the database is at the schema version it expects. A replica whose schema does not match reports Down on /readyz. The hub creates its own weekly table partitions ahead of time as a background task. This routine task is not a schema migration and needs no operator action. Partition retention is off by default (STREAMING_HUB_PARTITION_RETENTION_ENABLED=false), so the hub keeps stored events until you turn it on. When you turn it on, STREAMING_HUB_PARTITION_RETENTION_HORIZON sets the retention window (default 2160h, 90 days). The horizon must be at least STREAMING_HUB_PARTITION_FUTURE_BUFFER_WEEKS × 7 days, or boot fails. The shortest horizon is therefore 7 days (168h with a 1-week buffer).

Health and graceful shutdown

Streaming Hub exposes two distinct probe endpoints. Wire each to the matching Kubernetes probe:

Endpoint	Probe	Behavior
`/healthz`	Liveness	Returns `200` unconditionally once the process is serving, independent of Postgres, Kafka, or application-ingest consumer health. A failing liveness restarts the pod, so it must not depend on dependencies.
`/readyz`	Readiness	Folds the full probe set for the role. `Healthy` or `Degraded` → `200` (kept in rotation); `Down` → `503` (pulled from rotation). A `Degraded` replica keeps serving.

/readyz distinguishes two failure classes. A runtime-probe failure (Postgres unreachable, consumer dead) takes the replica Down and out of rotation. A degrader (elevated latency, consumer lag, a thin partition buffer) clamps the replica to Degraded but keeps it serving. An impaired replica should not refuse traffic. The probe set is role-aware. A missing ingest consumer never makes a delivery-role pod unready. On SIGTERM the hub drains gracefully. It flips /readyz to NotReady first, before it stops serving. It then waits a bounded pre-stop window, so the orchestrator can pull the pod from the service before connection teardown. The /healthz endpoint stays 200 throughout, so the orchestrator does not kill the pod mid-drain. It then stops its components in dependency order.

Set the deployment’s terminationGracePeriodSeconds at or above the hub’s drain ceiling, not a fixed number. The ceiling is the pre-stop window, plus the shutdown timeout, plus the shutdown timeout again, plus 15 seconds of fixed margin. At the defaults (a 5-second pre-stop window and a 30-second shutdown timeout) it is 80 seconds.A grace period below the ceiling risks a SIGKILL of a still-draining replica. That is safe for correctness, but it forfeits the clean drain. The hub reclaims and redelivers in-flight jobs, and the consumer deduplicates them.

/version (build identity) and /runtime (a cheap Go-runtime snapshot) round out the unauthenticated operational surface for incident triage.

Essential configuration

Streaming Hub reads its configuration from STREAMING_HUB_* environment variables, plus shared PLUGIN_AUTH_*, RATE_LIMIT_*, and OTEL_* variables and the unprefixed ENV_NAME and ALLOW_RATELIMIT_FAIL_OPEN. The variables you set most often:

Variable	Default	Purpose
`STREAMING_HUB_ENV`	`local`	Deployment environment (`local` \| `staging` \| `production`). `staging` and `production` refuse the development bypass flags.
`STREAMING_HUB_ROLE`	`all`	Deployable slice: `all` \| `ingest` \| `delivery`. Gates workers and Kafka clients, never routes.
`STREAMING_HUB_HTTP_LISTEN_ADDR`	`:8080`	The single control-plane bind address.
`STREAMING_HUB_POSTGRES_DSN`	(required)	The hub-owned PostgreSQL DSN. Boot fails if empty.
`STREAMING_HUB_KAFKA_BROKERS`	(empty)	Bootstrap broker list for the internal stream.
`STREAMING_HUB_REDIS_ADDRESS`	(required)	`host:port` of the Redis or Valkey that counts the per-tenant rate limits. Every role needs it. Boot fails if empty. TLS is on by default (`STREAMING_HUB_REDIS_TLS`).
`RATE_LIMIT_ENABLED`	`false`	Must be `true`. Boot fails otherwise.
`ALLOW_RATELIMIT_FAIL_OPEN`	`false`	`true` serves the rate-limited routes unmetered while Redis is unreachable. `false` answers them `503`.
`STREAMING_HUB_DASHBOARD_CURSOR_HMAC_KEY`	(required)	Standard base64 of exactly 32 random bytes. It signs the dashboard page cursor. Boot fails if it is empty or malformed.
`STREAMING_HUB_TENANT_ID`	(required)	The tenant every control-plane request acts as. Boot fails if empty. See the warning below.
`STREAMING_HUB_MANIFEST_SOURCES`	(empty)	Comma-separated producer base URLs whose manifests populate the catalog in BYOC mode. Invalid URLs fail boot.
`STREAMING_HUB_MANIFEST_CLIENT_ID`	(empty)	Optional client id used to fetch BYOC producer manifests. Set it together with `STREAMING_HUB_MANIFEST_CLIENT_SECRET`; leaving both empty uses anonymous fetches.
`STREAMING_HUB_MANIFEST_CLIENT_SECRET`	(empty)	Optional client secret for manifest fetches. Setting only one manifest credential fails boot.
`STREAMING_HUB_KEK_SOURCE`	`env`	Key-encryption-key provider (`env` \| `secretsmanager`).
`STREAMING_HUB_KEK_REF`	(required in `staging` and `production`)	The name of the env var holding the KEK material — never the material itself.
`PLUGIN_AUTH_ADDRESS`	`http://localhost:4000`	The Access Manager authorization base URL for control-plane requests.
`PLUGIN_AUTH_ENABLED`	`true`	Master auth switch. `false` is a local bypass, refused in `staging` and `production`.
`STREAMING_HUB_AUTODISABLE_ENABLED`	`true`	`false` turns auto-disable off.
`STREAMING_HUB_SHUTDOWN_TIMEOUT`	`30s`	Graceful-drain window. It must be below `35s`, or boot fails.
`STREAMING_HUB_AWS_HUB_PRINCIPAL_ARN`	(empty)	The hub’s public IAM principal embedded in AWS setup artifacts. Required for AWS sinks.
`STREAMING_HUB_AWS_SETUP_TEMPLATE_URL`	(empty)	The public CloudFormation template URL for the AWS quick-create link.
`OTEL_EXPORTER_OTLP_ENDPOINT`	(empty)	The OTLP collector endpoint telemetry exports to (no `STREAMING_HUB_` prefix).

When Access Manager returns no verdict on a request, the hub answers 503 with the code service_unavailable on every authorized route, GET /admin/dlq included. Retry with backoff. Removing a URL from STREAMING_HUB_MANIFEST_SOURCES stops future refreshes from that producer but keeps its last-known-good catalog rows. Manifest refresh is control-plane discovery only. A producer outage does not block event ingest or delivery matching. Secret values never belong in committed configuration. You reference the KEK by the name of the env var that the deploy layer injects it into (STREAMING_HUB_KEK_REF). The hub reads the material from that named variable and never logs it. Deployment-level Kafka SASL and TLS material, the dashboard cursor key, and Redis credentials must come from your secret store.

STREAMING_HUB_TENANT_ID must match the ce-tenantid your producers stamp. Ingest stores each event under its ce-tenantid. Subscriptions created through /v1 belong to STREAMING_HUB_TENANT_ID. When the two differ, creates succeed and events are stored, but no producer event matches a subscription or reaches a pull read. The hub reports no error. A value other than default emits one startup warning.Access Manager is the only authorization gate: every principal it grants the Streaming Hub subscriptions permission acts on this tenant. Grant it only to this tenant’s principals.

DLQ forensics

GET /admin/dlq is the operator forensics surface for dead-letter observations. It is cross-tenant by design. Access Manager authorizes it on the admin_dlq resource with the get action, and it returns records across all tenants. It is therefore not part of the customer-facing /v1 API. The dead-letter observations it reads are observability only. They come from upstream producers’ dead-letter topics, and the hub never re-delivers them. Use it to investigate why records failed upstream. It does not replay them.

Topic reconciler

The topic reconciler is a read-only drift detector, enabled by default (STREAMING_HUB_RECONCILER_ENABLED). On each pass it compares the live broker topics, the event catalog, and the distinct subscription targets. It flags three kinds of drift:

ghost topics: a followed topic with no catalog entry.
dead subscriptions: a subscribed event type and major with no live catalog entry.
lag-versus-retention breaches.

It detects, never corrects. It emits count-only gauges and structured logs, and writes no broker or database state. Disabling it removes an operational alarm and never affects delivery.

Observability

Streaming Hub exports its metrics, traces, and logs through OTLP, pointed at the collector in OTEL_EXPORTER_OTLP_ENDPOINT. That is where the real streaming_hub_* metrics live.

The /metrics endpoint is almost empty by design: it serves only the static streaming_hub_build_info gauge. Scrape the hub’s real metrics from your OTLP collector, not from /metrics.

Tenant identity is never a metric label. It lives on span attributes and log fields, so the metric cardinality stays bounded no matter how many tenants a deployment serves.

Introduction

Access Manager

Console

Streaming Hub

Multi-tenancy

Observability

Narya

Lifecycle Management

Licensing

Operating Streaming Hub

Deploying with Helm

Running database migrations

Health and graceful shutdown

Essential configuration

DLQ forensics

Topic reconciler

Observability

Next steps

How Streaming Hub works

Managing subscriptions

​Deploying with Helm

​Running database migrations

​Health and graceful shutdown

​Essential configuration

​DLQ forensics

​Topic reconciler

​Observability

​Next steps

How Streaming Hub works

Managing subscriptions

Deploying with Helm

Running database migrations

Health and graceful shutdown

Essential configuration

DLQ forensics

Topic reconciler

Observability

Next steps