Skip to main content
An event moves through Streaming Hub in three stages: the hub ingests it from the internal stream, matches it against your subscriptions, and dispatches it to each matched destination. This page walks through each stage and the delivery guarantees that fall out of it.

​
Ingesting events from the stream

Streaming Hub consumes the platform stream as CloudEvents 1.0 messages in binary content mode: the CloudEvents context attributes travel in Kafka record headers (each prefixed ce-) and the event body is the record value. The hub reads routing and identity from the headers without deserializing the payload. The hub subscribes only to v3 application fact topics:
ce-source is one lowercase, dot-free segment using letters, digits, underscores, or hyphens. The topic carries no resource, event, or schema-major suffix; the hub dispatches by CloudEvents headers. Command topics (.commands) and dead-letter topics (.dlq) are outside the follow set. Midaz, Tracer, Matcher, Lender, and Consignado application fact topics now match this follow-set contract. A second consumer reads the .dlq topics that producers write. That consumer serves observability only. The follow-set grammar excludes a trailing .dlq, so a dead-letter record never becomes a delivery to one of your subscriptions. Operators read those records through GET /admin/dlq — see DLQ forensics. Every record the hub accepts must carry this context set: For each record the hub returns one of three verdicts: The tenant admitted at ingest comes only from the validated ce-tenantid attribute the parser extracted. Ingest has no tenant-roster drop gate. Stored events match only subscriptions with the same tenant id.
Three related strings look alike but are distinct — never conflate them:
  • ce-type — the source-qualified CloudEvents type: studio.lerian.<source>.<resource>.<event> (for example, studio.lerian.lender.loan_application.approved).
  • Kafka topic — one v3 fact topic per application: lerian.streaming.<source> (for example, lerian.streaming.lender). Resource, event, and schema major do not appear in the topic.
  • X-Lerian-Event-Type — the bare <resource>.<event> key stamped on a delivered webhook (loan_application.approved). It intentionally omits the producer. Use a subscription origin pin when the same key can come from more than one application.
See the event streaming overview for the full wire contract.

​
Consume-once deduplication

The internal stream is at-least-once, so the same record can arrive more than once. On the way into the event inbox the hub deduplicates on ce-id: a duplicate id writes nothing and is counted as a dedup drop. Records are persisted per Kafka partition in one transaction, and the stream position for a partition is committed only after that partition’s write commits — so a failure on one partition never loses or double-commits events on another.

​
Subscription matching

Once an event is stored, the hub resolves which of your subscriptions should receive it. Subscription matching is internal and catalog-free: it evaluates each event directly on (tenant, origin, event type, schema major). The origin dimension is optional on the subscription; when absent, the same event key from any producing application can match. An event no subscription wants produces zero deliveries — a correct outcome, not an error. A subscription is admitted by matching only when it is both enabled and in verification_state = active. These are two independent conditions — see the subscription model — and matching requires both.

​
Dispatching to your destinations

For every match, the hub creates a delivery job and hands it to a worker pool (eight workers by default). Workers claim due jobs atomically with a short lease and interleave across tenants, so no single tenant’s backlog starves the others. Each worker loads the stored payload, decrypts the destination’s signing secret or credential in memory, and makes exactly one delivery attempt to the sink. Delivery is at-least-once: the hub may deliver the same event more than once (through retries or redeliveries after a worker restart). Every delivery carries a stable X-Lerian-Event-Id (the ce-id) for you to deduplicate on, and a per-attempt X-Lerian-Delivery-Id that changes on every retry. Treat a repeated X-Lerian-Event-Id as a duplicate and acknowledge it without reprocessing. See Consuming events for the consumer side of this contract.

​
Retries and back-off

When an attempt fails, the hub schedules the next one on a fixed back-off curve with full jitter:
The curve spans roughly 28 hours across its eight steps. Each subscription can override it with its own schedule (up to 12 steps, each capped at 10 hours); a malformed override falls back to the default curve rather than failing delivery.

​
Dead-lettering

An event whose retries are exhausted is dead-lettered — the hub stops attempting it and records the terminal outcome. A job that instead keeps crashing a worker mid-attempt — never recording an outcome — is reclaimed a bounded number of times (five by default) and then dead-lettered as poison, so a single toxic job can never occupy a worker forever.

​
Circuit breaker

Each (tenant, destination) pair has a circuit breaker. When a destination fails repeatedly, its breaker opens and further jobs for that destination are shed — rescheduled without an attempt — so a single broken endpoint does not burn worker capacity or hammer a struggling target. The breaker recovers on its own once the destination starts succeeding again.

​
Auto-disabling a broken destination

A destination that stays broken for a long time is automatically disabled. Auto-disable trips only when the open failure span is both:
  • sustained — it has been failing for at least the failure window (120 hours by default); and
  • spread — at least 12 hours separate the first and last failure in the span.
Both gates must hold, and a single successful delivery clears the span, so a brief blip never accumulates toward the verdict. When auto-disable trips it flips the subscription’s enabled flag to false — it never touches verification_state. Delivery stops (matching requires enabled), and the hub records why. You recover an auto-disabled subscription with POST /v1/subscriptions/:id/verify, which re-probes the destination and, on success, re-enables it in place. Auto-disable is governed by a kill switch (STREAMING_HUB_AUTODISABLE_ENABLED), so an operator can ship it dark. See recovering an auto-disabled subscription.

​
Ordering guarantees

The internal stream is partitioned by tenant, so events for one tenant normally ride a single partition and the hub preserves their first-in, first-out order end to end. The hub attributes and deduplicates on the CloudEvents headers, not the Kafka record key, so a producer that spreads (salts) a hot tenant across several partitions changes only physical placement — the hub still attributes and dedups correctly. The one trade-off is ordering: a salted tenant’s events span multiple partitions with no cross-partition order guarantee, so that tenant forfeits strict FIFO. The arrival sequence the hub assigns reflects the order events arrived at the hub, not the order they were produced. A tenant that is not salted keeps single-partition FIFO throughout.

​
Next steps

Managing subscriptions

The subscription model, onboarding flows, and secret rotation.

Consuming events

Verify signatures, deduplicate, and pull events.