job.completed when the result reaches storage, and job.failed when the run stops. Subscribe to those events instead of polling GET /v1/fetcher/{id}.
This page covers the consumer side: what arrives, what it means, and what to do with it.
Where the events arrive
The Worker publishes to a durable topic exchange.
RABBITMQ_JOB_EVENTS_EXCHANGE names it, and the shipped value is fetcher.job.events.
Bind your own queue to the routing keys you care about. The local infrastructure definition binds two queues as an example:
fetcher.job.completed.queue and fetcher.job.failed.queue.
Job events are a product contract, not an optional feature. The Worker refuses to start without streaming enabled and without an exchange name. See Deployment for the operator side.
job.completed
This event means one thing: the extraction ran to the end, and the encrypted result now sits in object storage. The Worker publishes it after two earlier steps. First it writes the result object, then it records the terminal status on the job. Only then does it emit the event.
metadata carries the metadata you sent on the job, so source and any correlation identifier come back to you unchanged.
sizeBytes and rowCount describe the plaintext result, before encryption. path is the object key of the stored result.
job.failed
This event means the run stopped and Fetcher stored no result. Extraction is fail-fast, so the first datasource that fails ends the whole job. A partial result never reaches storage. The event fires for any failure along the path: an unresolvable connection name, a datasource error, a schema mismatch, or a storage write that did not complete.
result block and no completedAt.
metadata.error.message passes through a redaction step first. Fetcher replaces four leak shapes with [redacted]: connection URIs, the address operand of a Go network error, the Addr: operand of a MongoDB driver error, and an IPv4 literal. The rest of the text survives, so the message stays actionable. Route it to operators.
The CloudEvents envelope
Every message travels in CloudEvents binary mode, version 1.0. The context attributes ride as AMQP headers.
A single-tenant deployment still carries a tenant value. It emits the literal
single-tenant, so one consumer handles both deployment shapes with the same code.
CloudEvents source configuration
STREAMING_CLOUDEVENTS_SOURCE has no default value. The Worker requires it whenever streaming is on, and it stops at startup when the value is empty. The shipped example uses //lerian.fetcher/worker.
Fetcher copies the value into ce-source verbatim. Give each Worker deployment its own source value when several producers share one broker, and route on that header.
Delivery contract
Delivery is at-least-once. Deduplicate on
ce-id.
- The Worker writes the event to a durable outbox before it publishes. A broker outage delays the event, it does not lose it.
- A repairer scans for terminal events that never published and re-emits them every 30 seconds.
ce-idstays identical across every re-emission of the same job and status. Nothing else is stable enough to key on.- Order is not a guarantee. Two jobs can complete in one order and arrive in another.
- The Worker records the terminal job status before it emits.
GET /v1/fetcher/{id}remains the authority on job state.
Verifying what you receive
Two independent signatures protect the path between the Worker and you. Both use HMAC-SHA256 with the external HMAC key. HKDF-SHA256 derives that key from the
APP_ENC_KEY master key. The Fetcher repository ships a small tool that prints it, so a consumer verifies signatures without ever holding the master key.
The message. The Worker signs each published message and stamps three headers on it: x-message-signature, t for the signing timestamp, and signature-version. The signed payload binds the timestamp, the signature version, the tenant, the job identifier, the exchange, and the routing key, followed by the message body. Replaying the same body under another tenant or another route fails verification.
The result. result.hmac is the keyed HMAC-SHA256 over the plaintext result JSON, computed before encryption. The integrity block states the same value with its algorithm. Verify it after you decrypt and before you trust the rows.
The protection block describes the stored bytes: encrypted is true, the storage adapter applied it, and the mode is adapter-managed. It describes the result only, never the datasource credentials.
Next steps
Extraction jobs
What a job requests, and the states it moves through.
Fetcher REST API
Create a job, read a job, and manage connections.
Configuration
The streaming, exchange, and encryption variables behind these events.
Architecture
The Manager, the Worker, and the Engine they both run over.