> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetcher architecture
> How Fetcher runs: a Manager service, a Worker service, and the dependency-free extraction Engine that both of them host in process.
Fetcher ships as two services. The **Manager** exposes the HTTP API and dispatches work. The **Worker** consumes that work and produces results.
Both services are **hosts**. Neither one contains the extraction logic. They run the same **Engine** in process, through the same ports, under the same rules. That single fact explains the rest of this page, and it explains why your own application can run Fetcher extraction without either service.
## The two services
***
| Service | Role |
| ----------- | ------------------------------------------------------------------------------------------------------------------ |
| **Manager** | HTTP API for connections and jobs. Keeps metadata in MongoDB. Publishes extraction jobs to RabbitMQ. |
| **Worker** | Queue consumer. Runs the extraction, protects the result, writes it to object storage, and emits a terminal event. |
### Manager
The Manager follows a hexagonal layout. HTTP adapters map requests onto services, and services reach infrastructure only through ports. Read paths and write paths live in separate service packages.
It serves twelve operations across two path prefixes:
| Path | Operations |
| ------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `/v1/fetcher` | Create an extraction job. Get a job by identifier. |
| `/v1/management/connections` | List and create connections. Validate a schema mapping. |
| `/v1/management/connections/{id}` | Get, update, delete, test, and discover the schema of one connection. |
| `/v1/management/connections/unassigned` and `.../{id}/assign` | Two migration operations for connections that predate product scoping. |
Three infrastructure dependencies back the API:
* **MongoDB** holds connection records and job records.
* **RabbitMQ** carries a new job to the Worker on the queue that `RABBITMQ_FETCHER_WORK_QUEUE` names.
* **Valkey or Redis** backs the schema cache and the rate limiter on the connection test. The schema cache falls back to process memory when Redis is unreachable.
A create-job request answers `202 Accepted`. A repeat of the same request inside a five-minute window answers `200 OK` with the job that already exists.
### Worker
The Worker has no primary HTTP server. It consumes the work queue and runs `RABBITMQ_NUMBERS_OF_WORKERS` jobs in parallel, five by default. A small health server on `HEALTH_PORT` carries the probes and the metrics endpoint.
For each job the Worker:
1. Resolves the connections the job maps and runs the extraction through the embedded Engine.
2. Signs the plaintext JSON with HMAC-SHA256, using a key derived from the master key.
3. Encrypts the signed payload with AES-GCM and writes it to S3-compatible object storage.
4. Publishes `job.completed` or `job.failed` to the exchange that `RABBITMQ_JOB_EVENTS_EXCHANGE` names.
The Worker drives the Engine in direct mode. The Engine hands back the exact bytes, and the Worker owns the protection and the storage of those bytes. Both terminal events are mandatory routes. The Worker refuses to start when `STREAMING_ENABLED` is false or the events exchange is blank, because a silent no-op emitter would swallow the contract.
The Worker speaks the S3 protocol. SeaweedFS works as a storage target through its S3-compatible gateway, which is how the local Docker Compose stack runs. Set retention on results with a lifecycle policy on the bucket.
### Messages between them
Every message the Manager publishes carries an HMAC-SHA256 signature. The signed payload binds the timestamp, the signature version, the tenant ID, the job ID, the exchange, and the routing key. The body comes last. A replay of the same body under a different tenant or a different route fails verification. The signer refuses any key shorter than 32 bytes and compares signatures in constant time.
## The Engine underneath
***
The Engine owns what an extraction **means**. A host owns how it **runs**.
| The Engine decides | The host provides |
| ------------------------------- | ---------------------------------------------- |
| Connection lifecycle rules | Where connections live |
| Schema discovery and validation | The driver that reads the catalog |
| Query planning and execution | The database connectors |
| Resource limits and timeouts | The transport that surfaces them |
| Tenant scope on every operation | The tenant identity from the request |
| Error categories | The HTTP status or queue behavior per category |
The Engine is a separate Go module: `github.com/LerianStudio/fetcher/pkg/engine`. Its `go.mod` carries no `require` block at all, and a CI job fails the build if one appears. A second build-enforced test walks every transitive dependency and rejects anything that is neither the Go standard library nor engine-module-local. No HTTP framework, no queue client, no database driver, and no cloud SDK can enter.
That constraint is the point. The Engine adds no third-party dependency, so a host application can embed it without a single new entry in its own dependency tree.
### Ports
A host wires the Engine through a small set of ports. One is always mandatory. A second is mandatory only when encrypted persistence is on. The rest are optional.
| Port | Required | Without it |
| ------------------------ | ----------- | ---------------------------------------------------------------------------------------------------------------------- |
| `ConnectorRegistry` | Yes | The Engine refuses to construct. |
| `CredentialProtector` | Conditional | Required when encrypted persistence is on. The Engine refuses to construct rather than store a plaintext credential. |
| `ConnectionStore` | No | The Engine constructs, but every operation that touches a connection fails. In practice it can only report its limits. |
| `SchemaCache` | No | Every schema call discovers live from the datasource. |
| `ResultSink` | No | Store mode is unavailable. Extraction returns the rows inline. |
| `ExecutionStore` | No | The Engine tracks no durable execution state. The host owns status. |
| `ActiveExecutionChecker` | No | The Engine applies no conflict gate. Connection updates and deletes proceed. |
| `Observability` | No | Span hooks disappear. Behavior does not change. |
A port supplied as a typed nil counts as absent. The Engine detects that at construction, so a misconfigured host fails at startup instead of panicking on the first call.
The module also ships in-memory implementations of the storage-facing ports — the connector registry, the connection store, the schema cache, the result sink, and the execution store. That covers the infrastructure a test needs, so you can exercise the Engine with no MongoDB, no Redis, no RabbitMQ, and no object storage. It ships no `CredentialProtector`, so a test that turns encrypted persistence on has to supply its own.
### Failure classification
The Engine returns one of eleven stable categories: `validation`, `not_found`, `unauthorized`, `forbidden`, `limit_exceeded`, `conflict`, `unavailable`, `connect`, `timeout`, `canceled`, and `internal`. A host maps each category onto its own transport. The Manager maps `conflict` to HTTP 409, for example.
Errors that cross the Engine boundary carry fixed text. The Engine discards the driver error behind a failure, because that text can embed a DSN, a credential, or driver internals.
## Health and readiness
***
Both services mount `/health`, `/readyz`, `/readyz/tenant/{id}`, and `/metrics` before authentication, so Kubernetes and load balancers reach them without a token.
* `/health` answers 503 until the startup self-probe succeeds. The kubelet then restarts a pod that could not reach its dependencies at boot.
* `/readyz` runs every dependency probe in parallel on each request, with no cache and a per-dependency timeout. The Manager probes MongoDB, RabbitMQ, and Redis. The Worker probes MongoDB, RabbitMQ, and the storage bucket. In multi-tenant mode both services add the Tenant Manager and the multi-tenant Redis.
* On `SIGTERM` both services answer `/readyz` with 503 for `READYZ_DRAIN_DELAY_SEC` seconds, 12 by default, before they tear down connections. Kubernetes removes the pod from the Service while it still serves traffic.
## Embedding instead of deploying
***
The Manager and the Worker are hosts, not privileged layers. Provide a connector registry and a connection store, and your application gets the same query planning, the same limits, the same tenant checks, and the same result shape in process. It needs no queue, no object storage, and no separate deployment.
Lerian products embed the Engine exactly this way. Read [Core concepts](/en/fetcher/fetcher-core-concepts) for the model the Engine implements.
## Next steps
***
Register, test, update, and delete a datasource connection.
The environment variables that shape a Fetcher deployment.