Fetcher architecture

Fetcher ships as two services. The Manager exposes the HTTP API and dispatches work. The Worker consumes that work and produces results. Both services are hosts. Neither one contains the extraction logic. They run the same Engine in process, through the same ports, under the same rules. That single fact explains the rest of this page, and it explains why your own application can run Fetcher extraction without either service.

​

The two services

---

​

Manager

The Manager follows a hexagonal layout. HTTP adapters map requests onto services, and services reach infrastructure only through ports. Read paths and write paths live in separate service packages. It serves twelve operations across two path prefixes: Three infrastructure dependencies back the API:

• MongoDB holds connection records and job records.
• RabbitMQ carries a new job to the Worker on the queue that RABBITMQ_FETCHER_WORK_QUEUE names.
• Valkey or Redis backs the schema cache and the rate limiter on the connection test. The schema cache falls back to process memory when Redis is unreachable.

A create-job request answers 202 Accepted. A repeat of the same request inside a five-minute window answers 200 OK with the job that already exists.

​

Worker

The Worker has no primary HTTP server. It consumes the work queue and runs RABBITMQ_NUMBERS_OF_WORKERS jobs in parallel, five by default. A small health server on HEALTH_PORT carries the probes and the metrics endpoint. For each job the Worker:

1. Resolves the connections the job maps and runs the extraction through the embedded Engine.
2. Signs the plaintext JSON with HMAC-SHA256, using a key derived from the master key.
3. Encrypts the signed payload with AES-GCM and writes it to S3-compatible object storage.
4. Publishes job.completed or job.failed to the exchange that RABBITMQ_JOB_EVENTS_EXCHANGE names.

The Worker drives the Engine in direct mode. The Engine hands back the exact bytes, and the Worker owns the protection and the storage of those bytes. Both terminal events are mandatory routes. The Worker refuses to start when STREAMING_ENABLED is false or the events exchange is blank, because a silent no-op emitter would swallow the contract.

​

Messages between them

Every message the Manager publishes carries an HMAC-SHA256 signature. The signed payload binds the timestamp, the signature version, the tenant ID, the job ID, the exchange, and the routing key. The body comes last. A replay of the same body under a different tenant or a different route fails verification. The signer refuses any key shorter than 32 bytes and compares signatures in constant time.

​

The Engine underneath

---

The Engine owns what an extraction means. A host owns how it runs. The Engine is a separate Go module: github.com/LerianStudio/fetcher/pkg/engine. Its go.mod carries no require block at all, and a CI job fails the build if one appears. A second build-enforced test walks every transitive dependency and rejects anything that is neither the Go standard library nor engine-module-local. No HTTP framework, no queue client, no database driver, and no cloud SDK can enter. That constraint is the point. The Engine adds no third-party dependency, so a host application can embed it without a single new entry in its own dependency tree.

​

Ports

A host wires the Engine through a small set of ports. One is always mandatory. A second is mandatory only when encrypted persistence is on. The rest are optional. A port supplied as a typed nil counts as absent. The Engine detects that at construction, so a misconfigured host fails at startup instead of panicking on the first call. The module also ships in-memory implementations of the storage-facing ports — the connector registry, the connection store, the schema cache, the result sink, and the execution store. That covers the infrastructure a test needs, so you can exercise the Engine with no MongoDB, no Redis, no RabbitMQ, and no object storage. It ships no CredentialProtector, so a test that turns encrypted persistence on has to supply its own.

​

Failure classification

The Engine returns one of eleven stable categories: validation, not_found, unauthorized, forbidden, limit_exceeded, conflict, unavailable, connect, timeout, canceled, and internal. A host maps each category onto its own transport. The Manager maps conflict to HTTP 409, for example. Errors that cross the Engine boundary carry fixed text. The Engine discards the driver error behind a failure, because that text can embed a DSN, a credential, or driver internals.

​

Health and readiness

---

Both services mount /health, /readyz, /readyz/tenant/{id}, and /metrics before authentication, so Kubernetes and load balancers reach them without a token.

• /health answers 503 until the startup self-probe succeeds. The kubelet then restarts a pod that could not reach its dependencies at boot.
• /readyz runs every dependency probe in parallel on each request, with no cache and a per-dependency timeout. The Manager probes MongoDB, RabbitMQ, and Redis. The Worker probes MongoDB, RabbitMQ, and the storage bucket. In multi-tenant mode both services add the Tenant Manager and the multi-tenant Redis.
• On SIGTERM both services answer /readyz with 503 for READYZ_DRAIN_DELAY_SEC seconds, 12 by default, before they tear down connections. Kubernetes removes the pod from the Service while it still serves traffic.

​

Embedding instead of deploying

---

The Manager and the Worker are hosts, not privileged layers. Provide a connector registry and a connection store, and your application gets the same query planning, the same limits, the same tenant checks, and the same result shape in process. It needs no queue, no object storage, and no separate deployment. Lerian products embed the Engine exactly this way. Read Core concepts for the model the Engine implements.

​

Next steps

---