> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetcher deployment
> Deploy the Fetcher Manager and Worker: MongoDB, RabbitMQ with a dead-letter queue, S3-compatible storage, Valkey or Redis, worker concurrency, autoscaling, and the startup checks that stop a misconfigured service.
A standalone Fetcher deployment is two services and four dependencies. This page covers what each dependency does, how to size the Worker, and what stops a service at startup.
If you only need extraction inside one Go application, you deploy none of this. See [Embedding the Engine](/en/fetcher/fetcher-embedding-the-engine).
## What you deploy
***
| Component | Role | Scale on |
| ------------------- | -------------------------------------------------------------- | ---------------------------- |
| **Manager** | HTTP API for connections and jobs. Publishes work to RabbitMQ. | Request rate. |
| **Worker** | Queue consumer. Runs extractions and writes results. | Queue depth. |
| **MongoDB** | Connection records and job records. | Metadata volume. |
| **RabbitMQ** | Work queue, dead-letter queue, and the job events exchange. | Job rate. |
| **Object storage** | Encrypted extraction results. S3-compatible. | Result volume and retention. |
| **Valkey or Redis** | Manager schema cache and connection-test rate limiter. | Manager only. |
## MongoDB
***
Both services connect to the same MongoDB deployment. The Manager writes connection and job records, and the Worker updates job state.
In multi-tenant mode, each tenant gets its own database, resolved from the JWT claims on the request. That path fails closed: a request that carries a tenant identity with no tenant database returns an error instead of touching the shared database.
## RabbitMQ
***
Fetcher uses three exchanges and four queues. The bundled local stack declares them for you. In your own environment, declare them before the services start.
| Object | Type | Purpose |
| ------------------------------------------------------------ | ------------------------- | ----------------------------------------------------------- |
| `fetcher.extract-external-data.queue` | queue | Extraction jobs, from Manager to Worker. |
| `fetcher.extract-external-data.exchange` | direct exchange | Bound to the work queue with routing key `fetcher.job.key`. |
| `fetcher.dlx` and `fetcher.dlq` | direct exchange and queue | Dead letters from the work queue. |
| `fetcher.job.events` | topic exchange | Terminal job events. |
| `fetcher.job.completed.queue` and `fetcher.job.failed.queue` | queues | Bound to `job.completed` and `job.failed`. |
### The dead-letter queue
The work queue carries `x-dead-letter-exchange: fetcher.dlx` and routing key `fetcher.dlq.key`. A message the Worker rejects lands in `fetcher.dlq`. That queue holds messages for 7 days and caps at 10,000 messages in the bundled configuration.
Watch the dead-letter queue. A message there is a job the Worker could not process.
### Job events
The Worker publishes `job.completed` and `job.failed` for every terminal job. Both routes are required, so a delivery failure is not silent. Consumers bind to the two queues above, or bind their own.
## Object storage
***
The Worker writes each stored result to S3-compatible object storage through the AWS S3 protocol. AWS S3, MinIO, and SeaweedFS all work. Reach SeaweedFS through its S3 gateway, not through its native filer API.
Two settings decide the connection posture:
* The scheme on `OBJECT_STORAGE_ENDPOINT` controls TLS. Use `https://` outside local development.
* `OBJECT_STORAGE_USE_PATH_STYLE=true` is what MinIO and SeaweedFS expect.
### Result retention
Set retention on the bucket, with the object store's own lifecycle policy. An S3 lifecycle rule on `OBJECT_STORAGE_KEY_PREFIX` expires results after the window your retention policy requires. Give Fetcher its own bucket or its own prefix, so the rule applies to extraction results and to nothing else.
Fetcher encrypts every stored result before it reaches the bucket. See [Security](/en/fetcher/fetcher-security).
## Valkey or Redis
***
The Manager uses Valkey or Redis for two things. It caches discovered datasource schemas, and it rate-limits the connection test. The `SCHEMA_CACHE_TTL_SECONDS` variable sets the cache lifetime, and it defaults to 5 minutes.
The schema cache falls back to process memory when Redis is unreachable. The Manager keeps serving. Its readiness probe reports the Redis state separately, so the fallback stays visible.
The Worker does not use the schema cache.
## Worker concurrency and autoscaling
***
`RABBITMQ_NUMBERS_OF_WORKERS` sets how many jobs one Worker process runs in parallel. It defaults to 5.
Size it against your datasources, not against the Worker. Each parallel job opens connections to the databases it reads. Every relational datasource pool caps at 25 open and 10 idle connections, and MongoDB caps at 100. Within one extraction, the engine runs at most 4 datasource steps at a time.
For horizontal scale, add Worker replicas and drive them from the depth of the extraction work queue. The bundled local stack runs the KEDA operator (2.16.0) against that queue, so the same trigger carries over to Kubernetes.
Every extraction also carries fixed engine limits. They are 10 datasources, 20 tables per datasource, and 50 fields per table. A run also stops at a 5-minute timeout or a 256 MiB serialized result. A request can lower any of them, and no request can raise one.
## Deployment mode and TLS
***
`DEPLOYMENT_MODE` declares the flavor of the deployment: `saas`, `byoc`, or `local`. It tags the `/readyz` response, and in `saas` it enforces TLS.
**SaaS mode requires TLS on every platform dependency.** Set `DEPLOYMENT_MODE=saas` and a plaintext MongoDB, Redis, RabbitMQ, S3, or Tenant Manager URL stops the service. It stops before it opens any connection, and the error names the rule. Your own tenants' datasources stay out of scope for this check.
Leave `ALLOW_INSECURE_TLS` unset in production. It permits plaintext connections to MongoDB, Redis, PostgreSQL, and RabbitMQ, and the bundled local environment is the only place for it.
## Startup checks
***
Fetcher fails closed. Each check below stops the process before it serves traffic.
| Trigger | What the operator sees |
| ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `APP_ENC_KEY` missing, not base64, or under 32 bytes | The process exits at startup and never binds a port. The log reads `master key too short: got 0 bytes, minimum 32 required`. |
| `STREAMING_ENABLED` not `true` on the Worker | Startup aborts with `STREAMING_ENABLED=true is required for mandatory job event notifications`. |
| `RABBITMQ_JOB_EVENTS_EXCHANGE` blank on the Worker | Startup aborts with `RABBITMQ_JOB_EVENTS_EXCHANGE is required for mandatory job event notifications`. |
| `OBJECT_STORAGE_BUCKET` missing on the Worker | The storage repository does not build, and the Worker does not start. |
| `MULTI_TENANT_ENABLED=true` without `MULTI_TENANT_SERVICE_API_KEY` | Startup aborts on both services with an error naming the variable. |
| Two per-service tenant keys that normalize to the same token | Startup aborts with an error naming the colliding token. |
| Multi-tenant mode with authentication off | The Manager router refuses to build: `tenant middleware requires effective authentication`. |
| `DEPLOYMENT_MODE=saas` with a plaintext platform dependency | Startup aborts before any connection opens. |
**A dependency that is down at boot does not produce a silently broken pod.** `/health` returns 503 until the startup self-probe succeeds. The kubelet then restarts the pod instead of sending it traffic.
## Rolling updates
***
On `SIGTERM`, both services enter a drain. `/readyz` answers 503 for `READYZ_DRAIN_DELAY_SEC` seconds, which defaults to 12, before connections tear down. Kubernetes removes the pod from Service endpoints while it still serves in-flight work.
Set your termination grace period above the drain window. See [Observability](/en/fetcher/fetcher-observability) for what the probes report during a drain.
## Next steps
***
Every environment variable, per component.
Keys, signing, encryption at rest, and host validation.
Probes, drain behavior, metrics, and tracing.
Job lifecycle, terminal states, and events.