Fetcher deployment

A standalone Fetcher deployment is two services and four dependencies. This page covers what each dependency does, how to size the Worker, and what stops a service at startup. If you only need extraction inside one Go application, you deploy none of this. See Embedding the Engine.

​

What you deploy

---

​

MongoDB

---

Both services connect to the same MongoDB deployment. The Manager writes connection and job records, and the Worker updates job state. In multi-tenant mode, each tenant gets its own database, resolved from the JWT claims on the request. That path fails closed: a request that carries a tenant identity with no tenant database returns an error instead of touching the shared database.

​

RabbitMQ

---

Fetcher uses three exchanges and four queues. The bundled local stack declares them for you. In your own environment, declare them before the services start.

​

The dead-letter queue

The work queue carries x-dead-letter-exchange: fetcher.dlx and routing key fetcher.dlq.key. A message the Worker rejects lands in fetcher.dlq. That queue holds messages for 7 days and caps at 10,000 messages in the bundled configuration. Watch the dead-letter queue. A message there is a job the Worker could not process.

​

Job events

The Worker publishes job.completed and job.failed for every terminal job. Both routes are required, so a delivery failure is not silent. Consumers bind to the two queues above, or bind their own.

​

Object storage

---

The Worker writes each stored result to S3-compatible object storage through the AWS S3 protocol. AWS S3, MinIO, and SeaweedFS all work. Reach SeaweedFS through its S3 gateway, not through its native filer API. Two settings decide the connection posture:

• The scheme on OBJECT_STORAGE_ENDPOINT controls TLS. Use https:// outside local development.
• OBJECT_STORAGE_USE_PATH_STYLE=true is what MinIO and SeaweedFS expect.

​

Result retention

Set retention on the bucket, with the object store’s own lifecycle policy. An S3 lifecycle rule on OBJECT_STORAGE_KEY_PREFIX expires results after the window your retention policy requires. Give Fetcher its own bucket or its own prefix, so the rule applies to extraction results and to nothing else. Fetcher encrypts every stored result before it reaches the bucket. See Security.

​

Valkey or Redis

---

The Manager uses Valkey or Redis for two things. It caches discovered datasource schemas, and it rate-limits the connection test. The SCHEMA_CACHE_TTL_SECONDS variable sets the cache lifetime, and it defaults to 5 minutes. The schema cache falls back to process memory when Redis is unreachable. The Manager keeps serving. Its readiness probe reports the Redis state separately, so the fallback stays visible. The Worker does not use the schema cache.

​

Worker concurrency and autoscaling

---

RABBITMQ_NUMBERS_OF_WORKERS sets how many jobs one Worker process runs in parallel. It defaults to 5. Size it against your datasources, not against the Worker. Each parallel job opens connections to the databases it reads. Every relational datasource pool caps at 25 open and 10 idle connections, and MongoDB caps at 100. Within one extraction, the engine runs at most 4 datasource steps at a time. For horizontal scale, add Worker replicas and drive them from the depth of the extraction work queue. The bundled local stack runs the KEDA operator (2.16.0) against that queue, so the same trigger carries over to Kubernetes. Every extraction also carries fixed engine limits. They are 10 datasources, 20 tables per datasource, and 50 fields per table. A run also stops at a 5-minute timeout or a 256 MiB serialized result. A request can lower any of them, and no request can raise one.

​

Deployment mode and TLS

---

DEPLOYMENT_MODE declares the flavor of the deployment: saas, byoc, or local. It tags the /readyz response, and in saas it enforces TLS. Leave ALLOW_INSECURE_TLS unset in production. It permits plaintext connections to MongoDB, Redis, PostgreSQL, and RabbitMQ, and the bundled local environment is the only place for it.

​

Startup checks

---

Fetcher fails closed. Each check below stops the process before it serves traffic.

​

Rolling updates

---

On SIGTERM, both services enter a drain. /readyz answers 503 for READYZ_DRAIN_DELAY_SEC seconds, which defaults to 12, before connections tear down. Kubernetes removes the pod from Service endpoints while it still serves in-flight work. Set your termination grace period above the drain window. See Observability for what the probes report during a drain.

​

Next steps

---