Fetcher’s Manager and Worker are configured entirely through environment variables — each component reads its own .env file (components/manager/.env, components/worker/.env). This page covers the variables that matter for an MVP deployment; the .env.example files in the repository are the exhaustive reference.
Encryption keys
| Variable | Component | Description |
|---|
APP_ENC_KEY | Manager + Worker | Required. Base64-encoded 32-byte master key. Generate with make generate-master-key. Both services must share the same value. |
APP_ENC_KEY_VERSION | Manager + Worker | Key rotation counter (default 1). Increment when rotating; Fetcher tracks which version encrypted each credential. |
make derive-key KEY="<master-key>".
Manager API
| Variable | Description | Default |
|---|
SERVER_PORT | HTTP port for the Manager API | 4006 |
SCHEMA_CACHE_TTL_SECONDS | How long discovered schemas stay cached | 300 |
PLUGIN_AUTH_ENABLED | Delegate authentication/authorization to Access Manager | false |
PLUGIN_AUTH_ADDRESS | Access Manager address when auth is enabled | — |
LOG_LEVEL | Log verbosity | debug (local) |
Infrastructure
Both services connect to MongoDB (metadata) and RabbitMQ (job queue); the Manager also uses Valkey/Redis for caching, and the Worker writes results to object storage.
| Variable | Component | Description |
|---|
MONGO_HOST, MONGO_PORT, MONGO_NAME, MONGO_USER, MONGO_PASSWORD | Manager + Worker | MongoDB metadata store |
RABBITMQ_HOST, RABBITMQ_PORT_AMQP, RABBITMQ_DEFAULT_USER, RABBITMQ_DEFAULT_PASS | Manager + Worker | RabbitMQ broker |
RABBITMQ_FETCHER_WORK_QUEUE | Manager + Worker | Extraction work queue (default fetcher.extract-external-data.queue) |
RABBITMQ_NUMBERS_OF_WORKERS | Worker | Concurrent extraction workers (default 5) |
REDIS_HOST, REDIS_PORT, REDIS_PASSWORD | Manager | Valkey/Redis cache |
OBJECT_STORAGE_ENDPOINT, OBJECT_STORAGE_REGION, OBJECT_STORAGE_BUCKET, OBJECT_STORAGE_ACCESS_KEY_ID, OBJECT_STORAGE_SECRET_KEY | Worker | S3-compatible result storage (SeaweedFS by default; AWS S3 and MinIO work the same way) |
ENGINE_MAX_RESULT_BYTES | Worker | Cap on the serialized result artifact per job (default 256 MiB) |
MONGO_TLS_CA_CERT, REDIS_TLS, RABBITMQ_TLS) are available per dependency.
Job event streaming
The Worker publishes mandatory job.completed and job.failed notifications and fails closed at startup if streaming is not enabled:
| Variable | Description | Default |
|---|
STREAMING_ENABLED | Must be true for the Worker to start | false |
STREAMING_BROKERS | Kafka/Redpanda bootstrap servers | localhost:9092 |
STREAMING_CLOUDEVENTS_SOURCE | CloudEvents source for Worker events | — |
RABBITMQ_JOB_EVENTS_EXCHANGE | RabbitMQ exchange for job events | fetcher.job.events |
single-tenant.
Multi-tenancy
| Variable | Description | Default |
|---|
MULTI_TENANT_ENABLED | Enable multi-tenant mode | false |
MULTI_TENANT_URL | Tenant Manager service URL | — |
MULTI_TENANT_MAX_TENANT_POOLS | Max concurrent tenant connection pools | 100 |
MULTI_TENANT_IDLE_TIMEOUT_SEC | Idle tenant connection timeout | 300 |
MULTI_TENANT_CIRCUIT_BREAKER_THRESHOLD | Circuit breaker failure threshold | 5 |
MULTI_TENANT_CIRCUIT_BREAKER_TIMEOUT_SEC | Circuit breaker reset timeout | 30 |
Enabling multi-tenant mode also activates the SSRF host-safety guard: tenant-supplied connection hosts are checked against loopback, private (RFC 1918), and cloud-metadata address ranges before Fetcher connects. With MULTI_TENANT_ENABLED=false (the default), all multi-tenant code paths are bypassed with zero performance impact.
Readiness and TLS posture
| Variable | Description | Default |
|---|
DEPLOYMENT_MODE | saas, byoc, or local — tags /readyz and drives SaaS TLS enforcement | local |
ALLOW_INSECURE_TLS | Permit plaintext connections to MongoDB, Redis, and RabbitMQ. Fetcher fails closed on insecure connection URIs by default — set this only for local development. | unset (fail closed) |
READYZ_DRAIN_DELAY_SEC | Graceful-drain window after SIGTERM, during which /readyz reports unhealthy before connections tear down | 12 |
GET /health (liveness), GET /readyz (readiness with parallel dependency probes), and GET /version.
Internal datasources
Fixed, operator-owned databases can be declared as env vars instead of API-registered connections, using the DATASOURCE_{NAME}_* family:
| Variable | Required | Example |
|---|
DATASOURCE_{N}_CONFIG_NAME | yes | billing_db |
DATASOURCE_{N}_TYPE | yes | postgresql / mysql / oracle / mongodb / sql_server |
DATASOURCE_{N}_HOST | yes | db.internal.example.com |
DATASOURCE_{N}_PORT | yes | 5432 |
DATASOURCE_{N}_DATABASE | yes | billing |
DATASOURCE_{N}_USER | yes | fetcher_ro |
DATASOURCE_{N}_PASSWORD | yes | from your secret manager |
DATASOURCE_{N}_OPTIONS | no | authSource=admin (MongoDB only) |
DATASOURCE_{N}_SSLMODE | no | require (PostgreSQL) |
_SSLMODE is the validated option; custom CA or client-certificate plumbing for internal datasources is not available.
Observability
Fetcher emits OpenTelemetry traces and metrics when telemetry is enabled:
| Variable | Description | Default |
|---|
ENABLE_TELEMETRY | Turn OTel emission on/off | false |
OTEL_EXPORTER_OTLP_ENDPOINT | OTLP collector endpoint | — |
OTEL_RESOURCE_SERVICE_NAME | Service name in telemetry (fetcher / fetcher-worker) | per component |
