Fetcher security

Fetcher holds credentials for databases it does not own, and it moves rows that came out of them. This page describes what protects each of those, and what an operator has to do.

​

One master key, four derived keys

---

APP_ENC_KEY is the only key you supply. Generate it with make generate-master-key, which produces a base64-encoded 32-byte value. Set the same value on the Manager and on the Worker. Fetcher never uses that key directly. It expands it with HKDF-SHA256 (RFC 5869) into four independent keys, one per purpose. Separation is the point. A consumer that holds the external key can check a result signature. It cannot decrypt a stored credential, and it cannot forge a message between the two services.

​

Key version and rotation

---

APP_ENC_KEY_VERSION labels the key that is in force. Every connection record stores the version that encrypted its password, so an operator can tell which key a record belongs to. Increment the version when you change the master key. Two consequences follow a master key change:

1. Stored credentials. A connection encrypted under the previous key belongs to the previous key. Register those connections again under the new key.
2. External verification. The external HMAC key changes with the master key. Derive the new key, and give it to every consumer that checks signatures. Earlier results verify against the earlier key.

Generate the external key with make derive-key KEY="<your-base64-master-key>". The tool also reads APP_ENC_KEY from the environment or the key from standard input, and it prints a 64-character hex key.

​

Credentials at rest

---

A datasource password never reaches MongoDB in the clear. The Manager encrypts it with AES-256-GCM under the derived credential key, then stores the ciphertext and the key version. Internal datasources declared through DATASOURCE_{NAME}_* are the deliberate exception. They come from the operator’s own environment, and Fetcher marks them internal with an empty key version.

​

Results at rest

---

The Worker protects a stored result in two steps:

1. It signs the plaintext JSON with HMAC-SHA256 under the derived external key, and records the algorithm and the signature with the result.
2. It encrypts the payload with AES-GCM under the derived storage key, with a fresh 12-byte random nonce, and stores the result base64-encoded.

The signature covers the plaintext, so a consumer verifies the data it received and not the envelope around it. The repository ships a verification guide at scripts/crypto/derive-key/verification-guide.md. Direct mode returns the rows inline without encryption. The engine reports them as plaintext and attaches a SHA-256 digest over the exact bytes.

​

Signed messages between the services

---

Every RabbitMQ message the Manager publishes to the Worker carries an HMAC-SHA256 signature under the derived internal key. The signature covers more than the body. It binds:

• the timestamp and the signature version,
• the tenant identifier,
• the job identifier,
• the exchange and the routing key,
• the message body.

That binding is what stops replay. A captured message replayed under a different tenant fails verification, because the signature covers the tenant identifier. The same message replayed onto a different exchange or routing key fails for the same reason. The publisher also strips any caller-supplied security header before it signs, so a client cannot inject its own. The signer refuses a key shorter than 32 bytes, and it compares signatures in constant time.

​

Datasource host validation

---

A tenant that registers its own connection could point it at your internal network. With MULTI_TENANT_ENABLED=true, Fetcher checks the host before it connects. Validation runs at two layers:

1. At request parse. Fetcher rejects an IP literal in a blocked range, with no DNS lookup.
2. In the datasource factory. Fetcher checks the hostname against a blocklist that covers localhost, cloud metadata names, and the .local, .internal, and .cluster.local suffixes. It then resolves the hostname and checks every address it gets back.

Fetcher refuses a blocked host with a forbidden-host error. The blocked ranges cover loopback, private, and cloud-metadata addresses, and they live in lib-commons so every Lerian product shares one list. Operator-configured internal datasources are exempt by construction. They come from your environment, not from a tenant request.

​

Tenant isolation

---

The engine scopes every operation by tenant identifier, and that identifier is the only isolation boundary it has. A malformed tenant identifier fails before Fetcher touches any resource. In multi-tenant mode, each tenant gets its own metadata database, resolved from the JWT claims on the request through the tenant middleware. Access fails closed. A request that carries a tenant identity with no resolved tenant database returns an error rather than reading the shared database.

​

Authentication and probe surfaces

---

PLUGIN_AUTH_ENABLED=true puts the Access Manager middleware in front of the API. Requests then carry a bearer token, and Fetcher authorizes each operation against a resource and an action. /health, /readyz, /readyz/tenant/:id, /metrics, and /version mount before that middleware, so Kubernetes and load-balancer probes stay unauthenticated.

​

Errors never leak connection material

---

Fetcher discards the raw driver error at the engine boundary and returns a fixed message in its place. A raw driver error can embed a DSN or a credential, so a caller sees failed to connect to datasource instead of the string the driver produced. Failures arrive classified into stable categories — validation, unauthorized, forbidden, limit exceeded, connect, timeout, and others — so a host maps them to its own status codes without parsing text.

​

Next steps

---