Fetcher use cases

Each scenario below states the problem first, then what changes when Fetcher owns the extraction path.

​

Stop rebuilding database access in every service

---

The problem. Lerian hit this internally. Each product team built its own data-access logic against the same external databases, and each copy aged separately. Fetcher exists to centralize that work. The same pattern appears in any company that runs several internal systems against the same operational databases. Every team writes its own connection pool, its own secret handling, and its own query builder. That is the same work, done five times. What changes. One connection record serves every consumer. Fetcher encrypts the password once, tests the connection on demand, and applies the same ten filter operators to all five database types. A new consumer registers no drivers and holds no credentials. It calls an API. Adding a datasource type becomes one team’s work instead of five teams’ work.

​

Feed reconciliation and reporting from operational databases

---

The problem. Reconciliation and report generation both need rows that live outside the platform. Those rows sit in a bank’s operational PostgreSQL, in a partner’s MySQL, or in a MongoDB customer store. The consuming product must reach them without becoming a database client itself. What changes. The consumer submits a field selection per datasource and per table. Fetcher validates that selection against the live schema before it runs anything. A renamed column therefore fails at plan time, not mid-extraction. The Worker writes the result to object storage and publishes job.completed or job.failed. The consumer reacts to the event. Fetcher also resolves a fixed set of Lerian-internal datasources from environment variables, so a report over Midaz data needs no registered connection.

​

Publish an extract that a third party can verify

---

The problem. You send a data extract to an auditor, a regulator, or a partner. They need proof that the file came from your systems and that nobody edited it in transit. What changes. The Worker signs the plaintext JSON with HMAC-SHA256 before it encrypts the payload, then stores the result with AES-GCM encryption at rest. Your recipient derives the external verification key from your master key with the make derive-key command, and checks the signature independently. That derived key verifies signatures only. It cannot decrypt stored credentials, and it cannot forge internal messages between the Manager and the Worker.

​

Add extraction to a service you already run

---

The problem. You need extraction inside one Go service. Standing up a Manager, a Worker, MongoDB, RabbitMQ, and object storage costs more than the feature is worth. What changes. Your service imports the Engine module and calls it in-process. The Engine has no third-party dependencies, so the import adds no operational surface. With no result sink wired, the Engine runs in direct mode and returns the rows inline as indented JSON, with a SHA-256 digest over the exact bytes. That output is deterministic. Identical input produces byte-identical JSON and the same digest, whatever order the parallel datasource steps finished in. You can persist those bytes and sign them yourself.

​

Serve many tenants from one deployment

---

The problem. You run one Fetcher for many customers. Each customer registers its own database hosts. One customer must never read another customer’s rows, and no customer may point a connection at your internal network. What changes. Multi-tenant mode gives each tenant its own metadata database, resolved from the JWT claims on the request. Access fails closed: a request that carries a tenant ID with no tenant database returns an error instead of reading the shared database. The same switch turns on host validation. Fetcher rejects a tenant-supplied host that resolves to a private, loopback, or cloud-metadata address. It checks an IP literal at request-parse time, and a hostname again in the datasource factory. Operator-configured internal datasources stay exempt.

​

Next steps

---