Skip to main content
Fetcher reads data out of databases that your application does not own. It connects to PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB. It discovers which tables and fields exist. It then extracts the rows you ask for, through one API and one credential store. You give Fetcher a connection and a field selection. Fetcher validates that selection against the live schema, plans the work, and runs it. It returns the rows to you inline, or writes them to object storage as an encrypted, signed artifact.

​
Open source

Fetcher ships under the Elastic License 2.0. The full source code lives on GitHub, and every engine source file carries an SPDX-License-Identifier: Elastic-2.0 header. Fetcher and Midaz are the two Lerian products.

​
Two ways to run it

The duality below is the shape of the product. Pick the row that matches your problem. The Engine is a separate Go module from the services module (github.com/LerianStudio/fetcher/v2). It carries zero third-party dependencies, and CI fails the build if a require block appears in its go.mod. Your host application supplies the parts the Engine needs through small interfaces, called ports. The Manager and the Worker are themselves hosts over that same Engine. Both shapes therefore run identical extraction rules.
The Engine ships with an in-memory harness (pkg/engine/memory) that covers the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You can run a real extraction with no MongoDB, no RabbitMQ, and no object storage. The harness does not cover credential protection: encrypted persistence is off by default, and turning it on requires your host application to supply a CredentialProtector. See Getting started.

​
What Fetcher gives you

  1. Connection management. Store, validate, and test database connections. Fetcher encrypts each password with AES-256-GCM before it writes the record.
  2. Schema discovery. Fetcher detects tables, columns, and data types across all five database types, with a cache and an always-fresh read.
  3. Data extraction. One query interface with field projection, ten filter operators, and multi-table, multi-schema, multi-datasource requests.
  4. Job orchestration. Asynchronous processing with a 5-minute duplicate window, status tracking, and job.completed and job.failed events.

​
Datasources

Fetcher accepts five datasource types. Send the identifier upper-case on the API — request validation matches the five values exactly, and postgresql is rejected with 400. Operator-declared internal datasources (DATASOURCE_{NAME}_TYPE) and the type query filter on the connection list do accept any case.

​
Safe by construction

  • Tenant scoping. Every engine operation carries a tenant ID, and that ID is the only isolation boundary. A malformed tenant ID fails before Fetcher touches any resource.
  • Redacted errors. Fetcher discards the raw driver error at its boundary, so a DSN, a credential, or a driver internal cannot reach a caller.
  • Fail-fast extraction. The first failing step stops the run. Fetcher never returns a partial result.
  • Fixed limits. 10 datasources per extraction, 20 tables per datasource, 50 fields per table, and a 5-minute timeout. A request can lower any limit, and never raise one.
  • Host validation. In multi-tenant mode, Fetcher rejects a tenant-supplied host that resolves to a private, loopback, or cloud-metadata address.

​
Next steps

Use cases

Concrete problems Fetcher solves, and what changes when you adopt it.

Getting started

Two paths to a first extraction: zero infrastructure, or full Docker stack.

Core concepts

Connections, schema discovery, extraction jobs, filters, and results.

Security

Master key, derived keys, message signing, and host validation.