Skip to main content
Fetcher is Lerian’s data extraction service. It gives the platform — and your own applications — a single, secure interface to connect to external databases, discover their schemas, and extract data from them: PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB, all through one API. Fetcher is under the Elastic License 2.0. The source code is publicly available on GitHub.

​
Fetcher in the Lerian platform

Lerian products regularly need data that lives outside the platform — transaction extracts in a bank’s operational database, statements in a partner’s MySQL, customer records in MongoDB. Before Fetcher, each product implemented its own access logic, with duplicated code and inconsistent security. Fetcher centralizes that job. Products like Reporter and Matcher use it to pull external data into their workflows — report generation, reconciliation — and your own services can use the same API for anything that needs controlled, audited access to external databases. Fetcher provides four core capabilities:
  1. Connection management: Store, validate, and test database connections, with credentials encrypted at rest.
  2. Schema discovery: Automatic detection of tables, columns, and data types across all supported databases.
  3. Data extraction: A unified query interface with field projection, filtering, and multi-table, multi-schema, multi-datasource support.
  4. Job orchestration: Asynchronous processing with duplicate detection, status tracking, and completion events.

​
Architecture

Fetcher runs as two cooperating services over an embeddable core:
ComponentRole
ManagerHTTP API server for connection and job management. Stores metadata in MongoDB and dispatches extraction jobs through RabbitMQ.
WorkerAsynchronous job processor. Consumes jobs from the queue, extracts data from the configured databases, encrypts the results into object storage (SeaweedFS or any S3-compatible service), and publishes job.completed / job.failed events.
EngineThe extraction core (pkg/engine) both services run on — connection lifecycle, schema discovery, query planning, execution, limits, and tenant safety.
The Engine is also importable on its own: it ships as a distinct Go module with zero third-party dependencies, so a host application can embed Fetcher’s extraction capabilities in-process — no separate service, queue, or storage stack required. Lerian products embed it exactly this way; the standalone Manager and Worker are themselves hosts running over the same Engine.

​
Supported databases

DatabaseVersionsHighlights
PostgreSQL12+Multi-schema support, JSONB auto-parsing, connection pooling
MySQL8.0+JSON field auto-parsing, multi-table extraction, connection pooling
Oracle19c+Owner/schema namespaces, multi-table extraction, connection pooling
SQL Server2019+Multi-schema support, table filter matching, connection pooling
MongoDB7.0+Schemaless inference, statistical sampling, aggregation pipelines

​
Security posture

Fetcher is built to touch other people’s databases safely:
  • One master key, three derived keys. A single APP_ENC_KEY secret derives — via HKDF — independent keys for credential encryption (AES-256-GCM), internal message signing between Manager and Worker (HMAC-SHA256), and signing of extracted documents so consumers can verify their authenticity.
  • SSRF protection. In multi-tenant deployments, connection hosts are validated against private and cloud-metadata address ranges before Fetcher ever dials them.
  • Fail-closed transport. Insecure (non-TLS) connections to infrastructure are rejected by default outside local development.
  • Multi-tenant isolation. When enabled, each tenant gets isolated metadata databases, cache namespaces, and object-storage paths, resolved from JWT-based tenant context — with zero overhead when disabled.

​
Next steps

Getting started

Run Fetcher locally and execute your first extraction job.

Core concepts

Connections, schema discovery, extraction jobs, and results.