> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# What is Fetcher?

> Extract data from external databases with Fetcher — Lerian's source-available, unified data extraction service for PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB.

export const GSourceAvailable = ({children}) => <Tooltip headline="Source-available" tip="A licensing model where the source code is publicly accessible for inspection, modification, and self-hosting, while certain commercial uses may require a license." cta="See glossary" href="/en/glossary">
    {children}
  </Tooltip>;

**Fetcher** is Lerian's data extraction service. It gives the platform — and your own applications — a single, secure interface to connect to external databases, discover their schemas, and extract data from them: PostgreSQL, MySQL, Oracle, SQL Server, and MongoDB, all through one API.

Fetcher is <GSourceAvailable>source-available</GSourceAvailable> under the [Elastic License 2.0](https://github.com/LerianStudio/fetcher/blob/main/LICENSE). The source code is publicly available on [GitHub](https://github.com/LerianStudio/fetcher).

## Fetcher in the Lerian platform

***

Lerian products regularly need data that lives outside the platform — transaction extracts in a bank's operational database, statements in a partner's MySQL, customer records in MongoDB. Before Fetcher, each product implemented its own access logic, with duplicated code and inconsistent security.

Fetcher centralizes that job. Products like **Reporter** and **Matcher** use it to pull external data into their workflows — report generation, reconciliation — and your own services can use the same API for anything that needs controlled, audited access to external databases.

Fetcher provides four core capabilities:

1. **Connection management**: Store, validate, and test database connections, with credentials encrypted at rest.
2. **Schema discovery**: Automatic detection of tables, columns, and data types across all supported databases.
3. **Data extraction**: A unified query interface with field projection, filtering, and multi-table, multi-schema, multi-datasource support.
4. **Job orchestration**: Asynchronous processing with duplicate detection, status tracking, and completion events.

## Architecture

***

Fetcher runs as two cooperating services over an embeddable core:

| Component   | Role                                                                                                                                                                                                                                           |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Manager** | HTTP API server for connection and job management. Stores metadata in MongoDB and dispatches extraction jobs through RabbitMQ.                                                                                                                 |
| **Worker**  | Asynchronous job processor. Consumes jobs from the queue, extracts data from the configured databases, encrypts the results into object storage (SeaweedFS or any S3-compatible service), and publishes `job.completed` / `job.failed` events. |
| **Engine**  | The extraction core (`pkg/engine`) both services run on — connection lifecycle, schema discovery, query planning, execution, limits, and tenant safety.                                                                                        |

The **Engine** is also importable on its own: it ships as a distinct Go module with zero third-party dependencies, so a host application can embed Fetcher's extraction capabilities in-process — no separate service, queue, or storage stack required. Lerian products embed it exactly this way; the standalone Manager and Worker are themselves hosts running over the same Engine.

### Supported databases

| Database       | Versions | Highlights                                                          |
| -------------- | -------- | ------------------------------------------------------------------- |
| **PostgreSQL** | 12+      | Multi-schema support, JSONB auto-parsing, connection pooling        |
| **MySQL**      | 8.0+     | JSON field auto-parsing, multi-table extraction, connection pooling |
| **Oracle**     | 19c+     | Owner/schema namespaces, multi-table extraction, connection pooling |
| **SQL Server** | 2019+    | Multi-schema support, table filter matching, connection pooling     |
| **MongoDB**    | 7.0+     | Schemaless inference, statistical sampling, aggregation pipelines   |

## Security posture

***

Fetcher is built to touch other people's databases safely:

* **One master key, three derived keys.** A single `APP_ENC_KEY` secret derives — via HKDF — independent keys for credential encryption (AES-256-GCM), internal message signing between Manager and Worker (HMAC-SHA256), and signing of extracted documents so consumers can verify their authenticity.
* **SSRF protection.** In multi-tenant deployments, connection hosts are validated against private and cloud-metadata address ranges before Fetcher ever dials them.
* **Fail-closed transport.** Insecure (non-TLS) connections to infrastructure are rejected by default outside local development.
* **Multi-tenant isolation.** When enabled, each tenant gets isolated metadata databases, cache namespaces, and object-storage paths, resolved from JWT-based tenant context — with zero overhead when disabled.

## Next steps

***

<CardGroup cols={2}>
  <Card title="Getting started" icon="rocket" href="/en/fetcher/fetcher-getting-started">
    Run Fetcher locally and execute your first extraction job.
  </Card>

  <Card title="Core concepts" icon="book" href="/en/fetcher/fetcher-core-concepts">
    Connections, schema discovery, extraction jobs, and results.
  </Card>
</CardGroup>