> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Fetcher core concepts

> The objects Fetcher works with — connections, schema discovery, extraction jobs, filters, and signed results.

Fetcher's model is small: you register **connections** to external databases, Fetcher **discovers their schemas**, and you submit **extraction jobs** that produce **signed, encrypted results**. This page walks through each piece.

## Connections

***

A connection is a stored, named reference to an external database — its type, host, port, database name, and credentials. Fetcher supports five connection types: `POSTGRESQL`, `MYSQL`, `ORACLE`, `SQL_SERVER`, and `MONGODB`.

* **Credentials are encrypted at rest** with AES-256-GCM before they reach storage; plaintext passwords never persist.
* **Connections are testable**: a dedicated endpoint opens a real connection and reports latency, so you can validate a connection before any job depends on it.
* **Connections are protected while in use**: updating or deleting a connection with active jobs is rejected.

Besides connections registered through the API, operators can define **internal datasources** directly via environment variables (`DATASOURCE_{NAME}_*`) — useful for fixed, platform-owned databases. See [Configuration](/en/fetcher/fetcher-configuration).

## Schema discovery

***

Fetcher automatically detects tables, columns, and data types on the databases it connects to — the same discovery works across all five database types. For relational databases it reads the catalog, including multi-schema namespaces (PostgreSQL schemas, Oracle owners, SQL Server schemas). For MongoDB, where collections have no declared schema, Fetcher infers one by statistical sampling of documents.

Two things build on discovery:

* **Schema validation**: before running a job, you can validate that the tables and fields you plan to extract actually exist in the datasources.
* **Schema caching**: discovered schemas are cached (with a configurable TTL) so repeated jobs don't re-introspect the database every time.

## Extraction jobs

***

A job is an asynchronous request to extract data. Its heart is the `mappedFields` map — *which fields, from which tables, from which datasources*:

```json theme={null}
{
  "dataRequest": {
    "mappedFields": {
      "my_postgres": {
        "accounts": ["id", "email", "created_at"]
      },
      "my_mongo": {
        "transactions": ["*"]
      }
    }
  }
}
```

* **Field projection**: name the fields you want, or use `["*"]` for all fields.
* **Multi-everything**: a single job can span multiple datasources, multiple tables per datasource, and multiple schemas.
* **JSON handling**: JSON/JSONB fields in relational databases are parsed automatically.

### Filters

Jobs can filter rows per table with a set of 10 operators: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `in`, `nin`, and `like`.

```json theme={null}
{
  "dataRequest": {
    "filters": {
      "my_postgres": {
        "transactions": {
          "status": { "in": ["completed", "pending"] }
        }
      }
    }
  }
}
```

### Job lifecycle

| Status       | Meaning                                                  |
| ------------ | -------------------------------------------------------- |
| `pending`    | Accepted and queued; not yet picked up by a Worker       |
| `processing` | A Worker is extracting the data                          |
| `completed`  | Results are stored and the completion event is published |
| `failed`     | Extraction failed; the failure event is published        |

Jobs are **deduplicated**: an identical request within a 5-minute window returns the existing job instead of creating a new one. On terminal states, the Worker publishes `job.completed` / `job.failed` notification events so consumers can react without polling.

## Results

***

The Worker writes extraction results to object storage — SeaweedFS by default, or any S3-compatible service:

* **Encrypted at rest**, with a configurable retention TTL.
* **Signed**: each extracted document carries an HMAC-SHA256 signature. Consumers can derive the verification key from the master key and check the authenticity of what they read — the data provably came from Fetcher and was not tampered with.

## Services and the Engine

***

Two deployment shapes share one core:

* **Standalone services** — the **Manager** (HTTP API, metadata in MongoDB, job dispatch via RabbitMQ) and the **Worker** (queue consumer, extraction, result storage). This is the shape the [getting started guide](/en/fetcher/fetcher-getting-started) runs.
* **Embedded Engine** — the extraction core (`pkg/engine`) as an importable Go module with zero third-party dependencies. Host applications wire their own storage and infrastructure to the Engine's interfaces and run extractions in-process. Without a result store, the Engine runs in *Direct mode*: results come back inline with a SHA-256 integrity digest.

The rule of thumb: the Engine owns *what* extraction means — planning, validation, limits, tenant safety; the host (Manager/Worker, or your application) owns *how* it runs — queues, storage, auth, lifecycle.

## Tenancy

***

Fetcher runs single-tenant by default. In multi-tenant mode, every operation is scoped to a tenant resolved from JWT-based tenant context, and each tenant gets isolated metadata databases, cache namespaces, and object-storage paths. Multi-tenant code paths add zero overhead when the mode is disabled. See [Multi-tenancy](/en/multi-tenancy) for the platform-wide model.

## Next steps

***

<CardGroup cols={2}>
  <Card title="Getting started" icon="rocket" href="/en/fetcher/fetcher-getting-started">
    Run Fetcher locally and execute your first extraction job.
  </Card>

  <Card title="Configuration" icon="gear" href="/en/fetcher/fetcher-configuration">
    The environment variables that shape a Fetcher deployment.
  </Card>
</CardGroup>