> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetcher Engine overview
> The Fetcher Engine is an importable Go module that runs data extraction in-process: the three-layer model, the enforced import boundary, and Direct versus Store mode.
The **Fetcher Engine** is the extraction core of Fetcher, packaged as a Go module you can import. Host applications such as Matcher and Reporter run the Engine in-process instead of operating a separate Fetcher deployment. The standalone [Manager and Worker](/en/fetcher/what-is-fetcher) are themselves hosts over the same Engine.
The Engine owns the rules of extraction: connection lifecycle, schema discovery and validation, query planning, extraction execution, result and error contracts, limits, and tenant safety. It owns no infrastructure at all.
The Engine is a **distinct Go module** from the Fetcher services. Its module path is `github.com/LerianStudio/fetcher/pkg/engine`, and the services module is `github.com/LerianStudio/fetcher/v2`. The Engine has its own version line with path-prefixed tags (`pkg/engine/vX.Y.Z`). An import of the Engine inherits none of the service dependencies.
Fetcher is source-available under the Elastic License 2.0, and the Engine carries the same license. You read the extraction rules you embed.
## The three-layer model
***
| Layer | Package | Owns |
| -------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Engine core** | `pkg/engine` | The *rules* of extraction. It depends only on host-provided port interfaces, never on infrastructure. |
| **Compatibility adapters** | `pkg/enginecompat/*` | Bridges between the Engine ports and Fetcher's real infrastructure — MongoDB, Redis, and the datasource drivers. |
| **Host application** | Manager, Worker, or your own service | The operational shell: authentication, license enforcement, HTTP routes, queues, state stores, storage, telemetry, and process lifecycle. |
The guiding principle: **the Engine owns what makes Fetcher *Fetcher*, and host applications own *how* Fetcher runs.**
Every product that embeds the Engine therefore shares one canonical owner of datasource and extraction behavior. A rule change in the core reaches every host at the next module bump, and no host re-implements it.
## The import boundary
***
The Engine module declares **zero third-party dependencies**. Its `go.mod` has no `require` block at all, and a build-enforced test keeps it that way.
Two guards run inside the module on every `go test ./...`, and the module CI workflow runs them with the Go workspace turned off:
* **The allowlist.** Every transitive dependency of `pkg/engine` must be either a Go standard-library package or a package local to the Engine module. Any other import family fails the build, even a family that no denylist names.
* **A named denylist.** Enumerated classes fail on top of the allowlist. They cover HTTP frameworks, message brokers, database drivers, object-storage SDKs, tenant-runtime middleware, the auth and license libraries, and standard-library shells such as `database/sql`, `net/http`, and `os/exec`.
A separate CI step greps the Engine `go.mod` for a `require` line and fails the job when it finds one.
### Why this matters when you embed
* **No dependency conflicts.** The Engine cannot pull a driver, a broker client, or an HTTP framework into your module graph. Your host keeps full control of its own versions.
* **No hidden I/O.** The core cannot open a socket, a file, or a database on its own. Every byte in and out crosses a port you supplied.
* **A stable substitution seam.** Run the whole Engine against the in-memory harness in tests. Swap real adapters in production, with no change to the calling code.
## Direct mode and Store mode
***
An extraction plan carries a mode with three values: `direct`, `store`, and the zero value `auto`.
The Engine resolves `auto` from the ports you wired. With a `ResultSink` configured it picks store mode, and without one it picks direct mode. So the mode is a consequence of your composition, not a separate switch. An explicit `store` request with no sink fails up front as a validation error, before the Engine touches any datasource.
The two modes return different shapes. Exactly one arm of the result is non-nil, and the JSON omits the unused arm entirely.
### Direct mode
The Engine runs the plan steps and merges the rows into one map. The map keys are the datasource config name and then the qualified table. The Engine serializes that map once as indented JSON and returns the bytes inline.
```json theme={null}
{
"pg-main": {
"public.users": [
{
"email": "a@example.com",
"id": 1
}
]
}
}
```
The inline result carries this metadata:
| Field | Value in direct mode |
| --------------- | ------------------------------------------------------ |
| `data` | The serialized payload above |
| `format` | `json` |
| `rowCount` | Total rows across all tables |
| `plaintextSize` | Byte size of the payload |
| `integrity` | Algorithm `SHA-256` plus the hex digest of the payload |
| `protection` | `encrypted: false`, applied by `engine` |
Direct-mode output is deterministic. The serializer sorts the map keys. The same input therefore gives you byte-identical JSON and the same digest, whatever order the parallel steps finished in.
Your host owns everything after that. The Fetcher Worker, for example, signs the plaintext with HMAC-SHA256 and encrypts it before it stores the bytes.
### Store mode
The Engine opens a stream on your `ResultSink` and writes the result incrementally, in constant memory. It never holds the whole result. One writer drains the extraction goroutines strictly by ascending plan-step order, and the integrity digest covers exactly the bytes written.
The wire shape is NDJSON — one JSON object per line, newline-terminated, with no enclosing array:
```json theme={null}
{"config":"pg-main","table":"public.users","row":{"id":1,"email":"a@example.com"}}
```
The call returns a reference instead of bytes:
| Field | Meaning |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `path` | The **logical** location in your storage. The reference exposes no physical backend type — your adapter resolves the path. |
| `format` | Output format of the persisted bytes |
| `rowCount` | Total rows in the persisted result |
| `sizeBytes` | Serialized size written |
| `integrity` and `protection` | What your sink reported. When the sink reports no integrity, the Engine stamps the SHA-256 digest it computed over the streamed bytes. The Engine validates only that `protection.appliedBy` is one of `engine`, `adapter`, or `host`. |
On an abort — a write error, an exceeded size limit, or a canceled context — the Engine abandons the writer and never calls `Close`. A partial result therefore never becomes a returned reference. Treat an unclosed writer as a discarded write.
## Failure and result contracts
***
* **Fail-fast across datasources.** The first failing step stops the run. The Engine never returns a partial result.
* **Redacted errors.** A driver error can embed a DSN, a credential, or driver internals, so the Engine discards it and returns a fixed message instead.
* **Eleven error categories.** `validation`, `not_found`, `unauthorized`, `forbidden`, `limit_exceeded`, `conflict`, `unavailable`, `connect`, `timeout`, `canceled`, and `internal`. Your host maps them to its own transport codes. `connect` stays distinct from `unavailable`, and `timeout` stays distinct from `canceled`.
* **Five execution statuses.** `pending`, `running`, `completed`, `failed`, and `canceled`. The last three are terminal. A host cancellation records `canceled`, and a deadline overrun records `failed`.
* **Closed connectors.** Every connector the Engine opens closes again, on the success path and on every failure path.
## Limits and tenant scope
***
The Engine never runs unbounded. When you supply no limits, it applies these defaults:
| Limit | Default |
| --------------------------- | --------- |
| Datasources per extraction | 10 |
| Tables per datasource | 20 |
| Fields per table | 50 |
| Parallel datasource workers | 4 |
| Extraction timeout | 5 minutes |
| Serialized result size | 256 MiB |
A request can **lower** any limit and never raise it. An override above the default fails with a validation error that names the breached field. A zero or negative override keeps the default.
The Engine enforces the result-size ceiling twice: a cheap per-step lower bound that aborts early, and an authoritative check on the final indented payload. An over-limit result never reaches you inline and never reaches your sink.
Tenant scope is equally narrow. Every operation carries a tenant ID and nothing else — the Engine has no organization or product concept. It validates the tenant ID before any resource access, and it rejects an empty or malformed value there.
## Next steps
***
Import it, provide the ports, and construct it with a runnable example.
Every port, whether it is required, and what happens without it.