Skip to main content
The Fetcher Engine is the extraction core of Fetcher, packaged as a Go module you can import. Host applications such as Matcher and Reporter run the Engine in-process instead of operating a separate Fetcher deployment. The standalone Manager and Worker are themselves hosts over the same Engine. The Engine owns the rules of extraction: connection lifecycle, schema discovery and validation, query planning, extraction execution, result and error contracts, limits, and tenant safety. It owns no infrastructure at all.
The Engine is a distinct Go module from the Fetcher services. Its module path is github.com/LerianStudio/fetcher/pkg/engine, and the services module is github.com/LerianStudio/fetcher/v2. The Engine has its own version line with path-prefixed tags (pkg/engine/vX.Y.Z). An import of the Engine inherits none of the service dependencies.Fetcher is source-available under the Elastic License 2.0, and the Engine carries the same license. You read the extraction rules you embed.

​
The three-layer model

The guiding principle: the Engine owns what makes Fetcher Fetcher, and host applications own how Fetcher runs. Every product that embeds the Engine therefore shares one canonical owner of datasource and extraction behavior. A rule change in the core reaches every host at the next module bump, and no host re-implements it.

​
The import boundary

The Engine module declares zero third-party dependencies. Its go.mod has no require block at all, and a build-enforced test keeps it that way. Two guards run inside the module on every go test ./..., and the module CI workflow runs them with the Go workspace turned off:
  • The allowlist. Every transitive dependency of pkg/engine must be either a Go standard-library package or a package local to the Engine module. Any other import family fails the build, even a family that no denylist names.
  • A named denylist. Enumerated classes fail on top of the allowlist. They cover HTTP frameworks, message brokers, database drivers, object-storage SDKs, tenant-runtime middleware, the auth and license libraries, and standard-library shells such as database/sql, net/http, and os/exec.
A separate CI step greps the Engine go.mod for a require line and fails the job when it finds one.

​
Why this matters when you embed

  • No dependency conflicts. The Engine cannot pull a driver, a broker client, or an HTTP framework into your module graph. Your host keeps full control of its own versions.
  • No hidden I/O. The core cannot open a socket, a file, or a database on its own. Every byte in and out crosses a port you supplied.
  • A stable substitution seam. Run the whole Engine against the in-memory harness in tests. Swap real adapters in production, with no change to the calling code.

​
Direct mode and Store mode

An extraction plan carries a mode with three values: direct, store, and the zero value auto. The Engine resolves auto from the ports you wired. With a ResultSink configured it picks store mode, and without one it picks direct mode. So the mode is a consequence of your composition, not a separate switch. An explicit store request with no sink fails up front as a validation error, before the Engine touches any datasource. The two modes return different shapes. Exactly one arm of the result is non-nil, and the JSON omits the unused arm entirely.

​
Direct mode

The Engine runs the plan steps and merges the rows into one map. The map keys are the datasource config name and then the qualified table. The Engine serializes that map once as indented JSON and returns the bytes inline.
The inline result carries this metadata: Direct-mode output is deterministic. The serializer sorts the map keys. The same input therefore gives you byte-identical JSON and the same digest, whatever order the parallel steps finished in. Your host owns everything after that. The Fetcher Worker, for example, signs the plaintext with HMAC-SHA256 and encrypts it before it stores the bytes.

​
Store mode

The Engine opens a stream on your ResultSink and writes the result incrementally, in constant memory. It never holds the whole result. One writer drains the extraction goroutines strictly by ascending plan-step order, and the integrity digest covers exactly the bytes written. The wire shape is NDJSON — one JSON object per line, newline-terminated, with no enclosing array:
The call returns a reference instead of bytes: On an abort — a write error, an exceeded size limit, or a canceled context — the Engine abandons the writer and never calls Close. A partial result therefore never becomes a returned reference. Treat an unclosed writer as a discarded write.

​
Failure and result contracts

  • Fail-fast across datasources. The first failing step stops the run. The Engine never returns a partial result.
  • Redacted errors. A driver error can embed a DSN, a credential, or driver internals, so the Engine discards it and returns a fixed message instead.
  • Eleven error categories. validation, not_found, unauthorized, forbidden, limit_exceeded, conflict, unavailable, connect, timeout, canceled, and internal. Your host maps them to its own transport codes. connect stays distinct from unavailable, and timeout stays distinct from canceled.
  • Five execution statuses. pending, running, completed, failed, and canceled. The last three are terminal. A host cancellation records canceled, and a deadline overrun records failed.
  • Closed connectors. Every connector the Engine opens closes again, on the success path and on every failure path.

​
Limits and tenant scope

The Engine never runs unbounded. When you supply no limits, it applies these defaults: A request can lower any limit and never raise it. An override above the default fails with a validation error that names the breached field. A zero or negative override keeps the default. The Engine enforces the result-size ceiling twice: a cheap per-step lower bound that aborts early, and an authoritative check on the final indented payload. An over-limit result never reaches you inline and never reaches your sink. Tenant scope is equally narrow. Every operation carries a tenant ID and nothing else — the Engine has no organization or product concept. It validates the tenant ID before any resource access, and it rejects an empty or malformed value there.

​
Next steps

Embed the Engine

Import it, provide the ports, and construct it with a runnable example.

Port reference

Every port, whether it is required, and what happens without it.