> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Embedding the Fetcher Engine
> Import the Fetcher Engine Go module, provide the ports it needs, and construct it with engine.New — with a self-contained example that runs without any infrastructure.
Embedding the [Fetcher Engine](/en/fetcher/fetcher-engine-overview) takes three steps: **import it, provide the ports it needs, construct it with `engine.New`.** No infrastructure ships with the import. You wire only the parts your host application actually uses.
## 1. Install
***
```bash theme={null}
go get github.com/LerianStudio/fetcher/pkg/engine
```
The Engine is a distinct Go module from the Fetcher services (`github.com/LerianStudio/fetcher/v2`). It carries zero third-party dependencies and has its own version line with path-prefixed tags (`pkg/engine/vX.Y.Z`). The import gives your module graph none of the service dependencies.
## 2. Provide the ports
***
The Engine depends only on host-provided interfaces. One port is always required. A second is required only when encrypted persistence is on. The rest are opt-in, and the Engine degrades gracefully without them.
| Port | Required? | Without it |
| ------------------------ | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `ConnectorRegistry` | **Always** | `engine.New` fails and no Engine exists |
| `CredentialProtector` | Only with `WithEncryptedPersistence(true)` | Construction fails when encrypted persistence is on; credentials reach the store unprotected when it is off |
| `ConnectionStore` | Optional | Every operation except `Limits()` fails |
| `ResultSink` | Optional | Store mode is unavailable, and extraction runs in direct mode |
| `SchemaCache` | Optional | Schema discovery always hits the live datasource |
| `ExecutionStore` | Optional | No durable execution-state tracking |
| `ActiveExecutionChecker` | Optional | No conflict gating on connection updates and deletes |
| `Observability` | Optional | Tracing hooks become no-ops |
The [port reference](/en/fetcher/fetcher-engine-ports) documents each contract and each degradation in full. Read it before you decide which ports to skip — "optional" does not mean "harmless".
For tests and first runs, the **`pkg/engine/memory`** harness provides in-memory implementations of the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You need no MongoDB, Redis, RabbitMQ, or object storage to exercise the Engine. The harness ships no `CredentialProtector`, so a test that turns encrypted persistence on must supply one.
## 3. Construct, plan, execute
***
This example is self-contained. It uses the in-memory harness, so it runs with zero infrastructure.
```go theme={null}
package main
import (
"context"
"fmt"
"log"
"github.com/LerianStudio/fetcher/pkg/engine"
"github.com/LerianStudio/fetcher/pkg/engine/memory"
)
func main() {
ctx := context.Background()
// Provide ports. In production these are your real adapters (see pkg/enginecompat);
// here the in-memory harness stands in so the example runs with zero infrastructure.
store := memory.NewConnectionStore()
registry := memory.NewConnectorRegistry()
// Construct the Engine. WithConnectorRegistry is the only required option.
eng, err := engine.New(
engine.WithConnectorRegistry(registry),
engine.WithConnectionStore(store),
)
if err != nil {
log.Fatal(err)
}
// Every operation is scoped to a tenant — the sole isolation dimension.
tenant, err := engine.NewTenantContext("tenant-123")
if err != nil {
log.Fatal(err)
}
// Register a connector for the datasource type, then persist a connection.
conn := memory.NewTemplateConnector(memory.ConnectorBehavior{
Schema: engine.SchemaSnapshot{
ConfigName: "pg-main",
Tables: []engine.TableSnapshot{{Name: "public.users", Fields: []string{"id", "email"}}},
},
Rows: map[string][]map[string]any{
"public.users": {{"id": 1, "email": "a@example.com"}},
},
})
registry.Register("postgres", memory.NewConnectorFactory(conn))
if _, err = eng.CreateConnection(ctx, tenant, engine.NewConnectionInput(engine.ConnectionInputParams{
ConfigName: "pg-main",
Type: "postgres",
Host: "localhost",
Port: 5432,
})); err != nil {
log.Fatal(err)
}
// Plan validates the request against the live schema and enforces limits.
plan, err := eng.PlanExtraction(ctx, tenant, engine.ExtractionRequest{
MappedFields: map[string]engine.FieldSelection{
"pg-main": {"public.users": {"id", "email"}},
},
})
if err != nil {
log.Fatal(err)
}
// Execute. With no ResultSink wired, the Engine runs in Direct mode and returns
// inline JSON bytes plus a SHA-256 integrity digest.
result, err := eng.ExecuteExtraction(ctx, plan)
if err != nil {
log.Fatal(err)
}
fmt.Printf("rows=%d bytes=%d\n", result.Direct.RowCount, len(result.Direct.Data))
}
```
### What the example shows
* **One required option.** `WithConnectorRegistry` is the only port `engine.New` insists on. The connection store here is a convenience, but skip it and every operation fails.
* **Construction-time validation.** `engine.New` rejects a port passed as a typed nil, not only a literal nil. A misconfigured host fails at construction instead of panicking at first use.
* **Tenant scope on every call.** `NewTenantContext` builds the only isolation dimension the Engine knows. It carries a tenant ID and an optional request ID — no organization and no product.
* **Plan, then execute.** `PlanExtraction` validates the request against the live schema and applies the limits. `ExecuteExtraction` reads the plan.
* **Mode by composition.** The example wires no `ResultSink`, so the Engine picks direct mode and returns the bytes inline with a SHA-256 digest. Add a sink and the same code returns a storage reference instead. See [Direct mode and Store mode](/en/fetcher/fetcher-engine-overview#direct-mode-and-store-mode).
## Moving to production
***
Swap the memory harness for real adapters. Fetcher's own services are the reference implementation, and their source is public. They bridge the Engine ports to real infrastructure under `pkg/enginecompat`:
| Adapter package | Bridges |
| ----------------------------------- | ------------------------------------------------------------------------------------- |
| `pkg/enginecompat/connectioncompat` | Connection store, connection access, tenant context, and the active-execution checker |
| `pkg/enginecompat/schemacompat` | Schema cache, schema connector, and snapshot building |
| `pkg/enginecompat/datasource` | The datasource drivers behind the connector contract |
| `pkg/enginecompat/tablenorm` | Qualified table-name normalization |
Read how the services wire them:
* **Connection CRUD** — `components/manager/internal/bootstrap/connection_engine.go`
* **Schema discovery and caching** — `components/manager/internal/bootstrap/schema_engine.go`
* **Plan and execute extraction** — `components/worker/internal/bootstrap/extraction_engine.go`
Two rules carry over from the harness to production:
1. **Your adapters own tenant scope.** The Engine passes a tenant context to every port call and enforces the boundary at its own edge. A store that ignores the tenant ID leaks data across tenants, and the Engine cannot catch that for you.
2. **Your adapters own secrets.** The Engine calls `Protect` and `Reveal` and records only the returned key version as metadata. Key derivation, rotation, and storage stay in your host.
## Next steps
***
Every port, its contract, and the behavior you get without it.
The three-layer model, the import boundary, and the two result modes.