Embedding the Fetcher Engine

Embedding the Fetcher Engine takes three steps: import it, provide the ports it needs, construct it with engine.New. No infrastructure ships with the import. You wire only the parts your host application actually uses.

1. Install

go get github.com/LerianStudio/fetcher/pkg/engine

The Engine is a distinct Go module from the Fetcher services (github.com/LerianStudio/fetcher/v2). It carries zero third-party dependencies and has its own version line with path-prefixed tags (pkg/engine/vX.Y.Z). The import gives your module graph none of the service dependencies.

2. Provide the ports

The Engine depends only on host-provided interfaces. One port is always required. A second is required only when encrypted persistence is on. The rest are opt-in, and the Engine degrades gracefully without them.

Port	Required?	Without it
`ConnectorRegistry`	Always	`engine.New` fails and no Engine exists
`CredentialProtector`	Only with `WithEncryptedPersistence(true)`	Construction fails when encrypted persistence is on; credentials reach the store unprotected when it is off
`ConnectionStore`	Optional	Every operation except `Limits()` fails
`ResultSink`	Optional	Store mode is unavailable, and extraction runs in direct mode
`SchemaCache`	Optional	Schema discovery always hits the live datasource
`ExecutionStore`	Optional	No durable execution-state tracking
`ActiveExecutionChecker`	Optional	No conflict gating on connection updates and deletes
`Observability`	Optional	Tracing hooks become no-ops

The port reference documents each contract and each degradation in full. Read it before you decide which ports to skip — “optional” does not mean “harmless”. For tests and first runs, the pkg/engine/memory harness provides in-memory implementations of the storage-facing ports: the connector registry, the connection store, the schema cache, the result sink, and the execution store. You need no MongoDB, Redis, RabbitMQ, or object storage to exercise the Engine. The harness ships no CredentialProtector, so a test that turns encrypted persistence on must supply one.

3. Construct, plan, execute

This example is self-contained. It uses the in-memory harness, so it runs with zero infrastructure.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/LerianStudio/fetcher/pkg/engine"
	"github.com/LerianStudio/fetcher/pkg/engine/memory"
)

func main() {
	ctx := context.Background()

	// Provide ports. In production these are your real adapters (see pkg/enginecompat);
	// here the in-memory harness stands in so the example runs with zero infrastructure.
	store := memory.NewConnectionStore()
	registry := memory.NewConnectorRegistry()

	// Construct the Engine. WithConnectorRegistry is the only required option.
	eng, err := engine.New(
		engine.WithConnectorRegistry(registry),
		engine.WithConnectionStore(store),
	)
	if err != nil {
		log.Fatal(err)
	}

	// Every operation is scoped to a tenant — the sole isolation dimension.
	tenant, err := engine.NewTenantContext("tenant-123")
	if err != nil {
		log.Fatal(err)
	}

	// Register a connector for the datasource type, then persist a connection.
	conn := memory.NewTemplateConnector(memory.ConnectorBehavior{
		Schema: engine.SchemaSnapshot{
			ConfigName: "pg-main",
			Tables:     []engine.TableSnapshot{{Name: "public.users", Fields: []string{"id", "email"}}},
		},
		Rows: map[string][]map[string]any{
			"public.users": {{"id": 1, "email": "a@example.com"}},
		},
	})
	registry.Register("postgres", memory.NewConnectorFactory(conn))

	if _, err = eng.CreateConnection(ctx, tenant, engine.NewConnectionInput(engine.ConnectionInputParams{
		ConfigName: "pg-main",
		Type:       "postgres",
		Host:       "localhost",
		Port:       5432,
	})); err != nil {
		log.Fatal(err)
	}

	// Plan validates the request against the live schema and enforces limits.
	plan, err := eng.PlanExtraction(ctx, tenant, engine.ExtractionRequest{
		MappedFields: map[string]engine.FieldSelection{
			"pg-main": {"public.users": {"id", "email"}},
		},
	})
	if err != nil {
		log.Fatal(err)
	}

	// Execute. With no ResultSink wired, the Engine runs in Direct mode and returns
	// inline JSON bytes plus a SHA-256 integrity digest.
	result, err := eng.ExecuteExtraction(ctx, plan)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("rows=%d bytes=%d\n", result.Direct.RowCount, len(result.Direct.Data))
}

What the example shows

One required option. WithConnectorRegistry is the only port engine.New insists on. The connection store here is a convenience, but skip it and every operation fails.
Construction-time validation. engine.New rejects a port passed as a typed nil, not only a literal nil. A misconfigured host fails at construction instead of panicking at first use.
Tenant scope on every call. NewTenantContext builds the only isolation dimension the Engine knows. It carries a tenant ID and an optional request ID — no organization and no product.
Plan, then execute. PlanExtraction validates the request against the live schema and applies the limits. ExecuteExtraction reads the plan.
Mode by composition. The example wires no ResultSink, so the Engine picks direct mode and returns the bytes inline with a SHA-256 digest. Add a sink and the same code returns a storage reference instead. See Direct mode and Store mode.

Moving to production

Swap the memory harness for real adapters. Fetcher’s own services are the reference implementation, and their source is public. They bridge the Engine ports to real infrastructure under pkg/enginecompat:

Adapter package	Bridges
`pkg/enginecompat/connectioncompat`	Connection store, connection access, tenant context, and the active-execution checker
`pkg/enginecompat/schemacompat`	Schema cache, schema connector, and snapshot building
`pkg/enginecompat/datasource`	The datasource drivers behind the connector contract
`pkg/enginecompat/tablenorm`	Qualified table-name normalization

Read how the services wire them:

Connection CRUD — components/manager/internal/bootstrap/connection_engine.go
Schema discovery and caching — components/manager/internal/bootstrap/schema_engine.go
Plan and execute extraction — components/worker/internal/bootstrap/extraction_engine.go

Two rules carry over from the harness to production:

Your adapters own tenant scope. The Engine passes a tenant context to every port call and enforces the boundary at its own edge. A store that ignores the tenant ID leaks data across tenants, and the Engine cannot catch that for you.
Your adapters own secrets. The Engine calls Protect and Reveal and records only the returned key version as metadata. Key derivation, rotation, and storage stay in your host.

Overview

Getting started

Concepts

Engine

Integration

Operations

Embedding the Fetcher Engine

1. Install

2. Provide the ports

3. Construct, plan, execute

What the example shows

Moving to production

Next steps

Port reference

Engine overview

​1. Install

​2. Provide the ports

​3. Construct, plan, execute

​What the example shows

​Moving to production

​Next steps

Port reference

Engine overview

1. Install

2. Provide the ports

3. Construct, plan, execute

What the example shows

Moving to production

Next steps