engine.New.
This page is the reference for all eight. The Without it column is the point of the page. Graceful degradation is the contract you plan against, so read it before you skip a port.
The ports at a glance
engine.New rejects a port passed as a typed nil, not only a literal nil. An interface value that wraps a nil pointer fails at construction with a clear validation error, instead of panicking at first use.ConnectorRegistry
Required: always. This is the only port the Engine validates unconditionally. The registry resolves a connector factory by datasource type. It performs no I/O, it resolves deterministically by type, and it reports
ok=false for a type nobody registered. Building and connecting a connector happens later, through the factory it returned.
Without it: engine.New returns a validation error with the message connector registry is required. You get no Engine value at all, so extraction is impossible.
CredentialProtector
Required: only with
WithEncryptedPersistence(true).
The protector encrypts and decrypts credential material for a tenant. The Protect call returns the protected bytes plus the key version that protected them. The Reveal call decrypts with a given key version, so your host can resolve a rotated key.
Your host owns key derivation, rotation, and storage. The Engine only calls the seam and records the returned key version as secret-free metadata.
Without it:
- With encrypted persistence on,
engine.Newfails withcredential protector is required when encrypted persistence is enabled. The Engine refuses to construct rather than persist plaintext credentials. - With encrypted persistence off, the port is genuinely optional, and credentials reach the
ConnectionStorewithout protection.
ConnectionStore
Required: optional at construction, load-bearing at runtime. The store persists and resolves connection descriptors owned by a tenant. It is the only persistence seam the connection operations use — the Engine embeds no MongoDB, no SQL, and no host repository. It exposes nine operations: create, find, find-by-id, update, update-by-id, delete, delete-by-id, list, and list-paged. Two obligations fall on your implementation. It must scope every record by the tenant ID, so one tenant never sees another tenant’s connections. It must not return secret material — the connection descriptor carries none. Without it:
engine.New succeeds, and then nearly everything fails. Every operation that touches a connection returns the validation error connection store is not configured. That covers all nine connection operations plus plan, execute, discover-schema, fresh-schema discovery, validate-schema, and test-connection. An Engine without a connection store can report only its limits.
ExecutionStore
Required: optional. The store upserts execution lifecycle state for a tenant. The Engine writes the transitions synchronously and inline:
running, then completed, failed, or canceled.
Those writes are best-effort by design. The Engine discards a save error, so optional persistence can never corrupt an extraction result. A result-sink write failure behaves differently and does fail the execution. The two are deliberately distinct.
Without it: the Engine runs with no durable execution tracking, and your host owns execution state externally.
ResultSink
Required: optional. It selects the result mode. The sink persists result payloads to host-managed storage. Store-mode extraction calls
OpenResultStream, so the Engine writes the result incrementally in constant memory. PersistResult remains for whole-payload writes.
The stream shape is contractual NDJSON — one JSON object per line, newline-terminated, with no enclosing array:
Close. Treat an unclosed writer as a discarded write, because a partial result must never become a returned reference.
Without it: store mode is unavailable. The default auto mode resolves to direct, so extraction returns inline bytes and persists nothing. An explicit store-mode request fails up front with store mode requires a configured result sink, before the Engine builds any connector.
SchemaCache
Required: optional. The cache stores and returns schema snapshots per tenant and config name. The Engine treats it as an accelerator, never as a source of truth. A failed cache read degrades to fresh discovery. A failed cache write still returns the discovered schema to the caller. The always-fresh discovery call ignores the cache on every invocation, even when you wired one, so the live-datasource contract of the Manager schema endpoint holds. Without it: the Engine discovers schema live from the datasource on every call.
ActiveExecutionChecker
Required: optional. The checker reports whether a connection currently has active executions. The Engine consults it before it mutates a connection. A host can therefore keep the Manager behavior, which blocks a change while jobs run against that connection. The port is deliberately logical, not a durable job store. Your host decides how to answer: a job repository, an in-memory tracker, a distributed lock, or always false. The Engine never imports a job repository to ask the question. The connection identity it passes is the config name inside the tenant scope. Your answer must be tenant-scoped, so one tenant’s running work never blocks another tenant’s mutation. Without it: the Engine performs no conflict gating, and connection updates and deletes proceed unconditionally.
Observability
Required: optional. The contract has one method.
StartSpan takes a context and an operation name, and returns a derived context plus an end function the Engine defers. One method is the whole point: the Engine core never imports a tracing library, and your host adapts its own tracer behind the seam.
Without it: span creation returns the incoming context and a no-op end function. Tracing hooks disappear with no other change in behavior.
Next steps
Embed the Engine
Import, provide the ports, and construct with a runnable example.
Engine overview
The three-layer model, the import boundary, and the two result modes.