Skip to main content
Fetcher’s model is small: you register connections to external databases, Fetcher discovers their schemas, and you submit extraction jobs that produce signed, encrypted results. This page walks through each piece.

​
Connections

A connection is a stored, named reference to an external database — its type, host, port, database name, and credentials. Fetcher supports five connection types: POSTGRESQL, MYSQL, ORACLE, SQL_SERVER, and MONGODB.
  • Credentials are encrypted at rest with AES-256-GCM before they reach storage; plaintext passwords never persist.
  • Connections are testable: a dedicated endpoint opens a real connection and reports latency, so you can validate a connection before any job depends on it.
  • Connections are protected while in use: updating or deleting a connection with active jobs is rejected.
Besides connections registered through the API, operators can define internal datasources directly via environment variables (DATASOURCE_{NAME}_*) — useful for fixed, platform-owned databases. See Configuration.

​
Schema discovery

Fetcher automatically detects tables, columns, and data types on the databases it connects to — the same discovery works across all five database types. For relational databases it reads the catalog, including multi-schema namespaces (PostgreSQL schemas, Oracle owners, SQL Server schemas). For MongoDB, where collections have no declared schema, Fetcher infers one by statistical sampling of documents. Two things build on discovery:
  • Schema validation: before running a job, you can validate that the tables and fields you plan to extract actually exist in the datasources.
  • Schema caching: discovered schemas are cached (with a configurable TTL) so repeated jobs don’t re-introspect the database every time.

​
Extraction jobs

A job is an asynchronous request to extract data. Its heart is the mappedFields map — which fields, from which tables, from which datasources: 
{
  "dataRequest": {
    "mappedFields": {
      "my_postgres": {
        "accounts": ["id", "email", "created_at"]
      },
      "my_mongo": {
        "transactions": ["*"]
      }
    }
  }
}
  • Field projection: name the fields you want, or use ["*"] for all fields.
  • Multi-everything: a single job can span multiple datasources, multiple tables per datasource, and multiple schemas.
  • JSON handling: JSON/JSONB fields in relational databases are parsed automatically.

​
Filters

Jobs can filter rows per table with a set of 10 operators: eq, ne, gt, gte, lt, lte, between, in, nin, and like. 
{
  "dataRequest": {
    "filters": {
      "my_postgres": {
        "transactions": {
          "status": { "in": ["completed", "pending"] }
        }
      }
    }
  }
}

​
Job lifecycle

StatusMeaning
pendingAccepted and queued; not yet picked up by a Worker
processingA Worker is extracting the data
completedResults are stored and the completion event is published
failedExtraction failed; the failure event is published
Jobs are deduplicated: an identical request within a 5-minute window returns the existing job instead of creating a new one. On terminal states, the Worker publishes job.completed / job.failed notification events so consumers can react without polling.

​
Results

The Worker writes extraction results to object storage — SeaweedFS by default, or any S3-compatible service:
  • Encrypted at rest, with a configurable retention TTL.
  • Signed: each extracted document carries an HMAC-SHA256 signature. Consumers can derive the verification key from the master key and check the authenticity of what they read — the data provably came from Fetcher and was not tampered with.

​
Services and the Engine

Two deployment shapes share one core:
  • Standalone services — the Manager (HTTP API, metadata in MongoDB, job dispatch via RabbitMQ) and the Worker (queue consumer, extraction, result storage). This is the shape the getting started guide runs.
  • Embedded Engine — the extraction core (pkg/engine) as an importable Go module with zero third-party dependencies. Host applications wire their own storage and infrastructure to the Engine’s interfaces and run extractions in-process. Without a result store, the Engine runs in Direct mode: results come back inline with a SHA-256 integrity digest.
The rule of thumb: the Engine owns what extraction means — planning, validation, limits, tenant safety; the host (Manager/Worker, or your application) owns how it runs — queues, storage, auth, lifecycle.

​
Tenancy

Fetcher runs single-tenant by default. In multi-tenant mode, every operation is scoped to a tenant resolved from JWT-based tenant context, and each tenant gets isolated metadata databases, cache namespaces, and object-storage paths. Multi-tenant code paths add zero overhead when the mode is disabled. See Multi-tenancy for the platform-wide model.

​
Next steps

Getting started

Run Fetcher locally and execute your first extraction job.

Configuration

The environment variables that shape a Fetcher deployment.