Skip to main content
This guide takes you from a clean checkout to a running Fetcher environment and a completed extraction job. Everything runs locally with Docker Compose. By the end you will:
  • Have Manager, Worker, and their infrastructure running locally
  • Register and test a database connection
  • Create an extraction job and read its status

​
Prerequisites

  • Docker and Docker Compose
  • Make
  • Go (only for development — the toolchain version is declared in the repo’s go.mod)

​
Set up and run

1

Clone the repository

git clone https://github.com/LerianStudio/fetcher.git
cd fetcher
2

Set up environment files

make set-env
This creates the .env files for each component from their .env.example templates.
3

Generate the master encryption key

make generate-master-key
Copy the generated key and set it as APP_ENC_KEY in both components/manager/.env and components/worker/.env. This key is required — the services will not start without it, and both must use the same value: the Worker needs it to decrypt connection credentials and verify message signatures.
4

Start all services

make up
5

Verify the API is up

  • REST API: http://localhost:4006
  • Swagger UI: http://localhost:4006/swagger/index.html
  • RabbitMQ management: http://localhost:3008

​
Run your first extraction

An extraction has three moves: register a connection, create a job, poll the job until it finishes.

​
1. Register a database connection

curl -X POST http://localhost:4006/v1/management/connections \
  -H "Content-Type: application/json" \
  -d '{
    "configName": "my_postgres",
    "type": "POSTGRESQL",
    "host": "host.docker.internal",
    "port": 5432,
    "databaseName": "mydb",
    "userName": "postgres",
    "password": "postgres"
  }'
The password is encrypted before it is stored. Test the connection before using it: 
curl -X POST http://localhost:4006/v1/management/connections/{id}/test

​
2. Create an extraction job

Tell Fetcher which fields to extract from which tables, per datasource: 
curl -X POST http://localhost:4006/v1/fetcher \
  -H "Content-Type: application/json" \
  -d '{
    "dataRequest": {
      "mappedFields": {
        "my_postgres": {
          "accounts": ["id", "email", "created_at"]
        }
      }
    },
    "metadata": {
      "source": "quickstart"
    }
  }'
The API answers 202 Accepted with a job ID. If an identical job was submitted within the last 5 minutes, Fetcher detects the duplicate and returns the existing job with 200 OK instead of creating a new one.

​
3. Poll the job status

curl http://localhost:4006/v1/fetcher/{id}
A job moves through pending → processing → completed (or failed). On completion, the Worker has encrypted the results into object storage and published a job.completed event.
When authentication is enabled (PLUGIN_AUTH_ENABLED=true), requests also carry an Authorization: Bearer <token> header. The local quickstart runs with auth disabled.

​
Next steps

Core concepts

Understand connections, schema discovery, jobs, filters, and results.

Configuration

The environment variables that shape a Fetcher deployment.