> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lerian.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# Discovery

> Use Discovery to detect external data sources, inspect their schemas, and pull transactions into Matcher automatically.

Discovery automates data source detection and extraction through Matcher's embedded extraction engine. Instead of manually uploading files, Discovery connects to external systems, identifies available data, and extracts transactions directly into Matcher.

## What Discovery solves

***

Manual file uploads create friction at every step. Teams export files, transfer them, monitor for failures, and re-upload when something goes wrong. This process is time-consuming, error-prone, and breaks when data volume grows.

Discovery replaces the manual pipeline. It connects to external systems through the extraction engine, detects available data sources automatically, and pulls transactions into Matcher on demand. When a new data source appears (a new bank connection, a new payment processor), Discovery finds it without reconfiguration.

## How Discovery works

***

Discovery runs inside Matcher. There is no separate extraction service to deploy. The embedded engine manages connections to external databases and runs extractions locally. Discovery exposes those connections and coordinates the extraction process, handing results directly to Ingestion.

The workflow has seven steps:

1. **Check status**: Confirm Discovery and its embedded engine are available.
2. **Browse connections**: See all data sources the embedded engine has access to.
3. **Inspect a connection**: Review the schema to understand what fields are available.
4. **Test a connection**: Validate the connection before committing to an extraction.
5. **Create an extraction**: Request that Matcher pull data from a specific source.
6. **Monitor progress**: Track extraction status as data flows in.
7. **Refresh connections**: Rescan when new data sources appear.

## Discovery workflow

***

### Check Discovery status

Before you start, verify that Discovery and the embedded extraction engine are operational.

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/status" \
  -H "Authorization: Bearer $TOKEN"
```

<Tip>
  API Reference: [Get Discovery status](/en/reference/products/matcher/discovery-status)
</Tip>

### Browse connections

List all data sources available through the embedded extraction engine.

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/connections" \
  -H "Authorization: Bearer $TOKEN"
```

The response lists each connection with its name, type (database, API, file store), and current status.

<Tip>
  API Reference: [List connections](/en/reference/products/matcher/list-discovery-connections)
</Tip>

### Get a connection

Retrieve a single discovered connection by its internal identifier:

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/connections/{connectionId}" \
  -H "Authorization: Bearer $TOKEN"
```

`GET /v1/discovery/connections/{connectionId}` returns the full `ConnectionResponse` (name, type, status, and metadata) for one connection. Use it when you already hold a `connectionId`, for example from a source binding's query rail. It gives the current details without a list of every connection.

<Tip>
  API Reference: [Get discovery connection](/en/reference/products/matcher/retrieve-discovery-connection)
</Tip>

### Inspect a connection

Before you extract, review the schema of a specific connection to understand what data fields are available.

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/connections/{connectionId}/schema" \
  -H "Authorization: Bearer $TOKEN"
```

Use schema inspection to confirm that required fields (transaction IDs, amounts, dates, references) exist before building field mappings.

<Tip>
  API Reference: [Get connection schema](/en/reference/products/matcher/get-connection-schema)
</Tip>

### Test a connection

Validate that Matcher can reach and read from a connection before creating an extraction.

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/connections/{connectionId}/test" \
  -H "Authorization: Bearer $TOKEN"
```

A successful test confirms connectivity and read access. Always test before creating an extraction, especially for new or recently modified connections.

<Tip>
  API Reference: [Test connection](/en/reference/products/matcher/test-discovery-connection)
</Tip>

### Create an extraction

Request that Matcher pull transaction data from a specific connection into the current context.

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/connections/{connectionId}/extractions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tables": {
      "transactions": {}
    },
    "startDate": "2026-06-01",
    "endDate": "2026-06-30"
  }'
```

The response returns an extraction ID. Use it to monitor progress.

<Tip>
  API Reference: [Create extraction](/en/reference/products/matcher/create-extraction)
</Tip>

### Monitor extraction progress

Track the status of an active extraction by polling its status with `GET`.

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/extractions/{extractionId}" \
  -H "Authorization: Bearer $TOKEN"
```

Extraction status transitions from `PENDING` → `SUBMITTED` → `EXTRACTING` → `COMPLETE` (or `FAILED`/`CANCELLED`). The response carries the extraction `status`, an `errorMessage` when it failed, and the linked `ingestionJobId` once the extraction bridges into ingestion.

<Tip>
  API Reference: [Get extraction](/en/reference/products/matcher/retrieve-extraction)
</Tip>

### Refresh available connections

When you register a new data source with the embedded engine, trigger a refresh so Discovery picks it up.

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/refresh" \
  -H "Authorization: Bearer $TOKEN"
```

<Tip>
  API Reference: [Refresh connections](/en/reference/products/matcher/refresh-discovery)
</Tip>

### List connector types

List the connector (datasource) types the engine registry has registered for this deployment. Each entry carries a backend-derived `category` (`database` or `rest`). The registry is live. Only connectors registered at boot appear. This list excludes aggregator vendors (Pluggy/Belvo). Provision those through the aggregator-connections surface below.

```bash theme={null}
curl -X GET "https://api.matcher.example.com/v1/discovery/connector-types" \
  -H "Authorization: Bearer $TOKEN"
```

#### Response

```json theme={null}
{
  "types": [
    { "type": "POSTGRESQL", "category": "database" },
    { "type": "MYSQL", "category": "database" }
  ]
}
```

## Aggregator connections (Open Finance)

***

Open-Finance data-aggregator connections (Pluggy or Belvo) let Matcher pull transactions from bank aggregators. Credential material (`clientId`/`secret`) is **sealed on write and never returned**. Every read is secret-free by construction.

### Create an aggregator connection

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/aggregator-connections" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "vendor": "pluggy",
    "configName": "pluggy-main",
    "baseUrl": "https://api.pluggy.ai",
    "accountRef": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
    "clientId": "...",
    "secret": "..."
  }'
```

Send five required fields: `vendor`, `configName`, `baseUrl`, `clientId`, and `secret`. The `accountRef` field is optional. Omit it to create a connection awaiting the end customer's consent in the vendor-hosted flow. Then bind the returned item or link id with `PUT`.

The `vendor` field is one of `pluggy` or `belvo`. The `configName` field is the tenant-scoped name the webhook token-mint endpoint binds to. A successful create returns **201** with a secret-free connection.

#### Response

```json theme={null}
{
  "vendor": "pluggy",
  "configName": "pluggy-main",
  "baseUrl": "https://api.pluggy.ai",
  "accountRef": "a1b2c3d4-5678-90ab-cdef-1234567890ab"
}
```

### List, get, update, and delete

```bash theme={null}
# List (cursor-paginated, secret-free)
curl -X GET "https://api.matcher.example.com/v1/discovery/aggregator-connections" \
  -H "Authorization: Bearer $TOKEN"

# Get one by id
curl -X GET "https://api.matcher.example.com/v1/discovery/aggregator-connections/{id}" \
  -H "Authorization: Bearer $TOKEN"

# Update (PUT). vendor is immutable. Supply clientId+secret together to rotate
# the sealed credential, or omit both to keep the stored secret intact.
curl -X PUT "https://api.matcher.example.com/v1/discovery/aggregator-connections/{id}" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "configName": "pluggy-main",
    "baseUrl": "https://api.pluggy.ai",
    "accountRef": "a1b2c3d4-5678-90ab-cdef-1234567890ab"
  }'

# Delete (soft-delete; frees the config name for reuse). Returns 204.
curl -X DELETE "https://api.matcher.example.com/v1/discovery/aggregator-connections/{id}" \
  -H "Authorization: Bearer $TOKEN"
```

### Test an aggregator connection

Run a live connectivity check against an existing, bound connection's already-sealed credential, addressed by `configName`. Matcher reads the vendor from the stored connection. This call takes no credential and returns none.

Invalid stored credentials give an expected test result: `200` with `"healthy": false`, not an error. A missing connection, an unbound connection, or an existing vendor without a connectivity-test path (Belvo today) surfaces through the standard error response. No test runs. Use the list response's `testable` field before offering the action.

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/aggregator-connections/test" \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "configName": "pluggy-main"
  }'
```

#### Response

```json theme={null}
{
  "vendor": "pluggy",
  "configName": "pluggy-main",
  "healthy": true
}
```

## Aggregator webhook tokens

***

Aggregators push data change signals to Matcher via webhooks. Mint an opaque token bound to an aggregator connection, then configure the returned URL in the vendor dashboard.

### Mint a webhook token

Matcher returns the raw token and its provider-facing URL **exactly once**. Matcher stores only the token's SHA-256 hash.

```bash theme={null}
curl -X POST "https://api.matcher.example.com/v1/discovery/webhooks/tokens" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "vendor": "pluggy",
    "connection_config_name": "pluggy-main"
  }'
```

#### Response

```json theme={null}
{
  "vendor": "pluggy",
  "token": "<raw-token-shown-once>",
  "webhook_url": "https://api.matcher.example.com/v1/discovery/webhooks/pluggy/<raw-token>"
}
```

### Receiving webhooks

The vendor calls `POST /v1/discovery/webhooks/{provider}/{webhookToken}` (no operator JWT). Two layers authenticate it: the opaque path token **plus** a per-provider source check. That check is a valid HMAC-SHA256 of the raw body in the `X-Webhook-Signature` header, **or** membership in the provider's source-IP allowlist. Both layers fail closed.

A valid first delivery returns **202 Accepted**. Matcher then pulls the signaled data asynchronously into the ingestion pipeline. A replay of an already-processed event returns **200 OK**.

## Best practices

***

<AccordionGroup>
  <Accordion title="Always test connections before extracting">
    A failed extraction mid-run is harder to recover from than a failed test. Test every connection before creating an extraction, especially when connecting to a new source or after a credential rotation.
  </Accordion>

  <Accordion title="Inspect schemas before mapping fields">
    Field names vary across systems. A bank might call the transaction date `value_date` while your ledger uses `posting_date`. Check the schema before configuring field mappings to avoid silent mismatches.
  </Accordion>

  <Accordion title="Monitor extractions actively for large datasets">
    Large extractions take time. Don't assume completion. Poll the extraction status and confirm the record count before starting a match run. Starting a run on incomplete data generates incorrect exceptions.
  </Accordion>

  <Accordion title="Refresh connections when sources change">
    Discovery doesn't scan for new connections automatically. When you add a new payment processor, or register a new database with the embedded engine, trigger a refresh. Otherwise, Discovery won't show the new source.
  </Accordion>

  <Accordion title="Scope extractions to the reconciliation period">
    Use date range parameters to extract only the data relevant to the current reconciliation period. Extracting unbounded data increases processing time and may pull records that belong to already-closed contexts.
  </Accordion>
</AccordionGroup>

## Next steps

***

<Card title="External sources" icon="building-columns" href="/en/products/matcher/integrations/matcher-external-sources" horizontal>
  Configure the external data sources that Discovery connects to.
</Card>

<Card title="Field mapping" icon="arrows-left-right" href="/en/products/matcher/configuration/matcher-field-mapping" horizontal>
  Map fields from extracted data to Matcher's transaction model.
</Card>

<Card title="Discovery API reference" icon="code" href="/en/reference/products/matcher/discovery-status" horizontal>
  Full API reference for Discovery endpoints.
</Card>